If we have an expected checksum, call `fstatat(repo_dfd, checksum)`
early on before we do much else. This actually duplicates code,
but future work here is going to split up the metadata/content
commit paths, so they'll need to diverge anyways.
Closes: #881
Approved by: jlebon
First, the streaming metadata API is pretty dumb, since metadata
should be small. Really we should have supported a `GBytes`
version. Currently, this API *is* used when we do local pulls,
so this commit has test coverage. However, I plan to change
the object import to avoid using this. But that's fine, since
I can't think of why someone would use this API.
Next, the only difference between `ostree_repo_write_metadata()` and
`ostree_repo_write_metadata_trusted()` is whether or not we pass
an output checksum; so just dedup the implementations.
Also while I'm here break out the input length validation and do
it early in the streaming case.
Closes: #881
Approved by: jlebon
Using `${sysroot}` to mean the physical storage root: We don't want to write to
`${sysroot}/etc/ostree/remotes.d`, since nothing will read it, and really
`${sysroot}` should just have `/ostree` (ideally). Today the Anaconda rpmostree
code ends up writing there. Fix this by adding a notion of "physical" sysroot.
We determine whether the path is physical by checking for `/sysroot`, which
exists in deployment roots (and there shouldn't be a `${sysroot}/sysroot`).
In order to unit test this, I added a `--sysroot` argument to `remote add`.
However, doing this better would require reworking the command line parsing for
the `remote` argument to support specifying `--repo` or `--sysroot`, and I
didn't quite want to do that yet in this patch.
Closes: https://github.com/ostreedev/ostree/issues/892Closes: #896
Approved by: jlebon
Having a failable accessor is annoying, since it's really common
to reference both. Instead, open the repo once when we load
the sysroot, and provide a non-failable accessor.
This is also prep for `ostree_repo_open_at()`, which collapses the separation
between `ostree_repo_new()` and `ostree_repo_open()`.
Closes: #886
Approved by: jlebon
This is prep for introducing a fd-relative `ostree_repo_new_at()`.
Previously, `ostree_repo_is_system()` compared `GFile` paths, but
there's a much simpler check we can do first - if this repository
was created via `OstreeSysroot`, it must be a system repo.
Closes: #886
Approved by: jlebon
This is a de-scoping of work I did in preparation for
rpm-ostree [live updates](https://github.com/projectatomic/rpm-ostree/pull/652).
Originally I was going to expose this as a public API.
However, I decided to do things differently, but the cleanup here for new code
style and fd-relative is nice to have anyways.
We rework things to use `OstreeDeployment*`, which the caller is expected to
already have, rather than `GFile*`s pointing to the config directories.
Closes: #741
Approved by: jlebon
The summary file can get large, but it compresses well (something
which is not true of other files in the ostree repo which are
already compressed). By sending Accept-Encoding: gzip (and
handling the compressed results) we send a lot less data.
I set up the flathub repo (http://flathub.org/repo) to enable
gzip for the summary file (only), and the result is that the
331514 byte large summary was transferred in 122889 bytes.
On my (fast) network this decreased the time i took to do
"flatpak remote-ls flathub" by about 100msec.
This fixes https://github.com/ostreedev/ostree/issues/802Closes: #882
Approved by: cgwalters
It's hard right now to do a full port to the new libglnx tmpfile
API since there are complex cases in the commit path which deal
with symlinks as well.
Let's make things more gradual by introducing the important part (struct with
autocleanup) here in libotutil, port what we can. This will make a future
complete port easier.
Closes: #871
Approved by: jlebon
A commit can now include a "ostree.endoflife-rebase" metadata key
pointing to a new ref.
When updating, the sysroot upgrader will see this and proceed to
pull and deploy the new ref instead. The origin file in the new
deployment will point to the new ref.
This functionality is planned to be used in Endless OS. We will create
a lesser tested branch for brand new, cutting edge hardware support,
and ship that on hardware platforms that require the latest drivers.
However, once our slower-moving official release is later updated to
support the new hardware, we will use this functionality to migrate
those bleeding-edge users over to the official release.
Closes: #874
Approved by: cgwalters
The whole ostree-remote.h file is only included in the public ostree.h
header if OSTREE_ENABLE_EXPERIMENTAL_API is defined, so there’s no need
to change the set of methods defined in it according to whether we’re
compiling with experimental API.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #875
Approved by: cgwalters
Make it an internal, not static, API; like _ostree_repo_add_remote(). It
will be used in many the same situations.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #875
Approved by: cgwalters
Return whether the remote already existed. This is an internal API, so
it’s not an API break. The return value will be useful in upcoming
commits for working out whether to later remove a remote again.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #875
Approved by: cgwalters
Add a name argument to the internal OstreeRemote constructor,
since this member (and several derived from it) is non-nullable,
and hence must always be set at construction time.
This changes the only call sites of the constructor to use the new API,
which is internal.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #875
Approved by: cgwalters
Just for internal documentation; g-ir-scanner doesn’t read or understand
them.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #875
Approved by: cgwalters
Now that we’ve got a public, sealed OstreeRemote structure, we can start
carefully exposing members of it as API.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #875
Approved by: cgwalters
If somehow a repo has gpg verification on but doesn't have signatures
present for the existing commit, ostree would error out if it needs to
scan the commit object (e.g. if there are no updates available).
An instance of this is currently happening in Fedora AH, in which
signatures are not shipped in the ISO due to filesystem restrictions.
Another possible scenario is if a content provider switches from not
signing commits to signing them; even if older commits are retroactively
signed, clients' local commit objects would error out if they needed
scanning.
This patch adds a check to ensure that we always attempt to fetch the
detached metadata and wait for its result (whether it exists or not)
before moving on to scan their corresponding commit objects.
See also: https://github.com/projectatomic/rpm-ostree/issues/630Closes: #873
Approved by: cgwalters
Porting a lot of this file would be hard since in many cases we do processing in
the `out:` section, so let's do what we can.
Closes: #870
Approved by: jlebon
Unbreaks mounting in CentOS. Newer systemd in Fedora pulls didn't need this, I
think due to `RequiresMountsFor=`. Anyways, this is what the fstab generator
does, and it's clearly right ✓.
Closes: https://github.com/ostreedev/ostree/issues/867Closes: #869
Approved by: jlebon
Follow up to a previous patch that addressed a double-close; I
realized we already had a helper for doing "open dfd iter, do nothing
if we get ENOENT". Raise it to libotuil, and port all consumers.
Closes: #863
Approved by: jlebon
I noticed an instance of this while working on https://github.com/ostreedev/ostree/pull/861
Which apparently I cargo-culted into the new system generator bits.
Let's break this out as a small concise change.
Closes: #866
Approved by: jlebon
If one wants to set up a mount for `/var` in `/etc/fstab`, it
won't be mounted since `ostree-prepare-root` set up a bind mount for
`/var` to `/sysroot/ostree/$stateroot/var`, and systemd will take
the already extant mount over what's in `/etc/fstab`.
There are a few options to fix this, but what I settled on is parsing
`/etc/fstab` in a generator (exactly like `systemd-fstab-generator` does),
except here we look for an explicit mount for `/var`, and if one *isn't* found,
synthesize the default ostree mount to the stateroot. Another nice property is
that if an admin creates a `var.mount` unit in `/etc` for example, that will
also override our mount.
Note that today ostree doesn't hard depend on systemd, so this behavior only
kicks in if we're built with systemd *and* libmount support (for parsing
`/etc/fstab`). I didn't really test that case though.
Initially I started writing this as a "pure libc" program, but at one point
decided to use `libostree.so` to find the booted deployment. That didn't work
out because `/boot` wasn't necessarily mounted and hence we couldn't find the
bootloader config. A leftover artifact from this is that the generator code
calls into libostree via the "cmd private" infrastructure. But it's an easy way
to share code, and doesn't hurt.
Closes: #859
Approved by: jlebon
This is what caused the merge of
https://github.com/projectatomic/rpm-ostree/pull/652
to blow up, since https://github.com/ostreedev/ostree/pull/848
landed right before we tried to merge it.
When I was writing that PR I remember having an uncertain feeling
since we were doing a `mkdirat` above, but at the time I thought
we'd have test suite coverage...turns out we didn't.
For backwards compatibility, we need to continue to do a `mkdirat` here of the
parent. However...I can't think of a reason anyone would *want* that behavior.
Hence, let's add a special trick - if the destination name is `.`, we skip
`mkdirat()`. That way rpm-ostree for example can open a dfd for `/etc` and avoid
the `mkdir`.
Fold the subpath tests into `test-basic.sh` since it's not worth a separate
file. Add a test case for checking out a file.
Closes: #854
Approved by: jlebon
Looking at `perf record ostree checkout`, some things stand out; e.g.:
```
+ 27.63% 0.07% ostree libgio-2.0.so.0.5000.3 [.] g_file_enumerator_iterate
+ 22.74% 0.28% ostree libostree-1.so.1.0.0 [.] ostree_repo_file_tree_query_child
+ 13.74% 0.08% ostree libostree-1.so.1.0.0 [.] ot_variant_bsearch_str
```
The GIO abstractions are already fairly heavyweight, and `OstreeRepoFile` mallocs
a lot too.
Make things more efficient here by dropping the GIO bits for reading ostree data -
we just read from the variants directly and iterate over them. The end result
here is that according to perf we go from ~40% of our time in the kernel to
~70%, and things like `g_file_enumerator_iterate()` drop entirely out of the
hot set.
Closes: #848
Approved by: jlebon
Since we now have a cleaner separation of "toplevel checkout prep"
versus "recursive checkout", handle the special case of checking out
a single file at first rather than later.
Prep for future work in optimizing this function more.
Closes: #848
Approved by: jlebon
Only non-mechanical bit here was creating a local autoptr for a bit
where we'd previously done an unref for a struct member.
Closes: #847
Approved by: jlebon
Rather than `g_output_stream_splice()`, where the input is a regular
file.
See https://github.com/GNOME/libglnx/pull/44 for some more information.
I didn't try to measure the performance difference, but seeing the
read()/write() to/from userspace mixed in with the pointless `poll()` annoyed me
when reading strace.
As a bonus, we will again start using reflinks (if available) for `/etc`,
which is a regression from the https://github.com/ostreedev/ostree/pull/797
changes (which before used `glnx_file_copy_at()`).
Also, for the first time we'll use reflinks when doing commits from file-backed
content. This happens in `rpm-ostree compose tree` today for example.
Update submodule: libglnx
Closes: #817
Approved by: jlebon
Continuing to chip away at this. Using `g_file_enumerator_iterate()`
here helps notably.
I started on the much bigger `ostree_diff_dirs_with_options()` but
it's a lot messier - for later.
Closes: #844
Approved by: jlebon
Previously it was static to ostree-repo.c. Make it usable throughout
libostree so it can be used by an upcoming commit, but also expose the
typedef and reference counting functions so that opaque OstreeRemote
pointers can be used by user code, in anticipation of exposing more of
its API publicly in future.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #832
Approved by: cgwalters
This allows consumers of libostree to check at configure time whether it
supports the feature they want.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #832
Approved by: cgwalters
There are currently no unstable APIs, but some will be added in
following commits. They will be built and exposed in the libostree
global symbol list iff configured with --enable-experimental-api.
Distributions should not package OSTree with --enable-experimental-api.
This is designed for previewing new APIs on controlled platforms; any of
the APIs hidden behind this option may be changed or removed at any
point.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #832
Approved by: cgwalters
Read the http-headers (a(ss)) option in
ostree_repo_remote_fetch_summary_with_options like
ostree_repo_pull_with_options and add the headers to the fetcher. This
allows things like providing additional authorization headers to the
HTTP requests.
Closes: #839
Approved by: cgwalters
Allow GI bindings to delete refs through ostree_repo_transaction_set_ref
and ostree_repo_transaction_set_refspec by setting the checksum to NULL.
Closes: #834
Approved by: cgwalters
• Commit timestamps, so it’s easy to work out whether a given commit is
newer than the one we have locally
• Summary file timestamp, so it’s easy to work out whether the summary
file is more up to date than another summary file
• Summary file expiry time, so clients can work out when they should
expect the summary file to next be updated, and hence can query for
it at roughly the right time
The expiry time requires input from the user, so is currently never set
automatically. Programs using libostree can set it if they wish.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #826
Approved by: cgwalters
While running the testsuite under valgrind a small memory leak showed up:
==16487== 65 bytes in 1 blocks are definitely lost in loss record 773 of 1,123
==16487== at 0x4C2BBAF: malloc (vg_replace_malloc.c:299)
==16487== by 0x6048E08: g_malloc (gmem.c:94)
==16487== by 0x6062EAE: g_strdup (gstrfuncs.c:363)
==16487== by 0x54CE3E6: write_object (ostree-repo-commit.c:776)
==16487== by 0x54CF2D4: ostree_repo_write_metadata (ostree-repo-commit.c:1528)
==16487== by 0x54CF505: _ostree_repo_write_directory_meta (ostree-repo-commit.c:1712)
==16487== by 0x54D0AB4: write_dfd_iter_to_mtree_internal (ostree-repo-commit.c:2650)
==16487== by 0x54D0E2D: ostree_repo_write_dfd_to_mtree (ostree-repo-commit.c:2793)
==16487== by 0x1190C4: ostree_builtin_commit (ot-builtin-commit.c:474)
==16487== by 0x11F2EE: ostree_run (ot-main.c:200)
==16487== by 0x116F32: main (main.c:78)
The reason for this is that ot_checksum_instream_get_string returns a chunk of newly allocated memory which never got freed.
Make actual_checksum something that gets autocleanend and own the memory
assigned to it in all cases.
Signed-off-by: Sjoerd Simons <sjoerd.simons@collabora.co.uk>
Closes: #827
Approved by: pwithnall
If we're freeing the segment, it's basically always better to use
`autoptr()`. Fewer lines, more reliable, etc.
Noticed an instance of this in the pull code while reviewing a different PR,
decided to do a grep for it and fix it tree wide.
Closes: #836
Approved by: pwithnall
Colin Walters
Alexander Larsson
Daniel Drake
Philip Withnall
Jonathan Lebon
Dan Nicholson
Sjoerd Simons