Just for internal documentation; g-ir-scanner doesn’t read or understand
them.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #875
Approved by: cgwalters
Now that we’ve got a public, sealed OstreeRemote structure, we can start
carefully exposing members of it as API.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #875
Approved by: cgwalters
If somehow a repo has gpg verification on but doesn't have signatures
present for the existing commit, ostree would error out if it needs to
scan the commit object (e.g. if there are no updates available).
An instance of this is currently happening in Fedora AH, in which
signatures are not shipped in the ISO due to filesystem restrictions.
Another possible scenario is if a content provider switches from not
signing commits to signing them; even if older commits are retroactively
signed, clients' local commit objects would error out if they needed
scanning.
This patch adds a check to ensure that we always attempt to fetch the
detached metadata and wait for its result (whether it exists or not)
before moving on to scan their corresponding commit objects.
See also: https://github.com/projectatomic/rpm-ostree/issues/630Closes: #873
Approved by: cgwalters
Porting a lot of this file would be hard since in many cases we do processing in
the `out:` section, so let's do what we can.
Closes: #870
Approved by: jlebon
Unbreaks mounting in CentOS. Newer systemd in Fedora pulls didn't need this, I
think due to `RequiresMountsFor=`. Anyways, this is what the fstab generator
does, and it's clearly right ✓.
Closes: https://github.com/ostreedev/ostree/issues/867Closes: #869
Approved by: jlebon
Follow up to a previous patch that addressed a double-close; I
realized we already had a helper for doing "open dfd iter, do nothing
if we get ENOENT". Raise it to libotuil, and port all consumers.
Closes: #863
Approved by: jlebon
I noticed an instance of this while working on https://github.com/ostreedev/ostree/pull/861
Which apparently I cargo-culted into the new system generator bits.
Let's break this out as a small concise change.
Closes: #866
Approved by: jlebon
If one wants to set up a mount for `/var` in `/etc/fstab`, it
won't be mounted since `ostree-prepare-root` set up a bind mount for
`/var` to `/sysroot/ostree/$stateroot/var`, and systemd will take
the already extant mount over what's in `/etc/fstab`.
There are a few options to fix this, but what I settled on is parsing
`/etc/fstab` in a generator (exactly like `systemd-fstab-generator` does),
except here we look for an explicit mount for `/var`, and if one *isn't* found,
synthesize the default ostree mount to the stateroot. Another nice property is
that if an admin creates a `var.mount` unit in `/etc` for example, that will
also override our mount.
Note that today ostree doesn't hard depend on systemd, so this behavior only
kicks in if we're built with systemd *and* libmount support (for parsing
`/etc/fstab`). I didn't really test that case though.
Initially I started writing this as a "pure libc" program, but at one point
decided to use `libostree.so` to find the booted deployment. That didn't work
out because `/boot` wasn't necessarily mounted and hence we couldn't find the
bootloader config. A leftover artifact from this is that the generator code
calls into libostree via the "cmd private" infrastructure. But it's an easy way
to share code, and doesn't hurt.
Closes: #859
Approved by: jlebon
By checking the mount status, we avoid remounting things if we don't
need to. And printing a single line per mount helps debugging when
things go wrong.
Closes: #859
Approved by: jlebon
I really have no idea what I was thinking with that list of mount points. It
seems arbitrary. Sadly `git log` doesn't help, and there's no comments.
Basically, the only mounts we should care about are those that libostree
creates. Which are just `/sysroot` and `/var`. Systemd will handle the other
things like `/tmp`, it's not our job, and we shouldn't touch them.
Closes: #859
Approved by: jlebon
In current systemd, there is:
[systemd-volatile-root](https://www.freedesktop.org/software/systemd/man/systemd-volatile-root.service.html)
which was introduced by [this commit](91214a37ef).
I'd like to make further changes to how we handle `/var`, and I don't
want to reason about the interaction of our "tmpfs var" with too many
other things.
The comment about having "all /var handling in one place" was always inaccurate
given that we rely on systemd for mounting. And in general, I don't want to
duplicate too many things systemd does - it does them well, documents them, etc.
As far as I know, it was basically just Owen who was using this for the GNOME
hardware testing effort, and I'm sure he could easily switch over to
`systemd.volatile=state`.
Closes: #856
Approved by: owtaylor
This is what caused the merge of
https://github.com/projectatomic/rpm-ostree/pull/652
to blow up, since https://github.com/ostreedev/ostree/pull/848
landed right before we tried to merge it.
When I was writing that PR I remember having an uncertain feeling
since we were doing a `mkdirat` above, but at the time I thought
we'd have test suite coverage...turns out we didn't.
For backwards compatibility, we need to continue to do a `mkdirat` here of the
parent. However...I can't think of a reason anyone would *want* that behavior.
Hence, let's add a special trick - if the destination name is `.`, we skip
`mkdirat()`. That way rpm-ostree for example can open a dfd for `/etc` and avoid
the `mkdir`.
Fold the subpath tests into `test-basic.sh` since it's not worth a separate
file. Add a test case for checking out a file.
Closes: #854
Approved by: jlebon
As $DEITY intended.
I was reading the `prepare-root.c` code and the indentation damage was
distracting. Squash tabs that have leaked into various places in the code. I
didn't yet touch the `src/libostree` bits as that has higher potential for
conflict.
Closes: #852
Approved by: jlebon
Looking at `perf record ostree checkout`, some things stand out; e.g.:
```
+ 27.63% 0.07% ostree libgio-2.0.so.0.5000.3 [.] g_file_enumerator_iterate
+ 22.74% 0.28% ostree libostree-1.so.1.0.0 [.] ostree_repo_file_tree_query_child
+ 13.74% 0.08% ostree libostree-1.so.1.0.0 [.] ot_variant_bsearch_str
```
The GIO abstractions are already fairly heavyweight, and `OstreeRepoFile` mallocs
a lot too.
Make things more efficient here by dropping the GIO bits for reading ostree data -
we just read from the variants directly and iterate over them. The end result
here is that according to perf we go from ~40% of our time in the kernel to
~70%, and things like `g_file_enumerator_iterate()` drop entirely out of the
hot set.
Closes: #848
Approved by: jlebon
Since we now have a cleaner separation of "toplevel checkout prep"
versus "recursive checkout", handle the special case of checking out
a single file at first rather than later.
Prep for future work in optimizing this function more.
Closes: #848
Approved by: jlebon
Only non-mechanical bit here was creating a local autoptr for a bit
where we'd previously done an unref for a struct member.
Closes: #847
Approved by: jlebon
Rather than `g_output_stream_splice()`, where the input is a regular
file.
See https://github.com/GNOME/libglnx/pull/44 for some more information.
I didn't try to measure the performance difference, but seeing the
read()/write() to/from userspace mixed in with the pointless `poll()` annoyed me
when reading strace.
As a bonus, we will again start using reflinks (if available) for `/etc`,
which is a regression from the https://github.com/ostreedev/ostree/pull/797
changes (which before used `glnx_file_copy_at()`).
Also, for the first time we'll use reflinks when doing commits from file-backed
content. This happens in `rpm-ostree compose tree` today for example.
Update submodule: libglnx
Closes: #817
Approved by: jlebon
Continuing to chip away at this. Using `g_file_enumerator_iterate()`
here helps notably.
I started on the much bigger `ostree_diff_dirs_with_options()` but
it's a lot messier - for later.
Closes: #844
Approved by: jlebon
Previously it was static to ostree-repo.c. Make it usable throughout
libostree so it can be used by an upcoming commit, but also expose the
typedef and reference counting functions so that opaque OstreeRemote
pointers can be used by user code, in anticipation of exposing more of
its API publicly in future.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #832
Approved by: cgwalters
This allows consumers of libostree to check at configure time whether it
supports the feature they want.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #832
Approved by: cgwalters
There are currently no unstable APIs, but some will be added in
following commits. They will be built and exposed in the libostree
global symbol list iff configured with --enable-experimental-api.
Distributions should not package OSTree with --enable-experimental-api.
This is designed for previewing new APIs on controlled platforms; any of
the APIs hidden behind this option may be changed or removed at any
point.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #832
Approved by: cgwalters
Read the http-headers (a(ss)) option in
ostree_repo_remote_fetch_summary_with_options like
ostree_repo_pull_with_options and add the headers to the fetcher. This
allows things like providing additional authorization headers to the
HTTP requests.
Closes: #839
Approved by: cgwalters
Allow GI bindings to delete refs through ostree_repo_transaction_set_ref
and ostree_repo_transaction_set_refspec by setting the checksum to NULL.
Closes: #834
Approved by: cgwalters
Rather than hard-coding the names as strings. This makes the code a
little more maintainable.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #826
Approved by: cgwalters
If a summary metadata key is well-known, like ostree.static-deltas, or
ostree.summary.last-modified, format it a little more nicely.
This is especially important for timestamps like last-modified, since
otherwise they’re formatted as a big-endian uint64, which is basically
unusable for the user.
Non-formatted output can still be retrieved using the OSTREE_DUMP_RAW
flag, and the non-formatted key name is always printed for clarity.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #826
Approved by: cgwalters
This allows a locally generated summary file to be viewed. It accepts
the same arguments as `ostree remote summary` (i.e. --raw).
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #826
Approved by: cgwalters
• Commit timestamps, so it’s easy to work out whether a given commit is
newer than the one we have locally
• Summary file timestamp, so it’s easy to work out whether the summary
file is more up to date than another summary file
• Summary file expiry time, so clients can work out when they should
expect the summary file to next be updated, and hence can query for
it at roughly the right time
The expiry time requires input from the user, so is currently never set
automatically. Programs using libostree can set it if they wish.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #826
Approved by: cgwalters
While running the testsuite under valgrind a small memory leak showed up:
==16487== 65 bytes in 1 blocks are definitely lost in loss record 773 of 1,123
==16487== at 0x4C2BBAF: malloc (vg_replace_malloc.c:299)
==16487== by 0x6048E08: g_malloc (gmem.c:94)
==16487== by 0x6062EAE: g_strdup (gstrfuncs.c:363)
==16487== by 0x54CE3E6: write_object (ostree-repo-commit.c:776)
==16487== by 0x54CF2D4: ostree_repo_write_metadata (ostree-repo-commit.c:1528)
==16487== by 0x54CF505: _ostree_repo_write_directory_meta (ostree-repo-commit.c:1712)
==16487== by 0x54D0AB4: write_dfd_iter_to_mtree_internal (ostree-repo-commit.c:2650)
==16487== by 0x54D0E2D: ostree_repo_write_dfd_to_mtree (ostree-repo-commit.c:2793)
==16487== by 0x1190C4: ostree_builtin_commit (ot-builtin-commit.c:474)
==16487== by 0x11F2EE: ostree_run (ot-main.c:200)
==16487== by 0x116F32: main (main.c:78)
The reason for this is that ot_checksum_instream_get_string returns a chunk of newly allocated memory which never got freed.
Make actual_checksum something that gets autocleanend and own the memory
assigned to it in all cases.
Signed-off-by: Sjoerd Simons <sjoerd.simons@collabora.co.uk>
Closes: #827
Approved by: pwithnall
If we're freeing the segment, it's basically always better to use
`autoptr()`. Fewer lines, more reliable, etc.
Noticed an instance of this in the pull code while reviewing a different PR,
decided to do a grep for it and fix it tree wide.
Closes: #836
Approved by: pwithnall
If one of the progress keys is set in a pull operation, a ::changed
signal is emitted on the progress object, and the callback for that
could query any of the progress keys — so they all need to be set,
otherwise we get an assertion failure in ostree_async_progress_get() due
to a named key not existing.
Spotted by Dan Nicholson in PR #819.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #835
Approved by: cgwalters
The summary URL override is looked up with "&s", which directly
exchanges the data to a pointer without allocation. This was causing a
segfault calling ostree_repo_remote_fetch_summary_with_options from
pygobject.
Closes: #829
Approved by: jlebon
g_date_time_new_from_unix_utc() will not always return a valid GDateTime
— if the input timestamp is too big, GDateTime cannot represent it, and
the constructor returns NULL.
Add some missing checks for these situations. We don’t ever expect
timestamps to be this big, but they could be as a result of corruption
or a malicious repository.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #825
Approved by: cgwalters
The keyring isn't large, so let's just fall back to copying it
rather than requiring `renameat()`.
Prep for `ostree_repo_open_at()`.
Closes: #821
Approved by: jlebon
Use the new well-known `status` key for OstreeAsyncProgress to get and
set the status atomically with other keys in an OstreeAsyncProgress
instance.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #819
Approved by: cgwalters
Rework how the status is handled in OstreeAsyncProgress so that it’s now
a well-known key in the hash table. This means that it can be retrieved
and set atomically with other keys using
ostree_async_progress_[get|set]().
The behaviour of ostree_async_progress_[get|set]_status() is preserved,
with the caveat that `status` can now also be accessed using the other
API on OstreeAsyncProgress, and has to be accessed with the right
GVariant type.
Internally, a NULL status is represented by an empty status string
(since ostree_async_progress_[get|set]_variant() deliberately don’t
allow NULL variants to be set against keys, since that would break the
ostree_async_progress_get() API).
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #819
Approved by: cgwalters
This will eliminate most of the potential races in progress reporting.
ostree_repo_pull_default_console_progress_changed() still calls three
getters, so there may still be races there, however.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #819
Approved by: cgwalters
OstreeAsyncProgress is thread-safe: it can have keys changed by one
thread while another is getting the same keys (modulo some locking
contention). However, the thread safety is done at the function call
level: if some code calls an OstreeAsyncProgress getter several times,
the key fetches are not atomic with respect to each other.
In the case of contention on the lock, this can result in consumers of
OstreeAsyncProgress data seeing an inconsistent state between the
properties they query, which could result in progress reporting
inaccuracies.
In the uncontested case, this results in the OstreeAsyncProgress lock
being locked and unlocked many times more than necessary.
Try to improve this by adding new API, which supports getting and
setting multiple keys atomically:
• ostree_async_progress_get()
• ostree_async_progress_set()
The new API uses GVariants and varargs: keys are passed as a
GVariantType string followed by arguments as for g_variant_new() or
g_variant_get(), followed by the next key, etc.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #819
Approved by: cgwalters
OstreeAsyncProgress currently does some contortions to try and avoid
allocating space for guints and guint64s (on 64-bit platforms), but this
means it uses two GHashTables. A GHashTable allocates 8 buckets even
when empty. Given that the largest usage of OstreeAsyncProgress in
libostree puts 13 uints and 5 uint64s in it, this optimisation does not
save significant (if any) memory.
Instead, change OstreeAsyncProgress to store values internally as
GVariants, and expose this with some new API:
• ostree_async_progress_get_variant()
• ostree_async_progress_set_variant()
Each GVariant is allocated on the heap. As they are immutable, they are
thread-safe once returned by a getter.
The existing API continues to work as before, except in the case where a
key is set/got as both a uint and a uint64 — there will now be a
collision (and a GVariant type checking failure) whereas previously
there was no collision. Nothing in OSTree uses OstreeAsyncProgress this
way though.
The new API can be used to share more complex data via the progress API.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #819
Approved by: cgwalters
Instead of using G_OPTION_ARG_STRING, use G_OPTION_ARG_FILENAME, which
handles filename encoding conversion differently from the locale
conversion which G_OPTION_ARG_STRING. This will fix argument handling on
systems where the filename encoding is not the same as the locale
encoding (which is fairly unlikely since most systems use UTF-8).
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #810
Approved by: cgwalters
I happened to glance at the top of my most recent patch and
noticed that I used an `throw_errno()` function in a non-errno place.
I scanned the patch for other instances of this but didn't find one.
Closes: #811
Approved by: jlebon
I was planning to change some of the object loading code in the
future, so here's some porting.
Note that I rewrote `_ostree_repo_has_loose_object()` since it
used an error return across multiple functions.
Honestly I'm not sure about this `TEMP_FAILURE_RETRY()` business...
in reality we're going to end up with a ton of code linked in
process that doesn't do it. Unix sucks =( But I'm keeping
what was there out of consistency.
Closes: #809
Approved by: jlebon
This did a `closedir` in the `goto out` section before, but it
turns out more nicely if we follow the usual pattern of doing
the `open(O_DIRECTORY)` in the callee function and handle `ENOENT`
there.
Closes: #809
Approved by: jlebon
This is a variant of the efforts in https://github.com/ostreedev/ostree/pull/741
Working on `rpm-ostree livefs`, I realized though I needed to just
check out *new* files directly into the live `/etc` (and possibly
delete obsolete files).
The way the current `/etc` merge works is fundamentally different from
that. So my plan currently is to probably do something like:
- Compute diff
- Check out each *new* file individually (as a copy)
- Optionally delete obsolete files
Also, a few other things become more important - in the current deploy code, we
copy all of the files, then relabel them. But we shouldn't expose to *live*
systems the race conditions of doing that, plus we should only relabel files we
checked out.
By converting the deploy's /etc code to use this, we fix the same TODO item
there around atomically having the label set up as we create files. And further,
if we kill the `/var` relabeling which I think is unnecessary since Anaconda
does it, we could delete large chunks of code there.
In the implementation, there are two types of things: regular files, and
symlinks. For regular files, in the `O_TMPFILE` case, we have the ability to
do *everything* atomically (including SELinux labeling) before linking it into
place. So let's just use that. For symlinks, we use `setfscreatecon()`.
Closes: #797
Approved by: jlebon
We really have an astonishing variety of similar functions which write files and
symlinks. I was working on a different PR and the duplication between the
union-mode and add-mode/none-mode checkout functions bothered me.
I realized that the "handle EEXIST" tri-state maps directly to the
`GLnxLinkTmpfileReplaceMode`, so deduping things makes even more sense.
Closes: #801
Approved by: jlebon
I was reading a strace the other day and noticed we were loading the same
`.dirmeta` object many times. Unlike the other object types, `.dirmeta` objects
don't accumulate much over time; there are only so many directory metadata types.
(Without SELinux involved it'd probably be 5-6 I'd guess offhand).
For `fedora-atomic/25/x86_64/docker-host` there are currently 34 `.dirmeta` in
the tree.
But how many times during a checkout did we load those 34 dirmeta objects?
With a quick strace:
```
$ strace -s 2048 -f -o strace.log ostree --repo=repo-build checkout -U fedora-atomic/25/x86_64/docker-host host-test-checkout
$ grep dirmeta strace.log | wc -l
7165
```
After, as you'd expect, we just loaded `34` from disk. We do
6 system calls (`openat+fstat+fstat+read+read+close`) per dirmeta,
so we dropped a total of 42780 system calls - which is about 20% of the total
system calls made.
`perf record` tells me that we're spending ~40 of our time in the kernel during
a checkout, so reducing syscall traffic helps. Though most of that appears to be
in the VFS and XFS layers for `linkat` (which isn't surprising).
So how much did perf improve? Well, on my workstation, I get a lot of
fluctuation in timing, sometimes by 30%, so this was well within the noise. But
it's well worth speeding up checkout, and I think this optimization will shine
more as we improve performance elsewhere.
Closes: #795
Approved by: jlebon
These are leftovers from the packfile code and should have been
deleted in commit: 2a0601efc7
I noticed this now since I wanted to add a new type of caching.
Closes: #795
Approved by: jlebon
I never really liked the term "osname". I feel "stateroot" is a *lot* clearer,
since the osname/stateroot mostly just holds `/var`. Further it avoids the `os`
prefix which is already overloaded.
Some of the existing docs already talked about "operating system state", which
further reinforces this.
There's *lot* more things than this which reference the term "osname", but I
don't want to change *everything* yet in this patch in case we decide to do
something different - this just gets the highlights.
Closes: #794
Approved by: jlebon
This is intended to be used for copying `/usr/etc` → `/etc` for
deployments.
A TODO here is to use `glnx_file_copy_at()` if the repo mode allows
it - then we'd use reflinks if available.
Closes: #804
Approved by: jlebon
There aren't many users of `g_file_enumerator_iterate()` left - those
remaining are usually good candidates for porting. There's some more
porting to do in this file; a mix of trivial and harder. This
one is a good candidate for an individual commit.
Closes: #803
Approved by: jlebon
I plan to make some future changes here, and wanted to do this
first.
Random side note; how about converting the do/while loops for `EINTR` to
`TEMP_FAILURE_RETRY()`? We're very inconsistent about that...
Closes: #792
Approved by: jlebon
This could help others who want to integrate with other init
systems/initramfs.
Commit-message-by: Colin Walters <walters@verbum.org>
Closes: #784
Approved by: cgwalters
`perf record ostree checkout ...` for a bare-user repo was telling
me we were spending a good 13% of our time in the depchain of `ot_lgexattrat()`.
The problem here is that traversing the `/proc` path turns out to be
somewhat expensive - there are LSM (SELinux) checks, etc.
For regular files, opening and just getting the xattr, then closing is still
quite cheap. For symlinks, we'll always need to open anyways.
This appears to shave about ~0.1 seconds off of a checkout of
`fedora-atomic/25/x86_64/docker-host` on my workstation.
Oh, and this was the last user of `ot_lgexattrat()` so we can kill it, which is
nice - the xattr code should really live in libglnx.
Closes: #796
Approved by: jlebon
This is only about 40%, and mostly simpler functions. It's
nice to switch to `g_autoptr(GMatchInfo)` instead of our inline version.
I decided to add more usage of `ot_transfer_out_value()`, though it'd
be nice to try to have a copy of that in libglnx (or possibly glib).
Closes: #791
Approved by: jlebon
Logic error introduced after refactoring; we hoisted the
`is_bare_user_symlink` variable to the top, but its computation
below. But the `is_bare` symlink depended on it.
Closes: https://github.com/ostreedev/ostree/issues/798Closes: #799
Approved by: jlebon
I think the majority of OSTree usage calls pull with refs, not
explicit commits. We even added special "override syntax" with
`@` (e.g. `ostree pull foo@ab12c34`) as a hybrid.
However, some users may want to still pull explicit commits
for whatever reason. The old static delta logic looked at
the previous commit of the ref. However, in https://github.com/ostreedev/ostree/pull/710
we enhanced the logic to look at all local commits.
It's now a lot more natural to teach the delta logic
to support revisions, e.g. `ostree pull someorigin ab12c34`.
This also fixes the problem that before, `--require-static-deltas`
was completely ignored when processing revisions.
This is a nontrivial refactoring of the logic, but the end
result feels a lot more readable to me.
Closes: https://github.com/ostreedev/ostree/issues/783Closes: #787
Approved by: cgwalters
Previously we'd assert and dump core if one used `checkout -H` without
`-U` on a bare-user repo, because we'd hit the bare-user symlink case.
Rework the code to handle this, and add tests. I hit this when I was going to
suggest to someone to use `-H` to ensure they were getting hardlinks.
Closes: #779
Approved by: jlebon
Testing a fetch of `fedora-atomic/.../docker-host` from
an nginx instance over `https://127.0.0.1` using Fedora 25
versions. Average over 3 runs:
Before: ~24.6 seconds
After: ~19 seconds
Speedup: ~30%
Closes: https://github.com/ostreedev/ostree/issues/778Closes: #780
Approved by: jlebon
In particular the 26-variable monster 👹 in `install_deployment_kernel()` is
slain🗡. I didn't touch every function here, trying to keep things gradual.
Closes: #781
Approved by: jlebon
It was reported that in the range request handling, we called `remove_pending()`
twice (once in processing it, and once potentially in the local_error cleanup),
and this could be viewed as a use-after-free. However, right now the range
cleanup and `local_error` being set are mututally exclusive.
Further, the task object already holds a strong reference, so I observed the
refcount was 2. For both of these reasons, there is no use-after-free in
practice.
Reported-By: "Siddharth Sharma" <siddharth@redhat.com>
Closes: #774
Approved by: jlebon
I was planning to change one here, decided to do a conversion
of some of the simpler functions in this file to keep up momentum.
Closes: #776
Approved by: jlebon
I was working on `rpm-ostree livefs` which does some ostree-based
filesystem diffs, and noticed that we were ending up with `/proc`
not being labeled in our base trees.
Reading the selinux-policy source, indeed we have:
```
/proc -d <<none>>
/proc/.* <<none>>
```
This dates pretty far back. We really don't want unlabeled
content in ostree. In this case it's mostly OK since the kernel
will assign a label, but again *everything* should be labeled via
OSTree so that it's all consistent, which will fix `ostree diff`.
Notably, `/proc` is the *only* file path that isn't covered when composing a
Fedora Atomic Host. So I added a hack here to hardcode it (although I'm a bit
uncertain about whether it should really be `proc_t` on disk before systemd
mounts or not).
Out of conservatism, I made this a flag, so if we hit issues down the line, we
could easily change rpm-ostree to stumble on as it did before.
Closes: #768
Approved by: jlebon
Being bitten by lack of PR testing here. There are two bugs:
- First and foremost, I forgot that GObject will call the property setters with
the defaults. This meant we were getting both path="/var/tmp/blah" and fd=-1,
and we were accepting -1 as a fd, which then got converted into AT_FDCWD
which was wrong.
- Since these properties are construct only and mutually exclusive, don't
try to handle one resetting the other. Assert that exactly one of them is set.
Closes: #769
Approved by: jlebon
This is inspired by the [Coccinelle](http://coccinelle.lip6.fr/) usage
in systemd. I also took it a bit further and added infrastructure
to have spatches which should never apply. This acts as a blacklist.
The reason to do the latter is that coccinelle is *way* more powerful than the
regular expresssions we have in `make syntax-check`.
I started with blacklisting `g_error_free()` directly. The reason that's bad is
it leaves a dangling pointer.
Closes: #754
Approved by: jlebon
I happened to read this file and realized there's a lot of cruft left over from
the time when I liked `GFile` and `malloc()`ing like 50 times just to make a
pathname string. Delete it.
Closes: #767
Approved by: jlebon
Actually trying to use this in rpm-ostree, it kept returning successfully when I
didn't expect it to... The first conditional was always succeeding even when I
was asking for a newer minor.
Closes: #766
Approved by: jlebon
I'd like to do this in rpm-ostree at least. Originally I was looking at porting
to `G_DECLARE_FINAL_TYPE` but eh, this is easier for now and won't bump our GLib
dependency which might matter for our embedded users.
For now I just did a few replacements in the `remote` command line. A full port
can come as we do other code cleanups.
This will actually break the flatpak build right now, but
that's easy to fix. And we concluded in e.g.
https://bugs.freedesktop.org/show_bug.cgi?id=95065#c5
it's a bug for downstream projects to do that.
Closes: #756
Approved by: jlebon
I didn't touch everything since at least `commit_loose_object_trusted`
does this:
```
out:
if (G_UNLIKELY (error && *error))
g_prefix_error (error, "Writing object %s.%s: ", checksum, ostree_object_type_to_string (objtype));
```
Which...it'd be interesting to make into an autocleanup. But for now just
keeping up with converting things bit by bit.
Closes: #761
Approved by: jlebon
In [this commit](6ce80f9685)
for some reason I added a `sepolicy` member to the sysroot. I
have no idea why I did that, and it's conceptually wrong
since the policy is specific to a *deployment*.
This bit me when I was working on [a pull request](https://github.com/ostreedev/ostree/pull/763)
elsewhere, since at that point it was `NULL`.
We already pass around the sepolicy in the deployment code, so just stop caching
it.
Closes: #764
Approved by: jlebon
I was playing around in a FAH vagrant box, and hit:
```
Receiving delta parts: 3/4 453.2 kB/s 1.8 MB/145.8 MB
error: opcode set-read-source: No such file object b6e54ba3471b9c116ce6b9bfbf9e55fec60d35cfdb9ae5ae1ee219af02a591b7
```
This is because this host version doesn't yet have
https://github.com/ostreedev/ostree/pull/710
which incidentally fixed this for the case where the OS vendor is using
summary files.
Some organizations may not be using summary files - at least we still try to
support that case. So let's copy the logic very recently added in that commit to
handle the legacy case too.
No new tests since this is a nice-to-have - we really do
expect people to be using summary files now.
Closes: #739
Approved by: jlebon
This adds to file permission masks the same bitmask that will
be applied to file objects in bare-user* repos. This will be
needed in the testsuite to ensure that the things we commit
will be expressable in bare-user-only repos.
Closes: #750
Approved by: cgwalters
This mode is similar to bare-user, but does not store the permission,
ownership (uid/gid) and xattrs in an xattr on the file objects in the
repo. Additionally it stores symlinks as symlinks rather than as
regular files+xattrs, like the bare mode. The later is needed because
we can't store the is-symlink in the xattr.
This means that some metadata is lost, such as the uid. When reading a
repo like this we always report uid, gid as 0, and no xattrs, so
unless this is true in the commit the resulting repository will
not fsck correctly.
However, it the main usecase of the repository is to check out with
--user-mode, then no information is lost, and the repository can
work on filesystems without xattrs (such as tmpfs).
Closes: #750
Approved by: cgwalters
There are a lot of things suboptimal about this approach, but
on the other hand we need to get our CI back up and running.
The basic approach is to - in the test suite, detect if we're on overlayfs. If
so, set a flag in the repo, which gets picked up by a few strategic places in
the core to turn on "ignore xattrs".
I also had to add a variant of this for the sysroot work.
The core problem here is while overlayfs will let us read and
see the SELinux labels, it won't let us write them.
Down the line, we should improve this so that we can selectively ignore e.g.
`security.*` attributes but not `user.*` say.
Closes: https://github.com/ostreedev/ostree/issues/758Closes: #759
Approved by: jlebon
More sophisticated users of libostree like rpm-ostree need control over things
like the system repository. Previously we introduced a "no cleanup" flag to
`ostree_sysroot_simple_write_deployment()`, but that's a high level API that
does filtering on its own.
Since rpm-ostree needs more control, let's expose the bare essentials of the
"sysroot commit" operation with an extensible options structure, where one of
the options is whether or not to do post-transaction repository operations.
Closes: #745
Approved by: jlebon
I happened to be reading this one recently for a reason I forget,
and it's a relatively easy conversion.
Also one not conflicting with any outstanding patches.
Closes: #752
Approved by: jlebon
Use `g_auto()` more sanely with a struct implmenting the "is initialized"
pattern. This is way less ugly for callers, and fixes bugs like
us calling `setfscreatecon()` even if an error occurred beforehand.
Also fold in the logic for "NULL or not loaded" sepolicy into the setup rather
than requiring callers to inline it.
Prep for more users of this function.
Closes: #746
Approved by: jlebon
I'm porting other code away from `GFile`, and while we don't use this
internally, it will let us do so at a later date. I'm averse to changing the
code right now as we don't have good CI coverage of this.
Closes: #746
Approved by: jlebon
The first options are owner_uid/owner_gid, which makes it possible to use diff
on local files where --owner-uid/gid have been passed to commit.
Closes: #740
Approved by: cgwalters
For future work I'm going to tweak how we handle cleanup, and
the private cleanup flags didn't really end up being used - we
only specify "prune repo or not". So fold that into a boolean for now.
The sysroot deploy logic then has a single "do_postclean" boolean, which is all
I want to expose as public API.
Closes: #744
Approved by: jlebon
Add an OpenSSL backend to the checksum input stream, which is where we do a lot
of checksumming (object commit, static deltas).
The raw OpenSSL performance is
[approximately double](https://gist.github.com/cgwalters/169349fd1c06fd4fb4d3a7ce33303222) on
my laptop; not only does OpenSSL have e.g. hand-tuned x86_64 assembly, the
current implementation uses the
[Intel SHA extensions](https://en.wikipedia.org/wiki/Intel_SHA_extensions).
Another reason to do this is I was idly thinking about adding
[Curve25519](https://en.wikipedia.org/wiki/Curve25519) signatures (like e.g.
Alpine does) instead of/in addition to GPG. The rationale for that is
that GPG is pretty heavyweight, both in code footprint and the simple
fact that EC keys are way smaller.
I didn't benchmark ostree with this; we have bigger performance problems
really like the fact we just malloc way too much. But, it's a step
in the right direction I think in combination with the libcurl work
where we're linking to openssl anyways.
Closes: #738
Approved by: jlebon
The current `OstreeChecksumInputStream` is public due to a historical
mistake. I'd like to add an OpenSSL checksum backend, but that's
harder without breaking this API.
Let's ignore it and create a new private version, so it's easier to do the
GLib/OpenSSL abstraction in one place.
Closes: #738
Approved by: jlebon
I just happened to be reading this code; still testing
the waters for the new code style. If we decide to do more
of this it's likely worth trying out a [Coccinelle](http://coccinelle.lip6.fr/)
patch or something.
Closes: #733
Approved by: jlebon
[Previously](https://github.com/ostreedev/ostree/pull/728) we added compile-time
checking for versions, but there are use cases for runtime checking as well,
because in a number of API calls we use `GVariant` as an API extension
mechanism.
Closes: #735
Approved by: jlebon
This goes farther along the path of deprecating it; consumers should
have been ported at this point. Though probably a lot of people
may still use `rpm-ostree-toolbox`.
Closes: #723
Approved by: jlebon
OSTree currently provides no way to inspect the versioning
information at run time, being only available at compile
time through pkg-config.
This is a problem for e.g. Flatpak, that needs to check
whether the 'update-frequency' option is available. Checking
at compile time isn't great since it's not looking for new
symbols, but only if an optional feature is present.
This commit, then, adds a new header that is generated
at compile time, exposing OSTree's versioning information.
Closes: #728
Approved by: cgwalters
It's been almost a month, I think the current git is working well and
not too risky. We have some new API additions which I think the
various consumers of them are going to want.
Closes: #726
Approved by: jlebon
After commit 80b3edc64731a5f0 introducing the option to set
a custom timeout, adapt the ostree program to be able to update
that.
Closes: #725
Approved by: jlebon
When using Flatpak with GNOME Software, it is important to
show the progress of the download and install as close as
possible to the real progress.
However, OSTree forces the frequency to call the async
progress callback to 1 second, which causes an unpleasant
effect on the UI, specially when the download size is so
small that everything happens in less than 1 second.
Fix that by adding making OSTree read a custom 'update-frequency'
option and set the timeout source timeout to that. If
no custom frequency is passed, we assume the default 1
second timeout, maintaining the current behavior.
Closes: #725
Approved by: jlebon
At least in all Linux kernels up to today, one can never `link()` across
devices, so we might as well verify that up front. This will help for a future
patch to add a new type of union-add checkout, since Linux checks for `EEXIST`
before `EXDEV`.
Closes: #714
Approved by: jlebon
I've seen code in a few places that I think on balance is definitely better this
way. Some of our functions have huge variable declaration sections.
This change includes one small example where we could start using declarations
after statements.
A concern I had was - how does this interact with `__attribute__((cleanup))` and
early returns? I tested it, and AFAICS the behavior is what you'd expect - the
cleanup function isn't called if its variable isn't reachable.
Closes: #718
Approved by: jlebon
Add a ostree_raw_file_to_archive_z2_stream_with_options() variant of
ostree_raw_file_to_archive_z2_stream(), to allow a compression-level
option to be passed in and passed through to zlib.
This is useful when building archive-z2 files on the fly for
transmission over a non-bandwidth-limited channel, such as a local
network. In this case, CPU time is more valuable than bandwidth, so we
want a low compression level.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #721
Approved by: cgwalters
I had to rebuild `glib` with `-fsanitize=address` in order to get a stack trace
to finally get this one. However, *installing* that glib "system wide"
in my container breaks everything (including `rpm-ostree`, `dnf`, `pkg-config` etc.)
that wasn't built with ASAN.
So my test scenario right now is to extract the libs and do e.g.:
```
make && env LD_LIBRARY_PATH=$HOME/src/distgit/fedora/glib2/asan-libs make check TESTS=tests/test-basic.sh
```
Closes: #719
Approved by: jlebon
The previous logic for static deltas was to use as a FROM
revision the current branch tip. However, we want
to support deltas between branches in an automatic
fashion.
If a summary file is available, we already have an
enumerated list of deltas - so the logic introduced
here is to search it, and find the newest commit
we have locally that matches the TO revision target.
This builds on some thoughts from
https://github.com/ostreedev/ostree/pull/151#issuecomment-232390232
Closes: https://github.com/ostreedev/ostree/pull/151Closes: #710
Approved by: giuseppe
Got a report that a Fedora Atomic Host built for ppc64le didn't work with the
`linux16`, it needed `linux`. See the comments for more links.
Closes: #716
Approved by: vathpela
In https://github.com/ostreedev/ostree/pull/408, we disabled the use of
static deltas when mirroring. Later,
https://github.com/ostreedev/ostree/pull/506 loosened this up again so
that we could use static deltas when mirroring into bare{-user} repos.
However, the issue which originally spurrred #408 is even more generic
than that: we want to avoid static deltas for any archive repo, not just
when doing a mirror pull. This patch tightens this up, and also
relocates the decision code to make it easier to read.
Closes: #715
Approved by: cgwalters
The C API (ostree_repo_static_delta_generate) knows what to do
with it, but this parameter was never exposed via command line
tool.
Closes: https://github.com/ostreedev/ostree/issues/695Closes: #703
Approved by: jlebon
Due to the async nature of `GSubprocess` it grabs `SIGCHLD` which
affects other software which might be using libostree, such as
QtOTA.
Closes: https://github.com/ostreedev/ostree/issues/696Closes: #702
Approved by: jlebon
This makes it easier to script downloading updates in the background,
and only do deployments just before rebooting.
Partially addresses https://github.com/ostreedev/ostree/issues/640Closes: #642
Approved by: jlebon
Particularly when HTTP requests fail, I really want a lot more information.
We could theoretically stuff it into the `GError` message field, but
that gets ugly *fast*.
Using the systemd journal allows us to log things in a structured fashion.
Right now e.g. rpm-ostree won't be aware of this additional information,
but I think we could teach it to be down the line.
In the short term, users can learn to find it from `systemctl status rpm-ostreed`
or `journalctl -b -r -u rpm-ostreed`, etc.
One thing I'd like to do next is log successful fetches of e.g. commit objects
as well with more information about the originating server (things like the
final URL if we were redirected, did we use TLS pinning, what was the negotiated
TLS version+cipher, etc).
Closes: #708
Approved by: jlebon
https://github.com/ostreedev/ostree/pull/705 broke the build
on CentOS 7 which only has util-linux 2.23.
When I was thinking about this, I realized that there must really be a way to
make this safe even for older versions. Looking at that version of util-linux,
all we need to do is invert the order of frees so we `mnt_free_table()` *before*
`mnt_free_cache()`, like util-linux does:
https://github.com/karelzak/util-linux/blob/stable/v2.23/sys-utils/eject.c#L1131
We still use the `_unref()` versions if available. I also fixed
the ordering there too for double plus redundant safety.
Closes: #712
Approved by: jlebon
Should fix everything from
<https://kojipkgs.fedoraproject.org//packages/ostree/2017.2/3.fc25/data/logs/x86_64/build.log>
Anything that uses autocleanups should *always* be initialized directly I think,
even if a few lines down we directly assign, since this way it's more robust
against refactoring.
And the `freopen()` warnings are right - IMO we should *always* check return
values.
Closes: #711
Approved by: jlebon
We saw a random ostree SEGV start popping up in our CI environment:
https://github.com/projectatomic/rpm-ostree/pull/641#issuecomment-281870424
Looking at this code more and comparing it to what util-linux does, I noticed we
had a write-after-free, since `mnt_unref_table()` will invoke
`mnt_unref_cache()` on its cache, and that function does:
```
if (cache) {
cache->rfcount--;
```
unconditionally.
Fix this by using `unref()`.
Closes: #705
Approved by: jlebon
I learned today that `docker version` does this and I really like
the idea. While we have the patient open, also add the gitrev
with code taken from https://github.com/projectatomic/rpm-ostree/pull/584Closes: #691
Approved by: giuseppe
It's just simpler, and I'm not sure people are going to care
much about the difference by default.
We already folded in the fallback sizes into the download totals, so folding in
the count makes things consistent; previously you could see e.g.
`3/3 parts, 100MB/150MB` and be confused.
Closes: #678
Approved by: giuseppe
I don't know why I added support for this; it makes no sense really. If we have
large metadata objects something has gone badly wrong.
The delta compiler has always only processed fallbacks for regular
content files.
Dropping support in the fetcher for this will simplify later handling of
fallback progress accounting.
Closes: #678
Approved by: giuseppe
There were a few bugs here.
- We need to keep track of the size of the delta parts we've already processed,
in order to make progress reliable at all in the face of interruptions. Add
a new `fetched-delta-part-size` async progress variable for this.
- The total before disregarded what we'd already downloaded, which was confusing.
Now, a progress percentage is `fetched/total`.
- Correctly handle "unknown bytes/sec" in the progress display.
However, to be fully correct we need to show the fallback objects too. That
would require tracking in the pull code when we fetch an object as a fallback
versus "normally". This would be simpler really if we could assume in a run we
were *only* processing a delta, but currently we don't do that.
Related: https://github.com/ostreedev/ostree/issues/475Closes: #678
Approved by: giuseppe
Doing `g_variant_print (superblock)` is unreadable and not very useful,
since we show the checksums as byte arrays.
However, do show the checksums for fallback objects. This makes it easier to see
which objects are fallbacks (and inspect why).
Closes: #678
Approved by: giuseppe
In https://github.com/ostreedev/ostree/pull/634 we introduced
a subtle regression - the unreadable object was added to the *new*
reachable objects, when it shouldn't have been. Because it
was a *from* object, clients already had it.
This became more obvious now that I'm working on fixing delta
progress - I noticed my deltas were always starting out with 40MB
fetched, which turned out to be a non-world-readable initramfs object.
This code should simply *skip* the unreadable object, and the delta processing
below properly iterates over "new objects", so we'll pick it up from there.
Closes: #678
Approved by: giuseppe
We should get a release out to try to keep with at least a once-a-month cadence.
This one has some exciting stuff like libcurl and Rust, and various bugfixes.
Also importantly I want to cut this *before* we land some other bigger stuff, so
rpm-ostree can start using the reload_config API etc.
Closes: #685
Approved by: jlebon
These allow us to avoid copying a lot of data around
in userspace. Instead we splice the data directly from
the fd to the destination fd.
Closes: #684
Approved by: cgwalters
Switching between local branches should be supported too.
Signed-off-by: Anton Gerasimov <anton@advancedtelematic.com>
Closes: #683
Approved by: cgwalters
Clarify the documentation for functions like
ostree_repo_get_remote_boolean_option(), stating what out_value will be
set to on error.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #676
Approved by: cgwalters
When fetching over a fast enough connection, we can be receiving files
faster than we write them. This can then lead to EMFILE when we have
enough files open. This was made very easy to notice with the upcoming
libcurl backend, which makes use of pipelining.
Closes: #675
Approved by: cgwalters
For rpm-ostree, we already link to libcurl indirectly via librepo, and
only having one HTTP library in process makes sense.
Further, libcurl is (I think) more popular in the embedded space. It
also supports HTTP/2.0 today, which is a *very* nice to have for OSTree.
This seems to be working fairly well for me in my local testing, but it's
obviously brand new nontrivial code, so it's going to need some soak time.
The ugliest part of this is having to vendor in the soup-url code. With
Oxidation we could follow the path of Firefox and use the
[Servo URL parser](https://github.com/servo/rust-url). Having to redo
cookie parsing also sucked, and that would also be a good oxidation target.
But that's for the future.
Closes: #641
Approved by: jlebon
The libcurl backend does all the work in the main thread/loop, which
seems to starve the idle scanning worker more. With the libcurl
backend, we're a lot more likely to have at least one outstanding
metadata request.
But it can more easily transiently happen with libcurl that all of our current
fetches are content. To be accurate here, just show Estimating if we're scanning
too.
Closes: #654
Approved by: jlebon
Now that we have queuing in the higher level pull logic, we don't
need to do this anymore.
It's tempting to keep it since the code diff is so small (without
completely rewriting things), but dropping it here will make
it easier to see when things go wrong at a higher level.
Note that I kept an assertion.
Closes: #654
Approved by: jlebon
Working on the libcurl backend, I didn't want to reimplement another queue. I
think the queue logic is really better done at the high level, since the fetcher
knows how we want to prioritize metadata over content, etc.
Adding another queue here is duplication, but things will look nicer when we can
actually delete the libsoup one in the next commit.
Closes: #654
Approved by: jlebon
The gzip default is 6. When I was writing this code, I chose 9 under
the assumption that for long-term archival, the extra compression was
worth it.
Turns out level 9 is really, really not worth it. Here's run at level 9
compressing the current Fedora Atomic Host into archive:
```
ostree --repo=repo pull-local repo-build fedora-atomic/25/x86_64/docker-host
real 2m38.115s
user 2m31.210s
sys 0m3.114s
617M repo
```
And here's the new default level of 6:
```
ostree --repo=repo pull-local repo-build fedora-atomic/25/x86_64/docker-host
real 0m53.712s
user 0m43.727s
sys 0m3.601s
619M repo
619M total
```
As you can see, we run almost *three times* faster, and we take up *less
than one percent* more space.
Conclusion: Using level 9 is dumb. And here's a run at compression level 1:
```
ostree --repo=repo pull-local repo-build fedora-atomic/25/x86_64/docker-host
real 0m24.073s
user 0m17.574s
sys 0m2.636s
643M repo
643M total
```
I would argue actually many people would prefer even this for "devel" repos.
For production repos, you want static deltas anyways. (However, perhaps
we should support a model where generating a delta involves re-compressing
fallback objects with a bit stronger compression level).
Anyways, let's make everyone's life better and switch the default to 6.
Closes: #671
Approved by: jlebon
For a long time we've cached the remote configs in the repo, which
mostly makes sense for the `repo/config` file, but less sense
for `/etc/ostree/remotes.d`, because we want to support admins
interactively editing them.
One can delete the repo instance and create a new one, but that's a bit ugly.
Let's introduce an API for this so rpm-ostree can reload remotes after
admins/scripts edit them in `/etc`. We also might as well reload
any other entries in the config.
Structurually now, `ostree_repo_open()` deals with file descriptors, and then
calls `ostree_repo_reload_config()`. Except for the uncompressed cache, which is
the only thing that deals with FDs that can be configured. But we want to delete
that anyways.
No tests, since...we don't have a daemon in this codebase, don't want to shave
that yak just today.
Closes: #662
Approved by: jlebon
We weren't running it before. Also I switched it to use GLib. Preparation for
some oxidation work (having an implementation of bupsplit in Rust).
I exported another function to do the raw rollsum operation which is what this
test suite uses.
Closes: #655
Approved by: jlebon
I was working on https://bugzilla.redhat.com/show_bug.cgi?id=1393545
and it was annoying that I couldn't know what the new (unsigned)
commit has was until verification succeeded. I could pull it
manually without GPG, but then it'd be sitting in the repo.
Now:
```
Updating from: fedora-atomic:fedora-atomic/25/x86_64/docker-host
Receiving metadata objects: 0/(estimating) -/s 0 bytes
error: Commit 2fb89decd2cb5c3bd73983f0a7b35c7437f23e3aaa91698fab952bb224e46af5: GPG verification enabled, but no signatures found (use gpg-verify=false in remote config to disable)
```
Closes: #663
Approved by: giuseppe
Add an arg description for -P, otherwise it's not immediately obvious
that it takes an argument.
Mention that - is supported for --log-file.
Closes: #657
Approved by: cgwalters
There are use cases for having a single repo with branches
with different lifecycles; a simple example of what I was
trying to do in CentOS Atomic Host work is have "stable"
and "devel" branches, were we want to prune devel, but
retain *all* of stable.
This patch is split into two parts - first we add a low level "delete all
objects not in this set" API, and change the current prune API
to use this.
Next, we move more logic into the "ostree prune" command. This paves the way for
demonstrating how more sophisticated algorithms/logic could be developed outside
of the ostree core.
Also, the --keep-younger-than logic already lived in the commandline, so it
makes sense to keep extending it there.
Closes: https://github.com/ostreedev/ostree/issues/604Closes: #646
Approved by: jlebon
This is prep for the libcurl porting. `GTlsCertificate/GTlsDatabase` are
abstract classes implemented in glib-networking for gnutls. curl's APIs take
file paths as strings, so it's easier to work on both if we move the GLib TLS
bits into the libsoup code.
Closes: #651
Approved by: giuseppe
I was making some other changes in this code, and noticed that we were adding
checksums without object types into the same hash table for metadata. We should
*never* do this with both metadata content objects, since in theory a content
object could have the same hash as metadata.
I don't actually think it's possible in practice for pure metadata to collide,
since they have different structures, but let's do this anyways since it's
conceptually right.
Closes: #651
Approved by: giuseppe
I was trying to debug `test-pull-c`, and typing `Ctrl-C` in gdb
ended up sending `SIGINT` to trivial-httpd as well, killing it.
Daemonize a bit more properly to avoid this. I also followed the standard
`/dev/null` guidelines.
Closes: #643
Approved by: jlebon
For the pending libcurl port, the backend is a bit more
sensitive to the main context setup. The delta superblock
fetch here is a synchronous request in the section that's
supposed to be async.
Now, libsoup definitely supports mixing sync and async requests, but it wasn't
hard to help the libcurl port here by making this one async. Now fetchers are
either sync or async.
Closes: #636
Approved by: jlebon
Working on the libcurl backend, I hit the issue that the trivial-httpd program
depends on libsoup. I briefly considered having two versions, but libcurl is
client only, and moreover trivial-httpd is no longer trivial - it has various
features which are used by the test suite extensively.
Hence, what we'll do is build it as a separate binary which links to libsoup,
and use it during the tests. We *also* currently still provide `ostree
trivial-httpd` since some things use it like `rpm-ostree-toolbox` and the
Cockpit tests.
After those are ported to use some other webserver, I plan to add a build-time
option to drop it.
Closes: #636
Approved by: jlebon
The previous commit introduced a single low level API - however,
we can do things in a more optimal way for the curl backend if
we drop the "streaming API" variant. Currently, we only use
it to synchronously splice into a memory buffer...which is pretty
silly when we could just do that in the backend.
The only tweak here is that we have an "add NUL character" flag that is
(possibly) needed when fetching into a membuf.
The code here ends up being better I think, since we avoid the double return
value for the `_finish()` invocation, and now most of the fetcher code (in the
soup case) writes to a `GOutputStream` consistently.
This will again make things easier for a curl backend.
Closes: #636
Approved by: jlebon
Conceptually these now lay on top of the core API, and don't reference libsoup.
This is preparation for libcurl porting, but it's also just generally better.
Closes: #636
Approved by: jlebon
This is in preparation for the libcurl port. We're basically making public what
we had internally. The next step here is to create `ostree-fetcher-util.[ch]`
that only operates in terms of this lower level API.
Also drop the `_mirrored` from the function name since it's
the default now.
Closes: #636
Approved by: jlebon
Without the element-type annotations, bindings don't know how to handle
the elements of the hash table. Since the table is created with destroy
functions, the caller does not own the elements, so transfer container
is used.
Closes: #635
Approved by: cgwalters
ostree_object_name_serialize returns a floating ref, so sink it before
adding it to the hash table so it can properly be freed later when the
hash table is destroyed.
This is particularly a problem for pygobject, which sinks the refs on
variants as it marshals them to native python types. If the ref isn't
already sunk, then the ref count won't increase and a critical warning
will be raised when both the hash table and pygobject try to unref it.
Closes: #635
Approved by: cgwalters
This will prevent including in the delta the bits to update files that
are not world readable, so that we don't run into a permissions problem
when applying the deltas from a bare-user repository that has a bare
repository set as its parent.
This is the case for Endless when updating flatpak runtimes, as the
temporary directory created in ~/.local/share/flatpak/system-cache will
be of type bare-user with its parent set to /var/lib/flatpak which is a
bare repository in EOS, as it's shared with the one at /ostree/repo.
https://phabricator.endlessm.com/T14159Closes: #634
Approved by: cgwalters
I was having this thought today about making more of the OS readonly,
and ultimately if we got to the point where all ostree operations are
through the repo and sysroot dfds, we could have rpm-ostree be the
only process holding those fds open, and have a read-only bind mount
on top.
Anyways, we're not there, likely won't be soon, but this gets us
closer to being fully fd relative.
Closes: #628
Approved by: jlebon
This is a migration from the origin version. It's
nicer to have it in the remote, since that's what one
needs to change. Then tools don't need to mess with
the origin file.o
In fact in this scenario one can keep the "media source" like
`file:///install/repo` or whatever, since conceptually that's where it
came from. We're just providing a better error.
Closes: https://github.com/ostreedev/ostree/issues/626Closes: #627
Approved by: jlebon
These are out parameters, so add the (out) annotation and switch
(nullable) to (optional) since the latter is used for the purpose of
optional out parameters.
Closes: #629
Approved by: cgwalters
We were leaking in a few places that I noticed in an ASAN run. Also,
this was one of the last non-autoptr cleanup sections we have in
`out:` cleanup sections, making us a lot closer to a potential
full-tree rewrite to `return FALSE`.
Closes: #624
Approved by: jlebon
I installed `parallel` in my dev container, which got me
the sysroot locking tests, which caught this leak when
built with ASAN.
Closes: #623
Approved by: jlebon
The "remote cookies" code broke this. While I'm not sure anyone is
actually using ostree-without-http, it isn't too hard to keep the
build time conditional going. Further, this work is preparatory for
libcurl porting.
Closes: #621
Approved by: jlebon
Due to the way glib-mkenums runs the preprocessor itself, it
doesn't pick up the `AC_USE_SYSTEM_EXTENSIONS()` that we have in
`configure.ac`.
This blew up in an obscure way when I later wanted to `#include
"libglnx.h"` in one of the headers, since it needs the `basename()`
from `string.h` which is only available with `_GNU_SOURCE`.
Closes: #616
Approved by: jlebon
This is what we do for non-local (i.e. HTTP) pulls; we wnat to
correctly handle being interrupted during partial pulls.
Closes: https://github.com/ostreedev/ostree/issues/579Closes: #613
Approved by: jlebon
This is a follow up to conversation on list - in practice, if we're
backing away from summary signing, then it makes sense to remove the
special casing for checksums in deltas around summary signatures.
This is also related to the recent change to enable GPG checking for
commits in deltas - now we have a more coherent story between the
previous pull path and deltas.
I didn't do any performance checking, and while it's slightly annoying
that we're now doing sha256 on the delta content twice (once for the
part and once per object)...sha256 is pretty fast, I think most users
are I/O bound anyways, and it'd drop even farther if we started using
openssl.
Closes: #612
Approved by: jlebon
We should be religious about the "only set output variables on
success", otherwise it makes leaks more likely.
But the real leak was us simply not using autoptr in one place.
Closes: #598
Approved by: jlebon
And "move semantics" via `g_steal_pointer()`. Just a minor code
cleanup I noticed when I was hunting for a leak, which ended up being
elsewhere.
Closes: #598
Approved by: jlebon
glnx_make_lock_file requires that the dfd passed in survives the
lifetime of the lock. Since dfd_iter.fd gets cleaned up after the
function returns, this isn't the case. dfd_iter.fd should be equivalent
to tmpdir_dfd, since we iter on ".", and that survives past the
function, so just use that instead.
Closes: #591
Approved by: cgwalters
The fact that we weren't doing this is at best an oversight, and
for some deployment models a security vulnerability. Having both
`gpg-verify` and `gpg-verify-summary` shows that we were intending
them to be orthogonal/independent.
Lately I've been advocating moving towards pinned TLS instead of
gpg-signed summaries, and if we follow that path, performing GPG
verification of commit objects even if using deltas is more important,
as it provides an at-rest verifiable authenticity and integrity
mechanism.
Content providers which are signing their summary files and/or using
TLS (particularly pinned TLS) for transport should treat this as a
nice-to-have. However, for providers which are serving content over
plain HTTP and relying on GPG, this is a critical update.
Closes: https://github.com/ostreedev/ostree/issues/517Closes: #589
Approved by: jlebon
`-fsanitize=address` complained that the `refcount > 0` assertions
were reading without atomics. We can fix this by reworking them
to read the previous value.
Closes: #582
Approved by: jlebon
It turns out this is basically racy with the presence of other
threads. It was really cosmetic so let's stop doing it and make
`-fsanitize=thread` happy.
Closes: #582
Approved by: jlebon
This is actually fine in practice, but it triggers this
`-fsanitize=undefined` warning I saw in the test suite log:
```
src/libostree/ostree-repo-static-delta-compilation.c:160:10: runtime error: null pointer passed as argument 1, which is declared to never be null
```
Closes: #584
Approved by: jlebon
You'd expect
ostree commit --tree=ref=A --tree=ref=B
to produce a commit with the union of the trees given. Instead you'd get
a commit with the contents of just the latter commit. This was due to an
optimisation where we'd skip filling out the `files` and `subdirs`
members of the mtree, just filling in the metadata instead. This backfires
becuase this same code relies on checking the `files` and `subdirs` members
itself to work out whether the mtree is empty.
This commit removes the optimisation, fixing the bug. Maybe there's a way
to keep the optimisation and still fix the bug but it's not obvious to
me.
Closes: #581
Approved by: cgwalters
Conceptually we've been moving towards having our GPG verification
paths be per-remote. The code internally supports this, but we
didn't expose an API to use it conveniently.
This came up when trying to add a new `gpgkeypath` option, since
right now rpm-ostree manually finds keyrings for the remote, and
hence it wasn't looking at the keypath, and said "Unknown key"
in status.
Adding an API fixes this nicely.
Closes: #576
Approved by: giuseppe
For Project Atomic, we already have RPM signatures which use files in
`/etc/pki/rpm-gpg`. It's convenient to simply bind the OSTree remote
configuration to those file paths, rather than having duplicate key
data.
This does mean that we need to parse the files for verification, so we
end up importing them into the verifier's temporary keyring, which is
a bit ugly, but it's what other projects do.
Closes: https://github.com/ostreedev/ostree/issues/573Closes: #575
Approved by: giuseppe
When doing commit --tree=ref=XXX while at the same time applying some
form of modifier, ostree dies trying to read the xattrs using the
raw syscalls. We fix this by falling back to ostree_repo_file_get_xattrs()
in this case.
Also adds a testcase for this.
Closes: #577
Approved by: cgwalters
What in the code is called "scanning" is ensuring (potentially
recursively) have an object, and if not, fetching it. And then if
it's metadata, parsing it and finding new objects to fetch.
This logic has grown fairly complex. What I'm trying to fix
right now is that if we're doing a pull-local to a remote repository
via `sshfs` (FUSE) we still end up scanning, which is inefficient.
We can take advantage of the "commitpartial" logic here - if a commit
isn't partial, it's complete, hence we don't need to scan it.
At the same time, I'm changing the logic here to *always* do scans for
dirtree objects. This will fix cases where multiple commits share
dirtree objects. We have "commitpartial" metadata, but no such concept
of partial/complete for dirtrees.
But, we'll only ever scan dirtrees if we scan commits, which is
what the section above fixes.
Closes: https://github.com/ostreedev/ostree/issues/543Closes: #564
Approved by: alexlarsson
Some deployments may want to gate access to content based on things
like OAuth. In this model, the client system would normally compute a
token and pass it to the server via an API.
We could theoretically support this in the remote config too, but
that'd be a bit weird for OAuth as the information is dynamic.
Therefore this cleans up the code a little bit to more clearly handle
the case that the fetcher is initialized from both remote config
data plus pull options.
Closes: #574
Approved by: giuseppe
Otherwise it's possible for us to exhaust available file descriptors
or (on 32 bit) run up against mmap limits.
In the rollsum case, we didn't need to hold open the "from" object
at all. And in the bsdiff case, we weren't even looking at either of
the files until we started processing.
Also, while we have the patient open, switch to using O_TMPFILE
if available.
Closes: #567
Approved by: giuseppe
Private Cloudfront instances return 403 for objects which don't exist
rather then a 404.
Change the fetcher to assume 403 is ok for download that are "optional"
rather then erroring out at that step (e.g. trying to download a static
delta if the remote repo doesn't have those)
Closes: #531
Approved by: cgwalters
Optionally read cookie jars for a remote to be used when downloading
data. This can be used for private repositories which require specific
cookies to be present, e.g. repositories hosted on Amazon cloudfront
using signed cookies.
Closes: #531
Approved by: cgwalters
We should just download the commit objects directly, as it's
obviously a lot more efficient than deltas.
I had to generate a summary file in more places in the tests,
since once created, it needs to be updated.
Closes: https://github.com/ostreedev/ostree/issues/528Closes: #566
Approved by: jlebon
I was doing a chain of mirroring like A -> B -> C
And repo B had A as a remote. When I added B as
a remote to C, the summary file of B had a ref
upstream:foo/bar/baz, which caused all pulls from
B to C to fail, since the summary file is only
expected to have refs, not refspecs.
Closes: https://github.com/ostreedev/ostree/issues/561Closes: #565
Approved by: jlebon
Various bootloader add kernel commandline options dynamically, filter
these out when grabbing boot options from /proc/cmdline. Specifically
grub adds BOOT_IMAGE and systemd-boot adds initrd.
Closes: #560
Approved by: cgwalters
Found by valgrind memcheck. g_variant_new_from_bytes takes a ref to the
bytes, so we need to release the original ref.
Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #556
Approved by: cgwalters
ostree_repo_pull_with_options() and ostree_repo_remote_change() don't
sink floating GVariant arguments, and doing so now would be an
ABI change; so don't rely on them to do so.
Leak found with valgrind memcheck.
Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #556
Approved by: cgwalters
g_variant_get_strv is (transfer container): the caller is expected to
free the array, but not the individual strings.
Leak found with valgrind memcheck.
Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #556
Approved by: cgwalters
It appears the result of assign_bootserials() is never actually used,
but I haven't changed it to return void right now.
Leak found with valgrind memcheck.
Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #556
Approved by: cgwalters
The cleanup attribute doesn't clean up before calling a noreturn
function like exit(). Explicitly clean up the pointer variables
(but don't assume that a simple g_object_unref() would be OK either,
in case the behaviour of the cleanup attribute changes).
This isn't a real leak since we're about to exit anyway, but if we
don't fix it then valgrind memcheck will make the tests fail.
Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #559
Approved by: cgwalters
We are relying on the GPtrArray to free its contents, but we only
give it a free-function on one code path.
Found by valgrind memcheck.
Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #559
Approved by: cgwalters
If !existed, then we add arg to kargs->order, where it will be freed
by that array's free-function. However, if the kernel argument did
already exist, we have to either free arg ourselves (and make sure
the old key is what appears in the hash table), or do a linear search
on kargs->order to replace the old key with the new.
Leak found by valgrind memcheck.
Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #559
Approved by: cgwalters
Improve the heuristic to use only the part before the first '.' when
looking for a similar file in the current directory.
last versions of dracut generate reproducible initramfs files, but we
still fallback to the full file download if there is any minimal
change that causes a different checksum and file name.
This change extends that case to deal better with similar files that
have a different suffix.
This is the difference generating a static delta from
fedora-atomic/f24/x86_64/docker-host to fedora-atomic/f24/x86_64/testing/docker-host
before the patch:
fallback for 111ec866aa7ce3688407fa4a1ae7c9fca93dcee0b851fc9434c59ff947830cc7 (47.0 MB)
fallback for c6a898265de22b02c89ea2f35d132628d0ee1c0a058052ed14fee5799c17904c (47.0 MB)
fallback for fbce656249ece77260887ed873e445561b9d43bcb28a32e759c0b1bab89e7137 (6.6 MB)
fallback for cfdb51457e47e0a0fe0bac38991a21279d2646ff2f019630c7b52a0cd3451397 (6.6 MB)
part 0 n:1972 compressed:11239809 uncompressed:33747412
part 1 n:1079 compressed:9683681 uncompressed:55641397
part 2 n:1507 compressed:15050265 uncompressed:44448838
part 3 n:101 compressed:1865881 uncompressed:31896086
part 4 n:278 compressed:2452585 uncompressed:52811323
part 5 n:18 compressed:67621 uncompressed:100220
uncompressed=218645276 compressed=40359842 loose=545102
rollsum=49 objects, 2117254 bytes
bsdiff=4067 objects
after the patch:
part 0 n:843 compressed:19844109 uncompressed:95443178
part 1 n:1223 compressed:11188609 uncompressed:33330401
part 2 n:990 compressed:15762905 uncompressed:61214132
part 3 n:1441 compressed:20614573 uncompressed:31534195
part 4 n:163 compressed:2734997 uncompressed:51356423
part 5 n:285 compressed:2480813 uncompressed:52902904
part 6 n:14 compressed:59125 uncompressed:75341
uncompressed=325856574 compressed=72685131 loose=533283
rollsum=51 objects, 57235332 bytes
bsdiff=4073 objects
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Closes: #554
Approved by: cgwalters
If we have a partial commit it is not an error for a dirmeta to be
missing (in fact, that is likely), so instead of returning a not-found
error from ostree_repo_traverse_commit() we ignore the error and
continue.
In particular, this means we don't stop early at the first
missing dirmeta, which previously caused ostree_repo_prune() to
thing the dirmetas after that to be unreached and thus purged.
Also, we remove the special casing in ostree_repo_prune() to
not report errors for commitpartial, because these should not
be reported anymore.
This fixes https://github.com/ostreedev/ostree/issues/541Closes: #542
Approved by: cgwalters
If there is a transaction active, then we put writes to detached
metadata into the staging dir, and when reading it we look there
first. This allows transactions to be aborted half-way without
writing the detached metadata into the repository (possibly
overwriting any old metadata from there).
This fixes https://github.com/ostreedev/ostree/issues/526Closes: #539
Approved by: giuseppe
If the detached metadata is not in the repo, try in the parent
repo if that is set.
Without this a commit will not gpg validate in the child repo
Closes: #539
Approved by: giuseppe
Commit 1d4f1b8878 started using hardlinks
checkouts of symlinks. However, symlinks are not stored as symlink in the
repo for bare-user repos, so this breaks user-mode checkouts of such repos.
We fix this by checking for !is_symlink in the bare-user case.
This fixes:
https://github.com/ostreedev/ostree/issues/537Closes: #538
Approved by: giuseppe
I was noticing a recent performance issue with checkouts
which seemed to be mostly us going back to doing a `fsync()` on
directories.
Regardless, while looking at that, I saw we were spending time
creating new symlinks. Even though symlinks are small, it's still
better to hardlink them.
Going way back in time, the reason we weren't doing this is
because we were hitting `EMFILE` on ext4, but that was for
gnome-continuous which creates *many* build roots. Even
there though, they're just a cache, and we handle `EMFILE`.
For ostree-for-host-system, we don't expect to have many roots (just 3
at most transiently), so hardlinking symlinks does make sense.
Closes: #521
Approved by: jlebon
Just noticed this while inspecting the code. The deployments retrieved
by `_ostree_sysroot_list_deployment_dirs_for_os` will forcibly already
have a matching osname since it indirectly uses that same variable to
construct them. Having a check there makes it look like there may be
subtle corner cases, when there aren't.
Closes: #529
Approved by: cgwalters
If this is true, don't initiate, abort of commit a transaction, instead
it is assumed that the caller initiated the transaction, and that it
will eventually be commited.
This allows you to do multiple pulls or a combination of pulls and
commits in a single transaction.
Closes: #525
Approved by: cgwalters
This pulls in a new compilation flag for wrpseudo compatibility. Also
note we need to add some includes since glnx-libcontainer went away,
and with it some includes for `sys/mount.h` etc.
Closes: #522
Approved by: cgwalters
We were seeing some weird potential memory corruption in this code
when using it for `rpm-ostree-toolbox installer`, which is almost
certainly not its fault, but let's use it as an excuse to port
(mostly) to fd-relative and away from GFile.
Dropping the last GFile use here is a bit tricky as it does have a
nice high level wrapper around inotify.
Closes: #512
Approved by: jlebon
My previous change in https://github.com/ostreedev/ostree/pull/425
actually broke things so we basically used the repository defaults =(
This is a subtle mess since we're only trying to flip things off
for the *new* API.
Clean this up so that the "default repo inheritance" lives only in one
place - in the compat layer for the old checkout API. The new
checkout API defaults to off period, so the repository state is
irrelevant.
Closes: #520
Approved by: jlebon
/var/log is another one of those core directories that should be made
available and properly labeled during early boot before tmpfiles.d
starts up.
Related: https://bugzilla.redhat.com/show_bug.cgi?id=1265295Closes: #513
Approved by: cgwalters
While looking at a slow update issue (which I'm guessing is
unpredictable I/O latency in an OpenStack instance), I noticed
in one of the traces we were inside a fsync here.
Dropping the fsync here is just another of a long series of unwinding
them - we `syncfs()` the sysroot fd and `/boot` and we have a big
`sync()` anyways.
Closes: #508
Approved by: jlebon
More fsync pruning. Since we have a public API for writing the origin
file and it did a fsync before, let's preserve that. But when writing
deployments as part of a full transaction, we rely on the global
`syncfs()`, so add an internal function for origin file writing that
doesn't.
Closes: #509
Approved by: giuseppe
If ostree_sysroot_unload() was called explicitly, then sysroot_fd
would be closed again at finalization time, possibly closing a
random file descriptor belonging to some other part of the
application.
Closes: #507
Approved by: cgwalters
0 was used as an "unset" flag for tmp_dir_fd, which is technically
incorrect. For cache_dir_fd, -1 was used as the sentinal but 0
was checked for, resulting in close(-1).
Closes: #507
Approved by: cgwalters
In https://github.com/ostreedev/ostree/pull/408 we fixed a
bug where we would crash when trying to execute deltas into
an archive repo (which isn't presently supported).
But that was overly aggressive - we obviously *can* execute deltas
when mirroring into a bare repo. This should fix a regression with
the way flatpak uses mirroring to pull from a user repo into the
system.
Closes: #506
Approved by: alexlarsson
This way e.g. flatpak can detect which timestamp it should use.
See `flatpak/common/flatpak-utils.c:flatpak_zero_mtime`.
Closes: #501
Approved by: cgwalters
Since forever, we've been doing two cleanups. In
8ece4d6d51
I thought we were doing just one and wanted to go to zero (if specified),
but I actually just dropped one cleanup.
In https://github.com/projectatomic/rpm-ostree/pull/452
@jlebon pointed out the duplication. Fix this by creating a new internal
deploy wrapper that takes cleanup flags.
(Since we already had the "piecemeal cleanup" API internally, let's
frame it in terms of that, rather than passing down a boolean).
Closes: #500
Approved by: jlebon
If the current repo is already up to date (we have no content to
fetch), it's possible for the fetcher to not request any URIs. So
create and then finalize it quickly.
Finalization involves calling `g_main_loop_quit()` +
`g_thread_wait()`. However, if `g_main_loop_quit()` is run *before*
`g_main_loop_run()`, we'll deadlock because GMainLoop assumes in
`_run()` to start things.
This is a common trap - ideally, GMainLoop would record if `_quit()`
was called before `_run()` or something, but doing that now would
likely break people who are expecting quit() -> run() to restart.
In general, we've moved in various GLib-consuming apps to an
explicit "main context iteration with termination condition" model;
see `pull_termination_condition()` in the pull code.
This fixes this race condition.
I verified that an assertion in `_finalize` that more than
zero URIs were requested was hit in multiple test cases, and this patch
has survived a while of make check loops.
Closes: https://github.com/ostreedev/ostree/issues/496Closes: #499
Approved by: jlebon
This should help avoid address space exhaustion on 32 bit systems,
and in general is obviously going to improve efficiency.
Closes: #498
Approved by: alexlarsson
Quoting Dan Nicholson in
<https://github.com/ostreedev/ostree/pull/330#issuecomment-245499099>
mtime of 0 has been the semantics of ostree deployments from basically
the beginning of the project. We (and others, see
flatpak/flatpak@b5204c9) rely on that fact when generating trees.
In particular, this affects caches that use the mtime of the
associated file or directory to determine if the cache is valid. By
arbitrarily changing the mtime of the files to something else, all
the caches we setup in the build are now invalidated. Preseeding
caches is really important to the user experience as it avoids
having the user wait while they're regenerated on first run.
Now, we could change our build infrastructure to preset all the
mtimes to 1 to match this change, but what does that do for our
existing users who are on an ostree that deploys with mtimes of 0?
We could just revert this change at Endless (and the associated one
in Flatpak), and that would be fine for our users. However, if we
point non-Endless users to our apps, they'll have the great
experience of waiting 10 seconds the first time they launch it while
the fontconfig cache is rebuilt unnecessarily.
Closes: #495
Approved by: jlebon
musl libc's implementation of `realpath` works by opening the path and then
doing a lookup in `/proc/self/fd` to find the canonical path. This fails
if `/proc` is not mounted. This causes problems for us if
`ostree-prepare-root` is `init` as `/proc` won't be mounted.
We have to mount `/proc` anyway for `/proc/cmdline` so this fix just
expands the scope over which `/proc` is mounted to include both our
`realpath` calls.
See also:
* http://www.openwall.com/lists/musl/2016/06/08/2 and
* http://git.musl-libc.org/cgit/musl/tree/src/misc/realpath.c?id=e738b8cbe64b6dd3ed9f47b6d4cd7eb2c422b38dCloses: #485
Approved by: cgwalters
I've seen it fail with musl which needs `/proc` to be mounted for it to
work. The error messages we're rather confusing before. At least this
now points to the right location.
Closes: #485
Approved by: cgwalters
While converting the mirrorlist code from using GSList to GPtrArray, I
completely missed the fact that there is now a much cleaner way to do
this.
Closes: #484
Approved by: cgwalters
This commit adds mirrorlist support to the fetcher. Users can now
prepend url or/and contenturl by mirrorlist= to interpret the link as a
mirrorlist.
If an object is not found, the fetcher will automatically try the next
mirror in the order given in the list (assuming the order returned by
the server is significant).
Closes: #469
Approved by: cgwalters
This made sense back when we used a main loop even when we needed to
fetch objects synchronously. Nowadays, we no longer actually update
progress before the FETCHING_OBJECTS phase, which is only for async
requests.
This allows us to get rid of fetch_uri_contents_membuf_sync() and to
generalize fetch_uri_contents_utf8_sync() so that it only requires a
fetcher. This will be needed later.
Closes: #469
Approved by: cgwalters
I've found this useful when monitoring multiple logs at the same time to
test the upcoming content & meta URL splitting.
Closes: #469
Approved by: cgwalters
Allow users to pass a --contenturl during `remote add` and store it in
the remote config.
Fish out the contenturl setting from the remote config and use it when
downloading static deltas and objects (except for commit signatures).
The idea here is that items in the trust chain (summary & sigs) can be
fetched from a more secure e.g. TLS-pinned location, while objects
themselves are fetched from another location. Once mirrorlist support is
added, this use-case will become even more advantageous.
Closes: #469
Approved by: cgwalters
OSTree function ostree_repo_pull_with_options starts a
series of operations that makes heavy use of the PullData's
cancellable.
This isn't effective, however, since nowhere in the code
the OtPullData.cancellable field is set. This is visible,
for example, when trying to cancel a Flatpak pull and nothing
happens, because the cancellable is not properly passed
to the pull data.
Fix that by setting the cancellable field of the pull data. It
owns a reference for safety reasons, and unreferences it at the
end of the operation.
ostreedev/ostree#482
Closes: #483
Approved by: cgwalters
This simplifies the build system by removing the need for
`libswitchroot-mountutil.la`.
Original idea by @cgwalters in #477.
Closes: #478
Approved by: cgwalters
The `warn()` libc extension has exactly the same behaviour as our own
`perrorv` function, but is available in (at least) glibc and musl. As an
added bonus the similar function `err()` which will exit with an error
code afterwards.
This implementation is tidier and allows us to get rid of our own
`perrorv`. It paves the way to removing `ostree-mount-util.c` to simplify
the build scripts.
Original idea by @cgwalters in #477.
Closes: #478
Approved by: cgwalters
POSIX and GNU define conflicting versions of `strerror_r`. The GNU
version returns the string but doesn't necessilary write into buf.
The POSIX version writes into buf and returns the length but doesn't
necessilary append a terminate the string with a NUL if it's too long
to fit in buf.
This commit fixes building ostree-prepare-root with musl libc. The
stripped static build with musl on my machine is 30K vs. 724K with glibc
static and 11K with glibc shared.
Closes: #477
Approved by: cgwalters
We hold a fd open on this, and it's basically now expected
to be immortal. Confer that status.
This was showing up in flatpak crashers, because we'd get
an unexpected errno.
(I didn't test this fixes the crasher, but it's clearly right)
https://bugzilla.redhat.com/show_bug.cgi?id=1347293Closes: #476
Approved by: alexlarsson
I noticed seeing the output of `prune` twice in rpm-ostree, and had
always wondered why. When reading the rpm-ostree code to fix
something else, reasons, I noticed the reason - we were pruning once
here, and then once after rpm-ostree regenerates its "base" refs.
There's no reason to clean twice, so let's add a flag so rpm-ostree
can suppress doing it inside libostree.
Closes: #474
Approved by: giuseppe
It's close-on-exec, not close-on-fork. I was clearly confused when
writing this; it works just fine to reference the fd in the child and
`fchdir()` before exec. So drop the unnecessary duplication.
Just noticed this while reading the code for a random other reason.
Closes: #473
Approved by: giuseppe
This is a proper fix for:
https://bugzilla.gnome.org/show_bug.cgi?id=755787
With this patch, an admin (system builder) can now:
1) Edit /usr/lib/ostree-boot/uEnv.txt
2) Deploy the new tree. OSTree will append system's uEnv.txt
to the OSTree's managed uEnv.txt (loader/uEnv.txt).
It is common for u-boot systems to read in an extra env
from external /uEnv.txt. The same file OSTree uses to pass
in its env. With this patch /uEnv.txt now contains OSTree's
env + custom env added by system builders.
Closes: #466
Approved by: cgwalters
Fixes this warning:
src/libostree/ostree-repo-pull.c:2162: Warning: OSTree: ostree_repo_pull_with_options: unknown parameter 'remote_name_or_baseurl' in documentation comment, should be 'remote_name'
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Closes: #472
Approved by: jlebon
When doing a prune, we should not try to delete objects in parent
repos, since it'll fail. There is a bigger discussion about the
semantics of `parent=` to be had, but this will fix trying to use
`ostree prune --repo=/ostree/repo/extensions/rpmostree/pkgcache`.
Closes: https://github.com/ostreedev/ostree/issues/467Closes: #471
Approved by: jlebon
In CentOS, these happened to appear in a repo that is served
via rsync, and having them not be world-readable caused mirroring
tools to fail.
They aren't secret, so don't make them so.
Closes: #468
Approved by: giuseppe
We used to only look at the "gpg-verify" and "gpg-verify-summary"
options when we're passed a local URL. Make these options also have an
effect in the configured remote case.
Closes: #465
Approved by: cgwalters
Programs like `useradd` try to `open(/etc/passwd, O_RDWR)` to append,
which didn't work with rofiles-fuse. Thinking about this, I realized
that there's a simpler algorithm for "can we write to this file" which
is "does it have a hardlink count <= 1"?
Switching to this both drops complexity (we no longer need to keep a
hash table of files we created), and also lets useradd work.
Closes: #462
Approved by: jlebon
The documentation says this is ignored, implying that you should pass
NULL to it. However, the function immediately returns in this case even
though the argument isn't used anywhere.
Closes: #458
Approved by: cgwalters
Why not to use libexecdir?
Because this directory does not exist on some distros or paths
between distros varies. There are several reasons why a well
known path is prefered, for example when generating a kernel
command line:
init=${ostree}/usr/lib/ostree-boot/ostree-prepare-root
In addition this saves us some typing in a console when wanting
to access the "ostree" cmd line.
Closes: #449
Approved by: cgwalters
For local pulls there's no benefit pulling the static delta over the
individual object files since there's no HTTP overhead. Furthermore,
processing deltas always generates the objects whereas a standard pull
ensures that the exact object files are copied. Using deltas also
prevents hardlinking the objects if the repos exist on the same
filesystem.
Closes: #447Closes: #448
Approved by: cgwalters
Add the --require-static-deltas option like pull to ensure static deltas
are processed for local pulls.
Closes: #447Closes: #448
Approved by: cgwalters
You need to supply an argument to F_DUPFD_CLOEXEC or fcntl will return
EINVAL. Use 3 as the minimum fd number as is standard.
Closes: #454Closes: #448
Approved by: cgwalters
Lots and lots of preparation led to this moment - when nothing
apparent changes for users! Woo!
But seriously, having the extra dependency is a minor annoyance, and
in the big picture I think the libgsystem idea was wrong - we need to
land things in GLib, and use git submodules for API-unstable or
Linux-specific sharing. For a lot of OSTree, the libgsystem `GFile*`
orientation was also wrong, we really want fd-relative.
Closes: #444
Approved by: jlebon
I hit an error with [CAHC](https://wiki.centos.org/SpecialInterestGroup/Atomic/Devel)
where we were doing time-based pruning. `ostree summary -u` started failing,
and it took me a bit to realize it was because we were pruning
even the tip of old branches, which I was not at all expecting,
and I don't think users will too.
Perhaps in the future we could add some sort of --prune-ref-tips or
something if people wanted it, but I doubt it.
Closes: #445
Approved by: jlebon
Since we already had a "recursive copy" implementation here, let's
reuse it rather than the libgsystem `gs_shutil_cp_a()`. Part of the
libglnx porting.
Closes: #428
Approved by: jlebon
It handles ownership of the `DIR*` for us more cleanly, and
is just a better API.
This is in preparation for further changes to this code to do SELinux
labeling while copying.
Closes: #428
Approved by: jlebon
I forgot to actually remove `config_file` in the previous
commit, the txn lock hasn't been used in a long time, and
for the uncompressed cache, everything uses the fd already.
Closes: #433
Approved by: giuseppe
I think we'd end up closing stdin...yuck. Need to investigate
having this be fatal, but CentOS 7 `gcc-4.8.5` doesn't understand
`-Werror=int-conversion`.
Closes: #430
Approved by: giuseppe
We have a better API now, drop use of the internal helper, which also
depended on libgsystem.
This required bumping libglnx to pull in a fix.
Closes: #429
Approved by: giuseppe
Since this is a new API, and adding booleans is the most likely thing
we'll do, let's stick some explicit padding for them in here now.
We could use the `unused_ints[]` but it'd be out of order, and this
will more clearly remind people about the padding. The efficiency hit
versus bitfields is annoying, but oh well, not a real world problem.
Closes: #427
Approved by: giuseppe
Since we're adding a new API, we have the opportunity to fix
the defaults. We expect clients to do a `syncfs()` or equivalent
on their own now, since it's way more efficient.
Flip the checkout fsync default to off.
Closes: #425
Approved by: giuseppe
This was the last use of libgsystem tmpfile APIs. The change here is
a bit uglier than it needs to be because we support creating the delta
in place as well as in the repo, so we needed to abstract over
locations.
Closes: #424
Approved by: giuseppe
Inlining this is uglier, but this code kind of sucks anyways =( We're
going to need to do some fd-relative porting in here at some point.
Closes: #424
Approved by: giuseppe
In one case, we already had relative fds and hence this was
nicer. Unfortunately the other areas got uglier. More fd-relative
porting to do later.
Closes: #424
Approved by: giuseppe
This allows ostree-prepare-root outside of the initramfs context where the
real rootfs is already mounted at /. We can't use `mount --move` in this
case because we would be trying to move / into a subdirectory of itself.
Closes: #403
Approved by: cgwalters
...for simplicity. This way we don't need to keep concatenating
deploy_path to everything. We can just refer relative to the current
working directory.
We need to do this after bind-mounting it over itself otherwise our cwd
is still on the non-bind-mounted filesystem below.
Closes: #403
Approved by: cgwalters
Typically we have our ready made-up up root at
`/sysroot/ostree/deploy/.../` (`deploy_path`) and the real rootfs at
`/sysroot` (`root_mountpoint`). We want to end up with our made-up root
at `/sysroot/` and the real rootfs under `/sysroot/sysroot` as systemd
will be responsible for moving `/sysroot` to `/`.
We need to do this in 3 moves to avoid trying to move `/sysroot` under
itself:
1. `/sysroot/ostree/deploy/...` -> `/sysroot.tmp`
2. `/sysroot` -> `/sysroot.tmp/sysroot`
3. `/sysroot.tmp` -> `/sysroot`
This is a refactoring to group all these operations together so I can
implement an alternative in terms of `pivot_root`.
Closes: #403
Approved by: cgwalters
This supports running ostree on embedded platforms without an initrd.
Specificially I'm trying to do bringup on an NVidia Tegra based Jetson TK1
dev board.
Closes: #403
Approved by: cgwalters
In general we want to support "idempotentcy" or "state
synchronization" across interruption. If a repo is only partially
created due to a crash or whatever, it's hard for a user to know that.
Let's just make `ostree_repo_create()` idempotent. Since all we're
doing is a set of `mkdirat()` invocations, it's quite simple.
This also involved porting to fd-relative, which IMO makes the
code a lot clearer.
Closes: #422
Approved by: 14rcole
If a static delta is generated between 2 commits with the same content,
then the delta will contain 1 part with no checksums. While useless,
this is a valid delta that shouldn't raise an assertion. If the delta
part has no checksums, then there are no objects to recreate and the
processing can be skipped.
Closes: #420
Approved by: cgwalters
These two functions are not safe for gobject introspection, so annotate
them to be skipped:
1) ostree_repo_import_archive_to_mtree
2) ostree_repo_export_tree_to_archive
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Closes: #417
Approved by: cgwalters
Provide a gobject introspection safe version for
`ostree_repo_checkout_tree_at'.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Closes: #417
Approved by: cgwalters
In general this is even cleaner now, though it was better after I
extracted a helper function for the "write tempfile with contents"
bits that were shared between metadata and regular file codepaths.
Closes: #369
Approved by: jlebon
When reworking the ostree core [to use O_TMPFILE](https://github.com/ostreedev/ostree/pull/369),
I hit an issue in the way the untrusted delta codepath ends up trying
to re-open the file to checksum it. That's not possible with
`O_TMPFILE` since the fd (which we opened `O_WRONLY`) is the only
accessible reference to the content.
Fix this by changing the delta processing code to update a checksum as
we're doing writes, which is also faster, and ends up simplifying the
code as well.
What would be an even larger simplification here is if we e.g. used a
separate thread calling `write_object()` or something like that; the
main issue I see there is somehow bridging the fact that function
wants a `GInputStream*` but the delta code is generating stream of
writes.
Closes: #392
Approved by: jlebon
This one is a bit subtle; we're generating a hash that contains
pointers to the strings we parsed, so we need to carefully track
ownership.
Closes: #410
Approved by: giuseppe
This shows up as a leak in valgrind; the callee isn't sinking. In
general through the power of cleanup attributes we can do explicit
cleanup rather than relying on floating refs.
Closes: #410
Approved by: giuseppe
Right now our valgrind runs also end up valgrinding the
`trivial-httpd` code, so while it doesn't matter, let's fix this leak
anyways. We need to avoid calling `_exit()` since that won't run the
cleanup functions.
Closes: #410
Approved by: giuseppe
We don't presently support this, since the static delta code assumes
it can just `mmap()` file objects. We could at some point implement
this, but for now just skip executing deltas when doing
`archive -> archive` mirroring.
I noticed this when trying to mirror a repo in Jenkins in
[CAHC](https://wiki.centos.org/SpecialInterestGroup/Atomic/Devel).
Closes: #408
Approved by: giuseppe
The checksum_b64_inplace variants can't be used in bindings. Provide
versions that allocate and return the output rather than working on a
passed in buffer. These can then be used in GI bindings to get the
ostree modified base64 encodings.
Closes: #398
Approved by: cgwalters
Where we're referencing local files. It's what
ostree_repo_write_directory_to_mtree() is doing internally anyways,
and I consider the `GFile*` based APIs to be generally deprecated for
the most part.
Closes: #394
Approved by: giuseppe
When trying to switch ostree to `O_TMPFILE`, I hit the fact that
by default it uses mode `000`. It still works to write to the
open fd of course, but it *doesn't* work to set xattrs because
that code path for some reason in the kernel checks the mode bits.
This only broke for bare-user repos where we tried to set the xattr
before calling `fchmod()`, so just invert those two operations.
Closes: #391
Approved by: jlebon
Conceptually the session thread owns the session, so let's clear out
everything predictably there, rather than sometimes having it happen
on the main thread.
Also, this moves up clearing the pending/outstanding queues *before*
we unreference the session, since conceptually they need to reference
it as well.
Based on a patch from: Matthew Barnes <mbarnes@redhat.com>
Closes: #383
Approved by: mbarnes
I don't think this fixes the bug I was seeing, but it makes me more
comfortable to know we have a strong ref to the main context across
the thread lifetime, and we only unset the default right before
we go away.
If something in `thread_closure_unref()` used
`g_main_context_get_thread_default()` for example it'd be wrong
before.
Closes: #383
Approved by: mbarnes
Systems like pulp may want to keep retrying in a loop if the server
throws a (hopefully transient) 500, and we need test coverage of
handling these errors versus our existing 404 and 206 coverage.
Closes: #383
Approved by: mbarnes
This should fix the memory leaks in #352
This is a subset of the changes, the other part is in my pull code rewrite
Closes: #382
Approved by: cgwalters
G_DEFINE_AUTOPTR_CLEANUP_FUNC is a new function in GLib 2.44, but
libglnx contains a backported version of it. A few source files were
however using G_DEFINE_AUTOPTR_CLEANUP_FUNC either without including
libglnx.h, or without including it early enough.
This fix is similar to the one in commit d368624.
Closes#376Closes: #377
Approved by: smcv
I often want to have "idempotent" systems that iterate to a known
state. If after generating a commit, the system is interrupted, I'd
like the next run to still generate a delta. But we don't want to
regenerate if one exists, hence this option.
Closes: #375
Approved by: jlebon
I've seen a few people hit this and wonder why checkouts are slow/take
space. Really, ensuring this happens is the *point* of OSTree.
Physical copies should be a last resort fallback for very unusual
situations (one of those is rpm-ostree checking out the db since
librpm doesn't know how to read from libostree).
Even I hit the fact that `/var` is a mountpoint disallowing hardlinks
with `/ostree` once and was confused. =)
Add this to the rofiles-fuse test case because it creates a mount
point.
Closes: #368
Approved by: jlebon
The program is called ro*files* and ostree creates physical
copies of directories, so changing them is fine.
I hit this when trying to do a copy checkout onto an rofiles-fuse
mount.
Closes: #368
Approved by: jlebon
Add the functionality to use the same name for refs in local and remote
repos. This helps users keep track of local refs of remote origin, much
like local and remote git branches.
Previously, when a local ref is specified, resolve_refspec would fall
back to searching through remote repos if the ref is not found locally.
This function now takes an extra flag to specify whether it should
search through remote repos. Additionally, ostree_repo_resove_rev_ext
was added to call resolve_refspec with fallback_remote being false, so
refs --create would no longer complain when trying to create a local
ref of the same name as a remote one.
Fix remote repo parsing not being handled correctly on refs --create.
Closes: #363
Approved by: jlebon
In flatpak i was using a parent repo, and it failed to update
with ENOENT when dispatching an set-read-source opcode, because the
object it referenced was in the parent repo.
This fixes that by making _ostree_repo_read_bare_fd look
at parent_repo.
Closes: #362
Approved by: cgwalters
We noticed that once a ref folder is created, there is no existing
command that can remove it. For example, once "foo/bar" is created,
even if the user deletes foo or all the refs under foo, the folder
will persist.
Now when the user attempts to create a ref "foo" either through commit
or refs --create, if a folder "foo" exists but is empty of refs, the
folder is removed and the new ref "foo" is created.
New unit tests in tests-ref.sh verify this functionality.
Closes: #354
Approved by: cgwalters
Import `gs_file_enumerator_iterate()` for the next six months or
so...after RHEL 7.3 is released I'm strongly considering hard
requiring 2.46 or so.
Likely at some point we should figure out how to share more "glib
backport" code with NetworkManager at least.
Closes: #341
Approved by: jlebon
This was the last caller of libgsystem that isn't
`gs_file_get_path_cached()`. I think the use case ostree has where
the same code can be called via command line and via a shared library
*and* via a daemon is rather unusual, so let's just copy the code for
logging from libgsystem into here.
For example rpm-ostree hard depends on a daemon mode, so it'll just
use `sd_journal` directly.
Closes: #341
Approved by: jlebon
I was easily reproducing a hang on pulls with thousands of requests on
current git master. The initial symptom seemed to be that there are
multiple code paths where we don't invoke
`session_thread_process_pending_queue()`. We really need to do
that any time we remove something from the outstanding queue,
to ensure it gets filled again.
A further issue is that we were tying the lifecycle of the pending
object to the `GTask`, but the task could be unref'd from the main
thread (via a `GSource` on the main thread), and that introduced
threadsafety issues, because the hash table and other data suddenly
could be concurrently modified.
Both of these need to be fixed together. First, we introduce
`Arc<Pending>`, and ensure that both the main and worker threads hold
references.
Second, we ensure that we re-process the queue *immediately* whenever
a task is done, inside the worker thread, rather than doing it
incidentally via an unref. This architecture is quite similar to what
the outside pull code is doing.
Closes: #350
Approved by: jlebon
The previous code was subject to a divide by zero if less than a
second had passed. Rework it so we only do the divide if more than a
second has passed.
Closes: #349
Approved by: Mathnerd314
I find the "-z2" is really a long ago relic of the past when I changed
the format. We no longer have anything to do with the original
`archive`, so let's start allowing people to type `--mode=archive`
which just looks saner.
At some point later I'll update the docs too, but it'll be an annoying
transition period as we'll have to say "On older OSTree, use -z2" etc.
Closes: #346
Approved by: giuseppe
Added the ability to create a ref (much like a git tag) for an
existing commit through "ostree refs EXISTING --create=NEWREF".
Previously the only way to create a new ref was by creating a new commit,
but refs --create allows multiple refs to point to the same commit.
The command will fail if:
- None/more than one existing ref is specified
- The specified EXISTING tag does not exist, or was not specified
- The specified NEWREF already exists, or is the name of a folder
Add unit tests in tests-ref.sh to verify above functionality
Closes: #340
Approved by: jlebon
Just noticed this while reading some code, we didn't have many manual
`out: close()` bits left, this pushes us over the edge to autocleanup
almost everywhere.
Closes: #332
Approved by: jlebon
The recent memleak fixes motivated me to look at the bitrotted code to
run invocations of `ostree` in the test suite underneath valgrind.
There are a few things here. First, update suppressions file from
libhif, since I recently worked on it.
When running *uninstalled* as we now support, we need
`libtool --mode=execute` in the mix so it expands out to
the uninstalled binary and we don't valgrind the intermediate shell.
However, it's harder than that because we chdir into a tmpdir,
which defeats the libtool logic. AFAICS, the only fix for this
is to determine the realbin path before we chdir, and then unfortunately
we need to change every use of `ostree` to `${OSTREE}` =(
Then this immediately breaks for me on RHEL7 because my ancient
copy of `valgrind-3.10.0-16.el7.x86_64` is unaware of syscall 306, i.e.
`syncfs`.
But let's do this first before I dive into that.
Closes: #292
Approved by: krnowak
As the docs say, `g_regex_match()` still allocates a match even if it
returns `FALSE`. Using `g_autoptr` is just plain better.
Closes: #292
Approved by: krnowak
This changes around a few things that didn't work for me:
* Section names seem to be ostree-* instead of libostree-*
* Also XML files are ostree-* (they didn't show up at all)
- gtk-doc doesn't seem to parse const _OSTREE_PUBLIC correctly
* pull documentation is now on the actual functions rather than stubs
* Update gitignore with some more files
And there some changes to make gtk-doc give fewer warnings (not finished)
Closes: #327
Approved by: cgwalters
This centralizes the ifdef's in one file, which will make it
easier to write new pull backends.
ostree-repo-pull.c is now built unconditionally
Closes: #327
Approved by: cgwalters
1 is a better choice than 0 because some programs use 0
as a special value; for example, GNU Tar warns of an
"implausibly old timestamp" with 0.
Closes: #330
Approved by: cgwalters
This kills another GSystem consumer...I think down the line I'd like
to do something like "detect whether file is > 1k if so, mmap,
otherwise just readall()" so we can use this helper in more places.
Closes: #319
Approved by: jlebon
We have a lot of "allow_noent" type wrapper functions since
a common pattern is to allow files to not exist, but still
throw cleanly on other issues.
This is another instance of that, and cleans up duplicated error
handling code.
Part of this is prep for moving away from `GFile` consumers.
Closes: #319
Approved by: jlebon
On 32-bit systems the modifier for printing 64bit values should be
%llu instead of %lu. Just use appriopriate macros that do the right
thing.
Closes: #329
Approved by: giuseppe
The --gpg-sign parameter will be used for signing the summary, not the
commit. This probably was copy-pasted from the ostree commit command.
Closes: #320
Approved by: cgwalters
The "no atime" thing was mostly useful only before "relative atime"
updates landed. Users who care about performance will turn it off
entirely anyways.
Closes: #316
Approved by: jlebon
In addition to generic fd relative porting,
this is a necessary preparatory step for libglnx porting, because
when I tried to use `g_mapped_file_new` I hit an issue with
it using a different error domain from GIO.
Thankfully libglnx consistently uses the GIO error domain, and here
we're now using it for the `open()` call.
Closes: #317
Approved by: jlebon
This is similar to changes Krzesimir has been doing recently - we
really don't need the ergonomics of floating refs since we have
autocleanups.
We should continue to change most of our code to sink refs.
Specifically here it was pretty broken that the `_map()` API was
sinking but the other two weren't, and this broke some refactoring I
was trying to do later.
Closes: #317
Approved by: jlebon
Bug 765429 said that not having a time estimate can be annoying
when working with large pulls.
There isn't any complex time estimation logic here - we just take
the number of bytes remaining and do a linear projection of
the bytes per second rate at the current point in time.
Closes: #318
Approved by: cgwalters
I happened to have the github page open with my commit after it was
merged, and then noticed a bug.
We still need to throw if we hit a non-`ENOENT` error.
Closes: #315
Approved by: giuseppe
This can be useful for validating the 3rd party data that is put in
the extensions directory and is signed with the same keys as commits
or the summary file.
Closes: #310
Approved by: cgwalters
Moved out setting up a GPG verifier to a separate function, as I would
like to use it for the any data verification function in the following
commit.
Closes: #310
Approved by: cgwalters
I plan to add a function for verifying any data which may return the
error about lack of trusted signatures, so let's avoid the redundancy
and put the check in the separate function.
Closes: #310
Approved by: cgwalters
It is quite similar to the already existing
ostree_raw_file_to_content_stream function, so I factored the common
part to a separate function. The difference is that we cannot report
the size of the resulting stream.
Can be useful for serving a "bare" repository as a faked "archive-z2"
repository.
Closes: #308
Approved by: cgwalters
This resolves the ambiguity for a message with a body but no subject,
and also hints that OSTree is using two bytes to store the empty
commit subject/body (when instead they could be stored, or rather,
not stored, as part of the metadata)
Closes: #305
Approved by: cgwalters
In practice, a lot of subjects are empty, because the commit date
and branch are sufficient identification. For example, rpm-ostree
does not use subjects. It also doesn't use the command-line ostree
commit tool, so this was not a problem there, but this makes the
behavior consistent.
Also adds a test that empty subjects and omitting the subject
are equivalent. The --timestamp is so that the commits do not
have different timestamps.
Closes: #305
Approved by: cgwalters
This allows automation to prefill the subject/body for the editor.
It could also be useful if some kind of "amend commit" functionality
was implemented.
Closes: #305
Approved by: cgwalters
Previously, the behavior was to run the editor if no subject or body was specified.
This led to using commit -s '' in scripts, to avoid running the editor.
In practice, manually editing commit messages is rare, whereas automated scripting
is common, so instead default to not running the editor and only run it when -e
is given.
Closes: #305
Approved by: cgwalters
Various places need to include libglnx.h for the autoptr backport
fallbacks to be there before ostree-autocleanups.h is included.
This fixes the build on centos7·
Closes: #309
Approved by: giuseppe
There's no need to allocate the variant builder on a heap, so allocate
it on the stack and avoid a memory leak at the same time.
Closes: #307
Approved by: cgwalters
To avoid breaking the ostree users we hide the autoptr setup behind an
OSTREE_WITH_AUTOCLEANUPS variable. The variable is set to 0 by
default, so users that want to start using them, have to define the
variable themselves to 1.
Probably after some transition period, the variable will be set 1 by
default.
This is conditional on GLib >= 2.44.0 being available. It does not
bump ostree's dependency on GLib.
Closes: #295
Approved by: cgwalters
Apparently I got the bracketing wrong in
862e6ecdcc58f025696b1394adfc0fcf7322df23:
src/libostree/ostree-repo.c: In function 'ostree_repo_delete_object':
src/libostree/ostree-repo.c:3538:11: warning: missing braces around
initializer [-Wmissing-braces]
g_auto(GVariantBuilder) builder = {0,};
Closes: #298
Approved by: cgwalters
When we added enum type generation, the generated symbols used
the `ostree_` prefix, and at the time that implied they were public.
So we started (if built with libsoup) exporting
`ostree_fetcher_config_flags_get_type`.
I think it's not worth confusing ABI checkers, so let's export the
dummy symbol forever, and switch enums to be private by default.
We should revisit this and also export some auto-generated enum
types for public enums, but that's a separate patch.
Closes: #296
Approved by: gatispaeglis
This tries to avoid leaking GVariantBuilders and GVariants in some
situations. The leaks were usually happening when some error occurred
or because of unclear variant ownership situation.
The former is mostly about making sure that g_variant_builder_clear is
called on builders that didn't finish their variant building process.
The latter is surely more work - sometimes the result of
g_variant_builder_end() should not be passed directly to a function,
but rather stored in a g_autoptr(GVariant), sunk and then passed to a
function. IMO, with an advent of g_autoptr, GVariants should be always
sunk instead of relying on some receiver function sinking it. This
would make an easy-to-follow policy of always sinking your
variants. Functions could then assume that the passed variant is
already sunk. These leaks are still happenning in commands, but they
are less harmful, since that code will not be used by some daemon as a
library routine.
Closes: #291
Approved by: cgwalters
This adds a _with_options variant of the
ostree_repo_remote_fetch_summary function, so we can tell the fetcher
to use a specific URL instead taking it from the remote config.
Closes: #290
Approved by: cgwalters
The base URI created in this line was always (sans erroneous
situations) overwritten in the code block below without freeing it
previously, so it leaked.
Closes: #290
Approved by: cgwalters
Use markdown to actually format the tuple member descriptions as a
list. Currently they are hardly legible as gtk-doc formats them into a
single sentence.
Also update the additional metadata docs in the summary objects.
Closes: #289
Approved by: cgwalters
It's very useful for third-party applications to have someplace to store
their data guaranteed to be on the same device as the repo (thus
ensuring hardlinks) while still being shielded away from any of OSTree's
timely garbage collections.
We create a new "extensions/" subdirectory where apps can include
whatever they wish in "extensions/myapp/". This subdirectory is
completely unmanaged by ostree.
NB: I didn't bother making it a member of the OstreeRepo proper since we
don't really use it for anything else yet.
Closes: #286
Approved by: cgwalters
The deployments and booted deployment were never unreffed.
Also, remote (void) from the call to the unload function - it returns
nothing, so there is no need to fool the compiler.
Closes: #287
Approved by: cgwalters
I was briefly looking at building with clang mostly since it
detects unused variables with cleanup attributes, but then
I hit this fatal error.
It's a hard compile error with `-Wformat-nonliteral` since clang
doesn't know it's a format string.
This was already supported by the commit modifier API, just needed to
expose it. This will also be used to test the libarchive API in a future
test.
Closes: #275
Approved by: cgwalters
- Make hardlink handling more generic. The previous strategy worked for
tar archives, but not for cpio. It now works for both.
- Add support for SEL labeling (through the OstreeRepoCommitModifier)
- Add support for xattr_callback (through the OstreeRepoCommitModifier)
- Add support for filter (through the OstreeRepoCommitModifier)
- Add a use_ostree_convention option
Closes: #275
Approved by: cgwalters
I ended up deciding to move this one into libglnx, seems like
something other libglnx-using software might want to do, even though
xdg-app doesn't right now.
Closes: #282
Approved by: jlebon
The filesystem commit code will never give us potentially hostile
filenames, and when importing from archives, we do some validation.
However, we should be extra paranoid and also add error messages in
the mtree in case someone tries to import a hostile
libarchive-supported format.
Closes: #283
Approved by: jlebon
We were arbitrarily only deleting content after exactly one day. Some
use cases may want something else; make it configurable.
Closes: #170
Approved by: jlebon
We had a policy of cleaning up all files in `$repo/tmp` older
than one day, but we should really clean up previous bootid staging
directories too, as they can potentially take up a lot of disk space.
https://bugzilla.gnome.org/show_bug.cgi?id=760531Closes: #170
Approved by: jlebon
Setting this causes commit to error out. There are other ways we
could do this in a more sophisticated fashion, such as via SystemTap
etc. But this has low-tech applicablity, works as non-root.
The reason I'm adding this is so that we can add test cases for
cleanup of the `tmp/staging-` directory.
Closes: #170
Approved by: jlebon
I have a cache drive I often mount read-only, and the previous commit
for opening `tmp/cache` broke since `errno == EROFS`, not `EPERM`.
It turns out we already had the concept of a "writable" repo, so just
piggy back off that.
Closes: #281
Approved by: giuseppe
To GLnxConsoleRef. There were some subtleties here, for example we
used to reference `GSConsole` inside the progress changed function,
which at first seems like an ABI hazard, because e.g. rpm-ostree or
xdg-app could still be passing a `GSConsole` instance there. Luckily,
it turns out to be compatible to just start calling libglnx here.
Another issue was that due to libglnx's use of the cleanup function,
we needed to ensure we always called `ostree_async_progress_finish()`
*before* the cleanup function was invoked.
Closes: #280
Approved by: giuseppe
Commit
1810de2b51
lost an optimization where we would try hardlinks for the
kernel/initramfs in `/boot`. This would be a noticeable space savings
on single-partition systems.
Closes: #277
Approved by: gatispaeglis
In the case we have a repo with a parent, and the child repo has a
remote called "foo", but some option is unset. Then when we look up
the parent repo for a value before using the default we will fail due
to the parent not having the "foo" remote. As soon as we find the
requested remote at some point in the hierarchy we need to ignore further
errors and use the default value.
Closes: #274
Approved by: giuseppe
We looked for and locked old temporary directories so we can
reuse them if not in use. However, once we found one that
we can reuse we didn't stop iterating, and eventually we
reached the end. This means we can lock multiple dirs.
Closes: #273
Approved by: giuseppe
There was some leftover intermediate cruft here I noticed
while reviewing another patch:
- We had an output `GFile*` for that was never used
- We required the caller to allocate the loose pathbuf, but
none of them ever reused it
- We had an extra intermediate function
Also while looking at this, I'm now uncertain whether some of the
callers of `_ostree_repo_has_loose_object` should really be invoking
`ostree_repo_has_object()`, but let's leave that aside for now.
Closes: #272
Approved by: alexlarsson
This lets you set a prefix for the resulting archive patsh.
Especially useful in combination with --subpath, for instance
--subpath=subdir --prefix=subdir to extract just subdir.
Closes: #265
Approved by: cgwalters
We keep forgetting to update `apidoc/ostree-sections.txt`, so let's
start enforcing it. Of course it turns out we had some bugs here
like symbols marked as public but never implemented, etc. Those
are fixed in the prior commits.
Closes: #263
Approved by: giuseppe
This was removed in 14d682305b - long
before we had a stable library.
I again claim removing unimplemented symbols from the header is not an
API/ABI break.
Closes: #263
Approved by: giuseppe
If a symbol falls in a git merkle tree forest, but no one hears it,
did it ever exist?
I'm claiming this isn't an API/ABI break because nothing could
have actually used this. `git log -S checksum_update_meta` leads
me to 38ef75e6e0 which was before
we really had a stable shared library.
Closes: #263
Approved by: giuseppe
This allows you to replace the default
$sysroot/$sysconfdir/ostree/repos.d string value, and to use a similar
feature for repos that are not the system repo.
In particular, this allows us to support /etc/xdg-app/remotes.d for
xdg-app.
Closes: #247
Approved by: cgwalters
in write_directory_content_to_mtree_internal dfd_iter can be NULL,
for instance if commiting from --tree=ref=FOO. Don't blindly de-ref
it to avoid crashing.
Closes: #256
Approved by: cgwalters
ostree pull-local crashed for me in thread_closure_unref () doing:
g_mutex_clear (&thread_closure->output_stream_set_lock);
Seems like we never initialize this mutex.
Closes: #254
Approved by: cgwalters
I think it's cleaner if we use `remote_repo_local` to know
that we have a local repo. In reality, it might be nicest
if we didn't even create an `OstreeFetcher` for this case,
but untangling the code is tricky.
Closes: #239
Approved by: alexlarsson
Otherwise we get undefined behaviour if the client didn't explicitly set
any flags.
Also, add documentation for all the other options supported by
ostree_repo_pull_with_options().
Closes: #252
Approved by: cgwalters
These are useful for ostree users (like xdg-app) that have custom
options for remotes. In particular they are useful when we later make them
all respect self->parent_repo.
Closes: #236
Approved by: cgwalters
Force the otherwise disabled gpg verifications on.
Note: You need to pass --remote=foo so we know what gpg keys to verify
against.
Closes: #237
Approved by: cgwalters
Not only does this not make sense from a performance perspective, but
it also doesn't work because we can't use a url as a path element.
Closes: #237
Approved by: cgwalters
ostree-grub-generator can be used to customize
the generated grub.cfg file. Compile time
decision ostree-grub-generator vs grub2-mkconfig
can be overwritten with the OSTREE_GRUB2_EXEC
envvar - useful for auto tests and OS installers.
Why this alternative approach:
1) The current approach is less flexible than using a
custom 'ostree-grub-generator' script. Each system can
adjust this script for its needs, instead of using the
hardcoded values from ostree-bootloader-grub2.c.
2) Too much overhead on embedded to generate grub.cfg
via /etc/grub.d/ configuration files. It is still
possible to do so, even with this patch applied.
No need to install grub2 package on a target device.
3) The grub2-mkconfig code path has other issues:
https://bugzilla.gnome.org/show_bug.cgi?id=761180
Task: https://bugzilla.gnome.org/show_bug.cgi?id=762220Closes: #228
Approved by: cgwalters
This is a follow up to #227 to allow ostree to open the editor also for
orphan commits when no subject or body is given on the cmdline.
Closes: #229
Approved by: cgwalters
The API supports this, and it's not hard for us to do in the command
line as well. One possible use case is separating "content
generation" in a separate server.
Related: https://github.com/ostreedev/ostree/pull/223Closes: #227
Approved by: jlebon
When I'm doing local development builds, it's quite common for me not
to want to accumulate history. There are also use cases for this on
build servers as well.
In particular, using this, one could write a build system that didn't
necessarily need to have access to (a copy of) the OSTree repository.
Instead, the build system would determine the last commit ID on the
branch, and pass that to a worker node, then sync the generated
content back.
The API supported generating custom commits that don't necessarily
reference the previous commit on the same branch, let's just expose
this in the command line for convenience.
I plan to also support this rpm-ostree.
Closes: #223
Approved by: jlebon
Now that OSTree is used as G_LOG_DOMAIN, set the main handler to match
so the appropriate messages are filtered. It would probably be more
appropriate to spell out "OSTree" in the code, but since G_LOG_DOMAIN is
being defined globally in the project, might as well reuse it here.
https://bugzilla.gnome.org/show_bug.cgi?id=764237Closes: #225
Approved by: cgwalters
If you have a repo where a needed object has been inadvertantly removed,
all you'll get is a "No such metadata object" error with no clue about
where it was referenced from.
Add some debug messages to provide clues about which objects are being
traversed and found.
https://bugzilla.gnome.org/show_bug.cgi?id=764006Closes: #224
Approved by: cgwalters
When prune fails, it can be really difficult to figure out why. This at
least lets you know which objects are being considered.
https://bugzilla.gnome.org/show_bug.cgi?id=764006Closes: #224
Approved by: cgwalters
This can be used as a fingerprint to determine whether two
OstreeSePolicy objects are equivalent.
Also add documentation for ostree_sepolicy_get_name().
Closes: #219
Approved by: cgwalters
The dirtree object is required for traversing, so don't use the
load_variant_if_exists() function. This will return a
G_IO_ERROR_NOT_FOUND to the caller rather than trying to ref a NULL
variant in ostree_repo_commit_traverse_iter_init_dirtree() if the object
is missing.
https://bugzilla.gnome.org/show_bug.cgi?id=764091
If a commit only pull has been done, then the commit object exists in
the object store in addition to the commitpartial file. Traversing this
partial commit will likely fail, but that's expected. If traverse
returns a G_IO_ERROR_NOT_FOUND in this case, continue with pruning.
https://bugzilla.gnome.org/show_bug.cgi?id=764091
I'm trying to improve the developer experience on OSTree-managed
systems, and I had an epiphany the other day - there's no reason we
have to be absolutely against mutating the current rootfs live. The
key should be making it easy to rollback/reset to a known good state.
I see this command as useful for two related but distinct workflows:
- `ostree admin unlock` will assume you're doing "development". The
semantics hare are that we mount an overlayfs on `/usr`, but the
overlay data is in `/var/tmp`, and is thus discarded on reboot.
- `ostree admin unlock --hotfix` first clones your current deployment,
then creates an overlayfs over `/usr` persistent
to this deployment. Persistent in that now the initramfs switchroot
tool knows how to mount it as well. In this model, if you want
to discard the hotfix, at the moment you roll back/reboot into
the clone.
Note originally, I tried using `rofiles-fuse` over `/usr` for this,
but then everything immediately explodes because the default (at least
CentOS 7) SELinux policy denies tons of things (including `sshd_t`
access to `fusefs_t`). Sigh.
So the switch to `overlayfs` came after experimentation. It still
seems to have some issues...specifically `unix_chkpwd` is broken,
possibly because it's setuid? Basically I can't ssh in anymore.
But I *can* `rpm -Uvh strace.rpm` which is handy.
NOTE: I haven't tested the hotfix path fully yet, specifically
the initramfs bits.
In some cases (such as `ostree-sysroot-cleanup.c`), the surrounding
code would be substantially cleaner if it was also ported to
fd-relative, but I'm going to do that in a separate patch.
That way these patches are easier to review for mechanical
correctness. I used an Emacs keyboard macro as the poor man's
[Coccinelle](http://coccinelle.lip6.fr/).
⚠️ There is a notable spiked pit trap here around
`posix_fallocate()` and `errno`. This has bit other projects,
see e.g.
7bb87460e6
Otherwise the port was straightforward.
Since we hard-depend on GLib 2.40, we can start using GSubprocess.
This is part of dropping our dependency on libgsystem, which is
deprecated in favor of libglnx (as well as migrating things to GLib).
I'd like to encourage people to make OSTree-managed systems more
strictly read-only in multiple places. Ideally everywhere is
read-only normally besides `/var/`, `/tmp/`, and `/run`.
`/boot` is a good example of something to make readonly. Particularly
now that there's work on the `admin unlock` verb, we need to protect
the system better against things like `rpm -Uvh kernel.rpm` because
the RPM-packaged kernel won't understand how to do OSTree right.
In order to make this work of course, we *do* need to remount `/boot`
as writable when we're doing an upgrade that changes the kernel
configuration. So the strategy is to detect whether it's read-only,
and if so, temporarily mount read-write, then remount read-only when
the upgrade is done.
We can generalize this in the future to also do `/etc` (and possibly
`/sysroot/ostree/` although that gets tricky).
One detail: In order to detect "is this path a mountpoint" is
nontrivial - I looked at copying the systemd code, but the right place
is to use `libmount` anyways.
It used to be allowed to run something like "ostree remote refs" on
a read-only (e.g. system) repo. However, the summary cache caused that to
break. This commit just makes it not save the cache if we get some kind
of permission error when writing it. It'll still work, even without the
cache.
https://bugzilla.gnome.org/show_bug.cgi?id=763855
It allows an optimization to skip the download of the summary file
if its .sig file is unchanged.
Downloading the .sig file is much cheaper than downloading the summary
file from repositories with many branches.
https://bugzilla.gnome.org/show_bug.cgi?id=762973
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
This will allow daemons like rpm-ostree to detect if there are any new
deployments efficiently, in combination with using inotify. If there
are any changes, rpm-ostree wants publish them on DBus.
While we're here, add some changes to start doing unit C testing of
the sysroot API.
And change the command line to use it. rpm-ostree had a copy
of this code, and thus there's a clear reason to have an API.
While we're moving this into API, ensure the mtime on deploy is bumped
after an osname is created, so that daemons like rpm-ostree can notice
changes. (In reality, creating the directory should do this, but
let's be double sure)
This allows other processes (e.g. rpm-ostreed) to monitor for external
changes (e.g. if someone does `ostree admin undeploy`) in a relatively
sane fashion.
Specifically, I'm trying to fix:
https://github.com/projectatomic/rpm-ostree/issues/220
It accepts a `flags` argument to control its behavior. Differently
from `ostree_repo_list_refs`, the `refspec_prefix` is not removed from
the results.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
I plan to use this in rpm-ostree at least for two reasons:
- To find the mtime on the repo
- To use the tmp/ directory to stage content (but we should eventually
add a better API)
As rpm-ostree evolves, it keeps driving API additions to libostree.
This creates a relatively tight coupling.
However, if delivering via e.g. RPM, unless one manually remembers to
increment the `Requires:` in the spec file, it's possible for the two
to become desynchronized.
RPM handles versioned symbols and will ensure a dependency if the
application starts using a newer version.
To implement this, switch to `-fvisibility=hidden`, along with an
annotation in the header, and finally add a `.sym` file.
This matches what other projects like systemd and libvirt do.
Although rather than attempting to retroactively version symbols, glom
them all onto the current one.
I see when analyzing a delta here that due to byteswapping a negative
compression ratio of 540%, 66%, and 28%. Let's arbitrarily pick 20%
as a threshold for detecting byetswapping.
If the average object size is greater than 4GiB, let's assume we're
dealing with opposite endianness. I'm fairly confident no one is
going to be shipping peta- or exa- byte size ostree deltas, period.
Past the gigabyte scale you really want bittorrent or something.
xdg-app passed this a filename directly, and in this case it should be
used as is. This regressed to always look for "superblock" in the same
directory as the passed in filename.
https://bugzilla.gnome.org/show_bug.cgi?id=762617
Some Docker layers are just metadata in the `layer.json`. If one is
mapping Docker layers to OSTree commits, one needs to create a dummy
root directory, because OSTree doesn't support metadata-only commits.
Let's just push that logic down here because it's easier than special
casing it in higher levels.
One of the design goals with deltas was not just wire efficiency,
but also having all the data up front about how much data would
be transferred before starting.
Let's expose that better by adding a `dry-run` option to the pull API.
This requires static deltas to be useful. Basically we simply call
the progress callback once with the data from the superblock.
For a production release repository, most OS vendors would want
to just always use static deltas. Add the ability for the pulls to
require it.
(I think I'll also add a summary key for this actually in addition,
so the repo manager can force it too)
If ostree is run in a test setup where it operates as root in a tmp
directory, it might cause issues to flag the deployments as immutable.
The test harness might simply be doing an `rm -rf` (effectively the case
for gnome-desktop-testing-runner), which will then fail.
We add a new debug option to the ostree_sysroot object using GLib's
GDebugKey functionality to allow our tests to communicate to ostree that
we don't want immutable deployments.
This is a more flexible version of the previous
ostree_repo_write_archive_to_mtree() which took a file reference.
This has an extensible options structure, and in particular
now supports `ignore_unsupported_content`.
I plan to use this for importing Docker images which contain device
nodes. (There's no reason for container images to have those, so
we'll just ignore them).
Also here, just like the export variant, the caller is responsible for
setting up libarchive.
I was going to add new API for importing, and it was really confusing
that what I think of now as import and export both had "write" in the
name. It's just clearer to talk about the direction.
At the same time, include `Export` in the options structure.
This isn't an ABI break as the API isn't in a release.
I was getting a compilation error with the GCC hardening flags which
look for a missing mode with `O_CREAT`. The right fix here is to drop
`O_CREAT`, as truncate() should throw `ENOENT` if the file doesn't
exist.
I don't know why we didn't do this a long time ago. This extends the
pull API to allow grabbing a specific commit, and will set the branch
to it. There's some support for this in the deploy engine, but there
are a lot of reasons to support it for raw pulls (such as subset
mirroring cases).
In fact I'm thinking we should also have the override-version logic
here too.
NOTE: One thing I debated here is inventing a new syntax on the
command line. Git doesn't seem to have this functionality (probably
because it'd be rarely used). The '@' character at least doesn't
conflict with anything.
Anyways, I wanted this for some other test cases. Without this,
writing tests that go between different commits is more awkward as one
must generate the content in one repo, then pull downstream, then
generate more content, then pull again. But now I can just keep track
of commit IDs and do exactly what I want without synchronizing the
tests.
At the moment I'm looking at using rpm-ostree to manage RPM inputs
which can then be converted into Docker images. It's most convenient
if we can stream directly out of libostree rather than doing a
checkout + tar combination.
There are also backup/debugging etc. reasons to implement `export` as
well.
While it's not strictly tied to OSTree, let's move
https://github.com/cgwalters/rofiles-fuse in here because:
- It's *very* useful in concert with OSTree
- It's tiny
- We can reuse OSTree's test, documentation, etc. infrastructure
One thing to consider also is that at some point we could experiment
with writing a FUSE filesystem for OSTree. This could internalize a
better equivalent of `--link-checkout-speedup`, but on the other hand,
the cost of walking filesystem trees for these types of operations is
really quite small.
But if we did decide to do more FUSE things in OSTree, this is a step
towards that too.
Now we display stats on the individual parts, such as the blob size
and the number of each type of opcode. Most interesting to me is
things like how many bsdiff opcodes there are vs new objects, etc.
We had code to deal with opening/checksumming/decompressing static
deltas in a few places. I'd like to teach `ostree static-delta show`
how to display more information, and this will allow it to just use
`_ostree_static_delta_part_open()` too.
Right now though, almost all of the details of deltas are private, so
we can't do the "honest thing" and have the command line just use the
shared library.
Eventually some of this should appear in the API, but for now add
command line which is useful for debugging.
I'd like to incrementally convert all of `ostree-repo*.c` to
fd-relative usage, so that we can sanely introduce
`ostree_repo_new_at()` which doesn't involve GFile.
This one is medium risk, but passes the test suite.
The original intention here was that we'd keey around a copy of the
file so that grub2 could eventually learn how to do atomic updates by
checking for a "fully written" marker in the *new* file, and if it
didn't exist, falling back to grub2.cfg.old.
I haven't yet proposed that upstream, but we might as well stop
deleting the file since it's useful as a backup at least.
Reported-by: Gatis Paeglis
This is a better followup to dc9239dd7b
since I wanted to do fsync-less checkouts in rpm-ostree too, and
replicating the "turn off fsync temporarily" was in retrospect just a
hack.
We can simply add a boolean to the checkout options.
https://github.com/GNOME/ostree/pull/172
Originally, a lot of the `fsync()` calls here were added for the
wrong reason - I was chasing a bug that ended up being the extlinux
bootloader not parsing 64 bit ext4 filesystems. But since it looked
like corruption, I tried adding a lot more `fsync()` calls.
All we should have to do is use `syncfs()`. If that doesn't work,
it's a kernel bug.
I'm making this change because skipping the individual fsyncs can be a
major performance win - it's easier for the FS to optimize, we do more
in parallel, etc.
https://bugzilla.gnome.org/show_bug.cgi?id=757117
A fast way to generate new OSTree content using an existing
tree is to checkout (as hard links), add/replace files, then
call `ostree_repo_scan_hardlinks()`, then commit.
But `ostree_repo_scan_hardlinks()` scans the entire repo, which
can be slow if you have a lot of content.
All we really need is a mapping of (device,inode) -> checksum
just for the objects we checked out, then use that mapping
for commits.
This patch adds API so that callers can create a mapping via
`ostree_repo_devino_cache_new()`, then pass it to
`ostree_repo_checkout_tree_at()` which will populate it, and then
`ostree_repo_write_directory_to_mtree()` can consume it.
I plan to use this in rpm-ostree for package layering work.
Notes:
- The old `ostree_repo_scan_hardlinks()` API still works.
- I tweaked the cache to be a set with the checksum colocated with
the key, to avoid a separate malloc block per entry.
https://github.com/GNOME/ostree/pull/167
The logic for checking which bootversion to use tries to access
sysroot->bootversion if the user didn't specify an explicit bootversion
on the command-line nor through the env var. However, at that point, the
sysroot object is not yet initialized, so it will always return 0, even
when it's 1.
This would cause e.g. `grub2-mkconfig` to have no output for the BLS
entries whenever the entries were under `/boot/loader.1`.
Related: RHBZ1293986
The tmp directory is lazily created for each fetcher instance, since
it may require superuser permissions and some instances only need
_ostree_fetcher_request_uri_to_membuf() which keeps everything in
memory buffers.
This is a continuation of earlier work to drop the individual fsync on
files/directories in favor of relying on `syncfs()` for speed.
As part of that cleanup, I'm porting it to be fd-relative.
I feel relatively confident about this change given that this area of
the code has notable test suite coverage, although that code runs as
non-root.
I'm porting the deployment code to be fd-relative, but part of the
logic was using `GFile` to talk to `OstreeRepoFile` to determine the
"bootcsum" (boot config checksum) before checking out the file tree.
We can avoid having both code paths by checking out the tree first,
then looking at it on the filesystem.
Extract existing code from ostree_repo_prune and add an argument COMMIT,
that controls which commit purge. If not set, the old behavior is kept.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Philip Withnall
Jonathan Lebon
Colin Walters
Dan Nicholson
Sjoerd Simons
Francesco Giannelli
Daniel J Walsh
Anton Gerasimov
Alexander Larsson
Erik Larsson
Georges Basile Stavracas Neto
André Klitzing
Christian Hergert
Gatis Paeglis
Krisztian Litkey
Paul van Tilburg
Mario Sanchez Prada
Jasper St. Pierre
William Manley
Sjoerd Simons
Simon McVittie
Giuseppe Scrivano
Owen W. Taylor
Mathnerd314
Bastien Nocera
Yu Qi Zhang
Krzesimir Nowak
Sam Spilsbury
Gatis Paeglis
Stef Walter
Matthew Barnes