This adds a `MS_SILENT` flag to all `mount(2)` calls, reducing the
amount of kernel logs produced on each boot.
Those messages do not contain actionable details, and in the "mount
plus read-only remount" case they can easily become highly redundant.
In general, we're probably going to need to change most of our
`g_return_if_fail` to `g_assert`. The analyzer flags that
the function can return `NULL`, but the caller isn't prepared for
this.
In practice, let's abort.
In general, we're probably going to need to change most of our
`g_return_if_fail` to `g_assert`. The analyzer flags that
the function can return `NULL`, but the caller isn't prepared for
this.
In practice, let's abort.
In general, we're probably going to need to change most of our
`g_return_if_fail` to `g_assert`. The analyzer flags that
the function can return `NULL`, but the caller isn't prepared for
this.
In practice, let's abort.
In general, we're probably going to need to change most of our
`g_return_if_fail` to `g_assert`. The analyzer flags that
the function can return `NULL`, but the caller isn't prepared for
this.
In practice, let's abort.
This defines `OstreeRepoAutoTransaction` as a boxed type, in order
to support auto-generating bindings for it.
That first requires adding internal reference-counting to it, to
allow freely copying/freeing references to a single transaction guard.
This rewords errors and log messages in the functions which take care
of preparing sysroot in initramfs.
Depending on the boot flow, it is possible to reach this logic
with a sysroot mounted (unexpectedly) as read-only.
In that case, let's clearly point out the problematic mountpoint.
This enhances the auto-transaction logic, augmenting the scope of a
transaction guard.
It allows committing or aborting a transaction through its guard.
It also supports tracking the completion status of a transaction
guard, avoiding double commits/aborts, while retaining the auto-cleanup
logic.
https://bugzilla.redhat.com/show_bug.cgi?id=1945274 is an issue where a privileged
kubernetes daemonset is writing a socket into `/etc`. This makes ostree upgrades barf.
Now, they should clearly move it to `/run`. However, one option is for us to
just ignore it instead of erroring out. Some brief investigation shows that
e.g. `git add somesocket` is a silent no-op, which is an argument in favor of ignoring it.
Closes: https://github.com/ostreedev/ostree/issues/2446
This is nicer than having the caller parse the commit
object, or indirect via the `OstreeRepoFile*` object of the root.
Will be used in ostree-rs-ext around tar parsing.
This is part of `OstreeCommitModifier`, but I'm not using
that in some of the ostree-ext Rust code.
It just makes more sense as a direct policy API, where it should
have been in the first place. There's already support for
setting a policy object on a commit modifier, so that's all the
old API needs to do now.
There's a general Unix philosophy that "silence is golden".
However, when one is explicitly invoking an error check it's nice
to see explicit success.
We already print various statistics, so ending with a happy
note has no extra cost.
The logic for `--selinux-policy` ended up in the `--tree=dir`
path, but there's no reason for that. Fix the imported
labeling with `--tree=tar`. Prep for use with containers.
We had this bug because the previous logic was trying to avoid
duplicating the code for generic `--selinux-policy` and
the case of `--selinux-policy-from-base --tree=dir`.
It's a bit more code, but it's cleaner if we dis-entangle them.
This will be helpful for the "ostree native container" work in
https://github.com/ostreedev/ostree-rs-ext/
Basically in order to reuse GPG/signapi verification, we need
to support adding a remote, even though it can't be used via
`ostree pull`. (At least, not until we merge ostree-rs-ext into ostree, but
even then I think the principle stands)
for deltafiles the legacy_transaction_resuming flag is not used,
which will mark the commit as done, even if files are missing.
using already existing commitstate_is_partial function as fix
We're waaay overdue for this, it's been the default
in rpm-ostree for years, and solves several important bugs
around not capturing `/etc` while things are running.
Also, `ostree admin upgrade --stage` (should) become idempotent.
Closes: https://github.com/ostreedev/ostree/issues/2389
We have a bunch of APIs to do GPG verification of a commit,
but that doesn't generalize to signapi. Further, they
require the caller to check the signature status explicitly
which seems like a trap.
This much higher level API works with both GPG and signapi.
The intention is to use this in things that are doing "external
pulls" like the ostree-ext tar import support. There we will
get the commitmeta from the tarball and we want to verify it
at the same time we import the commit.
Followup to PRs related to https://github.com/ostreedev/ostree/issues/2410
Since the test suite now covers this the test was failing on
a Fedora SELinux enabled host where we see `security.selinux`
even if not in the commit.
I was seeing an `EPERM` here which was confusing.
It turned out the real error was `EEXIST`.
Since we're referring to the original error, but we do a
lot of computation in the middle, we need to save errno.
This fixes some aspects of OstreeRepoAutoTransaction and re-aligns
it with the logic in flatpak. Specifically:
* link to the underlying repo through refcounting
* bridge internal errors to warning messages
* verify the input pointer type
This is a preparation step before exposing this logic as a public API.
As pointed out in the original review, `gpg-list-keys` fits better
alongside the existing `gpg-import`.
Changes were done with:
```
git grep -l list-gpg-keys | xargs sed -i 's/list-gpg-keys/gpg-list-keys/'
for src in $(git ls-files '*list-gpg-keys*'); do
dst=${src/list-gpg-keys/gpg-list-keys}
git mv "$src" "$dst"
done
```
Calculate the advanced and direct update URLs for the key discovery
portion[1] of the OpenPGP Web Key Directory specification, and include
the URLs in the key listing in ostree_repo_remote_get_gpg_keys(). These
URLs can be used to locate updated GPG keys for the remote.
1. https://datatracker.ietf.org/doc/html/draft-koch-openpgp-webkey-service#section-3.1
This will be used to implement the PGP Web Key Directory (WKD) URL
generation. This is a slightly cleaned up implementation[1] taken from
the zbase32 author's original implementation[2]. It provides a single
zbase32_encode API to convert a set of bytes to the zbase32 encoding.
I believe this should be acceptable for inclusion in ostree. The license
in the source files is BSD style while the original repo LICENSE file
claims the Creative Commons CC0 1.0 Universal license, which is public
domain.
1. https://github.com/dbnicholson/libbase32/tree/for-ostree
2. https://github.com/zooko/libbase32
This provides a wrapper for the `ostree_repo_remote_get_gpg_keys`
function to show the GPG keys associated with a remote. This is
particularly useful for validating that GPG key updates have been
applied. Tests are added, which checks the
`ostree_repo_remote_get_gpg_keys` API by extension.
This function enumerates the trusted GPG keys for a remote and returns
an array of `GVariant`s describing them. This is useful to see which
keys are collected by ostree for a particular remote. The same
information can be gathered with `gpg`. However, since ostree allows
multiple keyring locations, that's only really useful if you have
knowledge of how ostree collects GPG keyrings.
The format of the variants is documented in
`OSTREE_GPG_KEY_GVARIANT_FORMAT`. This format is primarily a copy of
selected fields within `gpgme_key_t` and its subtypes. The fields are
placed within vardicts rather than using a more efficient tuple of
concrete types. This will allow flexibility if more components of
`gpgme_key_t` are desired in the future.
Currently the verifier decides whether to include the global keyrings
based on whether the specified remote has its own keyring or not. Allow
callers to exclude the global keyrings even when that's not the case.
This will be used in a subsequent commit in order to get the GPG keys
only associated with a remote.
In order to use the GPG verifier, it needs to be seeded with GPG keys
after instantation. Currently this is only used for verifying data, but
it will also be used for getting a list of trusted GPG keys in a
subsequent commit.
In configure the systemd unit path is optional, but in the code it's
assumed to be defined. Add an `#ifdef` that throws an error when it's
not defined like the handling of `HAVE_LIBMOUNT` below it.
We struggled for a long time with enablement of our "internal units",
trying to follow the philosophy that units should only be enabled
by explicit preset.
See https://bugzilla.redhat.com/show_bug.cgi?id=1451458
and https://github.com/coreos/rpm-ostree/pull/1482
etc.
And I just saw chat (RH internal on a proprietary system sadly) where
someone hit `ostree-remount.service` not being enabled in CentOS8.
Thinking about this more, I realized we've shipped a systemd generator
for a long time and while its only role until now was to generate `var.mount`,
but by using it to force on our internal units, we don't require
people to deal with presets anymore.
Basically we're inverting things so that "if ostree= is on the kernel
cmdline, then enable our units" and not "enable our units, but have
them use ConditionKernelCmdline=ostree to skip".
Drop the weird gyrations we were doing around `ostree-finalize-staged.path`
too; forking `systemctl start` is just asking for bugs.
So after this, hopefully we won't ever again have to think about
distribution presets and our units.
This will be ignored, so let's make it very clear
people are doing something wrong. Motivated by a bug
in a build pipeline that injected `/var/lib/rpm` into an ostree
commit which ended up crashing rpm-ostree because it was an empty db
which it wasn't expecting.
It *also* turns out rpm-ostree is incorrectly dumping content in the
deployment `/var` today, which is another bug.
Use `g_error` and `g_assert*` rather than `g_return*` when checking the
locking preconditions so that failures result in the program
terminating. Since this code is protecting filesystem data, we'd rather
crash than delete or corrupt data unexpectedly.
`g_error` is used when the error is due to the caller requesting an
invalid transition like attempting to pop a lock type that hasn't been
taken. It also provides a semi-useful message about what happened.
Previously each thread maintained its own lock file descriptor
regardless of whether the thread was using the same `OstreeRepo` as
another thread. This was very safe but it made certain multithreaded
procedures difficult. For example, if a main thread took an exclusive
lock and then spawned worker threads, it would deadlock if one of the
worker threads tried to acquire the lock.
This moves the file descriptor from thread local storage to the
`OstreeRepo` structure so that threads using the same `OstreeRepo` can
share the lock. A mutex guards against threads altering the lock state
concurrently.
Fixes: #2344
This simplifies the lock state management considerably since the
previously pushed type doesn't need to be tracked. Instead, 2 counters
are kept to track how many times each lock type has been pushed. When
the number of exclusive locks drops to 0, the lock transitions back to
shared.
Previous to this we'd trip an assertion `abort()` deep in the curl code if e.g.
a user did `ostree remote add foo htttp://...` etc.
Motivated by considering supporting "external remotes" where code outside
ostree does a pull, but we want to reuse the signing verification infrastructure.
Several tests generate summaries and then expect to use the generated
summary immediately. However, this can cause intermittent test failures
when they inadvertantly get a cached summary file. This typically
happens when the test is run on a filesystem that doesn't support user
extended attributes. In that case, the caching code can only use the
last modified time, which only has 1 second granularity. If tests don't
carefully manage the summary modification times or the repo cache then
they are likely subject to races in some test environments.
This introduces an environment variable `OSTREE_SKIP_CACHE` that
prevents the repo from using a cache directory. This is enabled by
default in tests and disabled for tests that are a explicitly trying to
test the caching behavior.
Fixes: #2313Fixes: #2351
This API is push rather than pull, which makes it much more
suitable to use cases like parsing a tar file from external
code.
Now, we have a large mess in this area internally because
the original file writing code was pull based, but static
deltas hit the same problem of wanting a push API, so I added
this special `OstreeRepoBareContent` just for writing regular
files from a push API.
Eventually...I'd like to deprecate the pull based API,
and rework things so that for regular files the push API
is the default, and then `write_content_object()` would
be split up into archive/bare cases.
In this world the `ostree_repo_write_content()` API would
then need to hackily bridge pull to push and it'd be
less efficient.
Anyways for now due to this bifurcation, this API only
works on non-archive repositories, but that's fine for
now because that's what I want for the `ostree-ext-container`
bits.
Continuation of the addition of `ostree_repo_write_regfile_inline()`.
This will be helpful for ostree-rs-ext and importing from tar, it's
quite inefficient and awkward for small files to end up creating
a whole `GInputStream` and `GFileInfo` and etc. for small files.
When working on ostree-ext and importing from tar, it's
quite inefficient and awkward for small files to end up creating
a whole `GInputStream` and `GFileInfo` and etc. for small files.
Plus the gtk-rs binding API to map from `impl Read` to Gio
https://docs.rs/gio/0.9.1/gio/struct.ReadInputStream.html
requires that the input stream is `Send` but the Rust `tar` API
isn't.
This is only 1/3 of the problem; we also need similar APIs
to directly create a symlink, and to stream large objects via
a push-based API.
I was going to add some new API and I noticed that this function
never returns an error; presumably at one point it did, but
not anymore. It simplifies the code flow noticeably
to remove that.
Enable support for setting and getting xattrs. Allow modifications
to xattrs only on user.ima xattr.
Signed-off-by: Stefan Berger <stefanb@linux.ibm.com>
Co-authored-by: Colin Walters <walters@verbum.org>
GRUB starting with version 2.02 permits the use of the linux and
initrd commands for both EFI boot in *-efi installations, as
well as 32-bit BIOS boot in i386-pc installations.
This makes the use of the -16 and -efi suffixes for BIOS and EFI
boot obsolete on systems with a modern GRUB installation.
The --with-modern-grub configure flag makes ostree use the
unsuffixed linux/initrd commands when generating a GRUB
configuration, while defaulting to the previous behaviour for
users not wanted this option.
I was doing some rpm-ostree work and I wanted to compare two
OSTree commits to see if the kernel has changed. I think
this should be a lot more natural.
Add `ostree commit --bootable` which calls into a new generic
library API `ostree_commit_metadata_for_bootable()` that
discovers the kernel version and injects it as an `ostree.linux`
metadata key. And for extra clarity, add an `ostree.bootable`
key.
It's interesting because the "core" OSTree layer is all about
generic files, but this is adding special APIs around bootable
OSTree commits (as opposed to e.g. flatpak as well as
things like rpm-ostree's pkgcache refs).
Eventually, I'd like to ensure everyone is using this and
hard require this metadata key for the `ostree admin deploy`
flow - mainly to prevent accidents.
Currently if a file path contains a special character such as '\', and
that character is encoded into a file:// URI that is passed to
ostree_repo_pull_with_options(), the percent encoding will remain in the
path passed to g_file_new() (in the case of backslash %5C) and the pull
will then fail with a file not found error. This is an important edge
case to handle because by default on many Linux distributions a
filesystem with no label is mounted at a path based on its UUID, and
this is then passed to systemd-escape by Flatpak (when
--enable-auto-sideloading was used at compile time) to create a symbolic
link such as this which contains backslashes:
$ ls -l /run/flatpak/sideload-repos/
total 0
lrwxrwxrwx 1 mwleeds mwleeds 55 Mar 9 14:21
'automount-run-media-mwleeds-29419e8f\x2dc680\x2d4e95\x2d9a31\x2d2cc907d421cb'
-> /run/media/mwleeds/29419e8f-c680-4e95-9a31-2cc907d421cb
And Flatpak then passes libostree a file:// URI containing that path, to
implement sideloading (pulling content from the USB drive).
This results in an error like:
Error: While pulling app/org.videolan.VLC/x86_64/stable from remote
flathub:
/run/flatpak/sideload-repos/automount-run-media-mwleeds-29419e8f%5Cx2dc680%5Cx2d4e95%5Cx2d9a31%5Cx2d2cc907d421cb/.ostree/repo:
opendir(/run/flatpak/sideload-repos/automount-run-media-mwleeds-29419e8f%5Cx2dc680%5Cx2d4e95%5Cx2d9a31%5Cx2d2cc907d421cb/.ostree/repo):
No such file or directory
This patch avoids such errors by using g_file_new_for_uri() instead of
g_file_new_for_path(), so that GLib handles the %-decoding for us.
Bug report by user:
https://community.endlessos.com/t/can-not-install-vlc-from-usb-drive-3-9-3/16353
If the `summary_sig_not_modified` branch is taken above, both
`signatures` and `summary` are loaded from the cache. This makes the
`_ostree_repo_load_cache_summary_if_same_sig()` call below redundant (it
checks `signatures` matches the file it was just loaded from, and then
loads `summary` again) — but that call also currently overwrites
`summary` without clearing the old value.
Fix this by only making that call if `signatures` was retrieved, but the
server said the local `summary` cache was invalid.
Signed-off-by: Philip Withnall <pwithnall@endlessos.org>
We have an `allow_noent` boolean that controls this, but
were missing the `(nullable)` annotation, so the Rust bindings
panic when the ref doesn't exist instead of being `Option<GString>`.
The local pull path was erroring on any missing commit, but that
prevents a depth pull where the source repo has truncated history. As in
the remote case, this also tries to pull in a tombstone commit if the
source repo supports it.
Fixes: #2266
When pulling with depth, missing parent commits are ignored. However,
the check was applying to any commit, which means that it would succeed
even if the requested commit was missing. This might happen on a
corrupted remote repo or when using ref data from a stale summary.
To achieve this, the semantics of the `commit_to_depth` hash table is
changed slightly to only ever includes parent commits. This makes it
easy to detect when a parent commit is being referenced (although there
is a minor bug there when multiple refs are being pulled) while keeping
references to commits that need their `commitpartial` files cleaned up.
It also means that the table is only populated on depth pulls, which
saves some memory and processing in the common depth=0 case.
Fixes: #2265
The recent change in https://github.com/coreos/fedora-coreos-config/pull/659
broke some of our tests that do `mount -o remount,rw /sysroot` but
leave `/boot` read-only.
We had code for having `/boot` read-only before `/sysroot` but
in practice we had a file descriptor for `/sysroot` that we opened
before the remount that would happen later on.
Clean things up here so that in the library, we also remount
`/boot` at the same time we remount `/sysroot` if either are readonly.
Delete the legacy code for remounting `/boot` rw if we're not in
a mount namespace. I am fairly confident most users are either
using the `ostree` CLI, or they're using the mount namespace.
Just like we hold a fd for `/sysroot`, also do so for `/boot`
instead of opening and closing it in a few places.
This is a preparatory cleanup for further work.
I was being very conservative initially here, but I think it's
really safe to just unconditionally set up the mount namespace.
This avoids having to check twice for a read-only `/sysroot`
(once in the binary and once in the library).
In some cases such as backups or mirroring you may want to pull commits
from one repo to another even if there commits that have incorrect
bindings. Fixing the commits in the source repository to have correct
bindings may not be feasible, so provide a pull option to disable
verification.
For Endless we have several repositories that predate collection IDs and
ref bindings. Later these repositories gained collection IDs to support
the features they provide and ref bindings as the ostree tooling was
upgraded. These repositories contain released commits that were valid to
the clients they were targeting at the time. Correcting the bindings is
not really an option as it would mean invalidating the repository
history.
The cache shouldn't be affected by the user passing in some other
summary as it may not be the "official one".
I ran into this in flatpak where the passed summary was correct, but
the re-saving of the cache updated the mtime of the cached file which
led to later http If-Modified-Since calls failing to update.
As detailed in
gitlab.gnome.org/GNOME/glib/-/issues/600#note_877282, volatile
isn't actually needed in these contexts because the atomic operations
already give us strong enough guarantees. In GCC 11, this triggers a
diagnostic due to the volatile qualifier getting dropped anyway.
There is a WIP to do the same in glib:
https://gitlab.gnome.org/GNOME/glib/-/merge_requests/1719
This obsoletes this downstream patch:
https://src.fedoraproject.org/rpms/ostree/c/b8c5a6fb
I got:
src/libostree/ostree-repo.c:5232: Warning: OSTree: ostree_repo_gpg_sign_data: unknown parameter 'out_signature' in documentation comment, should be 'out_signatures'
The timeout timer should always be one-shot, so let's just always
destroy it in the callback. The main context has its own ref on it, so
it won't be freed behind its back.
This *should* fix a leak that was brought up in
https://bugzilla.redhat.com/show_bug.cgi?id=1891761.
Reported-by: Milan Crha <mcrha@redhat.com>
...with the `sysroot.bootloader` configuration option. This can be useful
when converting a system to use `ostree` which doesn't currently have a
bootloader configuration that `ostree` can automatically detect, and is
also useful in combination with the `--sysroot` option when provisioning a
rootfs for systems other than the one you're running `ostree admin deploy`
on.
It's easier to extend and it centralises the config parsing. In other
places we will no longer need to use `g_str_equal` to match these values,
a `switch` statement will be sufficient.
Luca BRUNO
Colin Walters
Simon McVittie
Jonathan Lebon
Buddelmann, Richard RB
Dan Nicholson
刘建强
Dan Nicholson
Christian Kellner
Timothée Ravier
Stefan Berger
Felix Krull
Kenneth J. Miller
Phaedrus Leeds
Philip Withnall
OpenShift Merge Robot
William Manley
Alexander Larsson