This prevents writing content into 'bare-split-xattrs` repository,
while carving some space for experimenting via a temporary
`OSTREE_EXP_WRITE_BARE_SPLIT_XATTRS` environment flag.
This adds two new object types for storing xattrs separately from
content objects.
`.file-xattrs` are regular files storing xattrs content, encoded as
GVariant. Each object is keyed by the checksum of its content, allowing
for multiple references.
`.file-xattrs-link` are hardlinks which are associated to file objects.
Each object is keyed by the same checksum of the corresponding file
object. The target of the hardlink is an existing file-xattrs object.
In case of reaching the limit of too many links, this object could be
a plain file too.
Recently we have noticed exceedingly long execution times
for multiple invocations of ostree prune. This is a result of
calculating full reachability on each invocation.
The --commit-only flag provides an alternative strategy. It will only
traverse and delete commit objects to avoid the more expensive
reachability calculations. This allows us to chain multiple --commit-only
commands cheaply, and then follow with a more expensive ostree prune
invocation at the end to clean up orphaned meta and content objects.
This patch makes it so that we mark the .commit file from a static delta
as partial before writing the commit to the staging directory. This
exactly mirrors what we do in meta_fetch_on_complete() when writing the
commit on that codepath, which should lend some credibility to the
correctness of this patch.
I have checked that this fixes an issue Flatpak users have been
encountering (https://github.com/flatpak/flatpak/issues/3479) which
results in error messages like "error: Failed to install
org.freedesktop.Sdk.Extension.texlive: Failed to read commit
c7958d966cfa8b80a42877d1d6124831d7807f93c89461a2a586956aa28d438a: No
such metadata object
8bdaa943b957f3cf14d19301c59c7eec076e57389e0fbb3ef5d30082e47a178f.dirtree"
Here's the sequence of events that lead to the error:
1. An install operation is started that fetches static deltas.
2. The fetch is interrupted for some reason such as network connectivity
dropping.
3. The .commit and .commitmeta files for the commit being pulled are
left in the staging dir, e.g.
"~/.local/share/flatpak/repo/tmp/staging-dfe862b2-13fc-49a2-ac92-5a59cc0d8e18-RURckd"
4. There is no `.commitpartial` file for the commit in
"~/.local/share/flatpak/repo/state/"
5. The next time the user attempts the install, libostree reuses the
existing staging dir, pulls the commit and commitmeta objects into
the repo from the staging dir on the assumption that it's a complete
commit.
6. Flatpak then tries to deploy the commit but fails in
ostree_repo_read_commit() in flatpak_dir_deploy(), leading to the
error message "Failed to read commit ..."
7. This happens again any subsequent time the user attempts the install,
until the incomplete commit is removed with "flatpak repair --user".
I will try to also add a workaround in Flatpak so this is fixed even
when Flatpak links against affected versions of libostree.
I'm working on enhancing the ostree-rs-ext test suite and I hit
a bug where walking a mtree and creating a parent would fail to
load lazy intermediate directories, e.g.:
/ -> usr -> bin
If we walked we'd load `/` but keep `usr` lazy, and then invalidation
would crash because it wasn't loaded.
If we're going to mutate a subdir, we need to have all the parents
loaded.
I know this is missing tests, but...it's a bit tedious to do with
the existing C tests. Eventually soon we'll execute on merging
all 3 repos, and better share test suites.
This allows to use Secure Execution with LUKS encrypted boot disk,
key and cryptab are stored only in 'sd-boot' encrypted image.
Signed-off-by: Nikita Dubrovskii <nikita@linux.ibm.com>
If system contains ibm-z-hostkey (fetched during ignition), than
ostree generates 'sd-boot' image and reboots into Secure Execution
Signed-off-by: Nikita Dubrovskii <nikita@linux.ibm.com>
I think I'm hitting issues due to this while using the Rust bindings:
https://github.com/coreos/rpm-ostree/pull/3406#issuecomment-1033084956
The bindings for those APIs use `from_glib_full` which says:
> Because ownership can only be transferred if something is already
> referenced, this is unsuitable for floating references.
The commit metadata `version` key is well established but getting it for
a remote commit is cumbersome since the commit object needs to be
fetched and loaded. Including it in the summary additional metadata
allows a much more convenient view of what each of the remote refs
represents.
This aligns all the assertion in the module. In particular, it gets
rid of all `g_return_val_if_fail` instances which may fail without
properly setting GError to the caller.
When we clean up from an error, for example copy_file_range() failing
while we generate a static delta (perhaps caused by
https://gitlab.gnome.org/GNOME/libglnx/-/issues/3 or by a
genuine write error), we might free a variant builder that has a
non-null parent. Previously, this caused infinite recursion and a stack
overflow, repeatedly freeing the same object, but Luca Bruno suggested
that the intention here appears to have been to free the parent object.
Partially resolves https://github.com/ostreedev/ostree/issues/2525
(the other bug reported in that issue needs to be resolved by updating
libglnx to a version where libglnx#3 has been fixed).
Signed-off-by: Simon McVittie <smcv@collabora.com>
This tightens up the logic for opening a file while inspecting its
xattrs. The only codepath fetching xattrs from a FD is the one
handling 'bare' mode.
It also rearranges the else-assert flow, mostly for future-proofing.
Support for that file was added previously, but the testing lived in
rpm-ostree only. Let's add it here too.
In the process add a hidden `--lock-finalization` to `ostree admin
deploy` to make testing easier (though it could also be useful to update
managers driving OSTree via the CLI).
Otherwise, any future staged deployment will also automatically be
locked even if not requested. Likely we should fold the locking into the
primary `staged-deployment` serialized GVariant instead.
This reworks `ostree ls` top-level logic so that cancellation
tokens and error details are plumbed through all codepaths.
It also gets rid of all previous goto jumps.
We do implicitly have this data because we log timings via structured
metadata in a later journal entry, but it's quite common to lose
the structured metadata because a lot of tooling just grabs the default
syslog-compatible text from `journalctl`.
Let's be louder when we hit this case as a general rule too; I think
most people shipping ostree systems want to see if it's happening.
https://bugzilla.redhat.com/show_bug.cgi?id=2003532
Basically there's a systemd bug where it's losing the `_netdev`
aspect of Ceph filesystem mounts. This means the network is taken
down before Ceph is unmounted. In turn, our invocation of `sync()`
blocks on Ceph, which won't succeed.
And this in turn manifests as a failure to transition to the new
deployment.
I initially did this patch to just rip out the global `sync()`. I
am pretty sure we don't need it anymore. We've been doing individual
`syncfs()` on `/sysroot` and `/boot` for a while now, and those
are the only filesystems we should be touching. But *proving* that
is a whole other thing of course.
To be conservative, let's instead just add a timeout of 5s on
our invocation of `sync()`. It doesn't return any information on
success/error anyways.
To allow testing without the `sync()` invocation, we also support
a new `OSTREE_SYSROOT_OPT_SKIP_SYNC=1` environment variable. For
staged deployments, this needs to be injected via e.g. systemd unit
overrides into `ostree-finalize-staged.service`.
Implementing this is a bit hairy - we need to spawn a thread. I
debated blocking in arecursive mainloop, but I think `g_cond_wait_until()`
is also fine here.
In fixing https://github.com/coreos/rpm-ostree/pull/3323
I felt that it was a bit ugly we're installing `/usr/bin/ostree-container`.
It's kind of an implementation detail. We want users to use
`ostree container`.
Let's support values outside of $PATH too.
For example, this also ensures that TAB completion for `ost` expands
to `ostree ` with a space.
This removes a 'g_setenv()' call, which could potentially be unsafe
in a multi-thread context.
The current libselinux codebase does not seem to check for
`LIBSELINUX_DISABLE_PCRE_PRECOMPILED`, so I think this has no effects
nowadays.
Additionally, I could not find any reference to it in libselinux
git history, so I'm not sure if it ever played any role at all.
My current understanding is that this is coming from version
incompatibilities between an older libselinux in the build environment
and a newer policy (with precompiled regexs) in the target.
But from the ML discussion I found, I think it eventually got
solved in a different way, possibly by avoiding the policy binary
caches.
Refs:
* https://www.spinics.net/lists/selinux/msg14822.html
* https://github.com/ostreedev/ostree/pull/2513#discussion_r781042884
This swaps the order of a couple of input sanity checks, in order
to fix a minor memory leak due to an early-return on the error
path.
Memory for the result is now allocated only after input has been
sanity-checked.
It fixes a static analysis warning highlighted by Coverity.
This adds build-time configuration logic to automatically detect
and switch between libfuse 2.x and 3.x.
Signed-off-by: Simon McVittie <smcv@collabora.com>
Co-authored-by: Luca BRUNO <luca.bruno@coreos.com>
Fixes `Argument with 'nonnull' attribute passed null` by making
the code not exist at all anymore.
In upstream libsoup this code is gone too; it uses `GUri` from glib
which we probably could now too, but one thing at a time.
This adds some logic to detect and dispatch unknown subcommands to
extensions available in `$PATH`. Additional commands can be
implemented by adding relevant `ostree-$verb` binaries to the system.
As an example, if a `/usr/bin/ostree-extcommand` extension is provided,
the execution of `ostree extcommand --help` will be dispatched to that
as `ostree-extcommand extcommand --help`.
This is trying to address:
https://pagure.io/fedora-iot/issue/48
Basically we changed rpm-ostree to start doing a shared lock during
commit by default, but this broke because pungi is starting a process
doing a commit for each architecture, and then trying to regenerate
the summary after each one.
This patch is deleting a big comment with a rationale for why
summary regeneration should be exclusive. Point by point:
> This makes sure the commits and deltas don't get
> deleted while generating the summary.
But prune operations require an exclusive lock, which means that
data still can't be deleted when the summary grabs a shared lock.
> It also means we can be sure refs
> won't be created/updated/deleted during the operation, without having to
> add exclusive locks to those operations which would prevent concurrent
> commits from working.
First: The status quo *has* prevented concurrent commits from working!
There is no real locking solution to this problem. What we really
need to do here is regenerate the summary after each commit *or*
when the caller decides to do it and e.g. include deltas at the same
time.
It's OK if multiple threads race to regenerate the summary;
last-one-wins behavior here is totally fine.
We should only try to remount `/sysroot` if we're actually handling the
sysroot repo and the repo isn't writable. We already have public APIs to
check each of those, so let's use them.
Closes: #2485
This moves read-only sysroot checks upfront, so that they are not
intermixed with mount operations.
It has no immediate side-effects, but allow these check to be
independent from the rest of the mounting logic (and future changes
to it).
This adds a `MS_SILENT` flag to all `mount(2)` calls, reducing the
amount of kernel logs produced on each boot.
Those messages do not contain actionable details, and in the "mount
plus read-only remount" case they can easily become highly redundant.
In general, we're probably going to need to change most of our
`g_return_if_fail` to `g_assert`. The analyzer flags that
the function can return `NULL`, but the caller isn't prepared for
this.
In practice, let's abort.
In general, we're probably going to need to change most of our
`g_return_if_fail` to `g_assert`. The analyzer flags that
the function can return `NULL`, but the caller isn't prepared for
this.
In practice, let's abort.
In general, we're probably going to need to change most of our
`g_return_if_fail` to `g_assert`. The analyzer flags that
the function can return `NULL`, but the caller isn't prepared for
this.
In practice, let's abort.
In general, we're probably going to need to change most of our
`g_return_if_fail` to `g_assert`. The analyzer flags that
the function can return `NULL`, but the caller isn't prepared for
this.
In practice, let's abort.
This defines `OstreeRepoAutoTransaction` as a boxed type, in order
to support auto-generating bindings for it.
That first requires adding internal reference-counting to it, to
allow freely copying/freeing references to a single transaction guard.
This rewords errors and log messages in the functions which take care
of preparing sysroot in initramfs.
Depending on the boot flow, it is possible to reach this logic
with a sysroot mounted (unexpectedly) as read-only.
In that case, let's clearly point out the problematic mountpoint.
This enhances the auto-transaction logic, augmenting the scope of a
transaction guard.
It allows committing or aborting a transaction through its guard.
It also supports tracking the completion status of a transaction
guard, avoiding double commits/aborts, while retaining the auto-cleanup
logic.
https://bugzilla.redhat.com/show_bug.cgi?id=1945274 is an issue where a privileged
kubernetes daemonset is writing a socket into `/etc`. This makes ostree upgrades barf.
Now, they should clearly move it to `/run`. However, one option is for us to
just ignore it instead of erroring out. Some brief investigation shows that
e.g. `git add somesocket` is a silent no-op, which is an argument in favor of ignoring it.
Closes: https://github.com/ostreedev/ostree/issues/2446
This is nicer than having the caller parse the commit
object, or indirect via the `OstreeRepoFile*` object of the root.
Will be used in ostree-rs-ext around tar parsing.
This is part of `OstreeCommitModifier`, but I'm not using
that in some of the ostree-ext Rust code.
It just makes more sense as a direct policy API, where it should
have been in the first place. There's already support for
setting a policy object on a commit modifier, so that's all the
old API needs to do now.
There's a general Unix philosophy that "silence is golden".
However, when one is explicitly invoking an error check it's nice
to see explicit success.
We already print various statistics, so ending with a happy
note has no extra cost.
The logic for `--selinux-policy` ended up in the `--tree=dir`
path, but there's no reason for that. Fix the imported
labeling with `--tree=tar`. Prep for use with containers.
We had this bug because the previous logic was trying to avoid
duplicating the code for generic `--selinux-policy` and
the case of `--selinux-policy-from-base --tree=dir`.
It's a bit more code, but it's cleaner if we dis-entangle them.
This will be helpful for the "ostree native container" work in
https://github.com/ostreedev/ostree-rs-ext/
Basically in order to reuse GPG/signapi verification, we need
to support adding a remote, even though it can't be used via
`ostree pull`. (At least, not until we merge ostree-rs-ext into ostree, but
even then I think the principle stands)
for deltafiles the legacy_transaction_resuming flag is not used,
which will mark the commit as done, even if files are missing.
using already existing commitstate_is_partial function as fix
We're waaay overdue for this, it's been the default
in rpm-ostree for years, and solves several important bugs
around not capturing `/etc` while things are running.
Also, `ostree admin upgrade --stage` (should) become idempotent.
Closes: https://github.com/ostreedev/ostree/issues/2389
We have a bunch of APIs to do GPG verification of a commit,
but that doesn't generalize to signapi. Further, they
require the caller to check the signature status explicitly
which seems like a trap.
This much higher level API works with both GPG and signapi.
The intention is to use this in things that are doing "external
pulls" like the ostree-ext tar import support. There we will
get the commitmeta from the tarball and we want to verify it
at the same time we import the commit.
Followup to PRs related to https://github.com/ostreedev/ostree/issues/2410
Since the test suite now covers this the test was failing on
a Fedora SELinux enabled host where we see `security.selinux`
even if not in the commit.
I was seeing an `EPERM` here which was confusing.
It turned out the real error was `EEXIST`.
Since we're referring to the original error, but we do a
lot of computation in the middle, we need to save errno.
This fixes some aspects of OstreeRepoAutoTransaction and re-aligns
it with the logic in flatpak. Specifically:
* link to the underlying repo through refcounting
* bridge internal errors to warning messages
* verify the input pointer type
This is a preparation step before exposing this logic as a public API.
As pointed out in the original review, `gpg-list-keys` fits better
alongside the existing `gpg-import`.
Changes were done with:
```
git grep -l list-gpg-keys | xargs sed -i 's/list-gpg-keys/gpg-list-keys/'
for src in $(git ls-files '*list-gpg-keys*'); do
dst=${src/list-gpg-keys/gpg-list-keys}
git mv "$src" "$dst"
done
```
Calculate the advanced and direct update URLs for the key discovery
portion[1] of the OpenPGP Web Key Directory specification, and include
the URLs in the key listing in ostree_repo_remote_get_gpg_keys(). These
URLs can be used to locate updated GPG keys for the remote.
1. https://datatracker.ietf.org/doc/html/draft-koch-openpgp-webkey-service#section-3.1
This will be used to implement the PGP Web Key Directory (WKD) URL
generation. This is a slightly cleaned up implementation[1] taken from
the zbase32 author's original implementation[2]. It provides a single
zbase32_encode API to convert a set of bytes to the zbase32 encoding.
I believe this should be acceptable for inclusion in ostree. The license
in the source files is BSD style while the original repo LICENSE file
claims the Creative Commons CC0 1.0 Universal license, which is public
domain.
1. https://github.com/dbnicholson/libbase32/tree/for-ostree
2. https://github.com/zooko/libbase32
This provides a wrapper for the `ostree_repo_remote_get_gpg_keys`
function to show the GPG keys associated with a remote. This is
particularly useful for validating that GPG key updates have been
applied. Tests are added, which checks the
`ostree_repo_remote_get_gpg_keys` API by extension.
This function enumerates the trusted GPG keys for a remote and returns
an array of `GVariant`s describing them. This is useful to see which
keys are collected by ostree for a particular remote. The same
information can be gathered with `gpg`. However, since ostree allows
multiple keyring locations, that's only really useful if you have
knowledge of how ostree collects GPG keyrings.
The format of the variants is documented in
`OSTREE_GPG_KEY_GVARIANT_FORMAT`. This format is primarily a copy of
selected fields within `gpgme_key_t` and its subtypes. The fields are
placed within vardicts rather than using a more efficient tuple of
concrete types. This will allow flexibility if more components of
`gpgme_key_t` are desired in the future.
Currently the verifier decides whether to include the global keyrings
based on whether the specified remote has its own keyring or not. Allow
callers to exclude the global keyrings even when that's not the case.
This will be used in a subsequent commit in order to get the GPG keys
only associated with a remote.
In order to use the GPG verifier, it needs to be seeded with GPG keys
after instantation. Currently this is only used for verifying data, but
it will also be used for getting a list of trusted GPG keys in a
subsequent commit.
In configure the systemd unit path is optional, but in the code it's
assumed to be defined. Add an `#ifdef` that throws an error when it's
not defined like the handling of `HAVE_LIBMOUNT` below it.
We struggled for a long time with enablement of our "internal units",
trying to follow the philosophy that units should only be enabled
by explicit preset.
See https://bugzilla.redhat.com/show_bug.cgi?id=1451458
and https://github.com/coreos/rpm-ostree/pull/1482
etc.
And I just saw chat (RH internal on a proprietary system sadly) where
someone hit `ostree-remount.service` not being enabled in CentOS8.
Thinking about this more, I realized we've shipped a systemd generator
for a long time and while its only role until now was to generate `var.mount`,
but by using it to force on our internal units, we don't require
people to deal with presets anymore.
Basically we're inverting things so that "if ostree= is on the kernel
cmdline, then enable our units" and not "enable our units, but have
them use ConditionKernelCmdline=ostree to skip".
Drop the weird gyrations we were doing around `ostree-finalize-staged.path`
too; forking `systemctl start` is just asking for bugs.
So after this, hopefully we won't ever again have to think about
distribution presets and our units.
This will be ignored, so let's make it very clear
people are doing something wrong. Motivated by a bug
in a build pipeline that injected `/var/lib/rpm` into an ostree
commit which ended up crashing rpm-ostree because it was an empty db
which it wasn't expecting.
It *also* turns out rpm-ostree is incorrectly dumping content in the
deployment `/var` today, which is another bug.
Use `g_error` and `g_assert*` rather than `g_return*` when checking the
locking preconditions so that failures result in the program
terminating. Since this code is protecting filesystem data, we'd rather
crash than delete or corrupt data unexpectedly.
`g_error` is used when the error is due to the caller requesting an
invalid transition like attempting to pop a lock type that hasn't been
taken. It also provides a semi-useful message about what happened.
Previously each thread maintained its own lock file descriptor
regardless of whether the thread was using the same `OstreeRepo` as
another thread. This was very safe but it made certain multithreaded
procedures difficult. For example, if a main thread took an exclusive
lock and then spawned worker threads, it would deadlock if one of the
worker threads tried to acquire the lock.
This moves the file descriptor from thread local storage to the
`OstreeRepo` structure so that threads using the same `OstreeRepo` can
share the lock. A mutex guards against threads altering the lock state
concurrently.
Fixes: #2344
This simplifies the lock state management considerably since the
previously pushed type doesn't need to be tracked. Instead, 2 counters
are kept to track how many times each lock type has been pushed. When
the number of exclusive locks drops to 0, the lock transitions back to
shared.
Previous to this we'd trip an assertion `abort()` deep in the curl code if e.g.
a user did `ostree remote add foo htttp://...` etc.
Motivated by considering supporting "external remotes" where code outside
ostree does a pull, but we want to reuse the signing verification infrastructure.
Several tests generate summaries and then expect to use the generated
summary immediately. However, this can cause intermittent test failures
when they inadvertantly get a cached summary file. This typically
happens when the test is run on a filesystem that doesn't support user
extended attributes. In that case, the caching code can only use the
last modified time, which only has 1 second granularity. If tests don't
carefully manage the summary modification times or the repo cache then
they are likely subject to races in some test environments.
This introduces an environment variable `OSTREE_SKIP_CACHE` that
prevents the repo from using a cache directory. This is enabled by
default in tests and disabled for tests that are a explicitly trying to
test the caching behavior.
Fixes: #2313Fixes: #2351
This API is push rather than pull, which makes it much more
suitable to use cases like parsing a tar file from external
code.
Now, we have a large mess in this area internally because
the original file writing code was pull based, but static
deltas hit the same problem of wanting a push API, so I added
this special `OstreeRepoBareContent` just for writing regular
files from a push API.
Eventually...I'd like to deprecate the pull based API,
and rework things so that for regular files the push API
is the default, and then `write_content_object()` would
be split up into archive/bare cases.
In this world the `ostree_repo_write_content()` API would
then need to hackily bridge pull to push and it'd be
less efficient.
Anyways for now due to this bifurcation, this API only
works on non-archive repositories, but that's fine for
now because that's what I want for the `ostree-ext-container`
bits.
Continuation of the addition of `ostree_repo_write_regfile_inline()`.
This will be helpful for ostree-rs-ext and importing from tar, it's
quite inefficient and awkward for small files to end up creating
a whole `GInputStream` and `GFileInfo` and etc. for small files.
Luca BRUNO
Saqib Ali
Jonathan Lebon
Phaedrus Leeds
Colin Walters
Christian Hergert
Nikita Dubrovskii
Dan Nicholson
Simon McVittie
Joseph Marrero
Ryan Gonzalez
Valentin David
Buddelmann, Richard RB
刘建强
Dan Nicholson
Christian Kellner
Timothée Ravier