I was doing some rpm-ostree work and I wanted to compare two
OSTree commits to see if the kernel has changed. I think
this should be a lot more natural.
Add `ostree commit --bootable` which calls into a new generic
library API `ostree_commit_metadata_for_bootable()` that
discovers the kernel version and injects it as an `ostree.linux`
metadata key. And for extra clarity, add an `ostree.bootable`
key.
It's interesting because the "core" OSTree layer is all about
generic files, but this is adding special APIs around bootable
OSTree commits (as opposed to e.g. flatpak as well as
things like rpm-ostree's pkgcache refs).
Eventually, I'd like to ensure everyone is using this and
hard require this metadata key for the `ostree admin deploy`
flow - mainly to prevent accidents.
We have an `allow_noent` boolean that controls this, but
were missing the `(nullable)` annotation, so the Rust bindings
panic when the ref doesn't exist instead of being `Option<GString>`.
It's cleaner if this is an build option rather than being
kludged into the CI layer.
Notably we can't use `LD_PRELOAD` anymore with ASAN, so update
our tests to check for `ASAN_OPTIONS`.
The recent change in https://github.com/coreos/fedora-coreos-config/pull/659
broke some of our tests that do `mount -o remount,rw /sysroot` but
leave `/boot` read-only.
We had code for having `/boot` read-only before `/sysroot` but
in practice we had a file descriptor for `/sysroot` that we opened
before the remount that would happen later on.
Clean things up here so that in the library, we also remount
`/boot` at the same time we remount `/sysroot` if either are readonly.
Delete the legacy code for remounting `/boot` rw if we're not in
a mount namespace. I am fairly confident most users are either
using the `ostree` CLI, or they're using the mount namespace.
Just like we hold a fd for `/sysroot`, also do so for `/boot`
instead of opening and closing it in a few places.
This is a preparatory cleanup for further work.
I was being very conservative initially here, but I think it's
really safe to just unconditionally set up the mount namespace.
This avoids having to check twice for a read-only `/sysroot`
(once in the binary and once in the library).
To pull in the fix for `/var/tmp` on NixOS but also
on general principle.
Update submodule: libglnx
```
Colin Walters (1):
xattrs: Add better error prefixing
Rebecca Turner (2):
glnx-fdio: try $TMPDIR if /var/tmp doesn't exist
glnx-fdio: use $TMPDIR if set
```
Alternative to https://github.com/ostreedev/ostree/pull/2197
Python's (usually) zero-sized `__init__.py` files can provoke
us hitting the hardlink limits on some filesystems (`EMLINK`).
At least one Fedora rpm-ostree user hit this.
The benefits of hardlinking here are quite marginal; lots
of hardlinks can behave suboptimally in particular filesystems
like BTRFS too.
This builds on prior code which made this an option, introduced
in https://github.com/ostreedev/ostree/commit/673cacd633f9d6b653cdea530657d3e780a41bbd
Now we just do it uncondtionally.
Also this provoked a different bug in a very obscure user mode checkout
case; when the "real" permissions were different from the "physical"
permissions, we would still hardlink. Fix the test case for this.
When support for devicetree was added, it created a problem
because old and new ostree versions would compute different
checksums for the "boot data". The scenario here is:
- Have system with ostree < 2020.4
- Reboot into system with ostree 2020.5
- Try to perform an operation that would retain
that previous booted deployment (common)
Currently ostree iterates over all the deployments
that will be retained and calls `install_deployment_kernel()`,
even for the booted one (which is a bit silly), but
just to verify that all boot data for the targeted
deployments are installed.
This then re-computes the checksum and we'd trip this
assertion.
In practice though, we don't strictly require them to match;
the only thing that will happen if they don't is that we'll
end up with another copy of the kernel/initramfs - and
that only temporarily until the previous deployment
gets GC'd.
Longer term, I think what we really want to do anyways
is probably closer to like a little ostree repo for `/boot`
so that we can e.g. still hardlink kernels there even if
the initramfs changes, or hardlink both kernel/initramfs
if just the devicetree changes, etc.
Closes: https://github.com/ostreedev/ostree/issues/2154
The extreme special case of "zero mode" files like `/etc/shadow`
comes up again. What we want is for "user mode" checkouts to
override it to make the file readable; otherwise when operating
as non-root without `CAP_DAC_OVERRIDE` it becomes very difficult
to work with.
Previously, we were hardlinking these files, but then it intersects
with *another* special case around zero sized files, which is
*also* true for `/etc/shadow`.
Trying to avoid hardlinking there unveiled this bug - when
we go to do a copy checkout, we need to override the mode.
Looking at
https://github.com/coreos/coreos-assembler/issues/1703
a user is getting a bare:
`error: fsetxattr: Permission denied`
I don't think it's these code paths since a deploy
isn't happening but on inspection I noticed we didn't
have error prefixing here.
This adds infrastructure to the Rust test suite for destructive
tests, and adds a new `transactionality` test which runs
rpm-ostree in a loop (along with `ostree-finalize-staged`) and
repeatedly uses either `kill -9`, `reboot` and `reboot -ff`.
The main goal here is to flush out any "logic errors".
So far I've validated that this passes a lot of cycles
using
```
$ kola run --qemu-image=fastbuild-fedora-coreos-ostree-qemu.qcow2 ext.ostree.destructive-rs.transactionality --debug --multiply 8 --parallel 4
```
a number of times.
I was thinking a bit more recently about the "live" changes
stuff https://github.com/coreos/rpm-ostree/issues/639
(particularly since https://github.com/coreos/rpm-ostree/pull/2060 )
and I realized reading the last debates in that issue that
there's really a much simpler solution; do exactly the same
thing we do for `ostree admin unlock`, except mount it read-only
by default.
Then, anything that wants to modify it does the same thing
libostree does for `/sysroot` and `/boot` as of recently; create
a new mount namespace and do the modifications there.
The advantages of this are numerous. First, we already have
all of the code, it's basically just plumbing through a new
entry in the state enumeration and passing `MS_RDONLY` into
the `mount()` system call.
"live" changes here also naturally don't persist, unlike what
we are currently doing in rpm-ostree.
We were committing the whole tempdir, which seems to fail
in Travis because the GPG agent Unix domain socket ends up there too,
and ostree refuses to commit sockets.
This came in with 5af403be0c but
was never implemented.
I noticed this now because the Rust ostree bindings generate a
wrapper for it which the linker tries to use.
This is the dual of https://github.com/ostreedev/ostree/pull/2129/commits/1f3c8c5b3de978f6e069c24938967f823cce7ee8
where we output more detail when signapi fails to validate.
Extend the API to return a string for success, which we output
to stdout.
This will help the test suite *and* end users validate that the expected
thing is happening.
In order to make this cleaner, split the "verified commit" set
in the pull code into GPG and signapi verified sets, and have
the signapi verified set contain the verification string.
We're not doing anything with the verification string in the
pull code *yet* but I plan to add something like
`ostree pull --verbose` which would finally print this.
To aid debuggability, when we find a commit that isn't signed
by our expected key, output a specific error message with the
key.
(And then add code to switch to just printing the count beyond 3
because the test suite injects 100 keys and hopefully no one
ever actually does that)
I'm thinking about adding an implementation of ed25519 signatures
with OpenSSL (so we can ship the feature with Fedora CoreOS
without requiring an additional library) and in preparation for
that it's essential that we validate that libsodium-generated
signatures and OpenSSL-generated signatures are compatible.
I don't know if they are yet actually, but the goal of this
new test is to add a pre-generated repository with a signed
commit generated by libsodium.
This will catch if e.g. there's ever a change in libsodium,
or if existing libsodium implementation versions (e.g. the
one in Debian) might differ from what we ship here.
Add a standard key for this. We actually had a case in OpenShift
builds recently where a `ppc64le` image was pushed over an `x86_64`
one and this started failing at runtime with a not immediately
obvious error.
I'll probably end up changing rpm-ostree at least to use
the RPM architecture for this key and fail if it doesn't match
the booted value.
Possibly that should live in ostree but it would involve adding
architecture schema here, which gets into a big mess. Let's
just standardize the key.
xref https://github.com/coreos/coreos-assembler/commit/e02ef2683d688607e7b5ad9ea6a0c00c50a682a5
One OpenShift user saw this from rpm-ostree:
```
client(id:cli dbus:1.583 unit:machine-config-daemon-host.service uid:0) added; new total=1
Initiated txn UpdateDeployment for client(id:cli dbus:1.583 unit:machine-config-daemon-host.service uid:0): /org/projectatomic/rpmostree1/rhcos
Txn UpdateDeployment on /org/projectatomic/rpmostree1/rhcos failed: File header size 4294967295 exceeds size 0
```
which isn't very helpful. Let's add some error
prefixing here which would at least tell us which
object was corrupted.
There's a lot going on here. First, this is intended to run
nicely as part of the new [cosa/kola ext-tests](https://github.com/coreos/coreos-assembler/pull/1252).
With Rust we can get one big static binary that we can upload,
and include a webserver as part of the binary. This way we don't
need to do the hack of running a container with Python or whatever.
Now, what's even better about Rust for this is that it has macros,
and specifically we are using [commandspec](https://github.com/tcr/commandspec/)
which allows us to "inline" shell script. I think the macros
could be even better, but this shows how we can intermix
pure Rust code along with using shell safely enough.
We're using my fork of commandspec because the upstream hasn't
merged [a few PRs](https://github.com/tcr/commandspec/pulls?q=is%3Apr+author%3Acgwalters+).
This model is intended to replace *both* some of our
`make check` tests as well.
Oh, and this takes the obvious step of using the Rust OSTree bindings
as part of our tests. Currently the "commandspec tests" and "API tests"
are separate, but nothing stops us from intermixing them if we wanted.
I haven't yet tried to write destructive tests with this but
I think it will go well.
Follow the precedent set in https://github.com/coreos/rpm-ostree/pull/2106
and rename the directory, to more clearly move away from the
"uninstalled" test model. Prep for Rust-based tests.
We recently disabled the read-only /sysroot handling:
https://github.com/ostreedev/ostree/pull/2108/commits/e35b82fb891daee823fcce421ae8f1442b630ea2
The core problem was that a lot of services run early in the
real root and want write access to things like `/var` and `/etc`.
In trying to do remounts while the system is running we introduce
too many race conditions.
Instead, just make the `/etc` bind mount in the initramfs right
after we set up the main root. This is much more natural really,
and avoids all race conditions since nothing is running in the
sysroot yet.
The main awkward part is that since we're not linking
`ostree-prepare-root` to GLib (yet) we have a hacky parser
for the config file. But, this is going to be fine I think.
In order to avoid parsing the config twice, pass state from
`ostree-prepare-root` to `ostree-remount` via a file in `/run`.
The goal here is to move the code towards a model
where the *client* can explicitly specify which signature types
are acceptable.
We retain support for `sign-verify=true` for backwards compatibility.
But in that configuration, a missing public key is just "no signatures found".
With `sign-verify=ed25519` and no key configured, we can
explicitly say `No keys found for required signapi type ed25519`
which is much, much clearer.
Implementation side, rather than maintaining `gboolean sign_verify` *and*
`GPtrArray sign_verifiers`, just have the array. If it's `NULL` that means
not to verify.
Note that currently, an explicit list is an OR of signatures, not AND.
In practice...I think most people are going to be using a single entry
anyways.
In a CI run I think one of these `ostree show` commands is failing.
While that output would be useful, the actual `err.txt` usually
has what we want too.
There's a lot of historical baggage associated with GPG verification
and `ostree pull` versus `ostree pull-local`. In particular nowadays,
if you use a `file://` remote things are transparently optimized
to e.g. use reflinks if available.
So for anyone who doesn't trust the "remote" repository, you should
really go through through the regular
`ostree remote add --sign-verify=X file://`
path for example.
Having a mechanism to say "turn on signapi verification" *without*
providing keys goes back into the "global state" debate I brought
up in https://github.com/ostreedev/ostree/issues/2080
It's just much cleaner architecturally if there is exactly one
path to find keys: from a remote config.
So here in contrast to the GPG code, for `pull-local` we explictily
disable signapi validation, and the `ostree_repo_pull()` API just
surfaces flags to disable it, not enable it.
In https://bugzilla.redhat.com/show_bug.cgi?id=1827712
some OpenShift CI is seeing `/boot` being unmounted before
`ostree-finalize-staged.service` runs or completes.
We finally tracked this down to a bug elsewhere, but
I think we should add this because it clearly shows
our requirements.
Previously in the pull code, every time we went to verify
a commit we would re-initialize an `OstreeSign` instance
of each time, re-parse the remote configuration
and re-load its public keys etc.
In most cases this doesn't matter really because we're
pulling one commit, but if e.g. pulling a commit with
history would get a bit silly.
This changes things so that the pull code initializes the
verifiers once, and reuses them thereafter.
This is continuing towards changing the code to support
explicitly configured verifiers, xref
https://github.com/ostreedev/ostree/issues/2080
This cleans up the verification code; it was weird how
we'd get the list of known names and then try to create
an instance from it (and throw an error if that failed, which
couldn't happen).
Per https://github.com/ostreedev/ostree/issues/2080#issuecomment-623614483
A huge benefit of ed25519 (and ECC in general) is that keys are very
short - short enough that it's completely reasonable to inline
them into a command line argument.
And I think that's a good model; it makes the keys very visible.
For example, someone could easily copy-paste a commandline
argument from a webpage (secured via TLS) that says to run
`ostree remote add --sign-verify=ed25519=inline:KEY`.
With just `After=` we'll still try to run in the scenario
where `sysroot.mount` fails because the rootfs didn't appear.
And this will end up spewing an error which can confuse people
into thinking something is wrong at the ostree level.
This has come up numerous times w/{Fedora,RHEL} CoreOS, most
recently while looking at
https://bugzilla.redhat.com/show_bug.cgi?id=1803130
I think we should encourage removing the writable bits from
executables. This has happened to me:
https://thomask.sdf.org/blog/2019/11/09/take-care-editing-bash-scripts.html
And not having the writable bit may help prevent hardlink
corruption with OSTree in some cases.
We can't do this by default, but add a convenient CLI flag
for it.
Same motivation as
https://github.com/coreos/rpm-ostree/pull/2060
I tried `InaccessiblePaths=/var` first and was very sad to find
out we have one tiny exception that breaks it. Otherwise it'd
be so elegant. Maybe in the future we split out that one thing
to a separate `ostree-finalized-stage-var.service` that's just
`ExecStart=/bin/rm -vf /var/.updated` and is otherwise
`ProtectSystem=strict` etc.
All of the underlying libostree APIs have supported passing `NULL`
for a merge deployment for...a long time. But we never plumbed
it up into the CLI.
Add a `--no-merge` option to aid people who want to do a "factory reset":
https://github.com/ostreedev/ostree/issues/1793
`ostree-repo-pull.c` is rather monstrous; I plan to split it
up a bit. There's actually already a `pull-private.h` but
that's just for the binding verification API. I think that one
isn't really pull specific. Let's move it into the "catchall"
`repo.c`.
As far as I can tell we're not gating on this right now. From
a quick glance, it looks like Debian stable has `libsodium-dev`
but only Ubuntu eoan does which we're not testing right now.
`libsodium` is an implementation detail. In particular, I'd like
to consider using OpenSSL for ed25519 (if libsodium isn't configured
and openssl is).
So switch the name of the exposed feature and adjust the tests.
Colin Walters