This API is push rather than pull, which makes it much more
suitable to use cases like parsing a tar file from external
code.
Now, we have a large mess in this area internally because
the original file writing code was pull based, but static
deltas hit the same problem of wanting a push API, so I added
this special `OstreeRepoBareContent` just for writing regular
files from a push API.
Eventually...I'd like to deprecate the pull based API,
and rework things so that for regular files the push API
is the default, and then `write_content_object()` would
be split up into archive/bare cases.
In this world the `ostree_repo_write_content()` API would
then need to hackily bridge pull to push and it'd be
less efficient.
Anyways for now due to this bifurcation, this API only
works on non-archive repositories, but that's fine for
now because that's what I want for the `ostree-ext-container`
bits.
Continuation of the addition of `ostree_repo_write_regfile_inline()`.
This will be helpful for ostree-rs-ext and importing from tar, it's
quite inefficient and awkward for small files to end up creating
a whole `GInputStream` and `GFileInfo` and etc. for small files.
When working on ostree-ext and importing from tar, it's
quite inefficient and awkward for small files to end up creating
a whole `GInputStream` and `GFileInfo` and etc. for small files.
Plus the gtk-rs binding API to map from `impl Read` to Gio
https://docs.rs/gio/0.9.1/gio/struct.ReadInputStream.html
requires that the input stream is `Send` but the Rust `tar` API
isn't.
This is only 1/3 of the problem; we also need similar APIs
to directly create a symlink, and to stream large objects via
a push-based API.
I was going to add some new API and I noticed that this function
never returns an error; presumably at one point it did, but
not anymore. It simplifies the code flow noticeably
to remove that.
It was added for the collections bits, but we made that stable.
It's now just cruft and we're very unlikely to reuse the infrastructure
again.
Motivated by a unit test failure when running from a tarball:
https://github.com/ostreedev/ostree/issues/2313
I was recently looking at systemd's CI and came across this.
AFAICS it's a proprietary analysis but it looks useful.
We still have `clang-analyzer` for example for FOSS analysis.
This was my first experiment with using Rust in this way; I gained
a lot of knowledge from it. But, we don't really gain
anything from the code as it is today - while it is "bit fiddling"
code, the C code is well tested.
We have a lot of compile-time options, and trimming them will be
helpful.
We've also gotten pushback on hard requiring Rust client side.
Instead, what I'd like to do is hopefully soon create an `ostree-system`
crate that uses the existing `ostree` library and can contain
code drained from the rpm-ostree Rust and used by other projects perhaps.
So the goal here is really more Rust, but we need to focus our
efforts on where it's most valuable.
I was doing some rpm-ostree work and I wanted to compare two
OSTree commits to see if the kernel has changed. I think
this should be a lot more natural.
Add `ostree commit --bootable` which calls into a new generic
library API `ostree_commit_metadata_for_bootable()` that
discovers the kernel version and injects it as an `ostree.linux`
metadata key. And for extra clarity, add an `ostree.bootable`
key.
It's interesting because the "core" OSTree layer is all about
generic files, but this is adding special APIs around bootable
OSTree commits (as opposed to e.g. flatpak as well as
things like rpm-ostree's pkgcache refs).
Eventually, I'd like to ensure everyone is using this and
hard require this metadata key for the `ostree admin deploy`
flow - mainly to prevent accidents.
We have an `allow_noent` boolean that controls this, but
were missing the `(nullable)` annotation, so the Rust bindings
panic when the ref doesn't exist instead of being `Option<GString>`.
It's cleaner if this is an build option rather than being
kludged into the CI layer.
Notably we can't use `LD_PRELOAD` anymore with ASAN, so update
our tests to check for `ASAN_OPTIONS`.
The recent change in https://github.com/coreos/fedora-coreos-config/pull/659
broke some of our tests that do `mount -o remount,rw /sysroot` but
leave `/boot` read-only.
We had code for having `/boot` read-only before `/sysroot` but
in practice we had a file descriptor for `/sysroot` that we opened
before the remount that would happen later on.
Clean things up here so that in the library, we also remount
`/boot` at the same time we remount `/sysroot` if either are readonly.
Delete the legacy code for remounting `/boot` rw if we're not in
a mount namespace. I am fairly confident most users are either
using the `ostree` CLI, or they're using the mount namespace.
Just like we hold a fd for `/sysroot`, also do so for `/boot`
instead of opening and closing it in a few places.
This is a preparatory cleanup for further work.
I was being very conservative initially here, but I think it's
really safe to just unconditionally set up the mount namespace.
This avoids having to check twice for a read-only `/sysroot`
(once in the binary and once in the library).
To pull in the fix for `/var/tmp` on NixOS but also
on general principle.
Update submodule: libglnx
```
Colin Walters (1):
xattrs: Add better error prefixing
Rebecca Turner (2):
glnx-fdio: try $TMPDIR if /var/tmp doesn't exist
glnx-fdio: use $TMPDIR if set
```
Alternative to https://github.com/ostreedev/ostree/pull/2197
Python's (usually) zero-sized `__init__.py` files can provoke
us hitting the hardlink limits on some filesystems (`EMLINK`).
At least one Fedora rpm-ostree user hit this.
The benefits of hardlinking here are quite marginal; lots
of hardlinks can behave suboptimally in particular filesystems
like BTRFS too.
This builds on prior code which made this an option, introduced
in https://github.com/ostreedev/ostree/commit/673cacd633f9d6b653cdea530657d3e780a41bbd
Now we just do it uncondtionally.
Also this provoked a different bug in a very obscure user mode checkout
case; when the "real" permissions were different from the "physical"
permissions, we would still hardlink. Fix the test case for this.
When support for devicetree was added, it created a problem
because old and new ostree versions would compute different
checksums for the "boot data". The scenario here is:
- Have system with ostree < 2020.4
- Reboot into system with ostree 2020.5
- Try to perform an operation that would retain
that previous booted deployment (common)
Currently ostree iterates over all the deployments
that will be retained and calls `install_deployment_kernel()`,
even for the booted one (which is a bit silly), but
just to verify that all boot data for the targeted
deployments are installed.
This then re-computes the checksum and we'd trip this
assertion.
In practice though, we don't strictly require them to match;
the only thing that will happen if they don't is that we'll
end up with another copy of the kernel/initramfs - and
that only temporarily until the previous deployment
gets GC'd.
Longer term, I think what we really want to do anyways
is probably closer to like a little ostree repo for `/boot`
so that we can e.g. still hardlink kernels there even if
the initramfs changes, or hardlink both kernel/initramfs
if just the devicetree changes, etc.
Closes: https://github.com/ostreedev/ostree/issues/2154
The extreme special case of "zero mode" files like `/etc/shadow`
comes up again. What we want is for "user mode" checkouts to
override it to make the file readable; otherwise when operating
as non-root without `CAP_DAC_OVERRIDE` it becomes very difficult
to work with.
Previously, we were hardlinking these files, but then it intersects
with *another* special case around zero sized files, which is
*also* true for `/etc/shadow`.
Trying to avoid hardlinking there unveiled this bug - when
we go to do a copy checkout, we need to override the mode.
Looking at
https://github.com/coreos/coreos-assembler/issues/1703
a user is getting a bare:
`error: fsetxattr: Permission denied`
I don't think it's these code paths since a deploy
isn't happening but on inspection I noticed we didn't
have error prefixing here.
This adds infrastructure to the Rust test suite for destructive
tests, and adds a new `transactionality` test which runs
rpm-ostree in a loop (along with `ostree-finalize-staged`) and
repeatedly uses either `kill -9`, `reboot` and `reboot -ff`.
The main goal here is to flush out any "logic errors".
So far I've validated that this passes a lot of cycles
using
```
$ kola run --qemu-image=fastbuild-fedora-coreos-ostree-qemu.qcow2 ext.ostree.destructive-rs.transactionality --debug --multiply 8 --parallel 4
```
a number of times.
I was thinking a bit more recently about the "live" changes
stuff https://github.com/coreos/rpm-ostree/issues/639
(particularly since https://github.com/coreos/rpm-ostree/pull/2060 )
and I realized reading the last debates in that issue that
there's really a much simpler solution; do exactly the same
thing we do for `ostree admin unlock`, except mount it read-only
by default.
Then, anything that wants to modify it does the same thing
libostree does for `/sysroot` and `/boot` as of recently; create
a new mount namespace and do the modifications there.
The advantages of this are numerous. First, we already have
all of the code, it's basically just plumbing through a new
entry in the state enumeration and passing `MS_RDONLY` into
the `mount()` system call.
"live" changes here also naturally don't persist, unlike what
we are currently doing in rpm-ostree.
We were committing the whole tempdir, which seems to fail
in Travis because the GPG agent Unix domain socket ends up there too,
and ostree refuses to commit sockets.
This came in with 5af403be0c but
was never implemented.
I noticed this now because the Rust ostree bindings generate a
wrapper for it which the linker tries to use.
This is the dual of https://github.com/ostreedev/ostree/pull/2129/commits/1f3c8c5b3de978f6e069c24938967f823cce7ee8
where we output more detail when signapi fails to validate.
Extend the API to return a string for success, which we output
to stdout.
This will help the test suite *and* end users validate that the expected
thing is happening.
In order to make this cleaner, split the "verified commit" set
in the pull code into GPG and signapi verified sets, and have
the signapi verified set contain the verification string.
We're not doing anything with the verification string in the
pull code *yet* but I plan to add something like
`ostree pull --verbose` which would finally print this.
Colin Walters