I'm aiming to do some more work on the Rust side around `fsck`
like functionality, and this is a useful primitive. There isn't
a great Rust crate for xattrs, and I think it's better to share this
code.
Previously, the reference count was left uninitialized as a result of
bypassing the constructor, and the intended abort-on-error usually
wouldn't have happened.
Fixes: 8a9737a "repo/private: move OstreeRepoAutoTransaction to a boxed type"
Resolves: https://github.com/ostreedev/ostree/issues/2592
Signed-off-by: Simon McVittie <smcv@collabora.com>
This prevents writing content into 'bare-split-xattrs` repository,
while carving some space for experimenting via a temporary
`OSTREE_EXP_WRITE_BARE_SPLIT_XATTRS` environment flag.
This aligns all the assertion in the module. In particular, it gets
rid of all `g_return_val_if_fail` instances which may fail without
properly setting GError to the caller.
This enhances the auto-transaction logic, augmenting the scope of a
transaction guard.
It allows committing or aborting a transaction through its guard.
It also supports tracking the completion status of a transaction
guard, avoiding double commits/aborts, while retaining the auto-cleanup
logic.
This is part of `OstreeCommitModifier`, but I'm not using
that in some of the ostree-ext Rust code.
It just makes more sense as a direct policy API, where it should
have been in the first place. There's already support for
setting a policy object on a commit modifier, so that's all the
old API needs to do now.
This simplifies the lock state management considerably since the
previously pushed type doesn't need to be tracked. Instead, 2 counters
are kept to track how many times each lock type has been pushed. When
the number of exclusive locks drops to 0, the lock transitions back to
shared.
This API is push rather than pull, which makes it much more
suitable to use cases like parsing a tar file from external
code.
Now, we have a large mess in this area internally because
the original file writing code was pull based, but static
deltas hit the same problem of wanting a push API, so I added
this special `OstreeRepoBareContent` just for writing regular
files from a push API.
Eventually...I'd like to deprecate the pull based API,
and rework things so that for regular files the push API
is the default, and then `write_content_object()` would
be split up into archive/bare cases.
In this world the `ostree_repo_write_content()` API would
then need to hackily bridge pull to push and it'd be
less efficient.
Anyways for now due to this bifurcation, this API only
works on non-archive repositories, but that's fine for
now because that's what I want for the `ostree-ext-container`
bits.
Continuation of the addition of `ostree_repo_write_regfile_inline()`.
This will be helpful for ostree-rs-ext and importing from tar, it's
quite inefficient and awkward for small files to end up creating
a whole `GInputStream` and `GFileInfo` and etc. for small files.
When working on ostree-ext and importing from tar, it's
quite inefficient and awkward for small files to end up creating
a whole `GInputStream` and `GFileInfo` and etc. for small files.
Plus the gtk-rs binding API to map from `impl Read` to Gio
https://docs.rs/gio/0.9.1/gio/struct.ReadInputStream.html
requires that the input stream is `Send` but the Rust `tar` API
isn't.
This is only 1/3 of the problem; we also need similar APIs
to directly create a symlink, and to stream large objects via
a push-based API.
If `glnx_make_lock_file` falls back to `flock`, on NFS this uses POSIX
locks (`F_SETLK`). As such, we need to be able to handle `EACCES` as
well as `EAGAIN` (see `fnctl(2)`).
I think this is what coreos-ostree-importer has been hitting, which runs
on RHEL7 in the Fedora infra and does locking over an NFS share where
multiple apps could concurrently pull things into the repo.
I've only noticed this by inspection. But I think it's possible for
`cleanup_txn_dir` to get called with the `staging-...-lock` file since
it matches the prefix.
Make the checking here stronger by verifying that it's a directory. If
it's not a directory (lockfile), then follow the default pruning expiry
logic so that we still cleanup stray lockfiles eventually.
The [dev-overlay](332c6ab3b9/src/cmd-dev-overlay)
script shipped in coreos-assembler mostly exists to deal
with the nontrivial logic around SELinux policy. Let's make
the use case of "commit some binaries overlaying a base tree, using
the base's selinux policy" just require a magical
`--selinux-policy-from-base` argument to `ostree commit`.
A new C API was added to implement this in the case of `--tree=ref`;
when the base directory is already checked out, we can just reuse
the existing logic that `--selinux-policy` was using.
Requires: https://github.com/ostreedev/ostree/pull/2039
Using fs-verity is natural for OSTree because it's file-based,
as opposed to block based (like dm-verity). This only covers
files - not symlinks or directories. And we clearly need to
have integrity for the deployment directories at least.
Also, what we likely need is an API that supports signing files
as they're committed.
So making this truly secure would need a lot more work. Nevertheless,
I think it's time to start experimenting with it. Among other things,
it does *finally* add an API that makes files immutable, which will
help against some accidental damage.
This is basic enablement work that is being driven by
Fedora CoreOS; see also https://github.com/coreos/coreos-assembler/pull/876
Append a byte encoding the OSTree object type for each object in the
metadata. This allows the commit metadata to be fetched and then for the
program to see which objects it already has for an accurate calculation
of which objects need to be downloaded.
This slightly breaks the `ostree.sizes` `ay` metadata entries. However,
it's unlikely anyone was asserting the length of the entries since the
array currently ends in 2 variable length integers. As far as I know,
the only users of the sizes metadata are the ostree test suite and
Endless' eos-updater[1]. The former is updated here and the latter
already expects this format.
1. https://github.com/endlessm/eos-updater/
If the object was already in the repo then the sizes metadata entry was
skipped. Move the sizes entry creation after the data has been computed
but before the early return for an existing object.
The object sizes hash table was only being cleared when the repo was
finalized. That means that performing multiple commits while the repo
was open would reuse all the object sizes metadata for each commit.
Clear the hash table when the sizes metadata is setup and when it's
added to a commit. This still does not fix the issue all the way since
it does nothing to prevent the program from constructing multiple
commits simultaneously. To handle that, the object sizes hash table
should be attached to the MutableTree since that has the commit state.
However, the MutableTree is gone when the commit is actually created.
The hash table would have to be transferred to the root file when
writing the MutableTree. That would be an awkward addition to
OstreeRepoFile, though. Add a FIXME to capture that.
After the corruption has been fixed with "ostree fsck -a --delete", a
second run of the "ostree fsck" command will print X partial commits
not verified and exit with a zero.
The zero exit code makes it hard to detect if a repair operation needs
to be run. When ever fsck creates a partial commit it should add a
reason for the partial commit to the state file found in
state/<hash>.commitpartial. This will allow a future execution of the
fsck to still return an error indicating that the repository is still
in the damaged state, awaiting repair.
Additional reason codes could be added in the future for why a partial
commit exists.
Text from: https://github.com/ostreedev/ostree/pull/1880
====
cgwalters commented:
To restate, the core issue is that it's valid to have partial commits
for reasons other than fsck pruned bad objects, and libostree doesn't
have a way to distinguish.
Another option perhaps is to write e.g. fsck-partial into the
statefile state/<hash>.commitpartial which would mean "partial, and
expected to exist but was pruned by fsck" and fsck would continue to
error out until the commit was re-pulled. Right now the partial stamp
file is empty, so it'd be fully compatible to write a rationale into
it.
====
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Closes: #1910
Approved by: cgwalters
We want a case where we can disable the min-free-space check. Initially,
it felt like to add a OSTREE_REPO_PULL_FLAGS_DISABLE_FREE_SPACE_CHECK but
the problem is prepare_transaction() does not have a OstreeRepoPullFlags
parameter which we can parse right here. On top of it, prepare_transaction()
enforces min-free-space check and won't let the transaction proceed if
the check failed.
This is pretty bad in conjunction with "inherit-transaction" as what
Flatpak uses. There is no way to disable this check unless we remove
it altogether from prepare_transaction.
This issue came out to light when flatpak wasn't able to write metadata
after fetching from remote:
[uajain@localhost ~]$ flatpak remote-info flathub org.kde.Platform//5.9
error: min-free-space-size 500MB would be exceeded
Metadata objects helps in housekeeping and restricting them means
restricting crucial UX (like search, new updates) functionalities
in clients like gnome-software. The error banners originated from
these issues are also abrupt and not much helpful to the user. This
is the specific instance of the issue this patches tries to address.
See https://github.com/flatpak/flatpak/issues/2139 for discussion.
Closes: #1779
Approved by: mwleeds
When falling back to copying objects when importing them into a
bare-user repo, we only actually need to transfer over the
`user.ostreemeta` xattr.
This allows the destination repo to be on a separate filesystem that
might not even support `security.selinux`. (I hit this while importing
over virtio-9p).
Closes: #1771
Approved by: cgwalters
I found this useful while hacking on rpm-ostree but I think it might be
useful enough to upstream. This stat is really helpful for validating
that a pipeline is hitting the devino cache sweet spot.
Closes: #1772
Approved by: cgwalters
Jonathan Lebon
Colin Walters
Simon McVittie
Daniel Kolesa
Luca BRUNO
Joseph Marrero
Dan Nicholson
Felix Krull
OpenShift Merge Robot
Dan Nicholson
John Hiesey
Jason Wessel
Umang Jain