Previously we were doing e.g. `ot_util_filename_validate()` specifically inline
in dirtree objects, but only *after* writing them into the staging directory (by
default). In (non-default) cases such as not using a transaction, such an object
could be written directly into the repo.
A notable gap here is that `pull-local --untrusted` was *not* doing
this verification, just checksums. We harden that (and also the
static delta writing path, really *everything* that calls
`ostree_repo_write_metadata()` to also do "structure" validation
which includes path traversal checks. Basically, let's try hard
to avoid having badly structured objects even in the repo.
One thing that sucks in this patch is that we need to allocate a "bounce buffer"
for metadata in the static delta path, because GVariant imposes alignment
requirements, which I screwed up and didn't fulfill when designing deltas. It
actually didn't matter before because we weren't parsing them, but now we are.
In theory we could check alignment but ...eh, not worth it, at least not until
we change the delta compiler to emit aligned metadata which actually may be
quite tricky. (Big picture I doubt this really matters much right now
but I'm not going to pull out a profiler yet for this)
The pull test was extended to check we didn't even write a dirtree
with path traversal into the staging directory.
There's a bit of code motion in extracting
`_ostree_validate_structureof_metadata()` from `fsck_metadata_object()`.
Then `_ostree_verify_metadata_object()` builds on that to do checksum
verification too.
Closes: #1412
Approved by: jlebon
While we do protect against path traversal during pull, let's also validate
during checkout; it's a cheap operation and provides good last-mile protection.
Closes: #1412
Approved by: jlebon
It can be helpful to be able to choose which OstreeRepoFinder instances
to use when using the find-remotes command. For example, if the tests
need to run in an environment that can't have an Avahi daemon, this
allows you to disable the Avahi (LAN) finder. This commit adds the
--finders option for this purpose.
Closes: #1407
Approved by: cgwalters
Previously when initramfs-* was not found in a deployment's
boot directory, it was assumed that rootfs is prepared for
ostree booting by a kernel patch.
With this patch, the behaviour changes to be - if initramfs-*
is not found, assume that system is using a static
ostree-prepare-root as init process. Booting without initramfs
is a common use case on embedded systems. This approach is
also more convenient, than having to patch the kernel.
Closes: #1401
Approved by: cgwalters
If the current deployment has "rootwait root=/dev/sda2",
but the new deployment does not need "rootwait" anymore,
there is no way to clear this arg at the moment (as opposed
to "karg=root=", which overrides any earlier argument with
the same name). With "--karg-none" users can now clear all
the previous args and set new "root=":
ostree admin deploy --karg-none --karg=root=LABEL=rootfs
Closes: #1401
Approved by: cgwalters
With the current approach, when ostree-prepare-root is used
on the kernel command line as init=, it always assumes that
the next value in the argument list is a path to the sysroot.
The code for falling back to a default path (if none is provided),
would only work if init= is the last arg in the argument list.
We can not rely on that and have to explicitly provide the
path to the sysroot. Which defeats the purpose of a default
path selection code.
To keep command line neater assume that sysroot is on / when
using ostree-prepare-root as init. This probably is what most
people want anyways. Also _ostree_kernel_args* API assumes
that args are space separated list. Which is problematic for:
"init=${ostree}/usr/lib/ostree/ostree-prepare-root /" as it
gets split in two.
Closes: #1401
Approved by: cgwalters
Clients of libostree such as rpm-ostree make extensive use of the
`ostree commit -b foo --tree=ref=foo` pattern in their tests, e.g. to
simulate an update.
What I'm trying to solve here is that it's often the case that we want
to keep metadata from the previous commit without having to be too
verbose (i.e. reading from the parent, then passing it as an argument).
The new `--keep-metadata` switch makes this really easy. I intend to use
this in the rpm-ostree testsuite to make sure we always carry over the
`source-title` metadata as well as during set up for tests that require
`rpmostree.rpmdb.pkglist` metadata.
I initially implemented this in a small wrapper script that uses the API
directly, though we make use of so many other `ostree commit` functions
that it'd require re-implementing a lot of it.
Closes: #1402
Approved by: cgwalters
Apparently there testing systems that literally install *all*
packages. Having `ostree-grub2` currently causes grub2 to fail
on a non-ostree managed system. Let's just gracefully exit
if there's no system repository.
https://bugzilla.redhat.com/show_bug.cgi?id=1532668Closes: #1399
Approved by: jlebon
This tripped up the `docbook-dtds` `%post` in my experiments
with doing rpm-ostree for buildroots.
I cloned and built [xfstests](https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git)
but haven't yet investigated actually running it.
In the meantime let's do the obvious fix here; we need to distinguish
between "copyup enabled" and "actually did a copyup" in the open path
at least, since if we didn't do a copyup we don't need to re-open.
Closes: #1396
Approved by: jlebon
Allways include ostree-repo-pull-private.h to get rid of the following
build error when HAVE_LIBCURL_OR_LIBSOUP is not defined:
src/libostree/ostree-repo-pull.c:1493:1: error: no previous prototype
for '_ostree_repo_verify_bindings' [-Werror=missing-prototypes]
Signed-off-by: Marcus Folkesson <marcus.folkesson@gmail.com>
Closes: #1389
Approved by: cgwalters
Let's do a new release with the locking preview, the http2 disable options and
other misc bugfixes to close out the year.
Closes: #1386
Approved by: jlebon
We had this basically forced on in the CLI; down the line I'd really like to
make this an API option to commit or so, but given that we found a use case in
the rpm-ostree test suite for "unbound" commits, let's support creating them
from the cmdline.
See: https://github.com/ostreedev/ostree/pull/1379Closes: #1380
Approved by: jlebon
It'd all be really nice if there was some sort of `O_TMPFILE` for symlinks, but
anyways the way we were doing a generic "make temp file than rename" actually
defeats some of the point of `O_TMPFILE`. It's now fully safe to do "copy to
self", so let's do that for regfiles.
Closes: #1378
Approved by: jlebon
Today the rpm-ostree test suite uses `refs --create` to save
commits. I think this is a legitimate use case, and other
people may be doing something similar.
On the other hand, I think we should probably be changing the rpm-ostree test
suite to create "unbound" commits. But let's be maximially compatible here since
we hit a real-world case where something needed to change.
Closes: #1379
Approved by: pwithnall
For the [rpm-ostree jigdo ♲📦](https://github.com/projectatomic/rpm-ostree/issues/1081) work.
We're basically doing "pull" via a non-libostree mechanism, and this
should be fully supported. As I mentioned earlier we should try to
have `ostree-repo-pull.c` only use public APIs; this gets us closer
to that.
Closes: #1376
Approved by: jlebon
I want some time to play with this more with different callers and work through
test scenarios. Let's disable the locking by default for now, but make it easy
to enable.
Closes: #1375
Approved by: jlebon
Typically you’d use --branch and --bind-ref together to add additional
bindings as well as creating a main --branch for the commit. However,
you might also want to occasionally use --orphan --bind-ref to create a
commit with bindings for one or more refs, but not actually create any
of those refs pointing to the commit (you might create them as a later
step).
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1347
Approved by: cgwalters
This new option verifies that the refs listed in the ref-bindings for
each commit all point to that commit (i.e. there aren’t multiple commits
listing the same ref in their ref-bindings, and there aren’t any commits
with non-empty ref-bindings which aren’t pointed at by a ref).
This is useful when generating a new repository from scratch, but not
useful when adding new commits to an existing repository (since the old
commits will still, correctly, have ref-bindings from when the refs
pointed at them). That’s why it has to be enabled explicitly using
--verify-back-refs, rather than being on by default.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1347
Approved by: cgwalters
Try and clarify what happens with the prefixes, and that they always
return refspecs.
I’m still not 100% sure this is right.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1347
Approved by: cgwalters
It seems ostree_repo_list_refs() can return refspecs as hash table keys,
as well as just ref names. Handle that by parsing them before trying to
use them as ref names.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1347
Approved by: cgwalters
Since an OSTree client will refuse to pull from a remote which it has
locally configured with a collection ID, if the commit on that remote
has incorrect or missing bindings, we’d better verify them as part of
fsck.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1347
Approved by: cgwalters
It will be used by the fsck utility in future. We could expose it
publicly in future too, if needed.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1347
Approved by: cgwalters
Mostly adding this for use in test cases; it allows us to add e.g.
integers, and we need to deal with byteswapping those.
Someone mind also find it useful to add fully structured metadata, although most
of those users should be using a real language and not shell script.
Closes: #1372
Approved by: jlebon
In the non-`CONSUME` path for regfiles (which happens currently for
`bare-user`), we go to a lot of contortions to make an "object stream",
only to immediately parse it again.
Fixing this will also enable the `G_IS_FILE_DESCRIPTOR_BASED()` fast path in
commit, since the input stream will actually reference the file descriptor and
not be an `_OstreeChainInputStream`.
There's a slight concern here in that we're no longer checksumming *literally*
the object stream passed in for the stream case, but I mention in the comment,
the data should be the same, and if it's not somehow we're not adding risk,
since the checksum is still covering the data we actually care about.
Prep for further changes to break up the `write_content_object()` path into
separate paths for archive, as well as regfile vs symlink in non-archive.
Closes: #1371
Approved by: jlebon
A while ago I did `truncate -s 0 /path/to/repo/00/123.commit`, and expected a
checksum error, but I actually got a validation error due to us loading the
commit into a variant and trying to parse out the parent checksum, etc.
I first started by changing the `load_and_fsck_one_object()` function to
checksum before loading, but the problem is that we do a traverse of all objects
first. Fixing this is going to require an `OSTREE_REPO_COMMIT_TRAVER_FLAG_FSCK`
or something.
In the meantime at least though, let's add a public API to fsck a single object
which *does* checksum cleanly before parsing the object, and change the `fsck`
command to use it.
We then change the fsck binary to do this while iterating over the refs
and finding the commit object. This way we'll at least get a checksum
first for commit objects, even if not dirtree/dirmeta.
Closes: #1364
Approved by: jlebon
This commit fixes an infinite loop that happens if you try to list the
remotes of a repo that has a parent repo set. It also adds a unit test
to ensure the right behavior, which is that both the child remotes and
parent remotes are listed.
Closes: #1366
Approved by: cgwalters
One major thing we can do to speed up local commits is multithreading. In
preparation for that, split up the recursion function so that the subdirectory
case is separate from the content (regfile/symlink) case. Then for non-subdirs,
we can easily peel off worker threads and gather the final checksums and update
the mtree from the main thread.
The diff here looks large but it's pretty straightforward; amazingly this change
compiled the very first time I tried it!
Closes: #1365
Approved by: jlebon
This seems to work around
https://github.com/ostreedev/ostree/issues/1362
Though I'm not entirely sure why yet. But at least with this it'll be easier for
people to work around things locally.
Closes: #1368
Approved by: jlebon
Add exclusive repository locking to all the pruning entry points. This
ensures that objects and deltas will not be removed while another
process is writing to the repository.
Closes: #1343
Approved by: cgwalters
Define an auto cleanup handler for use with repo locking. This is based
on the existing auto transaction cleanup. A wrapper for
ostree_repo_lock_push() is added with it. The intended usage is like so:
g_autoptr(OstreeRepoAutoLock) lock = NULL;
lock = ostree_repo_auto_lock_push (repo, lock_type, cancellable, error);
if (!lock)
return FALSE;
The functions and type are marked to be skipped by introspection since I
can't see them being usable from bindings.
Closes: #1343
Approved by: cgwalters
Currently ostree has no method of guarding against concurrent pruning.
When there are multiple repo writers, it's possible to have a pull or
commit race against a prune and end up with missing objects.
This adds a file based repo locking mechanism. The intention is to take
a shared lock when writing objects and an exclusive lock when deleting
them. In order to make use of the locking throughout the library in a
fine grained fashion, the lock acts recursively with a stack of lock
states. If the lock becomes exclusive, it will stay in that state until
the stack is unwound past the initial exclusive push. The file locking
is similar to GLnxLockFile in that it uses open file descriptor locks
but falls back to flock when needed.
The lock also attempts to be thread safe by storing the lock state in
thread local storage with GPrivate. This means that each thread will
have an independent lock for each repository it opens. There are some
drawbacks to that, but it seemed impossible to manage the lock state
coherently in the face of multithreaded access.
The API is a push/pop interface in accordance with the recursive nature
of the locking. The push interface uses an enum that's translated to
LOCK_SH or LOCK_EX as needed. Both interfaces use an internal timeout
field to decide whether to manage the lock in a blocking or non-blocking
fashion. The intention is to allow ostree applications as well as
administrators to control this timeout. For now, the default is a 30
second timeout.
Note that the timeout is handled synchronously in thread since the lock
is maintained in thread local storage. I.e., the thread that acquires
the lock needs to be the same thread that runs the operation. There may
be a way to offer an asynchronous version, but it's not clear exactly
how that would work since it would likely involve a separate thread that
invokes a callback when the locking operation completes.
https://bugzilla.gnome.org/show_bug.cgi?id=759442Closes: #1343
Approved by: cgwalters
I was getting a bare `error: Creating temp file: No such file or directory` when
debugging `test-concurrency.py`; with this I get
`error: Writing content object: Creating temp file: No such file or directory`
which helps me pin it down.
Closes: #1343
Approved by: cgwalters
Colin Walters
Matthew Leeds
Gatis Paeglis
Jonathan Lebon
Will Thompson
Marcus Folkesson
Philip Withnall
Dan Nicholson