We already set all file mtimes to 0 so that they are constant
over all checkouts, and can be made constant with a known value from
the system where the ostree was created.
However, this was not happening for directories. Zero their mtimes too.
This is important for shipping a fontconfig cache in the ostree;
the fontconfig cache files embed a directory mtime.
Some use cases for checkouts don't need to fsync during checkout.
Installer programs for example will just do a global fsync at the end.
In the future, the default "ostree admin" core could also be
rearchitected to only do a transaction commit right before reboot, and
do the fsync then.
https://bugzilla.gnome.org/show_bug.cgi?id=742482
This format is pretty much the same as the "bare" format, except the
file ownership and xattrs is not stored in the actual filesystem object, but
rather on the side in a user xattr. This means two things:
1) An unprivileged user can store such a repo independent of the types
of files in it or their xattrs. And you can later (as root)
reconstruct the real filesystem tree with ownership. Although you
can't do that using hardlink-sharing. This also means ostree
fsck does a full verification.
2) Such a repository can be checked out with user-mode (checkout -U)
as an unprivileged user using hardlinks for space sharing.
Additionally, symlinks are stored as regular files (with the content
being the symlink target) because user xattrs are not supported on
symlinks. We know at checkout time if the file is a symlink because
the original st_mode is stored in the xattr metadata.
https://bugzilla.gnome.org/show_bug.cgi?id=741125
Applying xattrs on a symlink during checkout failed since
it was setting the xattrs on the final filename, not the
temporary name.
This made the "checkout union 1" test in test-basic.sh
fail.
https://bugzilla.gnome.org/show_bug.cgi?id=741125
When commiting a symlink we do store the uid/gid of the actual
symlink (i.e. not target). However, this was not restored
on non-user-mode checkout as it should.
This commit fixes that, and additionally it ensures xattrs
on symlinks are not set in user-mode checkout.
https://bugzilla.gnome.org/show_bug.cgi?id=741125
fixes a coredump when using a command like:
$ ostree --repo=repo checkout -U --subpath=/usr/lib/passwd \
fedora-atomic/rawhide/x86_64/docker-host usrlib-new
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
While we did support disabling the uncompressed-objects-cache
per-repository:
1) We didn't actually respect that operation when doing
CHECKOUT_MODE_USER on archive-z2 repositories
2) It'd be better to automatically detect we can't write to the
repo and disable the uncompressed cache then.
Finally, fsync to ensure all entries are on disk, unless disabled.
We support disabling this for cases like server-side buildroot
construction where we don't need to be robust against power loss
The previous commit here changed things so that we do mkdir(x, 0700),
then fchmod later only if we created the directory.
However the logic was incorrect; we still need to chmod even in
MODE_USER if we created the directory.
This is a bit more efficient in that we're not walking full paths, and
it helps avoid security/reliability issues if an attacker (or just a
misbehaving process) has the ability to mutate paths in the middle.
This large patch moves the core xattr logic down into libgsystem,
which allows the gs_shutil_cp_a() API to copy them. In turn, this
allows us to just use that API instead of rolling our own recursive
copy here.
As noted in the new comment though, one case that we are explicitly
regressing is where the new /etc removes a parent directory that's
needed by a modified file. This seems unlikely for most vendors now,
but let's do that as a separate bug.
https://bugzilla.gnome.org/show_bug.cgi?id=711058
For the cases where we can't hardlink, use at-relative walking of the
path where possible. We still don't have lsetxattrat, so we also need
to deal with pathnames, but that is now only for symlinks.
Again, the advantages of this are a lot less malloc() of pathnames in
ostree, and much less time spent traversing paths inside the kernel.
https://bugzilla.gnome.org/show_bug.cgi?id=707733
Nothing external uses it. We keep ostree_get_xattrs_for_file() public
because it's convenient for external consumers to get xattrs in
exactly the format we desire.
https://bugzilla.gnome.org/show_bug.cgi?id=707733
Clean up how we deal with the uncompressed object cache; we now use
openat()/linkat() and such just like we do for the main objects/.
Use linkat() between the objects and the destination, if possible.
https://bugzilla.gnome.org/show_bug.cgi?id=707733
This avoids repeatedly traversing the target pathname, and is just
more efficient.
Part of a prelude to using fd-relative API for the source object path
too.
The way we recurse into subdirectories in parallel makes it far too
easy to hit up against the arbitrary Linux fd limit of 1024.
Since the fix here is about dropping parallelism, let's just go all
the way for now and make a plain old synchronous API =(
This does simplify both internal callers which wanted a sync API
anyways.
https://bugzilla.gnome.org/show_bug.cgi?id=706380
We'll always have "bare" mode for keeping files-as-hardlinks as root.
But "archive" was my second attempt at a format for non-root file
storage, used by the gnome-ostree buildsystem which runs as non-root.
It was really handy to have a "tar" like mode where I can create
tarballs as a user, that contain files owned by root for example.
The "archive" mode stored content files as two pieces in the
filesystem; ".file" contained metadata, and ".filecontent" was the
actual content, uncompressed. The nice thing about this was that to
check out a tree as non-root, you could just hardlink into the repo.
However, archive was fairly bad for serving via HTTP; it required
*two* HTTP requests per content object, greatly magnifing the already
inefficient fetch process. So "archive-z2" was introduced.
To allow gnome-ostree to still check out trees as a user, the
"uncompressed-object-cache" was introduced, and that's how things have
been working for a while.
So we should just be able to kill this code. Specifically note just
how much better the stage_object() function became.
https://bugzilla.gnome.org/show_bug.cgi?id=706057
Daniel Drake
Colin Walters
Alexander Larsson
Giuseppe Scrivano
Vadim Rutkovsky
Tobias Hunger