Checkout was one of the first complex code paths I tried to convert to
*at(). I ended up keeping both, because I hit the "xattrs for a
symlink" problem. Later, Florian gave me a workaround, and we started
using it here, but the GFile * parameters weren't deleted. They're
not used, so do so now.
Starting down the path of not using libgsystem. The main win here
will be code sharing between ostree/rpm-ostree as well as going down
the path of not using GFile * for local files.
The previous diff algorithm was file tree based, and only looked
at modified files that lived at the same path.
However, components like the Linux kernel have versioned
subdirectories, e.g. /usr/lib/modules/$kver/.../ext4.ko. We want to
be able to detect these "modified renames" so that we can compute
diffs (rollsum, bsdiff).
This does an rsync-style prepared delta basically. On my test data,
it shaves ~6MB of uncompressed data. Not a huge amount, but I expect
this to be more useful for things like binaries which embed data, etc.
There's still some silliness here, but there is now only one opcode
open-splice-and-close, that writes a single chunk from the payload.
This is really all we need for metadata, and small content objects are
also fine with this.
We get some deduplication between content objects by creating a
dictionary for (uid,gid,mode) tuples and xattrs.
This still keeps the operation/payload code in, so we could do
rollsums in a future update easily.
I was hitting a bug in libguestfs/guestmount/FUSE where it blew up
with EINVAL on directories containing lots of files (more than
32000?). We really want to use prefixed subdirs just like the real
objects/ directory does.
This allows us to share more code between the paths, is more
efficient, etc.
gnome-continuous uses the ostree_repo_scan_hardlinks() mode to
avoid re-checksumming everything. However, when I ported the commit
code to use openat() and friends, this optimization was lost.
Re add it. The difference is about 15s versus 5 minutes.
This follows up from the previous commit; now that pull knows how to
do the efficient link() or copy for local files, we can just have
pull-local call into ostree_repo_pull().
As part of this:
- pull() can also accept a file:/// URI instead
of a remote name (since pull local supports anonymous pulls)
- pull() knows an "override-remote-name" option, since pull-local
supported writing a ref out even if there wasn't a remote with
that name
It's always been suboptimal to have both pull and pull-local; as we go
beyond the raw object data into things like deltas and summary files,
the logic to perform e.g. mirroring should only be in one place.
This will be used by Pulp's OSTree content plugin at least to perform
promotions.
When doing a pull --mirror from an archive-z2 repository into another
archive-z2 repository, currently we gunzip/checksum/gzip each content
object. The re-gzip process in particular is fairly expensive.
This does assume that the upstream content is trusted and correct.
It'd be nice in the future to do at least a CRC check, if not the full
checksum. (Could we append CRC data to the end of filez objects?)
We could also choose to only do this optimization if fetching over
TLS.
before: 1626 metadata, 20320 content objects fetched; 299634 KiB transferred in 62 seconds
after : 1626 metadata, 20320 content objects fetched; 299634 KiB transferred in 11 seconds
For future delta work where we do more interesting things than just
"tar of new objects", this lays the groundwork for doing streaming
writes into content objects.
It's also more efficient, as we avoid many intermediate allocations
and virtual calls. Just a single `g_output_stream_write_all` for the
splice case.
Conflicts:
src/libostree/ostree-repo-private.h
src/libostree/ostree-repo-static-delta-processing.c
We could just make everything relative to this, but the objects/ and
tmp/ are accessed very often, so I think it's worth holding individual
fds.
This fd can cover everything else: refs, deltas, etc.
subscription-manager has a daemon that runs in a confined domain,
and it doesn't have permission to write usr_t, which is the default
label of /ostree/deploy/$osname/deploy.
A better long term fix is probably to move the origin file into the
deployment root as /etc/ostree/origin.conf or so.
In the meantime, let's ensure the .origin files are labeled as
configuration.
If an object already existed and we somehow tried to pull it, the
caller would still expect a returned checksum.
This appears to happen with static deltas for some reason; we might be
including duplicate metadata objects. Regardless, this is a bug that
should be fixed.
We have a chain of checksums from the root up until here. While doing
checksums of the objects individually would be a good redundancy check
for test cases and the like, when doing a pull there's no good reason
to burn cycles on SHA256.
This caused deadlocks and/or EMFILE due to the interaction between
threads and fds. What we really want here is a better pull-based
model for parsing content objects.
Another idea would be to change static deltas so that content objects
have a special opcode that includes their metadata first, and then do
rollsums etc. only over actual content.
See projectatomic/rpm-ostree#42 for rationale. There are two high
level use cases:
- If the OS comes unconfigured, this is a way to point it at a repo of your choice.
- To switch between repositories while keeping the same branch easily.
At some point, we might want to expose a uniform way to refer
to deployments by an index. At the moment undeploy is the only
command that does.
I plan to introduce another command which optionally takes an index,
so prepare a helper function for this.
This is a noticeable cleanup, and fixes another big user of GFile* in
performance/security sensitive codepaths.
I'm specifically making this change because the static deltas code was
leaking temporary files, and cleaning that up nicely would be best if
we were fd relative.
This will be used for a later change to use openat() for the fetching
code. Note that we drop the code to use mmap() - it was an attempt to
avoid keeping a fd open, but we do correctly close anyways.
Regression from 86764dbf00
This function is kind of fiendish now that we have 3 cases, each of
which want to be optimized somewhat to only load what's necessary
(e.g. don't open the file if we don't have an output for stream
requested).
Clean things up so that BARE_USER and BARE are separate conditionals
that share as much as possible, and fix the bug that asserted we
were in BARE mode.
I tested this by running test-basic-user.sh by hand.
Some use cases for checkouts don't need to fsync during checkout.
Installer programs for example will just do a global fsync at the end.
In the future, the default "ostree admin" core could also be
rearchitected to only do a transaction commit right before reboot, and
do the fsync then.
https://bugzilla.gnome.org/show_bug.cgi?id=742482
This is just an efficiency optimization. We're getting fairly close
to all of the hot code paths using `*at()`.
Note that we end up maintaining a half-duplicate code path set here,
because we still need to support commits from an arbitrary GFile *,
which in a possible common case is an OSTree commit.
I think it's worth it though.
rpm-ostree had code to check for this, which didn't actually work.
I don't see a no backwards compatibility concern in changing this, as
it's unlikely a caller would try to sensibly disambiguate FAILED.
We were already using openat() for the contents, but not the xattrs.
Now that libgsystem 2014.3 has gs_fd_get_all_xattrs(), make use of it.
Clean things up a bit so we only open the fd once.
For Anaconda, I needed OSTREE_REPO_REMOTE_CHANGE_ADD_IF_NOT_EXISTS,
with the GFile *sysroot argument to avoid ugly hacks. We want to
write the content provided via "ostreesetup" as a remote to the target
chroot only in the case where it isn't provided as part of the tree
content itself.
This is also potentially useful in idempotent systems management tools
like Ansible.
https://bugzilla.gnome.org/show_bug.cgi?id=741577
rpm-ostree at least has the option to generate a tree with just that
instead of /boot, but while we were enumerating the latter, we'd still
return paths from /boot.
https://bugzilla.gnome.org/show_bug.cgi?id=740947
In Anaconda, we're using "ostree admin --sysroot=/mnt/sysimage
instutil set-kargs", and it was working before, but newer versions of
lorax strip out /etc/system-release which grub2 wants.
That was wrong anyways as we want the /etc/system-release from the
target root.
(Man, grub2 sucks...give me a declarative config file format I can just
write)
https://bugzilla.gnome.org/show_bug.cgi?id=740697
We potentially need a lot of argument types for pull. Rather than
have a C function with tons of arguments, let's use a GVariant a{sv}
as a handy extensible (and immutable) bag of properties.
This is prepratory work for adding an option to pull to traverse
history.
https://bugzilla.gnome.org/show_bug.cgi?id=737844
We need basic support for UEFI - many newer servers don't support
BIOS compatibility mode anymore.
However, this patch only implements non-atomic because UEFI is FAT, and
we can't do the previous design for OSTree of atomic swap of
/boot/loader.
The Fedora/RHEL UEFI layout has the kernels on a "real" /boot
partition, and /boot/efi/EFI/$vendor just holds the grub2 UEFI binary
and grub.cfg.
Following this, /boot/loader is still on the OS boot partition, and we
still atomically swap it. This potentially paves the way to atomic
upgrades in the future.
https://bugzilla.gnome.org/show_bug.cgi?id=724246
Some package systems need to be run as root, so the process linking to
libostree may also be root. However, it's reasonable to have the
target repository be owned by a uid other than root.
This patch makes it Just Work by chowning the file content to match.
Note this only operates on archive-z2 repositories, because you can't
usefully serve bare repositories via HTTP.
https://bugzilla.gnome.org/show_bug.cgi?id=738954
Colin Walters