If we pass XATTR_REPLACE then the attribute must already exist, which
is not our intent. Passing zero creates the attribute if necessary,
or replaces it when it already exists.
https://bugzilla.gnome.org/show_bug.cgi?id=705893
* Specifying global options after the command for a more natural:
# ostree commit --repo=/path/to/repo ...
* Support asking for --help without --repo
# ostree commit --help
* Support short form of -h
* Support specifying --repo without equals sign
# ostree --repo /path/to/repo commit ...
* Support global --help and -h
# ostree --help
* Ditto for ostree admin sub commands
* Removed some leaky code
https://bugzilla.gnome.org/show_bug.cgi?id=705903
While the actual commit object format is presently the same, for a
number of reasons we'd like to change it fairly radically. Among
other things, we need to drop our a{sv} types in objects, to protect
against GVariant changing format.
Since now gnome-ostree now longer uses related objects, and nothing
ever used metadata, just drop them both.
If the admin encounters corruption and does:
$ ostree admin fsck --delete
We want them to be able to recover the objects easily from the
network; with this patch, they do:
$ ln -s dummyvalue /ostree/repo/transaction
$ ostree refs --delete remotename:branchname
$ ostree pull remotename
This patch avoids the need for the refs --delete; we might as well
force scan the commit, and with this patch we still print that it
changed.
We may revive this later, but commits in their current form aren't
very useful for humans to read, so it doesn't make sense to have a
tool to show a history of useless stuff.
More interesting things are diffs between commits, object statistics,
etc.
Otherwise it's really easy to keep accumulating deployments. Also, we
may want to run this after rebooting, so we're back down to one
operating system.
And disable devino scan by default. For the gnome-ostree build case,
our commits are from "make install DESTDIR=", so they won't be
hardlinks into the repo. In that case, particularly as the repository
size grows, the cost of building up the devino -> checksum mapping
becomes a problem.
The compose step will use this option though.
While the first was useful way back in the day when we were importing
Debian bits and /sbin/init was expecting to find /dev/.initctl as a
named pipe, that's no longer an issue with systemd since it uses
dynamic Unix sockets.
Likewise, character and block devices in /dev are now dynamically
created by the devtmpfs from the kernel.
Less complexity and code here if we just support directories, regular
files, and symbolic links.
Calling it "cleanup" is better since it does more than repo pruning.
We were also doing a prune twice; ot_admin_cleanup() already does one,
so drop the bits to do it in cleanup.c.
It isn't useful at the moment, since the deploy stuff all changed. It
will make sense to bring back later, but for now let's not carry
broken untested code.
While the systemd integration effectively requires /sysroot, it will
help people trying to use OSTree with other initramfs systems
(e.g. initramfs-tools) if we don't hardcode that requirement in this
tool.
If pull is interrupted, we may have downloaded an arbitrary subset of
the requested objects. Previously, we handled this by scanning for
all objects each time.
However, there's an easy optimization - this patch creates a lock file
in the repo. If we don't see that file when starting a pull, we know
we don't need to stat() every file; presence of a dirtree object for
example implies the existence of everything it references.
Originally, the idea was that clients would replicate "OS/tree"s from
a build server, but we'd run things like "ldconfig" on the client.
This was to allow adding e.g. the nVidia binary driver.
However, the triggers were the only thing in the system at the moment
that really had expected knowledge of the *contents* of the OS, like
the location of binaries.
For now, it's architecturally cleaner if we move the burden of
triggers to the tree builder (e.g. gnome-ostree or RPM). Eventually
we may want OSTree to assist with this type of thing (perhaps
something like RPM %ghost), but this is the right thing to do now.
See https://wiki.gnome.org/OSTree/DeploymentModel2
This is a major rework of the on-disk filesystem layout, and the boot
process. OSTree now explicitly supports upgrading kernels, and these
upgrades are also atomic.
The core concept of the new model is the "deployment list", which is
an ordered list of bootable operating system trees. The deployment
list is reflected in the bootloader configuration; which has a kernel
argument that tells the initramfs (dracut) which operating system root
to use.
Invidiual notable changes that come along with this:
1) Operating systems should now come with their etc in usr/etc; OSTree
will perform a 3-way merge at deployment time, and place etc in
the actual root. This avoids the need for a bind mount, and is
just a lot cleaner.
2) OSTree no longer bind mounts /root, /home, and /tmp. It is expected
that the the OS/ has these as symbolic links into /var.
At the moment, OSTree only supports managing syslinux; other
bootloader backends will follow.
This version of the code doesn't know what to do with new-model trees,
so just abort if we see that. We'll likely never care about upgrades
from 1.0 to 2.0.
A simple HTTP server implementation is so few lines of code when one
is linking to libsoup anyways, so let's just have one here in ostree
that will be used for the test suite.
This allows us to run the archive tests that previously required
apache even in gnome-ostree.
Even if very suboptimally, for now; we copy the files, then copy them
again.
The obvious long term plan is to merge pull-local and pull together,
but truly optimizing that requires the pull code to know how to use
the OstreeRepo APIs when operating on local repositories (as
pull-local does), rather than assuming the remote is an archive-z
fetched over HTTP.
We could drop into g_main_loop_run() after the worker
thread had called g_main_loop_quit().
Fix this by following the pattern suggested by Ryan of a while loop
around our termination condition, and g_main_context_iteration().
Linux creates a copy of the soure mount flags when creating a bind
mount; if the source is read-only, then the bind mount is.
The problem is that systemd will remount the rootfs read/write, but
each mount (/home, /var etc.) will still be read-only. We need to
remount every bind mount except for /usr to read-write too.
This only "worked" with the old ostree-switch-root because it
effectively force mounted the rootfs read-write always, ignoring the
"ro" flag.
This installs a Dracut module which parses the ostree= kernel command
line argument, and if given, sets up the OS/ at /sysroot, which
systemd's switch-root then moves into. This only works if dracut is
configured to use systemd itself.
Rather than attempting to hack up the "switch-root" functionality of
systemd, this binary allows us to simply prepare the root before we
switch into it.
This is useful for the gnome-ostree build system where each build is
one commit, but it's split up into /runtime /devel /debug etc. trees.
Ideally we wouldn't have a /debug subdirectory for "noarch"
components for example.
So add an option to not error out if the given path doesn't exist in
the commit.
When multiple threads need to uncompress an object, there was
a race condition where thread A could get EEXIST, unlink,
then thread B calls linkat(), then thread A tries to link() but
fails.
We can just loop in this case.
While this still isn't fully atomic (that depends on the bootloader),
this better ensures that the deployed kernel is booted with the
intended tree. For example, if we get ENOSPC when writing out the
kernel, we won't have swapped the symlink.
It's possible (likely even) that /tmp is on a separate filesystem; in
that case, a raw rename() is going to fail.
Saw this on the ostree.gnome.org builder.
Previously, I've observed bugs where we either:
1) Exit too early, leaving undownloaded objects
2) Hang while downloading
This rewrite hopefully fixes both.
We shouldn't overwrite deploy/<osname>/previous unless we actually
have something new.
This fixes the bug where two consecutive:
$ ostree admin upgrade foo
$ ostree admin upgrade foo
would end up pruning the data for foo/previous from the repo.
This is admittedly a hack, since the story is that services should be
handling /var on their own. But on the other hand the current systemd
story is that admins should create it to enable it. Possibly a better
fix is --enable-journal-always or something for systemd.
We had a bit of legacy code that looked for "current" if the deploy
path was unspecified; this needs to be deploy/OSNAME/current.
Secondly, we need to point dracut at a tmpdir outside of the root when
doing FUSE, because directories with mode 0700 and owned by root:root
aren't writable.
Also update libgsystem to the latest.
Will be used by the QA scripts, since we need to inspect the kernel
configuration and generate a correct grub conf from that, rather than
trying to have update-bootloader reuse an existing config, since there
won't be one initially.
In some cases we want the ability to run triggers independently of
checking out a tree. For example, due to kernel limitations which
impact the gnome-ostree build system, we may need to run triggers on
first boot via systemd.
Secondarily, if the user installs a system extension which adds a new
shared library to /usr/lib for example, the system will need to run
the triggers again.
Also, I think I want to take triggers out of the core and put them in
ostree admin anyways.
The real vision of OSTree is to "multiple versions of multiple
operating systems". Up until now, it's worked to install gnome-ostree
inside a host distribution, but several things don't work quite right
if you try to do completely different systems.
In the new model, there's the concept of an "osname" which encompasses
a few properties:
1) Its own /var
2) A set of trees deployed in /ostree/deploy/OSNAME/
3) Its own "current" and "previous" links.
Now it no longer really makes sense to boot with "ostree=current".
Instead, you specify e.g. "ostree=gnome/current".
This is an incompatible change to the deployment code - you will need
to run init-os gnome and redeploy.
All "ostree admin" subcommands now take an OSNAME argument.
This counts as an implicit "hold" on the first items, which we must
complete. Otherwise there are race conditions where the queue may
signal idle when in fact the main process is still working.
Should be used when initializing a new root filesystem for a "pure
OSTree" system; for example, what "ostbuild privhelper-deploy-qemu"
does when creating a filesystem image loopback.
Previously we'd open(path, O_NOATIME) and do a series of small read()
calls to just parse the header. I think this will trigger kernel readahead
into the compressed portion, but we don't care about that.
This should be more efficient.
We were blocking for easily 1/10 or 1/5 of a second in fdatasync(),
which drastically slows down the whole process.
This threading isn't quite as good as the ostree-pull command, but it
lets us avoid the dependency on libsoup everywhere, and it's simpler.
After a while of pull-deploy cycles, you start to accumulate a lot of
them. While the deployment read-only part is hardlinked, the -etc
space adds up.
Additionally, the repository itself just gets large.
The new command "ostree admin prune" deletes everything except the
"current" and "previous" deployments.
The previous code (unintentionally) only traversed from refs; so data
only reachable from previous commits would be deleted. That shouldn't
be the default, but we do want to offer it as an option.
So add a --refs-only option.
At a minimum, it should list the available subcommands. This is
still not perfect, since there is no way to get at the help output
of the subcommands - getting that right needs more refactoring.
Signed-off-by: Colin Walters <walters@verbum.org>
This seems to work around a likely Linux kernel VFS bug, where I
randomly see ENOENT on link() when we *definitely* called mkdir() at
an earlier point in time.
This is an incompatible change to archive-z, thus it is now renamed to
archive-z2 and ostree will no longer parse archive-z.
I noticed in perf that we were spending some time zlib-decompressing
file headers, which is just inefficient. Rather than do this, keep
the headers uncompressed, and just zlib-compress content.
This gives us something closer to the advantages of archive and
archive-z when using the latter. Concretely we get deduplication
among multiple checkouts, along with the "devino" hash table trick
during commits to avoid checksumming content again.
This is enabled by default.
For similar reasons as metadata, this avoids having the main thread
blocked in fdatasync(), and even better - we can achieve much higher
parallelism if we have multiple threads blocked on fdatasync().
We need this hack for "archive mode" repositories; otherwise,
what ends up happening is that we get 10000+ requests pending
for .filemeta files, which we can't process until we also get
the .filecontent.
Note this hack is unneccessary when fetching from archive-z
repositories.
Create a worker thread for processing metadata, reserving the main
thread for HTTP requests.
This can create a very significant efficiency win for large pull
requests since we are much more likely to keep a full pipeline open.
The status display is also nicer now.
This is where loose content objects are stored as one compressed file,
instead of the two separate ones for regular archive mode. This mode
would be suitable for HTTP servers, beause only one HTTP request is
necessary, and the result would be compressed.
Cleanly separate metadata/content APIs, rather than defaulting to
raw streams. This helps most use cases.
Also, drop support for staging content without knowing the total
length. This complicated the code, and for things like streaming
HTTP, we should be able to figure this out from Content-Length.
They're not a large efficiency win at the moment, because we don't
do any delta compression.
At the moment, they simply served to compress data, but we will change
the archive mode to do that by default.
Configuration associated with a specific revision is stored in a folder
named <revision>-etc. In a similar spirit, add /ostree/current-etc, pointing
to the -etc folder for the revision named by current. This allows
easy editing of configuration from the host distribution, and allows
diffing current/etc and current-etc for configuration changes.
https://bugzilla.gnome.org/show_bug.cgi?id=684049
First, move deployments to /ostree/deploy. Having them in the
toplevel clutters the filesystem layout too much.
When we deploy a tree like /ostree/deploy/NAME, there is now also a
writable directory /ostree/deploy/NAME-etc. This is mounted as
read-write inside the system.
On an initial install, that directory is copied from
/ostree/deploy/NAME/etc. On subsequent deployments, we find any
changes made in the current deployment's /etc, and apply that set of
changes to the new deployment's /etc.
See https://live.gnome.org/OSTree/EverythingInEtcIsABug
Colin Walters
Stef Walter
Giovanni Campagna
Jasper St. Pierre
Matthias Clasen