We shouldn't overwrite deploy/<osname>/previous unless we actually
have something new.
This fixes the bug where two consecutive:
$ ostree admin upgrade foo
$ ostree admin upgrade foo
would end up pruning the data for foo/previous from the repo.
This is admittedly a hack, since the story is that services should be
handling /var on their own. But on the other hand the current systemd
story is that admins should create it to enable it. Possibly a better
fix is --enable-journal-always or something for systemd.
We had a bit of legacy code that looked for "current" if the deploy
path was unspecified; this needs to be deploy/OSNAME/current.
Secondly, we need to point dracut at a tmpdir outside of the root when
doing FUSE, because directories with mode 0700 and owned by root:root
aren't writable.
Also update libgsystem to the latest.
Will be used by the QA scripts, since we need to inspect the kernel
configuration and generate a correct grub conf from that, rather than
trying to have update-bootloader reuse an existing config, since there
won't be one initially.
In some cases we want the ability to run triggers independently of
checking out a tree. For example, due to kernel limitations which
impact the gnome-ostree build system, we may need to run triggers on
first boot via systemd.
Secondarily, if the user installs a system extension which adds a new
shared library to /usr/lib for example, the system will need to run
the triggers again.
Also, I think I want to take triggers out of the core and put them in
ostree admin anyways.
The real vision of OSTree is to "multiple versions of multiple
operating systems". Up until now, it's worked to install gnome-ostree
inside a host distribution, but several things don't work quite right
if you try to do completely different systems.
In the new model, there's the concept of an "osname" which encompasses
a few properties:
1) Its own /var
2) A set of trees deployed in /ostree/deploy/OSNAME/
3) Its own "current" and "previous" links.
Now it no longer really makes sense to boot with "ostree=current".
Instead, you specify e.g. "ostree=gnome/current".
This is an incompatible change to the deployment code - you will need
to run init-os gnome and redeploy.
All "ostree admin" subcommands now take an OSNAME argument.
This counts as an implicit "hold" on the first items, which we must
complete. Otherwise there are race conditions where the queue may
signal idle when in fact the main process is still working.
Should be used when initializing a new root filesystem for a "pure
OSTree" system; for example, what "ostbuild privhelper-deploy-qemu"
does when creating a filesystem image loopback.
Previously we'd open(path, O_NOATIME) and do a series of small read()
calls to just parse the header. I think this will trigger kernel readahead
into the compressed portion, but we don't care about that.
This should be more efficient.
We were blocking for easily 1/10 or 1/5 of a second in fdatasync(),
which drastically slows down the whole process.
This threading isn't quite as good as the ostree-pull command, but it
lets us avoid the dependency on libsoup everywhere, and it's simpler.
After a while of pull-deploy cycles, you start to accumulate a lot of
them. While the deployment read-only part is hardlinked, the -etc
space adds up.
Additionally, the repository itself just gets large.
The new command "ostree admin prune" deletes everything except the
"current" and "previous" deployments.
The previous code (unintentionally) only traversed from refs; so data
only reachable from previous commits would be deleted. That shouldn't
be the default, but we do want to offer it as an option.
So add a --refs-only option.
At a minimum, it should list the available subcommands. This is
still not perfect, since there is no way to get at the help output
of the subcommands - getting that right needs more refactoring.
Signed-off-by: Colin Walters <walters@verbum.org>
This seems to work around a likely Linux kernel VFS bug, where I
randomly see ENOENT on link() when we *definitely* called mkdir() at
an earlier point in time.
This is an incompatible change to archive-z, thus it is now renamed to
archive-z2 and ostree will no longer parse archive-z.
I noticed in perf that we were spending some time zlib-decompressing
file headers, which is just inefficient. Rather than do this, keep
the headers uncompressed, and just zlib-compress content.
This gives us something closer to the advantages of archive and
archive-z when using the latter. Concretely we get deduplication
among multiple checkouts, along with the "devino" hash table trick
during commits to avoid checksumming content again.
This is enabled by default.
For similar reasons as metadata, this avoids having the main thread
blocked in fdatasync(), and even better - we can achieve much higher
parallelism if we have multiple threads blocked on fdatasync().
We need this hack for "archive mode" repositories; otherwise,
what ends up happening is that we get 10000+ requests pending
for .filemeta files, which we can't process until we also get
the .filecontent.
Note this hack is unneccessary when fetching from archive-z
repositories.
Create a worker thread for processing metadata, reserving the main
thread for HTTP requests.
This can create a very significant efficiency win for large pull
requests since we are much more likely to keep a full pipeline open.
The status display is also nicer now.
This is where loose content objects are stored as one compressed file,
instead of the two separate ones for regular archive mode. This mode
would be suitable for HTTP servers, beause only one HTTP request is
necessary, and the result would be compressed.
Cleanly separate metadata/content APIs, rather than defaulting to
raw streams. This helps most use cases.
Also, drop support for staging content without knowing the total
length. This complicated the code, and for things like streaming
HTTP, we should be able to figure this out from Content-Length.
They're not a large efficiency win at the moment, because we don't
do any delta compression.
At the moment, they simply served to compress data, but we will change
the archive mode to do that by default.
Configuration associated with a specific revision is stored in a folder
named <revision>-etc. In a similar spirit, add /ostree/current-etc, pointing
to the -etc folder for the revision named by current. This allows
easy editing of configuration from the host distribution, and allows
diffing current/etc and current-etc for configuration changes.
https://bugzilla.gnome.org/show_bug.cgi?id=684049
First, move deployments to /ostree/deploy. Having them in the
toplevel clutters the filesystem layout too much.
When we deploy a tree like /ostree/deploy/NAME, there is now also a
writable directory /ostree/deploy/NAME-etc. This is mounted as
read-write inside the system.
On an initial install, that directory is copied from
/ostree/deploy/NAME/etc. On subsequent deployments, we find any
changes made in the current deployment's /etc, and apply that set of
changes to the new deployment's /etc.
See https://live.gnome.org/OSTree/EverythingInEtcIsABug
I run builds on my laptop, but it also crashes about 1/4 of the time
while suspending. It's definitely undesrirable to get e.g. empty
.dirtree objects because they corrupt builds. Concretely, I was
getting empty contents committed for xorg-util-macros.
Now, we used to write out temporary files using g_file_replace() which
does a fsync() during close, but then switched to a more "manual"
g_file_append_to().
We could switch back to g_file_replace(), but the problem is, we don't
want to call fsync() on temporary files in the case where we already
have the object. Attempting to add an object we already have is a
*very* common case.
This is both the old and new code sequence for the case where an
object is already stored:
open(temp, O_WRONLY)
write() write() write()
close()
lstat(objects/3a/9fe332...) = 0
unlink(temp)
In the *new* code, here's the case where an object *isn't* stored:
open(temp, O_WRONLY)
write() write() write()
close()
lstat(objects/3a/9fe332...) = -1
open(temp, O_RDONLY)
fdatasync()
close()
rename(temp, objects/3a/9fe332)
Compare with the *old* code path for when an object isn't stored:
open(temp, O_WRONLY)
write() write() write()
close()
lstat(objects/3a/9fe332...) = -1
link(temp, objects/3a/9fe332)
unlink(temp)
The problem with this is we really need to fdatasync(). Also doing
just rename() instead of the weird link()/unlink() helps us express to
the filesystem that we want atomic semantics. For example, BTRFS has
special handling for rename().
We really don't have a sane story for private files. This is a
defensive step ensuring that with old versions of gnome-ostree,
components that mistakenly have un-world-readable files don't break
pulls.
Use the new update-cache mode of pango-querymodules, which
automatically finds the correct cache file location.
Updated to look in both places by Colin Walters <walters@verbum.org>
https://bugzilla.gnome.org/show_bug.cgi?id=682411
This command regenerates kernel-dependent files. It is meant to
be run after kernel upgrades in the host distribution, to keep
the ostree installation running, and attempts to figure out the
right version automatically (and without changing the ostree checkout)
As the manual page doesn't say, but the in-code kernel documentation
shows, hardlinking for normal users can fail for a variety of
reasons (including very common situations such as non regular file
or non writable file), if the owner of the file does not match
the user linking (e.g. when checking out a shadow repo with a root-
owned master).
If that happens, fail back silently to copying instead of aborting
the whole operation.
https://bugzilla.gnome.org/show_bug.cgi?id=682298
Common code was duplicating the command name and not shifting
arguments properly, which thus required the builtins to be aware
of it, instead of being treated like subcommands.
The qemu helper really wants to copy kernel modules, but not
update the system bootloader. Allow it to reuse ostadmin for
this.
Note that our previous path of shelling out to "cp -al" broke because
it refused to make cross-device links.
# Please enter the commit message for your changes. Lines starting
# with '#' will be ignored, and an empty message aborts the commit.
# On branch master
# Your branch is ahead of 'origin/master' by 1 commit.
#
# Changes to be committed:
# (use "git reset HEAD <file>..." to unstage)
#
# modified: src/libotutil/ot-gio-utils.c
# modified: src/libotutil/ot-gio-utils.h
# modified: src/ostadmin/ot-admin-builtin-deploy.c
#
# Untracked files:
# (use "git add <file>..." to include in what will be committed)
#
# embedded-dependencies/glib/
# embedded-dependencies/libsoup/
The old command line syntax assumed you wanted the latest from
a given revision, but to bisect, you really want the ability
to deploy any given one.
Old:
$ ostadmin deploy gnomeos-3.6-i686-{runtime,devel}
New:
$ ostadmin deploy trees/gnomeos-3.6-i686-runtime
$ ostadmin deploy trees/gnomeos-3.6-i686-devel
And note that the deploy directory now changes to:
/ostree/trees/gnomeos-3.6-i686-devel
And now it's possible to:
$ ostadmin deploy trees/gnomeos-3.6-i686-devel trees/gnomeos-3.6-i686-devel^^^
This helps make things less confusing when we try to create a file not
owned by our uid, such as running ostree-pull as non-root on a
repository owned by root.
The "status text" code was assuming that libsoup was only doing one
thing at a time, but that's never been the case. Change the code to
display the status of all active requests.
This can be a large performance win in certain circumstances:
* Cold buffer cache (we don't block the whole process)
* Requiring a copy instead of hardlink
The previous fix to just ignore symbolic links for hard linking isn't
really good enough, since it can happen for empty files too.
Since this is an optimization, when we get EMLINK, let's instead just
fall back to copying. This also applies to EXDEV.
Rather than always doing:
1) make temporary link
2) unlink() target
3) rename()
Just try making the link, and only do the second two if the file
already exists. This reduces system call traffic a lot.
We should explicitly distinguish between the case where we have a git
branch we're following, versus an immutable tag. In the latter case,
we can entirely avoid running 'git fetch' for it once we have it.
This is a noticeable speedup in our current scenario of pinning WebKit
to a tag.
The git mirroring code now has a --fetch-skip-secs=X option which
allows us to basically run it in a loop, without hitting remote git
repositories too often.
Colin Walters
Giovanni Campagna
Jasper St. Pierre
Matthias Clasen
Stef Walter
Adrian Perez