We shouldn't g_print() from a library, particularly when the
expectation is that the client has an async progress set up.
This should fix the pull output extending the status line.
If a MITM attacker (or just network corruption) causes a temporary
downloaded object in tmp/ to be corrupted, we'll end up
continually trying to commit it, and fail.
Fix this unlinking the temp file immediately after opening it. This
will ensure that if we exit due to an error (or crash), the kernel
will clean up the space for us.
https://bugzilla.gnome.org/show_bug.cgi?id=725924
The idea with this is that things like yum should be able to look for
it and determine whether or not they should assume that they can
change things on the system.
https://bugzilla.gnome.org/show_bug.cgi?id=725380
This is a bit more efficient in that we're not walking full paths, and
it helps avoid security/reliability issues if an attacker (or just a
misbehaving process) has the ability to mutate paths in the middle.
Mixing async and threads has proved to be too much for my little mind.
It has race conditions that I've tried repeatedly to fix, but failed.
The threading here was scanning metadata objects - and there are
two parts to that:
1) Physically loading them from disk
2) Parsing them
Now #1 has been partially addressed by avoiding a storm of lstat() if
we're starting from a known working state. If pull gets interrupted,
then we do need to rescan all objects. Also, we can address this with
local metadata packfiles.
The other potentially slow bit is that we recurse across the metadata,
blocking the main thread. We could ameliorate that in the future by
scheduling metadata parsing as idle "chunks".
Anyways, let's move the needle back to reliability, and readd speed
more carefully.
https://bugzilla.gnome.org/show_bug.cgi?id=706456
We don't want to allow MITM attackers to intercept upgrade requests
and provide clients with older OS versions vulnerable to security
flaws.
Only "ostree admin upgrade" gets this behavior for now - whether we
want to do it for "ostree admin switch" is another question.
It's better if this is independent from the OstreeSysroot; for
example, a policy is active in a given deployment root at once, not
for a sysroot globally.
We can also collect SELinux-related API in one place.
Unfortunately at the moment there can be only one instance of this
class per process.
We're seeing some hangs while ostree is fetching updates.
I imagine the fact that SoupSessionAsync has no timeout by default
could be the cause of this.
Set timeout values to 60 seconds, which is the default for the new
SoupSession API which we may switch to later.
https://bugzilla.gnome.org/show_bug.cgi?id=724310
tmpfiles.d configurations generally require write access to some places
that are read-only until ostree-remount runs.
Make sure ostree-remount has run first.
Thanks to Cosimo Cecchi for finding and diagnosing this problem.
https://bugzilla.gnome.org/show_bug.cgi?id=724183
The instructions one finds on the internets are apparently wrong, we
really need to keep the default here, since gpgme uses it to actually
find the helper binary it runs.
This fixes the GPG tests for me on EL7 at least.
This has a very basic level of functionality (deltas can be generated,
and applied offline). There is only some stubbed out pull code to
fetch them via HTTP.
But, better to commit this now and improve it from a known starting
point, rather than have it languish in a branch.
First, /var needs to be labeled at least once. We should probably
rearrange things so that /var is only created (and labeled) on the
first deployment, but this patch adds a /var/.ostree-selabeled file
instead.
Second, when doing the /etc merge, we compare the xattrs of the old
/usr/etc versus the current /etc. The problem with that is that the
policy has different labels for /usr/etc on disk than the real /etc.
The correct fix for this is a bit invasive - we have to take the
physical content of the old /usr/etc, but compare the labels as if
they were really in /etc.
Instead for now, just ignore changes to xattrs. If the file
content/mode changes, then we take the new file (including any changed
xattrs).
Bottom line: just doing chcon -t blah_t /etc/foo.conf may be lost on
upgrade (for now).
This will be used by guestmount - it's WAY faster. We only take disks
as a unit, so it's safe. If the process fails halfway through, we
just start over from scratch the next time anyways.
The trees as shipped come with /usr/etc, which should just be labeled
as usr_t. When we do a deployment, we need to relabel the copies of
the files we're making in /etc.
SELinux support is compile and runtime optional.
The intent of this code I'm fairly certain was to use *.gpg from the
trusted.gpg.d, directory. But right now, we're only using
"pubring.gpg" from that directory, which is odd.
Let's fix this to use all keys ending in .gpg, which will also
include pubring.gpg.
Only send _IDLE messages if and only if we state transition the main
thread (from idle -> !idle or !idle -> idle). This ensures that we
don't send IDLE, then get it back, and process that when we're !idle.
This is a redesign (again) of the pull code. It is simpler and
survives 20 minutes of testing in a loop, whereas the old code would
only go from 30 seconds to 2 minutes.
The problem with the old code was that there was a race where we might
determine idle state even when there are content requests in flight
between the metadata thread and the main one.
This code majorly reworks things - there's now only one IDLE message,
sent in a circle from the main thread, through the metadata scanner,
and back to the main one.
Crucially it's only sent when the *main* thread is idle. Previously
we were looking at whether the metadata scanner is idle, but that
doesn't make a lot of sense. First let's make sure the main thread is
idle, then verify that the metadata one is.
This closes the loop because we'll have ensured we get any pending
requests.
https://bugzilla.gnome.org/show_bug.cgi?id=706456
The "ordered hash" code was really just for kernel arguments. And it
turns out it needs to be a multihash (for e.g. multiple console=
arguments).
So turn the OstreeOrderedHash into OstreeKernelArgs, and move the bits
to split key=value and such into there.
Now we're not making this public API yet - the public OstreeSysroot
just takes char **kargs. To facilitate code reuse between ostree/ and
libostree/, make it a noinst libtool library. It'll be duplicated in
the binary and library, but that's OK for now. We can investigate
making OstreeKernelArgs public later.
https://bugzilla.gnome.org/show_bug.cgi?id=721136
The official way to add bootloader arguments to the current deployment
is to redeploy with --karg. However, doing so tripped up an
optimization made inside the deployment code to just swap the
bootlinks if we're keeping the same "bootcsum".
Change this optimization to look at the pair of (bootcsum, options).
We can't use #ifdef in the headers, since then g-ir-scanner won't pick
up the functions (unless we included config.h). Let's instead always
have the symbols, but just set an error if we were built without
support for it, just like how pull works.
This large patch moves the core xattr logic down into libgsystem,
which allows the gs_shutil_cp_a() API to copy them. In turn, this
allows us to just use that API instead of rolling our own recursive
copy here.
As noted in the new comment though, one case that we are explicitly
regressing is where the new /etc removes a parent directory that's
needed by a modified file. This seems unlikely for most vendors now,
but let's do that as a separate bug.
https://bugzilla.gnome.org/show_bug.cgi?id=711058
Previously the progress meter would bump in large chunks after we
completed a download. Instead, poll in progress files via fstat() for
their size, and add those to the running total.
The libostree already treats passing NULL for osname as "booted
osname, if any". We should do the same inside the tools. The upgrade
builtin had this logic duplicated there; we should be able to safely
remove it.
https://bugzilla.gnome.org/show_bug.cgi?id=710970
When booted into an ostree you can deploy without passing
an --os option. That was crashing though, because
ot_admin_complete_deploy_one is called with NULL
osname but it was not handling it properly.
https://bugzilla.gnome.org/show_bug.cgi?id=710970
Several APIs in libostree were moved there from the commandline code,
and have hardcoded g_print() for progress and notifications. This
isn't useful for people who want to write PackageKit backends, custom
GUIs and the like.
From what I can tell, there isn't really a winning precedent in GLib
for progress notifications.
PackageKit has the model where the source has GObject properties that
change as async ops execute, which isn't bad...but I'd like something
a bit more general where say you can have multiple outstanding async
ops and sensibly track their state.
So, OstreeAsyncProgress is basically a threadsafe property bag with a
change notification signal.
Use this new API to move the GSConsole usage (i.e. g_print()) out from
libostree/ and into ostree/.
Add a --generate-sizes option to commit to add size information to the
commit metadata. This will be used by higher level code which wants
to determine the total size necessary for downloading.
The gpgme examples use this, but from what I can tell we don't really
need to because we don't need detailed results; we only care whether
we signed it at all.
We need to use the full shutil_rm_rf() in order to actually delete
complete directories.
Test suite code based on a patch from Sjoerd Simons <sjored@luon.net>
https://bugzilla.gnome.org/show_bug.cgi?id=710097
Adapted from Google protobufs. For several cases, we want to support
e.g. file sizes up to guint64, but paying the cost of 8 bytes for each
number is too high.
This will be used for static deltas and sizes metadata.
If a deployment is somehow in the list twice, the hash table will free
the *new* value with g_hash_table_insert which gets all broken. Just
use g_hash_table_replace().
This commit changes the sysroot API so that one can create arbitrary
new deployment checkouts, then commit them as one step. This is to
enable things like an automatic bisection tool which say create 50
deployments at once, then when done clean them up.
This also moves some printfs from the library into src/ostree.
I plan to rename all of these APIs to use the term 'loose', so that it
makes more sense after pack files are introduced. External users
should not use them; instead use _load_variant() or _read_commit().
This uses gpgv for verification against DATADIR/ostree/pubring.gpg by
default. The keyring can be overridden by specifying OSTREE_GPG_HOME.
Add a unit test for commit signing with gpg key and verifying on pull;
to implement this we ship a test GPG key generated with no password
for Ostree Tester <test@test.com>.
Change all of the existing tests to disable GPG verification.
Add an optional dependency on gpgme to add GPG signatures into the
detached metadata, with the key "ostree.gpgsigs", as an "aay", an
array of signatures (treated as binary data).
The commit command gains a --gpg-sign=<key-id> argument. Also add an
argument --gpg-homedir to set the GPG homedir where we look for
keyrings.
I was getting hangs in the test suite, and looking at the previous
commit, we were calling the async completion functions out of the
finalizer for the URI, which is weird. I didn't analyze what's going
wrong, but what we really should be doing is processing our internal
queue after we've downloaded a file, and the request is about to be
finalized.
I suspect doing queue management from the finalizer created a circular
reference type situation.
This patch deduplicates the queue processing bits too.
https://bugzilla.gnome.org/show_bug.cgi?id=708126
On a large ostree repository pulling over http slows to a crawl. Pulling
from localhost results in:
5944 metadata, 63734 content objects fetched; 850509 KiB transferred in
1106 seconds
In other words about 800KiB/s. Some profiling shows that essentially
all of the CPU goes into libsoup doing its request bookkeeping instead
of into the actual downloading.
Adding a simple queue to limit to number of active request sent into
libsoup makes for a dramatic improvement:
5944 metadata, 63734 content objects fetched; 850509 KiB transferred
in 89 seconds
So around 9450 KiB/s.
https://bugzilla.gnome.org/show_bug.cgi?id=708126
If we had two deployments with different boot checksums, and were
trying to remove the one that was the same and add a new one (the
normal case), we'd end up assuming due to comparison with 0 that
we only needed to do the fast subbootversion swap.
Fix this by actually putting 1 where we really mean 1.
And update the tests to verify the fix; I have double-verified by
undoing the fix, and noting that the test fails.
https://bugzilla.gnome.org/show_bug.cgi?id=708351
The actual deployment checksum shouldn't be in there, because we may
just swap bootlinks, rendering the name of the old bootloader entry
file invalid. Thankfully nothing actually parsed the names of these
files, so let's just use the index.
This is just something I noticed on inspection; we should catch any
changes to /boot in the sync(), even though theoretically gio should
have done fdatasync().
Now that we have a real GObject for the sysroot, we have a convenient
place to keep track of 4 pieces of state:
* The current deployment list
* The current bootversion
* The current subbootversion
* The current booted deployment (if any)
Avoid requiring callers to pass all of this around and load it
piecemeal; instead the new thing is ostree_sysroot_load().
I ran into Jeremy Katz today, and he gave me permission to relicense
the small bits of switch-root.c to LGPLv2+. This combined with
permission from Peter Jones allows OSTree to become fully LGPLv2+.
Not a big deal, it's just a lot clearer to only have one license, and
it makes it easier to turn application code into library code.
read_commit resolves the ref to a commit, and a lot of consumers want
the resolved commit for their own purposes; this prevents them from
calling resolve_rev themselves.
https://bugzilla.gnome.org/show_bug.cgi?id=707727
We want an OstreeRepoFile to be the way to reference a "filesystem
tree" that's stored in the repo, which is a combination of a DIR_TREE
and a DIR_META. The idea is that once you write an mtree to the repo
using ostree_repo_write_mtree, it becomes serialized and you get an
OstreeRepoFile in return.
Change any APIs that care about DIR_TREE / DIR_META checksums to care
about OstreeRepoFiles instead, which right now is mostly is
ostree_repo_write_commit.
https://bugzilla.gnome.org/show_bug.cgi?id=707727
We want an OstreeRepoFile to be the way to represent a filesystem tree
inside an ostree repository. In order to do this, we need to drop the
commit from an OstreeRepoFile, and make that go to callers.
Switch all current users of ostree_repo_file_new_root to
ostree_repo_read_commit, and make the actual constructor private.
https://bugzilla.gnome.org/show_bug.cgi?id=707727
Previously I thought we'd have to ditch the current commit
format to avoid a{sv} due to
See https://bugzilla.gnome.org/show_bug.cgi?id=673012
But I realized that we don't really have to care about
unpacking/repacking commit objects, so let's just re-expose the
existing metadata a{sv} in commits in the API.
Also, add support for "detached" metadata that can be updated at any
time post-commit. This is specifically designed for GPG signatures.
https://bugzilla.gnome.org/show_bug.cgi?id=707379
...get_thread_default returns NULL when the thread default is also the global
default, so this only shows up when running in a thread (eg g_task_run_in_thread)
This is a relic from long ago when we were trying to stage objects
before finally committing them all in one go in the pull code.
We're no longer doing that, so stop trying to make the directory.
This also fixes trying to use ostree as non-root to read the
root-owned repo, since we'd fail to create the pending dir.
For the cases where we can't hardlink, use at-relative walking of the
path where possible. We still don't have lsetxattrat, so we also need
to deal with pathnames, but that is now only for symlinks.
Again, the advantages of this are a lot less malloc() of pathnames in
ostree, and much less time spent traversing paths inside the kernel.
https://bugzilla.gnome.org/show_bug.cgi?id=707733
Nothing external uses it. We keep ostree_get_xattrs_for_file() public
because it's convenient for external consumers to get xattrs in
exactly the format we desire.
https://bugzilla.gnome.org/show_bug.cgi?id=707733
Do as many operations as we can using the original file descriptor
while we have it open, rather than writing, closing, then reopening.
This necessitated very explicitly special casing symbolic links,
mainly due to the lack of lsetxattrat().
https://bugzilla.gnome.org/show_bug.cgi?id=707733
Clean up how we deal with the uncompressed object cache; we now use
openat()/linkat() and such just like we do for the main objects/.
Use linkat() between the objects and the destination, if possible.
https://bugzilla.gnome.org/show_bug.cgi?id=707733
Instead, use OstreeRepoFile as a handle for the parent commit.
We need to add an accessor for the metadata checksum, as that
hasn't been exposed before.
https://bugzilla.gnome.org/show_bug.cgi?id=707727
We can't quite do it for bare repositories yet because we need to have
a way to go from struct stat -> GFileInfo, and that's buried in gio's
private GLocalFile class.
Just use openat() for locating variants, rather than doing the lstat()
+ open(). This also drops several malloc+object allocations from the
lookup path.
Add new internal API to both fstatat() and write a pathname for the
given object. Use it in commit, and also wrapped in the old
GFile-based API.
This is more efficient.
Rather than having separate write_ref calls, make clients start a
transaction, add some refs, and then commit it. While this doesn't
make it 100% atomic, it makes it easier for us to use an atomic
model, and it means we don't do as much I/O updating the summary
file and such.
https://bugzilla.gnome.org/show_bug.cgi?id=707644
An earlier version of this API acted like git in that some objects
would be staged in a temporary directory which would be then committed
in one go by moving files around. The API doesn't match most users
expectations though, as while the stage is nice as a high-level API
it isn't really suited for low-level APIs.
While the stage was removed, the APIs were never renamed. Rename
them now so that they match expectations.
https://bugzilla.gnome.org/show_bug.cgi?id=707644
This avoids repeatedly traversing the target pathname, and is just
more efficient.
Part of a prelude to using fd-relative API for the source object path
too.
Update libgsystem submodule for a bugfix.
This is both more efficient from a kernel perspective, and avoids us
calling gs_file_get_path_cached() on tmp_dir constantly, which
triggered another bug due to lack of locking.
It's now isolated almost entirely to ostree-core.c, except
ostree-repo.c needs to know how to create archive-z2 file headers. So
give it a private API for that.
See the new comment in the source; basically if we're fetching content
over http, then someone with the capability to MITM the network could
create a transient setuid binary on disk with arbitrary content. If
they also had a process running on the system (such as an application)
it could be escalated to root.
https://bugzilla.gnome.org/show_bug.cgi?id=707139
There's not a good reason to write small things such as repo/config to
the filesystem, only to read them back in again. Change the
non-partial API to just return a stream, then read it into a memory
buffer.
https://bugzilla.gnome.org/show_bug.cgi?id=707157
I think originally we had the .part/.done separation because we were
trying to support partial downloads of files like repo/config and
repo/refs.
But now that the http server configuration won't give us partial
results, we don't need to support caching those files between runs.
And thus, there's no reason to have the .part/.done and do the dance
with renaming them.
When fetching objects/ and other things that use _with_async, we
continue to use _append_to(), and if the returned range tells us we
have all the bytes, then we hand the full file over to the caller.
Don't attempt to shortcut in the case where the last run told us we
already have the object; the object fetcher code will not make a
request.
While we're here, also clean up use of GError and consistently use the
cancellable from the pending.
https://bugzilla.gnome.org/show_bug.cgi?id=707157
Extracting the code for parse_ostree_cmdline() and running it on some
test input (on RHEL6.4 glibc), I can reproduce the odd behavior from
getline() where it apparently returns the size of the default malloc
buffer in the size output, and some non-zero value.
This behavior would be OK except that it breaks the logic for
stripping off the trailing newline, which in turn breaks booting
because we return "ostree=foo\n".
This has worked so far in gnome-ostree because syslinux apparently
injects initrd=/path/to/initrd as a final kernel argment.
Anyways, we don't handle NUL characters here in /proc/cmdline, so
let's just call strlen () to be safe.
https://bugzilla.gnome.org/show_bug.cgi?id=707192
Behave similar to git when 'ostree commit' is run without
a --subject or --body. Bring up an editor. The first line becomes
the subject and following lines become the --body after an optional
blank line.
Use similar logic to git in determining EDITOR
https://bugzilla.gnome.org/show_bug.cgi?id=707063
Fold in fetch_uri to fetch_uri_utf8(), and rename the latter to
include _sync as a suffix, since it's synchronous.
Improve the status line to show when we're fetching a synchronous URI;
previously we just showed "Scanning metadata".
https://bugzilla.gnome.org/show_bug.cgi?id=707023
Use a consistent temporary filename to download uri's.
Check for downloaded files before fetching from uri.
Download to hash.part file, then copy/move to hash.done when complete.
Add argument support to setup_fake_remote_repo1 function.
Add test for pull resume.
To implement this, pass --force-range-requests into the trivial-httpd,
which will only serve half of the objects to clients at a time.
https://bugzilla.gnome.org/show_bug.cgi?id=706344
Pull the cleanup code to a helper function, and ensure we delete
leftover temporary files also when aborting a transaction. Mainly
this will happen if a local 'ostree commit' fails.
While we're here, also change it to use gs_shutil_rm_rf() which also
handles directories, should we start using those.
Reviewed-by: Jeremy Whiting <jpwhiting@kde.org>
It turns out every builtin (with one special exception) that takes a
repo argument did the same thing; let's just centralize it. The
special exception was "ostree init --repo=foo" where foo is expected
to *not* actually be a repo. In that case, simply skip the
ostree_repo_check() invocation.
https://bugzilla.gnome.org/show_bug.cgi?id=706762
We removed support for writing "related objects" from ostree commits
in ostree git c9b61cbfee because it just
didn't work out as an idea. This also removes the API and code from
"ostree pull".
Note there was no test suite coverage.
https://bugzilla.gnome.org/show_bug.cgi?id=706342
OSTree now supports multiple bootloader backends so
notify which bootloader configuration was detected.
https://bugzilla.gnome.org/show_bug.cgi?id=706548
Signed-off-by: Javier Martinez Canillas <javier.martinez@collabora.co.uk>
Currently, when deploying an OSTree that does not contain a
bootloader configuration it fails with the following message:
"No known bootloader configuration detected"
A bootloader configuration is not strictly necessary if the
bootloader used is able to parse /boot/loader/entries on boot.
So, failing to deploy seems to be a little harsh. It is better
to just not write the bootloader configuration if a previous
one was not found but still swap the bootversion.
https://bugzilla.gnome.org/show_bug.cgi?id=706477
Signed-off-by: Javier Martinez Canillas <javier.martinez@collabora.co.uk>
Before, we were writing the "bootversion", which is either 0 or 1, for
all entries. This is completely wrong; the idea of the "version"
field is to compare between entries.
Fix this by writing out the inverted index - internally, index 0 is
the *first* boot entry, so we give it the highest version number, and
index N is the last, so give it version 0.
Then fix the deployment sorting code to correctly reverse the version
number comparison, so we read back the right order.
In practice before this bug didn't matter because "normally" you only
have at most two deployments.
https://bugzilla.gnome.org/show_bug.cgi?id=706546
The way we recurse into subdirectories in parallel makes it far too
easy to hit up against the arbitrary Linux fd limit of 1024.
Since the fix here is about dropping parallelism, let's just go all
the way for now and make a plain old synchronous API =(
This does simplify both internal callers which wanted a sync API
anyways.
https://bugzilla.gnome.org/show_bug.cgi?id=706380
This patch adds support to generate files that
can be used by Universal Bootloader (U-Boot).
U-Boot allows to modify boards default boot commands by
reading and executing a bootscript file or importing a
plain text file that contains environment variables that
could parameterize the boot command or a bootscript.
OSTree generates a uEnv.txt file that contains booting
information that is taken from Boot Loader Specification
snippets files as defined in the new OSTree deployment model:
https://wiki.gnome.org/OSTree/DeploymentModel2
On deploy or upgrade an uEnv.txt env var file is created
in the path /boot/loader.${bootversion}/uEnv.txt. Also, a
/boot/uEnv.txt symbolic link to loader/uEnv.txt is created
so U-Boot can always import the file from a fixed path.
Since U-Boot does not support a menu to list a set of
Operative Systems, the most recent bootloader configuration
from the list is used.
To boot an OSTree using the generated uEnv.txt file, a
board has to parameterize its default boot command using the
following variables defined by OSTree:
${kernel_image}: path to the Linux kernel image
${ramdisk_image}: path to the initial ramdisk image
${bootargs}: parameters passed to the kernel command line
Alternatively, for boards that don't support this scheme,
a bootscript that overrides the default boot command can be used.
An example of such a bootscript could be:
setenv scriptaddr 40008000
setenv kernel_addr 0x40007000
setenv ramdisk_addr 0x42000000
ext2load mmc 0:1 ${scriptaddr} uEnv.txt
env import -t ${scriptaddr} ${filesize}
ext2load mmc 0:1 ${kernel_addr} ${kernel_image}
ext2load mmc 0:1 ${ramdisk_addr} ${ramdisk_image}
bootm ${kernel_addr} ${ramdisk_addr}
Signed-off-by: Javier Martinez Canillas <javier.martinez@collabora.co.uk>
https://bugzilla.gnome.org/show_bug.cgi?id=706370
ot-bootloader-syslinux.c has a join_lines() function that is rather
generic and can be used in other places. Let's add it as a helper
function.
Signed-off-by: Javier Martinez Canillas <javier.martinez@collabora.co.uk>
https://bugzilla.gnome.org/show_bug.cgi?id=706370
We'll always have "bare" mode for keeping files-as-hardlinks as root.
But "archive" was my second attempt at a format for non-root file
storage, used by the gnome-ostree buildsystem which runs as non-root.
It was really handy to have a "tar" like mode where I can create
tarballs as a user, that contain files owned by root for example.
The "archive" mode stored content files as two pieces in the
filesystem; ".file" contained metadata, and ".filecontent" was the
actual content, uncompressed. The nice thing about this was that to
check out a tree as non-root, you could just hardlink into the repo.
However, archive was fairly bad for serving via HTTP; it required
*two* HTTP requests per content object, greatly magnifing the already
inefficient fetch process. So "archive-z2" was introduced.
To allow gnome-ostree to still check out trees as a user, the
"uncompressed-object-cache" was introduced, and that's how things have
been working for a while.
So we should just be able to kill this code. Specifically note just
how much better the stage_object() function became.
https://bugzilla.gnome.org/show_bug.cgi?id=706057
Debian uses /lib/systemd/system for system unit files, while i'm
putting ostree under the /usr prefix which means the hardcoded path
fails. Leave it to configure to work out the right location for systemd
units (method copied from pollkit).
Furthermore instead of installing the unit in local-fs.target.wants by
hand add a [Install] section so systemctl enable does the right thing
https://bugzilla.gnome.org/show_bug.cgi?id=705864
We have APIs to load metadata as variants, and files as parsed
content/info/xattrs, but for some cases such as static deltas, all we
want is to operate on all objects in their canonical representation.
https://bugzilla.gnome.org/show_bug.cgi?id=706031
We used to have a version of this, but since I'm trying to use
GBytes more, this became a more common operation, and it's annoying
to type out the whole G_VARIANT_TYPE ("ay") each time, and pass
TRUE for trusted.
https://bugzilla.gnome.org/show_bug.cgi?id=706031
This will be used to test resuming interrupted downloads for
ostree-pull.
With this option, if a whole file is asked for, only half of the file
is given. Then the client should retry with a range request, and
we'll give them the other half.
https://bugzilla.gnome.org/show_bug.cgi?id=705925
If we pass XATTR_REPLACE then the attribute must already exist, which
is not our intent. Passing zero creates the attribute if necessary,
or replaces it when it already exists.
https://bugzilla.gnome.org/show_bug.cgi?id=705893
* Specifying global options after the command for a more natural:
# ostree commit --repo=/path/to/repo ...
* Support asking for --help without --repo
# ostree commit --help
* Support short form of -h
* Support specifying --repo without equals sign
# ostree --repo /path/to/repo commit ...
* Support global --help and -h
# ostree --help
* Ditto for ostree admin sub commands
* Removed some leaky code
https://bugzilla.gnome.org/show_bug.cgi?id=705903
While the actual commit object format is presently the same, for a
number of reasons we'd like to change it fairly radically. Among
other things, we need to drop our a{sv} types in objects, to protect
against GVariant changing format.
Since now gnome-ostree now longer uses related objects, and nothing
ever used metadata, just drop them both.
If the admin encounters corruption and does:
$ ostree admin fsck --delete
We want them to be able to recover the objects easily from the
network; with this patch, they do:
$ ln -s dummyvalue /ostree/repo/transaction
$ ostree refs --delete remotename:branchname
$ ostree pull remotename
This patch avoids the need for the refs --delete; we might as well
force scan the commit, and with this patch we still print that it
changed.
We may revive this later, but commits in their current form aren't
very useful for humans to read, so it doesn't make sense to have a
tool to show a history of useless stuff.
More interesting things are diffs between commits, object statistics,
etc.
Otherwise it's really easy to keep accumulating deployments. Also, we
may want to run this after rebooting, so we're back down to one
operating system.
And disable devino scan by default. For the gnome-ostree build case,
our commits are from "make install DESTDIR=", so they won't be
hardlinks into the repo. In that case, particularly as the repository
size grows, the cost of building up the devino -> checksum mapping
becomes a problem.
The compose step will use this option though.
While the first was useful way back in the day when we were importing
Debian bits and /sbin/init was expecting to find /dev/.initctl as a
named pipe, that's no longer an issue with systemd since it uses
dynamic Unix sockets.
Likewise, character and block devices in /dev are now dynamically
created by the devtmpfs from the kernel.
Less complexity and code here if we just support directories, regular
files, and symbolic links.
Calling it "cleanup" is better since it does more than repo pruning.
We were also doing a prune twice; ot_admin_cleanup() already does one,
so drop the bits to do it in cleanup.c.
It isn't useful at the moment, since the deploy stuff all changed. It
will make sense to bring back later, but for now let's not carry
broken untested code.
While the systemd integration effectively requires /sysroot, it will
help people trying to use OSTree with other initramfs systems
(e.g. initramfs-tools) if we don't hardcode that requirement in this
tool.
If pull is interrupted, we may have downloaded an arbitrary subset of
the requested objects. Previously, we handled this by scanning for
all objects each time.
However, there's an easy optimization - this patch creates a lock file
in the repo. If we don't see that file when starting a pull, we know
we don't need to stat() every file; presence of a dirtree object for
example implies the existence of everything it references.
Originally, the idea was that clients would replicate "OS/tree"s from
a build server, but we'd run things like "ldconfig" on the client.
This was to allow adding e.g. the nVidia binary driver.
However, the triggers were the only thing in the system at the moment
that really had expected knowledge of the *contents* of the OS, like
the location of binaries.
For now, it's architecturally cleaner if we move the burden of
triggers to the tree builder (e.g. gnome-ostree or RPM). Eventually
we may want OSTree to assist with this type of thing (perhaps
something like RPM %ghost), but this is the right thing to do now.
See https://wiki.gnome.org/OSTree/DeploymentModel2
This is a major rework of the on-disk filesystem layout, and the boot
process. OSTree now explicitly supports upgrading kernels, and these
upgrades are also atomic.
The core concept of the new model is the "deployment list", which is
an ordered list of bootable operating system trees. The deployment
list is reflected in the bootloader configuration; which has a kernel
argument that tells the initramfs (dracut) which operating system root
to use.
Invidiual notable changes that come along with this:
1) Operating systems should now come with their etc in usr/etc; OSTree
will perform a 3-way merge at deployment time, and place etc in
the actual root. This avoids the need for a bind mount, and is
just a lot cleaner.
2) OSTree no longer bind mounts /root, /home, and /tmp. It is expected
that the the OS/ has these as symbolic links into /var.
At the moment, OSTree only supports managing syslinux; other
bootloader backends will follow.
This version of the code doesn't know what to do with new-model trees,
so just abort if we see that. We'll likely never care about upgrades
from 1.0 to 2.0.
A simple HTTP server implementation is so few lines of code when one
is linking to libsoup anyways, so let's just have one here in ostree
that will be used for the test suite.
This allows us to run the archive tests that previously required
apache even in gnome-ostree.
Even if very suboptimally, for now; we copy the files, then copy them
again.
The obvious long term plan is to merge pull-local and pull together,
but truly optimizing that requires the pull code to know how to use
the OstreeRepo APIs when operating on local repositories (as
pull-local does), rather than assuming the remote is an archive-z
fetched over HTTP.
We could drop into g_main_loop_run() after the worker
thread had called g_main_loop_quit().
Fix this by following the pattern suggested by Ryan of a while loop
around our termination condition, and g_main_context_iteration().
Linux creates a copy of the soure mount flags when creating a bind
mount; if the source is read-only, then the bind mount is.
The problem is that systemd will remount the rootfs read/write, but
each mount (/home, /var etc.) will still be read-only. We need to
remount every bind mount except for /usr to read-write too.
This only "worked" with the old ostree-switch-root because it
effectively force mounted the rootfs read-write always, ignoring the
"ro" flag.
This installs a Dracut module which parses the ostree= kernel command
line argument, and if given, sets up the OS/ at /sysroot, which
systemd's switch-root then moves into. This only works if dracut is
configured to use systemd itself.
Rather than attempting to hack up the "switch-root" functionality of
systemd, this binary allows us to simply prepare the root before we
switch into it.
This is useful for the gnome-ostree build system where each build is
one commit, but it's split up into /runtime /devel /debug etc. trees.
Ideally we wouldn't have a /debug subdirectory for "noarch"
components for example.
So add an option to not error out if the given path doesn't exist in
the commit.
When multiple threads need to uncompress an object, there was
a race condition where thread A could get EEXIST, unlink,
then thread B calls linkat(), then thread A tries to link() but
fails.
We can just loop in this case.
While this still isn't fully atomic (that depends on the bootloader),
this better ensures that the deployed kernel is booted with the
intended tree. For example, if we get ENOSPC when writing out the
kernel, we won't have swapped the symlink.
It's possible (likely even) that /tmp is on a separate filesystem; in
that case, a raw rename() is going to fail.
Saw this on the ostree.gnome.org builder.
Previously, I've observed bugs where we either:
1) Exit too early, leaving undownloaded objects
2) Hang while downloading
This rewrite hopefully fixes both.
We shouldn't overwrite deploy/<osname>/previous unless we actually
have something new.
This fixes the bug where two consecutive:
$ ostree admin upgrade foo
$ ostree admin upgrade foo
would end up pruning the data for foo/previous from the repo.
Colin Walters
Daniel Drake
Cosimo Cecchi
Roddy Shuler
Daniel Narvaez
Jeremy Whiting
Jasper St. Pierre
Sjoerd Simons
Vivek Dasmohapatra
Tobias Hunger
Stef Walter
Jeremy Whiting
Javier Martinez Canillas
Sjoerd Simons