This originally was a way that we detected the case where a pull was
interrupted. Later, we added `.commitpartial` files which also cover
this case.
See also https://github.com/GNOME/ostree/pull/85
We still want to honor their existence (and unlink them) in case an
old version of ostree was in use, but I believe it's safe to stop
creating them now.
The only case where this would break is if you have a version of
ostree that predates commitpartial in your rollback history, but such
old versions are no longer in use by operating systems I support at
least.
Closes: https://github.com/GNOME/ostree/pull/100
An OSTree user noticed that `ostree fsck` would produce `missing
object` errors in the case of interrupted pulls.
It's possible to do e.g. `ostree pull --subpath=/usr/share/rpm ...`,
which gets you just that portion of the commit. The use case for this
was being able to see what changes would appear in an update before
actually downloading all of it.
(I think this would be better covered by static deltas, but those
aren't final yet, and `--subpath` predates it)
Further, `.commitpartial` is used as a successor to the `transaction`
symlink for more precise knowledge in the case where a pull was
interrupted that we needed to resume scanning.
So it makes sense for `ostree fsck` to be aware of it.
First, this is just a general continuation of the `GFile -> openat`
transition.
Second, it's preparatory work for fsck to gain awareness of partial
commits.
If a system administrator happens to type `ostree admin upgrade`
multiple times, currently that will lead to a potentially corrupted
system.
I originally attempted to do locking *internally* in `libostree`, but
that didn't work out because currently a number of the commands
perform multi-step operations that all need to be serialized. All of
the current code in `ostree admin deploy` is an example.
Therefore, allow callers to perform locking, as most of the higher
level logic is presently implemented there.
At some point, we can revisit having internal locking, but it will be
difficult. A more likely approach would be similar to Java's approach
with concurrency on iterators - a "fail fast" method.
To make room for "remote gpg-import", which will be non-trivial.
ot-builtin-remote.c was already a little too crowded anyway.
Also while we're at it, port this bit of code away from libgsystem.
Always request detached metadata for commit objects, even if we already
have the commit object. This ensures we fetch any post facto detached
metadata updates such as new GPG signatures.
https://bugzilla.gnome.org/748220
These fsyncs were added for what turned out to be a fairly bogus
reason; I was hitting read errors from extlinux after upgrades and out
of conservatisim tried adding fsync calls, but the *actual* problem
was that extlinux didn't support 64 bit ext4. Now that at least for
Project Atomic hosts we're just targeting grub2, we can drop these
fsync calls and rely on `syncfs()` being both faster and catching any
errors.
For some sort of crazy reason, the `sync()` system call doesn't
actually return an error code, even though from what I can tell in the
kernel it wouldn't be terribly hard to add.
Regardless though, it is better for userspace apps to use `syncfs()`
to avoid flushing filesystems unrelated to what they want to sync. In
the case of OSTree, this does matter - for example you might have a
network mount point backing your database, and we don't want to block
upgrades on syncing it.
This change is safe because we're doing syncfs in *addition* to the
previous global `sync()` (a revision from an earlier patch).
Now because OSTree only touches the `/` mount point which covers the
repository, the deployment roots (including their copy of `/etc`), as
well as `/boot`, we should at some point later be able to drop the
`sync()` call. Note that on initial system installs we do relabel
`/var` but that shouldn't happen at ostree time - any new directories
are taken care of via `systemd-tmpfiles` on boot.
This way external programs like rpm-ostree can do fd-relative
operations on the deployment directories, like inspecting the RPM
database.
Closes: https://github.com/GNOME/ostree/pull/91
Rather than returning a new OstreeRepo instance in each call to
ostree_sysroot_get_repo(), cache one internally so the same instance
is returned each time.
Emitted during a pull operation upon GPG verification (if enabled).
Applications can connect to this signal to output the verification
results if desired.
First, git doesn't do this, and whatever Linus thinks is right or
something.
Second specifically to OSTree, it's quite common to not have
intermediate commits. If one wants to reset a ref in order to prune
data after a deployment, the parentage check will fail.
Closes: https://github.com/GNOME/ostree/pull/87
do not write directly to the summary file but use a temporary file
first. It avoids to create an empty file if "ot_util_variant_save"
fails.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
I was looking at https://bugzilla.gnome.org/show_bug.cgi?id=738954
which wants us to ensure we chown() the refs. As part of that,
I did a generic conversion to use `*at()` (which naturally gives
us more low level control so we can call `fchown` etc.
This patch also sneaks in a change to respect the repo's
`disable_fsync` flag - if fsync is not set, then we never
`fdatasync()` (unlike the `g_file_replace_contents()` default. Also
unlike it, if fsync is enabled, we *always* sync even if the file
didn't exist.
This will be used by rpm-ostree to unset the immutable bit temporarily
in order to do package layering. We could add an API to deploy a tree
without the immutable bit, but this is simpler.
rpm-ostree currently uses ostree_repo_checkout_tree(), which as a side
effect will use the uncompressed objects cache by default. This is
rather annoying if you're using rpm-ostree on a server-side
repository, because if you then rsync the repo, you'll be syncing out
the uncompressed objects unless you exclude them.
We added the ability to disable the uncompressed cache in the
repository config to fix this, but it's better to allow application
control over this. The uncompressed cache will in some future version
become opt in as well.
This new API further:
- Drops the `GFile` usage in favor of `openat` APIs
- Improves ergonomics by avoiding callers having to query the source
`GFileInfo` (and carry around a copy of `OSTREE_GIO_FAST_QUERYINFO`)
- Has a more extensible options structure
Per the comment, I rather crudely have the `ostree checkout` builtin
call both APIs to ensure some testing coverage.
However, I'd like to in the future have easier-to-set-up testing code
that calls `libtest.sh` to set up dummy data.
It's valid for the remote server to say 200 OK and give us the entire
file instead of a 206 Partial Content, and in that case we should blow
away the previous cached data, rather than blindly appending to it and
thus creating multiple copies of the data inside the file.
This problem primarily occurs when we do have the complete file, and
we're interrupted, then try again, where the new process didn't record
the download was already complete. We do a range request for bytes
past the end, and some web servers (e.g. Akamai) will return 200 OK
with the whole content again, rather than a 416 Requested Range Not
Satisfiable.
Thus we could also fix this by saner caching strategy - since we know
the file is complete, rename it again to $checksum.done or something
before it's processed. (Or really, rework how we do caching more
intelligently in general).
This fixes the issue that interrupted pulls failed with such
webservers, although repeated attempts would eventually succeed
because we'd unlink files that failed to pull.
Related: https://bugzilla.redhat.com/show_bug.cgi?id=1207292
The use case for non-default sysroots that I know of are:
1) The current test suite
2) Installers (Anaconda)
3) Inspecting VM disks
For 2) and 3), it'll quickly be obvious if they're not running as
root, and these are more obscure cases. We want to allow 1), and this
is a simple way to do it.
https://bugzilla.gnome.org/show_bug.cgi?id=747164
If the starting index is beyond the end of the list, it's a programming
error. Previously, the code was trying to raise a runtime error, but
actually causing a segfault.
This was detected by test code in test-mutable-tree.c, which is removed
in this commit because it should now not be possible to crash here.
https://bugzilla.gnome.org/747032
Indicates the command requires superuser privilege. Fails early with
a more helpful message than would otherwise be returned by libostree.
Currently all admin commands except 'status' require superuser.
Commands that need to write files within the repo directory can call
this early to ensure the directory is writable for the current user.
If not, it fails with a helpful "You need to be root to perform this
command" message.
Uses OstreeGpgVerifyResult to catch duplicate signatures.
If the commit has already been signed with the given GPG key ID, fail
with a G_IO_ERROR_EXISTS error code.
Wrappers a referenced gpgme_verify_result_t so detailed verify results
can be examined independently of executing a verify operation.
_ostree_gpg_verifier_check_signature() now returns this object instead
of a single valid/invalid boolean, but the idea is for OstreeRepo to also
return this object for commit signature verification so it can be utilized
at the CLI layer (and possibly by other programs).
The object count comes from g_hash_table_size(), so it's not a 0 based
index. In order to maintain the mod calculations correctly, just print
out index + 1.
https://bugzilla.gnome.org/show_bug.cgi?id=746360
Similar to c2b01ad. For some reason I was thinking the commit data
still needed to be written to disk prior to verifying, but it's just
another artifact of spawning gpgv2 (predates using GPGME).
Makes for a nice cleanup in fetch_metadata_to_verify_delta_superblock()
as well.
In anticipation of API enhancements for GPG signature verification, which
would otherwise require a non-functional stub version were GPGME excluded.
GPGME is a pretty lightweight dependency, and the motivation to exclude
it is not clear.
ostree-prepare-root was logging normal, informational messages
to stderr which the systemd unit points to the console.
To achieve silent boot, log these ordinary messages to stdout only.
The function returns a gboolean, replace g_return_if_fail with
g_return_val_if_fail.
Add similar checks to the other functions.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
The signature data is in memory to begin with, so there's no need to
write it to disk only to immediately read it back.
Also, because the GPGME multi-keyring workaround is somewhat expensive
to setup and teardown, concatenate all signatures into a single GBytes
so _ostree_gpg_verifier_check_signature() is only called once. We're
currently only looking for one valid signature anyway.
This sets the stage for more advanced signature management.
(Also, talking to GPG over pipes sucks.)
Previously we were spawning gpgv2 with a bunch of --keyring options
for /usr/share/ostree/trusted.gpg.d/ and whatever other keyring files
were explicitly added. GPGME has no public API for multiple keyrings,
so we work around the issue by setting up a temp directory to serve as
a fake "home" directory for the crypto engine and then concatenate all
the keyring files into a single public keyring (pubring.gpg).
Unfortunately at present we do this on every signature verification.
There's a desire to cache this concatenation, but the problem is the
user may be unprivileged. So it seems the cache would have to be per
user under $XDG_CACHE_HOME, which OSTree doesn't otherwise use. I'm
open to suggestions.
We do at least clean up the temp directory when finished, and I have
further API changes planned to OstreeGpgVerifier to help mitigate the
performance impact.
it will be used by the next patch that adds "--generate-static-delta"
to the commit command.
As part of the patch, update the list of supported "params" in the
documentation.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
g_variant_builder_add() does not replace identical keys in a VARDICT
variant, so signing a commit multiple times results in multiple copies
of "ostree.gpgsigs" in the metadata. And since g_variant_lookup_value()
stops on the first match, subsequent signatures have no net effect.
Instead of GVariantBuilder use GVariantDict, which behaves more like a
hash table.
Checkout was one of the first complex code paths I tried to convert to
*at(). I ended up keeping both, because I hit the "xattrs for a
symlink" problem. Later, Florian gave me a workaround, and we started
using it here, but the GFile * parameters weren't deleted. They're
not used, so do so now.
Starting down the path of not using libgsystem. The main win here
will be code sharing between ostree/rpm-ostree as well as going down
the path of not using GFile * for local files.
Convenience feature to avoid having to pass --repo options repeatedly.
Before falling back to the default system repository path, check for a
repository path defined by the OSTREE_REPO environment variable.
We already set all file mtimes to 0 so that they are constant
over all checkouts, and can be made constant with a known value from
the system where the ostree was created.
However, this was not happening for directories. Zero their mtimes too.
This is important for shipping a fontconfig cache in the ostree;
the fontconfig cache files embed a directory mtime.
The previous diff algorithm was file tree based, and only looked
at modified files that lived at the same path.
However, components like the Linux kernel have versioned
subdirectories, e.g. /usr/lib/modules/$kver/.../ext4.ko. We want to
be able to detect these "modified renames" so that we can compute
diffs (rollsum, bsdiff).
This does an rsync-style prepared delta basically. On my test data,
it shaves ~6MB of uncompressed data. Not a huge amount, but I expect
this to be more useful for things like binaries which embed data, etc.
There's still some silliness here, but there is now only one opcode
open-splice-and-close, that writes a single chunk from the payload.
This is really all we need for metadata, and small content objects are
also fine with this.
We get some deduplication between content objects by creating a
dictionary for (uid,gid,mode) tuples and xattrs.
This still keeps the operation/payload code in, so we could do
rollsums in a future update easily.
I was hitting a bug in libguestfs/guestmount/FUSE where it blew up
with EINVAL on directories containing lots of files (more than
32000?). We really want to use prefixed subdirs just like the real
objects/ directory does.
This allows us to share more code between the paths, is more
efficient, etc.
gnome-continuous uses the ostree_repo_scan_hardlinks() mode to
avoid re-checksumming everything. However, when I ported the commit
code to use openat() and friends, this optimization was lost.
Re add it. The difference is about 15s versus 5 minutes.
This follows up from the previous commit; now that pull knows how to
do the efficient link() or copy for local files, we can just have
pull-local call into ostree_repo_pull().
As part of this:
- pull() can also accept a file:/// URI instead
of a remote name (since pull local supports anonymous pulls)
- pull() knows an "override-remote-name" option, since pull-local
supported writing a ref out even if there wasn't a remote with
that name
It's always been suboptimal to have both pull and pull-local; as we go
beyond the raw object data into things like deltas and summary files,
the logic to perform e.g. mirroring should only be in one place.
This will be used by Pulp's OSTree content plugin at least to perform
promotions.
When doing a pull --mirror from an archive-z2 repository into another
archive-z2 repository, currently we gunzip/checksum/gzip each content
object. The re-gzip process in particular is fairly expensive.
This does assume that the upstream content is trusted and correct.
It'd be nice in the future to do at least a CRC check, if not the full
checksum. (Could we append CRC data to the end of filez objects?)
We could also choose to only do this optimization if fetching over
TLS.
before: 1626 metadata, 20320 content objects fetched; 299634 KiB transferred in 62 seconds
after : 1626 metadata, 20320 content objects fetched; 299634 KiB transferred in 11 seconds
For future delta work where we do more interesting things than just
"tar of new objects", this lays the groundwork for doing streaming
writes into content objects.
It's also more efficient, as we avoid many intermediate allocations
and virtual calls. Just a single `g_output_stream_write_all` for the
splice case.
Conflicts:
src/libostree/ostree-repo-private.h
src/libostree/ostree-repo-static-delta-processing.c
We could just make everything relative to this, but the objects/ and
tmp/ are accessed very often, so I think it's worth holding individual
fds.
This fd can cover everything else: refs, deltas, etc.
prepare-root works with the mount that has been set up at /sysroot.
It creates a bind-mount within /sysroot (the deployment) and then moves
that mount to /sysroot.
Now we have 2 mounts both at /sysroot, and once we do switch_root, we will
never be able to unmount both of them. I'm not sure if this is ultimately
a kernel bug, but either way, ostree could do a bit more tidying up
after itself.
http://thread.gmane.org/gmane.linux.file-systems/92411
Easy way to reproduce:
1. Boot with rd.break param
2. At initramfs shell, run: ostree-prepare-root /sysroot
3. Observe two /sysroot mounts in /proc/mounts
Fix this by setting up the mounts at /sysroot.tmp, and unmounting the
original /sysroot before our new mount is MS_MOVEd on top of it.
Do not write directly to objects/ but maintain pulled files under tmp/
with a "tmpobject-$CHECKSUM.$OBJTYPE" name until they are syncfs'ed to
disk.
Move them under objects/ at ostree_repo_commit_transaction cleanup
time.
Before (test done on a local network):
$ LANG=C sudo time ./ostree --repo=repo pull origin master
0 metadata, 3 content objects fetched; 83820 KiB; 4 delta parts
fetched, transferred in 417 seconds
16.42user 6.73system 6:57.19elapsed 5%CPU (0avgtext+0avgdata
248428maxresident)k
24inputs+794472outputs (0major+233968minor)pagefaults 0swaps
After:
$ LANG=C sudo time ./ostree --repo=repo pull origin master
0 metadata, 3 content objects fetched; 83820 KiB; 4 delta parts
fetched, transferred in 9 seconds
14.70user 2.87system 0:09.99elapsed 175%CPU (0avgtext+0avgdata
256168maxresident)k
0inputs+794472outputs (0major+164333minor)pagefaults 0swaps
https://bugzilla.gnome.org/show_bug.cgi?id=728065
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
subscription-manager has a daemon that runs in a confined domain,
and it doesn't have permission to write usr_t, which is the default
label of /ostree/deploy/$osname/deploy.
A better long term fix is probably to move the origin file into the
deployment root as /etc/ostree/origin.conf or so.
In the meantime, let's ensure the .origin files are labeled as
configuration.
If an object already existed and we somehow tried to pull it, the
caller would still expect a returned checksum.
This appears to happen with static deltas for some reason; we might be
including duplicate metadata objects. Regardless, this is a bug that
should be fixed.
We have a chain of checksums from the root up until here. While doing
checksums of the objects individually would be a good redundancy check
for test cases and the like, when doing a pull there's no good reason
to burn cycles on SHA256.
This caused deadlocks and/or EMFILE due to the interaction between
threads and fds. What we really want here is a better pull-based
model for parsing content objects.
Another idea would be to change static deltas so that content objects
have a special opcode that includes their metadata first, and then do
rollsums etc. only over actual content.
This will avoid too many open files at the same time that could cause
an EMFILE error.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
(cherry picked from commit bc092b06f0e34e93f7d6102957bf55fd7ffd1b9e)
See projectatomic/rpm-ostree#42 for rationale. There are two high
level use cases:
- If the OS comes unconfigured, this is a way to point it at a repo of your choice.
- To switch between repositories while keeping the same branch easily.
At some point, we might want to expose a uniform way to refer
to deployments by an index. At the moment undeploy is the only
command that does.
I plan to introduce another command which optionally takes an index,
so prepare a helper function for this.
This is a noticeable cleanup, and fixes another big user of GFile* in
performance/security sensitive codepaths.
I'm specifically making this change because the static deltas code was
leaking temporary files, and cleaning that up nicely would be best if
we were fd relative.
This will be used for a later change to use openat() for the fetching
code. Note that we drop the code to use mmap() - it was an attempt to
avoid keeping a fd open, but we do correctly close anyways.
You create these with something like:
ostree static-delta generate --empty --to=master
These will be automatically used during pull if no previous revision
exists in the target repo.
These work very much like the normal static deltas except they
are named just by the "to" revision. I.e:
deltas/94/f7d2dc23759dd21f9bd01e6705a8fdf98f90cad3e0109ba3f6c091c1a3774d
for a from-scratch to 94f7d2dc23759dd21f9bd01e6705a8fdf98f90cad3e0109ba3f6c091c1a3774d delta.
https://bugzilla.gnome.org/show_bug.cgi?id=721799
The current layout uses a prefix of two bytes as the initial dir
and a second directory inside that with the superblock. This
updates the list code to handle that.
https://bugzilla.gnome.org/show_bug.cgi?id=721799
Regression from 86764dbf00
This function is kind of fiendish now that we have 3 cases, each of
which want to be optimized somewhat to only load what's necessary
(e.g. don't open the file if we don't have an output for stream
requested).
Clean things up so that BARE_USER and BARE are separate conditionals
that share as much as possible, and fix the bug that asserted we
were in BARE mode.
I tested this by running test-basic-user.sh by hand.
Some use cases for checkouts don't need to fsync during checkout.
Installer programs for example will just do a global fsync at the end.
In the future, the default "ostree admin" core could also be
rearchitected to only do a transaction commit right before reboot, and
do the fsync then.
https://bugzilla.gnome.org/show_bug.cgi?id=742482
This is just an efficiency optimization. We're getting fairly close
to all of the hot code paths using `*at()`.
Note that we end up maintaining a half-duplicate code path set here,
because we still need to support commits from an arbitrary GFile *,
which in a possible common case is an OSTree commit.
I think it's worth it though.
rpm-ostree had code to check for this, which didn't actually work.
I don't see a no backwards compatibility concern in changing this, as
it's unlikely a caller would try to sensibly disambiguate FAILED.
We were already using openat() for the contents, but not the xattrs.
Now that libgsystem 2014.3 has gs_fd_get_all_xattrs(), make use of it.
Clean things up a bit so we only open the fd once.
For Anaconda, I needed OSTREE_REPO_REMOTE_CHANGE_ADD_IF_NOT_EXISTS,
with the GFile *sysroot argument to avoid ugly hacks. We want to
write the content provided via "ostreesetup" as a remote to the target
chroot only in the case where it isn't provided as part of the tree
content itself.
This is also potentially useful in idempotent systems management tools
like Ansible.
https://bugzilla.gnome.org/show_bug.cgi?id=741577
Must have glossed over these because the commands are so simple.
- List subcommands for "ostree remote --help".
- Only show options relevant to COMMAND for
"ostree remote COMMAND --help".
ostree_repo_pull_with_options() needs this, and I'd rather keep the
OstreeRemote struct definition tucked away in ostree-repo.c with its
own internal API.
OstreeRemote is a reference-counted struct that encompasses data about a
remote, whether read from a configuration file or created explicitly via
ostree_repo_remote_add().
OstreeRemotes are held in an internal table indexed by remote name.
This solves some problems caused by merging system-wide remote data into
the OstreeRepo's internal config key file.
Also fixes https://bugzilla.gnome.org/show_bug.cgi?id=740911
This format is pretty much the same as the "bare" format, except the
file ownership and xattrs is not stored in the actual filesystem object, but
rather on the side in a user xattr. This means two things:
1) An unprivileged user can store such a repo independent of the types
of files in it or their xattrs. And you can later (as root)
reconstruct the real filesystem tree with ownership. Although you
can't do that using hardlink-sharing. This also means ostree
fsck does a full verification.
2) Such a repository can be checked out with user-mode (checkout -U)
as an unprivileged user using hardlinks for space sharing.
Additionally, symlinks are stored as regular files (with the content
being the symlink target) because user xattrs are not supported on
symlinks. We know at checkout time if the file is a symlink because
the original st_mode is stored in the xattr metadata.
https://bugzilla.gnome.org/show_bug.cgi?id=741125
Applying xattrs on a symlink during checkout failed since
it was setting the xattrs on the final filename, not the
temporary name.
This made the "checkout union 1" test in test-basic.sh
fail.
https://bugzilla.gnome.org/show_bug.cgi?id=741125
When commiting a symlink we do store the uid/gid of the actual
symlink (i.e. not target). However, this was not restored
on non-user-mode checkout as it should.
This commit fixes that, and additionally it ensures xattrs
on symlinks are not set in user-mode checkout.
https://bugzilla.gnome.org/show_bug.cgi?id=741125
rpm-ostree at least has the option to generate a tree with just that
instead of /boot, but while we were enumerating the latter, we'd still
return paths from /boot.
https://bugzilla.gnome.org/show_bug.cgi?id=740947
In Anaconda, we're using "ostree admin --sysroot=/mnt/sysimage
instutil set-kargs", and it was working before, but newer versions of
lorax strip out /etc/system-release which grub2 wants.
That was wrong anyways as we want the /etc/system-release from the
target root.
(Man, grub2 sucks...give me a declarative config file format I can just
write)
https://bugzilla.gnome.org/show_bug.cgi?id=740697
Refactor command-line parsing to better utilize GOptionContext. This
eliminates most of the manual parsing and global options are now shown
in the help output.
Here's a sample:
$ ostree admin --help
Usage:
ostree admin [OPTION...] --print-current-dir|COMMAND
Builtin "admin" Commands:
cleanup
config-diff
deploy
init-fs
instutil
os-init
status
switch
undeploy
upgrade
Help Options:
-h, --help Show help options
Application Options:
--sysroot=PATH Create a new OSTree sysroot at PATH
-v, --verbose Print debug information during command processing
--version Print version information and exit
https://bugzilla.gnome.org/show_bug.cgi?id=740295
Corrections, clarifications, consistency.
Not a comprehensive overhaul of the manpages; they do still need work.
These are just flaws I've noted while studying the ostree command syntax.
https://bugzilla.gnome.org/show_bug.cgi?id=740097
Make _ostree_fetcher_request_uri_with_partial_async and
ostree_fetcher_stream_uri_async simple wrapper around the same
function, all the requests are created in the same place now.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Rename _ostree_fetcher_contents_membuf_sync to
ostree_fetcher_request_uri_to_membuf and drop unused argument
user_data.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
_ostree_fetcher_query_state_text() and_ostree_fetcher_get_n_requests()
have no callers, so remove them.
If they will be needed, they can be easily copied back from the git
history.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Use the pattern:
$PRETTY_NAME [$COMMIT_VERSION] (ostree[:$OSNAME][:$DEPLOYMENT_INDEX])
$OSNAME is only shown if there are multiple values.
$COMMIT_VERSION refers to the version tag in the commit's metadata.
$DEPLOYMENT_INDEX is only shown if no $COMMIT_VERSION is available.
https://bugzilla.gnome.org/show_bug.cgi?id=739416
src/libostree/ostree-repo-pull.c:1676:22: warning: 'flags' may be used uninitialized in this function [-Wmaybe-uninitialized]
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
We potentially need a lot of argument types for pull. Rather than
have a C function with tons of arguments, let's use a GVariant a{sv}
as a handy extensible (and immutable) bag of properties.
This is prepratory work for adding an option to pull to traverse
history.
https://bugzilla.gnome.org/show_bug.cgi?id=737844
fixes a coredump when using a command like:
$ ostree --repo=repo checkout -U --subpath=/usr/lib/passwd \
fedora-atomic/rawhide/x86_64/docker-host usrlib-new
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
We need basic support for UEFI - many newer servers don't support
BIOS compatibility mode anymore.
However, this patch only implements non-atomic because UEFI is FAT, and
we can't do the previous design for OSTree of atomic swap of
/boot/loader.
The Fedora/RHEL UEFI layout has the kernels on a "real" /boot
partition, and /boot/efi/EFI/$vendor just holds the grub2 UEFI binary
and grub.cfg.
Following this, /boot/loader is still on the OS boot partition, and we
still atomically swap it. This potentially paves the way to atomic
upgrades in the future.
https://bugzilla.gnome.org/show_bug.cgi?id=724246
Some package systems need to be run as root, so the process linking to
libostree may also be root. However, it's reasonable to have the
target repository be owned by a uid other than root.
This patch makes it Just Work by chowning the file content to match.
Note this only operates on archive-z2 repositories, because you can't
usefully serve bare repositories via HTTP.
https://bugzilla.gnome.org/show_bug.cgi?id=738954
For Anaconda, we have an ugly bootstrapping problem where we need to
add the remote to the repository's config, then do a pull+deploy, then
remove and re-add the config, because /etc/ostree/remotes.d doesn't
exist yet in the target system.
https://bugzilla.gnome.org/show_bug.cgi?id=738698
While we did support disabling the uncompressed-objects-cache
per-repository:
1) We didn't actually respect that operation when doing
CHECKOUT_MODE_USER on archive-z2 repositories
2) It'd be better to automatically detect we can't write to the
repo and disable the uncompressed cache then.
In this approach, we drop a /etc/grub.d/15_ostree file which is a
hybrid of shell/C that picks up bits from the GRUB2 library (e.g. the
block device script generation), and then calls into libostree's
GRUB2 code which knows about the BLS entries.
This is admittedly ugly. There exists another approach for GRUB2 to
learn the BLS specification. However, the spec has a few issues:
https://www.redhat.com/archives/anaconda-devel-list/2014-July/msg00002.html
This approach also gives a bit more control to the admin via the
naming of the 15_ostree symlink; they can easily disable it:
Or reorder the ostree entries ahead of 10_linux:
Also, this approach doesn't require patches for grub2, which is an
issue with the pressure to backport (rpm-)OSTree to EL7.
src/libostree/ostree-repo.c:1759: Warning: OSTree:
ostree_repo_import_object_from: unknown parameter 'checksum' in
documentation comment, should be 'sha256'
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Some operating systems may come with external tools for subscription
management that drive access to the content. In that case, the origin
file may not be useful (for example, it could refer to an installer
ISO).
This patch will allow OS installers to inject that state, with a
useful error message, directing the system administrator to an
external tool.
See: https://github.com/projectatomic/rpm-ostree/issues/31https://bugzilla.gnome.org/show_bug.cgi?id=737686
Now that we have a summary file, we can use it to allow a simple:
ostree pull --mirror
To download the latest commit on every branch. Also, for a case I'm
dealing with there's only one branch, but I don't want mirror users to
have to hardcode it.
https://bugzilla.gnome.org/show_bug.cgi?id=737807
And use it in pull-local. As one might expect, this is blazingly fast
if they're on the same filesystem.
I'll be using this to "promote" builds between different repositories.
Add command line arguments:
--import-proc-cmdline: import values from /proc/cmdline
--merge: import current values
--replace=ARG=VALUE: replace value
--append=ARG=VALUE: append a new argument
Extra command line arguments are treated like --append=, which
gives backwards compatibility.
https://bugzilla.gnome.org/show_bug.cgi?id=731051
The standard convention is that the output of --help should go to standard
output (so that it can be piped to a pager and searched.) See, e.g., the
GNU coding standards.
https://bugzilla.gnome.org/show_bug.cgi?id=737194
'ostree admin' and 'ostree admin instuil' with no arguments were meant to fail,
but the logic was wrong; add an assertion on the return value from all ostree
commands to catch similar problems in the future.
https://bugzilla.gnome.org/show_bug.cgi?id=737194
Previously, in the case where a parent directory of a modified config
file was removed, we would throw an exception. This happens when
switching from a tree that has some software (e.g. firewalld), to one
that does not.
While it's nice to have this warning that your config file probably no
longer applies, there's no need to make it so...fatal.
It's particularly problematic that the only easy workaround is to
remove the config files from your current tree - which breaks
rollback.
The solution then is for for us to take ownership of the parent
directories too into the new /etc. Admins can clean up these files
afterwards at any time.
https://bugzilla.gnome.org/show_bug.cgi?id=734293
This fixes a regression introduced with https://git.gnome.org/browse/ostree/commit/?id=7baa600e237b326899de2899a9bc54a6b863943c
The original code in "ostree admin upgrade" had a comment:
/* Here we perform cleanup of any leftover data from previous
* partial failures. This avoids having to call gs_shutil_rm_rf()
* at random points throughout the process. */
But since I deleted that initial cleanup call, we *do* need to do the
cleanup during the process run. It turns out there are only a few
places this is necessary.
https://bugzilla.gnome.org/show_bug.cgi?id=733030
While looking to fix a different bug here, I found the current
state of things where we had a mix of fd-relative API versus not
frustrating.
Change the code around to consistently use *at, and also add some more
tests.
For Fedora and potentially other distributions which use globally
distributed mirrors, metalink is a popular solution to redirect
clients to a dynamic set of mirrors.
In order to make metalink work though, it needs *one* file which can
be checksummed. (Well, potentially we could explode all refs into the
metalink.xml, but that would be a lot more invasive, and a bit weird
as we'd end up checksumming the checksum file).
This commit adds a new command:
$ ostree summary -u
To regenerate the summary file. Can only be run by one process at a
time.
After that's done, the metalink can be generated based on it, and the
client fetch code will parse and load it.
https://bugzilla.gnome.org/show_bug.cgi?id=729585
Otherwise, we're potentially holding up subsequent requests.
I was hitting this when testing the metalink code, where we want to
continue doing more fetches after hitting a 404.
https://bugzilla.gnome.org/show_bug.cgi?id=729585
Changes the pull API to allow pulling only a single directory instead
of the whole deployment. This option is utilized by the check-diff
option in rpm-ostree.
Add a new state directory to hold <checksum>.commitpartial files, so
we know that we've only downloaded partial state.
If we deployed a new tree inside an existing OS, inheriting kernel
args, we need to use append() instead of replace() to avoid collapsing
multiply specified kernel arguments like console=/dev/foo
console=/dev/bar.
Reported-by: Dusty Mabe <dusty@dustymabe.com>
We want to allow multiply specified arguments, as it's useful for
things like console= and rd.lvm.lv.
See: https://bugzilla.gnome.org/show_bug.cgi?id=721136
Reported-by: Dusty Mabe <dustymabe@gmail.com>
I noticed OSTree was a bit slower, did some investigation
and saw we were enumerating all objects for things like
$ ostree rev-parse blah
Since "blah" can never be an object (because of the 'l' and 'h'), just
return no matches.
The user might "ostree ls /usr/bin/bash/blah", which previously would
segfault.
A somewhat related future enhancement here would be for "ostree ls" to
follow symbolic links.
Reported-by: Dusty Mabe <dustymabe@gmail.com>
https://bugzilla.gnome.org/show_bug.cgi?id=733476
Prune has worked fine on bare repositories for some time, but now that
I finally try to delete data on the server side, I notice we weren't
actually enumerating content objects =/
That caused them to not be pruned.
https://bugzilla.gnome.org/show_bug.cgi?id=733458
The prune API duplicated logic to delete objects, and furthermore the
core API to delete an object didn't clean up detached metadata.
Fix the duplication by doing the obvious thing: prune should call
_delete.
https://bugzilla.gnome.org/show_bug.cgi?id=733452
This patch adds a function that will parse a partial checksum when
resolving a refspec. If the inputted refspec matches a truncated
existing checksum, it will return that checksum to be parsed. If
multiple truncated checksums match the partial refspec, it is not
unique and will return false. This addition is inspired by the same
functionality in Docker, which allows a user to reference a specific
commit without typing the entire checksum.
partial checksums: Add function to abstract comparison
This modifies the list_objects and list_objects_at functions
to take an additional argument for the string that a commit starts
with. If this string arg is not null, it will only list commit
objects beginning with that string. This allows for a new function
ostree_repo_list_commit_objects_starting_with to pass a partial string
and return a list of all matching commits. This improves on the
previous strategy of listing refs because it will list all commit objects,
even ones in past history. This update also includes bugfixes on
error handling and string comparison, and changes the output structure
of resolve_partial_checksum. The new strcuture will no longer return FALSE
without error. Also, the hashtable foreach now uses iter. Also
includes modified test file
/var needs to be read-write for a functioning system. Various
systemd services will fail to start if /var is read-only. After we
remount /var (or if we skip remounting /var because / is read-only),
if /var is still readonly, mount a tmpfs file system on /var.
While this isn't strictly part of ostree, keeping it here makes sense
because it keeps twiddling around with the /var mount in one place
for easier coordination and maintenance. This will likely need updating
if systemd gains better support for a readonly root filesystem.
https://bugzilla.gnome.org/show_bug.cgi?id=732979
Some organizations will want to use private Certificate Authorities to
serve content to their clients. While it's possible to add the CA
to the system-wide CA store, that has two drawbacks:
1) Compromise of that cert means it can be used for other web traffic
2) All of ca-certificates is trusted
This patch allows a much stronger scenario where *only* the CAs in
tls-ca-path are used for verification from the given repository.
https://bugzilla.gnome.org/show_bug.cgi?id=726256
Using docker as a model, this update creates separate manpages
for each OSTree command, such that the main manpage is not
cluttered and the separate pages can provide more in-depth detail
and exanples that might be useful to a user. Each page includes
synopsis, description, example, and a list of options if needd.
This update also alphabetizes the usage error output for ostree
and ostree admin so that it matches the list on the manpage.
We were using unsigned size when we should have been using signed,
this means we basically weren't checking for errors on write...ouch.
Luckily if we e.g. hit ENOSPC during a pull, the checksums wouldn't
match and we'd return an error anyways. However when writing an
object, we'd end up silently ignoring it =/
https://bugzilla.gnome.org/show_bug.cgi?id=732020
The generic GKeyFile error isn't quite informative enough here.
I hit this with the new compose process where we don't automatically
inject a configured remote into the generated disk images; we expect
people to add them.
https://bugzilla.gnome.org/show_bug.cgi?id=731346
There's several use cases for calling into ostree itself to do
mirroring, instead of using bare rsync. For example, it's a bit more
efficient as it doesn't require syncing the objects/ directory.
https://bugzilla.gnome.org/show_bug.cgi?id=728351
For the local repository on the system, it's not the usual case to
have the complete compose history. Rather than erroring out, provide
a bit more friendly message.
https://bugzilla.gnome.org/show_bug.cgi?id=731538
In the case of running ostree as non-root on a regular filesystem (not
tmpfs which doesn't support immutable), we should just silently do
nothing if we encounter EPERM. Cache the result to avoid spam in
strace.
https://bugzilla.gnome.org/show_bug.cgi?id=728006
We weren't installing the headers, but at the moment all symbols
starting with ostree_ were being exported. Fix that by prefixing
non-static symbols with '_'.
https://bugzilla.gnome.org/show_bug.cgi?id=731369
Finally, fsync to ensure all entries are on disk, unless disabled.
We support disabling this for cases like server-side buildroot
construction where we don't need to be robust against power loss
The previous S_IMMUTABLE commit broke ostree-remount; / is now not
actually writable. All we really wanted to know though was whether it
was *mounted* writable, so check that via statvfs() which is cleaner
anyways (i.e. not via access() which kernel people hate).
https://bugzilla.gnome.org/show_bug.cgi?id=728006
On some storage configurations, fsync() can be extremely expensive.
Developers and users with slow hard drives may want the ability to opt
for speed over safety.
Furthermore, many production servers have UPS and stable kernels, and
the risk of not fsync'ing in that scenario is fairly low. These users
should also be able to opt out.
This prevents people from creating new directories there and expecting
them to be persisted. The OSTree model has all local state to be in
/etc and /var.
This introduces a compile-time dependency on libe2fsprogs.
We're only doing this for the root directory at the moment.
https://bugzilla.gnome.org/show_bug.cgi?id=728006
If fetching GPG-signed commits over plain HTTP, a MitM attacker can
fill up the drive of targets by simply returning an enormous stream
for the commit object.
Related to this, an attacker can also cause OSTree to perform large
memory allocations by returning enormous GVariants in the metadata.
This helps close that attack by limiting all metadata objects to 10
MiB, so the initial fetch will be truncated.
But now the attack is only slightly more difficult as the attacker
will have to return a correctly formed commit object, then return a
large stream of < 10 MiB dirmeta/dirtree objects.
https://bugzilla.gnome.org/show_bug.cgi?id=725921
The current "transaction" symlink was introduced to fix issues with
interrupted pulls; normally we assume that if we have a metadata
object, we also have all objects to which it refers.
There used to be a "summary" which had all the available refs, but I
deleted it because it wasn't really used, and was still racy despite
the transaction bits.
We still want the pull process to use the transaction link, so don't
delete the APIs, just relax the restriction on object writing, and
introduce a new ostree_repo_set_ref_immediate().
They shouldn't be loaded for random test/personal repositories. Doing
so triggers another bug in that we return them from
ostree_repo_get_config() when then causes clients to write them out
permanently to disk with ostree_repo_write_config(). This caused test
suite failures.
This makes it easy to use for the case where rpm-ostree-toolbox is
injecting systemd services into the deployment root, and we don't
actually need to traverse the whole FS.
For many OS install scenarios, one runs through an installer which may
come with embedded data, and then the OS is configured post-install to
receive updates.
In this model, it'd be nice to avoid the post-install having to rewrite
the /ostree/repo/config file.
Additionally, it feels weird for admins to interact with "/ostree" -
let's make the system feel more like Unix and have our important
configuration in /etc.
https://bugzilla.gnome.org/show_bug.cgi?id=729343
The option --port-file=- is most useful when the stdout of the daemon
is programatically redirected and not going to a terminal. The
flush-after-a-line behavior of stdout is specific to terminals, so
we need an explicit flush.
https://bugzilla.gnome.org/show_bug.cgi?id=729609
Matthew Barnes
Colin Walters
Giuseppe Scrivano
Micah Abbott
Daniel Svensson
Sam Thursfield
Alexander Larsson
Dan Nicholson
Daniel Drake
James Antill
Owen W. Taylor
Anne LoVerso
Vadim Rutkovsky