prepare-root works with the mount that has been set up at /sysroot.
It creates a bind-mount within /sysroot (the deployment) and then moves
that mount to /sysroot.
Now we have 2 mounts both at /sysroot, and once we do switch_root, we will
never be able to unmount both of them. I'm not sure if this is ultimately
a kernel bug, but either way, ostree could do a bit more tidying up
after itself.
http://thread.gmane.org/gmane.linux.file-systems/92411
Easy way to reproduce:
1. Boot with rd.break param
2. At initramfs shell, run: ostree-prepare-root /sysroot
3. Observe two /sysroot mounts in /proc/mounts
Fix this by setting up the mounts at /sysroot.tmp, and unmounting the
original /sysroot before our new mount is MS_MOVEd on top of it.
Do not write directly to objects/ but maintain pulled files under tmp/
with a "tmpobject-$CHECKSUM.$OBJTYPE" name until they are syncfs'ed to
disk.
Move them under objects/ at ostree_repo_commit_transaction cleanup
time.
Before (test done on a local network):
$ LANG=C sudo time ./ostree --repo=repo pull origin master
0 metadata, 3 content objects fetched; 83820 KiB; 4 delta parts
fetched, transferred in 417 seconds
16.42user 6.73system 6:57.19elapsed 5%CPU (0avgtext+0avgdata
248428maxresident)k
24inputs+794472outputs (0major+233968minor)pagefaults 0swaps
After:
$ LANG=C sudo time ./ostree --repo=repo pull origin master
0 metadata, 3 content objects fetched; 83820 KiB; 4 delta parts
fetched, transferred in 9 seconds
14.70user 2.87system 0:09.99elapsed 175%CPU (0avgtext+0avgdata
256168maxresident)k
0inputs+794472outputs (0major+164333minor)pagefaults 0swaps
https://bugzilla.gnome.org/show_bug.cgi?id=728065
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
subscription-manager has a daemon that runs in a confined domain,
and it doesn't have permission to write usr_t, which is the default
label of /ostree/deploy/$osname/deploy.
A better long term fix is probably to move the origin file into the
deployment root as /etc/ostree/origin.conf or so.
In the meantime, let's ensure the .origin files are labeled as
configuration.
If an object already existed and we somehow tried to pull it, the
caller would still expect a returned checksum.
This appears to happen with static deltas for some reason; we might be
including duplicate metadata objects. Regardless, this is a bug that
should be fixed.
We have a chain of checksums from the root up until here. While doing
checksums of the objects individually would be a good redundancy check
for test cases and the like, when doing a pull there's no good reason
to burn cycles on SHA256.
This caused deadlocks and/or EMFILE due to the interaction between
threads and fds. What we really want here is a better pull-based
model for parsing content objects.
Another idea would be to change static deltas so that content objects
have a special opcode that includes their metadata first, and then do
rollsums etc. only over actual content.
This will avoid too many open files at the same time that could cause
an EMFILE error.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
(cherry picked from commit bc092b06f0e34e93f7d6102957bf55fd7ffd1b9e)
See projectatomic/rpm-ostree#42 for rationale. There are two high
level use cases:
- If the OS comes unconfigured, this is a way to point it at a repo of your choice.
- To switch between repositories while keeping the same branch easily.
At some point, we might want to expose a uniform way to refer
to deployments by an index. At the moment undeploy is the only
command that does.
I plan to introduce another command which optionally takes an index,
so prepare a helper function for this.
This is a noticeable cleanup, and fixes another big user of GFile* in
performance/security sensitive codepaths.
I'm specifically making this change because the static deltas code was
leaking temporary files, and cleaning that up nicely would be best if
we were fd relative.
This will be used for a later change to use openat() for the fetching
code. Note that we drop the code to use mmap() - it was an attempt to
avoid keeping a fd open, but we do correctly close anyways.
You create these with something like:
ostree static-delta generate --empty --to=master
These will be automatically used during pull if no previous revision
exists in the target repo.
These work very much like the normal static deltas except they
are named just by the "to" revision. I.e:
deltas/94/f7d2dc23759dd21f9bd01e6705a8fdf98f90cad3e0109ba3f6c091c1a3774d
for a from-scratch to 94f7d2dc23759dd21f9bd01e6705a8fdf98f90cad3e0109ba3f6c091c1a3774d delta.
https://bugzilla.gnome.org/show_bug.cgi?id=721799
The current layout uses a prefix of two bytes as the initial dir
and a second directory inside that with the superblock. This
updates the list code to handle that.
https://bugzilla.gnome.org/show_bug.cgi?id=721799
Regression from 86764dbf00
This function is kind of fiendish now that we have 3 cases, each of
which want to be optimized somewhat to only load what's necessary
(e.g. don't open the file if we don't have an output for stream
requested).
Clean things up so that BARE_USER and BARE are separate conditionals
that share as much as possible, and fix the bug that asserted we
were in BARE mode.
I tested this by running test-basic-user.sh by hand.
Some use cases for checkouts don't need to fsync during checkout.
Installer programs for example will just do a global fsync at the end.
In the future, the default "ostree admin" core could also be
rearchitected to only do a transaction commit right before reboot, and
do the fsync then.
https://bugzilla.gnome.org/show_bug.cgi?id=742482
This is just an efficiency optimization. We're getting fairly close
to all of the hot code paths using `*at()`.
Note that we end up maintaining a half-duplicate code path set here,
because we still need to support commits from an arbitrary GFile *,
which in a possible common case is an OSTree commit.
I think it's worth it though.
rpm-ostree had code to check for this, which didn't actually work.
I don't see a no backwards compatibility concern in changing this, as
it's unlikely a caller would try to sensibly disambiguate FAILED.
We were already using openat() for the contents, but not the xattrs.
Now that libgsystem 2014.3 has gs_fd_get_all_xattrs(), make use of it.
Clean things up a bit so we only open the fd once.
For Anaconda, I needed OSTREE_REPO_REMOTE_CHANGE_ADD_IF_NOT_EXISTS,
with the GFile *sysroot argument to avoid ugly hacks. We want to
write the content provided via "ostreesetup" as a remote to the target
chroot only in the case where it isn't provided as part of the tree
content itself.
This is also potentially useful in idempotent systems management tools
like Ansible.
https://bugzilla.gnome.org/show_bug.cgi?id=741577
Must have glossed over these because the commands are so simple.
- List subcommands for "ostree remote --help".
- Only show options relevant to COMMAND for
"ostree remote COMMAND --help".
ostree_repo_pull_with_options() needs this, and I'd rather keep the
OstreeRemote struct definition tucked away in ostree-repo.c with its
own internal API.
OstreeRemote is a reference-counted struct that encompasses data about a
remote, whether read from a configuration file or created explicitly via
ostree_repo_remote_add().
OstreeRemotes are held in an internal table indexed by remote name.
This solves some problems caused by merging system-wide remote data into
the OstreeRepo's internal config key file.
Also fixes https://bugzilla.gnome.org/show_bug.cgi?id=740911
This format is pretty much the same as the "bare" format, except the
file ownership and xattrs is not stored in the actual filesystem object, but
rather on the side in a user xattr. This means two things:
1) An unprivileged user can store such a repo independent of the types
of files in it or their xattrs. And you can later (as root)
reconstruct the real filesystem tree with ownership. Although you
can't do that using hardlink-sharing. This also means ostree
fsck does a full verification.
2) Such a repository can be checked out with user-mode (checkout -U)
as an unprivileged user using hardlinks for space sharing.
Additionally, symlinks are stored as regular files (with the content
being the symlink target) because user xattrs are not supported on
symlinks. We know at checkout time if the file is a symlink because
the original st_mode is stored in the xattr metadata.
https://bugzilla.gnome.org/show_bug.cgi?id=741125
Applying xattrs on a symlink during checkout failed since
it was setting the xattrs on the final filename, not the
temporary name.
This made the "checkout union 1" test in test-basic.sh
fail.
https://bugzilla.gnome.org/show_bug.cgi?id=741125
When commiting a symlink we do store the uid/gid of the actual
symlink (i.e. not target). However, this was not restored
on non-user-mode checkout as it should.
This commit fixes that, and additionally it ensures xattrs
on symlinks are not set in user-mode checkout.
https://bugzilla.gnome.org/show_bug.cgi?id=741125
rpm-ostree at least has the option to generate a tree with just that
instead of /boot, but while we were enumerating the latter, we'd still
return paths from /boot.
https://bugzilla.gnome.org/show_bug.cgi?id=740947
In Anaconda, we're using "ostree admin --sysroot=/mnt/sysimage
instutil set-kargs", and it was working before, but newer versions of
lorax strip out /etc/system-release which grub2 wants.
That was wrong anyways as we want the /etc/system-release from the
target root.
(Man, grub2 sucks...give me a declarative config file format I can just
write)
https://bugzilla.gnome.org/show_bug.cgi?id=740697
Refactor command-line parsing to better utilize GOptionContext. This
eliminates most of the manual parsing and global options are now shown
in the help output.
Here's a sample:
$ ostree admin --help
Usage:
ostree admin [OPTION...] --print-current-dir|COMMAND
Builtin "admin" Commands:
cleanup
config-diff
deploy
init-fs
instutil
os-init
status
switch
undeploy
upgrade
Help Options:
-h, --help Show help options
Application Options:
--sysroot=PATH Create a new OSTree sysroot at PATH
-v, --verbose Print debug information during command processing
--version Print version information and exit
https://bugzilla.gnome.org/show_bug.cgi?id=740295
Corrections, clarifications, consistency.
Not a comprehensive overhaul of the manpages; they do still need work.
These are just flaws I've noted while studying the ostree command syntax.
https://bugzilla.gnome.org/show_bug.cgi?id=740097
Make _ostree_fetcher_request_uri_with_partial_async and
ostree_fetcher_stream_uri_async simple wrapper around the same
function, all the requests are created in the same place now.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Rename _ostree_fetcher_contents_membuf_sync to
ostree_fetcher_request_uri_to_membuf and drop unused argument
user_data.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
_ostree_fetcher_query_state_text() and_ostree_fetcher_get_n_requests()
have no callers, so remove them.
If they will be needed, they can be easily copied back from the git
history.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Use the pattern:
$PRETTY_NAME [$COMMIT_VERSION] (ostree[:$OSNAME][:$DEPLOYMENT_INDEX])
$OSNAME is only shown if there are multiple values.
$COMMIT_VERSION refers to the version tag in the commit's metadata.
$DEPLOYMENT_INDEX is only shown if no $COMMIT_VERSION is available.
https://bugzilla.gnome.org/show_bug.cgi?id=739416
src/libostree/ostree-repo-pull.c:1676:22: warning: 'flags' may be used uninitialized in this function [-Wmaybe-uninitialized]
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
We potentially need a lot of argument types for pull. Rather than
have a C function with tons of arguments, let's use a GVariant a{sv}
as a handy extensible (and immutable) bag of properties.
This is prepratory work for adding an option to pull to traverse
history.
https://bugzilla.gnome.org/show_bug.cgi?id=737844
Daniel Drake
Giuseppe Scrivano
Colin Walters
Alexander Larsson
Matthew Barnes