So far a lot of submitted PR have added symbols into the first
section. Split the file into `-released` and `-devel` to make
this more obvious.
To further enforce things, we hardcode a checksum of the `-released`
file in `test-symbols.sh`. Only release commits should update that
checksum.
Did you notice I like checksums?
Closes: #931
Approved by: pwithnall
For the flatpak PR: https://github.com/flatpak/flatpak/pull/849
It's really more convenient if this works for HTTP pulls as well, since flatpak
does various types of pulling, and we can just set the flag everywhere.
Further, we might as well reject the content as early as possible.
Closes: #930
Approved by: alexlarsson
We saw this fail in a CI run. We've been trying to add strategic error prefixing
as a general rule, and this specific instance may help debug.
Closes: #929
Approved by: pwithnall
This is a continuation of https://github.com/ostreedev/ostree/pull/926
for directories instead of files.
See: https://github.com/flatpak/flatpak/issues/845
This option suppresses mode bits outside of `0775` for directory
checkouts. I think most people should start doing this by default,
and use explicit overrides for e.g. `/tmp` if doing a recommit based
on a checkout.
Closes: #927
Approved by: alexlarsson
This is an option which is intended mostly for flatpak;
see: https://github.com/flatpak/flatpak/issues/845
We're adding an option for pulling into *all*
repo modes that has an effect similar to the `bare-user-only`
change from https://github.com/ostreedev/ostree/pull/909
This way one can pull content into e.g. a root-owned `bare` repository and
ensure that there aren't any setuid or world-writable files.
Closes: #926
Approved by: alexlarsson
Thinking about the problem of flatpak converting from `bare-user` to `bare-user-only`
"in place" by creating a new repo and doing a `pull-local`, I realized
that we can optimize this process by doing hardlinks for both metadata
and regular files. The repo formats are *almost* compatible, the
exception being symlinks.
An earlier patch caused us to do hardlinks for metadata, this patch takes things
to the next step and special cases this specific conversion. In this case we
need to parse the source object to determine whether or not it's a symlink.
Closes: #922
Approved by: alexlarsson
Our previous logic for import-via-hardlink only tried if the repo modes match,
but we *can* hardlink metadata between e.g. `archive` and `bare-user` repos, and
that's quite useful thing to do. Our documentation encourages converting to/from
those repo modes locally for build systems.
Closes: #922
Approved by: alexlarsson
Before this, if one had repos of matching mode but different owners,
which could happen if one e.g. makes a `bare` non-root repo in
`/ostree/deploy/$stateroot/var/tmp`, every time we tried to call `linkat()`
we'd get `EPERM` and fall back to a copy.
Fix this by saving the repo owner uid, and avoid trying to call `linkat()` if we
know it's going to fail. Of course most commonly in this scenario we'll
immediately fail trying to `chown` the files to `0`, but this is prep for a
future patch to improve `bare-user` → `bare-user-only` imports where we'll be a
bit more sophisticated.
Closes: #922
Approved by: alexlarsson
This came up in: https://github.com/ostreedev/ostree/pull/881
Basically doing streaming for metadata is dumb. Split up the metadata/content
paths so we pass metadata around as `GVariant`. This drops the last internal
caller of `ostree_repo_write_metadata_stream_trusted()` which was the dumb
function mentioned.
Closes: #923
Approved by: jlebon
See https://github.com/ostreedev/ostree/pull/909 for more information on the
rationale. Basically there's no reason for flatpak (which uses `bare-user-only`)
to have world-writable dirs. Particularly with the presence of the system
helper.
An approach I considered instead was to parse and validate directory metadata
objects at commit time. We still may do that in addition; for file objects we *had*
to do it that way because the actual files would be laid down suid. But directories
live only as inert `.dirmeta` objects until we do a checkout (i.e. `mkdir()`), so
we can solve the problem at checkout time.
Closes: #914
Approved by: alexlarsson
Both callers of `commit_loose_object_trusted()` were passing
`OSTREE_OBJECT_TYPE_FILE`, so drop that parameter. This in turn
allows us to drop lots of checking of that inside the function.
Add a doc comment, and rename to `commit_loose_content_object()` for clarity.
Closes: #914
Approved by: alexlarsson
I noticed my previous patches incorrectly started doing `return glnx_throw*`
inside a `goto out;` function. Fix this by porting forward consistently to new
style. We just do the error prefixing in the caller.
Closes: #914
Approved by: alexlarsson
This copies the `ci/` directory from rpm-ostree, with much the same rationale;
among other things we don't want to depend on the Docker hub.
The specific reason I'm doing this is that I want to add a CentOS7 build, but
that means we can't use `projectatomic/ostree-tester`, and at that point we
might as well unwind it all.
Closes: #917
Approved by: jlebon
These exist in the wild for flatpak, and aren't really a problem. The canonical
permissions are still either `0755` or `0644`, we just support the additional
writable bit for the group (i.e. extend the set to include `0775` and `0664`)
now to avoid breaking some flatpak content.
Closes: #913
Approved by: alexlarsson
We didn't have coverage of this before, and adding the test infrastructure will
help ensure we have coverage for more changes here.
Closes: #913
Approved by: alexlarsson
Having every object in a bare-user repo (and checkouts) be executable
is ugly. I can't think of a good reason to do that; they should only
be executable if their input is. This does
for `bare-user` what we did for `bare-user-only` in
https://github.com/ostreedev/ostree/pull/909
It's also a stronger version of what we do with `checkout -U` in suppressing
suid - here we also strip world-writable files and the sticky bit (even though
that's meaningless today, it might not be in the future).
Closes: https://github.com/ostreedev/ostree/issues/907Closes: #908
Approved by: alexlarsson
For the flatpak use case where bare-user-only was introduced, we actually
don't want to support s{u,g} id files in particular.
Actually, I can't think of a reason to have anything outside of the
`0755 i.e. (u=rwx,g=rx,o=rx)` mask, so that's what we do here.
This will have the effect of treating existing `bare-user-only` repositories as
corrupted if they have files outside of that mask, but I think we should do this
now; most of the flatpak users will still be on `bare-user`, and we haven't
changed the semantics of that mode yet.
Note that in this patch we will also *reject* file content that doesn't
match this. This is somewhat asymmetric, since we aren't similarly rejecting
e.g. directory metadata. But, this will close off the biggest source
of the problem for flatpak (setuid binaries).
See: https://github.com/ostreedev/ostree/pull/908
See: https://github.com/flatpak/flatpak/pull/837Closes: #909
Approved by: alexlarsson
This reverts commit 1eff3e8343. There
are a few issues with it. It's not a critical thing for now, so
let's ugly up the git history and revisit when we have time to
debug it and add more tests.
Besides the below issue, I noticed that the simple `ostree remote add`
now writes to `/ostree/repo/config` because we *aren't* using the
`--sysroot` argument.
Closes: https://github.com/ostreedev/ostree/issues/901Closes: #902
Approved by: mike-nguyen
There was a lot of conditionals inside `write_object()` differentating
between metadata/content, and then for content, on the different repo
types. Further, in the metadata path since the logic is simpler, can
present a non-streaming API, and further use `OtTmpfile`, etc.
Splitting them up helps drop a lot of conditionals. We introduce a small
`CleanupUnlinkat` that allows us to fully convert to the new code style in both
functions.
This itself is still prep for fully switching to `GLnxTmpfile`.
Closes: #881
Approved by: jlebon
If we have an expected checksum, call `fstatat(repo_dfd, checksum)`
early on before we do much else. This actually duplicates code,
but future work here is going to split up the metadata/content
commit paths, so they'll need to diverge anyways.
Closes: #881
Approved by: jlebon
First, the streaming metadata API is pretty dumb, since metadata
should be small. Really we should have supported a `GBytes`
version. Currently, this API *is* used when we do local pulls,
so this commit has test coverage. However, I plan to change
the object import to avoid using this. But that's fine, since
I can't think of why someone would use this API.
Next, the only difference between `ostree_repo_write_metadata()` and
`ostree_repo_write_metadata_trusted()` is whether or not we pass
an output checksum; so just dedup the implementations.
Also while I'm here break out the input length validation and do
it early in the streaming case.
Closes: #881
Approved by: jlebon
Using `${sysroot}` to mean the physical storage root: We don't want to write to
`${sysroot}/etc/ostree/remotes.d`, since nothing will read it, and really
`${sysroot}` should just have `/ostree` (ideally). Today the Anaconda rpmostree
code ends up writing there. Fix this by adding a notion of "physical" sysroot.
We determine whether the path is physical by checking for `/sysroot`, which
exists in deployment roots (and there shouldn't be a `${sysroot}/sysroot`).
In order to unit test this, I added a `--sysroot` argument to `remote add`.
However, doing this better would require reworking the command line parsing for
the `remote` argument to support specifying `--repo` or `--sysroot`, and I
didn't quite want to do that yet in this patch.
Closes: https://github.com/ostreedev/ostree/issues/892Closes: #896
Approved by: jlebon
I saw a few instances of `glnx_set_error_from_errno() + return FALSE`,
and fixed them and did a bit of style conversion.
Closes: #895
Approved by: jlebon
Having a failable accessor is annoying, since it's really common
to reference both. Instead, open the repo once when we load
the sysroot, and provide a non-failable accessor.
This is also prep for `ostree_repo_open_at()`, which collapses the separation
between `ostree_repo_new()` and `ostree_repo_open()`.
Closes: #886
Approved by: jlebon
This is prep for introducing a fd-relative `ostree_repo_new_at()`.
Previously, `ostree_repo_is_system()` compared `GFile` paths, but
there's a much simpler check we can do first - if this repository
was created via `OstreeSysroot`, it must be a system repo.
Closes: #886
Approved by: jlebon
This is a de-scoping of work I did in preparation for
rpm-ostree [live updates](https://github.com/projectatomic/rpm-ostree/pull/652).
Originally I was going to expose this as a public API.
However, I decided to do things differently, but the cleanup here for new code
style and fd-relative is nice to have anyways.
We rework things to use `OstreeDeployment*`, which the caller is expected to
already have, rather than `GFile*`s pointing to the config directories.
Closes: #741
Approved by: jlebon
It's hard right now to do a full port to the new libglnx tmpfile
API since there are complex cases in the commit path which deal
with symlinks as well.
Let's make things more gradual by introducing the important part (struct with
autocleanup) here in libotutil, port what we can. This will make a future
complete port easier.
Closes: #871
Approved by: jlebon
I want to keep this a "pure copy-able" file into various projects
like rpm-ostree, bwrap, and flatpak. Pull in changes from rpm-ostree
to prep for that.
While we have the patient open, dedup the code for file matching a bit.
Closes: #877
Approved by: jlebon
We're starting to get a lot of contexts, and this is likely going to drive some
requirements for consolidation and improvements like not testing *every* context
on every PR, etc.
But for now since experimental is new, and under development, let's test it.
Closes: #875
Approved by: cgwalters
Porting a lot of this file would be hard since in many cases we do processing in
the `out:` section, so let's do what we can.
Closes: #870
Approved by: jlebon
Unbreaks mounting in CentOS. Newer systemd in Fedora pulls didn't need this, I
think due to `RequiresMountsFor=`. Anyways, this is what the fstab generator
does, and it's clearly right ✓.
Closes: https://github.com/ostreedev/ostree/issues/867Closes: #869
Approved by: jlebon
Since the version in CentOS is too old, and we get a spam of warnings, plus
things like detecting the git repo break.
Fixes: 50f73cbac3Closes: #868
Approved by: jlebon
Follow up to a previous patch that addressed a double-close; I
realized we already had a helper for doing "open dfd iter, do nothing
if we get ENOENT". Raise it to libotuil, and port all consumers.
Closes: #863
Approved by: jlebon
I noticed an instance of this while working on https://github.com/ostreedev/ostree/pull/861
Which apparently I cargo-culted into the new system generator bits.
Let's break this out as a small concise change.
Closes: #866
Approved by: jlebon
If one wants to set up a mount for `/var` in `/etc/fstab`, it
won't be mounted since `ostree-prepare-root` set up a bind mount for
`/var` to `/sysroot/ostree/$stateroot/var`, and systemd will take
the already extant mount over what's in `/etc/fstab`.
There are a few options to fix this, but what I settled on is parsing
`/etc/fstab` in a generator (exactly like `systemd-fstab-generator` does),
except here we look for an explicit mount for `/var`, and if one *isn't* found,
synthesize the default ostree mount to the stateroot. Another nice property is
that if an admin creates a `var.mount` unit in `/etc` for example, that will
also override our mount.
Note that today ostree doesn't hard depend on systemd, so this behavior only
kicks in if we're built with systemd *and* libmount support (for parsing
`/etc/fstab`). I didn't really test that case though.
Initially I started writing this as a "pure libc" program, but at one point
decided to use `libostree.so` to find the booted deployment. That didn't work
out because `/boot` wasn't necessarily mounted and hence we couldn't find the
bootloader config. A leftover artifact from this is that the generator code
calls into libostree via the "cmd private" infrastructure. But it's an easy way
to share code, and doesn't hurt.
Closes: #859
Approved by: jlebon
By checking the mount status, we avoid remounting things if we don't
need to. And printing a single line per mount helps debugging when
things go wrong.
Closes: #859
Approved by: jlebon
I really have no idea what I was thinking with that list of mount points. It
seems arbitrary. Sadly `git log` doesn't help, and there's no comments.
Basically, the only mounts we should care about are those that libostree
creates. Which are just `/sysroot` and `/var`. Systemd will handle the other
things like `/tmp`, it's not our job, and we shouldn't touch them.
Closes: #859
Approved by: jlebon
In current systemd, there is:
[systemd-volatile-root](https://www.freedesktop.org/software/systemd/man/systemd-volatile-root.service.html)
which was introduced by [this commit](https://github.com/systemd/systemd/commit/91214a37ef4eb8042d2598aa89bae52b410d11a7).
I'd like to make further changes to how we handle `/var`, and I don't
want to reason about the interaction of our "tmpfs var" with too many
other things.
The comment about having "all /var handling in one place" was always inaccurate
given that we rely on systemd for mounting. And in general, I don't want to
duplicate too many things systemd does - it does them well, documents them, etc.
As far as I know, it was basically just Owen who was using this for the GNOME
hardware testing effort, and I'm sure he could easily switch over to
`systemd.volatile=state`.
Closes: #856
Approved by: owtaylor
This is what caused the merge of
https://github.com/projectatomic/rpm-ostree/pull/652
to blow up, since https://github.com/ostreedev/ostree/pull/848
landed right before we tried to merge it.
When I was writing that PR I remember having an uncertain feeling
since we were doing a `mkdirat` above, but at the time I thought
we'd have test suite coverage...turns out we didn't.
For backwards compatibility, we need to continue to do a `mkdirat` here of the
parent. However...I can't think of a reason anyone would *want* that behavior.
Hence, let's add a special trick - if the destination name is `.`, we skip
`mkdirat()`. That way rpm-ostree for example can open a dfd for `/etc` and avoid
the `mkdir`.
Fold the subpath tests into `test-basic.sh` since it's not worth a separate
file. Add a test case for checking out a file.
Closes: #854
Approved by: jlebon
As $DEITY intended.
I was reading the `prepare-root.c` code and the indentation damage was
distracting. Squash tabs that have leaked into various places in the code. I
didn't yet touch the `src/libostree` bits as that has higher potential for
conflict.
Closes: #852
Approved by: jlebon
Looking at `perf record ostree checkout`, some things stand out; e.g.:
```
+ 27.63% 0.07% ostree libgio-2.0.so.0.5000.3 [.] g_file_enumerator_iterate
+ 22.74% 0.28% ostree libostree-1.so.1.0.0 [.] ostree_repo_file_tree_query_child
+ 13.74% 0.08% ostree libostree-1.so.1.0.0 [.] ot_variant_bsearch_str
```
The GIO abstractions are already fairly heavyweight, and `OstreeRepoFile` mallocs
a lot too.
Make things more efficient here by dropping the GIO bits for reading ostree data -
we just read from the variants directly and iterate over them. The end result
here is that according to perf we go from ~40% of our time in the kernel to
~70%, and things like `g_file_enumerator_iterate()` drop entirely out of the
hot set.
Closes: #848
Approved by: jlebon
Since we now have a cleaner separation of "toplevel checkout prep"
versus "recursive checkout", handle the special case of checking out
a single file at first rather than later.
Prep for future work in optimizing this function more.
Closes: #848
Approved by: jlebon
Only non-mechanical bit here was creating a local autoptr for a bit
where we'd previously done an unref for a struct member.
Closes: #847
Approved by: jlebon
Rather than `g_output_stream_splice()`, where the input is a regular
file.
See https://github.com/GNOME/libglnx/pull/44 for some more information.
I didn't try to measure the performance difference, but seeing the
read()/write() to/from userspace mixed in with the pointless `poll()` annoyed me
when reading strace.
As a bonus, we will again start using reflinks (if available) for `/etc`,
which is a regression from the https://github.com/ostreedev/ostree/pull/797
changes (which before used `glnx_file_copy_at()`).
Also, for the first time we'll use reflinks when doing commits from file-backed
content. This happens in `rpm-ostree compose tree` today for example.
Update submodule: libglnx
Closes: #817
Approved by: jlebon
Continuing to chip away at this. Using `g_file_enumerator_iterate()`
here helps notably.
I started on the much bigger `ostree_diff_dirs_with_options()` but
it's a lot messier - for later.
Closes: #844
Approved by: jlebon
I think tests/ should be just that, ci/ is separate. Also rename
the files to include "travis" since that's what we use them
for right now.
Closes: #843
Approved by: jlebon
We need our `make install` to override the ostree RPM, so do it all in one txn.
This sort of thing is where a more rigorous model like rdgo/gcontinuous use
becomes better, but we'll hack it with shell for now.
Closes: #824
Approved by: jlebon
These at the moment aren't in a container, and may need space. In the future
overlay2 will help here, we can more easily extend the rootfs.
Closes: #840
Approved by: jlebon
`test-pull-many.sh` is was just too slow to be a unit test. Generating
a bunch of files via shell is slow, the delta generation is slow, etc.
Every developer doesn't need to run it every time.
Somewhat address this by converting it into our installed test framework, which
moves it out of the developer fast paths. Another advantage to this is
that we can simply reuse the FAH tree content rather than synthesizing
new bits each time.
Closes: #840
Approved by: jlebon
The major reason to do this is that running tests *both* installed
and uninstalled in our CI is a mostly pointless waste of time.
Particularly given we have a few expensive tests.
We *do* have tests that only run uninstalled (since they require
the source code) like `test-symbols.sh`.
Hence, add `--enable-installed-tests=exclusive` to mean *only* do installed for
most tests.
We'll still have uninstalled coverage via the Travis/Debian configs, and we
could perhaps do another build with a subset of uninstalled tests, but I'm not
really concerned about it.
I'd like to do a renewed push for the InstalledTests model since
I feel it's just fundamentally better. (`g-d-t-r` kind of sucks,
but then so does the automake runner).
Also while we're here - fix the CI to use the correct context,
which started this mess.
Closes: #837
Approved by: dbnicholson
If we're freeing the segment, it's basically always better to use
`autoptr()`. Fewer lines, more reliable, etc.
Noticed an instance of this in the pull code while reviewing a different PR,
decided to do a grep for it and fix it tree wide.
Closes: #836
Approved by: pwithnall
The keyring isn't large, so let's just fall back to copying it
rather than requiring `renameat()`.
Prep for `ostree_repo_open_at()`.
Closes: #821
Approved by: jlebon
This was part of the philosophy behind https://wiki.gnome.org/Initiatives/GnomeGoals/InstalledTests -
libraries like libostree don't need to replicate everything in unit tests, we
can use the tests from our dependencies directly too.
We'll also get API break coverage testing too.
Closes: #818
Approved by: jlebon
I happened to glance at the top of my most recent patch and
noticed that I used an `throw_errno()` function in a non-errno place.
I scanned the patch for other instances of this but didn't find one.
Closes: #811
Approved by: jlebon
I was planning to change some of the object loading code in the
future, so here's some porting.
Note that I rewrote `_ostree_repo_has_loose_object()` since it
used an error return across multiple functions.
Honestly I'm not sure about this `TEMP_FAILURE_RETRY()` business...
in reality we're going to end up with a ton of code linked in
process that doesn't do it. Unix sucks =( But I'm keeping
what was there out of consistency.
Closes: #809
Approved by: jlebon
This did a `closedir` in the `goto out` section before, but it
turns out more nicely if we follow the usual pattern of doing
the `open(O_DIRECTORY)` in the callee function and handle `ENOENT`
there.
Closes: #809
Approved by: jlebon
This is a variant of the efforts in https://github.com/ostreedev/ostree/pull/741
Working on `rpm-ostree livefs`, I realized though I needed to just
check out *new* files directly into the live `/etc` (and possibly
delete obsolete files).
The way the current `/etc` merge works is fundamentally different from
that. So my plan currently is to probably do something like:
- Compute diff
- Check out each *new* file individually (as a copy)
- Optionally delete obsolete files
Also, a few other things become more important - in the current deploy code, we
copy all of the files, then relabel them. But we shouldn't expose to *live*
systems the race conditions of doing that, plus we should only relabel files we
checked out.
By converting the deploy's /etc code to use this, we fix the same TODO item
there around atomically having the label set up as we create files. And further,
if we kill the `/var` relabeling which I think is unnecessary since Anaconda
does it, we could delete large chunks of code there.
In the implementation, there are two types of things: regular files, and
symlinks. For regular files, in the `O_TMPFILE` case, we have the ability to
do *everything* atomically (including SELinux labeling) before linking it into
place. So let's just use that. For symlinks, we use `setfscreatecon()`.
Closes: #797
Approved by: jlebon
Our container-driven tests can't e.g. test SELinux sanely, and
have to support being run as root *and* non-root too.
Use redhat-ci to provision a VM and run tests directly there. These are
installed tests too.
Closes: https://github.com/ostreedev/ostree/issues/806Closes: #807
Approved by: jlebon
This could be shared more easily with e.g. rpm-ostree, but what I'm currently
working on is installed, privileged (potentially destructive, i.e. VM) tests
that will source this separately from the current `libtest.sh`. That does work
installed, but in practice is oriented around unit (uninstalled, unprivileged)
tests.
Closes: #807
Approved by: jlebon
We really have an astonishing variety of similar functions which write files and
symlinks. I was working on a different PR and the duplication between the
union-mode and add-mode/none-mode checkout functions bothered me.
I realized that the "handle EEXIST" tri-state maps directly to the
`GLnxLinkTmpfileReplaceMode`, so deduping things makes even more sense.
Closes: #801
Approved by: jlebon
I was reading a strace the other day and noticed we were loading the same
`.dirmeta` object many times. Unlike the other object types, `.dirmeta` objects
don't accumulate much over time; there are only so many directory metadata types.
(Without SELinux involved it'd probably be 5-6 I'd guess offhand).
For `fedora-atomic/25/x86_64/docker-host` there are currently 34 `.dirmeta` in
the tree.
But how many times during a checkout did we load those 34 dirmeta objects?
With a quick strace:
```
$ strace -s 2048 -f -o strace.log ostree --repo=repo-build checkout -U fedora-atomic/25/x86_64/docker-host host-test-checkout
$ grep dirmeta strace.log | wc -l
7165
```
After, as you'd expect, we just loaded `34` from disk. We do
6 system calls (`openat+fstat+fstat+read+read+close`) per dirmeta,
so we dropped a total of 42780 system calls - which is about 20% of the total
system calls made.
`perf record` tells me that we're spending ~40 of our time in the kernel during
a checkout, so reducing syscall traffic helps. Though most of that appears to be
in the VFS and XFS layers for `linkat` (which isn't surprising).
So how much did perf improve? Well, on my workstation, I get a lot of
fluctuation in timing, sometimes by 30%, so this was well within the noise. But
it's well worth speeding up checkout, and I think this optimization will shine
more as we improve performance elsewhere.
Closes: #795
Approved by: jlebon
These are leftovers from the packfile code and should have been
deleted in commit: 2a0601efc7
I noticed this now since I wanted to add a new type of caching.
Closes: #795
Approved by: jlebon
I never really liked the term "osname". I feel "stateroot" is a *lot* clearer,
since the osname/stateroot mostly just holds `/var`. Further it avoids the `os`
prefix which is already overloaded.
Some of the existing docs already talked about "operating system state", which
further reinforces this.
There's *lot* more things than this which reference the term "osname", but I
don't want to change *everything* yet in this patch in case we decide to do
something different - this just gets the highlights.
Closes: #794
Approved by: jlebon
This is intended to be used for copying `/usr/etc` → `/etc` for
deployments.
A TODO here is to use `glnx_file_copy_at()` if the repo mode allows
it - then we'd use reflinks if available.
Closes: #804
Approved by: jlebon
Colin Walters