I'd like to move the new canonical kernel directory to `/usr/lib/modules/$kver`,
as Fedora has done. The `get_kernel_from_tree()` function now abstracts over
parsing the data (src vs destination filenames, as well as checksum) in
preparation for adding the new case.
In preparation for this, let's change the current test suite to use the
*current* directory of `/usr/lib/ostree-boot`, and also add coverage of `/boot`.
Closes: #1053
Approved by: jlebon
See: http://marc.info/?l=linux-fsdevel&m=149520244919284&w=2
XFS doesn't flush the journal on `syncfs()`. GRUB doesn't know how to follow the
XFS journal, so if the filesystem is in a dirty state (possible with xfs
`/boot`, extremely likely with `/`, if the journaled data includes content for
`/boot`, the system may be unbootable if a system crash occurs.
Fix this by doing a `FIFREEZE`+`FITHAW` cycle. Now, most people
probably would have replaced the `syncfs()` invocation with those two
ioctls. But this would have become (I believe) the *only* place in
libostree where we weren't safe against interruption. The failure
mode would be ugly; nothing else would be able to write to the filesystem
until manual intervention.
The real fix here I think is to land an atomic `FIFREEZETHAW` ioctl
in the kernel. I might try a patch.
In the meantime though, let's jump through some hoops and set up
a "watchdog" child process that acts as a fallback unfreezer.
Closes: https://github.com/ostreedev/ostree/issues/876Closes: #1049
Approved by: jlebon
The intended behaviour of ostree_repo_find_remotes() is to return
results which have the latest version of at least one of the requested
refs. Results which have some of the requested refs, but don’t have the
latest version of any of them, should be ignored. The logic to do this
was broken in the case that a result contained a positive number of the
requested refs, but none of them were the latest version. (It previously
worked when the result contained none of the requested refs.)
Fix the counting to work correctly in both cases.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1058
Approved by: cgwalters
Coverity spotted an infloop here since we were incrementing `i++`
instead of `j++`. But adding a test revealed other bugs - we need
to keep the arrays in sync.
Coverity CID: 1452204
Closes: #1041
Approved by: pwithnall
There are multiple use cases where we'd like to alias refs.
First, having a "stable" alias which gets swapped across major
versions: https://pagure.io/atomic-wg/issue/228
Another case is when a ref is obsoleted;
<https://pagure.io/atomic-wg/issue/303>
This second one could be done with endoflife rebase, but I think
this case is better on the server side, as we might later change
our minds and do actual releases there.
I initially just added some test cases for symlinks in the `refs/heads` dir to
ensure this actually works (and it did), but I think it's worth having APIs.
Closes: #1033
Approved by: jlebon
Define typedefs for read/write archives, and use the GLib
autocleanups for them. Prep for updating libglnx to drop its
custom autocleanup macros.
Closes: #1042
Approved by: jlebon
Coverity spotted that we had an off-by-one here since we were using
`i+1`. Fix this by adding a `-1` to the bounds check. Also use
`sizeof()` to ensure the data and size are coupled.
Coverity CID: 1452207
Closes: #1037
Approved by: jlebon
It's designed for test suites and non-critical random uses like this. This
silences a Coverity warning about weak randomness.
Closes: #1037
Approved by: jlebon
No real problems here, but Coverity likes to see consistent checking of return
values, and I agree with it.
Coverity CID: 1452213
Coverity CID: 1452211
Closes: #1037
Approved by: jlebon
Previous to this commit, the gjs tests were installed-only; and our
logic for handling the "--enable-installed-tests=exclusive" logic
actually also meant they weren't installed.
It did work for me locally with `--enable-installed-tests`.
However, to make things fully symmetric, let's enable the js tests to also be
run under `make check`.
Also remove `corrupt-repo-ref.js` from the PAPR invocation since it's not
actually a unit test, it's a utility helper.
Closes: #1022
Approved by: jlebon
See: https://github.com/projectatomic/rpm-ostree/issues/885
If we get a successful Apache directory listing HTML when fetching what we
intend to be a ref, we'd dump the HTML into the error.
I did some scanning of the pull code, and this was the only case
I saw offhand where we were dumping text out into an error. Which
makes sense, since most of our formats are binary, the exeptions I
think are just `repo/config` and `repo/refs/`.
Closes: #1015
Approved by: mbarnes
(Note this PR was reverted in <https://github.com/ostreedev/ostree/pull/902>;
this version should be better)
Using `${sysroot}` to mean the physical storage root: We don't want to write to
`${sysroot}/etc/ostree/remotes.d`, since nothing will read it, and really
`${sysroot}` should just have `/ostree` (ideally). Today the Anaconda rpmostree
code ends up writing there. Fix this by adding a notion of "physical" sysroot.
We determine whether the path is physical by checking for `/sysroot`, which
exists in deployment roots (and there shouldn't be a `${sysroot}/sysroot`).
In order to unit test this, I added a `--sysroot` argument to `remote add`.
However, doing this better would require reworking the command line parsing for
the `remote` argument to support specifying `--repo` or `--sysroot`, and I
didn't quite want to do that yet in this patch.
This second iteration of this patch fixes the bug we hit the first time;
embarassingly enough I broke `ostree remote list` finding system remotes.
The fix is to have `ostree_repo_open()` figure out whether it's the same
as `/ostree/repo` for now.
Down the line...we might consider having the `ostree remote` command line itself
instatiate an `OstreeSysroot` by default, but this maximizes compatibility; we
just have to pay a small cost that `ostree` usage outside of that case like
`ostree static-delta` in a releng Jenkins job or whatever will do this `stat()`
too.
Closes: https://github.com/ostreedev/ostree/issues/892Closes: #1008
Approved by: mbarnes
We have variants of `test-basic` for all 4 modes, but not for pull-test, which
for some reason was named `pull-archive`, but mostly pulls *into* bare repos.
The test code was structured like the basic one where it called into a
`pull-test.sh`, so let's actually use it for 2/3 bare modes. (I tried to extend
it to `bare-user-only` but it failed, going to look at that after this).
This is related to https://github.com/ostreedev/ostree/issues/991Closes: #998
Approved by: jlebon
When we [switched to using checkout + force_copy](e8efd1c8dc),
a side effect that went unnoticed at the time is that we started
setting directory mtimes to zero.
See the below bug where we long ago set the file times to zero, which got fixed,
so let's not regress things by setting the directory times to zero either. (Even
though AFAICS GNU tar doesn't complain about those)
This semantic is somewhat "overloaded" onto `force_copy`, but it avoids adding
yet another boolean; we don't have that many reserved boolean slots left. I
can't really think of many good use cases for `force_copy` *other* than the
`/etc` merge anyways.
https://bugzilla.redhat.com/show_bug.cgi?id=1229160
Closes: https://github.com/ostreedev/ostree/issues/995Closes: #997
Approved by: jlebon
The collection and ref bindings are stored in the commit metadata
under ostree.collection-binding and ostree.ref-binding,
respectively. They will be used to verify if the commit really comes
from the collection and ref we wanted to pull from.
Closes: #972
Approved by: cgwalters
And in general, if for some reason we can't write `user.` xattrs, provide an
error immediately rather than doing it during a later pull. This way the failure
cause is a lot more obvious.
Related: https://github.com/ostreedev/ostree/issues/991Closes: #993
Approved by: jlebon
In the storage PR I was trying to do a `pull-local` of the whole
`/ostree/repo` on the system, which ended up triggering a `g_critical()`
in the collections code, since we tried to parse a remote-prefixed ref
`fedora:fedora/26/x86_64/atomic-host` as a ref.
I'm not sure offhand what our behavior in this case *should* be. I
think git only clones local refs, but I need to check.
This corner case arises only with `pull-local`. But in any case,
while we were previously saying this is programmer error, since it's
so easy to pass various unchecked input into the pull machinery,
make invalid refs an explicit error.
Closes: #992
Approved by: jlebon
For ostree-as-host, we're the superuser, so we'll blow past
any reserved free space by default. While deltas have size
metadata, if one happens to do a loose fetch, we can fill
up the disk.
Another case is flatpak: the system helper has similar concerns
here as ostree-as-host, and for `flatpak --user`, we also
want to be nice and avoid filling up the user's quota.
Closes: https://github.com/ostreedev/ostree/issues/962Closes: #987
Approved by: jlebon
Some of the Jenkins jobs for Fedora Atomic Host broke after updating
to 2017.7, and it turns out that we regressed handling unreadable
files in `bare-user` mode. An example of this is `/etc/shadow`, which
ends up in the ostree-as-host content as `/usr/etc/shadow`.
Now there are better fixes here; we should probably delete it and create it
during the config merge if it doesn't exist. In general, having secret files in
ostree really isn't supported, so it doesn't make sense to include them.
But let's fix this regression - when operating as an unprivileged user we don't
have `CAP_DAC_OVERRIDE` and hence will fail to open un-user-readable objects.
(We still preserve the actual `0` mode of course in the xattr and will
apply it in `bare`)
Closes: #989
Approved by: jlebon
Previously, we only supported additions in the statoverride file;
it was mainly for adding the setuid bit without having that physically
on disk.
However, for testing a change to `bare-user` handling around *unreadable*
files (which happens for `/etc/shadow` in host content), I need a way
to write that into a repo in the test suite.
I'm not actually aware of a non-test-suite use case for this; a more
sophisticated user is going to be using the API directly, which can already do
this. But we need it for tests at least.
Closes: #989
Approved by: jlebon
This is a lot like `git clone --reference`, but we chose "localcache" as the
term "reference" is already used.
The main use case I'm targeting this for is the Fedora Atomic Host installer
case where we embed the repo content in the installer, but we may want to
kickstart and download newer content. There, while we want to get a newer ref,
we can still use the local repo as an object cache, since we have it sitting
there in memory anyways.
Another case is where one has a host ostree (say e.g. Fedora Atomic
Workstation), and one wants to create a local archive mirror of FAH. Then one
can use `pull --reference /ostree/repo` and pull the common objects (e.g.
contents of `bash.rpm` etc.)
Closes: https://github.com/ostreedev/ostree/issues/975Closes: #982
Approved by: jlebon
Previously, `ostree pull` was silent if not on a tty. I don't
see a reason not to print the final status line at least. This
is prep for more work in the test suite, so I can write assertions
on the output.
But it should also be nicer for people who e.g. do an `ostree pull` in a Jenkins
job or whatever.
Closes: #981
Approved by: jlebon
The `-z2` is annoying now since it's really a legacy; we've long
since supported typing `archive`. Convert the docs fully and
explain that.
Also do some (but not all) of the tests just to encourage newer tests to use
`archive` too.
Closes: #980
Approved by: jlebon
When updating a summary file, parse additional arguments to the `ostree
summary` command as additional metadata to be put into the summary.
Add some tests for this.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #961
Approved by: cgwalters
Previously the spurious ‘update’ atom was ignored; now that `ostree
summary --update` accepts extra arguments as additional metadata, it’s
causing an error. Drop it.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #961
Approved by: cgwalters
Test various pieces of core and command line utility functionality
relating to the newly-introduced concept of collections. Mostly focussed
around the find-remotes utility, and around handling of collection–refs.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
This is a more complex implementation of OstreeRepoFinder which resolves
ref names to remote URIs by looking for refs advertised by peers on the
local network using DNS-SD records and mDNS (Avahi). The idea is to
allow OS and app updates to be propagated over local networks, without
the internet.
It requires an OSTree server and code to generate the DNS-SD adverts in
order to be fully functional — support for this will be added
separately.
Unit tests are included.
Includes fixes by Krzesimir Nowak <krzesimir@kinvolk.io>.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
This will be used in an upcoming commit. It adds a basic bloom filter
implementation, using the SipHash family of hash functions.
The implementation (including its parameter choices and hash functions)
will become a protocol detail in future, so must not be changed so that
its output is bitwise incompatible between OSTree versions.
Unit tests are included.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
This is a basic implementation of OstreeRepoFinder which resolves ref
names to remote URIs by looking for them on any currently mounted
removable storage volumes. The idea is to support OS and app updates via
USB stick.
Unit tests are included.
This bumps libostree’s maximum GLib dependency from 2.44 to 2.50 for
g_drive_is_removable(). If GLib 2.50 is not available, the call which
needs it will be omitted and the OstreeRepoFinderMount implementation
will scan all volumes (not just removable ones); this is a performance
hit, but not a functionality hit.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
This is a basic implementation of OstreeRepoFinder which resolves ref
names to remote URIs by looking their collection IDs up in the local
configuration of remotes who have their collection-id key set.
Unit tests are included.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
These are tuples of (collection ID, ref name) which are a globally-unique
form of local ref. They use OstreeCollectionRef as an identifier, and hence
need to be accessed using new API, as the existing API uses string
identifiers and sometimes accepts refspecs. Remote names are not
supported as part an OstreeCollectionRef.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
There are a few places in the code where ad-hoc validation was being
performed. Might as well formalise it a bit more.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #948
Approved by: cgwalters
test-symbols.sh was looking for the DevelBuild string, which is actually
part of the output from `ostree --version`, not $OSTREE_FEATURES.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #948
Approved by: cgwalters
And use it in `test-symbols.sh`, to fix the `distcheck` case;
the previous change stopped distributing `libostree-devel.sym`
in release builds.
Closes: #944
Approved by: jlebon
I only checked the test passed, I didn't read the output closely, and
made it succeed without testing anything.
Fix the absolute/relative `/etc` references.
Closes: #937
Approved by: jlebon
We're hitting this in flathub, where we have a bunch of local builds,
but we also mirror a few refs from the gnome runtime repo into it.
Its fixable by re-doing the summary, but for a short time the
wrong version is visible.
Fixes https://github.com/ostreedev/ostree/issues/846Closes: #935
Approved by: cgwalters
The code here tried to truncate the string to the previous length,
but that doesn't work when recursing, since further calls change the
length.
What actually ended up happening was the string would get corrupted
after the first level of recursion.
Closes: #936
Approved by: jlebon
So far a lot of submitted PR have added symbols into the first
section. Split the file into `-released` and `-devel` to make
this more obvious.
To further enforce things, we hardcode a checksum of the `-released`
file in `test-symbols.sh`. Only release commits should update that
checksum.
Did you notice I like checksums?
Closes: #931
Approved by: pwithnall
For the flatpak PR: https://github.com/flatpak/flatpak/pull/849
It's really more convenient if this works for HTTP pulls as well, since flatpak
does various types of pulling, and we can just set the flag everywhere.
Further, we might as well reject the content as early as possible.
Closes: #930
Approved by: alexlarsson
This imports a function that is used in rpm-ostree, and it's also intended for
use by https://github.com/advancedtelematic/aktualizr to display
what deployment we're going to boot next after the reboot.
Updated-by: Colin Walters <walters@verbum.org>
Closes: #897
Approved by: OYTIS
This is a continuation of https://github.com/ostreedev/ostree/pull/926
for directories instead of files.
See: https://github.com/flatpak/flatpak/issues/845
This option suppresses mode bits outside of `0775` for directory
checkouts. I think most people should start doing this by default,
and use explicit overrides for e.g. `/tmp` if doing a recommit based
on a checkout.
Closes: #927
Approved by: alexlarsson
This is an option which is intended mostly for flatpak;
see: https://github.com/flatpak/flatpak/issues/845
We're adding an option for pulling into *all*
repo modes that has an effect similar to the `bare-user-only`
change from https://github.com/ostreedev/ostree/pull/909
This way one can pull content into e.g. a root-owned `bare` repository and
ensure that there aren't any setuid or world-writable files.
Closes: #926
Approved by: alexlarsson
Colin Walters
Philip Withnall
Krzesimir Nowak
Jonathan Lebon
Alexander Larsson
Anton Gerasimov