I feel like I'm drowning in a pile of experimental-but-almost-stable
features...
Anyways, since we made the feature opt-in in rpm-ostree in
https://github.com/projectatomic/rpm-ostree/pull/1352
let's mirror that a bit here with an environment variable so people
can play with it more easily.
The tests needed some tweaks; specifically we need to reload the
status fact after making changes. I'm still a bit uncertain
about the Ansible-as-tests.
But we add an upgrade test that uses the new environment variable.
Closes: #1583
Approved by: jlebon
This should give a more insightful error message if the user provides
a UID which is present on multiple keys.
This happens if you have an old key in your keyring which you are not
actively using any more, e.g. because it is too old. You still have
your old keys in your keyring, because you want to read old email
encrypted for that key, though.
The gpgme function used by ostree right now complains if a UID is found
on multiple keys:
https://www.gnupg.org/documentation/manuals/gpgme/Listing-Keys.html#index-gpgme_005fget_005fkey
The used API is too simple for that use case.
Note that it would be nicer if ostree picked the only valid signing key out
of the available keys rather than using the simplistic gpgme_get_key
function. It be nicer, of course, if there was such a gpgme function.
Closes: #1579
Approved by: cgwalters
Let's add a semi-colon between the "bootconfig swap" part and the
"deployment count change" to make it more clear they're separate
statements.
Closes: #1575
Approved by: cgwalters
While we do a `syncfs()` plus `FIFREEZE/THAW` for `/boot`, that
only comes during deployment finalization.
The code here today generally assumes that if the file exists
it's been fully written. So let's do a `fdatasync()` before
we do the `rename()`.
This just came out of looking through the code while working
on deployment staging. In that scenario there's a much larger
window between when we copy the kernel/initramfs and when we
sync `/boot`.
Closes: #1571
Approved by: jlebon
This is easier to `git grep` etc. versus ad-hoc English. Although
we still have some English for the prepare_transaction/commit which
acquire/release in separate phases.
Closes: #1572
Approved by: jlebon
These are further fixes based on running more of the rpm-ostree
test suite.
When dropping the staged deployment, we do need to do the
"post operations" such as bumping the sysroot mtime, so that
clients know something changed. We also need to regenerate
the deployment refs. And of course do a sysroot reload.
Also, add a "base cleanup" after creating a staged deployment
which also regenerates the refs.
Closes: #1570
Approved by: jlebon
There's no reason to do this. I didn't actually hit this problem,
but it's a corner case that just occurred to me while working on
the code.
I think callers should be adapted to skip trying to use staging
if there's no booted deployment.
Closes: #1568
Approved by: jlebon
This was pointed out in a previous PR review; we don't have
a need for the separate variables. Prep for adding an API for
this.
Closes: #1568
Approved by: jlebon
Doing so can break rpm-ostree, which wants to own the cleanup process
to ensure its baselayer refs are generated.
Further, doing the cleanup at shutdown time adds latency. It's also
going to be generally unnecessary as we expect repo pruning to have
been done when writing the refs.
Closes: #1567
Approved by: jlebon
The code has been sitting around for a while but since I disabled
it by default, I doubt anyone is really using it or relying on it.
This patch and turns on locking by default, and also drops the
API which was only public in the experimental API builds.
Conceptually these are two distinct things, and we
may actually want to split up the patches.
I don't think this will break anyone, but it's hard to say for sure.
It's also going to be hard to find out until we actually release
I suspect...
But anyone who is broken should be able to add `locking=false` into
their repo config. On the flip side Endless has been shipping with
this enabled and it is reported to help.
The reason to drop the APIs: I'm a bit concerned about the interactions over time
between libostree's use of the API and any apps that start using it.
For example, if an app specifies a SHARED lock in their code, then
later internally we decide to temporarily grab an `EXCLUSIVE`, but the
app had a second thread/process that was `EXCLUSIVE` already, and
that process was waiting on the first bit of code, then we could
deadlock. I can't think of a real world situation where this would happen
yet though.
We are likely to in the future have say `fsck` take an external lock,
`checkout` grab a shared one, etc.
Closes: #1555
Approved by: jlebon
Today rpm-ostree has some code to run a "sanitycheck" on a deployment.
I had initially deleted that when adapting it to use the staging code,
but I realized it should work fine; we just won't see the merged
config, but that's OK.
When I readded that code it started crashing because we didn't
actually return the new deployment object. We'll gain some coverage
here as I'll land the code to have rpm-ostree use staging, then bump
the rpm-ostree tests here.
Closes: #1559
Approved by: jlebon
Testing out the staged API with rpm-ostree, ostree-prepare-root.service
in the initramfs was failing. Turned out that was because we didn't
have a `root=` kernel argument. Which was because we didn't have
any kernel arguments at all except `ostree=`.
That in turn was because we weren't loading the bootloader config
from the merge deployment.
The serialized deployment data holds the unique identity of
(osname, checksum, deployserial) - look for the real merge deployment
in our deployment list which has the bootloader arguments we need.
This issue was entirely masked by the `ostree admin deploy` command
which itself explicitly loads the merge deployment's kernel arguments
in every case - it never passes the `NULL` default down. A followup
patch will fix that.
Closes: #1558
Approved by: jlebon
When comparing deployments to determine whether we need a new
bootversion, we should also check whether the commit "version" metadata
is the same. Otherwise, we may end up with the a bootconfig whose
`title` includes a version that doesn't match the one from the
deployment checksum.
Closes: https://github.com/projectatomic/rpm-ostree/issues/1343Closes: #1556
Approved by: cgwalters
This is a normal case when running unit tests in client code
on continuous integration infrastructure. When those tests are
running they will set G_DEBUG=fatal-warnings which will cause
the program to abort if a warning is emitted. Instead, emit
a debug message if the problem was that we couldn't connect to
the daemon.
Closes: #1542
Approved by: jlebon
Followup to: https://github.com/ostreedev/ostree/pull/1503
After starting some more work on on this in rpm-ostree, it is
actually simpler if the staged deployment just shows up in the list.
It's effectively opt-in today; down the line we may make it the default,
but I worry about breaking things that e.g. assume they can mutate
the deployment before rebooting and have `/etc` already merged.
There's not that many things in libostree that iterate over the deployment
list. The biggest change here is around the
`ostree_sysroot_write_deployments_with_options` API. I initially
tried hard to support a use case like "push a rollback" while retaining
the staged deployment, but everything gets very messy because that
function truly is operating on the bootloader list.
For now what I settled on is to just discard the staged deployment;
down the line we can enhance things.
Where we then have some new gymnastics is around implementing
the finalization; we need to go to some effort to pull the staged
deployment out of the list and mark it as unstaged, and then pass
it down to `write_deployments()`.
Closes: #1539
Approved by: jlebon
This is a version of ostree_repo_traverse_commit_union that also
remembers where the objects came from, by recording the parent
relationships in a hashtable. This can be used to later find which
commits each object was from, which we want to use in fsck.
Closes: #1533
Approved by: cgwalters
reintroduce the feature that was reverted with commit:
28c7bc6d0e
Differently than the original implementation, now we don't attempt any
test for reflinks support on the parent repository, since the test
requires write access to the repository.
Additionally, also check that the two repositories are on the same
device before attempting any reflink.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Closes: #1525
Approved by: cgwalters
In prep for staging work, where we'll need to load the origin
for the staged deployment too.
The function was previously trying to avoid operating on an
instantiated deployment, but the data we need is in the deployment
object at that point.
Closes: #1538
Approved by: jlebon
Prep for handling staged deployments better; if we're not passed
the staged one back, then we just want to delete it but not
touch the bootloader config.
Closes: #1538
Approved by: jlebon
The reason we were returning a hashtable is a bit lost to history,
there's no reason to do so now anyways. Also port to declare-and-initialize
style and add more comments.
Closes: #1538
Approved by: jlebon
Add API to write a deployment state to `/run/ostree/staged-deployment`,
along with a systemd service which runs at shutdown time.
This is a big change to the ostree model for hosts,
but it closes a longstanding set of bugs; many, many people have
hit the "losing changes in /etc" problem. It also avoids
the other problem of racing with programs that modify `/etc`
such as LVM backups:
https://bugzilla.redhat.com/show_bug.cgi?id=1365297
We need this in particular to go to a full-on model for
automatically updated host systems where (like a dual-partition model)
everything is fully prepared and the reboot can be taken
asynchronously.
Closes: https://github.com/ostreedev/ostree/issues/545Closes: #1503
Approved by: jlebon
A newly created archive-mode repository won't have a uncompressed-objects-cache
directory, and uncompressed_objects_dir is -1 to flag that. The special meaning of
-1 meaning "cwd" for libglnx means that the current directory was scanned as
if it was an objects directory, producing unexpected results, especially if there
were any two-letter files/subdirs in the current directory.
Closes: #1537
Approved by: jlebon
Currently OstreeRepoFinderResult, a data structure used by pull code
that supports P2P operations, has a hash table mapping refs to checksums
but doesn't include timestamp information. This means that clients have
no way of knowing just from the OstreeRepoFinderResult information if a
commit being offered by a peer remote is an update or downgrade until
they start pulling it. The client could check the summary or the commit
metadata for the timestamps, but this requires adding the temporary
remotes to the repo config, and ostree is already checking timestamps
before returning the results, so I think it makes more sense for them to
be returned rather than leaving it to the client. This limitation is
especially important for offline computers, because for online computers
the latest commit available from any remote is the latest commit,
period.
This commit adds a "ref_to_timestamp" hash table to
OstreeRepoFinderResult that is symmetric to "ref_to_checksum" in that it
shares the same keys. This is an API break, but it's part of the
experimental API, and none of the current users of that (flatpak,
eos-updater, and gnome-software) are affected. See the documentation for
more details on "ref_to_timestamp". One thing to note is the data
structure currently gets initialized in find_remotes_cb(), so only users
of ostree_repo_find_remotes_async() will get them, not users of, say,
ostree_repo_finder_resolve_all_async(). This is because the individual
OstreeRepoFinder implementations don't currently access the timestamps
(but I think this could be changed in the future if there's a need).
This commit will allow P2P support to be added to
flatpak_installation_list_installed_refs_for_update, which will allow
GNOME Software to update apps from USB drives while offline (it's
already possible online).
Closes: #1518
Approved by: cgwalters
In case of some kind of race or other weirdness we might be getting
non-matching versions of summary.sig and summary, where summary.sig
is the latest version. Currently we're saving them to the cache
directly after downloading them successfully, but they will then fail
to gpg validate. Then on the next run we'll keep using the cached files
even if they are incorrect, until summary.sig changes upstream.
This changes the order so that we verify the signatures before saving
to the cache, thus ensuring that we don't end up in a stuck state.
Fixes https://github.com/ostreedev/ostree/issues/1523Closes: #1529
Approved by: cgwalters
In ostree_repo_remote_fetch_summary_with_options(), if no summary is
found on the server and summary verification is enabled, the error
message implies that it's the summary signature that's missing, which is
misleading. This commit adds a more specific error message for the case
of a missing summary, which has the side effect of explicitly checking
for the case that signatures != NULL && summary == NULL after
repo_remote_fetch_summary(), even though that should never happen.
One effect of this is that if you run "flatpak remote-add" with an
incorrect URL you get a more helpful error message, and similarly for
other flatpak operations and other users of ostree.
Closes: #1522
Approved by: cgwalters
In libostree, the phrase "commit metadata" has two meanings-- one is the
first dictionary in a commit GVariant that stores metadata such as ref
bindings, and the other is the commit metadata in the summary file,
which stores the commit size, checksum, and timestamp. In
find_remotes_process_refs(), the entire commit GVariant was being
referred to as commit metadata, so this commit changes the variable
name and a comment to make things more consistent.
Closes: #1528
Approved by: cgwalters
In _try_clone_from_payload_link, don't try to do the clone in the
parent repo, because we don't want to modify that. parent repos are
typically used when you want a shared, immutable base.
For example in flatpak, the parent repo is the system repo which you
don't have write access to, so any modification to it will fail with
EACCES, making it impossible to install via the system helper.
Closes: #1524
Approved by: cgwalters
ostree_repo_pull_from_remotes_async() passes along some options to
ostree_repo_pull_with_options(), so document them.
Closes: #1519
Approved by: cgwalters
The main blocker for doing this before was the `goto out` handling
for remounting `/boot`. Handle that by factoring out the bits that
require it to a helper function, and do the C/GError equivalent of
"try/finally".
Not prep for anything right now, just decided to do this since I had the file
open.
Closes: #1515
Approved by: jlebon
For staged deploy, we want to pay the cost of creating copies from
`/usr/etc` → `/etc` at stage time, since it can be expensive. (We
want to minimize time spent during shutdown).
Split it up into two functions; the logic is also simply clearer.
Closes: #1514
Approved by: jlebon
Prep for deployment staging. We had the code to hande "explicit kargs" in one
place, but the "use merge deployment" karg bits mixed in with the "/etc merge"
logic. Those are separate things, and it's better to have karg handling in one
place.
Closes: #1514
Approved by: jlebon
A quick turnaround to include one PR: https://github.com/ostreedev/ostree/pull/1508
"switchroot: Ensure /run/ostree-booted is created even without initramfs"
This fixes ostree when booting without an initramfs. Thanks to @akiernan for the
bug report and helping review the fix! I'm working on enhancing
the test suite, which will help in adding some coverage here.
Also for this release I'm going to avoid adding a "stub" symbol section
to the `-released.sym` file; I don't believe it's necessary.
Closes: #1512
Approved by: jlebon
It's been over a month since 2018.2; we have a few features and various fixes,
and the "stage" work pending which is pretty invasive. Time for a new release!
Closes: #1506
Approved by: jlebon
Ensures it's labeled consistently. Prep for staged deployments which reworks the
logic around when the origin file is written.
Closes: #1505
Approved by: jlebon
Pulling some of this out of stage deploy work. It's generally better as it's
easier to change functions to have multiple callers.
Closes: #1505
Approved by: jlebon
We do already have `http-headers`, which potentially could be used to
allow clients to completely override the field, but it seems like the
more common use case is simply to append.
Closes: #1496
Approved by: cgwalters
Allow users to pass `<remote>:` to list all refs we have locally
belonging to `<remote>`. Also (re-)allow the similar `<remote>:.` syntax
for backwards compatibility with flatpak.
Closes: #1500
Approved by: cgwalters
I was looking at this code in prep for "staging" deployments,
and there are several cleanups to be made here. The first
thing I noticed is that we look for the `ostree=` kernel argument,
but the presence of that should be exactly equivalent to having
`/run/ostree-booted` exist. We just added a member variable for
that, so let's make use of it.
Related to this, we were erroring out if we had the karg but
didn't find a deployment. But this can happen if e.g. one is
using `ostree admin --sysroot` from an ostree-booted system! It's
actually a bit surprising no one has reported this so far; I guess
in the end people are either using non-ostree systems or running
from containers.
Let's add a member variable `root_is_sysroot` that we can use
to determine if we're looking at `/`. Then, our more precise
"should find a booted deployment" state is when both `ostree_booted`
and `root_is_sysroot` are TRUE.
Next, rather than walking all of the deployments after parsing,
we can inline the `fstatat()` while parsing. The mild ugly
thing about this is assigning to the sysroot member variable while
parsing, but I will likely clean that up later, just wanted to avoid
rewriting everything in one go.
Closes: #1497
Approved by: jlebon
Colin Walters
Tobias Mueller
Jonathan Lebon
Matthew Leeds
Sam Spilsbury
Alexander Larsson
Giuseppe Scrivano
Owen W. Taylor