Quite a while ago we added staged deployments, which solved
a bunch of issues around the `/etc` merge. However...a persistent
problem since then is that any failures in that process that
happened in the *previous* boot are not very visible.
We ship custom code in `rpm-ostree status` to query the previous
journal. But that has a few problems - one is that on systems
that have been up a while, that failure message may even get
rotated out. And second, some systems may not even have a persistent
journal at all.
A general thing we do in e.g. Fedora CoreOS testing is to check
for systemd unit failures. We do that both in our automated tests,
and we even ship code that displays them on ssh logins. And beyond
that obviously a lot of other projects do the same; it's easy via
`systemctl --failed`.
So to make failures more visible, change our `ostree-finalize-staged.service`
to have an internal wrapper around the process that "catches" any
errors, and copies the error message into a file in `/boot/ostree`.
Then, a new `ostree-boot-complete.service` looks for this file on
startup and re-emits the error message, and fails.
It also deletes the file. The rationale is to avoid *continually*
warning. For example we need to handle the case when an upgrade
process creates a new staged deployment. Now, we could change the
ostree core code to delete the warning file when that happens instead,
but this is trying to be a conservative change.
This should make failures here much more visible as is.
Without reproducible images, a rebuild of the initrd will create a
different image file (due to things like creation time of the files in
the cpio archive) even if the actual contents in it are exactly the
same, adding an unnecessary download during updates.
Adding 'reproducible=yes' avoids this and creates the same image files
for the same content.
In https://bugzilla.redhat.com/show_bug.cgi?id=1827712
some OpenShift CI is seeing `/boot` being unmounted before
`ostree-finalize-staged.service` runs or completes.
We finally tracked this down to a bug elsewhere, but
I think we should add this because it clearly shows
our requirements.
With just `After=` we'll still try to run in the scenario
where `sysroot.mount` fails because the rootfs didn't appear.
And this will end up spewing an error which can confuse people
into thinking something is wrong at the ostree level.
This has come up numerous times w/{Fedora,RHEL} CoreOS, most
recently while looking at
https://bugzilla.redhat.com/show_bug.cgi?id=1803130
Same motivation as
https://github.com/coreos/rpm-ostree/pull/2060
I tried `InaccessiblePaths=/var` first and was very sad to find
out we have one tiny exception that breaks it. Otherwise it'd
be so elegant. Maybe in the future we split out that one thing
to a separate `ostree-finalized-stage-var.service` that's just
`ExecStart=/bin/rm -vf /var/.updated` and is otherwise
`ProtectSystem=strict` etc.
This is another attempt to avoid having duplicated menu entries caused by
GRUB having support to parse BLS snippets and the 15_ostree script adding
menu entries as well.
The previous attempt was in commit 985a141002 ("grub2: Exit gracefully if
the configuration has BLS enabled") but that lead to users not having menu
entries at all, due having an old GRUB version that was not able to parse
the BLS snippets.
This happened because the GRUB bootloader is never updated in the ESP as
a part of the OSTree upgrade transaction.
The logic is similar to the previous commit, the 15_ostree script exits if
able to determine that the bootloader can parse the BLS snippets directly.
But this time it will not only check that a BLS configuration was enabled,
but also that a /boot/grub2/.grub2-blscfg-supported file exists. This file
has to be created by a component outside of OSTree that also takes care of
updating GRUB to a version that has proper BLS support.
We want to support extending the read-only state to cover `/sysroot`
and `/boot`, since conceptually all of the data there should only
be written via libostree. Or at least for `/boot` should *mostly*
just be written by ostree.
This change needs to be opt-in though to avoid breaking anyone.
Add a `sysroot/readonly` key to the repository config which instructs
`ostree-remount.service` to ensure `/sysroot` is read-only. This
requires a bit of a dance because `/sysroot` is actually the same
filesystem as `/`; so we make `/etc` a writable bind mount in this case.
We also need to handle `/var` in the "OSTree default" case of a bind
mount; the systemd generator now looks at the writability state of
`/sysroot` and uses that to determine whether it should have the
`var.mount` unit happen before or after `ostree-remount.service.`
Also add an API to instruct the libostree shared library
that the caller has created a new mount namespace. This way
we can freely remount read-write.
This approach extends upon in a much better way previous work
we did to support remounting `/boot` read-write.
Closes: https://github.com/ostreedev/ostree/issues/1265
This reverts commit 985a141002.
It turned out that some people have old bootloaders, and hence
get the "no entries" problem. That's much, much much worse
than double entries.
Since Fedora 30 grub2 has support to populate its menu entries from the
BootLoaderSpec fragments in /boot/loader/entries, so there's no need to
generate menu entries anymore using the /etc/grub.d/15_ostree script.
But since ostree doesn't update the bootloader, it may be that the grub2
installed is an old one that doesn't have BLS support.
For new installs, GRUB_ENABLE_BLSCFG=true is set in /etc/default/grub to
tell the /etc/grub.d/10_linux script if a blscfg command has to be added
to the generated grub2 config file.
So check if BLS is enabled in /etc/default/grub and only add the entries
if that's not the case. Otherwise the menu entries will be duplicated.
The approach has the drawback that if a user sets GRUB_ENABLE_BLSCFG=true
in /etc/default/grub without updating grub2, they will get an empty menu.
Since there won't be any entries created by the 30_ostree script and the
blscfg command won't work on the older grub2.
Unfortunately there is no way to know if the installed grub2 already has
BLS support or not.
Related: https://bugzilla.redhat.com/show_bug.cgi?id=1751272#c27Closes: #1929
Approved by: jlebon
In Fedora 31, `systemd-journal-flush.service` uses a new
`--smart-relinquish-var` switch which fixes the
`umount: /var: target is busy` bug by telling journald to stop logging
to `/var` and back to `/run` again during shutdown.
This interacted with `ostree-finalize-staged.service` in a tricky way:
since we weren't strongly ordered against it, when we happened to
finalize after `/var` is relinquished, we never persisted the output
from that service to disk. This then threw off `rpm-ostree status` when
trying to find the completion message to know that finalization went
well.
Just fix this by adding an explicit `After=` on that unit. That way we
shut down *before* `systemd-journal-flush.service` (the `/var`
relinquish bit happens in its `ExecStop=`).
For more info, see:
3ff7a50d661e187d2dd5https://bugzilla.redhat.com/show_bug.cgi?id=1751272Closes: #1926
Approved by: cgwalters
Really, all `ostree admin finalize-staged` needs is access to `/sysroot`
and `/boot`. So let's activate it right after `local-fs.target` so that
it gets deactivated later in the shutdown process. This should allow us
to conflict with less services still running and possibly writing things
to `/etc`.
Related: https://bugzilla.redhat.com/show_bug.cgi?id=1672283Closes: #1840
Approved by: cgwalters
Similar as available for u-boot (ce2995e1dc)
and syslinux (c5112c25e4), enable parsing
and writing devicetree filename into grub.cfg.
This is required by arm64-based devices running edk2 instead of u-boot
as the main bootloader (e.g. 96boards HiKey and HiKey960).
Signed-off-by: Ricardo Salveti <ricardo@foundries.io>
Closes: #1790
Approved by: cgwalters
Even with the previous docstring, I didn't understand at first the
relationship between the `ostree-grub-generator` script and
`ostree-bootloader-grub2.c`. Throw some more docs to clarify things a
bit.
Closes: #1791
Approved by: cgwalters
Rather than manually starting the `ostree-finalize-staged.service` unit,
we can leverage systemd's path units for this. It fits quite nicely too,
given that we already have a path we drop iif we have a staged
deployment.
To give some time for the preset to make it to systems, we don't yet
drop the explicit call to `systemctl start`. Though we do make it
conditional based on a DEBUG env var so that we can actually test it in
CI for now. Once we're sure this has propagated, we can drop the
`systemctl start` path and the env var together.
Closes: #1740
Approved by: cgwalters
Previously, we were preparing the root very late in the boot process;
right before we switch root. The issue with that is that most services
in the initrd that run `After=initrd-root-fs.target` expect that
`/sysroot` already points to the rootfs we'll be pivoting to. Running
this late violates that assumption.
This patch fixes this by making `ostree-prepare-root.service` instead
run right after `sysroot.mount` (the physical sysroot mounted by
systemd) but still before `initrd-root-fs.target` (which is the target
signalling that `/sysroot` is now valid and ready).
This should make it easier to integrate OSTree with other initrd
services such as Ignition.
Related: https://github.com/dustymabe/ignition-dracut/issues/20Closes: #1759
Approved by: cgwalters
For the same reasons as #1697. This is especially important in services
that are likely to be used as an `After/Before=` target in other units.
`ostree-prepare-root.service` is one such service.
Closes: #1759
Approved by: cgwalters
See https://github.com/projectatomic/rpm-ostree/issues/1568
Basically for people on e.g. rotational media, the default 90
second timeout can be too small.
We're in a tough situation here, because delaying shutdown
can be problematic too if the user is trying to shut down their
laptop to put in a backpack, etc.
There's potential optimizations here to make; I think we
could pre-copy the kernel/initramfs for example.
I suspect for some people the grub2 os-prober is a factor here too,
if that tries to e.g. inspect attached USB rotational hard drives.
But hopefully we'll get rid of that soon.
Closes: #1755
Approved by: jlebon
It's a neat way to point folks to the documentation (of course, better
would be to have man pages for each of those services). Also
consistently use Title Case everywhere.
Closes: #1750
Approved by: cgwalters
This is standard practice for units like this; e.g. it's what
`systemd-remount-fs.service` does. I think it may be part of
or the whole cause for
https://github.com/projectatomic/rpm-ostree/issues/1471
I haven't reproduced the problem exactly but it seems to me that
if the unit starts and is GC'd, then when systemd goes to execute
a later unit it might end up restarting it.
A noticeable side effect of this is that `systemctl status ostree-remount`
exits with code `0` as expected.
Closes: #1697
Approved by: jlebon
Follow systemd units in using emergency.target, not emergency.service
(which is the sole unit, by default, in emergency.target) so we can
easily reconfigure the units which are actived when entering
emergency mode.
Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
Closes: #1665
Approved by: cgwalters
Currently the BLS snippets are named ostree-$ID-$VARIANT_ID-$index.conf,
but the BLS config files are actually sorted by using the version field
which is the inverse of the index.
In most places, _ostree_sysroot_read_boot_loader_configs() is used to
get the BLS files and this function already returns them sorted by the
version field. The only place where the index trailing number is used is
in the ostree-grub-generator script that lists the BLS files to populate
the grub config file.
But for some bootloaders the BLS filename is the criteria for sorting by
taking the filename as a string version. So on these bootloaders the BLS
entries will be listed in the reverse order.
To avoid that, change the BLS snippets filename to have the version field
instead of the index and also to have the version before deployment name.
Make the filenames to be of the form ostree-$version-$ID-$VARIANT_ID.conf
so the version is before the deployment name.
Signed-off-by: Javier Martinez Canillas <javierm@redhat.com>
Closes: #1654
Approved by: cgwalters
The ostree-grub-generator populates the grub.cfg menu entries using the
BLS config files. But it uses the ls command that by default sorts the
entries alphabetically, so the order won't be correct if there are more
than 10 deployments, i.e:
$ ls -1 /boot/loader/entries/
ostree-fedora-workstation-0.conf
ostree-fedora-workstation-10.conf
ostree-fedora-workstation-1.conf
...
So instead the -v option should be used to make ls use version sorting:
$ ls -1 -v /boot/loader/entries/
ostree-fedora-workstation-0.conf
ostree-fedora-workstation-1.conf
...
ostree-fedora-workstation-10.conf
Signed-off-by: Javier Martinez Canillas <javierm@redhat.com>
Closes: #1653
Approved by: cgwalters
Add API to write a deployment state to `/run/ostree/staged-deployment`,
along with a systemd service which runs at shutdown time.
This is a big change to the ostree model for hosts,
but it closes a longstanding set of bugs; many, many people have
hit the "losing changes in /etc" problem. It also avoids
the other problem of racing with programs that modify `/etc`
such as LVM backups:
https://bugzilla.redhat.com/show_bug.cgi?id=1365297
We need this in particular to go to a full-on model for
automatically updated host systems where (like a dual-partition model)
everything is fully prepared and the reboot can be taken
asynchronously.
Closes: https://github.com/ostreedev/ostree/issues/545Closes: #1503
Approved by: jlebon
SPDX License List is a list of (common) open source
licenses that can be referred to by a “short identifier”.
It has several advantages compared to the common "license header texts"
usually found in source files.
Some of the advantages:
* It is precise; there is no ambiguity due to variations in license header
text
* It is language neutral
* It is easy to machine process
* It is concise
* It is simple and can be used without much cost in interpreted
environments like java Script, etc.
* An SPDX license identifier is immutable.
* It provides simple guidance for developers who want to make sure the
license for their code is respected
See http://spdx.org for further reading.
Signed-off-by: Marcus Folkesson <marcus.folkesson@gmail.com>
Closes: #1439
Approved by: cgwalters
Previously when initramfs-* was not found in a deployment's
boot directory, it was assumed that rootfs is prepared for
ostree booting by a kernel patch.
With this patch, the behaviour changes to be - if initramfs-*
is not found, assume that system is using a static
ostree-prepare-root as init process. Booting without initramfs
is a common use case on embedded systems. This approach is
also more convenient, than having to patch the kernel.
Closes: #1401
Approved by: cgwalters
Apparently there testing systems that literally install *all*
packages. Having `ostree-grub2` currently causes grub2 to fail
on a non-ostree managed system. Let's just gracefully exit
if there's no system repository.
https://bugzilla.redhat.com/show_bug.cgi?id=1532668Closes: #1399
Approved by: jlebon
This is simplest for now. Compare with similar logic from
`/usr/lib/tmpfiles.d/tmp.conf`:
```
R! /tmp/systemd-private-*
```
Closes: https://github.com/ostreedev/ostree/issues/393Closes: #1090
Approved by: jlebon
Why not to use libexecdir?
Because this directory does not exist on some distros or paths
between distros varies. There are several reasons why a well
known path is prefered, for example when generating a kernel
command line:
init=${ostree}/usr/lib/ostree-boot/ostree-prepare-root
In addition this saves us some typing in a console when wanting
to access the "ostree" cmd line.
Closes: #449
Approved by: cgwalters
ostree-grub-generator can be used to customize
the generated grub.cfg file. Compile time
decision ostree-grub-generator vs grub2-mkconfig
can be overwritten with the OSTREE_GRUB2_EXEC
envvar - useful for auto tests and OS installers.
Why this alternative approach:
1) The current approach is less flexible than using a
custom 'ostree-grub-generator' script. Each system can
adjust this script for its needs, instead of using the
hardcoded values from ostree-bootloader-grub2.c.
2) Too much overhead on embedded to generate grub.cfg
via /etc/grub.d/ configuration files. It is still
possible to do so, even with this patch applied.
No need to install grub2 package on a target device.
3) The grub2-mkconfig code path has other issues:
https://bugzilla.gnome.org/show_bug.cgi?id=761180
Task: https://bugzilla.gnome.org/show_bug.cgi?id=762220Closes: #228
Approved by: cgwalters
In this approach, we drop a /etc/grub.d/15_ostree file which is a
hybrid of shell/C that picks up bits from the GRUB2 library (e.g. the
block device script generation), and then calls into libostree's
GRUB2 code which knows about the BLS entries.
This is admittedly ugly. There exists another approach for GRUB2 to
learn the BLS specification. However, the spec has a few issues:
https://www.redhat.com/archives/anaconda-devel-list/2014-July/msg00002.html
This approach also gives a bit more control to the admin via the
naming of the 15_ostree symlink; they can easily disable it:
Or reorder the ostree entries ahead of 10_linux:
Also, this approach doesn't require patches for grub2, which is an
issue with the pressure to backport (rpm-)OSTree to EL7.
tmpfiles.d configurations generally require write access to some places
that are read-only until ostree-remount runs.
Make sure ostree-remount has run first.
Thanks to Cosimo Cecchi for finding and diagnosing this problem.
https://bugzilla.gnome.org/show_bug.cgi?id=724183
Colin Walters
Joseph Marrero
Jonathan Lebon
Matt Bilker
Simon McVittie
João Paulo Rechi Vita
Javier Martinez Canillas
Robert Fairley
Ricardo Salveti
Alex Kiernan
Marcus Folkesson
Gatis Paeglis
Will Thompson
Carlos Alberto Lopez Perez
Gatis Paeglis
Giuseppe Scrivano
Daniel Drake
Daniel Narvaez