Don't mention deprecation in the description for
`ostree_sysroot_deploy_tree` since there are legitimate use cases for it
(e.g. to create the first deployment via `ostree admin deploy`).
Instead, make the comment clearly redirect to the staging API when
booted into the sysroot.
Tighten up how we handle kargs here so it's more clear. When we call
`sysroot_finalize_deployment`, any karg overrides have already been set
on the bootconfig object of the deployment. So re-setting it here is
redundant and confusing.
I was thinking a bit more recently about the "live" changes
stuff https://github.com/coreos/rpm-ostree/issues/639
(particularly since https://github.com/coreos/rpm-ostree/pull/2060 )
and I realized reading the last debates in that issue that
there's really a much simpler solution; do exactly the same
thing we do for `ostree admin unlock`, except mount it read-only
by default.
Then, anything that wants to modify it does the same thing
libostree does for `/sysroot` and `/boot` as of recently; create
a new mount namespace and do the modifications there.
The advantages of this are numerous. First, we already have
all of the code, it's basically just plumbing through a new
entry in the state enumeration and passing `MS_RDONLY` into
the `mount()` system call.
"live" changes here also naturally don't persist, unlike what
we are currently doing in rpm-ostree.
These allow the `summary` and `summary.sig` files to be cached at a
higher layer (for example, flatpak) between related pull operations (for
example, within a single flatpak transaction). This avoids
re-downloading `summary.sig` multiple times throughout a transaction,
which increases the transaction’s latency and introduces the possibility
for inconsistency between parts of the transaction if the server changes
its `summary` file part-way through.
In particular, this should speed up flatpak transactions on machines
with high latency network connections, where network round trips have a
high impact on the latency of an overall operation.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
We use a similar trick to having a `sysroot -> .` symlink on the real root
here to support both /boot on root as well as on a separate filesystem. No
matter how it's mounted `/boot/xyz` will always refer to the file you'd
expect.
This is nicer than my previous attempts at this because there's no
configuration nor auto-detection required.
The "new style" code generally avoids `goto err` because it conflicts
with `__attribute__((cleanup))`. This fixes a compiler warning.
Signed-off-by: Frédéric Danis <frederic.danis@collabora.com>
If `glnx_make_lock_file` falls back to `flock`, on NFS this uses POSIX
locks (`F_SETLK`). As such, we need to be able to handle `EACCES` as
well as `EAGAIN` (see `fnctl(2)`).
I think this is what coreos-ostree-importer has been hitting, which runs
on RHEL7 in the Fedora infra and does locking over an NFS share where
multiple apps could concurrently pull things into the repo.
This came in with 5af403be0c but
was never implemented.
I noticed this now because the Rust ostree bindings generate a
wrapper for it which the linker tries to use.
This is the dual of 1f3c8c5b3d
where we output more detail when signapi fails to validate.
Extend the API to return a string for success, which we output
to stdout.
This will help the test suite *and* end users validate that the expected
thing is happening.
In order to make this cleaner, split the "verified commit" set
in the pull code into GPG and signapi verified sets, and have
the signapi verified set contain the verification string.
We're not doing anything with the verification string in the
pull code *yet* but I plan to add something like
`ostree pull --verbose` which would finally print this.
To aid debuggability, when we find a commit that isn't signed
by our expected key, output a specific error message with the
key.
(And then add code to switch to just printing the count beyond 3
because the test suite injects 100 keys and hopefully no one
ever actually does that)
Add a standard key for this. We actually had a case in OpenShift
builds recently where a `ppc64le` image was pushed over an `x86_64`
one and this started failing at runtime with a not immediately
obvious error.
I'll probably end up changing rpm-ostree at least to use
the RPM architecture for this key and fail if it doesn't match
the booted value.
Possibly that should live in ostree but it would involve adding
architecture schema here, which gets into a big mess. Let's
just standardize the key.
xref e02ef2683d
Add support for a devicetree directory at /usr/lib/modules/$kver/dtb/.
In ARM world a general purpose distribution often suppports multiple
boards with a single operating system. However, OSTree currently only
supports a single device tree, which does not allow to use the same
OSTree on different ARM machines. In this scenario typically the boot
loader selects the effective device tree.
This adds device tree directory support for the new boot artefact
location under /usr/lib/modules. If the file `devicetree` does not
exist, then the folder dtb will be checked. All devicetrees are hashed
into the deployment hash. This makes sure that even a single devicetree
change leads to a new deployment and hence can be rolled back.
The loader configuration has a new key "devicetreepath" which contains
the path where devicetrees are stored. This is also written to the
U-Boot variable "fdtdir". The boot loader is expected to use this path
to load a particular machines device tree from.
Closes: #1900
Signed-off-by: Stefan Agner <stefan.agner@toradex.com>
One OpenShift user saw this from rpm-ostree:
```
client(id:cli dbus:1.583 unit:machine-config-daemon-host.service uid:0) added; new total=1
Initiated txn UpdateDeployment for client(id:cli dbus:1.583 unit:machine-config-daemon-host.service uid:0): /org/projectatomic/rpmostree1/rhcos
Txn UpdateDeployment on /org/projectatomic/rpmostree1/rhcos failed: File header size 4294967295 exceeds size 0
```
which isn't very helpful. Let's add some error
prefixing here which would at least tell us which
object was corrupted.
The goal here is to move the code towards a model
where the *client* can explicitly specify which signature types
are acceptable.
We retain support for `sign-verify=true` for backwards compatibility.
But in that configuration, a missing public key is just "no signatures found".
With `sign-verify=ed25519` and no key configured, we can
explicitly say `No keys found for required signapi type ed25519`
which is much, much clearer.
Implementation side, rather than maintaining `gboolean sign_verify` *and*
`GPtrArray sign_verifiers`, just have the array. If it's `NULL` that means
not to verify.
Note that currently, an explicit list is an OR of signatures, not AND.
In practice...I think most people are going to be using a single entry
anyways.
There's a lot of historical baggage associated with GPG verification
and `ostree pull` versus `ostree pull-local`. In particular nowadays,
if you use a `file://` remote things are transparently optimized
to e.g. use reflinks if available.
So for anyone who doesn't trust the "remote" repository, you should
really go through through the regular
`ostree remote add --sign-verify=X file://`
path for example.
Having a mechanism to say "turn on signapi verification" *without*
providing keys goes back into the "global state" debate I brought
up in https://github.com/ostreedev/ostree/issues/2080
It's just much cleaner architecturally if there is exactly one
path to find keys: from a remote config.
So here in contrast to the GPG code, for `pull-local` we explictily
disable signapi validation, and the `ostree_repo_pull()` API just
surfaces flags to disable it, not enable it.
For the same reason as https://github.com/coreos/rpm-ostree/pull/2094.
What we care most about is that the new commit we pull is newer than the
one we're currently sitting on, not necessarily that it's newer than the
branch itself, which it might not be if e.g. we're trying to deploy a
commit older than the tip but still newer than the deployment (via
`--override-commit`).
The way `timestamp-check` works might be too restrictive in some
situations. Essentially, we need to support the case where users want to
pull an older commit than the current tip, but while still guaranteeing
that it is newer than some even older commit.
This will be used in Fedora CoreOS. For more information see:
https://github.com/coreos/rpm-ostree/pull/2094https://github.com/coreos/fedora-coreos-tracker/issues/481
Previously in the pull code, every time we went to verify
a commit we would re-initialize an `OstreeSign` instance
of each time, re-parse the remote configuration
and re-load its public keys etc.
In most cases this doesn't matter really because we're
pulling one commit, but if e.g. pulling a commit with
history would get a bit silly.
This changes things so that the pull code initializes the
verifiers once, and reuses them thereafter.
This is continuing towards changing the code to support
explicitly configured verifiers, xref
https://github.com/ostreedev/ostree/issues/2080
This cleans up the verification code; it was weird how
we'd get the list of known names and then try to create
an instance from it (and throw an error if that failed, which
couldn't happen).
The GI scanner decides if an `enum` is really a `bitfield` if it finds
any values that have left shifts. With an `enumeration`, the
introspecting language may error or convert to a different type if the
user tries to combine values. Change all Flags `enum`s to use
left-shifted values so that they're represented as `bitfield`s in the
GIR.
The primary bug here is that you can't combine `REFS_ONLY` and
`NO_PRUNE` when calling `OSTree.Repo.prune()` from an introspected
language.
This is an IABI break since the typelib will change from `enumeration`
to `bitfield`. `OstreeRepoImportFlags` is internal but the change is
included here to prepare for a subsequent name that would require bit
shifting to operate correctly as a flag.
Explicitly expose functions for querying the metadata format
and key name used by OstreeSign object:
- ostree_sign_metadata_format
- ostree_sign_metadata_key
This allows to use the same metadata format and key name
by 3-rd party applications using signapi.
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
I've only noticed this by inspection. But I think it's possible for
`cleanup_txn_dir` to get called with the `staging-...-lock` file since
it matches the prefix.
Make the checking here stronger by verifying that it's a directory. If
it's not a directory (lockfile), then follow the default pruning expiry
logic so that we still cleanup stray lockfiles eventually.
`ostree-repo-pull.c` is rather monstrous; I plan to split it
up a bit. There's actually already a `pull-private.h` but
that's just for the binding verification API. I think that one
isn't really pull specific. Let's move it into the "catchall"
`repo.c`.
Previously we would pass the `verification-key` and `verification-file`
to all backends, ignoring errors from loading keys until we
found one that worked.
Instead, change the options to be `verification-<engine>-key`
and `verification-<engine>-file`, and then
rework this to use standard error handling; barf explicitly if
we can't load the public keys for example. Preserve
the semantics of accepting the first valid signature. The
first signature error is captured, the others are currently
compressed into a `(and %d more)` prefix.
And now that I look at this more closely there's a lot of
duplication between the two code paths in pull.c for verifying;
will dedup this next.
I'm mainly doing this to sanity check the CI state right now.
However, I also want to more cleanly/clearly distinguish
the "sign" code from the "gpg" code.
Rename one function to include `gpg`.
For the other...I think what it's really doing is using the remote
config, so change it to include `remote` in its name.
If GPG support is disabled in a build time we should to check if any of
options "gpg_verify" or "gpg_verify_summary" is set to TRUE instead
of checking if they are passed via options while pulling from remote.
Fixed the failure with assertion of `ostree find-remotes --pull --mirror`
calling (`tests/test-pull-collections.sh`) if libostree has been compiled
without GPG support.
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
This code wasn't written with idiomatic GError usage; it's not standard
to construct an error up front and continually append to its
message. The exit from a function is usually `return TRUE`,
with error conditions before that.
Updating it to match style reveals what I think is a bug;
we were silently ignoring failure to parse key files.
When we're only pulling a subset of the refs available in the remote, it
doesn't make sense to copy the remote's summary (which may not be valid
for the local repo). This makes the check here match the one done
several lines above when we decide whether to error out if there's no
remote summary available.
This extends the fix in https://github.com/ostreedev/ostree/pull/935 for
the case of collection-refs.
Also, add a unit test for this issue, based on the existing one in
pull-test.sh.
Noticed this while writing tests for a core `ostree_sysroot_load()`
entrypoint. And decided to do the same for `ostree_repo_open()`,
and while there also noted we had a duplicate error prefixing
for the open (more recently `glnx_opendirat()` automatically
prefixes with the path).
Correctly return "error" from `ostree_repo_sign_commit()`
in case if GPG is not enabled.
Use glnx_* functions in signature related pull code for clear
error handling if GPG isn't enabled.
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
Do not mask implementation anymore since we have a working
engines integrated with pulling mechanism.
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
The "new style" code generally avoids `goto err` because it conflicts
with `__attribute__((cleanup))`. This fixes a compiler warning.
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
The "new style" code generally avoids `goto err` because it conflicts
with `__attribute__((cleanup))`. This fixes a compiler warning.
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
Improve error handling for signatures checks -- passthrough real
reasons from signature engines instead of using common messages.
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
Return the collected errors from signing engines in case if verification
failed for the commit.
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
Add more precise error handling for ed25519 initialization.
Check the initialization status at the beginning of every public
function provided by ed25519 engine.
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
Change the API of supporting functions `_load_public_keys()` and
`_ostree_repo_sign_verify()` -- pass repo object and remote name
instead of OtPullData object. This allows to use these functions
not only in pull-related places.
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
We don't need anymore stubs for verification options for remotes
in case if ostree built without GPG support.
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
Usage of 'g_warning()' inside keys loading funcrion lead to false
failure: the key loading attempt for the wrong engine breaks the
pulling process instead of trying to use this key with correct engine.
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
The initial implementation with single key for secret and public parts
doesn't allow to test pulling with several signing engines used.
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
Skip public keys verification if key is marked as invalid key.
Allow to redefine system-wide directories for ed25519 verification.
Minor bugfixes.
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
Add function `_load_public_keys()` to pre-load public keys according
remote's configuration. If no keys configured for remote, then use
system-wide configuration.
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
Allow to add public and secret key for ed25519 module as based64 string.
This allows to use common API for pulling and builtins without knowledge
of used signature algorithm.
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
Removed from public `ostree_sign_detached_metadata_append` function.
Renamed `metadata_verify` into `data_verify` to fit to real
functionality.
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
Return `const char *` instead of copy of the string -- this allow to
avoid unneeded copying and memory leaks in some constructions.
Minor code cleanup and optimisations.
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
If not provided key of file name with keys for remote, then try to use
system defaults:
- /etc/ostree/trusted.ed25519
- /etc/ostree/trusted.ed25519.d/*
- /usr/share/ostree/trusted.ed25519
- /usr/share/ostree/trusted.ed25519.d/*
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
If `verification-key` is set for remote it is used as a public key for
checking the commit pulled from that remote.
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
API changes:
- added function `ostree_sign_add_pk()` for multiple public keys using.
- `ostree_sign_set_pk()` now substitutes all previously added keys.
- added function `ostree_sign_load_pk()` allowed to load keys from file.
- `ostree_sign_ed25519_load_pk()` able to load the raw keys list from file.
- use base64 encoded public and private ed25519 keys for CLI and keys file.
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
Added the initial version of signing interface allowing to allowing to
sign and verify commits.
Implemented initial signing modules:
- dummy -- simple module allowing to sign/verify with ASCII string
- ed25519 -- module allowing to sign/verify commit with ed25519
(EdDSA) signature scheme provided by libsodium library.
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
The [dev-overlay](332c6ab3b9/src/cmd-dev-overlay)
script shipped in coreos-assembler mostly exists to deal
with the nontrivial logic around SELinux policy. Let's make
the use case of "commit some binaries overlaying a base tree, using
the base's selinux policy" just require a magical
`--selinux-policy-from-base` argument to `ostree commit`.
A new C API was added to implement this in the case of `--tree=ref`;
when the base directory is already checked out, we can just reuse
the existing logic that `--selinux-policy` was using.
Requires: https://github.com/ostreedev/ostree/pull/2039
Add G_IO_ERROR_PARTIAL_INPUT to the list of error codes caused by
transient networking errors which lead us to retry the request. When
attempting to install the spotify flatpak you often get the error
message "Connection terminated unexpectedly" and the download of the deb
file fails. In this case, libsoup is setting G_IO_ERROR_PARTIAL_INPUT
and sometimes a subsequent download attempt is successful, so we should
treat it as transient.
Ideally we would behave as wget does in this case and retry the download
picking up where we left off in the file rather than starting over, but
that would require changes to libsoup I think.
Sadly this patch does not fix the flatpak installation of spotify in the
face of such errors, because flatpak doesn't use libostree to download
extra data, but presumably it's possible we could encounter such an
error pulling from an ostree repo, so the patch is still correct.
These had been added assuming 2019.7 would be the next version, but now
it's 2020 and there's been a release. In the case of
`OstreeCommitSizesEntry`, I'd forgotten to move it forward from 2019.5
to 2019.7 in the time between when I started working on the feature and
it landed.
For repo structure directories like `objects`, `refs`, etc... we should
be more permissive and let the system's `umask` narrow down the
permission bits as wanted.
This came up in a context where we want to be able to have read/write
access on an OSTree repo on NFS from two separate OpenShift apps by
using supplemental groups[1] so we don't require SCCs for running as the
same UID (supplemental groups are part of the default restricted SCC).
[1] https://docs.openshift.com/container-platform/3.11/install_config/persistent_storage/persistent_storage_nfs.html#nfs-supplemental-groups
For some reason I haven't fully debugged (probably a recent
kernel change), in the case where the immutable bit isn't set,
trying to call `EXT2_IOC_SETFLAGS` without it set returns `EINVAL`.
Let's avoid calling the `ioctl()` if we don't have anything to do.
This fixes a slew of `make check` failures here in my toolbox
environment.
(kernel is `5.5.0-0.rc6.git0.1.fc32.x86_64` with `xfs`)
Using fs-verity is natural for OSTree because it's file-based,
as opposed to block based (like dm-verity). This only covers
files - not symlinks or directories. And we clearly need to
have integrity for the deployment directories at least.
Also, what we likely need is an API that supports signing files
as they're committed.
So making this truly secure would need a lot more work. Nevertheless,
I think it's time to start experimenting with it. Among other things,
it does *finally* add an API that makes files immutable, which will
help against some accidental damage.
This is basic enablement work that is being driven by
Fedora CoreOS; see also https://github.com/coreos/coreos-assembler/pull/876
Currently `ostree_gpg_verify_result_require_valid_signature` always
returns an error that the key used for the signature is missing from the
keyring. However, all that's been determined is that there are no valid
signatures. The error could also be from an expired signature, an
expired key, a revoked key or an invalid signature.
Provide values for these missing errors and return them from
`ostree_gpg_verify_result_require_valid_signature`. The description of
each result is appended to the error message, but since the result can
contain more than one signature but only a single error can be returned,
the status of the last signature is used for the error code. See the
comment for rationale.
Related: flatpak/flatpak#1450
This function parses the object listing in the `ostree.sizes` metadata
and returns an array of `OstreeCommitSizesEntry` structures.
Unfortunately, for reasons I don't understand, the linker wants to
resolve `_ostree_read_varuint64` from `ostree-core.c` even though it's
not used by `test-checksum.c` at all.
Append a byte encoding the OSTree object type for each object in the
metadata. This allows the commit metadata to be fetched and then for the
program to see which objects it already has for an accurate calculation
of which objects need to be downloaded.
This slightly breaks the `ostree.sizes` `ay` metadata entries. However,
it's unlikely anyone was asserting the length of the entries since the
array currently ends in 2 variable length integers. As far as I know,
the only users of the sizes metadata are the ostree test suite and
Endless' eos-updater[1]. The former is updated here and the latter
already expects this format.
1. https://github.com/endlessm/eos-updater/
If the object was already in the repo then the sizes metadata entry was
skipped. Move the sizes entry creation after the data has been computed
but before the early return for an existing object.
The object sizes hash table was only being cleared when the repo was
finalized. That means that performing multiple commits while the repo
was open would reuse all the object sizes metadata for each commit.
Clear the hash table when the sizes metadata is setup and when it's
added to a commit. This still does not fix the issue all the way since
it does nothing to prevent the program from constructing multiple
commits simultaneously. To handle that, the object sizes hash table
should be attached to the MutableTree since that has the commit state.
However, the MutableTree is gone when the commit is actually created.
The hash table would have to be transferred to the root file when
writing the MutableTree. That would be an awkward addition to
OstreeRepoFile, though. Add a FIXME to capture that.
We want to support extending the read-only state to cover `/sysroot`
and `/boot`, since conceptually all of the data there should only
be written via libostree. Or at least for `/boot` should *mostly*
just be written by ostree.
This change needs to be opt-in though to avoid breaking anyone.
Add a `sysroot/readonly` key to the repository config which instructs
`ostree-remount.service` to ensure `/sysroot` is read-only. This
requires a bit of a dance because `/sysroot` is actually the same
filesystem as `/`; so we make `/etc` a writable bind mount in this case.
We also need to handle `/var` in the "OSTree default" case of a bind
mount; the systemd generator now looks at the writability state of
`/sysroot` and uses that to determine whether it should have the
`var.mount` unit happen before or after `ostree-remount.service.`
Also add an API to instruct the libostree shared library
that the caller has created a new mount namespace. This way
we can freely remount read-write.
This approach extends upon in a much better way previous work
we did to support remounting `/boot` read-write.
Closes: https://github.com/ostreedev/ostree/issues/1265
This allows copying the state from one OstreeAsyncProgress object to
another, atomically, without invoking the callback. This is needed in
libflatpak, in order to chain OstreeAsyncProgress objects so that you
can still receive progress updates when iterating a different
GMainContext than the one that the OstreeAsyncProgress object was
created under.
See https://github.com/flatpak/flatpak/pull/3211 for the application of
this API.
Define an `OstreeKernelArgsEntry` structure, which holds
both the key and the value. The kargs order array stores
entries for each key/value pair, instead of just the keys.
The hash table is used to locate entries, by storing
entries in a pointer array for each key. The same public
interface is preserved, while maintaining ordering
information of each key/value pair when
appending/replacing/deleting kargs.
Fixes: #1859
To allow for FIPS mode, we need to also install the HMAC file from
`/usr/lib/modules` to `/boot` alongside the kernel image where the
`fips` dracut module will find it. For details, see:
https://github.com/coreos/fedora-coreos-tracker/issues/302
Note I didn't include the file in the boot checksum since it's itself a
checksum of the kernel, so we don't really gain much here other than
potentially causing an unnecessary bootcsum bump.
I was hitting `SIGSEGV` when running `cosa build` and narrowed it down
to #1954. What's happening here is that because we're using the default
context, when we unref it in the out path, it may not actually destroy
the `GSource` if it (the context) is still ref'ed elsewhere. So then,
we'd still get events from it if subsequent operations iterated the
context.
This patch is mostly a revert of #1954, except that we still keep a ref
on the `GSource`. That way it is always safe to destroy it afterwards.
(And I've also added a comment to explain this better.)
We're creating the timer source and then passing ownership to the
context, but because we didn't free the pointer, we would still call
`g_source_destroy` in the exit path. We'd do this right after doing
`unref` on the context too, which would have already destroyed and
unref'ed the source.
Drop that and just restrict the scope of that variable down to make
things more obvious.
Just noticed this after reviewing #1953.
In glib 2.62 this has been changed to emitting a warning. Use G_STRFUNC
instead, which has been available for a long time and is already used in
other places in ostree.
zipl is a bit special in that it parses the BLS config files
directly *but* we need to run the command to update the "boot block".
Hence, we're not generating a separate config file like the other
backends. Instead, extend the bootloader interface with a `post_bls_sync`
method that is run in the same place we swap the `boot/loader` symlink.
We write a "stamp file" in `/boot` that says we need to run this command.
The reason we use stamp file is to prevent the case where the system is
interrupted after BLS file is updated, but before zipl is triggered,
then zipl boot records are not updated.
This opens the door to making things eventually-consistent/reconcilable
by later adding a systemd unit to run `zipl` if we're interrupted via
a systemd unit - I think we should eventually take this approach
everywhere rather than requiring `/boot/loader` to be a symlink.
Author: Colin Walters <walters@verbum.org>
Tested-by: Tuan Hoang <tmhoang@linux.ibm.com>
Co-Authored-By: Tuan Hoang <tmhoang@linux.ibm.com>
More "scan-build doesn't understand GError and our out-param conventions"
AKA "these errors would be impossible with Rust's sum type Result<> approach".
Got this error when trying to rebase libostree in RHEL:
```
Error: CLANG_WARNING: [#def1]
libostree-2019.2/src/libostree/ostree-repo-checkout.c:375:21: warning: Access to field 'disable_xattrs' results in a dereference of a null pointer (loaded from variable 'repo')
```
I think what's happening is it sees us effectively testing
`if (repo == NULL)` via the `while (current_repo)`. Let's
tell it we're sure it's non-null right after the loop.
Tiny release. Just want to get out the important bugfixes instead of
backporting patches (notably the gpg-agent stuff and
`ostree-finalize-staged.service` ordering).
Closes: #1927
Approved by: cgwalters
After the corruption has been fixed with "ostree fsck -a --delete", a
second run of the "ostree fsck" command will print X partial commits
not verified and exit with a zero.
The zero exit code makes it hard to detect if a repair operation needs
to be run. When ever fsck creates a partial commit it should add a
reason for the partial commit to the state file found in
state/<hash>.commitpartial. This will allow a future execution of the
fsck to still return an error indicating that the repository is still
in the damaged state, awaiting repair.
Additional reason codes could be added in the future for why a partial
commit exists.
Text from: https://github.com/ostreedev/ostree/pull/1880
====
cgwalters commented:
To restate, the core issue is that it's valid to have partial commits
for reasons other than fsck pruned bad objects, and libostree doesn't
have a way to distinguish.
Another option perhaps is to write e.g. fsck-partial into the
statefile state/<hash>.commitpartial which would mean "partial, and
expected to exist but was pruned by fsck" and fsck would continue to
error out until the commit was re-pulled. Right now the partial stamp
file is empty, so it'd be fully compatible to write a rationale into
it.
====
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Closes: #1910
Approved by: cgwalters
If there are different deployments for the same commit version, the BLS
snippets will have the same title fields (but different version fields):
$ grep title *
ostree-1-testos.conf:title TestOS 42 20190902.0 (ostree)
ostree-2-testos.conf:title TestOS 42 20190902.0 (ostree)
ostree-3-testos.conf:title TestOS 42 20190902.0 (ostree)
But bootloaders could expect the title field to be unique for BLS files.
For example, the zipl bootloader used in the s390x architecture uses the
field to name the boot sections that are created from the BLS snippets.
So two BLS snippets having the same title would lead to zipl failing to
create the IPL boot sections because they would have duplicated names:
$ zipl
Using config file '/etc/zipl.conf'
Using BLS config file '/boot/loader/entries/ostree-3-testos.conf'
Using BLS config file '/boot/loader/entries/ostree-2-testos.conf'
Using BLS config file '/boot/loader/entries/ostree-1-testos.conf'
Error: Config file '/etc/zipl.conf': Line 0: section name 'TestOS 42 20190902.0 (ostree)' already specified
Avoid this by always including the deployment index along with the commit
version in the title field, so this will be unique even if there are BLS
files for deployments that use the same commit version:
$ grep title *
ostree-1-testos.conf:title TestOS 42 20190902.0 (ostree:2)
ostree-2-testos.conf:title TestOS 42 20190902.0 (ostree:1)
ostree-3-testos.conf:title TestOS 42 20190902.0 (ostree:0)
$ zipl
Using config file '/etc/zipl.conf'
Using BLS config file '/boot/loader/entries/ostree-3-testos.conf'
Using BLS config file '/boot/loader/entries/ostree-2-testos.conf'
Using BLS config file '/boot/loader/entries/ostree-1-testos.conf'
Building bootmap in '/boot'
Building menu 'zipl-automatic-menu'
Adding #1: IPL section 'TestOS 42 20190902.0 (ostree:0)' (default)
Adding #2: IPL section 'TestOS 42 20190902.0 (ostree:1)'
Adding #3: IPL section 'TestOS 42 20190902.0 (ostree:2)'
Preparing boot device: dasda (0120).
Done.
Closes: #1911
Approved by: cgwalters
Currently the BLS fragments fields write is non-determinisitc. The order
of the fields will depend on how the iterator of the options GHashTable
iterates over the key/value pairs.
But some bootloaders expect the fields to be written in a certain order.
For example the zipl bootloader (used in the s390x architecture) fails to
parse BLS files if the first field is not the 'title' field, since that's
used to name the zipl boot sections that are created from the BLS files.
Write the fields in a deterministic order, following what is used in the
example file of the BootLoaderspec document:
https://systemd.io/BOOT_LOADER_SPECIFICATION
Related: https://github.com/ostreedev/ostree/issues/1888Closes: #1904
Approved by: cgwalters
OSTree has some logic to preserve comment lines in the BLS fragments, but
the BLS fragments are always created on new deployments so the comments
are never carried.
Also, OSTree never writes BLS fragments with comments so these will only
be present in BLS files that were modified outside of OSTree. Something
that should be avoided in general.
Finally, there is a bug in the logic that causes BLS files to have lines
with only a newline character.
The ostree_bootconfig_parser_parse_at() function reads the bootconfig file
using glnx_fd_readall_utf8() but this function NUL terminates the returned
string with the file contents.
So when the string is later split using '\n' as delimiter, the last token
is set to '\0' and a wrong GVariant will be added to the lines GPtrArray
in the OstreeBootconfigParser struct.
This will lead to bootconfig files that contains lines with only a newline
character, since the key in the GVariant would be set to NUL and won't be
present in the options GHashTable of the OstreeBootconfigParser struct.
So let's just remove that logic since is never used and makes BLS files to
have wrong empty lines.
Before this patch:
$ tail -n 4 /boot/loader/entries/ostree-1-testos.conf | hexdump -C
00000000 74 69 74 6c 65 20 54 65 73 74 4f 53 20 34 32 20 |title TestOS 42 |
00000010 32 30 31 39 30 38 32 34 2e 30 20 28 6f 73 74 72 |20190824.0 (ostr|
00000020 65 65 29 0a 0a 0a 0a |ee)....|
00000027
After this patch:
$ tail -n 4 /boot/loader/entries/ostree-1-testos.conf | hexdump -C
00000000 76 65 72 73 69 6f 6e 20 31 0a 6f 70 74 69 6f 6e |version 1.option|
00000010 73 20 72 6f 6f 74 3d 4c 41 42 45 4c 3d 4d 4f 4f |s root=LABEL=MOO|
00000020 20 71 75 69 65 74 20 6f 73 74 72 65 65 3d 2f 6f | quiet ostree=/o|
00000030 73 74 72 65 65 2f 62 6f 6f 74 2e 31 2f 74 65 73 |stree/boot.1/tes|
00000040 74 6f 73 2f 61 65 34 36 34 39 36 38 30 64 33 65 |tos/ae4649680d3e|
00000050 38 33 62 32 34 65 34 37 66 38 64 66 31 30 38 31 |83b24e47f8df1081|
00000060 38 62 66 36 39 38 39 64 36 34 37 61 62 32 38 38 |8bf6989d647ab288|
00000070 64 31 63 30 39 38 30 36 65 34 61 33 36 61 34 65 |d1c09806e4a36a4e|
00000080 62 62 66 36 2f 30 0a 6c 69 6e 75 78 20 2f 6f 73 |bbf6/0.linux /os|
00000090 74 72 65 65 2f 74 65 73 74 6f 73 2d 61 65 34 36 |tree/testos-ae46|
000000a0 34 39 36 38 30 64 33 65 38 33 62 32 34 65 34 37 |49680d3e83b24e47|
000000b0 66 38 64 66 31 30 38 31 38 62 66 36 39 38 39 64 |f8df10818bf6989d|
000000c0 36 34 37 61 62 32 38 38 64 31 63 30 39 38 30 36 |647ab288d1c09806|
000000d0 65 34 61 33 36 61 34 65 62 62 66 36 2f 76 6d 6c |e4a36a4ebbf6/vml|
000000e0 69 6e 75 7a 2d 33 2e 36 2e 30 0a 74 69 74 6c 65 |inuz-3.6.0.title|
000000f0 20 54 65 73 74 4f 53 20 34 32 20 32 30 31 39 30 | TestOS 42 20190|
00000100 38 32 34 2e 30 20 28 6f 73 74 72 65 65 29 0a |824.0 (ostree).|
0000010f
Closes: #1904
Approved by: cgwalters
I've seen people confused by this error in the case where
`/boot` isn't mounted or the BLS fragments were deleted, etc.
If you understand ostree deeply it's clear but, let's do
better here and a direct error message for the case where
we can't find `/boot/loader` which is the majority of these.
The other case could happen if e.g. just the BLS fragment
for the booted deployment was deleted; let's reword that
one a bit too.
Closes: #1905
Approved by: rfairley
When running under qemu, unimplemented ioctls such as FIFREEZE
return ENOSYS, and this causes the deployment to fail.
Catch this and handle it like EOPNOTSUPP.
I'm not sure if qemu's behaviour is fully correct here (or if it should
return EOPNOTSUPP) but it's trivial to handle regardless.
Closes: #1901
Approved by: cgwalters
Add dummy stubs for GPG public functions to be compiled instead of
original code in case if support of GPG is disabled.
Need that to keep API backward compatibility.
Based on original code from file `ostree-gpg-verify-result.c`.
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
Closes: #1889
Approved by: cgwalters
Some gpg-named functions/variables should be used for any signature
system, so remove "gpg_" prefix from them to avoid confusion.
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
Closes: #1889
Approved by: cgwalters
Do not build the code related to GPG sign and verification if
GPGME support is disabled.
Public functions return error 'G_IO_ERROR_NOT_SUPPORTED' in case if
gpg-related check is rquested.
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
Closes: #1889
Approved by: cgwalters
There's a valid use case for enabling the timestamp downgrade check
while still also using override commits.
We'll make use of this in Fedora CoreOS, where the agent specifies the
exact commit to upgrade to, while still enforcing that it be newer.
Closes: #1891
Approved by: cgwalters
This way projects can dispatch at run-time based on ostree's
build time options, e.g. detect the availability of GPG.
Closes: #1890
Approved by: jlebon
Move the OstreeKernelArgs autoptr cleanup definition to
ostree-autocleanups.h, which will only expose the definitions when
building ostree or if glib is new enough. The include of
ostree-kernel-args.h needs to be moved before ostree-autocleanups.h in
ostree.h so that the OstreeKernelArgs type is declared when the autoptr
cleanup is defined. All the places it's used already pull in libglnx.h
first so that the compat macros are picked up if glib it too old during
the ostree build.
Closes: #1892
Approved by: jlebon
When a temporary directory is used for GPG operations, it's pretty clear
that the running agent will be useless after the directory is deleted.
Call the new `ot_gpgme_kill_agent ()` helper to kill gpg-agent rather
than leaving them it hanging around forever.
As it turns out, gnupg does have code to make gpg-agent automatically
exit when the homedir is removed (https://dev.gnupg.org/T2756), but
that's only available on gnupg 2.2 or newer. Possibly this code can be
dropped later when that's more widely deployed or users/distros have
been advised to backport the necessary changes.
Closes: #1799
Approved by: cgwalters
Introduce a new signature attribute for the key expiration timestamp and
display it when the key has a non-zero expiration time. Without this,
the error shown is `BAD signature`, which isn't correct.
Closes: #1872
Approved by: cgwalters
This change makes public the current kargs API in src/libostree/ostree-kernel-args.c
and adds documentations.
Upstreams the new kargs API from rpm-ostree/src/libpriv/rpmostree-kargs-process.c
Merges libostree_kernel_args_la_SOURCES to libostree_1_la_SOURCES in Makefile-libostree.am
Upstreams tests/check/test-kargs.c from rpm-ostree.
Closes: #1833Closes: #1869
Approved by: jlebon
Similar to ostree_repo_write_archive_to_mtree(), but takes
a file descriptor to read the archive from instead of mandating
a file path.
Usefull for importing archives into an OSTree repo over a socket
or from standard input in command line tools.
Closes: #1862
Approved by: jlebon
Use GIOErrorEnum as the return value for
_ostree_fetcher_http_status_code_to_io_error(), to avoid an
implicit cast from GIOError.
Closes: #1857
Approved by: cgwalters
Teach `ostree-finalize-staged.service` to check for a file in `/run` to
determine if it should do the finalization. This will be used in
RPM-OSTree, where we want to be able to separate out "preparing updates"
from "making update the default" for more fine-grained control. See:
https://github.com/projectatomic/rpm-ostree/issues/1748Closes: #1841
Approved by: cgwalters
This can happen if a deployment was staged and later cleaned up. Though
just as a helper when debugging issues, let's explicitly mention that
case.
Closes: #1841
Approved by: cgwalters
Rather than wrapping each instance of `sd_journal_*` with
`HAVE_SYSTEMD`, let's just add some convenience macros that are just
no-op if we're not compiling with systemd.
Closes: #1841
Approved by: cgwalters
Generate a grub2 config using the pending deployment, if a grub2
bootloader is detected in the sysroot. Allows grub2-mkconfig
to run if there are no previous deployments.
Fixes: #1774Closes: #1831
Approved by: jlebon
Otherwise, we'll be subject to whatever `umask` is currently. Normally,
processes should respect `umask` when creating files and directories,
but specifically for `ostree admin unlock` (or `rpm-ostree usroverlay`),
this poses a problem since e.g. a `/usr` with mode 0700 will break any
daemon that doesn't run as root and needs to read files under `/usr`,
such as polkitd.
This patch just does a `chmod()` after the `mkdir()`. An alternative
would be to do `umask(0000)` after forking into the child process
that'll call `mount()`, but that'd require also moving the `mkdir()`
calls into there, making for a more intrusive patch.
Closes: #1843
Approved by: cgwalters
Currently for a "normal" refspec you can choose to use
ostree_repo_resolve_rev_ext() instead of ostree_repo_resolve_rev() if
you only want to look at local refs (in refs/heads/) not remote ones.
This commit provides the analogous functionality for
ostree_repo_resolve_collection_ref() by adding a flag
OSTREE_REPO_RESOLVE_REV_EXT_LOCAL_ONLY and implementing it. This
will be used by Flatpak.
Closes: #1825
Approved by: jlebon
Currently the flag OSTREE_REPO_LIST_REFS_EXT_EXCLUDE_REMOTES for
ostree_repo_list_collection_refs() means that refs in refs/remotes/
should be excluded but refs in refs/mirrors/ should still be checked, in
addition to refs/heads/ which is always checked. However in some
situations you want to exclude both remote and mirrored refs and only
check local "owned" ones. So this
commit adds a new flag OSTREE_REPO_LIST_REFS_EXT_EXCLUDE_MIRRORS which
lets you exclude refs/mirrors/ from the listing.
This way we can avoid breaking API but still allow the listing of local
collection-refs.
The impetus for this change is that I'm changing Flatpak to make more
use of refs/mirrors, and we need a way to specify that a collection-ref
is local when using ostree_repo_resolve_collection_ref() in, for
example, the implementation of the repo command. The subsequent commit
will make the changes needed there.
Closes: #1825
Approved by: jlebon
My last commit "lib/repo-refs: Resolve collection-refs in-memory and in
parent repos" changed ostree_repo_resolve_collection_ref() to check the
in-memory set of refs *after* failing to find the ref on disk but that's
not what we want. We want to use the in-memory set of refs first,
because those are the most up to date commits, and then fall back to the
on-disk repo and finally fall back to checking any parent repo. This
commit makes such a change to the order of operations, which is
consistent with how ostree_repo_resolve_rev() works.
Aside from this change being logical, it also fixes some unit test
failures on an unmerged branch of flatpak:
https://github.com/flatpak/flatpak/pull/2705
Also, tweak the comments here.
Closes: #1825
Approved by: jlebon
On at least one user's computer, g_getenv("http_proxy") returns the
empty string, so check for that and treat it as no proxy rather than
printing a warning.
See https://github.com/flatpak/flatpak/issues/2790Closes: #1835
Approved by: cgwalters
Currently the P2P code requires you to trust every remote you have
configured to the same extent, because a remote controlled by a
malicious actor can serve updates to refs (such as Flatpak apps)
installed from other remotes.[1] The way this attack would play out is
that the malicious remote would deploy the same collection ID as the
victim remote, and would then be able to serve updates for it.
One possible remedy would be to make it an error to configure remotes
such that two have the same collection ID but differing GPG keys. I
attempted to do that in Flatpak[2] but it proved difficult because it is
valid to configure two remotes with the same collection ID, and they may
then each want to update their keyrings which wouldn't happen
atomically.
Another potential solution I've considered is to add a `trusted-remotes`
option to ostree_repo_find_remotes_async() which would dictate which
keyring to use when pulling each ref. However the
ostree_repo_finder_resolve_async() API would still remain vulnerable,
and changing that would require rewriting a large chunk of libostree's
P2P support.
So this commit represents a third attempt at mitigating this security
hole, namely to have the client specify which remote to use for GPG
verification at pull time. This way the pull will fail if the commits
are signed with anything other than the keys we actually trust to serve
updates.
This is implemented as an option "ref-keyring-map" for
ostree_repo_pull_from_remotes_async() and
ostree_repo_pull_with_options() which dictates the remote to be used for
GPG verification of each collection-ref. I think specifying a keyring
remote for each ref is better than specifying a remote for each
OstreeRepoFinderResult, because there are some edge cases where a result
could serve updates to refs which were installed from more than one
remote.
The PR to make Flatpak use this new option is here[3].
[1] https://github.com/flatpak/flatpak/issues/1447
[2] https://github.com/flatpak/flatpak/pull/2601
[3] https://github.com/flatpak/flatpak/pull/2705Closes: #1810
Approved by: cgwalters
When writing a delta to a file this may not always be recorded
in the filename, and it's useful data.
Ref: https://mail.gnome.org/archives/ostree-list/2019-February/msg00000.html
This also required teaching `show` to accept a file path.
Note...for some reason `test-deltas.sh` breaks when run from
a tty - we get `SIGTTIN` which implies something is reading from
the tty but it wasn't obvious to me what.
Closes: #1823
Approved by: jlebon
In Silverblue right now, the boot menu title looks like this:
Fedora 29.20190301.0 (Workstation Edition) 29.20190301.0 (ostree)
This is because RPM-OSTree's `mutate-os-release` feature is enabled,
which injects the OSTree version string directly into `VERSION` and
`PRETTY_NAME`. So appending the version string again is a bit redundant.
Let's just do a simple substring check here before adding the version to
the title.
Closes: #1829
Approved by: cgwalters
The sysroot.bootloader key configures the bootloader
that OSTree uses when deploying a sysroot. Having this key
allows specifying behavior not to use the default bootloader
backend code, which is preferable when creating a first
deployment from the sysroot (#1774).
As of now, the key can take the values "auto" or "none". If
the key is not given, the value defaults to "auto".
"auto" causes _ostree_sysroot_query_bootloader() to be used
when writing a new deployment, which is the original behavior
that dynamically detects which bootloader to use.
"none" avoids querying the bootloader dynamically. The BLS
config fragments are still written to
sysroot/boot/loader/entries for use by higher-level software.
More values can be supported in future to specify a single
bootloader, different behavior for the bootloader code, or
a list of bootloaders to try.
Resolves: #1774Closes: #1814
Approved by: jlebon
Rename ot_keyfile_get_string_as_list() to
ot_keyfile_get_string_list_with_separator_choice() which expresses
more clearly why the function is needed. Also shorten the
function comment.
Closes: #1814
Approved by: jlebon
Currently the behavior of ostree_repo_resolve_rev() is that it tries to
resolve a ref to a commit by checking the refs/ directories, but also by
checking for in-memory ref-checksum pairs which are part of an
in-progress transaction and also by checking the parent repo if one
exists. Currently ostree_repo_resolve_collection_ref() only checks the
refs/ directories, so this commit makes its behavior analagous since it
is the analagous API which supports collection-refs.
The impetus for this was that currently Flatpak uses
ostree_repo_resolve_rev() to load a commit after doing a P2P pull in
flatpak_dir_do_resolve_p2p_refs(), but that assumes the ref came from
the same remote that originally provided it, which might not be the case
if more than one remote has the same collection ID configured. And
changing Flatpak to use ostree_repo_resolve_collection_ref() doesn't
work without this patch.
Closes: #1821
Approved by: pwithnall
Add the OSTREE_REPO_REMOTE_CHANGE_REPLACE operation to the
OstreeRepoRemoteChange enum. This operation will add a remote or replace
an existing one. It respects the location of the remote configuration
file when replacing and the remotes config dir settings when adding a
new remote.
Closes: #1166
Approved by: cgwalters
We have a `http2=[0|1]` remote config option; let's have the
`--disable-http2` build option define the default for that. This way
it's easy to still enable http2 for testing even if
we have it disabled by default.
Closes: #1798
Approved by: jlebon
Similar as available for u-boot (ce2995e1dc)
and syslinux (c5112c25e4), enable parsing
and writing devicetree filename into grub.cfg.
This is required by arm64-based devices running edk2 instead of u-boot
as the main bootloader (e.g. 96boards HiKey and HiKey960).
Signed-off-by: Ricardo Salveti <ricardo@foundries.io>
Closes: #1790
Approved by: cgwalters
We want a case where we can disable the min-free-space check. Initially,
it felt like to add a OSTREE_REPO_PULL_FLAGS_DISABLE_FREE_SPACE_CHECK but
the problem is prepare_transaction() does not have a OstreeRepoPullFlags
parameter which we can parse right here. On top of it, prepare_transaction()
enforces min-free-space check and won't let the transaction proceed if
the check failed.
This is pretty bad in conjunction with "inherit-transaction" as what
Flatpak uses. There is no way to disable this check unless we remove
it altogether from prepare_transaction.
This issue came out to light when flatpak wasn't able to write metadata
after fetching from remote:
[uajain@localhost ~]$ flatpak remote-info flathub org.kde.Platform//5.9
error: min-free-space-size 500MB would be exceeded
Metadata objects helps in housekeeping and restricting them means
restricting crucial UX (like search, new updates) functionalities
in clients like gnome-software. The error banners originated from
these issues are also abrupt and not much helpful to the user. This
is the specific instance of the issue this patches tries to address.
See https://github.com/flatpak/flatpak/issues/2139 for discussion.
Closes: #1779
Approved by: mwleeds
The way _ostree_repo_import_object() is written, a hardlink copy is only
attempted if the source repo is trusted, so update the docs for
ostree_repo_import_object_from_with_trust() to reflect that.
Closes: #1777
Approved by: cgwalters
This allows specifying gpgpath as list of
paths that can point to a file or a directory. If a directory path
is given, paths to all regular files in the directory are added
to the remote as gpg ascii keys. If the path is not a directory,
the file is directly added (whether regular file, empty - errors
will be reported later when verifying gpg keys e.g. when pulling).
Adding the gpgkeypath property looks like:
ostree --repo=repo remote add --set=gpgpath="/path/key1.asc,/path/keys.d" R1 https://example.com/some/remote/ostree/repoCloses#773Closes: #1773
Approved by: cgwalters
When falling back to copying objects when importing them into a
bare-user repo, we only actually need to transfer over the
`user.ostreemeta` xattr.
This allows the destination repo to be on a separate filesystem that
might not even support `security.selinux`. (I hit this while importing
over virtio-9p).
Closes: #1771
Approved by: cgwalters
I found this useful while hacking on rpm-ostree but I think it might be
useful enough to upstream. This stat is really helpful for validating
that a pipeline is hitting the devino cache sweet spot.
Closes: #1772
Approved by: cgwalters
if a file ".wh..wh..opq" is present in a directory, delete anything
from lower layers that is already in that directory.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Closes: #1486
Approved by: cgwalters
This renames a config key to make its semantics more obvious. Despite
what the commit message says, it only applies when a set of repo finders
is not specified (either on the command line or in a library API call).
This also renames the corresponding ostree_repo_get function. We can do
this since it hasn't been released yet.
Closes: #1763
Approved by: pwithnall
Rather than manually starting the `ostree-finalize-staged.service` unit,
we can leverage systemd's path units for this. It fits quite nicely too,
given that we already have a path we drop iif we have a staged
deployment.
To give some time for the preset to make it to systems, we don't yet
drop the explicit call to `systemctl start`. Though we do make it
conditional based on a DEBUG env var so that we can actually test it in
CI for now. Once we're sure this has propagated, we can drop the
`systemctl start` path and the env var together.
Closes: #1740
Approved by: cgwalters
This commit disables searching on the local network for refs, unless
explicitly requested by the user either by changing the value of the
"core.repo-finders" config option, or by passing an OstreeRepoFinderAvahi to
ostree_repo_find_remotes_async() / ostree_repo_finder_resolve_async(),
or by specifying "lan" in the --finders option of the find-remotes
command.
The primary reason for this is that ostree_repo_find_remotes_async()
takes about 40% longer to complete with the LAN finder enabled, and that
API is used widely (e.g. in every flatpak operation). It's also probable
that some users don't want ostree doing potentially unexpected traffic
on the local network, even though everything pulled from a peer is GPG
verified.
Flathub will soon deploy collection IDs to everyone[1] so these code
paths will soon see a lot more use and that's why this change is being
made now.
Endless is the only potential user of the LAN updates feature, and we
can revert this patch on our fork of ostree. For it to be used outside
Endless OS we will need to upstream eos-updater-avahi and
eos-update-server into ostree.
[1] https://github.com/flathub/flathub/issues/676Closes: #1758
Approved by: cgwalters
Currently libostree essentially has two modes when it's pulling refs:
the "legacy" code paths pull only from the Internet, and the code paths
that are aware of collection IDs try to pull from the Internet, the
local network, and mounted filesystems (such as USB drives). The problem
is that while we eventually want to migrate everyone to using collection
IDs, we don't want to force checking LAN and USB sources if the user
just wants to pull from the Internet, since the LAN/USB code paths can
have privacy[1], security[2], and performance[3] implications.
So this commit implements a new repo config option called "repo-finders"
which can be configured to, for example, "config;lan;mount;" to check
all three sources or "config;mount;" to disable searching the LAN. The
set of values mirror those used for the --finders option of the
find-remotes command. This configuration affects pulls in three places:
1. the ostree_repo_find_remotes_async() API, regardless of whether or
not the user of the API provided a list of OstreeRepoFinders
2. the ostree_repo_finder_resolve_async() /
ostree_repo_finder_resolve_all_async() API
3. the find-remotes command
This feature is especially important right now since we soon want to
have Flathub publish a metadata key which will have Flatpak clients
update the remote config to add a collection ID.[4]
This effectively fixes https://github.com/flatpak/flatpak/issues/1863
but I'll patch Flatpak too, so it doesn't pass finders to libostree only
to then have them be removed.
[1] https://github.com/flatpak/flatpak/issues/1863#issuecomment-404128824
[2] https://github.com/ostreedev/ostree/issues/1527
[3] Based on how long the "ostree find-remotes" command takes to
complete, having the LAN finder enabled slows down that step of the
pull process by about 40%. See also
https://github.com/flatpak/flatpak/issues/1862
[4] https://github.com/flathub/flathub/issues/676Closes: #1758
Approved by: cgwalters
The idea is that if the process is running as root, it can change
ownership of newly written files to match the owner of the repo.
Unfortunately, it currently applies in the other direction, too - a
non-root user writing to a root owned repository. If the repo is
writable by the user but owned by root, it can still create files and
directories there, but it can't change ownership of them.
This feature comes from
https://bugzilla.gnome.org/show_bug.cgi?id=738954. As it turns out, this
feature was never completed. It only works on content objects and not
metadata objects, refs, deltas, summaries, etc. Rather than try to fix
all of those, remove the feature until someone has interest in
completing it.
Closes: #1754
Approved by: cgwalters
Actually testing the patch to add `--force-copy-zerosized` to
rpm-ostree tripped over the fact that it uses `--union-identical`,
and we just hit an assertion failure with that combination.
Fix this by copying over the logic we have for the hardlink case.
Closes: #1753
Approved by: jlebon
In rpm-ostree we've hit a few cases where hardlinking zero-sized
files causes us problems. The most prominent is lock files in
`/usr/etc`, such as `/usr/etc/selinux/semanage.LOCK`. If there
are two zero-sized lock files to grab, but they're hardlinked,
then locking will fail.
Another case here is if one is using ostree inside a container
and don't have access to FUSE (i.e. `rofiles-fuse`), then the
ostree hardlinking can cause files that aren't ordinarily hardlinked
to become so, and mutation of one mutates all. An example where
this is concerning is Python `__init__.py` files.
Now, these lock files should clearly not be in the tree to begin
with, but - we're not gaining a huge amount by hardlinking these
files either, so let's add an option to disable it.
Closes: #1752
Approved by: jlebon
Write to the journal when starting to finalize a staged deployment.
Combined with the "Transaction completed" message we already emit, this
makes it easy later on to determine whether the operation was successful
by inspecting the journal. This will be used by `rpm-ostree status`.
Closes: #1750
Approved by: cgwalters
There are use cases for libostree as a local content store
for content derived or delivered via other mechanisms (e.g. OCI
images, RPMs, etc.). rpm-ostree today imports RPMs into OSTree
branches, and puts the RPM header value as commit metadata.
Some of these can be quite large because the header includes
permissions for each file. Similarly, some OCI metadata is large.
Since there's no security issues with this, support committing
such content.
We still by default limit the size of metadata fetches, although
for good measure we make this configurable too via a new
`max-metadata-size` value.
Closes: https://github.com/ostreedev/ostree/issues/1721Closes: #1744
Approved by: jlebon
Currently the locking code checks if the value -1 was set for the config
key "lock-timeout-secs" and if so, a thread trying to acquire a lock
will block indefinitely. Positive values specify how long to attempt to
acquire a lock in a non-blocking way (the attempt is made once every
second). But when the value is read from the config file,
g_ascii_strtoull() is used, which converts it to an unsigned integer.
This commit makes libostree use g_ascii_strtoll() instead, so that it's
possible to set that key to -1 as intended.
Closes: #1737
Approved by: jlebon
Copying the xattrs on metadata objects is wrong in general, we
don't "own" them. Notably this would fail in the situation of
doing a pull from e.g. a `bare-user` source to a destination
that was on a different mount point (so we couldn't hardlink),
and the source had e.g. a `security.selinux` attribute.
Closes: #1734Closes: #1736
Approved by: jlebon
Just include the whole URL that failed if libcurl failed with something
elementary like CURLE_COULDNT_CONNECT or CURLE_COULDNT_RESOLVE_HOST.
Closes: #1731Closes: #1732
Approved by: cgwalters
There's some subtlety to this, we don't handle all cases.
But the 99% cases are using `--sysroot deploy` to create an
initial deployment, and then doing upgrades from inside
a booted deployment.
It was only the latter case that didn't work with a separate `/var`.
Fixing all of them would probably require libostree to learn
how to e.g. look at `/etc/fstab` (or worse, systemd mount units?)
and handle the mounting. I don't think we want to do anything
like that right now, since there are no active drivers for the
use case.
Closes: https://github.com/ostreedev/ostree/issues/1729Closes: #1730
Approved by: akiernan
Earlier, the actual reserved space (in blocks) were calculated inside the
transaction codepath ostree_repo_prepare_transaction(). However, while
reworking on ostree_repo_get_min_free_space_bytes() API, it was realized that
this calculation can be done independently from the transaction's codepaths, hence
enabling the usage for ostree_repo_get_min_free_space_bytes() API irrespective
of whether there is an ongoing transaction or not.
https://github.com/ostreedev/ostree/issues/1720Closes: #1722
Approved by: pwithnall
This commit defines a metadata key that tells clients to update their
remote config to add a collection ID. This functionality is currently
implemented in Flatpak for the key "xa.collection-id", but there are two
good reasons for moving the key to OSTree:
1) Servers such as Flathub shouldn't set xa.collection-id in their
metadata now or in the medium term future, because many users are still
using old versions of Flatpak and OSTree[1] which would hit various
bugs[2][3][4] on the P2P code paths that are enabled by collection IDs.
Defining a new key means that only clients running recent
(as-yet-unreleased) versions of Flatpak and OSTree will pay attention to
it and deploy the collection ID, leaving the users on old versions
unaffected.
2) OSTree is as "invested" in collection IDs as Flatpak, so there's no
reason the key should be defined in Flatpak rather than here. According
to Philip Withnall, the reason the key was put in Flatpak originally was
that at the time there was uncertainty about tying OSTree to collection
IDs.
[1] https://ahayzen.com/direct/flathub.html#downloadsbyflatpakstacked
[2] https://github.com/ostreedev/ostree/commit/e4e6d85ea
[3] https://github.com/flatpak/flatpak/commit/5813639f
[4] https://github.com/flatpak/flatpak/commit/5b21a5b7Closes: #1726
Approved by: pwithnall
There is no API method to remove a file or subdirectory from a MutableTree
besides directly manipulating the GHashTable returned by _get_files or
_get_subdirs. This isn't possible from an introspection binding that transforms
the returned GHashTable, and may also leave the tree checksum in an invalid
state. Introduce a new method so that removing files or subdirectories is
safe, and possible from bindings.
Closes: #1724
Approved by: jlebon
This fixes typos and grammar in the docs for OstreeRepo, and copies the
information about OSTREE_REPO_MODE_BARE_USER_ONLY from ostree-core.h
Closes: #1725
Approved by: jlebon
In the OstreeRepoFinderAvahi implementation,
ostree_avahi_service_build_repo_finder_result() is where the DNS-SD
records are processed and turned into OstreeRepoFinderResult objects.
Each result object is supposed to have a hash table mapping refs to
checksums, so this is accomplished by first adding a placeholder (a ref
mapping to a NULL checksum) for each ref matched by the bloom filter,
and later filling in the checksums using the remote's summary file,
which happens in get_checksums(). The problem is that there's no
guarantee all the checksums will be resolved (non-NULL), so the
ostree_repo_finder_result_new() call then hits an assertion failure in
is_valid_collection_ref_map() leading to a crash (in the case that one
or more refs had NULL checksums).
There are at least two situations where the ref checksum might not be
found in the peer remote's summary file:
1) The bloom filter match was a false positive. This is going to happen
sometimes by design.
2) The peer remote's summary is out of sync with its DNS-SD records.
This shouldn't normally happen but it's still good to be robust to the
possibility; in Endless OS nothing guarantees the atomicity of updating
the summary and DNS-SD records.
This commit changes libostree to be robust to the possibility of refs
missing from the peer remote's summary, by removing any that still have
a NULL checksum associated with them after the summary has been fetched
and processed.
The other OstreeRepoFinder implementations don't have this issue because
they use summary files directly and therefore always have access to the
checksum.
Closes: #1717
Approved by: pwithnall
when converted to bytes
In a subsequent commit, we add a public API to read the value of
min-free-space-* value in bytes. The value for free space check
is enforced in terms of block size instead of bytes. Therefore,
for consistency we check while preparing the transaction that the
value doesn't overflow when converted to bytes.
https://phabricator.endlessm.com/T23694Closes: #1715
Approved by: cgwalters
Debian and Debian-derived systems have their GRUB configuration file in
/boot/grub/grub.cfg, rather than /boot/grub2/grub.cfg. Detecting this
file is necessary to correctly generate GRUB boot configuration on
Debian systems.
Closes: #1714
Approved by: cgwalters
If a ref already exists, we are likely only a few commits behind the
current head of the ref, so it is probably better for bandwidth
consumption to pull the individual objects rather than the from-scratch
delta.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1709
Approved by: cgwalters
Since 9dc6ddce08 it has not been true that
'new_config' was simply ref'd: it's serialized, and then re-parsed into
a new GKeyFile.
Closes: #1707
Approved by: jlebon
Add an invalid-cache test error flag to ensure that the code that checks
for and recovers from a corrupted summary cache is hit. This helps make
sure that the recovery path is actually used without resorting to
G_MESSAGES_DEBUG.
Closes: #1698
Approved by: cgwalters
If for some reason the cached summary doesn't match the cached signature
then fetch the remote summary and verify again. Since commit c4c2b5eb
this is unlikely to happen since the summary will only be cached if it
matches the signature. However, if the summary cache has been corrupted
for any other reason then it's best to be safe and fetch the remote
summary again.
This is essentially the corollary to c4c2b5eb. Where that commit helps
you from getting into the corrupted summary cache in the first place,
this helps you get out of it. Without this the client can get wedged
until a prune or the remote server republishes the summary.
Closes: #1698
Approved by: cgwalters
copy_option() unnecessarily passed ownership of the value
to g_variant_dict_insert_value, but that already refs, so it was leaked.
Closes: #1702
Approved by: cgwalters
We need to have the g_auto(GLnxDirFdIterator) inside the loop, or
we don't correctly clean up when iterating several times.
Closes: #1700
Approved by: cgwalters
Now that we have `auto-update-summary`, there is no point in having
`commit-update-summary`. The latter also only had an effect through
the `commit` CLI command, whereas the former is embedded directly in
libostree.
There is one corner case that slips through: `commit` would update the
summary file even if orphan commits were created, which we no longer do
here. I can't imagine anyone relying on this, so it seems safe to drop.
Closes: #1689Closes: #1693
Approved by: mwleeds
Mildly bikeshed, though I find the name `auto-update-summary` to be
easier to grok than `change-update-summary`. I think it's because it can
be read as "verb-verb-noun" rather than "noun-verb-noun".
Closes: #1693
Approved by: mwleeds
Normally, a configured remote will only serve refs with one associated
collection ID, but temporary remotes such as USB drives or LAN peers can
serve refs from multiple collection IDs which may use different GPG
keyrings. So the OstreeRepoFinderMount and OstreeRepoFinderAvahi classes
create dynamic OstreeRemote objects for each (uri, keyring) pair. So if
for example the USB mounted at /mnt/usb serves content from the
configured remotes "eos-apps" and "eos-sdk", the OstreeRepoFinderResult
array returned by ostree_repo_find_remotes_async() will have one result
with a remote called something like
file_mnt_usb_eos-apps.trustedkeys.gpg and the list of refs on the USB
that came from eos-apps, and another result with a remote
file_mnt_usb_eos-sdk.trustedkeys.gpg and the list of refs from eos-sdk.
Unfortunately while OstreeRepoFinderMount and OstreeRepoFinderAvahi
correctly only include refs in a result if the ref uses the associated
keyring, the find_remotes_cb() function used to clean up the set of
results looks at the remote summary file and includes every ref that's
in the intersection with the requested refs, regardless of whether it
uses a different remote's keyring. This leads to an error when you try
to pull from a USB containing refs from different collection IDs: the
pull using the wrong collection ID will error out with "Refspec not
found" and the result with the correct keyring will then be ignored "as
it has no relevant refs or they have already been pulled." So the pull
ultimately fails.
This commit fixes the issue by filtering refs coming from a dynamic
remote, so that only ones with the collection ID associated with the
keyring remote are examined. This only needs to be done for dynamic
remotes because you should be able to pull any ref from a configured
remote using its keyring. It's also only done when looking at the
collection map in the summary file, because LAN/USB remotes won't have a
"main" collection ID set (OSTREE_SUMMARY_COLLECTION_ID).
Closes: #1695
Approved by: pwithnall
This commits adds and implements a boolean repo config option called
"change-update-summary" which updates the summary file every time a ref
changes (additions, updates, and deletions).
The main impetus for this feature is that the `ostree create-usb` and
`flatpak create-usb` commands depend on the repo summary being up to
date. On the command line you can work around this by asking the user to
run `ostree summary --update` but in the case of GNOME Software calling
out to `flatpak create-usb` this wouldn't work because it's running as a
user and the repo is owned by root. That strategy also means flatpak
can't update the repo metadata refs for fear of invalidating the
summary.
Another use case for this relates to LAN updates. Specifically, the
component of eos-updater that generates DNS-SD records advertising ostree
refs depends on the repo summary being up to date.
Since ostree_repo_regenerate_summary() now takes an exclusive lock, this
should be safe to enable. However it's not enabled by default because of
the performance cost, and because it's more useful on clients than
servers (which likely have another mechanism for updating the summary).
Fixes https://github.com/ostreedev/ostree/issues/1664Closes: #1681
Approved by: jlebon
This ensures that commits aren't deleted and refs aren't added, removed,
or updated while the summary is being generated. This is in preparation
for adding a repo config option that will automatically regenerate the
summary on every ref change.
Closes: #1681
Approved by: jlebon
Using `MAX(0, $x)` here is useless since we're comparing against an
unsigned integer. Just unpack this and only subtract if it's safe to do
so.
Also, explicitly check for `fd >= 0` rather than just `!= -1` to be sure
it's a valid fd. And finally, explicitly check the return value of
`g_input_stream_read_all` as is done everywhere else in the tree and
make it clear that we're purposely ignoring the return value of `_flush`
here, but not in other places.
Discovered by Coverity.
Closes: #1692
Approved by: cgwalters
I initially was going to add a `G_DEFINE_AUTOPTR_CLEANUP_FUNC` for
`FetchStaticDeltaData`, but it honestly didn't seem worth mucking around
ownership everywhere and potentially getting it wrong.
Discovered by Coverity.
Closes: #1692
Approved by: cgwalters
In `write_metadata_object()`, make sure when creating tombstone commits
that we're actually passed an expected checksum to use.
In `write_dir_entry_to_mtree_internal()`, sanity check that `dfd_iter`
is indeed not `NULL` before trying to dereference it.
Discovered by Coverity.
Closes: #1692
Approved by: cgwalters
Since min_free_space_size_mb is considered before min_free_space_percent
in min_free_space_calculate_reserved_blocks(), it has to be considered
first when generating the error message in order for it to be accurate.
Closes: #1691
Approved by: jlebon
Previously, we would error out if both of the options were mentioned
in the config file (even if one of them is disabled with 0). There
were few suggestions that this behavior was not quite right.
Therefore, instead of throwing error and exiting, it's preferred to
warn the user. Hence, the solution that worked out is:
* Allow both options to exist simulateneously
* Check each config's value and decide:
* If both are present and are non-zero, warn the user. Also, prefer
to use min-free-space-size over the another.
* If both are absent, then use -percent=3% as fallback
* Every other case is valid hence, no warning
https://phabricator.endlessm.com/T13698
(cherry picked from commit be68991cf80f0aa1da7d36ab6e1d2c4d6c7cd3fb)
Signed-off-by: Robert McQueen <rob@endlessm.com>
Closes: #1685
Approved by: cgwalters
This is the inverse of https://github.com/ostreedev/ostree/pull/1558
aka commits cadece6c4f398ca61d21e497bd6e3fbb549f9cf6 and
3358698c86d80821d81443c906621c92672f99fb
Needed to fix `rpm-ostree kargs` test suite with default staging; skipping
a test here for now as eventually what we'll do is turn on the rpm-ostree
suite fully here.
Closes: #1677
Approved by: jlebon
Running `ostree commit --tree=ref=a --tree=dir=b` involves reading the
whole of a into an `OstreeMutableTree` before composing `b` on top. This
is inefficient if a is a complete rootfs and b is just touching one file.
We process O(size of a + size of b) directories rather than
O(number of touched dirs).
This commit makes `ostree commit` more efficient at composing multiple
directories together. With `ostree_mutable_tree_fill_empty_from_dirtree`
we create a lazy `OstreeMutableTree` which only reads the underlying
information from disk when needed. We don't need to read all the
subdirectories just to get the checksum of a tree already checked into the
repo.
This provides great speedups when composing a rootfs out of multiple other
rootfs as we do in our build system. We compose multiple containers
together with:
ostree commit --tree=ref=base-rootfs --tree=ref=container1 --tree=ref=container2
and it is much faster now.
As a test I ran
time ostree --repo=... commit --orphan --tree=ref=big-rootfs --tree=dir=modified_etc
Where modified_etc contained a modified sudoers file under /etc. I used
`strace` to count syscalls and I seperatly took timing measurements. To
test with a cold cache I ran
sync && echo 3 | sudo tee /proc/sys/vm/drop_caches
Results:
| | Before | After |
| -------------------- | ------ | ----- |
| Time (cold cache) | 8.1s | 0.12s |
| Time (warm cache) | 3.7s | 0.08s |
| `openat` calls | 53589 | 246 |
| `fgetxattr` calls | 78916 | 0 |
I'm not sure if this will have some negative interaction with the
`_ostree_repo_commit_modifier_apply` which is short-circuited here. I
think it was disabled for `--tree=ref=` anyway, but I'm not certain. All
the tests pass anyway.
I originally implemented this in terms of the `OstreeRepoFile` APIs, but
it was *way* less efficient, opening and reading many files unnecessarily.
Closes: #1643
Approved by: cgwalters
It is important that we use user-friendly error strings. The reason
being error strings are seen by users such as in GNOME Software's
error banner.
Closes: #1671
Approved by: jlebon
For `rpm-ostree ex livefs` we have a use case of pushing a rollback
deployment. There's no reason this should require deleting the staged
deployment (and doing so actually breaks livefs which tries to access
it as a data source).
I was initially very conservative here, but I think it ends up
being fairly easy to retain the staged deployment. We need to handle
two cases:
First, when the staged is *intentionally* deleted; here, we just need
to unlink the `/run` file, and then everything will be sync'd up after
reloading.
Second, (as in the livefs case) where we're retaining it,
e.g. adding a deployment to the end. What I realized here is that
we can have the code keep `new_deployments` as view without staged,
and then when we do the final reload we'll end up re-reading it from
disk anyways.
Closes: #1672
Approved by: jlebon
This implements a TODO item from
`ostree_mutable_tree_get_contents_checksum`. We now no-longer invalidate
the dirtree contents checksum at `get_contents_checksum` time - we
invalidate it when the mtree is modified. This is implemented by keeping
a pointer to the parent directory in each `OstreeMutableTree`. This gives
us stronger invariants on `contents_checksum`.
For even stronger guarantees about invariants we could make
`ostree_repo_write_mtree` or similar a member of `OstreeMutableTree` and
remove `ostree_mutable_tree_set_metadata_checksum`.
I think I've fixed a bug here too. We now invalidate parent's contents
checksum when our metadata checksum changes, whereas we didn't before.
Closes: #1655
Approved by: cgwalters
If ostree_repo_prepare_transaction() fails, we should reset the
repository’s state so that the failed call was essentially idempotent.
Do that by calling ostree_repo_abort_transaction() on the failure path.
Typically, the way for preparing a transaction to fail is for its
GCancellable to be triggered, rather than because any of the operations
involved in preparing a transaction are particularly failure prone.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1647
Approved by: cgwalters
min-free-space-* act as a gating condition whether to we want hold onto caches in
repo/tmp. If it is found that the free-disk space is going below this threshold,
we flag it as an error and cleanup current boot's staging directory.
Closes: #1602
Approved by: jlebon
Here's a subtle bug in abort_transaction():
One of the policies of cleaning up is to skip the current boot's staging
directory. The responsible function for this is cleanup_tmpdir() which tries
to lock each of the tmpdir before deleting it. When it comes to the current
boot's staging dir, it tries to lock the directory(again!) but fails as there
is already a lockfile present. Just because the current boot's staging dir was
meant to be skipped, the bug never surfaced up and wasn't catastrohpic.
if (!_ostree_repo_try_lock_tmpdir (dfd, path, &lockfile, &did_lock, error))
return FALSE;
if (!did_lock)
return TRUE; /* Note early return */
...
if (g_str_has_prefix (path, self->stagedir_prefix))
return TRUE; /* Note early return */
The actual check for skipping staging dir for current boot was never reached
because the function returned at did_lock failure.
Therefore, execute cleanup_tmpdir() after releasing the lockfile in
abort_transaction() so that cleanup_tmpdir gets a chance to lock current boot's
staging directory and succeed.
Closes: #1602
Approved by: jlebon
Currently the BLS snippets are named ostree-$ID-$VARIANT_ID-$index.conf,
but the BLS config files are actually sorted by using the version field
which is the inverse of the index.
In most places, _ostree_sysroot_read_boot_loader_configs() is used to
get the BLS files and this function already returns them sorted by the
version field. The only place where the index trailing number is used is
in the ostree-grub-generator script that lists the BLS files to populate
the grub config file.
But for some bootloaders the BLS filename is the criteria for sorting by
taking the filename as a string version. So on these bootloaders the BLS
entries will be listed in the reverse order.
To avoid that, change the BLS snippets filename to have the version field
instead of the index and also to have the version before deployment name.
Make the filenames to be of the form ostree-$version-$ID-$VARIANT_ID.conf
so the version is before the deployment name.
Signed-off-by: Javier Martinez Canillas <javierm@redhat.com>
Closes: #1654
Approved by: cgwalters
If a function has a guint "out argument", passing a pointer to a gsize
is not, in general, valid. On an ILP64 platform there is no problem
since guint and gsize are identical, but on an LP64 platform it will
overwrite only the first word of the gsize, leaving the second word
unaffected. On little-endian machines, if the second word is
zero-initialized (as it is here), the result is numerically equal to
the guint, but on big-endian machines the result is around 4 billion
times what it should be, resulting in
ostree_repo_finder_config_resolve_async() reading past the end of
the array and causing undefined behaviour.
In practice this caused assertion failures (and consequently test
failures) on Debian's s390x (z/Architecture), ppc64 (64-bit PowerPC)
and sparc64 (64-bit SPARC) ports.
Closes: #1640
Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #1641
Approved by: cgwalters
A prelude to my understanding. Unfortunately `OstreeMutableTree` provides
little encapsulation, as each member has setters† so it's difficult to come
up with a list of invariants.
† `files` and `subdirs` only have getters, but the getters return mutable
references to the internals, so we still can't reason about invariants.
Closes: #1645
Approved by: jlebon
We special-case AVAHI_ERR_NO_DAEMON to not cause warnings, but if
we pass AVAHI_CLIENT_NO_FAIL to avahi_client_new, we never actually
see AVAHI_ERR_NO_DAEMON. Instead, we will get AVAHI_ERR_BAD_STATE
when we try to use the client.
Closes: #1618
Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #1639
Approved by: cgwalters
During the pull, there is an explicit check for free space on disk
vs. the size of uncompressed delta; But while writing the new content
objects that are generated, they have to honor min-free-space-* checks
too. We enforce this check in _bare_content_commit as that is where
we can know the final size of the new content object.
Closes: #1614
Approved by: jlebon
We were referencing the txn bits outside of the lock in the error
path. Generally shouldn't matter, but e.g. Rust wouldn't let us do this, and
race detector tooling will warn about it.
Closes: #1632
Approved by: jlebon
I generally like having variables include their units where applicable;
timer variables having `_secs` or `_ms`, etc.
Closes: #1632
Approved by: jlebon
Systemd units using ConditionNeedsUpdate run if the mtime of .updated in
the specified directory is newer than /usr. Since /usr has an mtime of
0, there's no way to have an older .updated file. Systemd units
typically specify ConditionNeedsUpdate=/etc or ConditionNeedsUpdate=/var
to support stateless systems like ostree.
Remove the file from the new deployment's /etc and the OS's /var
regardless of where they came from to ensure that these systemd units
run when booting new deployments. This will provide a method to run
services only on upgrade.
Closes: #1628https://bugzilla.gnome.org/show_bug.cgi?id=752950Closes: #1631
Approved by: cgwalters
Currently when I run `ostree prune` it hits a seg fault when the
hash_func is used (in this case g_str_hash) from the call stack
_ostree_repo_prune_tmp() -> g_hash_table_contains() ->
g_hash_table_lookup_node(). So the key, in this case dent->d_name, must
be corrupt in some way.
glnx_dirfd_iterator_next_dent() uses readdir() to get the dirent struct.
And according to the man page for readdir(3), "POSIX.1 explicitly notes
that this field should not be used as an lvalue" (in reference to
d_name). So this commit avoids modifying d_name in place and copies it
instead. This seems to avoid the seg fault.
Closes: #1627
Approved by: jlebon
If there’s a problem while aborting a transaction, store the error but
don’t report it until the end of the function — do a best effort at
clearing the rest of the transaction state first (since most of it
cannot fail).
If cleanup_tmpdir() fails (which, arguably, should not be a
showstopper), this allows a caller to recover and start a new
transaction in future.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1626
Approved by: jlebon
Similar to min-free-space-percent but it supports specific sizes
(in MB, GB or TB). Also, making min-free-space-percent and -size
mutually exclusive.
min-free-space-percent does not give a fine tuning of the free disk
space that a user might decide to keep. It can translate to very large
size (e.g. 1% = ~10GB on 1TB HDD) or very small (e.g. 1% = ~330MB on 32GB
system like Endless devices). Hence, it makes sense to introduce a config
option to honor specific size as per the user.
Closes: #1616
Approved by: jlebon
We need to include libglnx.h in places where ostree-autocleanups.h is
included, so that we get backports of G_DEFINE_AUTOPTR_CLEANUP_FUNC and
friends.
Closes: #1615
Approved by: jlebon
This reverts commit f1d9196076.
Since libglnx.h does not get installed, it can't be included in
ostree-autocleanups.h, which is included by ostree.h.
Closes: #1615
Approved by: jlebon
Currently the API that allows P2P operations (e.g. pulling an ostree ref
from a LAN or USB source) is hidden behind the configure flag
--enable-experimental-api. This commit makes the API public and makes
that flag essentially a no-op (leaving it in place in case we want to
use it again in the future). The P2P API has been tested over the last
several months and proven to work.
This means that since we're no longer using the "experimental" feature
flag, P2P builds of Flatpak will fail when using versions of OSTree from
this commit onwards, until Flatpak is patched in the near future. If you
want to build Flatpak < 0.11.8 with P2P enabled and link against OSTree
2018.6, you'll have to patch Flatpak. However, since Flatpak won't yet
have a hard dependency on OSTree 2018.6, it needs a new way to determine
if the P2P API in OSTree is available, so this commit adds a "p2p"
feature flag. This way the feature set is more semantically correct than
if we had continued to use the "experimental" feature flag.
In addition to making the P2P API public, this commit makes the P2P unit
tests run by default, removes the f27-experimental CI instance that's no
longer needed, changes a few man pages to reflect the changes, and
updates the bash completion script to accept the new commands and
options.
Closes: #1596
Approved by: cgwalters
This commit includes libglnx.h in ostree-autocleanups.h, so we get the
g_autoptr backports wherever they're needed. Also, remove the "#include
libglnx.h" lines elsewhere that are no longer needed.
Closes: #1596
Approved by: cgwalters
Use the recently introduced architecture for retrying network requests
on transient failure to do the same for delta superblock requests, now
that they’re queued.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1600
Approved by: jlebon
Just like all the other requests made for delta parts and objects by the
pull code, use a queue for delta superblocks. Currently this doesn’t do
any prioritisation or retries after transient failures, but it could do
in future.
This means that delta superblocks are now subject to the parallel
request limit in the fetcher, which was a problem highlighted here:
https://github.com/ostreedev/ostree/pull/1453#discussion_r168321706.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1600
Approved by: jlebon
Use the same G_IO_ERROR_* values for HTTP status codes in both fetchers.
The libsoup fetcher still handles a few more internal error codes than
the libcurl one; this could be built on in future.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1594
Approved by: jlebon
Various of the counters already have assertions like this; add some more
for total paranoia.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1594
Approved by: jlebon
This allows the retry code in ostree-repo-pull.c to recover from (for
example) timeouts at the libsoup layer in the stack, as well as from the
GSocket layer in the stack.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1594
Approved by: jlebon
Allow network requests to be re-queued if they failed with a transient
error, such as a socket timeout. Retry each request up to a limit
(default: 5), and only then fail the entire pull and propagate the error
to the caller.
Add a new ostree_repo_pull_with_options() option, n-network-retries, to
control the number of retries (including setting it back to the old
default of 0, if the caller wants).
Currently, retries are not supported for FetchDeltaSuperData requests,
as they are not queued. Once they are queued, adding support for retries
should be trivial. A FIXME comment has been left for this.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1594
Approved by: jlebon
When building the OstreeBloom code against old versions of glib, we have
to have the libglnx headers included so that it defines
G_DEFINE_AUTOPTR_CLEANUP_FUNC and friends for us.
This is similarly true for test-repo-finder-mount.c which indirectly
includes ostree-autocleanups.h.
Closes: #1605
Approved by: cgwalters
This commit rearranges a few things in ostree-repo-pull.c so that OSTree
will successfully compile with experimental API enabled and without
libsoup, libcurl, or avahi:
./autogen.sh --enable-experimental-api --without-soup --without-curl
--without-avahi
This is accomplished with two sets of changes:
1. Move ostree_repo_resolve_keyring_for_collection() so it can be used
even without libsoup or libcurl.
2. Add stub functions for ostree_repo_find_remotes_async() and
ostree_repo_pull_from_remotes_async(), and their _finish() counterparts,
so they return an error when libsoup or libcurl isn't available.
Closes: #1605
Approved by: cgwalters
This introduces no functional changes, but will make upcoming support
for retrying downloads easier to add.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1599
Approved by: jlebon
This introduces no functional changes, but will make upcoming support
for retrying downloads easier to add.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1599
Approved by: jlebon
This introduces no functional changes, but will make upcoming support
for retrying downloads easier to add.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1599
Approved by: jlebon
Rename from `fdata` to `fetch_data` to clarify things and make it
consistent with other similar functionality in the file.
This introduces no functional changes.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1599
Approved by: jlebon
This introduces no functional changes, but does make the code a little
cleaner.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1599
Approved by: jlebon
This introduces no functional changes; just makes the code a bit shorter
in a few places.
https://gcc.gnu.org/onlinedocs/gcc/Conditionals.html
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1599
Approved by: jlebon
This introduces no functional changes, but will make some upcoming
refactoring a little easier.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1599
Approved by: jlebon
The initial motivation for this is that the "staging" code currently
didn't rewrite the deployment refs, meaning that the staged commit
could be pruned.
Hence first, this new API ensures that deployments also
hold a strong ref to their commit, without relying on the magical
"deployment refs" that we inject. That has always been a weird
artifact of the strict layering separation between OstreeSysroot
and OstreeRepo.
I also plan to change rpm-ostree to start using this API to
hold references to base layers for client-side layering; it also
today generates various refs.
That said, if we still want to support multiple processes
writing to a single repo (as happens on EndlessOS today) we
still need to write refs; perhaps later we could add a concept
of "generators" or something that create refs based on whatever
logic?
Another minor thing this fixes is that we had a printf inside
the library; this propagates the pruned data to the higher level
which can log however it likes.
Closes: #1566
Approved by: jlebon
Prep for reworking how we do sysroot cleanup. We're going to
start doing more lowlevel pruning work there, and I wanted to avoid
duplicating the ref enumeration.
Closes: #1566
Approved by: jlebon
Likewise the corresponding support for syslinux introduced by commit
c5112c25e4, this one enables writing devicetree
filename into the uEnv.txt environment file for u-boot.
Since u-boot does not strictly defines variable names, here 'fdt_file' was
chosen as it appear to be one the most frequently adopted names in u-boot
default environments. Outer boot logic should of course comply with this choice
and use $fdt_file as the device tree file name to pass to boot commands.
This was tested on a custom board booting with u-boot.
Closes: #1590
Approved by: cgwalters
I feel like I'm drowning in a pile of experimental-but-almost-stable
features...
Anyways, since we made the feature opt-in in rpm-ostree in
https://github.com/projectatomic/rpm-ostree/pull/1352
let's mirror that a bit here with an environment variable so people
can play with it more easily.
The tests needed some tweaks; specifically we need to reload the
status fact after making changes. I'm still a bit uncertain
about the Ansible-as-tests.
But we add an upgrade test that uses the new environment variable.
Closes: #1583
Approved by: jlebon
This should give a more insightful error message if the user provides
a UID which is present on multiple keys.
This happens if you have an old key in your keyring which you are not
actively using any more, e.g. because it is too old. You still have
your old keys in your keyring, because you want to read old email
encrypted for that key, though.
The gpgme function used by ostree right now complains if a UID is found
on multiple keys:
https://www.gnupg.org/documentation/manuals/gpgme/Listing-Keys.html#index-gpgme_005fget_005fkey
The used API is too simple for that use case.
Note that it would be nicer if ostree picked the only valid signing key out
of the available keys rather than using the simplistic gpgme_get_key
function. It be nicer, of course, if there was such a gpgme function.
Closes: #1579
Approved by: cgwalters
Let's add a semi-colon between the "bootconfig swap" part and the
"deployment count change" to make it more clear they're separate
statements.
Closes: #1575
Approved by: cgwalters
While we do a `syncfs()` plus `FIFREEZE/THAW` for `/boot`, that
only comes during deployment finalization.
The code here today generally assumes that if the file exists
it's been fully written. So let's do a `fdatasync()` before
we do the `rename()`.
This just came out of looking through the code while working
on deployment staging. In that scenario there's a much larger
window between when we copy the kernel/initramfs and when we
sync `/boot`.
Closes: #1571
Approved by: jlebon
This is easier to `git grep` etc. versus ad-hoc English. Although
we still have some English for the prepare_transaction/commit which
acquire/release in separate phases.
Closes: #1572
Approved by: jlebon
These are further fixes based on running more of the rpm-ostree
test suite.
When dropping the staged deployment, we do need to do the
"post operations" such as bumping the sysroot mtime, so that
clients know something changed. We also need to regenerate
the deployment refs. And of course do a sysroot reload.
Also, add a "base cleanup" after creating a staged deployment
which also regenerates the refs.
Closes: #1570
Approved by: jlebon
There's no reason to do this. I didn't actually hit this problem,
but it's a corner case that just occurred to me while working on
the code.
I think callers should be adapted to skip trying to use staging
if there's no booted deployment.
Closes: #1568
Approved by: jlebon
This was pointed out in a previous PR review; we don't have
a need for the separate variables. Prep for adding an API for
this.
Closes: #1568
Approved by: jlebon
Doing so can break rpm-ostree, which wants to own the cleanup process
to ensure its baselayer refs are generated.
Further, doing the cleanup at shutdown time adds latency. It's also
going to be generally unnecessary as we expect repo pruning to have
been done when writing the refs.
Closes: #1567
Approved by: jlebon
The code has been sitting around for a while but since I disabled
it by default, I doubt anyone is really using it or relying on it.
This patch and turns on locking by default, and also drops the
API which was only public in the experimental API builds.
Conceptually these are two distinct things, and we
may actually want to split up the patches.
I don't think this will break anyone, but it's hard to say for sure.
It's also going to be hard to find out until we actually release
I suspect...
But anyone who is broken should be able to add `locking=false` into
their repo config. On the flip side Endless has been shipping with
this enabled and it is reported to help.
The reason to drop the APIs: I'm a bit concerned about the interactions over time
between libostree's use of the API and any apps that start using it.
For example, if an app specifies a SHARED lock in their code, then
later internally we decide to temporarily grab an `EXCLUSIVE`, but the
app had a second thread/process that was `EXCLUSIVE` already, and
that process was waiting on the first bit of code, then we could
deadlock. I can't think of a real world situation where this would happen
yet though.
We are likely to in the future have say `fsck` take an external lock,
`checkout` grab a shared one, etc.
Closes: #1555
Approved by: jlebon
Today rpm-ostree has some code to run a "sanitycheck" on a deployment.
I had initially deleted that when adapting it to use the staging code,
but I realized it should work fine; we just won't see the merged
config, but that's OK.
When I readded that code it started crashing because we didn't
actually return the new deployment object. We'll gain some coverage
here as I'll land the code to have rpm-ostree use staging, then bump
the rpm-ostree tests here.
Closes: #1559
Approved by: jlebon
OpenShift Merge Robot
Colin Walters
Jonathan Lebon
Philip Withnall
William Manley
Frédéric Danis
Denis Pynkin
Stefan Agner
Dan Nicholson
Felix Krull
Matthew Leeds
John Hiesey
Philip Chimento
Robert Fairley
Umang Jain
Jason Wessel
Javier Martinez Canillas
Daniel Drake
Rafael Fonseca
Allen Bai
Tristan Van Berkom
Richard Hughes
Laurent Bonnans
Ricardo Salveti
Matthias Clasen
Umang Jain
Luca Bruno
Alexander Larsson
Giuseppe Scrivano
Robert McQueen
Will Thompson
William Manley
Simon McVittie
Andrea Galbusera
Tobias Mueller