Option "--no-sign-verify" disable the signature verification including
GPG. So use it in tests instead of "--no-gpg-verification".
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
Add functions for keys generation to be used in signing-related tests:
- gen_ed25519_keys initializing variables ED25519PUBLIC, ED25519SEED and
ED25519SECRET with appropriate base64-encoded keys
- gen_ed25519_random_public print a random base64 public key (used in
tests with wrong keys)
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
Extend the ed25519 tests with checking the system-wide directory
keys loading code(with the help of redefinition).
Added test of ed25519 revoking keys mechanism.
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
`ostree sign` is able to use several public keys provided via arguments
and via file with keys.
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
API changes:
- added function `ostree_sign_add_pk()` for multiple public keys using.
- `ostree_sign_set_pk()` now substitutes all previously added keys.
- added function `ostree_sign_load_pk()` allowed to load keys from file.
- `ostree_sign_ed25519_load_pk()` able to load the raw keys list from file.
- use base64 encoded public and private ed25519 keys for CLI and keys file.
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
Add tests checking:
- sign mechanism is in working state
- module 'dummy' is able to sign/verify commit
- module 'ed25519' is able to sign/verify commit
- both modules could be used for the same commit
- 'ostree sign' builtin works with commits
- 'ostree commit' builtin able to sign commits
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
The [dev-overlay](332c6ab3b9/src/cmd-dev-overlay)
script shipped in coreos-assembler mostly exists to deal
with the nontrivial logic around SELinux policy. Let's make
the use case of "commit some binaries overlaying a base tree, using
the base's selinux policy" just require a magical
`--selinux-policy-from-base` argument to `ostree commit`.
A new C API was added to implement this in the case of `--tree=ref`;
when the base directory is already checked out, we can just reuse
the existing logic that `--selinux-policy` was using.
Requires: https://github.com/ostreedev/ostree/pull/2039
This test keeps occasionally failing in CI - as expected, because
we retry 8 times for an object but it's completely possible for
us to hit the <0.5% chance of 50% failure 8 times in a row.
Since the max errors from the server is 100, set retries to the
same thing.
Previously we made an effort to use the [Fedora Standard Test Interface](https://docs.fedoraproject.org/en-US/ci/standard-test-interface/).
This effort was not very successful; the primary thing that
it really died on is Ansible just didn't support rebooting
very well. I think that's since gotten better, but even
then, Ansible wasn't the best thing for a test framework
for us anyways.
In the meantime Fedora CoreOS happened emphasizing Ignition
and not "post-hoc reconciliation" models like Ansible over
ssh.
And, [coreos-assembler](https://github.com/coreos/coreos-assembler) happened too.
Furthermore, we really need to test OSTree's interaction
with Ignition as we've invented several special things there.
Then most recently, I've been working on having
cosa/kola support running externally defined tests:
https://github.com/coreos/coreos-assembler/pull/1215
There's a lot of things to clean up after this but at least this
works for me:
```
$ cd /srv/fcos
$ cosa kola run -- --parallel 4 --output-dir tmp/kola -E ~/src/github/ostreedev/ostree/ 'ext.ostree.*'
```
NOTE: This *does not* drop ostree binaries into the target. See:
https://github.com/coreos/coreos-assembler/pull/1252#issuecomment-600623315
This drops our dependency on Python in the installed tests, and
also fixes a few bugs that came up.
I disabled the `itest-bare-user-root.sh` one because it's
entangled with the shell script infrastructure for the unit tests.
Move the alternative builds into the Jenkinsfile.
Update it to do a FCOS build + kola run.
We drop the flatpak/rpm-ostree runs for now; the former
will needs some work to do the automatic virt bits. The
latter I think we can circle back to when we e.g. figure
out how to include rpm-ostree's tests in kola runs.
This checks whether gpg-import will properly update the keyring for a
key that already exists. In particular, we check that changing the key
expiration time or revoking it results in commit verification failure
after re-importing the keys.
Add explicit tests for
`ostree_gpg_verify_result_require_valid_signature` in addition to the
implicit tests via `ostree pull` and others. This allows checking the
error code raised.
Currently `ostree_gpg_verify_result_require_valid_signature` always
returns an error that the key used for the signature is missing from the
keyring. However, all that's been determined is that there are no valid
signatures. The error could also be from an expired signature, an
expired key, a revoked key or an invalid signature.
Provide values for these missing errors and return them from
`ostree_gpg_verify_result_require_valid_signature`. The description of
each result is appended to the error message, but since the result can
contain more than one signature but only a single error can be returned,
the status of the last signature is used for the error code. See the
comment for rationale.
Related: flatpak/flatpak#1450
Currently tests are always run against the full lgpl2.sig file with all
signatures, but it should also be possible to specify one or more of the
individual lgpgl2.sig<N> files.
Drop the current usage of passing the signature index in the test data
since it's always specific to the test function and instead provide an
optional array of signature files for the test fixture to sign with.
When the private keys were generated, gpg added an ultimate trust entry
since you normally want to trust your own keys. However, this throws off
the expired signature testing since gpgme considers it valid if the key
is fully or ultimately trusted.
The use of a trustdb for the test-gpg-verify-result is unlike any other
GPG verification in ostree. Under normal circumstances, a temporary GPG
homedir is created without any trust information, so all keys are
treated as having unknown trust.
Regenerate an empty trustdb.gpg in gpg-verify-data so that the tests
behave as ostree normally operates. After this the expired signature
testing correctly shows up as a non-valid signature. The trustdb was
regenerated by simply removing it and running any gpg operation with the
gpg-verify-data directory as the homedir.
The full block with all 5 signatures remains, but this allows passing
individual signatures through the GPG verification APIs. The split was
done with `gpgsplit`, and looking at the output of `gpg --list-packets`
of the split and unsplit files appears correct.
These can then be imported during a test to revoke a key without trying
to go through the gpg --generate-revocation dialog. Note that these need
to go in a subdirectory of the homedir since `gpgkeypath` will try to
import every regular file in the homedir.
gpg prints a warning about unsafe permissions if the homedir is group or
world readable. This is just noise in the test logs, so appease it by
making the homedir 700.
Use long GPG key IDs as it's safer and matches the format used by gpg
and gpgme. Add the associated fingerprints since these are needed by gpg
when manipulating keys.
The case-ignoring regex `^(C|en_US)` will match any locale that starts
with `c`. On my system this is `ca_AD.utf8`, which breaks the test
suite. Instead, use a single regex that includes the joining `.` rather
than 2 separate regexes. This also changes `head` to use the `-n`
option, which has been preferred for at least 10 years in the coreutils
version and is supported by busybox as well.
Append a byte encoding the OSTree object type for each object in the
metadata. This allows the commit metadata to be fetched and then for the
program to see which objects it already has for an accurate calculation
of which objects need to be downloaded.
This slightly breaks the `ostree.sizes` `ay` metadata entries. However,
it's unlikely anyone was asserting the length of the entries since the
array currently ends in 2 variable length integers. As far as I know,
the only users of the sizes metadata are the ostree test suite and
Endless' eos-updater[1]. The former is updated here and the latter
already expects this format.
1. https://github.com/endlessm/eos-updater/
Ensure all 3 of the checksum, compressed size and uncompressed size are
correct. For repeatable objects, skip xattrs and use canonical
permissions for the commit. For the sizes, read a varint rather than
assuming they will be a single byte. To work around bugs in gjs with
byte array unpacking, manually build the array byte by byte. Split out
some helper functions to use in subsequent tests.
When running with installed tests, ostree-prepare-root (probably)
exists in /usr/lib. Add heuristics to look for it based on the directory
we're running from.
Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
If running with systemd and libmount then /var mounting is deferred for
systemd. Skip the relevant tests in this case as it will always fail.
Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
Since we're not interested in any file inside /proc, exclude it from the
file listing in our fake root thus avoiding failures when processes die
during our execution and find(1) can't then look inside those
directories.
Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
Define an `OstreeKernelArgsEntry` structure, which holds
both the key and the value. The kargs order array stores
entries for each key/value pair, instead of just the keys.
The hash table is used to locate entries, by storing
entries in a pointer array for each key. The same public
interface is preserved, while maintaining ordering
information of each key/value pair when
appending/replacing/deleting kargs.
Fixes: #1859
Prep for fsverity, where I want to create a new group
`[fsverity]` in the keyfile that has default values. We should
treat the absence of a group the same as absence of a key
in these "with defaults" APIs.
When copying the tree, using musl and GNU coreutils, something gets confused
when setting the ownership of symlinks and the copy fails with:
cp: failed to preserve ownership for osdata-devel/bin: Not supported
Rework using tar to avoid the problem.
Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
When building with musl there's no locale command, also its default
locale is C.UTF-8, so just get C.UTF-8 if we can't find locale.
Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
When using musl, it appears that the default is line buffered output, so
when `head -1` reads from a pipe we have to handle the source end of the
pipe getting EPIPE.
Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
A number of tests expect explicit left/right single quotes in their
messages, which will never happen in the C locale. Change so we pick a
likely UTF-8 locale, or fail if we can't find one.
Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
To allow for FIPS mode, we need to also install the HMAC file from
`/usr/lib/modules` to `/boot` alongside the kernel image where the
`fips` dracut module will find it. For details, see:
https://github.com/coreos/fedora-coreos-tracker/issues/302
Note I didn't include the file in the boot checksum since it's itself a
checksum of the kernel, so we don't really gain much here other than
potentially causing an unnecessary bootcsum bump.
If we are built without libarchive support, this test fails:
error: This version of ostree is not compiled with libarchive support
...
ERROR: tests/test-export.sh - too few tests run (expected 5, got 0)
ERROR: tests/test-export.sh - exited with status 1
Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
Tiny release. Just want to get out the important bugfixes instead of
backporting patches (notably the gpg-agent stuff and
`ostree-finalize-staged.service` ordering).
Closes: #1927
Approved by: cgwalters
The ostree fsck test is aimed to check that it will still fail an fsck
if the repository has been repaired by fsck. It also checks that a
pull operation corrects the error and ostree fsck will exit with zero.
The test was modeled after the following script:
rm -rf ./f1
mkdir -p ./f1
./ostree --repo=./f1 init --mode=archive-z2
mkdir -p ./trial
echo test > ./trial/test
./ostree --repo=./f1 commit --tree=dir=./trial --skip-if-unchanged --branch=exp1 --subject="test Commit"
rm -rf ./f2
mkdir -p ./f2
./ostree --repo=./f2 init
./ostree --repo=./f2 pull-local ./f1
echo whoops > `find ./f2 |grep objects |grep \\.file `
./ostree fsck --repo=./f2 ; echo Exit: $?
./ostree fsck --delete --repo=./f2 ; echo Exit: $?
./ostree fsck --repo=./f2 ; echo Exit: $?
./ostree --repo=./f2 pull-local ./f1
./ostree fsck --repo=./f2 ; echo Exit: $?
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
fsck: Update test so that it will pass on fs without xattrs
The fsck test does not require xattrs to prove that it works. It is
simple enough to change it to use an archvie instead of a bare type
repository.
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Closes: #1910
Approved by: cgwalters
If there are different deployments for the same commit version, the BLS
snippets will have the same title fields (but different version fields):
$ grep title *
ostree-1-testos.conf:title TestOS 42 20190902.0 (ostree)
ostree-2-testos.conf:title TestOS 42 20190902.0 (ostree)
ostree-3-testos.conf:title TestOS 42 20190902.0 (ostree)
But bootloaders could expect the title field to be unique for BLS files.
For example, the zipl bootloader used in the s390x architecture uses the
field to name the boot sections that are created from the BLS snippets.
So two BLS snippets having the same title would lead to zipl failing to
create the IPL boot sections because they would have duplicated names:
$ zipl
Using config file '/etc/zipl.conf'
Using BLS config file '/boot/loader/entries/ostree-3-testos.conf'
Using BLS config file '/boot/loader/entries/ostree-2-testos.conf'
Using BLS config file '/boot/loader/entries/ostree-1-testos.conf'
Error: Config file '/etc/zipl.conf': Line 0: section name 'TestOS 42 20190902.0 (ostree)' already specified
Avoid this by always including the deployment index along with the commit
version in the title field, so this will be unique even if there are BLS
files for deployments that use the same commit version:
$ grep title *
ostree-1-testos.conf:title TestOS 42 20190902.0 (ostree:2)
ostree-2-testos.conf:title TestOS 42 20190902.0 (ostree:1)
ostree-3-testos.conf:title TestOS 42 20190902.0 (ostree:0)
$ zipl
Using config file '/etc/zipl.conf'
Using BLS config file '/boot/loader/entries/ostree-3-testos.conf'
Using BLS config file '/boot/loader/entries/ostree-2-testos.conf'
Using BLS config file '/boot/loader/entries/ostree-1-testos.conf'
Building bootmap in '/boot'
Building menu 'zipl-automatic-menu'
Adding #1: IPL section 'TestOS 42 20190902.0 (ostree:0)' (default)
Adding #2: IPL section 'TestOS 42 20190902.0 (ostree:1)'
Adding #3: IPL section 'TestOS 42 20190902.0 (ostree:2)'
Preparing boot device: dasda (0120).
Done.
Closes: #1911
Approved by: cgwalters
We would stop passing through `--` and args after it to the underlying
command in `ostree_run`. This made it impossible to use `--` to tell the
parser that following args starting with `-` really are positional.
AFAICT, that logic for `--` here came from a time when we parse options
manually in a big loop, in which case breaking out made sense (see
97558276e4).
There's an extra step here, which is that glib by default leaves the
`--` in the list of args, so we need to take care to remove it from the
list after parsing.
Closes: #1898Closes: #1899
Approved by: rfairley
This skips creating the default stuff in the physical sysroot.
I don't recall why I did that to be honest; it originated with
the first commit of this file. It might not have ever been
necessary.
In any case, it's not necessary now with Fedora CoreOS, so
prune it and let's have a clean `/`.
Keep the old behavior by default though to avoid breaking anyone.
Closes: #1894
Approved by: ajeddeloh
`ostree_repo_resolve_keyring_for_collection()` function fail the tests
if there is no GPG support.
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
Closes: #1889
Approved by: cgwalters
Skip tests or run them without GPG-related functionality if GPGME
wasn't enabled in a build time.
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
Closes: #1889
Approved by: cgwalters
Shell function `has_gpgme` shouldn't exit if GPG support is not detected
since it stop any test with error.
Added function `skip_without_gpgme` to skip the whole test if it is
useless without GPG support
Signed-off-by: Denis Pynkin <denis.pynkin@collabora.com>
Closes: #1889
Approved by: cgwalters
There's a valid use case for enabling the timestamp downgrade check
while still also using override commits.
We'll make use of this in Fedora CoreOS, where the agent specifies the
exact commit to upgrade to, while still enforcing that it be newer.
Closes: #1891
Approved by: cgwalters
The ability to expire subkeys using gpg's --quick-set-expire is only
available on gnupg 2.1.22. If expiring a subkey fails, assume this is
why and skip the tests that require it but run the actions that the
subsequent tests depend on. This was failing on the Debian Stretch CI
tests since stretch has gnupg 2.1.18.
Closes: #1892
Approved by: jlebon
This wasn't available when I originally wrote this, but it ensures that
the running gpg-agent in tmpgpghome is killed in case the tests exit
early.
Closes: #1892
Approved by: jlebon
Recent GJS changed how byte arrays are unpacked with some assumptions
that they are likely strings. Manually use get_child_value() and
get_byte() to ensure the correct value is parsed when checking the
`ostree.sizes` metadata.
The upstream test is currently passing fine with GJS 1.56.2, but at
Endless we (unfortunately) have a downstream change that adds the object
type as an additional byte in the array. This is parsed incorrectly by
`deep_unpack()`. We can carry this patch downstream, but this change
makes the test more robust regardless.
Closes: #1884
Approved by: cgwalters
The GIR for commit_transaction() only has a single argument for the
GCancellable. Calling it with 2 arguments prints a GJS warning:
Gjs-Message: 15:37:40.287: JS WARNING: [/home/dan/src/ostree/tests/test-sizes.js 56]: Too many arguments to method OSTree.Repo.commit_transaction: expected 1, got 2
Currently this is harmless, but it could become a hard error in GJS at
some point.
Closes: #1884
Approved by: cgwalters
Add `libtest_cleanup_gpg()` to the array of commands to run when
exiting. This provides 2 improvements:
1. You don't need to worry about whether the test will spawn a gpg-agent
and therefore require adding a call to `libtest_cleanup_gpg()`.
2. All the existing users were calling `libtest_cleanup_gpg()` at the
end of the script. If there was a failure and the script exited
early, then it wouldn't cleanup and there may be a stray gpg-agent
hanging around.
Closes: #1799
Approved by: cgwalters
Currently if a test script adds a trap on `EXIT` to run some cleanup, it
will stomp on the existing trap to run `save_core()`. Allow for scripts
to append actions that will run on exit by introducing an array that
will be iterated over by a single exit runner.
Closes: #1799
Approved by: cgwalters
Extend test-gpg-signed-commit.sh to test various key states. If gpg is
found that supports the required options, keys will be generated on the
fly and changed in various ways to exercise the output from
`ostree_gpg_verify_result_describe_variant` used in `ostree show`.
I tested this using gnupg 2.2.12, so I hope it works well enough on
various gpgs found in the wild.
Closes: #1872
Approved by: cgwalters
In case the tests want to use a custom GPG homedir, allow passing in the
homedir to use when cleaning up a running gpg-agent.
Closes: #1872
Approved by: cgwalters
Introduce a new signature attribute for the key expiration timestamp and
display it when the key has a non-zero expiration time. Without this,
the error shown is `BAD signature`, which isn't correct.
Closes: #1872
Approved by: cgwalters
This change makes public the current kargs API in src/libostree/ostree-kernel-args.c
and adds documentations.
Upstreams the new kargs API from rpm-ostree/src/libpriv/rpmostree-kargs-process.c
Merges libostree_kernel_args_la_SOURCES to libostree_1_la_SOURCES in Makefile-libostree.am
Upstreams tests/check/test-kargs.c from rpm-ostree.
Closes: #1833Closes: #1869
Approved by: jlebon
Currently if you want to update a non-alias ref, you need to first check
if it exists and use either `ostree refs --create` or `ostree reset` as
appropriate. That's unnecessarily complicated and is much less
convenient than the old `write-refs` builtin that simply called
`ostree_repo_set_ref_immediate()` without any checks.
Add a `--force` option to be used with `--create` that does not raise an
error when the destination ref already exists.
Closes: #1870
Approved by: jlebon
Split the creation of the directory containing grub.cfg, and the creation
of the file, so that a failure in the mkdir command will fail the test
and not attempt the touch command.
Closes: #1831
Approved by: jlebon
Fix CI failure due to mismatching quotes in the error output given
between Fedora and Debian test suites.
Example of the error in Debian Stretch:
https://api.travis-ci.org/v3/job/519335717/log.txtCloses: #1839
Approved by: cgwalters
When writing a delta to a file this may not always be recorded
in the filename, and it's useful data.
Ref: https://mail.gnome.org/archives/ostree-list/2019-February/msg00000.html
This also required teaching `show` to accept a file path.
Note...for some reason `test-deltas.sh` breaks when run from
a tty - we get `SIGTTIN` which implies something is reading from
the tty but it wasn't obvious to me what.
Closes: #1823
Approved by: jlebon
In Silverblue right now, the boot menu title looks like this:
Fedora 29.20190301.0 (Workstation Edition) 29.20190301.0 (ostree)
This is because RPM-OSTree's `mutate-os-release` feature is enabled,
which injects the OSTree version string directly into `VERSION` and
`PRETTY_NAME`. So appending the version string again is a bit redundant.
Let's just do a simple substring check here before adding the version to
the title.
Closes: #1829
Approved by: cgwalters
The sysroot.bootloader key configures the bootloader
that OSTree uses when deploying a sysroot. Having this key
allows specifying behavior not to use the default bootloader
backend code, which is preferable when creating a first
deployment from the sysroot (#1774).
As of now, the key can take the values "auto" or "none". If
the key is not given, the value defaults to "auto".
"auto" causes _ostree_sysroot_query_bootloader() to be used
when writing a new deployment, which is the original behavior
that dynamically detects which bootloader to use.
"none" avoids querying the bootloader dynamically. The BLS
config fragments are still written to
sysroot/boot/loader/entries for use by higher-level software.
More values can be supported in future to specify a single
bootloader, different behavior for the bootloader code, or
a list of bootloaders to try.
Resolves: #1774Closes: #1814
Approved by: jlebon
Add ot_keyfile_get_value_with_default_group_optional() which allows
getting values from keys where the group is optional in the config
file. This is preparatory to add the sysroot.bootloader repo config
key, where the sysroot group is optional.
Closes: #1814
Approved by: jlebon
Currently it's not an error to provide too many arguments to an ostree
config command. Change it so we print usage information in that case,
and update the unit tests.
Closes: #1743
Approved by: cgwalters
Currently there's a way to set a key to the empty string but there's no
way to unset it completely (remove the key from the group). This might
be helpful for instance if you want to temporarily set
"core.lock-timeout-secs" to a specific value for the duration of one
operation and then return it to the default after that operation
completes.
This commit implements an "unset" operation for the config command, adds
a unit test, and updates the man page.
Closes: #1743
Approved by: cgwalters
Since commit a06bd82cd we no longer use OVERLAYFS_SUPER_MAGIC or
statfs() so remove the includes for linux/magic.h and sys/vfs.h
Closes: #1815
Approved by: cgwalters
glib 2.59.2 uses a non-breaking space instead of a space to
separate the quantity and unit in g_format_size() so update
our test to handle both a plain space and a non-breaking space.
See https://gitlab.gnome.org/GNOME/glib/issues/1625Closes: #1818
Approved by: cgwalters
Denis Pynkin
OpenShift Merge Robot
Colin Walters
Jonathan Lebon
Dan Nicholson
John Hiesey
Alex Kiernan
Robert Fairley
Jason Wessel
Javier Martinez Canillas
Allen Bai
Rafael Fonseca
Tristan Van Berkom
Matthew Leeds
Robert Fairley
Jeremy Bicha