Commit Graph
100 Commits
Author SHA1 Message Date
Dan Nicholson 90a3bda1f8 bin/remote: Include update URLs in list-gpg-keys 2021-07-15 15:50:04 -06:00
Dan Nicholson 27dc5d7d38 lib/repo: Include WKD update URLs in GPG key listing
If the key UID contains a valid email address, include the GPG WKD
update URLs in GVariant returned by ostree_repo_remote_get_gpg_keys().
2021-07-15 15:50:04 -06:00
Dan Nicholson 4fa403aee5 libotutil: Add helper for GPG WKD update URLs
Calculate the advanced and direct update URLs for the key discovery
portion[1] of the OpenPGP Web Key Directory specification, and include
the URLs in the key listing in ostree_repo_remote_get_gpg_keys(). These
URLs can be used to locate updated GPG keys for the remote.

1. https://datatracker.ietf.org/doc/html/draft-koch-openpgp-webkey-service#section-3.1
2021-07-15 15:50:04 -06:00
Dan Nicholson fbff05e28d libotutil: Import implementation of zbase32 encoding
This will be used to implement the PGP Web Key Directory (WKD) URL
generation. This is a slightly cleaned up implementation[1] taken from
the zbase32 author's original implementation[2]. It provides a single
zbase32_encode API to convert a set of bytes to the zbase32 encoding.

I believe this should be acceptable for inclusion in ostree. The license
in the source files is BSD style while the original repo LICENSE file
claims the Creative Commons CC0 1.0 Universal license, which is public
domain.

1. https://github.com/dbnicholson/libbase32/tree/for-ostree
2. https://github.com/zooko/libbase32
2021-07-15 15:50:04 -06:00
Dan Nicholson 74fb0c5f78 bin/remote: Add list-gpg-keys subcommand
This provides a wrapper for the `ostree_repo_remote_get_gpg_keys`
function to show the GPG keys associated with a remote. This is
particularly useful for validating that GPG key updates have been
applied. Tests are added, which checks the
`ostree_repo_remote_get_gpg_keys` API by extension.
2021-07-15 15:50:04 -06:00
Dan Nicholson a50f6d0b9f lib/repo: Add ostree_repo_remote_get_gpg_keys()
This function enumerates the trusted GPG keys for a remote and returns
an array of `GVariant`s describing them. This is useful to see which
keys are collected by ostree for a particular remote. The same
information can be gathered with `gpg`. However, since ostree allows
multiple keyring locations, that's only really useful if you have
knowledge of how ostree collects GPG keyrings.

The format of the variants is documented in
`OSTREE_GPG_KEY_GVARIANT_FORMAT`. This format is primarily a copy of
selected fields within `gpgme_key_t` and its subtypes. The fields are
placed within vardicts rather than using a more efficient tuple of
concrete types. This will allow flexibility if more components of
`gpgme_key_t` are desired in the future.
2021-07-15 15:50:04 -06:00
Dan Nicholson c8715c123e lib/repo: Factor out GPG verifier preparation
In order to use the GPG verifier, it needs to be seeded with GPG keys
after instantation. Currently this is only used for verifying data, but
it will also be used for getting a list of trusted GPG keys in a
subsequent commit.
2021-07-15 15:50:04 -06:00
Dan Nicholson dba2cdcbac lib/repo: Factor out GPG verifier key imports
Currently the verifier only imports all the GPG keys when verifying
data, but it would also be useful for inspecting the trusted keys.
2021-07-15 15:50:04 -06:00
Dan Nicholson dc69f56de6 lib: Coerce flags enums to GIR bitfields
The GI scanner decides if an `enum` is really a `bitfield` if it finds
any values that have left shifts. With an `enumeration`, the
introspecting language may error or convert to a different type if the
user tries to combine values. Change all Flags `enum`s to use
left-shifted values so that they're represented as `bitfield`s in the
GIR.

The primary bug here is that you can't combine `REFS_ONLY` and
`NO_PRUNE` when calling `OSTree.Repo.prune()` from an introspected
language.

This is an IABI break since the typelib will change from `enumeration`
to `bitfield`. `OstreeRepoImportFlags` is internal but the change is
included here to prepare for a subsequent name that would require bit
shifting to operate correctly as a flag.
2020-04-30 15:13:31 -06:00
Dan Nicholson 3557e4e165 lib: Fix Since versions for 2020.1
These had been added assuming 2019.7 would be the next version, but now
it's 2020 and there's been a release. In the case of
`OstreeCommitSizesEntry`, I'd forgotten to move it forward from 2019.5
to 2019.7 in the time between when I started working on the feature and
it landed.
2020-02-20 15:47:41 -07:00
Dan Nicholson 11ad68647a ostree/trivial-httpd: Add log message for autoexit
This is useful when checking if the daemon actually exited since we
don't store the child PID anywhere.
2020-01-24 13:11:54 -07:00
Dan Nicholson fbf5a94e0a ostree/trivial-httpd: Fix --autoexit with --daemonize and --log-file
When --autoexit is used with --daemonize and --log-file, the program
never exits when the root directory is deleted. I believe what happens
is that g_file_new_for_path triggers the glib worker context to be
started to talk to GVfs. Once the program forks, the parent exits and
the thread iterating the worker context is gone. The file monitor then
never receives any events because the inotify helper also runs from the
worker context.

Move the fork earlier just after parsing and validating the command line
arguments. In order to handle setup errors in the child, a pipe is
opened and the parents waits until the child writes a status byte to it.
If the byte is 0, the parent considers the child setup successful and
exits and the child carries on as a daemon. Notably, the child doesn't
reopen stderr to /dev/null until after this so that it can send error
messages there.

Fixes: #1941
2020-01-24 13:11:54 -07:00
Dan Nicholson 68a11d4eeb ci/flatpak: Patch GPG error assertions from OSTree
Some of the flatpak tests assert on GPG error strings that come from
OSTree. Those are being changed here, so patch the cloned flatpak 1.4.1
to accommodate the new error strings. When this work lands, I'll send a
patch upstream to flatpak that will eventually trickle back here in a
tagged build.
2020-01-24 13:03:30 -07:00
Dan Nicholson b81a6b4ab2 tests/gpg: Add tests for importing updated remote GPG keys
This checks whether gpg-import will properly update the keyring for a
key that already exists. In particular, we check that changing the key
expiration time or revoking it results in commit verification failure
after re-importing the keys.
2020-01-24 13:02:17 -07:00
Dan Nicholson 01da2371c5 tests/gpg: Test ostree_gpg_verify_result_require_valid_signature
Add explicit tests for
`ostree_gpg_verify_result_require_valid_signature` in addition to the
implicit tests via `ostree pull` and others. This allows checking the
error code raised.
2020-01-24 13:02:17 -07:00
Dan Nicholson 0fbfc0b207 lib/gpg: Add more specific OstreeGpgError codes
Currently `ostree_gpg_verify_result_require_valid_signature` always
returns an error that the key used for the signature is missing from the
keyring. However, all that's been determined is that there are no valid
signatures. The error could also be from an expired signature, an
expired key, a revoked key or an invalid signature.

Provide values for these missing errors and return them from
`ostree_gpg_verify_result_require_valid_signature`. The description of
each result is appended to the error message, but since the result can
contain more than one signature but only a single error can be returned,
the status of the last signature is used for the error code. See the
comment for rationale.

Related: flatpak/flatpak#1450
2020-01-24 13:02:17 -07:00
Dan Nicholson 2c24f28ce4 tests/test-gpg-verify-result: Allow specifying signature files
Currently tests are always run against the full lgpl2.sig file with all
signatures, but it should also be possible to specify one or more of the
individual lgpgl2.sig<N> files.

Drop the current usage of passing the signature index in the test data
since it's always specific to the test function and instead provide an
optional array of signature files for the test fixture to sign with.
2020-01-24 13:02:09 -07:00
Dan Nicholson b786d1b4bc tests/gpg-verify-data: Empty out trustdb.gpg
When the private keys were generated, gpg added an ultimate trust entry
since you normally want to trust your own keys. However, this throws off
the expired signature testing since gpgme considers it valid if the key
is fully or ultimately trusted.

The use of a trustdb for the test-gpg-verify-result is unlike any other
GPG verification in ostree. Under normal circumstances, a temporary GPG
homedir is created without any trust information, so all keys are
treated as having unknown trust.

Regenerate an empty trustdb.gpg in gpg-verify-data so that the tests
behave as ostree normally operates. After this the expired signature
testing correctly shows up as a non-valid signature. The trustdb was
regenerated by simply removing it and running any gpg operation with the
gpg-verify-data directory as the homedir.
2020-01-24 13:02:09 -07:00
Dan Nicholson 7fe265b087 tests/gpg-verify-data: Split out signature data
The full block with all 5 signatures remains, but this allows passing
individual signatures through the GPG verification APIs. The split was
done with `gpgsplit`, and looking at the output of `gpg --list-packets`
of the split and unsplit files appears correct.
2020-01-24 13:02:09 -07:00
Dan Nicholson b825083549 tests/gpghome: Create revocation certificates for keys
These can then be imported during a test to revoke a key without trying
to go through the gpg --generate-revocation dialog. Note that these need
to go in a subdirectory of the homedir since `gpgkeypath` will try to
import every regular file in the homedir.
2020-01-24 13:02:09 -07:00
Dan Nicholson 63414e85c3 tests/libtest: Make temporary gpghome private
gpg prints a warning about unsafe permissions if the homedir is group or
world readable. This is just noise in the test logs, so appease it by
making the homedir 700.
2020-01-24 13:02:09 -07:00
Dan Nicholson 7f04c5d764 tests/libtest: Record long GPG key IDs and fingerprints
Use long GPG key IDs as it's safer and matches the format used by gpg
and gpgme. Add the associated fingerprints since these are needed by gpg
when manipulating keys.
2020-01-24 13:02:09 -07:00
Dan Nicholson c54a3b5daa lib/gpg: Prefer declare-and-initialize style
As noted in
https://github.com/ostreedev/ostree/pull/1872#discussion_r295408768.
2020-01-24 13:02:09 -07:00
Dan Nicholson a6994459c1 ci/rpmostree: Bump to 2019.4
The vmcheck tests in 2019.3 fail because of an SSH control socket issue
on overlayfs. This is fixed in 2019.4[1]. That has some other changes
such as using Python 3 in tests. The package dependencies have been
synced from the rpm-ostree CI for that.

Unfortunately, this is no longer a totally representative test of f29
since it has 2019.3 in updates. But that's the price you pay for
exercising someone else's CI from your own CI.

1. https://github.com/coreos/rpm-ostree/commit/c89f81c1385ef095616b0e7001926572a20057b2

Fixes: #1994
2020-01-21 20:33:58 -07:00
Dan Nicholson 5135a1e58a tests/core: Really pick C.UTF-8 locale
The case-ignoring regex `^(C|en_US)` will match any locale that starts
with `c`. On my system this is `ca_AD.utf8`, which breaks the test
suite. Instead, use a single regex that includes the joining `.` rather
than 2 separate regexes. This also changes `head` to use the `-n`
option, which has been preferred for at least 10 years in the coreutils
version and is supported by busybox as well.
2020-01-21 10:29:41 -07:00
Dan Nicholson 97c831dd5f bin/show: Add --print-sizes option to show sizes metadata
Use the new `ostree_commit_get_object_sizes()` API to read the
`ostree.sizes` commit metadata and print a summary.
2020-01-20 20:46:31 -07:00
Dan Nicholson 260bcd1193 core: Add ostree_commit_get_object_sizes API
This function parses the object listing in the `ostree.sizes` metadata
and returns an array of `OstreeCommitSizesEntry` structures.

Unfortunately, for reasons I don't understand, the linker wants to
resolve `_ostree_read_varuint64` from `ostree-core.c` even though it's
not used by `test-checksum.c` at all.
2020-01-20 20:46:29 -07:00
Dan Nicholson fcbb453443 core: Add OstreeCommitSizesEntry type
This will be used when reading out entries in the `ostree.sizes`
metadata. Each entry corresponds to an object in the metadata array.
2020-01-20 20:44:12 -07:00
Dan Nicholson 1bbe674d91 libarchive: Support commit sizes metadata
Call the helper to set the generate_sizes boolean so that object size
data is stored while writing the mtree.
2020-01-20 20:42:27 -07:00
Dan Nicholson a4592678aa tests/sizes: Check duplicate file doesn't add sizes entry
A duplicate file will resolve to the same object, so it shouldn't add
any entries to the sizes metadata.
2020-01-20 20:42:27 -07:00
Dan Nicholson 4f1b991246 tests/sizes: Test that sizes metadata is not reused
Ensure that the object sizes hash table is cleared after a commit and
not only when the repo is closed.
2020-01-20 20:42:27 -07:00
Dan Nicholson 1ea719b76b tests/sizes: Test sizes metadata with existing objects
Repeat the commit to make sure that the files are enumerated again for
the size metadata.
2020-01-20 20:42:27 -07:00
Dan Nicholson 44fb5e72a1 lib/commit: Make size entries for existing objects
If the object was already in the repo then the sizes metadata entry was
skipped. Move the sizes entry creation after the data has been computed
but before the early return for an existing object.
2020-01-20 20:42:27 -07:00
Dan Nicholson 8ec7d6322f lib/commit: Fix object sizes metadata for multiple commits
The object sizes hash table was only being cleared when the repo was
finalized. That means that performing multiple commits while the repo
was open would reuse all the object sizes metadata for each commit.

Clear the hash table when the sizes metadata is setup and when it's
added to a commit. This still does not fix the issue all the way since
it does nothing to prevent the program from constructing multiple
commits simultaneously. To handle that, the object sizes hash table
should be attached to the MutableTree since that has the commit state.
However, the MutableTree is gone when the commit is actually created.
The hash table would have to be transferred to the root file when
writing the MutableTree. That would be an awkward addition to
OstreeRepoFile, though. Add a FIXME to capture that.
2020-01-20 20:42:27 -07:00
Dan Nicholson 694b741a36 tests/sizes: Improve metadata validation
Ensure all 3 of the checksum, compressed size and uncompressed size are
correct. For repeatable objects, skip xattrs and use canonical
permissions for the commit. For the sizes, read a varint rather than
assuming they will be a single byte. To work around bugs in gjs with
byte array unpacking, manually build the array byte by byte. Split out
some helper functions to use in subsequent tests.
2020-01-20 20:42:27 -07:00
Dan Nicholson 37045b4b46 lib/commit: Only set generate_sizes for archive repos
Rather than checking throughout the code, only set the boolean when
appropriate.
2020-01-20 20:42:27 -07:00
Dan Nicholson 4df90d4012 repo: Stop using deprecated G_GNUC_FUNCTION
In glib 2.62 this has been changed to emitting a warning. Use G_STRFUNC
instead, which has been available for a long time and is already used in
other places in ostree.
2019-10-18 11:10:44 -06:00
Dan Nicholson e49060c207 lib/gpg: Use g_spawn_sync to kill gpg-agent
For reasons I don't understand, GSubprocess doesn't play nice with KDE's
plasmashell. I assume this has something to do with the GSubprocess
using the glib worker thread while plasmashell uses the glib main
loop. Instead, just use g_spawn_sync to fork and wait in the current
thread.

Fixes: #1913

Closes: #1917
Approved by: cgwalters
2019-09-07 18:42:22 +00:00
Dan Nicholson d14472a7f0 lib/gpg: Don't kill gpg-agent on newer gnupg
GnuPG 2.1.17 contains a bug fix so that `gpg-agent` is killed when the
entire GPG home directory is deleted[1]. If the host's GnuPG is new
enough, then we don't need to bother calling `gpg-connect-agent` to kill
the agent since it will be cleaned up on its own.

Get the GnuPG version from the GPGME OpenPGP engine info and parse it to
see if it matches this criteria.

1. https://dev.gnupg.org/T2756

Closes: #1915
Approved by: cgwalters
2019-09-06 18:04:05 +00:00
Dan Nicholson 522d31b2d4 lib/gpg: Only show gpg-connect-agent stderr on failures
When listing GPG keys, the temporary GPG homedir will be constructed by
simply copying the remote's trusted keys to the pubring.gpg file. In
that case, no GPG operations spawning gpg-agent will be run. When
gpg-connect-agent is run to cleanup the homedir, it will helpfully print
on stderr that it's starting gpg-agent like so:

gpg-connect-agent: no running gpg-agent - starting '/usr/bin/gpg-agent'
gpg-connect-agent: waiting for the agent to come up ... (5s)
gpg-connect-agent: connection to agent established

Send gpg-connect-agent's stderr to a pipe and only send it to the
application's stderr if an error was encountered.

Fixes: #1907

Closes: #1908
Approved by: cgwalters
2019-09-02 21:55:14 +00:00
Dan Nicholson cf7fc0eabb ci/rpmostree: Bump to 2019.3
The test-ucontainer.sh test in rpmostree 2019.1 is failing for
non-ostree reasons. This is fixed in [1], which is part of 2019.3. I
believe this is still a relevant test of f29 since that's what's shipped
in f29-updates.

1. https://github.com/projectatomic/rpm-ostree/commit/45b6186be0826889d805ba55bee3be75b042ed35

Closes: #1892
Approved by: jlebon
2019-07-27 03:20:21 +00:00
Dan Nicholson 19e764cea2 lib/kargs: Don't expose OstreeKernelArgs autoptr cleanup on old glib
Move the OstreeKernelArgs autoptr cleanup definition to
ostree-autocleanups.h, which will only expose the definitions when
building ostree or if glib is new enough. The include of
ostree-kernel-args.h needs to be moved before ostree-autocleanups.h in
ostree.h so that the OstreeKernelArgs type is declared when the autoptr
cleanup is defined. All the places it's used already pull in libglnx.h
first so that the compat macros are picked up if glib it too old during
the ostree build.

Closes: #1892
Approved by: jlebon
2019-07-27 03:20:21 +00:00
Dan Nicholson d2a9c8604e tests/gpg: Skip tests when subkeys can't be expired
The ability to expire subkeys using gpg's --quick-set-expire is only
available on gnupg 2.1.22. If expiring a subkey fails, assume this is
why and skip the tests that require it but run the actions that the
subsequent tests depend on. This was failing on the Debian Stretch CI
tests since stretch has gnupg 2.1.18.

Closes: #1892
Approved by: jlebon
2019-07-27 03:20:21 +00:00
Dan Nicholson 8ac348d19a tests/gpg: Use exit hook to kill agent in temporary GPG homedir
This wasn't available when I originally wrote this, but it ensures that
the running gpg-agent in tmpgpghome is killed in case the tests exit
early.

Closes: #1892
Approved by: jlebon
2019-07-27 03:20:21 +00:00
Dan Nicholson 615861443b tests/sizes.js: Fix byte array unpacking
Recent GJS changed how byte arrays are unpacked with some assumptions
that they are likely strings. Manually use get_child_value() and
get_byte() to ensure the correct value is parsed when checking the
`ostree.sizes` metadata.

The upstream test is currently passing fine with GJS 1.56.2, but at
Endless we (unfortunately) have a downstream change that adds the object
type as an additional byte in the array. This is parsed incorrectly by
`deep_unpack()`. We can carry this patch downstream, but this change
makes the test more robust regardless.

Closes: #1884
Approved by: cgwalters
2019-07-23 12:49:26 +00:00
Dan Nicholson 2312caad76 tests/sizes: Fix call to commit_transaction()
The GIR for commit_transaction() only has a single argument for the
GCancellable. Calling it with 2 arguments prints a GJS warning:

Gjs-Message: 15:37:40.287: JS WARNING: [/home/dan/src/ostree/tests/test-sizes.js 56]: Too many arguments to method OSTree.Repo.commit_transaction: expected 1, got 2

Currently this is harmless, but it could become a hard error in GJS at
some point.

Closes: #1884
Approved by: cgwalters
2019-07-23 12:49:26 +00:00
Dan Nicholson abb173352d tests: Always cleanup gpg-agent when exiting
Add `libtest_cleanup_gpg()` to the array of commands to run when
exiting. This provides 2 improvements:

1. You don't need to worry about whether the test will spawn a gpg-agent
   and therefore require adding a call to `libtest_cleanup_gpg()`.

2. All the existing users were calling `libtest_cleanup_gpg()` at the
   end of the script. If there was a failure and the script exited
   early, then it wouldn't cleanup and there may be a stray gpg-agent
   hanging around.

Closes: #1799
Approved by: cgwalters
2019-06-19 17:30:24 +00:00
Dan Nicholson 0dd27bbf4b tests/libtest: Allow appending actions to be run on EXIT
Currently if a test script adds a trap on `EXIT` to run some cleanup, it
will stomp on the existing trap to run `save_core()`. Allow for scripts
to append actions that will run on exit by introducing an array that
will be iterated over by a single exit runner.

Closes: #1799
Approved by: cgwalters
2019-06-19 17:30:24 +00:00
Dan Nicholson b6979e7572 lib: Kill GPG agent when cleaning up tmp homedirs
When a temporary directory is used for GPG operations, it's pretty clear
that the running agent will be useless after the directory is deleted.
Call the new `ot_gpgme_kill_agent ()` helper to kill gpg-agent rather
than leaving them it hanging around forever.

As it turns out, gnupg does have code to make gpg-agent automatically
exit when the homedir is removed (https://dev.gnupg.org/T2756), but
that's only available on gnupg 2.2 or newer. Possibly this code can be
dropped later when that's more widely deployed or users/distros have
been advised to backport the necessary changes.

Closes: #1799
Approved by: cgwalters
2019-06-19 17:30:24 +00:00
Dan Nicholson 4670837f7c lib/gpg: Add helper to kill GPG agent
With GnuPG 2, any time you do basically any operation, a gpg-agent will
be spawned for the GPG home directory in use. The classic way to kill a
gpg-agent is to use `gpg-connect-agent` and send the `killagent` command
as is done in libtest.sh.

Closes: #1799
Approved by: cgwalters
2019-06-19 17:30:24 +00:00
Dan Nicholson 0c63f30783 tests/test-gpg-signed-commit: Test more key states
Extend test-gpg-signed-commit.sh to test various key states. If gpg is
found that supports the required options, keys will be generated on the
fly and changed in various ways to exercise the output from
`ostree_gpg_verify_result_describe_variant` used in `ostree show`.

I tested this using gnupg 2.2.12, so I hope it works well enough on
various gpgs found in the wild.

Closes: #1872
Approved by: cgwalters
2019-06-19 17:17:11 +00:00
Dan Nicholson d8649f4d56 tests/libtest: Allow specifying GPG homedir to cleanup
In case the tests want to use a custom GPG homedir, allow passing in the
homedir to use when cleaning up a running gpg-agent.

Closes: #1872
Approved by: cgwalters
2019-06-19 17:17:11 +00:00
Dan Nicholson d0ec319b6b lib/gpg: Show if GPG key has been revoked
Rather then showing the incorrect `BAD signature`.

Closes: #1872
Approved by: cgwalters
2019-06-19 17:17:11 +00:00
Dan Nicholson d714e622b8 lib/gpg: Show information for expired keys
Introduce a new signature attribute for the key expiration timestamp and
display it when the key has a non-zero expiration time. Without this,
the error shown is `BAD signature`, which isn't correct.

Closes: #1872
Approved by: cgwalters
2019-06-19 17:17:11 +00:00
Dan Nicholson 901c2f5e5f lib/gpg: Factor out expiration timestamp formatting
This will be reused to format a key expiration message.

Closes: #1872
Approved by: cgwalters
2019-06-19 17:17:11 +00:00
Dan Nicholson d916383953 bin/refs: Allow overwriting existing ref
Currently if you want to update a non-alias ref, you need to first check
if it exists and use either `ostree refs --create` or `ostree reset` as
appropriate. That's unnecessarily complicated and is much less
convenient than the old `write-refs` builtin that simply called
`ostree_repo_set_ref_immediate()` without any checks.

Add a `--force` option to be used with `--create` that does not raise an
error when the destination ref already exists.

Closes: #1870
Approved by: jlebon
2019-06-12 15:00:19 +00:00
Dan Nicholson b33a4e9b1e remote-add: Add --force option to add or replace remote
This uses the OSTREE_REPO_REMOTE_CHANGE_REPLACE operation to add a
remote or replace an existing one. This is roughly the opposite of
--if-not-exists and will raise an error if both options are passed.

Closes: #1166
Approved by: cgwalters
2019-02-08 14:36:41 +00:00
Dan Nicholson 8431bb5406 man: Document remote add --if-not-exists option
Closes: #1166
Approved by: cgwalters
2019-02-08 14:36:41 +00:00
Dan Nicholson 15ba2007a1 repo: Add OstreeRepoRemoteChange replace operation
Add the OSTREE_REPO_REMOTE_CHANGE_REPLACE operation to the
OstreeRepoRemoteChange enum. This operation will add a remote or replace
an existing one. It respects the location of the remote configuration
file when replacing and the remotes config dir settings when adding a
new remote.

Closes: #1166
Approved by: cgwalters
2019-02-08 14:36:41 +00:00
Dan Nicholson 43d9cac4fc lib/commit: Don't chown objects to repo target owner
The idea is that if the process is running as root, it can change
ownership of newly written files to match the owner of the repo.
Unfortunately, it currently applies in the other direction, too - a
non-root user writing to a root owned repository. If the repo is
writable by the user but owned by root, it can still create files and
directories there, but it can't change ownership of them.

This feature comes from
https://bugzilla.gnome.org/show_bug.cgi?id=738954. As it turns out, this
feature was never completed. It only works on content objects and not
metadata objects, refs, deltas, summaries, etc. Rather than try to fix
all of those, remove the feature until someone has interest in
completing it.

Closes: #1754
Approved by: cgwalters
2018-10-12 12:34:57 +00:00
Dan Nicholson 4e6b13e8b6 repo: Add OSTREE_REPO_TEST_ERROR=invalid-cache env var
Add an invalid-cache test error flag to ensure that the code that checks
for and recovers from a corrupted summary cache is hit. This helps make
sure that the recovery path is actually used without resorting to
G_MESSAGES_DEBUG.

Closes: #1698
Approved by: cgwalters
2018-08-14 13:38:11 +00:00
Dan Nicholson 1b5cb52da2 tests: Test for recovery from corrupted summary cache
Check that recovery from a corrupted summary cache (cached summary
doesn't match cached signature) works.

Closes: #1698
Approved by: cgwalters
2018-08-14 13:38:11 +00:00
Dan Nicholson e5061f54d6 lib/pull: Fetch summary if cached version doesn't match signature
If for some reason the cached summary doesn't match the cached signature
then fetch the remote summary and verify again. Since commit c4c2b5eb
this is unlikely to happen since the summary will only be cached if it
matches the signature. However, if the summary cache has been corrupted
for any other reason then it's best to be safe and fetch the remote
summary again.

This is essentially the corollary to c4c2b5eb. Where that commit helps
you from getting into the corrupted summary cache in the first place,
this helps you get out of it. Without this the client can get wedged
until a prune or the remote server republishes the summary.

Closes: #1698
Approved by: cgwalters
2018-08-14 13:38:11 +00:00
Dan Nicholson 1c69f1ed31 lib/pull: Add debug message when loading summary from cache
This helps when debugging issues with the cached summary handling.

Closes: #1698
Approved by: cgwalters
2018-08-14 13:38:11 +00:00
Dan Nicholson ce4eb12ffb tests: Add tests for remote summary update races
There have been subtle bugs in the past when a client pulls while the
remote server is updating the summary. The client may get the old
summary and new signature or vice versa. Add tests to simulate this
behavior to make sure there aren't regressions in the future.

Closes: #1698
Approved by: cgwalters
2018-08-14 13:38:11 +00:00
Dan Nicholson ce58307757 deploy: Delete .updated file from /etc and /var on new deployments
Systemd units using ConditionNeedsUpdate run if the mtime of .updated in
the specified directory is newer than /usr. Since /usr has an mtime of
0, there's no way to have an older .updated file. Systemd units
typically specify ConditionNeedsUpdate=/etc or ConditionNeedsUpdate=/var
to support stateless systems like ostree.

Remove the file from the new deployment's /etc and the OS's /var
regardless of where they came from to ensure that these systemd units
run when booting new deployments. This will provide a method to run
services only on upgrade.

Closes: #1628
https://bugzilla.gnome.org/show_bug.cgi?id=752950

Closes: #1631
Approved by: cgwalters
2018-06-18 13:21:52 +00:00
Dan Nicholson df7f33e498 lib/prune: Take exclusive repository lock
Add exclusive repository locking to all the pruning entry points. This
ensures that objects and deltas will not be removed while another
process is writing to the repository.

Closes: #1343
Approved by: cgwalters
2017-12-05 02:32:47 +00:00
Dan Nicholson 6d978893f1 lib/commit: Add repository locking during transactions
Take a shared repo lock during a transaction to ensure that another
process doesn't delete objects.

Closes: #1343
Approved by: cgwalters
2017-12-05 02:32:47 +00:00
Dan Nicholson 7d863ed9e4 lib/repo: Add locking auto cleanup handler
Define an auto cleanup handler for use with repo locking. This is based
on the existing auto transaction cleanup. A wrapper for
ostree_repo_lock_push() is added with it. The intended usage is like so:

  g_autoptr(OstreeRepoAutoLock) lock = NULL;
  lock = ostree_repo_auto_lock_push (repo, lock_type, cancellable, error);
  if (!lock)
    return FALSE;

The functions and type are marked to be skipped by introspection since I
can't see them being usable from bindings.

Closes: #1343
Approved by: cgwalters
2017-12-05 02:32:47 +00:00
Dan Nicholson 4e78ddd2da lib/repo: Add repo locking mechanism
Currently ostree has no method of guarding against concurrent pruning.
When there are multiple repo writers, it's possible to have a pull or
commit race against a prune and end up with missing objects.

This adds a file based repo locking mechanism. The intention is to take
a shared lock when writing objects and an exclusive lock when deleting
them. In order to make use of the locking throughout the library in a
fine grained fashion, the lock acts recursively with a stack of lock
states. If the lock becomes exclusive, it will stay in that state until
the stack is unwound past the initial exclusive push. The file locking
is similar to GLnxLockFile in that it uses open file descriptor locks
but falls back to flock when needed.

The lock also attempts to be thread safe by storing the lock state in
thread local storage with GPrivate. This means that each thread will
have an independent lock for each repository it opens. There are some
drawbacks to that, but it seemed impossible to manage the lock state
coherently in the face of multithreaded access.

The API is a push/pop interface in accordance with the recursive nature
of the locking. The push interface uses an enum that's translated to
LOCK_SH or LOCK_EX as needed. Both interfaces use an internal timeout
field to decide whether to manage the lock in a blocking or non-blocking
fashion. The intention is to allow ostree applications as well as
administrators to control this timeout. For now, the default is a 30
second timeout.

Note that the timeout is handled synchronously in thread since the lock
is maintained in thread local storage. I.e., the thread that acquires
the lock needs to be the same thread that runs the operation. There may
be a way to offer an asynchronous version, but it's not clear exactly
how that would work since it would likely involve a separate thread that
invokes a callback when the locking operation completes.

https://bugzilla.gnome.org/show_bug.cgi?id=759442

Closes: #1343
Approved by: cgwalters
2017-12-05 02:32:47 +00:00
Dan Nicholson 681a62b92c ci: Really show test-suite.log on travis
The test-suite.log file is created in the top directory, not in test
(which isn't even a directory here).

Closes: #1352
Approved by: dbnicholson
2017-12-01 19:00:18 +00:00
Dan Nicholson afebb5c336 tests: Run python tests with stdout unbuffered
Set the PYTHONUNBUFFERED environment variable during tests so that
python leaves stdout unbuffered. This is helpful when reading logs for
failures since the interleaved stdout and stderr will generally come out
in the right order. It's not perfect since tap-driver.sh does some
special redirection to the log file, but it's an improvement.

Closes: #1352
Approved by: dbnicholson
2017-12-01 19:00:18 +00:00
Dan Nicholson 162edf71ed lib/repo: Don't delete new tmpdir if it can't be locked
If a newly allocated tmpdir can't be locked, set initialized to FALSE so
that glnx_tmpdir_cleanup doesn't delete it when new_tmpdir goes out of
scope.

Closes: #1346
Approved by: cgwalters
2017-11-17 18:25:22 +00:00
Dan Nicholson bf85f8d89e lib/repo: Handle race with existing tmpdir being deleted
Another tmpdir user may have deleted an existing tmpdir between the time
the current user called readdir and tried to open it.

Closes: #1346
Approved by: cgwalters
2017-11-17 18:25:22 +00:00
Dan Nicholson f246287010 lib/repo: Restore tmpdir reusing out parameter
This got lost in d0b0578 and now the caller always thinks it got a new
tmpdir.

Closes: #1346
Approved by: cgwalters
2017-11-17 18:25:22 +00:00
Dan Nicholson c60f319629 lib/repo: Add debug messages when allocating tmpdir
This code is pretty complex and has some races when reusing tmpdirs, so
print some messages for debugging.

Closes: #1346
Approved by: cgwalters
2017-11-17 18:25:22 +00:00
Dan Nicholson 3e8b7e29fa Revert "lib/pull: Skip ostree_repo_resolve_keyring_for_collection for bindings"
This reverts commit 519b30b7e1. Now that
the experimental GIR is being built correctly and OstreeRemote is a real
boxed type, this can be exposed again.

Closes: #1337
Approved by: pwithnall
2017-11-10 10:03:44 +00:00
Dan Nicholson ed242cdd3b lib: Include OstreeRemote and OstreeCollectionRef in GIR
Now that g-ir-scanner is being told about ENABLE_EXPERIMENTAL_API, it
can include these types correctly. Drop the __GI_SCANNER__ guards in the
header files so that all the declarations are found.

After this, you can actually construct the types normally:

>>> OSTree.CollectionRef.new('com.example.Foo', 'bar')
<OSTree.CollectionRef object at 0x7f2bba4c7528 (OstreeCollectionRef at 0x55c033ff2f30)>

Closes: #1337
Approved by: pwithnall
2017-11-10 10:03:44 +00:00
Dan Nicholson a256b2d1a3 lib/remote: Export ostree_remote_get_type symbol
Without this, you can't really use OstreeRemote as a GObject, which is a
requirement for bindings.

This was found when attempting to include OstreeRemote in the GIR, and
g-ir-scanner wasn't able to link it's temporary object due to an
"undefined reference to `ostree_remote_get_type'" error.

Closes: #1337
Approved by: pwithnall
2017-11-10 10:03:44 +00:00
Dan Nicholson 374f7fc973 bin/summary: Fix --raw option
I wanted to inspect a summary file the other day and was saddened to
find it was broken:

  $ ostree summary --raw
  error: No option specified; use -u to update summary

Fix the test to do the normal thing of passing just --raw without
--view. It's legal to pass --raw and --view, but it shouldn't be a
requirement.

Closes: #1336
Approved by: cgwalters
2017-11-09 17:15:59 +00:00
Dan Nicholson 2e9fea8fa5 tests: Don't symlink rofiles-fuse if it's disabled
Creating the symlink will cause make to try to build rofiles-fuse, which
will fail if it's disabled. Normally I wouldn't disable rofiles-fuse,
but it's triggering a hang in our ARM Xenial builder's kernel in splice.
I'm sure that's fuse's fault, but for now I just need to disable
rofiles-fuse there and found --disable-rofiles-fuse didn't actually
work.

Closes: #1325
Approved by: cgwalters
2017-11-03 22:20:00 +00:00
Dan Nicholson 519b30b7e1 lib/pull: Skip ostree_repo_resolve_keyring_for_collection for bindings
Since ostree_remote_get_type is not made available to g-ir-scanner, it
treats OstreeRemote as a bare struct. That's not kosher for bindings and
it issues the following warning:

  src/libostree/ostree-repo-pull.c:5560: Warning: OSTree:
  ostree_repo_resolve_keyring_for_collection: return value: Invalid
  non-constant return of bare structure or union; register as boxed type
  or (skip)

For now, just skip this API for bindings.

Closes: #1322
Approved by: pwithnall
2017-11-03 22:05:47 +00:00
Dan Nicholson 03bbe45530 lib/core: Fix documentation comment in ostree_validate_collection_id
g-ir-scanner was spitting this warning:

  src/libostree/ostree-core.c:281: Warning: OSTree:
  ostree_validate_collection_id: unknown parameter 'rev' in
  documentation comment, should be 'collection_id'

Closes: #1322
Approved by: pwithnall
2017-11-03 22:05:47 +00:00
Dan Nicholson 69c4737491 build: Define OSTREE_ENABLE_EXPERIMENTAL_API for g-ir-scanner
When compiling libostree, OSTREE_ENABLE_EXPERIMENTAL_API is managed via
config.h. However, g-ir-scanner can't use that since it gets confused
about the namespace of all the random macros. It won't include the
experimental APIs unless the macro is defined through another means.
Without this, none of the experimental APIs were being included in the
gir data.

Closes: #1322
Approved by: pwithnall
2017-11-03 22:05:47 +00:00
Dan Nicholson 6ec4cd3eb5 build: Ensure enumtypes.h is built before enumtypes.c
ostree-enumtypes.c includes ostree-enumtypes.h, so make needs to be told
about the dependency. Without it, parallel make could try to build
ostree-enumtypes.c before the header file exists.

I hit this when running `make -j OSTree-1.0.gir`.

Closes: #1322
Approved by: pwithnall
2017-11-03 22:05:47 +00:00
Dan Nicholson 63ce86d597 lib/repo: Properly handle NULL homedir when signing commit
Without this, ostree_repo_sign_commit throws a critical message when no
homedir is provided:

(ostree gpg-sign:5034): GLib-GIO-CRITICAL **: g_file_new_for_path: assertion 'path != NULL' failed

Closes: #1305
Approved by: cgwalters
2017-10-24 19:58:07 +00:00
Dan Nicholson 3f3d3d64ac build: Do *.am.inc substitutions from make correctly
The string needs to be escaped with a $ since we want to replace the
literal string $(path). Without this make will run the command with the
value of the variable, which won't match anything in the input Makefile
stub.

Closes: #1291
Approved by: smcv
2017-10-18 16:53:29 +00:00
Dan Nicholson 9b3e01be30 build: Fix bsdiff Makefile circular dependency
The intended use was to have the .am.inc generated from the .am like the
libglnx one. Without this, make was detecting a circular dependency and
dropping the rule:

  make: Circular bsdiff/Makefile-bsdiff.am.inc <- bsdiff/Makefile-bsdiff.am.inc dependency dropped.

Closes: #1276
Approved by: jlebon
2017-10-16 16:40:11 +00:00
Dan Nicholson a5b7660c94 lib/deploy: Ignore FIFREEZE/FITHAW errors when already in state
If the filesystem is already frozen, FIFREEZE returns EBUSY, and if the
filesystem is already thawed, FITHAW returns EINVAL. It's very unlikely
these issues would arise on a real ostree system since the sysroot would
be locked during the freeze/thaw cycle.

However, when multiple fake sysroots are used during the test suite (run
as root), the tests could race to run the freeze/thaw cycle without
locking. Furthermore, there's no reason why an independent process might
be trying to freeze the filesystem while ostree was deploying. Ignore
but warn for these errors since there's not much ostree can do about it,
anyways.

Closes: #1260
Approved by: cgwalters
2017-10-11 15:40:20 +00:00
Dan Nicholson 3b315e16d8 repo: Ensure new config doesn't set remotes in separate file
If the new configuration passed to ostree_write_config () tries to
update options for a remote defined in a separate config file, return an
error. Without this, the full configuration would contain duplicate
remote specifications, which would raise an error the next time the repo
is opened.

Closes: #1159
Approved by: cgwalters
2017-09-13 16:03:25 +00:00
Dan Nicholson adac42b6ef repo: Add add-remotes-config-dir option
This option allows a repo to explicitly opt out of adding new remotes in
a remotes configuration directory. This currently defaults to true for
system repos and false for non-system repos to maintain legacy behavior
that non-system repos don't add remotes in a configuration directory.
That would be problematic for flatpak, which specifies a remotes config
dir but adds remotes in ways that are incompatible with it.

So, what this really does is allow system repos to control whether they
want to add remotes in the config dir or not. That's important if your
flatpak repo is the system repo like at Endless.

Closes: #1134

Closes: #1155
Approved by: cgwalters
2017-09-11 10:53:20 +00:00
Dan Nicholson ccbbf77c71 sysroot: Reload config after setting sysroot kind
This allows any repo configuration defaults to be set based on whether
it's a system repo or not.

Closes: #1155
Approved by: cgwalters
2017-09-11 10:53:20 +00:00
Dan Nicholson 43c78c9006 repo: Fix non-system remotes-config-dir usage
Before commit e0346c1, a non-system repo could specify
remotes-config-dir and have remotes read from there. However, adding
remotes would only be done in the config dir for a system repo. Restore
that by respecting remotes-config-dir when no sysroot is found and
adding back the ostree_repo_is_system() check when adding remotes.

Closes: #1133

Closes: #1151
Approved by: cgwalters
2017-09-08 13:54:30 +00:00
Dan Nicholson fa4e4bf4df tests: Look for trivial-httpd in $libexecdir
Since b825aac, trivial-httpd is in $libexecdir/libostree by default and
not available through the ostree runner in PATH. Try to adjust find it
when running the tests installed.

Closes: #837
Approved by: dbnicholson
2017-05-08 18:34:10 +00:00
Dan Nicholson 1fe914755a tests: Install libtest-core.sh with installed tests
Without this, running the installed tests fails dramatically. Remove it
from EXTRA_DIST since dist_installed_test_data takes care of dist.

Closes: #837
Approved by: dbnicholson
2017-05-08 18:34:10 +00:00
Dan Nicholson 4e13361c8e pull: Allow additional HTTP headers for summary fetch
Read the http-headers (a(ss)) option in
ostree_repo_remote_fetch_summary_with_options like
ostree_repo_pull_with_options and add the headers to the fetcher. This
allows things like providing additional authorization headers to the
HTTP requests.

Closes: #839
Approved by: cgwalters
2017-05-08 16:50:45 +00:00
Dan Nicholson 37b8dae2c4 commit: Mark ostree_repo_transaction_set_ref* checksums nullable
Allow GI bindings to delete refs through ostree_repo_transaction_set_ref
and ostree_repo_transaction_set_refspec by setting the checksum to NULL.

Closes: #834
Approved by: cgwalters
2017-05-08 16:35:09 +00:00
Dan Nicholson 59897f2b84 pull: Fix crash specifying override URL in summary fetch
The summary URL override is looked up with "&s", which directly
exchanges the data to a pointer without allocation. This was causing a
segfault calling ostree_repo_remote_fetch_summary_with_options from
pygobject.

Closes: #829
Approved by: jlebon
2017-05-03 18:08:37 +00:00
Dan Nicholson 300752e55a repo: Fix list_objects annotations
Without the element-type annotations, bindings don't know how to handle
the elements of the hash table. Since the table is created with destroy
functions, the caller does not own the elements, so transfer container
is used.

Closes: #635
Approved by: cgwalters
2016-12-22 16:07:52 +00:00
Dan Nicholson dd3cda401b repo: Fix object list keys ownership
ostree_object_name_serialize returns a floating ref, so sink it before
adding it to the hash table so it can properly be freed later when the
hash table is destroyed.

This is particularly a problem for pygobject, which sinks the refs on
variants as it marshals them to native python types. If the ref isn't
already sunk, then the ref count won't increase and a critical warning
will be raised when both the hash table and pygobject try to unref it.

Closes: #635
Approved by: cgwalters
2016-12-22 16:07:52 +00:00