Commit Graph

2028 Commits

Author SHA1 Message Date
Jonathan Lebon 72a54fa877 lib/config: Deprecate commit-update-summary option
Now that we have `auto-update-summary`, there is no point in having
`commit-update-summary`. The latter also only had an effect through
the `commit` CLI command, whereas the former is embedded directly in
libostree.

There is one corner case that slips through: `commit` would update the
summary file even if orphan commits were created, which we no longer do
here. I can't imagine anyone relying on this, so it seems safe to drop.

Closes: #1689

Closes: #1693
Approved by: mwleeds
2018-08-01 19:59:07 +00:00
Jonathan Lebon 786ee6bdec lib/config: Rename change-update-summary to auto-...
Mildly bikeshed, though I find the name `auto-update-summary` to be
easier to grok than `change-update-summary`. I think it's because it can
be read as "verb-verb-noun" rather than "noun-verb-noun".

Closes: #1693
Approved by: mwleeds
2018-08-01 19:59:07 +00:00
Jonathan Lebon 3e96ec9811 lib/refs: Use GLNX_HASH_TABLE_FOREACH_KV helper
Closes: #1693
Approved by: mwleeds
2018-08-01 19:59:07 +00:00
Matthew Leeds daa57b4630 lib/repo-pull: Use correct keyring for dynamic remotes
Normally, a configured remote will only serve refs with one associated
collection ID, but temporary remotes such as USB drives or LAN peers can
serve refs from multiple collection IDs which may use different GPG
keyrings. So the OstreeRepoFinderMount and OstreeRepoFinderAvahi classes
create dynamic OstreeRemote objects for each (uri, keyring) pair. So if
for example the USB mounted at /mnt/usb serves content from the
configured remotes "eos-apps" and "eos-sdk", the OstreeRepoFinderResult
array returned by ostree_repo_find_remotes_async() will have one result
with a remote called something like
file_mnt_usb_eos-apps.trustedkeys.gpg and the list of refs on the USB
that came from eos-apps, and another result with a remote
file_mnt_usb_eos-sdk.trustedkeys.gpg and the list of refs from eos-sdk.

Unfortunately while OstreeRepoFinderMount and OstreeRepoFinderAvahi
correctly only include refs in a result if the ref uses the associated
keyring, the find_remotes_cb() function used to clean up the set of
results looks at the remote summary file and includes every ref that's
in the intersection with the requested refs, regardless of whether it
uses a different remote's keyring. This leads to an error when you try
to pull from a USB containing refs from different collection IDs: the
pull using the wrong collection ID will error out with "Refspec not
found" and the result with the correct keyring will then be ignored "as
it has no relevant refs or they have already been pulled." So the pull
ultimately fails.

This commit fixes the issue by filtering refs coming from a dynamic
remote, so that only ones with the collection ID associated with the
keyring remote are examined. This only needs to be done for dynamic
remotes because you should be able to pull any ref from a configured
remote using its keyring. It's also only done when looking at the
collection map in the summary file, because LAN/USB remotes won't have a
"main" collection ID set (OSTREE_SUMMARY_COLLECTION_ID).

Closes: #1695
Approved by: pwithnall
2018-08-01 13:57:10 +00:00
Matthew Leeds 6869bada49 config: Add a core/change-update-summary option
This commits adds and implements a boolean repo config option called
"change-update-summary" which updates the summary file every time a ref
changes (additions, updates, and deletions).

The main impetus for this feature is that the `ostree create-usb` and
`flatpak create-usb` commands depend on the repo summary being up to
date. On the command line you can work around this by asking the user to
run `ostree summary --update` but in the case of GNOME Software calling
out to `flatpak create-usb` this wouldn't work because it's running as a
user and the repo is owned by root. That strategy also means flatpak
can't update the repo metadata refs for fear of invalidating the
summary.

Another use case for this relates to LAN updates. Specifically, the
component of eos-updater that generates DNS-SD records advertising ostree
refs depends on the repo summary being up to date.

Since ostree_repo_regenerate_summary() now takes an exclusive lock, this
should be safe to enable. However it's not enabled by default because of
the performance cost, and because it's more useful on clients than
servers (which likely have another mechanism for updating the summary).

Fixes https://github.com/ostreedev/ostree/issues/1664

Closes: #1681
Approved by: jlebon
2018-07-30 17:19:12 +00:00
Matthew Leeds fb36b62f33 lib/repo: Take exclusive lock while generating summary
This ensures that commits aren't deleted and refs aren't added, removed,
or updated while the summary is being generated. This is in preparation
for adding a repo config option that will automatically regenerate the
summary on every ref change.

Closes: #1681
Approved by: jlebon
2018-07-30 17:19:12 +00:00
Jonathan Lebon 968e8805b0 lib: Fix some logic/error-checking code
Using `MAX(0, $x)` here is useless since we're comparing against an
unsigned integer. Just unpack this and only subtract if it's safe to do
so.

Also, explicitly check for `fd >= 0` rather than just `!= -1` to be sure
it's a valid fd. And finally, explicitly check the return value of
`g_input_stream_read_all` as is done everywhere else in the tree and
make it clear that we're purposely ignoring the return value of `_flush`
here, but not in other places.

Discovered by Coverity.

Closes: #1692
Approved by: cgwalters
2018-07-26 21:01:19 +00:00
Jonathan Lebon fcd31a195b lib: Fix some minor memory leaks
I initially was going to add a `G_DEFINE_AUTOPTR_CLEANUP_FUNC` for
`FetchStaticDeltaData`, but it honestly didn't seem worth mucking around
ownership everywhere and potentially getting it wrong.

Discovered by Coverity.

Closes: #1692
Approved by: cgwalters
2018-07-26 21:01:19 +00:00
Jonathan Lebon 9482922e5e lib: Check for NULL pointers in some more places
In `write_metadata_object()`, make sure when creating tombstone commits
that we're actually passed an expected checksum to use.

In `write_dir_entry_to_mtree_internal()`, sanity check that `dfd_iter`
is indeed not `NULL` before trying to dereference it.

Discovered by Coverity.

Closes: #1692
Approved by: cgwalters
2018-07-26 21:01:19 +00:00
Matthew Leeds be07c04e63 lib/repo-commit: Fix min-free-space error message
Since min_free_space_size_mb is considered before min_free_space_percent
in min_free_space_calculate_reserved_blocks(), it has to be considered
first when generating the error message in order for it to be accurate.

Closes: #1691
Approved by: jlebon
2018-07-25 13:16:18 +00:00
Colin Walters 93da568422 lib/pull: Fix minor memleak in error path
Spotted by a downstream Coverity build.

Closes: #1684
Approved by: jlebon
2018-07-20 20:32:17 +00:00
Umang Jain bbb253238a Post-release version bump
Closes: #1683
Approved by: cgwalters
2018-07-20 19:43:18 +00:00
Umang Jain 21318bbc1f Release 2018.7
Request via flatpak: mainly to port min-free-space-size

Closes: #1683
Approved by: cgwalters
2018-07-20 19:43:18 +00:00
Umang Jain 66079c7b65 lib/repo: Allow min-free-space-size and -percent to co-exist
Previously, we would error out if both of the options were mentioned
in the config file (even if one of them is disabled with 0). There
were few suggestions that this behavior was not quite right.

Therefore, instead of throwing error and exiting, it's preferred to
warn the user. Hence, the solution that worked out is:
* Allow both options to exist simulateneously
* Check each config's value and decide:
  * If both are present and are non-zero, warn the user. Also, prefer
    to use min-free-space-size over the another.
  * If both are absent, then use -percent=3% as fallback
  * Every other case is valid hence, no warning

https://phabricator.endlessm.com/T13698
(cherry picked from commit be68991cf80f0aa1da7d36ab6e1d2c4d6c7cd3fb)
Signed-off-by: Robert McQueen <rob@endlessm.com>

Closes: #1685
Approved by: cgwalters
2018-07-20 14:58:40 +00:00
Colin Walters fef07889d3 deploy: Fix overriding kernel args for staged deployments
This is the inverse of https://github.com/ostreedev/ostree/pull/1558
aka commits cadece6c4f398ca61d21e497bd6e3fbb549f9cf6 and
3358698c86d80821d81443c906621c92672f99fb

Needed to fix `rpm-ostree kargs` test suite with default staging; skipping
a test here for now as eventually what we'll do is turn on the rpm-ostree
suite fully here.

Closes: #1677
Approved by: jlebon
2018-07-09 18:04:57 +00:00
William Manley c7b12a8730 ostree repo commit: Speed up composing trees with `--tree=ref`
Running `ostree commit --tree=ref=a --tree=dir=b` involves reading the
whole of a into an `OstreeMutableTree` before composing `b` on top.  This
is inefficient if a is a complete rootfs and b is just touching one file.
We process O(size of a + size of b) directories rather than
O(number of touched dirs).

This commit makes `ostree commit` more efficient at composing multiple
directories together.  With `ostree_mutable_tree_fill_empty_from_dirtree`
we create a lazy `OstreeMutableTree` which only reads the underlying
information from disk when needed.  We don't need to read all the
subdirectories just to get the checksum of a tree already checked into the
repo.

This provides great speedups when composing a rootfs out of multiple other
rootfs as we do in our build system.  We compose multiple containers
together with:

    ostree commit --tree=ref=base-rootfs --tree=ref=container1 --tree=ref=container2

and it is much faster now.

As a test I ran

    time ostree --repo=... commit --orphan --tree=ref=big-rootfs --tree=dir=modified_etc

Where modified_etc contained a modified sudoers file under /etc.  I used
`strace` to count syscalls and I seperatly took timing measurements.  To
test with a cold cache I ran

    sync && echo 3 | sudo tee /proc/sys/vm/drop_caches

Results:

|                      | Before | After |
| -------------------- | ------ | ----- |
| Time (cold cache)    |   8.1s | 0.12s |
| Time (warm cache)    |   3.7s | 0.08s |
| `openat` calls       |  53589 |   246 |
| `fgetxattr` calls    |  78916 |     0 |

I'm not sure if this will have some negative interaction with the
`_ostree_repo_commit_modifier_apply` which is short-circuited here.  I
think it was disabled for `--tree=ref=` anyway, but I'm not certain.  All
the tests pass anyway.

I originally implemented this in terms of the `OstreeRepoFile` APIs, but
it was *way* less efficient, opening and reading many files unnecessarily.

Closes: #1643
Approved by: cgwalters
2018-07-09 13:10:51 +00:00
Umang Jain eeacbc6b29 repo: Reword min-free-space-size option's error strings
It is important that we use user-friendly error strings. The reason
being error strings are seen by users such as in GNOME Software's
error banner.

Closes: #1671
Approved by: jlebon
2018-07-06 19:59:11 +00:00
Umang Jain 4c023a9585 lib/repo-commit: Factor out min-free-space-size error reporting
Improves code readability.

Closes: #1671
Approved by: jlebon
2018-07-06 19:59:10 +00:00
Colin Walters 7468600029 deploy: Retain staged by default
For `rpm-ostree ex livefs` we have a use case of pushing a rollback
deployment.  There's no reason this should require deleting the staged
deployment (and doing so actually breaks livefs which tries to access
it as a data source).

I was initially very conservative here, but I think it ends up
being fairly easy to retain the staged deployment.  We need to handle
two cases:

First, when the staged is *intentionally* deleted; here, we just need
to unlink the `/run` file, and then everything will be sync'd up after
reloading.

Second, (as in the livefs case) where we're retaining it,
e.g. adding a deployment to the end.  What I realized here is that
we can have the code keep `new_deployments` as view without staged,
and then when we do the final reload we'll end up re-reading it from
disk anyways.

Closes: #1672
Approved by: jlebon
2018-07-06 15:23:52 +00:00
Umang Jain 0c8b86ea09 lib/repo: Minor fixes around min-free-space
Summary:
* Remove a useless if condition in prepare_transaction()
* Fix glnx_throw error propagation
* Integer overflow check while parsing min-free-space-size config
* Documentation fixes

Closes: #1663
Approved by: jlebon
2018-07-03 12:59:26 +00:00
William Manley 488365f9bf OstreeMutableTree: Invalidate parent contents checksum when metadata changes
This bug has existed before the previous commit, but thanks to the previous
commit it is now easy to fix.

Closes: #1655
Approved by: cgwalters
2018-06-29 21:31:08 +00:00
William Manley 5b0dd1002e OstreeMutableTree: Refactor: Add `parent` pointer
This implements a TODO item from
`ostree_mutable_tree_get_contents_checksum`.  We now no-longer invalidate
the dirtree contents checksum at `get_contents_checksum` time - we
invalidate it when the mtree is modified.  This is implemented by keeping
a pointer to the parent directory in each `OstreeMutableTree`.  This gives
us stronger invariants on `contents_checksum`.

For even stronger guarantees about invariants we could make
`ostree_repo_write_mtree` or similar a member of `OstreeMutableTree` and
remove `ostree_mutable_tree_set_metadata_checksum`.

I think I've fixed a bug here too.  We now invalidate parent's contents
checksum when our metadata checksum changes, whereas we didn't before.

Closes: #1655
Approved by: cgwalters
2018-06-29 21:31:08 +00:00
Philip Withnall abff8b8cfa lib/repo-commit: Abort a transaction if preparing it fails
If ostree_repo_prepare_transaction() fails, we should reset the
repository’s state so that the failed call was essentially idempotent.
Do that by calling ostree_repo_abort_transaction() on the failure path.

Typically, the way for preparing a transaction to fail is for its
GCancellable to be triggered, rather than because any of the operations
involved in preparing a transaction are particularly failure prone.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1647
Approved by: cgwalters
2018-06-29 19:32:44 +00:00
Colin Walters 7ead3c1aa8 sysroot: Reject attempts to pin the staged deployment
From https://github.com/projectatomic/rpm-ostree/pull/1434#discussion_r198936674

To support it we'd have to actually write it to disk, which...let's
not try that right now.

Closes: #1660
Approved by: jlebon
2018-06-29 01:52:30 +00:00
Umang Jain d686056254 lib/repo: Cleanup current boot's staging dir min-free-space-* checks are hit
min-free-space-* act as a gating condition whether to we want hold onto caches in
repo/tmp. If it is found that the free-disk space is going below this threshold,
we flag it as an error and cleanup current boot's staging directory.

Closes: #1602
Approved by: jlebon
2018-06-27 19:02:02 +00:00
Umang Jain 1074668ede lib/repo: cleanup_tmpdir should be executed after releasing lock file
Here's a subtle bug in abort_transaction():
One of the policies of cleaning up is to skip the current boot's staging
directory. The responsible function for this is cleanup_tmpdir() which tries
to lock each of the tmpdir before deleting it. When it comes to the current
boot's staging dir, it tries to lock the directory(again!) but fails as there
is already a lockfile present. Just because the current boot's staging dir was
meant to be skipped, the bug never surfaced up and wasn't catastrohpic.

if (!_ostree_repo_try_lock_tmpdir (dfd, path, &lockfile, &did_lock, error))
  return FALSE;
if (!did_lock)
  return TRUE; /* Note early return */
...
if (g_str_has_prefix (path, self->stagedir_prefix))
  return TRUE; /* Note early return */

The actual check for skipping staging dir for current boot was never reached
because the function returned at did_lock failure.

Therefore, execute cleanup_tmpdir() after releasing the lockfile in
abort_transaction() so that cleanup_tmpdir gets a chance to lock current boot's
staging directory and succeed.

Closes: #1602
Approved by: jlebon
2018-06-27 19:02:02 +00:00
Javier Martinez Canillas 9f48e212a3 deploy: Change BootLoaderSpec filenames so they can be used for sorting
Currently the BLS snippets are named ostree-$ID-$VARIANT_ID-$index.conf,
but the BLS config files are actually sorted by using the version field
which is the inverse of the index.

In most places, _ostree_sysroot_read_boot_loader_configs() is used to
get the BLS files and this function already returns them sorted by the
version field. The only place where the index trailing number is used is
in the ostree-grub-generator script that lists the BLS files to populate
the grub config file.

But for some bootloaders the BLS filename is the criteria for sorting by
taking the filename as a string version. So on these bootloaders the BLS
entries will be listed in the reverse order.

To avoid that, change the BLS snippets filename to have the version field
instead of the index and also to have the version before deployment name.

Make the filenames to be of the form ostree-$version-$ID-$VARIANT_ID.conf
so the version is before the deployment name.

Signed-off-by: Javier Martinez Canillas <javierm@redhat.com>

Closes: #1654
Approved by: cgwalters
2018-06-27 18:08:28 +00:00
Simon McVittie 47ae4f5c7e OstreeRepoFinderConfig: Fix guint/gsize confusion
If a function has a guint "out argument", passing a pointer to a gsize
is not, in general, valid. On an ILP64 platform there is no problem
since guint and gsize are identical, but on an LP64 platform it will
overwrite only the first word of the gsize, leaving the second word
unaffected. On little-endian machines, if the second word is
zero-initialized (as it is here), the result is numerically equal to
the guint, but on big-endian machines the result is around 4 billion
times what it should be, resulting in
ostree_repo_finder_config_resolve_async() reading past the end of
the array and causing undefined behaviour.

In practice this caused assertion failures (and consequently test
failures) on Debian's s390x (z/Architecture), ppc64 (64-bit PowerPC)
and sparc64 (64-bit SPARC) ports.

Closes: #1640
Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #1641
Approved by: cgwalters
2018-06-27 15:24:18 +00:00
William Manley 5190f1df42 OstreeMutableTree: Document ostree_mutable_tree_ensure_dir
Closes: #1645
Approved by: jlebon
2018-06-25 16:22:21 +00:00
William Manley ca8571a49b OstreeMutableTree: Document each private member of `OstreeMutableTree`
A prelude to my understanding.  Unfortunately `OstreeMutableTree` provides
little encapsulation, as each member has setters† so it's difficult to come
up with a list of invariants.

† `files` and `subdirs` only have getters, but the getters return mutable
  references to the internals, so we still can't reason about invariants.

Closes: #1645
Approved by: jlebon
2018-06-25 16:22:21 +00:00
Simon McVittie e120a6b119 avahi: Fail immediately if we can't talk to D-Bus or Avahi
We special-case AVAHI_ERR_NO_DAEMON to not cause warnings, but if
we pass AVAHI_CLIENT_NO_FAIL to avahi_client_new, we never actually
see AVAHI_ERR_NO_DAEMON. Instead, we will get AVAHI_ERR_BAD_STATE
when we try to use the client.

Closes: #1618
Signed-off-by: Simon McVittie <smcv@debian.org>

Closes: #1639
Approved by: cgwalters
2018-06-23 14:54:39 +00:00
Umang Jain 095376efa2 lib/repo: Enforce min-free-space-* size check for regfiles in deltas
During the pull, there is an explicit check for free space on disk
vs. the size of uncompressed delta; But while writing the new content
objects that are generated, they have to honor min-free-space-* checks
too. We enforce this check in _bare_content_commit as that is where
we can know the final size of the new content object.

Closes: #1614
Approved by: jlebon
2018-06-22 21:01:56 +00:00
Colin Walters 1174d9f5ba lib/repo: Fix 32 bit format string error 2018-06-21 11:33:23 -04:00
Jonathan Lebon 603c1258cc Post-release version bump 2018-06-21 11:23:40 -04:00
Colin Walters 31a356dca9 Release 2018.6 2018-06-19 15:48:14 -04:00
Colin Walters 5e9d382811 lib/repo: Do free space math under lock in error path
We were referencing the txn bits outside of the lock in the error
path. Generally shouldn't matter, but e.g. Rust wouldn't let us do this, and
race detector tooling will warn about it.

Closes: #1632
Approved by: jlebon
2018-06-19 18:29:31 +00:00
Colin Walters acab2c1ac6 lib/repo: Rename free_space_size variable to free_space_mb
I generally like having variables include their units where applicable;
timer variables having `_secs` or `_ms`, etc.

Closes: #1632
Approved by: jlebon
2018-06-19 18:29:31 +00:00
Colin Walters a2b08f9342 lib/repo: Fix double-set-error in min-free-space-size code
We need to pass `NULL` as the error, we only care if the key exists;
otherwise we'll try to set the error twice.

Closes: #1632
Approved by: jlebon
2018-06-19 18:29:31 +00:00
Colin Walters cc5254ac34 lib/archive: Tell g-ir-scanner to ignore the private libarchive bits
Squashes this warning:
```
src/libostree/ostree-libarchive-private.h:46: syntax error, unexpected typedef-name in '  g_autoptr(OtAutoArchiveRead) a = archive_read_new ();' at 'OtAutoArchiveRead'
```

Closes: #1629
Approved by: jlebon
2018-06-18 16:40:39 +00:00
Dan Nicholson ce58307757 deploy: Delete .updated file from /etc and /var on new deployments
Systemd units using ConditionNeedsUpdate run if the mtime of .updated in
the specified directory is newer than /usr. Since /usr has an mtime of
0, there's no way to have an older .updated file. Systemd units
typically specify ConditionNeedsUpdate=/etc or ConditionNeedsUpdate=/var
to support stateless systems like ostree.

Remove the file from the new deployment's /etc and the OS's /var
regardless of where they came from to ensure that these systemd units
run when booting new deployments. This will provide a method to run
services only on upgrade.

Closes: #1628
https://bugzilla.gnome.org/show_bug.cgi?id=752950

Closes: #1631
Approved by: cgwalters
2018-06-18 13:21:52 +00:00
Matthew Leeds 7dc3e45b3a lib/prune: Don't modify dirent->d_name in place
Currently when I run `ostree prune` it hits a seg fault when the
hash_func is used (in this case g_str_hash) from the call stack
_ostree_repo_prune_tmp() -> g_hash_table_contains() ->
g_hash_table_lookup_node(). So the key, in this case dent->d_name, must
be corrupt in some way.

glnx_dirfd_iterator_next_dent() uses readdir() to get the dirent struct.
And according to the man page for readdir(3), "POSIX.1 explicitly notes
that this field should not be used as an lvalue" (in reference to
d_name). So this commit avoids modifying d_name in place and copies it
instead. This seems to avoid the seg fault.

Closes: #1627
Approved by: jlebon
2018-06-15 19:01:46 +00:00
Alexander Larsson 677e181025 ostree_repo_resolve_rev: Resolve refs set in the transaction
This allows you to get at the current commit for a ref pending in the transaction.

Closes: #1624
Approved by: cgwalters
2018-06-14 17:58:47 +00:00
Philip Withnall 2d2f218669 lib/repo-commit: Delay propagation of errors from abort_transaction()
If there’s a problem while aborting a transaction, store the error but
don’t report it until the end of the function — do a best effort at
clearing the rest of the transaction state first (since most of it
cannot fail).

If cleanup_tmpdir() fails (which, arguably, should not be a
showstopper), this allows a caller to recover and start a new
transaction in future.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1626
Approved by: jlebon
2018-06-14 17:13:43 +00:00
Umang Jain 31809d32f2 lib/repo: Add min-free-space-size option
Similar to min-free-space-percent but it supports specific sizes
(in MB, GB or TB). Also, making min-free-space-percent and -size
mutually exclusive.

min-free-space-percent does not give a fine tuning of the free disk
space that a user might decide to keep. It can translate to very large
size (e.g. 1% = ~10GB on 1TB HDD) or very small (e.g. 1% = ~330MB on 32GB
system like Endless devices). Hence, it makes sense to introduce a config
option to honor specific size as per the user.

Closes: #1616
Approved by: jlebon
2018-06-13 18:57:37 +00:00
Matthew Leeds 6ea21696a3 Fix building against old glib versions
We need to include libglnx.h in places where ostree-autocleanups.h is
included, so that we get backports of G_DEFINE_AUTOPTR_CLEANUP_FUNC and
friends.

Closes: #1615
Approved by: jlebon
2018-06-07 17:28:49 +00:00
Matthew Leeds 1d830c1792 Revert "lib: Fix building against old glib versions"
This reverts commit f1d9196076.

Since libglnx.h does not get installed, it can't be included in
ostree-autocleanups.h, which is included by ostree.h.

Closes: #1615
Approved by: jlebon
2018-06-07 17:28:49 +00:00
Matthew Leeds 8fbf19c9f5 Make P2P API public (no longer experimental)
Currently the API that allows P2P operations (e.g. pulling an ostree ref
from a LAN or USB source) is hidden behind the configure flag
--enable-experimental-api. This commit makes the API public and makes
that flag essentially a no-op (leaving it in place in case we want to
use it again in the future). The P2P API has been tested over the last
several months and proven to work.

This means that since we're no longer using the "experimental" feature
flag, P2P builds of Flatpak will fail when using versions of OSTree from
this commit onwards, until Flatpak is patched in the near future. If you
want to build Flatpak < 0.11.8 with P2P enabled and link against OSTree
2018.6, you'll have to patch Flatpak.  However, since Flatpak won't yet
have a hard dependency on OSTree 2018.6, it needs a new way to determine
if the P2P API in OSTree is available, so this commit adds a "p2p"
feature flag. This way the feature set is more semantically correct than
if we had continued to use the "experimental" feature flag.

In addition to making the P2P API public, this commit makes the P2P unit
tests run by default, removes the f27-experimental CI instance that's no
longer needed, changes a few man pages to reflect the changes, and
updates the bash completion script to accept the new commands and
options.

Closes: #1596
Approved by: cgwalters
2018-06-04 19:20:10 +00:00
Matthew Leeds f1d9196076 lib: Fix building against old glib versions
This commit includes libglnx.h in ostree-autocleanups.h, so we get the
g_autoptr backports wherever they're needed. Also, remove the "#include
libglnx.h" lines elsewhere that are no longer needed.

Closes: #1596
Approved by: cgwalters
2018-06-04 19:20:09 +00:00
Philip Withnall bf1af263d6 lib/repo-pull: Support retries for delta superblocks
Use the recently introduced architecture for retrying network requests
on transient failure to do the same for delta superblock requests, now
that they’re queued.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1600
Approved by: jlebon
2018-05-30 19:57:13 +00:00
Philip Withnall f342e66c11 lib/repo-pull: Support queuing delta superblock requests
Just like all the other requests made for delta parts and objects by the
pull code, use a queue for delta superblocks. Currently this doesn’t do
any prioritisation or retries after transient failures, but it could do
in future.

This means that delta superblocks are now subject to the parallel
request limit in the fetcher, which was a problem highlighted here:
https://github.com/ostreedev/ostree/pull/1453#discussion_r168321706.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1600
Approved by: jlebon
2018-05-30 19:57:13 +00:00
Philip Withnall 197644c406 lib/fetcher: Factor out HTTP status code handling from soup and curl
Use the same G_IO_ERROR_* values for HTTP status codes in both fetchers.
The libsoup fetcher still handles a few more internal error codes than
the libcurl one; this could be built on in future.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1594
Approved by: jlebon
2018-05-30 16:23:57 +00:00
Philip Withnall 78f40136db lib/repo-pull: Add some missing assertions for progress statistics
Various of the counters already have assertions like this; add some more
for total paranoia.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1594
Approved by: jlebon
2018-05-30 16:23:57 +00:00
Philip Withnall 224f3cdd24 lib/fetcher-soup: Map more SoupStatus codes to known GIOErrors
This allows the retry code in ostree-repo-pull.c to recover from (for
example) timeouts at the libsoup layer in the stack, as well as from the
GSocket layer in the stack.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1594
Approved by: jlebon
2018-05-30 16:23:57 +00:00
Philip Withnall 938055392f lib/repo-pull: Support retrying requests on transient network errors
Allow network requests to be re-queued if they failed with a transient
error, such as a socket timeout. Retry each request up to a limit
(default: 5), and only then fail the entire pull and propagate the error
to the caller.

Add a new ostree_repo_pull_with_options() option, n-network-retries, to
control the number of retries (including setting it back to the old
default of 0, if the caller wants).

Currently, retries are not supported for FetchDeltaSuperData requests,
as they are not queued. Once they are queued, adding support for retries
should be trivial. A FIXME comment has been left for this.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1594
Approved by: jlebon
2018-05-30 16:23:57 +00:00
Matthew Leeds f31087137e Fix building P2P code against old glib versions
When building the OstreeBloom code against old versions of glib, we have
to have the libglnx headers included so that it defines
G_DEFINE_AUTOPTR_CLEANUP_FUNC and friends for us.

This is similarly true for test-repo-finder-mount.c which indirectly
includes ostree-autocleanups.h.

Closes: #1605
Approved by: cgwalters
2018-05-30 14:19:25 +00:00
Matthew Leeds 7412e65f7d Allow compiling OstreeRepoFinderAvahi without libsoup
Closes: #1605
Approved by: cgwalters
2018-05-30 14:19:25 +00:00
Matthew Leeds 2b32de38cf Allow compiling with experimental API but without curl/soup
This commit rearranges a few things in ostree-repo-pull.c so that OSTree
will successfully compile with experimental API enabled and without
libsoup, libcurl, or avahi:

./autogen.sh --enable-experimental-api --without-soup --without-curl
--without-avahi

This is accomplished with two sets of changes:

1. Move ostree_repo_resolve_keyring_for_collection() so it can be used
even without libsoup or libcurl.
2. Add stub functions for ostree_repo_find_remotes_async() and
ostree_repo_pull_from_remotes_async(), and their _finish() counterparts,
so they return an error when libsoup or libcurl isn't available.

Closes: #1605
Approved by: cgwalters
2018-05-30 14:19:25 +00:00
Philip Withnall 23d4beead1 lib/repo-pull: Factor out enqueue function for FetchStaticDeltaData
This introduces no functional changes, but will make upcoming support
for retrying downloads easier to add.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1599
Approved by: jlebon
2018-05-25 13:12:33 +00:00
Philip Withnall dc1fde704d lib/repo-pull: Factor out enqueue function for FetchObjectData
This introduces no functional changes, but will make upcoming support
for retrying downloads easier to add.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1599
Approved by: jlebon
2018-05-25 13:12:33 +00:00
Philip Withnall 8b8952ebee lib/repo-pull: Factor out enqueue function for ScanObjectQueueData
This introduces no functional changes, but will make upcoming support
for retrying downloads easier to add.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1599
Approved by: jlebon
2018-05-25 13:12:33 +00:00
Philip Withnall 6d21f83813 lib/repo-pull: Rename a variable
Rename from `fdata` to `fetch_data` to clarify things and make it
consistent with other similar functionality in the file.

This introduces no functional changes.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1599
Approved by: jlebon
2018-05-25 13:12:33 +00:00
Philip Withnall fc10354404 lib/repo-pull: Factor out free function for FetchDeltaSuperData
This introduces no functional changes, but does make the code a little
cleaner.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1599
Approved by: jlebon
2018-05-25 13:12:33 +00:00
Philip Withnall 66777b1b75 lib/repo-pull: Use GCC `?:` ternary shortcut where appropriate
This introduces no functional changes; just makes the code a bit shorter
in a few places.

https://gcc.gnu.org/onlinedocs/gcc/Conditionals.html

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1599
Approved by: jlebon
2018-05-25 13:12:33 +00:00
Philip Withnall 8593357902 lib/repo-pull: Use values from struct in enqueue_one_object_request()
This introduces no functional changes, but will make some upcoming
refactoring a little easier.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1599
Approved by: jlebon
2018-05-25 13:12:33 +00:00
Colin Walters 9131d8a4cc lib/sysroot: Add wrapper API to prune system repository
The initial motivation for this is that the "staging" code currently
didn't rewrite the deployment refs, meaning that the staged commit
could be pruned.

Hence first, this new API ensures that deployments also
hold a strong ref to their commit, without relying on the magical
"deployment refs" that we inject.  That has always been a weird
artifact of the strict layering separation between OstreeSysroot
and OstreeRepo.

I also plan to change rpm-ostree to start using this API to
hold references to base layers for client-side layering; it also
today generates various refs.

That said, if we still want to support multiple processes
writing to a single repo (as happens on EndlessOS today) we
still need to write refs; perhaps later we could add a concept
of "generators" or something that create refs based on whatever
logic?

Another minor thing this fixes is that we had a printf inside
the library; this propagates the pruned data to the higher level
which can log however it likes.

Closes: #1566
Approved by: jlebon
2018-05-24 12:56:11 +00:00
Colin Walters 371081d123 lib: Add a public helper method for pruning to find all ref'd commits
Prep for reworking how we do sysroot cleanup.  We're going to
start doing more lowlevel pruning work there, and I wanted to avoid
duplicating the ref enumeration.

Closes: #1566
Approved by: jlebon
2018-05-24 12:56:11 +00:00
Andrea Galbusera ce2995e1dc u-boot: add support for devicetree
Likewise the corresponding support for syslinux introduced by commit
c5112c25e4, this one enables writing devicetree
filename into the uEnv.txt environment file for u-boot.

Since u-boot does not strictly defines variable names, here 'fdt_file' was
chosen as it appear to be one the most frequently adopted names in u-boot
default environments. Outer boot logic should of course comply with this choice
and use $fdt_file as the device tree file name to pass to boot commands.

This was tested on a custom board booting with u-boot.

Closes: #1590
Approved by: cgwalters
2018-05-23 12:25:38 +00:00
Colin Walters 79c2d76e9f Post-release version bump
Closes: #1586
Approved by: jlebon
2018-05-11 15:52:05 +00:00
Colin Walters 986f176e7c Release 2018.5
Lots of stuff queued up.

Closes: #1585
Approved by: jlebon
2018-05-11 13:06:19 +00:00
Colin Walters 6873650cae lib/sysroot: Add OSTREE_EX_STAGE_DEPLOYMENTS environment variable
I feel like I'm drowning in a pile of experimental-but-almost-stable
features...

Anyways, since we made the feature opt-in in rpm-ostree in
https://github.com/projectatomic/rpm-ostree/pull/1352
let's mirror that a bit here with an environment variable so people
can play with it more easily.

The tests needed some tweaks; specifically we need to reload the
status fact after making changes.  I'm still a bit uncertain
about the Ansible-as-tests.

But we add an upgrade test that uses the new environment variable.

Closes: #1583
Approved by: jlebon
2018-05-11 12:39:17 +00:00
Tobias Mueller 5ee78ea42a repo: handle GPG_ERR_AMBIGUOUS_NAME in sign_data
This should give a more insightful error message if the user provides
a UID which is present on multiple keys.
This happens if you have an old key in your keyring which you are not
actively using any more, e.g. because it is too old.  You still have
your old keys in your keyring, because you want to read old email
encrypted for that key, though.

The gpgme function used by ostree right now complains if a UID is found
on multiple keys:
https://www.gnupg.org/documentation/manuals/gpgme/Listing-Keys.html#index-gpgme_005fget_005fkey
The used API is too simple for that use case.

Note that it would be nicer if ostree picked the only valid signing key out
of the available keys rather than using the simplistic gpgme_get_key
function.  It be nicer, of course, if there was such a gpgme function.

Closes: #1579
Approved by: cgwalters
2018-05-06 10:46:07 +00:00
Jonathan Lebon 589e97dc60 lib/commit: Fix function name typo in docstring
Closes: #1575
Approved by: cgwalters
2018-05-04 14:51:07 +00:00
Jonathan Lebon bd904b26e1 lib/deploy: Add semi-colon in post-deployment msg
Let's add a semi-colon between the "bootconfig swap" part and the
"deployment count change" to make it more clear they're separate
statements.

Closes: #1575
Approved by: cgwalters
2018-05-04 14:51:07 +00:00
Colin Walters d0a1832347 deploy: Use fdatasync() for new kernel/initramfs by default
While we do a `syncfs()` plus `FIFREEZE/THAW` for `/boot`, that
only comes during deployment finalization.

The code here today generally assumes that if the file exists
it's been fully written.  So let's do a `fdatasync()` before
we do the `rename()`.

This just came out of looking through the code while working
on deployment staging.  In that scenario there's a much larger
window between when we copy the kernel/initramfs and when we
sync `/boot`.

Closes: #1571
Approved by: jlebon
2018-05-02 19:37:52 +00:00
Colin Walters 9f8e2b8862 lib: Use `Locking:` term in docs
This is easier to `git grep` etc. versus ad-hoc English.  Although
we still have some English for the prepare_transaction/commit which
acquire/release in separate phases.

Closes: #1572
Approved by: jlebon
2018-05-02 17:28:29 +00:00
Colin Walters 5337ba51b2 lib/deploy: Do post-ops when removing staged commit
These are further fixes based on running more of the rpm-ostree
test suite.

When dropping the staged deployment, we do need to do the
"post operations" such as bumping the sysroot mtime, so that
clients know something changed.  We also need to regenerate
the deployment refs.  And of course do a sysroot reload.

Also, add a "base cleanup" after creating a staged deployment
which also regenerates the refs.

Closes: #1570
Approved by: jlebon
2018-05-02 16:22:34 +00:00
Colin Walters 25ba8db987 lib/deploy: Throw an error if trying to stage when not ostree-booted
There's no reason to do this.  I didn't actually hit this problem,
but it's a corner case that just occurred to me while working on
the code.

I think callers should be adapted to skip trying to use staging
if there's no booted deployment.

Closes: #1568
Approved by: jlebon
2018-05-02 13:27:01 +00:00
Colin Walters f5f8e2c540 lib/sysroot: Maintain one variable for "root is ostree booted"
This was pointed out in a previous PR review; we don't have
a need for the separate variables.  Prep for adding an API for
this.

Closes: #1568
Approved by: jlebon
2018-05-02 13:27:01 +00:00
Colin Walters 36ac768fb7 repo: Add checksum to error message opening unreadable object
This would have debugged trying to do a pull as non-root from
a sysroot repository.  See for example:
https://github.com/ostreedev/ostree/issues/1562

Closes: #1564
Approved by: jlebon
2018-05-01 19:22:10 +00:00
Colin Walters 11c12cc19a deploy: Don't prune repo at finalization time by default
Doing so can break rpm-ostree, which wants to own the cleanup process
to ensure its baselayer refs are generated.

Further, doing the cleanup at shutdown time adds latency.  It's also
going to be generally unnecessary as we expect repo pruning to have
been done when writing the refs.

Closes: #1567
Approved by: jlebon
2018-05-01 13:59:07 +00:00
Colin Walters 8c1542134c lib/repo: Enable locking by default, but drop external API
The code has been sitting around for a while but since I disabled
it by default, I doubt anyone is really using it or relying on it.

This patch and turns on locking by default, and also drops the
API which was only public in the experimental API builds.
Conceptually these are two distinct things, and we
may actually want to split up the patches.

I don't think this will break anyone, but it's hard to say for sure.
It's also going to be hard to find out until we actually release
I suspect...

But anyone who is broken should be able to add `locking=false` into
their repo config.  On the flip side Endless has been shipping with
this enabled and it is reported to help.

The reason to drop the APIs: I'm a bit concerned about the interactions over time
between libostree's use of the API and any apps that start using it.
For example, if an app specifies a SHARED lock in their code, then
later internally we decide to temporarily grab an `EXCLUSIVE`, but the
app had a second thread/process that was `EXCLUSIVE` already, and
that process was waiting on the first bit of code, then we could
deadlock. I can't think of a real world situation where this would happen
yet though.

We are likely to in the future have say `fsck` take an external lock,
`checkout` grab a shared one, etc.

Closes: #1555
Approved by: jlebon
2018-04-30 17:24:51 +00:00
Colin Walters cd17e364ae deploy: Return staged deployment
Today rpm-ostree has some code to run a "sanitycheck" on a deployment.
I had initially deleted that when adapting it to use the staging code,
but I realized it should work fine; we just won't see the merged
config, but that's OK.

When I readded that code it started crashing because we didn't
actually return the new deployment object.  We'll gain some coverage
here as I'll land the code to have rpm-ostree use staging, then bump
the rpm-ostree tests here.

Closes: #1559
Approved by: jlebon
2018-04-27 19:41:19 +00:00
Colin Walters de4c7105f0 lib/deploy: Fix staged deployments with no kargs
Testing out the staged API with rpm-ostree, ostree-prepare-root.service
in the initramfs was failing.  Turned out that was because we didn't
have a `root=` kernel argument.  Which was because we didn't have
any kernel arguments at all except `ostree=`.

That in turn was because we weren't loading the bootloader config
from the merge deployment.

The serialized deployment data holds the unique identity of
(osname, checksum, deployserial) - look for the real merge deployment
in our deployment list which has the bootloader arguments we need.

This issue was entirely masked by the `ostree admin deploy` command
which itself explicitly loads the merge deployment's kernel arguments
in every case - it never passes the `NULL` default down.  A followup
patch will fix that.

Closes: #1558
Approved by: jlebon
2018-04-27 17:48:21 +00:00
Jonathan Lebon dc4aa346a3 lib/deploy: Also compare deployment csum versions
When comparing deployments to determine whether we need a new
bootversion, we should also check whether the commit "version" metadata
is the same. Otherwise, we may end up with the a bootconfig whose
`title` includes a version that doesn't match the one from the
deployment checksum.

Closes: https://github.com/projectatomic/rpm-ostree/issues/1343

Closes: #1556
Approved by: cgwalters
2018-04-24 17:04:27 +00:00
Jonathan Lebon ab8d694361 lib/deploy: Factor out function to get deployment kargs
No functional change. Prep for next commit.

Closes: #1556
Approved by: cgwalters
2018-04-24 17:04:27 +00:00
Matthew Leeds 972a1921f5 man: Add man page for create-usb
Closes: #1543
Approved by: cgwalters
2018-04-19 13:35:15 +00:00
Sam Spilsbury 002e5eeac6 avahi: Don't complain with g_warning if the daemon wasn't running
This is a normal case when running unit tests in client code
on continuous integration infrastructure. When those tests are
running they will set G_DEBUG=fatal-warnings which will cause
the program to abort if a warning is emitted. Instead, emit
a debug message if the problem was that we couldn't connect to
the daemon.

Closes: #1542
Approved by: jlebon
2018-04-18 19:39:33 +00:00
Colin Walters 16d3359bf8 lib/sysroot: Move staged into deployment list, rework handling
Followup to: https://github.com/ostreedev/ostree/pull/1503
After starting some more work on on this in rpm-ostree, it is
actually simpler if the staged deployment just shows up in the list.

It's effectively opt-in today; down the line we may make it the default,
but I worry about breaking things that e.g. assume they can mutate
the deployment before rebooting and have `/etc` already merged.

There's not that many things in libostree that iterate over the deployment
list.  The biggest change here is around the
`ostree_sysroot_write_deployments_with_options` API.  I initially
tried hard to support a use case like "push a rollback" while retaining
the staged deployment, but everything gets very messy because that
function truly is operating on the bootloader list.

For now what I settled on is to just discard the staged deployment;
down the line we can enhance things.

Where we then have some new gymnastics is around implementing
the finalization; we need to go to some effort to pull the staged
deployment out of the list and mark it as unstaged, and then pass
it down to `write_deployments()`.

Closes: #1539
Approved by: jlebon
2018-04-18 18:59:15 +00:00
Alexander Larsson f258e9e5ff lib/repo: Add ostree_repo_traverse_commit_union_with_parents
This is a version of ostree_repo_traverse_commit_union that also
remembers where the objects came from, by recording the parent
relationships in a hashtable. This can be used to later find which
commits each object was from, which we want to use in fsck.

Closes: #1533
Approved by: cgwalters
2018-04-14 15:36:21 +00:00
Giuseppe Scrivano cdaf7cd838 commit, payload-reflink: do not write to the parent repo
reintroduce the feature that was reverted with commit:

28c7bc6d0e

Differently than the original implementation, now we don't attempt any
test for reflinks support on the parent repository, since the test
requires write access to the repository.

Additionally, also check that the two repositories are on the same
device before attempting any reflink.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

Closes: #1525
Approved by: cgwalters
2018-04-13 21:52:53 +00:00
Colin Walters a2dc731982 sysroot: Clean up origin loading function
In prep for staging work, where we'll need to load the origin
for the staged deployment too.

The function was previously trying to avoid operating on an
instantiated deployment, but the data we need is in the deployment
object at that point.

Closes: #1538
Approved by: jlebon
2018-04-13 15:26:29 +00:00
Colin Walters efdaf1495b deploy: Silently do nothing if passed same set of deployments
Prep for handling staged deployments better; if we're not passed
the staged one back, then we just want to delete it but not
touch the bootloader config.

Closes: #1538
Approved by: jlebon
2018-04-13 15:26:29 +00:00
Colin Walters 56de631721 deploy: Clean up bootserial assignment function
The reason we were returning a hashtable is a bit lost to history,
there's no reason to do so now anyways.  Also port to declare-and-initialize
style and add more comments.

Closes: #1538
Approved by: jlebon
2018-04-13 15:26:28 +00:00
Colin Walters eb506c759c Add concept of "staged" deployment
Add API to write a deployment state to `/run/ostree/staged-deployment`,
along with a systemd service which runs at shutdown time.

This is a big change to the ostree model for hosts,
but it closes a longstanding set of bugs; many, many people have
hit the "losing changes in /etc" problem.  It also avoids
the other problem of racing with programs that modify `/etc`
such as LVM backups:
https://bugzilla.redhat.com/show_bug.cgi?id=1365297

We need this in particular to go to a full-on model for
automatically updated host systems where (like a dual-partition model)
everything is fully prepared and the reboot can be taken
asynchronously.

Closes: https://github.com/ostreedev/ostree/issues/545

Closes: #1503
Approved by: jlebon
2018-04-12 14:55:12 +00:00
Owen W. Taylor 9199237b01 Don't scan uncompressed_objects_dir if it doesn't exist
A newly created archive-mode repository won't have a uncompressed-objects-cache
directory, and uncompressed_objects_dir is -1 to flag that. The special meaning of
-1 meaning "cwd" for libglnx means that the current directory was scanned as
if it was an objects directory, producing unexpected results, especially if there
were any two-letter files/subdirs in the current directory.

Closes: #1537
Approved by: jlebon
2018-04-12 13:53:15 +00:00
Colin Walters 54b3bbc00f sysroot: Split out a helper function to delete a deployment dir
Prep for staged deployments.

Closes: #1535
Approved by: jlebon
2018-04-11 19:11:07 +00:00
Matthew Leeds cb3360fca6 lib/repo: Add timestamps to OstreeRepoFinderResult
Currently OstreeRepoFinderResult, a data structure used by pull code
that supports P2P operations, has a hash table mapping refs to checksums
but doesn't include timestamp information. This means that clients have
no way of knowing just from the OstreeRepoFinderResult information if a
commit being offered by a peer remote is an update or downgrade until
they start pulling it. The client could check the summary or the commit
metadata for the timestamps, but this requires adding the temporary
remotes to the repo config, and ostree is already checking timestamps
before returning the results, so I think it makes more sense for them to
be returned rather than leaving it to the client. This limitation is
especially important for offline computers, because for online computers
the latest commit available from any remote is the latest commit,
period.

This commit adds a "ref_to_timestamp" hash table to
OstreeRepoFinderResult that is symmetric to "ref_to_checksum" in that it
shares the same keys. This is an API break, but it's part of the
experimental API, and none of the current users of that (flatpak,
eos-updater, and gnome-software) are affected. See the documentation for
more details on "ref_to_timestamp". One thing to note is the data
structure currently gets initialized in find_remotes_cb(), so only users
of ostree_repo_find_remotes_async() will get them, not users of, say,
ostree_repo_finder_resolve_all_async(). This is because the individual
OstreeRepoFinder implementations don't currently access the timestamps
(but I think this could be changed in the future if there's a need).

This commit will allow P2P support to be added to
flatpak_installation_list_installed_refs_for_update, which will allow
GNOME Software to update apps from USB drives while offline (it's
already possible online).

Closes: #1518
Approved by: cgwalters
2018-04-03 15:50:40 +00:00
Alexander Larsson c4c2b5ebd1 pull: Don't save summary to cache before validating signatures
In case of some kind of race or other weirdness we might be getting
non-matching versions of summary.sig and summary, where summary.sig
is the latest version. Currently we're saving them to the cache
directly after downloading them successfully, but they will then fail
to gpg validate. Then on the next run we'll keep using the cached files
even if they are incorrect, until summary.sig changes upstream.

This changes the order so that we verify the signatures before saving
to the cache, thus ensuring that we don't end up in a stuck state.

Fixes https://github.com/ostreedev/ostree/issues/1523

Closes: #1529
Approved by: cgwalters
2018-04-03 15:04:31 +00:00
Matthew Leeds b190982ee6 lib/repo-pull: Improve error message when no summary is found
In ostree_repo_remote_fetch_summary_with_options(), if no summary is
found on the server and summary verification is enabled, the error
message implies that it's the summary signature that's missing, which is
misleading. This commit adds a more specific error message for the case
of a missing summary, which has the side effect of explicitly checking
for the case that signatures != NULL && summary == NULL after
repo_remote_fetch_summary(), even though that should never happen.

One effect of this is that if you run "flatpak remote-add" with an
incorrect URL you get a more helpful error message, and similarly for
other flatpak operations and other users of ostree.

Closes: #1522
Approved by: cgwalters
2018-04-02 22:40:37 +00:00
Matthew Leeds 5d84f5e102 lib/repo-pull: Rename a variable for clarity
In libostree, the phrase "commit metadata" has two meanings-- one is the
first dictionary in a commit GVariant that stores metadata such as ref
bindings, and the other is the commit metadata in the summary file,
which stores the commit size, checksum, and timestamp. In
find_remotes_process_refs(), the entire commit GVariant was being
referred to as commit metadata, so this commit changes the variable
name and a comment to make things more consistent.

Closes: #1528
Approved by: cgwalters
2018-04-02 22:05:52 +00:00
Matthew Leeds 005d25cc75 lib: Fix a few comments
Closes: #1526
Approved by: cgwalters
2018-03-29 22:01:51 +00:00
Alexander Larsson 28c7bc6d0e Don't write to parent repo
In _try_clone_from_payload_link, don't try to do the clone in the
parent repo, because we don't want to modify that. parent repos are
typically used when you want a shared, immutable base.

For example in flatpak, the parent repo is the system repo which you
don't have write access to, so any modification to it will fail with
EACCES, making it impossible to install via the system helper.

Closes: #1524
Approved by: cgwalters
2018-03-29 14:11:38 +00:00
Matthew Leeds 2be4631738 lib/commit: Fix a memory leak of OtChecksum
Closes: #1521
Approved by: jlebon
2018-03-29 13:45:26 +00:00
Matthew Leeds 3c14546017 lib/repo-pull: Document all options in pull_from_remotes_async
ostree_repo_pull_from_remotes_async() passes along some options to
ostree_repo_pull_with_options(), so document them.

Closes: #1519
Approved by: cgwalters
2018-03-29 00:18:21 +00:00
Colin Walters 2648c968c4 lib/deploy: Port final bootconfig writing to new style
The main blocker for doing this before was the `goto out` handling
for remounting `/boot`.  Handle that by factoring out the bits that
require it to a helper function, and do the C/GError equivalent of
"try/finally".

Not prep for anything right now, just decided to do this since I had the file
open.

Closes: #1515
Approved by: jlebon
2018-03-26 16:29:37 +00:00
Colin Walters 7ec3d06014 lib/deploy: Split /etc merge into two stages
For staged deploy, we want to pay the cost of creating copies from
`/usr/etc` → `/etc` at stage time, since it can be expensive.  (We
want to minimize time spent during shutdown).

Split it up into two functions; the logic is also simply clearer.

Closes: #1514
Approved by: jlebon
2018-03-26 14:24:29 +00:00
Colin Walters 460fb7aebc lib/deploy: Set kargs in one place
Prep for deployment staging. We had the code to hande "explicit kargs" in one
place, but the "use merge deployment" karg bits mixed in with the "/etc merge"
logic. Those are separate things, and it's better to have karg handling in one
place.

Closes: #1514
Approved by: jlebon
2018-03-26 14:24:29 +00:00
Colin Walters 0b1d3b87e3 lib/deltas: Some misc declare-and-initialize porting
Also some `glnx_fstatat_allow_noent()`.  Not specifically prep for anything.

Closes: #1511
Approved by: jlebon
2018-03-23 18:16:35 +00:00
Colin Walters 155c936cd5 Release 2018.4
A quick turnaround to include one PR: https://github.com/ostreedev/ostree/pull/1508
"switchroot: Ensure /run/ostree-booted is created even without initramfs"

This fixes ostree when booting without an initramfs.  Thanks to @akiernan for the
bug report and helping review the fix!  I'm working on enhancing
the test suite, which will help in adding some coverage here.

Also for this release I'm going to avoid adding a "stub" symbol section
to the `-released.sym` file; I don't believe it's necessary.

Closes: #1512
Approved by: jlebon
2018-03-22 21:07:11 +00:00
Colin Walters 83b97ec569 lib/deltas: Squash some GCC maybe-uninitialized warnings
These show up in the RPM build, I didn't yet try to figure out why we're not
reproducing them outside of that.

Closes: #1510
Approved by: jlebon
2018-03-22 19:30:16 +00:00
Colin Walters 671b026fcc Release 2018.3
It's been over a month since 2018.2; we have a few features and various fixes,
and the "stage" work pending which is pretty invasive. Time for a new release!

Closes: #1506
Approved by: jlebon
2018-03-21 16:01:41 +00:00
Colin Walters 9ca3f76cd2 lib/deploy: Have internal origin writing API take sepolicy
Ensures it's labeled consistently. Prep for staged deployments which reworks the
logic around when the origin file is written.

Closes: #1505
Approved by: jlebon
2018-03-19 18:42:13 +00:00
Colin Walters ce2449ad2e lib/deploy: Use in-function error prefixing more
Pulling some of this out of stage deploy work. It's generally better as it's
easier to change functions to have multiple callers.

Closes: #1505
Approved by: jlebon
2018-03-19 18:42:13 +00:00
Colin Walters d4d193495f lib/deploy: Port various functions to declare-and-initialize
Just noticed this while working on the code.

Closes: #1499
Approved by: jlebon
2018-03-17 20:36:04 +00:00
Jonathan Lebon 4e4436beec lib/fetcher: Allow clients to append to User-Agent
We do already have `http-headers`, which potentially could be used to
allow clients to completely override the field, but it seems like the
more common use case is simply to append.

Closes: #1496
Approved by: cgwalters
2018-03-16 19:21:31 +00:00
Jonathan Lebon 296ef25e12 lib/core: Support <remote>: syntax when listing refs
Allow users to pass `<remote>:` to list all refs we have locally
belonging to `<remote>`. Also (re-)allow the similar `<remote>:.` syntax
for backwards compatibility with flatpak.

Closes: #1500
Approved by: cgwalters
2018-03-16 19:18:08 +00:00
Colin Walters 5b3f79d4bb sysroot: Rework how we find booted deployment
I was looking at this code in prep for "staging" deployments,
and there are several cleanups to be made here.  The first
thing I noticed is that we look for the `ostree=` kernel argument,
but the presence of that should be exactly equivalent to having
`/run/ostree-booted` exist.  We just added a member variable for
that, so let's make use of it.

Related to this, we were erroring out if we had the karg but
didn't find a deployment.  But this can happen if e.g. one is
using `ostree admin --sysroot` from an ostree-booted system!  It's
actually a bit surprising no one has reported this so far; I guess
in the end people are either using non-ostree systems or running
from containers.

Let's add a member variable `root_is_sysroot` that we can use
to determine if we're looking at `/`.  Then, our more precise
"should find a booted deployment" state is when both `ostree_booted`
and `root_is_sysroot` are TRUE.

Next, rather than walking all of the deployments after parsing,
we can inline the `fstatat()` while parsing.  The mild ugly
thing about this is assigning to the sysroot member variable while
parsing, but I will likely clean that up later, just wanted to avoid
rewriting everything in one go.

Closes: #1497
Approved by: jlebon
2018-03-15 17:43:19 +00:00
Colin Walters bb9cc8912a sysroot: Track whether /run/ostree-booted exists
Prep for further work around deployment staging.

Closes: #1497
Approved by: jlebon
2018-03-15 17:43:19 +00:00
Colin Walters 792c190a44 lib/deploy: Port deployment checkout func to new style
Not sure how we missed this one before.  No functional changes,
just prep for further work.

Closes: #1497
Approved by: jlebon
2018-03-15 17:43:19 +00:00
Joaquim Rocha 591f8a68b1 pull: Ignore the cancellable when aborting a transaction
In ostree_repo_abort_transaction, if we pass a cancellable and it gets
canceled, then the function may fail to fully clean up the transaction
state. This was happening e.g. when the ostree_repo_pull_with_options
call got cancelled.

To fix this, as suggested by Colin Walters, we set the passed
cancellable as NULL, in order for it to be ignored.

https://github.com/ostreedev/ostree/issues/1491

Closes: #1492
Approved by: jlebon
2018-03-12 19:18:57 +00:00
Jeremy Hiatt 3b7044f45e lib/repo: Fix multi-signature support when generating summary files
Ensure that the metadata object is built up with the signatures from all keys
passed to ostree_repo_add_gpg_signature_summary(). Previously only the signature
from the last key would end up in the metadata.

Closes: #1488

Closes: #1489
Approved by: jlebon
2018-03-08 19:58:41 +00:00
Giuseppe Scrivano 127d8bb846 commit: add logic for .payload-link
When a new object is added to the repository, create a
$PAYLOAD-SHA256.payload-link symlink file as well.  The target of the
symlink is the checksum of the object that was added the repository.

Whenever we add a new object file, in addition to lookup if the file is
already present with the same checksum we also check if an object with
the same payload is in the repository.

If a file with the same payload is already present in the repository, we
copy it with `glnx_regfile_copy_bytes` that internally attempts to
create a reflink (ioctl (..., FICLONE, ..)) to the target file if the
file system supports it.  This enables to have objects that share the
payload but have a different inode and xattrs.

By default the payload-link-threshold value is G_MAXUINT64 that disables
the feature.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

Closes: #1443
Approved by: cgwalters
2018-03-07 18:28:59 +00:00
Giuseppe Scrivano 118f1f7e40 ostree: introduce PAYLOAD_LINK object type
It will be used by successive commits to keep track of the payload
checksum for objects stored in the repository.

The goal is that files having the same payload but different xattrs can
take advantage of reflinks where supported.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

Closes: #1443
Approved by: cgwalters
2018-03-07 18:28:59 +00:00
Giuseppe Scrivano 418e4545de ostree-repo-private: remove declaration for _ostree_repo_find_object
it was removed with:

commit 8609cb036b
Author: Colin Walters <walters@verbum.org>
Date:   Thu Apr 21 15:14:51 2016 -0400

    repo: Simplify internal has_object() lookup code

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

Closes: #1443
Approved by: cgwalters
2018-03-07 18:28:59 +00:00
Colin Walters 969e4eb72e repo/refs: Clean up error prefixing
Add some "function global" prefixing in line with what we do in
other places now, and drop the "manual filename" prefixing that
is no longer necessary since
23f7df1500

Closes: https://github.com/ostreedev/ostree/issues/1467

Closes: #1485
Approved by: jlebon
2018-03-06 14:01:39 +00:00
Matthew Leeds fe6ae92ebc lib: Fix memory leaks of OstreeRemote
The _ostree_repo_get_remote() and _ostree_repo_get_remote_inherited()
methods transfer ownership of the returned OstreeRemote to the caller,
so this commit fixes a few call sites that weren't properly freeing it.

Closes: #1478
Approved by: cgwalters
2018-03-05 20:00:16 +00:00
Colin Walters 733c0498dc lib/repo: Do account for size with prune --no-prune
I think this got changed in a refactor.  We definitely want
to total up the amount of space that *would* be freed even
with `--no-prune` AKA `OSTREE_REPO_PRUNE_FLAGS_NO_PRUNE`.

It's actually a bit terrifying this is apparently the first test case for
the `--no-prune` option...

Closes: https://github.com/ostreedev/ostree/issues/1480

Closes: #1483
Approved by: jlebon
2018-03-05 16:58:12 +00:00
Matthew Leeds 1214395f0e lib/repo-finder-mount: Improve debug message
This makes it easier to tell which mount is being checked when repos are
found.

Closes: #1477
Approved by: cgwalters
2018-03-02 19:16:33 +00:00
Matthew Leeds 7727fdd9f7 lib/remote: Fix memory leak
Closes: #1476
Approved by: cgwalters
2018-03-02 18:49:44 +00:00
Matthew Leeds d0e4a4f03b lib/repo-finder-mount: Update comment about paths
This updates the gtk-doc comment for OstreeRepoFinderMount to match the
correct flatpak repo path, which was fixed in commit 6db6268df.

Closes: #1473
Approved by: cgwalters
2018-03-01 14:10:49 +00:00
Matthew Leeds 2381ca0aa4 lib/repo-pull: Fix free function for hash table
The "ref_original_commits" hash table uses string values, not variants,
so fix the free function passed to g_hash_table_new_full (). Since
g_variant_unref isn't NULL safe, this prevents an assertion failure when
a NULL value is inserted.

Dan Nicholson suggested this patch; I'm just submitting it because he's
busy.

Fixes https://github.com/ostreedev/ostree/issues/1433

Closes: #1474
Approved by: cgwalters
2018-03-01 14:10:24 +00:00
Jonathan Lebon 530043fcf6 lib/sysroot: Fix retrieving non-booted pending deployment
If we're booted into a deployment, then any queries for the pending
merge deployment of a non-booted OS will fail due all of them being
considered rollback.

Fix this by filtering by `osname` *before* determining if we've crossed
the booted deployment yet.

Closes: #1472
Approved by: cgwalters
2018-02-28 18:09:13 +00:00
Matthew Leeds 6db6268dfd lib/repo-finder-mount: Fix path to flatpak repo
OstreeRepoFinderMount checks mounts for a few well-known directories
such as "ostree/repo" and ".ostree/repo" to try to find remotes. One of
the hard-coded directories is "var/lib/flatpak" but that's the flatpak
directory, not the ostree repo used by flatpak, which is at
"var/lib/flatpak/repo". So this commit changes the path so the repo can
be found.

For recent versions of Endless, flatpak uses /ostree/repo as its
repository, so this commit won't make a difference there. But it may
help on other operating systems.

Closes: #1471
Approved by: cgwalters
2018-02-27 20:53:46 +00:00
Colin Walters 7f88fddcd4 sysroot: Add concept of deployment "pinning" 📌
Example user story: Jane rebases her OS to a new major version N, and wants to
keep around N-1 even after a few upgrades for a while so she can easily roll
back. I plan to add `rpm-ostree rebase --pin` to opt-in to this for example.

Builds on the new `libostree-transient` group to store pinning state there.

Closes: https://github.com/ostreedev/ostree/issues/1460

Closes: #1464
Approved by: jlebon
2018-02-26 19:06:59 +00:00
Colin Walters c40a47e965 sysroot: Add API to clean up transient keys in origin files
The `origin/unlocked` and `origin/override-commit` keys are examples of state
that's really transient; we don't want to maintain them across upgrades. Right
now there are bits for this in both `ostree admin upgrade` as well as in
rpm-ostree.

This new API will slightly clean up both cases, but it's really prep for adding
a concept of deployment "pinning" that will live in the new
`libostree-transient` group.

Closes: #1464
Approved by: jlebon
2018-02-26 19:06:59 +00:00
Colin Walters 2f5a34bed9 sysroot: Bump mtime when writing an origin file
This ensures that e.g. `rpm-ostreed` will get notified of the changes.

Closes: #1464
Approved by: jlebon
2018-02-26 19:06:59 +00:00
Colin Walters 779df11c9e build-sys: Post-release version bump
Closes: #1455
Approved by: jlebon
2018-02-15 14:44:11 +00:00
Colin Walters 81560cada6 Release 2018.2
There are enough fixes here, and there are some potentially larger patches
incoming like wmanley's checkout speedups and the payload link that will need
soak time in master.

Closes: #1455
Approved by: jlebon
2018-02-15 14:44:11 +00:00
Colin Walters 96eec98f39 fetcher: Drop max queue size assertion in libsoup/libcurl backends
Since f4d1334e19 the primary pull code maintains a
maximum queue. In that commit message I said `Note that I kept an assertion.`.
But I think this is wrong since while it covers a lot of the normal cases, if
one is e.g. trying to fetch a ton of refs, the primary pull code doesn't yet
queue those.  While it'd be nice to queue those, it isn't worth carrying
extra assertions in the backends that can still trigger.

Closes: https://github.com/ostreedev/ostree/issues/1451

Closes: #1453
Approved by: dbnicholson
2018-02-14 23:15:09 +00:00
Colin Walters 0041a7a1ed core: Add API (and standard concept for) content checksum
There are a few cases for knowing whether a commit has identical
content to another commit.  Some people want to do a "promotion workflow",
where the content of a commit on a tesitng branch is then "promoted"
to a production branch with `ostree commit --tree=ref`.

Another use case I just hit in rpm-ostree deals with
[jigdo](https://github.com/projectatomic/rpm-ostree/issues/1081) where we're
importing RPMs on both the client and server, and will be using the
content checksum, since the client/server cases inject different metadata
into the commit object.

Closes: https://github.com/ostreedev/ostree/issues/1315

Closes: #1449
Approved by: jlebon
2018-02-12 19:03:18 +00:00
Matthew Leeds 5848de93a4 lib/pull: Properly remove temporary remotes
For P2P pulls ostree adds temporary remotes and removes them in
find_remotes_cb(). However, if an OstreeRepoFinderResult gets freed
during the course of that function, the OstreeRemote in the result is
freed but a pointer to it remains in the remotes_to_remove array. This
means that when _ostree_repo_remove_remote() gets called on it at the
end of the function it will fail. In my case the resulting error was
"OSTree-CRITICAL **: _ostree_repo_remove_remote: assertion 'remote->name
!= NULL' failed" but I think it could also seg fault.

This commit adds a reference to the remote so it can be properly removed
when we're finished with it.

Closes: #1450
Approved by: giuseppe
2018-02-09 22:30:51 +00:00
Colin Walters 88d27fb3f1 repo: Create uncompressed-object-cache dir dynamically
Having the `uncompressed-object-cache` directory in `archive` repos by default
is clutter; the functionality should be considered deprecated.

Now we only create the directory if we're doing a checkout with the cache
enabled.

Closes: #1446
Approved by: jlebon
2018-02-08 21:27:11 +00:00
Jonathan Lebon 2e95e06616 lib/checkout: add filter API to skip over files
This is analogous to the filtering support for the commit API: we allow
library users to skip over checking out specific files. This is useful
in some tricky situations where we *know* that the files to be checked
out will conflict with existing files in subtle ways.

One such example is in rpm-ostree support for multilib. There, we want
to allow checking out a package onto an existing tree, but skipping over
files that are not coloured to our preferred value (e.g. not overwriting
an i686 version of `ldconfig` if we already have the `x86_64` version).
See https://github.com/projectatomic/rpm-ostree/pull/1227 for details.

Closes: #1441
Approved by: cgwalters
2018-02-06 15:38:20 +00:00
Colin Walters 4a98a86b72 deploy: SELinux-relabel installed kernel/initramfs data
When we changed around the kernel location in rpm-ostree, we
started installing the kernel into `/boot` as `modules_object_t`,
and the current policy didn't permit that.  For maximum compatibility,
relabel installed kernel/initramfs/dtb as `boot_t`.

https://bugzilla.redhat.com/show_bug.cgi?id=1536991

Closes: #1444
Approved by: jlebon
2018-02-02 22:32:49 +00:00
Marcus Folkesson 6bf4b3e1d8 Add SPDX-License-Identifier to source files
SPDX License List is a list of (common) open source
licenses that can be referred to by a “short identifier”.
It has several advantages compared to the common "license header texts"
usually found in source files.

Some of the advantages:
* It is precise; there is no ambiguity due to variations in license header
  text
* It is language neutral
* It is easy to machine process
* It is concise
* It is simple and can be used without much cost in interpreted
  environments like java Script, etc.
* An SPDX license identifier is immutable.
* It provides simple guidance for developers who want to make sure the
  license for their code is respected

See http://spdx.org for further reading.

Signed-off-by: Marcus Folkesson <marcus.folkesson@gmail.com>

Closes: #1439
Approved by: cgwalters
2018-01-30 20:03:42 +00:00
Philip Withnall 785da8d5a6 lib/core: Expand documentation for ostree_parse_refspec()
The old documentation had outdated and incomplete annotations, and
didn’t make it very clear that out_remote could legitimately return
NULL.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1437
Approved by: cgwalters
2018-01-29 18:21:00 +00:00
Matthew Leeds 660bc588ee lib/pull: Allow specific commits in P2P updates
Currently users of the find_remotes_async()/pull_from_remotes_async()
functions have no way to specify a commit hash to use instead of the
latest one available. This commit implements an "override-commit-ids"
option analogous to the one used by ostree_repo_pull_with_options().
It's accomplished by returning OstreeRepoFinderResult objects pointing
to the given commit checksum(s) regardless of which ones were available
from the remotes, but in the future this implementation could be
improved to take into account the commits advertised by the remotes.

One effect of this is that flatpak will have the ability to downgrade
apps that use collection IDs
(https://github.com/flatpak/flatpak/issues/1309).

Closes: #1425
Approved by: pwithnall
2018-01-24 14:15:53 +00:00
Colin Walters 42eea23864 bin/delta: Fix compilation with relative subdirs --filename
Currently we were parsing `opt_filename` twice...I dug through
the history a bit and it looks like it may have been an accident
from refactoring.

What we're fixing here concretely is that using relative subdirectories
like `--filename somesubdir/foo` broke because we were incorrectly
passing the `somesubdir/` again.

Closes: #1423

Closes: #1427
Approved by: jlebon
2018-01-22 14:02:34 +00:00
Colin Walters 654a9177d0 lib/pull: Port a few functions to new style
Prep for further work here. This diff is a bit noisy for the delta bits because
the identation was off originally as well.

Closes: #1424
Approved by: jlebon
2018-01-19 07:41:13 +00:00
William Manley 720e2ec9bc Add support for devicetree files alongside the kernel and initramfs
Much like the (optional) initramfs at
`/usr/lib/ostree-boot/initramfs-<SHA256>` or
`/usr/lib/modules/$kver/initramfs` you can now optionally include a
flattened devicetree (.dtb) file alongside the kernel at
`/usr/lib/ostree-boot/devicetree-<SHA256>` or
`/usr/lib/modules/$kver/devicetree`.

This is useful for embedded ARM systems which need the devicetree file
loaded by the bootloader for the kernel to discover and initialise
hardware.  See https://en.wikipedia.org/wiki/Device_tree for more
information.

This patch was mostly produced by copy-pasting code for initramfs handling
and renaming `s/initramfs/devicetree/g`.  It's not beautiful, but it is
fairly straightforward.

It may be useful to extend device-tree support in a number ways in the
future.  Device trees dependant on many details of the hardware they
support.  This makes them unlike kernels, which may support many different
hardware variants as long as the instruction-set matches.  This means that
a ostree tree created with a device-tree in this manner will only boot on
a single model of hardware.  This is sufficient for my purposes, but may
not be for others'.

I've tested this on my NVidia Tegra TK1 device which has u-boot running
in syslinux-compatible mode.

Closes: #1411
Approved by: cgwalters
2018-01-16 22:54:53 +00:00