Commit Graph
100 Commits
Author SHA1 Message Date
Philip Withnall 785da8d5a6 lib/core: Expand documentation for ostree_parse_refspec()
The old documentation had outdated and incomplete annotations, and
didn’t make it very clear that out_remote could legitimately return
NULL.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1437
Approved by: cgwalters
2018-01-29 18:21:00 +00:00
Philip Withnall cac42bb6f5 build: Fix typo in -Wparentheses warning
GCC supports -Wparentheses, not -Wparenthesis.

https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wno-parentheses

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1388
Approved by: jlebon
2017-12-21 22:34:26 +00:00
Philip Withnall bc3d80550b tests: Expand fsck unit tests to cover checks on bindings
Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1347
Approved by: cgwalters
2017-12-14 14:18:44 +00:00
Philip Withnall e48a1bcfe7 tests: Fix LC_ALL for systems which use .utf8 suffixes
libtest-core.sh tries to clear the locale to a UTF-8 supporting C
locale, either by setting it to C.UTF-8 (preferred) or just C.

Some systems, like Fedora 26, use the locale name C.utf8, rather than
C.UTF-8. Support that too.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1347
Approved by: cgwalters
2017-12-14 14:18:44 +00:00
Philip Withnall 5d1753f59b ostree/commit: Allow --orphan and --bind-ref to be specified together
Typically you’d use --branch and --bind-ref together to add additional
bindings as well as creating a main --branch for the commit. However,
you might also want to occasionally use --orphan --bind-ref to create a
commit with bindings for one or more refs, but not actually create any
of those refs pointing to the commit (you might create them as a later
step).

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1347
Approved by: cgwalters
2017-12-14 14:18:44 +00:00
Philip Withnall fb7692bd44 ostree/fsck: Add --verify-back-refs option
This new option verifies that the refs listed in the ref-bindings for
each commit all point to that commit (i.e. there aren’t multiple commits
listing the same ref in their ref-bindings, and there aren’t any commits
with non-empty ref-bindings which aren’t pointed at by a ref).

This is useful when generating a new repository from scratch, but not
useful when adding new commits to an existing repository (since the old
commits will still, correctly, have ref-bindings from when the refs
pointed at them). That’s why it has to be enabled explicitly using
--verify-back-refs, rather than being on by default.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1347
Approved by: cgwalters
2017-12-14 14:18:44 +00:00
Philip Withnall 38152d71aa lib/repo: Clarify documentation for ostree_repo_list_refs{,_ext}()
Try and clarify what happens with the prefixes, and that they always
return refspecs.

I’m still not 100% sure this is right.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1347
Approved by: cgwalters
2017-12-14 14:18:44 +00:00
Philip Withnall b0e7b26921 ostree/fsck: Handle refspecs from ostree_repo_list_refs()
It seems ostree_repo_list_refs() can return refspecs as hash table keys,
as well as just ref names. Handle that by parsing them before trying to
use them as ref names.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1347
Approved by: cgwalters
2017-12-14 14:18:44 +00:00
Philip Withnall 97bdb3b271 ostree/fsck: Verify commit bindings for each ref
Since an OSTree client will refuse to pull from a remote which it has
locally configured with a collection ID, if the commit on that remote
has incorrect or missing bindings, we’d better verify them as part of
fsck.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1347
Approved by: cgwalters
2017-12-14 14:18:44 +00:00
Philip Withnall 931cbe6fc9 lib/static-delta: Drop duplicated declaration from private header
Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1347
Approved by: cgwalters
2017-12-14 14:18:44 +00:00
Philip Withnall 609bd4748e lib/pull: Fix capitalisation in binding verification error messages
Make them suitable for output from fsck.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1347
Approved by: cgwalters
2017-12-14 14:18:44 +00:00
Philip Withnall 1b7d83114e lib/pull: Split verify_bindings() out into a cmdprivate method
It will be used by the fsck utility in future. We could expose it
publicly in future too, if needed.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1347
Approved by: cgwalters
2017-12-14 14:18:44 +00:00
Philip Withnall 1160d3a110 ostree/fsck: Factor out common commit checking code
This will make upcoming commits a bit cleaner.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1347
Approved by: cgwalters
2017-12-14 14:18:44 +00:00
Philip Withnall 4a58364cfa lib/repo: Fix a memory leak of options in ostree_repo_create()
Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1341
Approved by: dbnicholson
2017-11-14 23:13:14 +00:00
Philip Withnall f7568dbfc8 lib/repo: Add (transfer) annotations to various GHashTable arguments
By default, unless it’s const, an (out) GHashTable will be assumed to be
(transfer full). That means the binding needs to free all the items in
the hash table, plus the table itself.

However, all the GHashTables we use have free functions set already, so
freeing the hash table will free its items. This results in a
double-free.

Fix that by ensuring we annotate such (out) hash tables as (transfer
container). Also annotate some other hash tables as (transfer none)
where appropriate, for clarity.

This fixes OSTree.Repo.list_collection_refs() in the Python bindings.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1341
Approved by: dbnicholson
2017-11-14 23:13:14 +00:00
Philip Withnall 3cf53f7c58 lib/repo: Add gtk-doc comment to OstreeRepoCommitState
Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1335
Approved by: cgwalters
2017-11-08 15:47:11 +00:00
Philip Withnall 9c4870b5e1 lib/repo: Add OSTREE_REPO_COMMIT_STATE_NORMAL to represent most commits
This allows more explicit handling of commit state in code using
libostree, rather than hard-coding a commit state of 0 for ‘normal’.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1335
Approved by: cgwalters
2017-11-08 15:47:11 +00:00
Philip Withnall 7d2ad351c4 build: Add a TODO comment about improving glib-mkenums usage in future
Signed-off-by: Philip Withnall <withnall@endlessm.com>

https://github.com/ostreedev/ostree/pull/1329

Closes: #1330
Approved by: cgwalters
2017-11-07 15:14:56 +00:00
Philip Withnall 0760ce1281 lib/repo-finder-mount: Ignore mounts which have a ‘system’ file system
For example, tmpfs or a cgroup file system. This is basically an
optimisation of the list of file systems we check for repositories,
since we would never expect any of these file systems to be capable of
containing a repository.

Depends on the new API from
https://bugzilla.gnome.org/show_bug.cgi?id=788927.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1307
Approved by: cgwalters
2017-10-26 01:34:14 +00:00
Philip Withnall 886a5d7928 lib/repo-finder-mount: Fix propagating NULL errors
These two code paths tried to propagate errors which had never been set.
Set new errors instead.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1303
Approved by: cgwalters
2017-10-24 16:59:54 +00:00
Philip Withnall 90680e1b29 lib/repo-finder-avahi: Fix memory corruption of a GVariantIter
A GVariantIter* was being passed to a GVariant format string varargs,
rather than a GVariantIter**. This resulted in memory corruption.

So we can continue to reuse ref_map throughout the function, make it a
GVariantIter* rather than a stack-allocated GVariantIter.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1301
Approved by: cgwalters
2017-10-24 12:55:24 +00:00
Philip Withnall 4c0f67be0c lib/sysroot: Expand docs for ostree_sysroot_get_repo()
Add some missing annotations and clarify that it always returns an open
repository on success.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1300
Approved by: cgwalters
2017-10-23 16:17:20 +00:00
Philip Withnall 2531d8fe63 lib/repo-finder: Add OstreeRepoFinderOverride
This is another OstreeRepoFinder implementation; it returns results from
a given set of URIs. It’s designed to be used for implementing user
overrides to other repo-finders, or for implementing unit tests.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1281
Approved by: mwleeds
2017-10-19 19:11:58 +00:00
Philip Withnall a1f020ffa4 lib/repo-finder-avahi: Fix a leak in a GVariantIter loop
Use g_variant_iter_loop() rather than next(), since it automatically
handles freeing the child memory each iteration. Previously, we leaked
it for all but the last iteration.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1293
Approved by: cgwalters
2017-10-19 13:16:59 +00:00
Philip Withnall 9350e8a488 lib/repo: Clarify that ostree_repo_remote_fetch_summary() doesn’t verify
Make that a bit clearer in the documentation.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1253
Approved by: cgwalters
2017-10-05 12:48:28 +00:00
Philip Withnall 43977dffde docs: Clarify %NULL argument to ostree_sysroot_new()
It’s equivalent to ostree_sysroot_new_default().

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1246
Approved by: cgwalters
2017-10-04 12:54:23 +00:00
Philip Withnall 86e072bdbe lib/repo-commit: Import detached metadata even if hardlink exists
Spotted while reading through the code, it looks like the
copy_detached_metadata() call is accidentally omitted if a hardlink
already exists for the .commit object.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1242
Approved by: cgwalters
2017-10-03 16:31:13 +00:00
Philip Withnall 030e3efbc4 lib/bloom: Add some missing preconditions on n_bytes
These shouldn’t change the bloom filter’s behaviour at all, but make it
a bit more obvious what the programmatical limitations are on the sizes
it can deal with.

In reality, those sizes should never be reached because they won’t fit
in a DNS-SD record.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1239
Approved by: cgwalters
2017-10-02 18:08:55 +00:00
Philip Withnall feeb3548f4 ostree/summary: Generate an ostree-metadata ref when updating summary
This is the new way of publishing repository metadata, rather than as
additional-metadata in the summary file. The use of an ostree-metadata
ref means that the metadata from multiple upstream collections is not
conflated when doing P2P mirroring of many repositories.

The new ref is only generated if the repository has a collection ID set.
The old summary file continues to be generated for backwards
compatibility (and because it continues to be the canonical ref →
checksum map for the repository).

The new code is only used if configured with --enable-experimental-api.

Includes unit tests.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1158
Approved by: cgwalters
2017-10-02 13:39:41 +00:00
Philip Withnall 16aa4d728a ostree/summary: Eliminate redundant gotos from error handling path
There is no error handling to do, so just return everywhere instead.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1158
Approved by: cgwalters
2017-10-02 13:39:41 +00:00
Philip Withnall d1a5652bdd man: Fix a copypasta error in ostree-summary.xml
Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1158
Approved by: cgwalters
2017-10-02 13:39:41 +00:00
Philip Withnall 1f666eb07d lib/bloom: Fix a -Wconversion warning in OstreeBloom
Compiling with -Wconversion warns on this line, as the conversion from
guint64 to guint8 is implicit (but safe: there is no bug here, since the
implicit cast is applied after the modulus arithmetic).

Make the cast explicit to silence -Wconversion.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1231
Approved by: cgwalters
2017-10-01 12:24:46 +00:00
Philip Withnall 1673601510 lib/bloom: Fix bloom hashing on 32-bit architectures
There was an implicit cast from guint64 to gsize (which is 32-bit on
armhf, for example) before the modulus arithmetic which safely narrows
the index.

Fix that by using a guint64 intermediate variable and making the cast
explicit.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1231
Approved by: cgwalters
2017-10-01 12:24:46 +00:00
Philip Withnall 6c7302fcdd tests/repo: Drop modeline from top of file
As per commit 6e4146a3.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1203
Approved by: cgwalters
2017-09-28 14:08:40 +00:00
Philip Withnall 467fcff349 tests: Update some tests to use OSTREE_REPO_MODE_ARCHIVE not ARCHIVE_Z2
The latter is deprecated now.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1203
Approved by: cgwalters
2017-09-28 14:08:40 +00:00
Philip Withnall df1d0a5fc6 tests: Add a manual integration test for OstreeRepoFinderMount
Test an end-to-end flow of pulling refs from an online
repository → local OS repository → create a USB stick of
them → pull to a local OS repository on another machine.

This is a manual test, as it requires a throwaway USB stick which the
test can format as ext4 or vfat to test the flow works with both file
systems.

Run it as:

MOUNT_INTEGRATION_DEV=/dev/sdb1 make check \
  TESTS=tests/test-repo-finder-mount-integration.sh

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1203
Approved by: cgwalters
2017-09-28 14:08:40 +00:00
Philip Withnall 9d8c1ec7df lib/repo-pull: Fix remote names in refspecs from non-mirror P2P pulls
Propagate the refspec_name from the OstreeRemote returned by an
OstreeRepoFinder through to the set_ref() call.

This changes ostree_repo_pull_with_options() to accept the
previously-disallowed combination of passing override-remote-name in
options and also setting a remote name in remote_name_or_baseurl.
ostree_repo_pull_with_options() will continue to pull using the remote
config named in remote_name_or_baseurl as before; but will now use the
remote name from override-remote-name when it’s setting the refs at the
end of the pull. This is consistent with the documentation for
override-remote-name.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1202
Approved by: cgwalters
2017-09-27 16:38:07 +00:00
Philip Withnall 69e332a0c0 lib/remote: Store name of remote providing keyring for dynamic remotes
When pulling from a dynamic (peer to peer) remote, the remote’s name is
set to a unique, generated string which doesn’t exist in repo/config. If
doing a non-mirror pull, however, we don’t want to use this name in the
refspecs for newly created or updated refs — we want to use the name of
the remote which provided the keyring for the pull (this will be a
remote from repo/config whose collection ID matches that being used for
the peer to peer pull).

Store both names in OstreeRemote. The name to use for refspecs is stored
as refspec_name, and is typically NULL unless it differs from name.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1202
Approved by: cgwalters
2017-09-27 16:38:07 +00:00
Philip Withnall 22c1fdfbd3 lib/repo: Change resolve_keyring_for_collection() to return a remote
Instead of returning just the keyring filename, return the entire
OstreeRemote, which has the keyring filename as one of its members. This
will simplify some upcoming changes, and allows slightly improved debug
logging.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1202
Approved by: cgwalters
2017-09-27 16:38:07 +00:00
Philip Withnall 030d2b1525 lib/repo-pull: Fix a potential minor leak
If override-remote-name is specified in the options to
ostree_repo_pull_with_options(), but the remote_name_or_baseurl argument
is also set to a remote name, the override-remote-name would be leaked.

Note that this is currently an invalid configuration, so this leak is
basically never hit.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1202
Approved by: cgwalters
2017-09-27 16:38:07 +00:00
Philip Withnall 2f9f7222a4 lib/repo-pull: Use resolve() instead of list() in fetch_ref_contents()
This is more efficient in the non-collection case; in the collection
case, the implementation of ostree_repo_resolve_collection_ref() needs
to be rewritten to improve efficiency.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1182
Approved by: cgwalters
2017-09-27 14:44:00 +00:00
Philip Withnall 149aec1099 lib/repo-refs: Add first version of ostree_repo_resolve_collection_ref()
This is a parallel for ostree_repo_resolve_rev_ext() which works on
collection–refs. At the moment, the implementation is simple and uses
ostree_repo_list_collection_refs(). In future, it could be rewritten to
check the checksum directly rather than enumerating all
potentially-relevant checksums.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1182
Approved by: cgwalters
2017-09-27 14:44:00 +00:00
Philip Withnall 9546e6795e create-usb: Add a create-usb command to complement OstreeRepoFinderMount
This can be used to put OSTree repositories on USB sticks in a format
recognised by OstreeRepoFinderMount.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1182
Approved by: cgwalters
2017-09-27 14:44:00 +00:00
Philip Withnall f923c2e1ea src/pull: Support local pulls for collection–refs
Previously, collection–refs could only be pulled from a repository if it
had a summary file (which listed them). There was no way to pull from a
local repository which doesn’t have a summary file, and where the refs
were stored as refs/remotes/$remote/$ref, with a config section linking
that $remote to the queried collection ID.

Fix that by explicitly supporting pull_data->remote_repo_local in
fetch_ref_contents().

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1182
Approved by: cgwalters
2017-09-27 14:44:00 +00:00
Philip Withnall cd70aaac11 lib/repo-finder-mount: Add var/lib/flatpak as a well-known directory
Add this as an additional well-known directory which is checked on
mounted removable drives to see if it contains OSTree repos we can pull
refs from.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

https://github.com/ostreedev/ostree/issues/1210

Closes: #1213
Approved by: cgwalters
2017-09-25 18:06:51 +00:00
Philip Withnall 64b23fd089 lib/repo: Add ostree_repo_hash() and tests
Add a hash function for OstreeRepo instances, which relies on the repo
being open, and hence being able to hash the device and inode of its
root directory.

Add unit tests for this and ostree_repo_equal().

Signed-off-by: Philip Withnall <withnall@endlessm.com>

https://github.com/ostreedev/ostree/issues/1191

Closes: #1205
Approved by: cgwalters
2017-09-21 21:25:58 +00:00
Philip Withnall c62b36046b lib/repo: Drop outdated FIXME comment from ostree-repo.h
This was some incomplete planning from while the find_remotes() API was
being designed; now totally outdated.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1179
Approved by: cgwalters
2017-09-19 14:51:09 +00:00
Philip Withnall 15247641d9 lib/repo-finder-mount: Change the schema for finding repos on volumes
See issue #1174 for the rationale behind this. In summary:
 • It required two lists of collection–refs to be maintained: one in the
   repository, and one pointing to the repository.
 • It didn’t automatically work for live USBs of OSs based on OSTree
   (where there’s always a repository at /ostree/repo).
 • It was unnecessarily complex.

The new scheme allows a list of repositories to be searched, but without
needing a layer of indirection through their collection–refs. It adds
/ostree/repo and /.ostree/repo as well-known repository locations which
are always checked on a mounted volume (if they exist).

Update the unit tests accordingly.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

https://github.com/ostreedev/ostree/issues/1174

Closes: #1179
Approved by: cgwalters
2017-09-19 14:51:09 +00:00
Philip Withnall 981eb6c226 lib/repo: Add ostree_repo_equal() for comparing repos
This will compare their root directory inodes to see if they are the
same repository on disk. A convenience method for the users of the
public API who can’t access OstreeRepo.inode.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1179
Approved by: cgwalters
2017-09-19 14:51:09 +00:00
Philip Withnall 80eab25f8a build: Ensure ostree-tmpfiles.conf is distributed
Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1103
Approved by: cgwalters
2017-08-22 17:37:48 +00:00
Philip Withnall e3a4049c24 build: Add distcheck configure flags to fix systemd and bash-completion
When running distcheck, the systemd system-generator and bash-completion
scripts are installed in absolute paths (/usr and /lib) as looked up
from their pkg-config files. This breaks distcheck. Use a
${prefix}-relative path for both of them when configuring for distcheck.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1103
Approved by: cgwalters
2017-08-22 17:37:48 +00:00
Philip Withnall 2522db0164 lib/repo-finder-config: Add some more debug output
This makes diagnosing false negatives a little easier.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1075
Approved by: jlebon
2017-08-11 18:07:29 +00:00
Philip Withnall 75bce24cb9 lib/gpg-verify: Add an OstreeGpgError error domain
Add a new error domain for GPG signing/verification errors, and use it
throughout libostree for describing verification errors. This replaces
various uses of G_IO_ERROR_FAILED, and one instance of
G_IO_ERROR_NOT_FOUND (for which some code in ot-builtin-show.c had to be
changed to ensure it was still handled correctly).

The use of a separate error domain allows failures in GPG operations to
be handled separately from network failures (where the summary file
could not be found to be downloaded, for example) or timeouts.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1064

Closes: #1071
Approved by: mbarnes
2017-08-10 13:38:40 +00:00
Philip Withnall 8c148eb7e1 lib/repo-finder: Emit gpg-verify-summary=false in dynamic remote config
When returning results from finding repos, set gpg-verify-summary=false
in their configs, since any pulls from such remotes will necessarily
involve collection IDs, and hence should be using the unsigned summary
support. In the intended deployment mode for P2P transmission of OSTree
refs, summaries *cannot* be signed, so setting gpg-verify-summary=true
would cause all the pulls to fail.

The unsigned summary support is the move of repository metadata from
the summary file (not spliceable) to the well-known ostree-metadata ref
(spliceable, as it can exist for multiple collection IDs in the same
repository).

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1066
Approved by: cgwalters
2017-08-08 18:08:05 +00:00
Philip Withnall 1672e2eee0 lib/repo: Fix handling of missing summary files when downloading
The API for downloading a summary file can legitimately return NULL for
the summary file contents when it returns TRUE (success). This indicates
an error 404 — the summary file was not found.

Two call sites were not handling that correctly, which was causing later
assertion failures.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1061

Closes: #1065
Approved by: cgwalters
2017-08-08 15:33:45 +00:00
Philip Withnall 86dce4b252 ostree/parse-datetime: Ensure tm structs are initialised
Otherwise tm.tm_wday remains uninitialised and gets propagated
elsewhere.

Spotted by Coverity as issue #209265.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1059
Approved by: cgwalters
2017-08-08 14:11:07 +00:00
Philip Withnall e9b9123bae lib/sepolicy: Drop duplicate assignment
Spotted by Coverity as issue #1452619.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1059
Approved by: cgwalters
2017-08-08 14:11:07 +00:00
Philip Withnall 4f187b576d lib/repo-commit: Drop unreachable conditional branch
(remaining > 0) is asserted by the loop condition, and remaining is not
modified between that check and the G_UNLIKELY — so the condition in the
G_UNLIKELY will always be true.

Spotted by Coverity as issue #1452617.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1059
Approved by: cgwalters
2017-08-08 14:11:07 +00:00
Philip Withnall baa9534fda lib/repo-finder-avahi: Drop redundant conditional
summary_timestamp is checked for non-NULL-ness above, and the function
bails if it’s NULL.

Fixes Coverity issue #1452616.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1059
Approved by: cgwalters
2017-08-08 14:11:06 +00:00
Philip Withnall f35b409077 lib/repo-refs: Add ostree_repo_remote_list_collection_refs() API
This parallels ostree_repo_remote_list_refs(), but returns a map of
OstreeCollectionRef → checksum, and includes refs from collection IDs
other than the remote repository’s main collection ID.

Use this in OstreeRepoFinderConfig to ensure that refs are matched
against even if they’re stored in the repository summary file’s
collection map, rather than its main ref map. This fixes false negatives
when searching for refs in some situations.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1058
Approved by: cgwalters
2017-08-08 13:59:58 +00:00
Philip Withnall 11e165b154 lib/repo-pull: Add a missing precondition
This catches a few failure modes in the pull code a little earlier,
before the incorrectly-NULL repo makes its way into a closure and a
worker thread, where the cause of the problem is harder to track down.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1058
Approved by: cgwalters
2017-08-08 13:59:58 +00:00
Philip Withnall 0d2a9a7909 lib/repo-finder: Avoid a potential unref-of-NULL crash
As the comment explains, it’s possible for a result to be freed while
ref_to_checksum is NULL, even though normally the data structure
guarantees it’s non-NULL. This was causing crashes when results were
filtered out of a find-remotes call. Guard against that.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1058
Approved by: cgwalters
2017-08-08 13:59:58 +00:00
Philip Withnall be4832242d lib/repo-pull: Fix counting of latest commits when finding repos
The intended behaviour of ostree_repo_find_remotes() is to return
results which have the latest version of at least one of the requested
refs. Results which have some of the requested refs, but don’t have the
latest version of any of them, should be ignored. The logic to do this
was broken in the case that a result contained a positive number of the
requested refs, but none of them were the latest version. (It previously
worked when the result contained none of the requested refs.)

Fix the counting to work correctly in both cases.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1058
Approved by: cgwalters
2017-08-08 13:59:58 +00:00
Philip Withnall 82024f161b build: Ensure all experimental tests are distributed in tarballs
As with the previous commit, ensure that tests which are run when
configured with --enable-experimental-api, are always distributed; even
when running `make dist` from a source tree configured with
--disable-experimental-api.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1002
Approved by: cgwalters
2017-07-12 14:23:44 +00:00
Philip Withnall f6ba20d9d6 build: Ensure all .sym files are distributed in tarballs
Since we’re using a custom variable for listing the .sym files,
automake’s magic support for automatically distributing all files in
conditionals doesn’t work, and the devel and experimental .sym files
were only being distributed if `make dist` was run on a source tree
which had been configured with --enable-experimental-api or not a
release flag.

Fix that by explicitly listing all the .sym files in EXTRA_DIST.

Specifically, this fixes the case of trying to compile with
--enable-experimental-api from a release tarball which was disted with
--disable-experimental-api.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1001
Approved by: cgwalters
2017-07-12 13:04:32 +00:00
Philip Withnall acb14648d7 lib/repo: Add OSTREE_REPO_METADATA_REF as a well-known metadata store
As discussed in https://github.com/ostreedev/ostree/pull/946, the
summary file is becoming an unsigned cache of ref information; any
additional metadata for the repository needs to move elsewhere in order
to remain signed. Introduce OSTREE_REPO_METADATA_REF as the well-known
name of a ref where such metadata can live, as the metadata on
contentless commits.

Don’t yet update the documentation for summary-related methods to
mention this, since it’s still hidden behind the
--enable-experimental-api configure option.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #946
Approved by: cgwalters
2017-07-11 19:50:32 +00:00
Philip Withnall 7d57459e83 lib/repo-commit: Fix types of content size cache entries
Use goffset rather than gsize for file sizes. More importantly, get the
unpacked_size from g_file_info_get_size() (goffset) rather than from the
splice return value, which has type gssize.

This will make a difference on 32-bit systems, where goffset is defined
as off64_t, but gsize is 32 bits.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #999
Approved by: cgwalters
2017-07-11 14:55:55 +00:00
Philip Withnall acace571ef lib/repo: Fix repo-finder deleting remote configs when run
An inverted condition in _ostree_repo_add_remote() was causing the
OstreeRepoFinder to delete precisely the wrong remote
configurations from memory once it was finished. It’s supposed to delete
the ones which it transiently added; but was instead deleting all the
existing remote configurations.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #985
Approved by: cgwalters
2017-06-29 23:49:25 +00:00
Philip Withnall a03f0447cb lib/pull: Use ostree_repo_verify_summary() to verify summary on pull
Rather than duplicating the code. This introduces no functional changes.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #961
Approved by: cgwalters
2017-06-27 19:19:32 +00:00
Philip Withnall 4c238c350d lib/pull: Check whether summary is in normal form when loading it
Check that it’s in normal form and has the correct type when loading it,
since it could come from an untrusted source.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #961
Approved by: cgwalters
2017-06-27 19:19:32 +00:00
Philip Withnall eb4887d619 lib/pull: Don’t cache summary file until its signature is verified
This makes no difference to the validity of the code, since any summary
file loaded from the cache will be verified before being read anyway;
but it will make some upcoming changes a little simpler.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #961
Approved by: cgwalters
2017-06-27 19:19:32 +00:00
Philip Withnall e0ad9b2266 ostree/summary: Add support for adding additional metadata
When updating a summary file, parse additional arguments to the `ostree
summary` command as additional metadata to be put into the summary.

Add some tests for this.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #961
Approved by: cgwalters
2017-06-27 19:19:32 +00:00
Philip Withnall a432a2b420 tests: Fix incorrect summary --update usage in test-local-pull.sh
Previously the spurious ‘update’ atom was ignored; now that `ostree
summary --update` accepts extra arguments as additional metadata, it’s
causing an error. Drop it.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #961
Approved by: cgwalters
2017-06-27 19:19:32 +00:00
Philip Withnall 2f297ba4d3 lib/repo: Fix a typo in a documentation comment
Looks like a copy-paste error.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #961
Approved by: cgwalters
2017-06-27 19:19:32 +00:00
Philip Withnall d0d5f54ce2 lib/refs: Add runtime error checking for collection ID validity
Instead of treating it as a programming error — given that it’s user
input, that’s not really appropriate. This modifies write_ref() and
list_collection_refs() to implement validation.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall 11ad3ec529 tests: Add integration tests for collections
Test various pieces of core and command line utility functionality
relating to the newly-introduced concept of collections. Mostly focussed
around the find-remotes utility, and around handling of collection–refs.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall 3dd4848c96 ostree/builtins: Add support for collection–refs to a few utilities
These utilities were not needed for the initial port to support
OstreeCollectionRef, so have been delayed a bit and, in some cases, left
as FIXME comments for follow up later.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall 18456d25fb ostree/dump: Include collection IDs and mirrored refs in summary dumps
If a repository’s summary file includes a collection ID, output that. If
it includes refs from other collections (in the ‘collection map’), output
those and include the same metadata detail as for refs in the summary
file’s main refs map.

If collection IDs are specified in the summary file, this changes the
output format from `ostree summary -v` to use (collection ID, ref name)
tuples.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall b7b79fa78d refs: Add a --collections argument to the built-in refs command
Rather than change the output format used by the existing refs command
to output collection IDs in addition to ref names, this functionality has
been hidden behind an --collections argument. If it’s not specified `ostree
refs` will output the same content as before for a given repository. If
it is specified, the collection ID for each ref will be included in the
output as (collection ID, ref name).

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall 205a84b36e remote-add: Add a --collection-id argument to the built-in add command
This allows newly configured remotes to have their collection ID specified,
so that refs from them can be downloaded from peers as well as the
upstream collection, using the remote’s configuration.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall 144e325579 init: Add a --collection-id argument to the built-in init command
This allows new repositories to be configured with a collection ID which
can be used to uniquely identify refs which originated from this
repository.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall 37fc49f36d find-remotes: Add pull support to the find-remotes built-in command
This will pull the remotes after finding them. This potentially needs to
go in its own pull-from-remotes built-in command, but it will be fine
here for now.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall 6453203f54 find-remotes: Add a find-remotes built-in command
This is a wrapper around the new ostree_repo_find_remotes() method; it
tries to find available remotes which can serve updates for the
user-provided refs.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall e3d4eeacbc lib/repo-finder: Add Avahi based OstreeRepoFinder implementation
This is a more complex implementation of OstreeRepoFinder which resolves
ref names to remote URIs by looking for refs advertised by peers on the
local network using DNS-SD records and mDNS (Avahi). The idea is to
allow OS and app updates to be propagated over local networks, without
the internet.

It requires an OSTree server and code to generate the DNS-SD adverts in
order to be fully functional — support for this will be added
separately.

Unit tests are included.

Includes fixes by Krzesimir Nowak <krzesimir@kinvolk.io>.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall 7ee4e1295a lib/bloom: Add an internal bloom filter implementation
This will be used in an upcoming commit. It adds a basic bloom filter
implementation, using the SipHash family of hash functions.

The implementation (including its parameter choices and hash functions)
will become a protocol detail in future, so must not be changed so that
its output is bitwise incompatible between OSTree versions.

Unit tests are included.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall ae335f24dc lib/repo-finder: Add mount based OstreeRepoFinder implementation
This is a basic implementation of OstreeRepoFinder which resolves ref
names to remote URIs by looking for them on any currently mounted
removable storage volumes. The idea is to support OS and app updates via
USB stick.

Unit tests are included.

This bumps libostree’s maximum GLib dependency from 2.44 to 2.50 for
g_drive_is_removable(). If GLib 2.50 is not available, the call which
needs it will be omitted and the OstreeRepoFinderMount implementation
will scan all volumes (not just removable ones); this is a performance
hit, but not a functionality hit.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall d15f83c922 lib/repo-finder: Add config-file based OstreeRepoFinder implementation
This is a basic implementation of OstreeRepoFinder which resolves ref
names to remote URIs by looking their collection IDs up in the local
configuration of remotes who have their collection-id key set.

Unit tests are included.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall 292230301d lib/repo-finder: Add basic support for finding remote URIs by ref name
Add an initial OstreeRepoFinder interface (but no implementations),
which will find remote URIs by ref names and collection IDs, the
combination of which is globally unique.

The new API is used in a new ostree_repo_find_updates() function, which
resolves a list of ref names to update into a set of remote URIs to pull
them from, which can be treated as mirrors. It is an attempt to
generalise resolution of the URIs to pull from, and to generalise
determination of the order and parallelisation which they should be
downloaded from in.

Includes fixes by Krzesimir Nowak <krzesimir@kinvolk.io>.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall 7607d94713 lib/pull: Add collection support to ostree_repo_pull_with_options()
This adds a new collection-refs option of type a(sss), giving a list of
(collection ID, ref name, checksum) tuples to pull from the given remote.
This option is intended to supersede the refs and override-commit-ids
options, so is mutually exclusive with them.

This includes support for resolving the refs from the remote’s summary
file, or from its refs/heads and refs/mirrors directories.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall fbf8df8829 lib/refs: Add methods for setting/listing collection–refs
These are tuples of (collection ID, ref name) which are a globally-unique
form of local ref. They use OstreeCollectionRef as an identifier, and hence
need to be accessed using new API, as the existing API uses string
identifiers and sometimes accepts refspecs. Remote names are not
supported as part an OstreeCollectionRef.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall 4de736fdfa lib/repo: Add collection ID support to OstreeRepo
Add {get,set}_collection_id() methods to OstreeRepo and some documentation
about the concept of a collection ID which globally identifies an
upstream repository. See the documentation for more details.

This will be used in future commits. For now, the new API is marked as
experimental (--enable-experimental-api).

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall 0a20e7d43c lib/ref: Add OstreeCollectionRef type for globally unique refs
This is a type representing the tuple (collection ID, ref name), which is
guaranteed to be globally unique. It will be used in upcoming commits.

It introduces the concept of a ‘collection’ which is a unique, curated
set of refs which lie in the same trust domain (i.e. all signed by the
same key and validated by the same developer). Flathub might be a
collection, for example; or the set of OS refs coming from a particular
OS vendor.

It includes a function for validating collection IDs.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall bf1f8eb0fa lib/repo: Split out ref handling from regenerate_summary()
This will make some future additions to regenerate_summary() easier.
This commit introduces no functional changes.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall 9a79d13ce3 lib/remote: Fix ‘Since’ line for OstreeRemote
There was a typo in it when it was first introduced. Let’s not live in
the past.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #950
Approved by: cgwalters
2017-06-21 13:00:55 +00:00
Philip Withnall 20829a0582 lib/sym: Fix symbol versions for 2017.7 experimental symbols
There was a typo in the group name. It should be OK to change the
version since this is all hidden behind the --enable-experimental-api
configure option.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #950
Approved by: cgwalters
2017-06-21 13:00:55 +00:00
Philip Withnall 333d264c5b lib/core: Fix ‘Since’ line for ostree_validate_remote_name()
This was missed when cherry-picking it out of
https://github.com/ostreedev/ostree/pull/924#discussion_r123097919.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #950
Approved by: cgwalters
2017-06-21 13:00:55 +00:00
Philip Withnall 20dc9454b3 lib/core: Add ostree_validate_remote_name() for remote names
There are a few places in the code where ad-hoc validation was being
performed. Might as well formalise it a bit more.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #948
Approved by: cgwalters
2017-06-20 21:52:22 +00:00
Philip Withnall fd50c97322 build: Add ‘devel’ or ‘release’ to OSTREE_FEATURES for test-symbols.sh
test-symbols.sh was looking for the DevelBuild string, which is actually
part of the output from `ostree --version`, not $OSTREE_FEATURES.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #948
Approved by: cgwalters
2017-06-20 21:52:22 +00:00
Philip Withnall 21eec96bfd lib/pull: Fix construction of a refspec to use the correct separator
This code looks like it was supposed to build a refspec, but it used a
slash as a separator rather than a colon. The following code does
recover by supporting prefix matching with slashes, but it seems like
this was perhaps not the intention.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #912
Approved by: cgwalters
2017-06-13 01:03:49 +00:00
Philip Withnall 4418ab7fa9 lib/fetcher: Add cleanup function for OstreeFetcher
This is only used internally (the header is not public), so it doesn’t
have to go in ostree-autocleanups.h. It will be used in some following
commits.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #911
Approved by: cgwalters
2017-06-07 16:55:36 +00:00