Commit Graph
88 Commits
Author SHA1 Message Date
Philip Withnall 2522db0164 lib/repo-finder-config: Add some more debug output
This makes diagnosing false negatives a little easier.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1075
Approved by: jlebon
2017-08-11 18:07:29 +00:00
Philip Withnall 75bce24cb9 lib/gpg-verify: Add an OstreeGpgError error domain
Add a new error domain for GPG signing/verification errors, and use it
throughout libostree for describing verification errors. This replaces
various uses of G_IO_ERROR_FAILED, and one instance of
G_IO_ERROR_NOT_FOUND (for which some code in ot-builtin-show.c had to be
changed to ensure it was still handled correctly).

The use of a separate error domain allows failures in GPG operations to
be handled separately from network failures (where the summary file
could not be found to be downloaded, for example) or timeouts.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1064

Closes: #1071
Approved by: mbarnes
2017-08-10 13:38:40 +00:00
Philip Withnall 8c148eb7e1 lib/repo-finder: Emit gpg-verify-summary=false in dynamic remote config
When returning results from finding repos, set gpg-verify-summary=false
in their configs, since any pulls from such remotes will necessarily
involve collection IDs, and hence should be using the unsigned summary
support. In the intended deployment mode for P2P transmission of OSTree
refs, summaries *cannot* be signed, so setting gpg-verify-summary=true
would cause all the pulls to fail.

The unsigned summary support is the move of repository metadata from
the summary file (not spliceable) to the well-known ostree-metadata ref
(spliceable, as it can exist for multiple collection IDs in the same
repository).

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1066
Approved by: cgwalters
2017-08-08 18:08:05 +00:00
Philip Withnall 1672e2eee0 lib/repo: Fix handling of missing summary files when downloading
The API for downloading a summary file can legitimately return NULL for
the summary file contents when it returns TRUE (success). This indicates
an error 404 — the summary file was not found.

Two call sites were not handling that correctly, which was causing later
assertion failures.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1061

Closes: #1065
Approved by: cgwalters
2017-08-08 15:33:45 +00:00
Philip Withnall 86dce4b252 ostree/parse-datetime: Ensure tm structs are initialised
Otherwise tm.tm_wday remains uninitialised and gets propagated
elsewhere.

Spotted by Coverity as issue #209265.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1059
Approved by: cgwalters
2017-08-08 14:11:07 +00:00
Philip Withnall e9b9123bae lib/sepolicy: Drop duplicate assignment
Spotted by Coverity as issue #1452619.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1059
Approved by: cgwalters
2017-08-08 14:11:07 +00:00
Philip Withnall 4f187b576d lib/repo-commit: Drop unreachable conditional branch
(remaining > 0) is asserted by the loop condition, and remaining is not
modified between that check and the G_UNLIKELY — so the condition in the
G_UNLIKELY will always be true.

Spotted by Coverity as issue #1452617.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1059
Approved by: cgwalters
2017-08-08 14:11:07 +00:00
Philip Withnall baa9534fda lib/repo-finder-avahi: Drop redundant conditional
summary_timestamp is checked for non-NULL-ness above, and the function
bails if it’s NULL.

Fixes Coverity issue #1452616.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1059
Approved by: cgwalters
2017-08-08 14:11:06 +00:00
Philip Withnall f35b409077 lib/repo-refs: Add ostree_repo_remote_list_collection_refs() API
This parallels ostree_repo_remote_list_refs(), but returns a map of
OstreeCollectionRef → checksum, and includes refs from collection IDs
other than the remote repository’s main collection ID.

Use this in OstreeRepoFinderConfig to ensure that refs are matched
against even if they’re stored in the repository summary file’s
collection map, rather than its main ref map. This fixes false negatives
when searching for refs in some situations.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1058
Approved by: cgwalters
2017-08-08 13:59:58 +00:00
Philip Withnall 11e165b154 lib/repo-pull: Add a missing precondition
This catches a few failure modes in the pull code a little earlier,
before the incorrectly-NULL repo makes its way into a closure and a
worker thread, where the cause of the problem is harder to track down.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1058
Approved by: cgwalters
2017-08-08 13:59:58 +00:00
Philip Withnall 0d2a9a7909 lib/repo-finder: Avoid a potential unref-of-NULL crash
As the comment explains, it’s possible for a result to be freed while
ref_to_checksum is NULL, even though normally the data structure
guarantees it’s non-NULL. This was causing crashes when results were
filtered out of a find-remotes call. Guard against that.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1058
Approved by: cgwalters
2017-08-08 13:59:58 +00:00
Philip Withnall be4832242d lib/repo-pull: Fix counting of latest commits when finding repos
The intended behaviour of ostree_repo_find_remotes() is to return
results which have the latest version of at least one of the requested
refs. Results which have some of the requested refs, but don’t have the
latest version of any of them, should be ignored. The logic to do this
was broken in the case that a result contained a positive number of the
requested refs, but none of them were the latest version. (It previously
worked when the result contained none of the requested refs.)

Fix the counting to work correctly in both cases.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1058
Approved by: cgwalters
2017-08-08 13:59:58 +00:00
Philip Withnall 82024f161b build: Ensure all experimental tests are distributed in tarballs
As with the previous commit, ensure that tests which are run when
configured with --enable-experimental-api, are always distributed; even
when running `make dist` from a source tree configured with
--disable-experimental-api.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1002
Approved by: cgwalters
2017-07-12 14:23:44 +00:00
Philip Withnall f6ba20d9d6 build: Ensure all .sym files are distributed in tarballs
Since we’re using a custom variable for listing the .sym files,
automake’s magic support for automatically distributing all files in
conditionals doesn’t work, and the devel and experimental .sym files
were only being distributed if `make dist` was run on a source tree
which had been configured with --enable-experimental-api or not a
release flag.

Fix that by explicitly listing all the .sym files in EXTRA_DIST.

Specifically, this fixes the case of trying to compile with
--enable-experimental-api from a release tarball which was disted with
--disable-experimental-api.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1001
Approved by: cgwalters
2017-07-12 13:04:32 +00:00
Philip Withnall acb14648d7 lib/repo: Add OSTREE_REPO_METADATA_REF as a well-known metadata store
As discussed in https://github.com/ostreedev/ostree/pull/946, the
summary file is becoming an unsigned cache of ref information; any
additional metadata for the repository needs to move elsewhere in order
to remain signed. Introduce OSTREE_REPO_METADATA_REF as the well-known
name of a ref where such metadata can live, as the metadata on
contentless commits.

Don’t yet update the documentation for summary-related methods to
mention this, since it’s still hidden behind the
--enable-experimental-api configure option.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #946
Approved by: cgwalters
2017-07-11 19:50:32 +00:00
Philip Withnall 7d57459e83 lib/repo-commit: Fix types of content size cache entries
Use goffset rather than gsize for file sizes. More importantly, get the
unpacked_size from g_file_info_get_size() (goffset) rather than from the
splice return value, which has type gssize.

This will make a difference on 32-bit systems, where goffset is defined
as off64_t, but gsize is 32 bits.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #999
Approved by: cgwalters
2017-07-11 14:55:55 +00:00
Philip Withnall acace571ef lib/repo: Fix repo-finder deleting remote configs when run
An inverted condition in _ostree_repo_add_remote() was causing the
OstreeRepoFinder to delete precisely the wrong remote
configurations from memory once it was finished. It’s supposed to delete
the ones which it transiently added; but was instead deleting all the
existing remote configurations.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #985
Approved by: cgwalters
2017-06-29 23:49:25 +00:00
Philip Withnall a03f0447cb lib/pull: Use ostree_repo_verify_summary() to verify summary on pull
Rather than duplicating the code. This introduces no functional changes.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #961
Approved by: cgwalters
2017-06-27 19:19:32 +00:00
Philip Withnall 4c238c350d lib/pull: Check whether summary is in normal form when loading it
Check that it’s in normal form and has the correct type when loading it,
since it could come from an untrusted source.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #961
Approved by: cgwalters
2017-06-27 19:19:32 +00:00
Philip Withnall eb4887d619 lib/pull: Don’t cache summary file until its signature is verified
This makes no difference to the validity of the code, since any summary
file loaded from the cache will be verified before being read anyway;
but it will make some upcoming changes a little simpler.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #961
Approved by: cgwalters
2017-06-27 19:19:32 +00:00
Philip Withnall e0ad9b2266 ostree/summary: Add support for adding additional metadata
When updating a summary file, parse additional arguments to the `ostree
summary` command as additional metadata to be put into the summary.

Add some tests for this.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #961
Approved by: cgwalters
2017-06-27 19:19:32 +00:00
Philip Withnall a432a2b420 tests: Fix incorrect summary --update usage in test-local-pull.sh
Previously the spurious ‘update’ atom was ignored; now that `ostree
summary --update` accepts extra arguments as additional metadata, it’s
causing an error. Drop it.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #961
Approved by: cgwalters
2017-06-27 19:19:32 +00:00
Philip Withnall 2f297ba4d3 lib/repo: Fix a typo in a documentation comment
Looks like a copy-paste error.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #961
Approved by: cgwalters
2017-06-27 19:19:32 +00:00
Philip Withnall d0d5f54ce2 lib/refs: Add runtime error checking for collection ID validity
Instead of treating it as a programming error — given that it’s user
input, that’s not really appropriate. This modifies write_ref() and
list_collection_refs() to implement validation.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall 11ad3ec529 tests: Add integration tests for collections
Test various pieces of core and command line utility functionality
relating to the newly-introduced concept of collections. Mostly focussed
around the find-remotes utility, and around handling of collection–refs.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall 3dd4848c96 ostree/builtins: Add support for collection–refs to a few utilities
These utilities were not needed for the initial port to support
OstreeCollectionRef, so have been delayed a bit and, in some cases, left
as FIXME comments for follow up later.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall 18456d25fb ostree/dump: Include collection IDs and mirrored refs in summary dumps
If a repository’s summary file includes a collection ID, output that. If
it includes refs from other collections (in the ‘collection map’), output
those and include the same metadata detail as for refs in the summary
file’s main refs map.

If collection IDs are specified in the summary file, this changes the
output format from `ostree summary -v` to use (collection ID, ref name)
tuples.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall b7b79fa78d refs: Add a --collections argument to the built-in refs command
Rather than change the output format used by the existing refs command
to output collection IDs in addition to ref names, this functionality has
been hidden behind an --collections argument. If it’s not specified `ostree
refs` will output the same content as before for a given repository. If
it is specified, the collection ID for each ref will be included in the
output as (collection ID, ref name).

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall 205a84b36e remote-add: Add a --collection-id argument to the built-in add command
This allows newly configured remotes to have their collection ID specified,
so that refs from them can be downloaded from peers as well as the
upstream collection, using the remote’s configuration.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall 144e325579 init: Add a --collection-id argument to the built-in init command
This allows new repositories to be configured with a collection ID which
can be used to uniquely identify refs which originated from this
repository.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall 37fc49f36d find-remotes: Add pull support to the find-remotes built-in command
This will pull the remotes after finding them. This potentially needs to
go in its own pull-from-remotes built-in command, but it will be fine
here for now.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall 6453203f54 find-remotes: Add a find-remotes built-in command
This is a wrapper around the new ostree_repo_find_remotes() method; it
tries to find available remotes which can serve updates for the
user-provided refs.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall e3d4eeacbc lib/repo-finder: Add Avahi based OstreeRepoFinder implementation
This is a more complex implementation of OstreeRepoFinder which resolves
ref names to remote URIs by looking for refs advertised by peers on the
local network using DNS-SD records and mDNS (Avahi). The idea is to
allow OS and app updates to be propagated over local networks, without
the internet.

It requires an OSTree server and code to generate the DNS-SD adverts in
order to be fully functional — support for this will be added
separately.

Unit tests are included.

Includes fixes by Krzesimir Nowak <krzesimir@kinvolk.io>.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall 7ee4e1295a lib/bloom: Add an internal bloom filter implementation
This will be used in an upcoming commit. It adds a basic bloom filter
implementation, using the SipHash family of hash functions.

The implementation (including its parameter choices and hash functions)
will become a protocol detail in future, so must not be changed so that
its output is bitwise incompatible between OSTree versions.

Unit tests are included.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall ae335f24dc lib/repo-finder: Add mount based OstreeRepoFinder implementation
This is a basic implementation of OstreeRepoFinder which resolves ref
names to remote URIs by looking for them on any currently mounted
removable storage volumes. The idea is to support OS and app updates via
USB stick.

Unit tests are included.

This bumps libostree’s maximum GLib dependency from 2.44 to 2.50 for
g_drive_is_removable(). If GLib 2.50 is not available, the call which
needs it will be omitted and the OstreeRepoFinderMount implementation
will scan all volumes (not just removable ones); this is a performance
hit, but not a functionality hit.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall d15f83c922 lib/repo-finder: Add config-file based OstreeRepoFinder implementation
This is a basic implementation of OstreeRepoFinder which resolves ref
names to remote URIs by looking their collection IDs up in the local
configuration of remotes who have their collection-id key set.

Unit tests are included.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall 292230301d lib/repo-finder: Add basic support for finding remote URIs by ref name
Add an initial OstreeRepoFinder interface (but no implementations),
which will find remote URIs by ref names and collection IDs, the
combination of which is globally unique.

The new API is used in a new ostree_repo_find_updates() function, which
resolves a list of ref names to update into a set of remote URIs to pull
them from, which can be treated as mirrors. It is an attempt to
generalise resolution of the URIs to pull from, and to generalise
determination of the order and parallelisation which they should be
downloaded from in.

Includes fixes by Krzesimir Nowak <krzesimir@kinvolk.io>.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall 7607d94713 lib/pull: Add collection support to ostree_repo_pull_with_options()
This adds a new collection-refs option of type a(sss), giving a list of
(collection ID, ref name, checksum) tuples to pull from the given remote.
This option is intended to supersede the refs and override-commit-ids
options, so is mutually exclusive with them.

This includes support for resolving the refs from the remote’s summary
file, or from its refs/heads and refs/mirrors directories.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall fbf8df8829 lib/refs: Add methods for setting/listing collection–refs
These are tuples of (collection ID, ref name) which are a globally-unique
form of local ref. They use OstreeCollectionRef as an identifier, and hence
need to be accessed using new API, as the existing API uses string
identifiers and sometimes accepts refspecs. Remote names are not
supported as part an OstreeCollectionRef.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall 4de736fdfa lib/repo: Add collection ID support to OstreeRepo
Add {get,set}_collection_id() methods to OstreeRepo and some documentation
about the concept of a collection ID which globally identifies an
upstream repository. See the documentation for more details.

This will be used in future commits. For now, the new API is marked as
experimental (--enable-experimental-api).

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall 0a20e7d43c lib/ref: Add OstreeCollectionRef type for globally unique refs
This is a type representing the tuple (collection ID, ref name), which is
guaranteed to be globally unique. It will be used in upcoming commits.

It introduces the concept of a ‘collection’ which is a unique, curated
set of refs which lie in the same trust domain (i.e. all signed by the
same key and validated by the same developer). Flathub might be a
collection, for example; or the set of OS refs coming from a particular
OS vendor.

It includes a function for validating collection IDs.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall bf1f8eb0fa lib/repo: Split out ref handling from regenerate_summary()
This will make some future additions to regenerate_summary() easier.
This commit introduces no functional changes.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall 9a79d13ce3 lib/remote: Fix ‘Since’ line for OstreeRemote
There was a typo in it when it was first introduced. Let’s not live in
the past.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #950
Approved by: cgwalters
2017-06-21 13:00:55 +00:00
Philip Withnall 20829a0582 lib/sym: Fix symbol versions for 2017.7 experimental symbols
There was a typo in the group name. It should be OK to change the
version since this is all hidden behind the --enable-experimental-api
configure option.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #950
Approved by: cgwalters
2017-06-21 13:00:55 +00:00
Philip Withnall 333d264c5b lib/core: Fix ‘Since’ line for ostree_validate_remote_name()
This was missed when cherry-picking it out of
https://github.com/ostreedev/ostree/pull/924#discussion_r123097919.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #950
Approved by: cgwalters
2017-06-21 13:00:55 +00:00
Philip Withnall 20dc9454b3 lib/core: Add ostree_validate_remote_name() for remote names
There are a few places in the code where ad-hoc validation was being
performed. Might as well formalise it a bit more.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #948
Approved by: cgwalters
2017-06-20 21:52:22 +00:00
Philip Withnall fd50c97322 build: Add ‘devel’ or ‘release’ to OSTREE_FEATURES for test-symbols.sh
test-symbols.sh was looking for the DevelBuild string, which is actually
part of the output from `ostree --version`, not $OSTREE_FEATURES.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #948
Approved by: cgwalters
2017-06-20 21:52:22 +00:00
Philip Withnall 21eec96bfd lib/pull: Fix construction of a refspec to use the correct separator
This code looks like it was supposed to build a refspec, but it used a
slash as a separator rather than a colon. The following code does
recover by supporting prefix matching with slashes, but it seems like
this was perhaps not the intention.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #912
Approved by: cgwalters
2017-06-13 01:03:49 +00:00
Philip Withnall 4418ab7fa9 lib/fetcher: Add cleanup function for OstreeFetcher
This is only used internally (the header is not public), so it doesn’t
have to go in ostree-autocleanups.h. It will be used in some following
commits.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #911
Approved by: cgwalters
2017-06-07 16:55:36 +00:00
Philip Withnall 59ffce73c8 lib/repo: Omit deltas from the summary file if there are none
If there are no deltas to be listed in the summary file, don’t bother
including the key for them in the additional metadata section of the
file. This saves a few bytes in some cases.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #911
Approved by: cgwalters
2017-06-07 16:55:35 +00:00
Philip Withnall 7ce6777028 ostree/dump: Improve formatting for well-known commit metadata keys
This follows on from commit a946c3d4, which added formatting for
well-known summary metadata keys. This commit adds it for commits.

Currently, the only well-known commit metadata key is
ostree.commit.timestamp. Formatting this correctly is especially
important, since it’s a big-endian uint64, which is completely unusable
for mere mortals when presented as a number rather than a date.

Non-formatted output can still be retrieved using the OSTREE_DUMP_RAW
flag, and the non-formatted key name is always printed for clarity.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #911
Approved by: cgwalters
2017-06-07 16:55:35 +00:00
Philip Withnall 87413ee63b lib/pull: Fix an over-indented block
Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #911
Approved by: cgwalters
2017-06-07 16:55:35 +00:00
Philip Withnall c968c12ec3 lib/pull: Drop some trailing whitespace
Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #911
Approved by: cgwalters
2017-06-07 16:55:35 +00:00
Philip Withnall 14082e6b03 lib/pull: Simplify a for-loop initialisation
It’s a bit neater to initialise the loop iterator and maximum in the
same place.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #911
Approved by: cgwalters
2017-06-07 16:55:35 +00:00
Philip Withnall 52ede69df8 lib/pull: Fix a typo in a documentation comment
Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #911
Approved by: cgwalters
2017-06-07 16:55:35 +00:00
Philip Withnall 12479d8b05 lib/repo: Reindent some code in regenerate_summary() for clarity
This makes it a bit more easily separable from the rest of the code in
the function. No functional changes.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #911
Approved by: cgwalters
2017-06-07 16:55:35 +00:00
Philip Withnall 56188808b4 build: Use AM_TESTS_ENVIRONMENT rather than TESTS_ENVIRONMENT
TESTS_ENVIRONMENT is reserved for the user to be able to set when
running the tests. AM_TESTS_ENVIRONMENT is for the tests’ Makefile to
set itself.

https://www.gnu.org/software/automake/manual/html_node/Scripts_002dbased-Testsuites.html

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #880
Approved by: cgwalters
2017-05-23 17:51:27 +00:00
Philip Withnall 1feda846ef lib/remote: Fix compilation with --enable-experimental-api
The whole ostree-remote.h file is only included in the public ostree.h
header if OSTREE_ENABLE_EXPERIMENTAL_API is defined, so there’s no need
to change the set of methods defined in it according to whether we’re
compiling with experimental API.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #875
Approved by: cgwalters
2017-05-19 15:02:00 +00:00
Philip Withnall 242a0fd779 lib/repo: Make ost_repo_remove_remote() available internally
Make it an internal, not static, API; like _ostree_repo_add_remote(). It
will be used in many the same situations.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #875
Approved by: cgwalters
2017-05-19 15:01:59 +00:00
Philip Withnall b6ac28b0da lib/repo: Add return value to _ostree_repo_add_remote()
Return whether the remote already existed. This is an internal API, so
it’s not an API break. The return value will be useful in upcoming
commits for working out whether to later remove a remote again.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #875
Approved by: cgwalters
2017-05-19 15:01:59 +00:00
Philip Withnall ed7905d000 lib/remote: Add arguments to internal OstreeRemote constructor
Add a name argument to the internal OstreeRemote constructor,
since this member (and several derived from it) is non-nullable,
and hence must always be set at construction time.

This changes the only call sites of the constructor to use the new API,
which is internal.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #875
Approved by: cgwalters
2017-05-19 15:01:59 +00:00
Philip Withnall 2910b88081 lib/remote: Add internal annotations to OstreeRemote
Just for internal documentation; g-ir-scanner doesn’t read or understand
them.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #875
Approved by: cgwalters
2017-05-19 15:01:59 +00:00
Philip Withnall d2eaded90d lib/remote: Add a getter for OstreeRemote.name
Now that we’ve got a public, sealed OstreeRemote structure, we can start
carefully exposing members of it as API.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #875
Approved by: cgwalters
2017-05-19 15:01:59 +00:00
Philip Withnall 50f73cbac3 build: Add -C arguments to some git invocations
This moves the build system a little closer towards being safe for
builddir ≠ srcdir.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #832
Approved by: cgwalters
2017-05-08 18:48:07 +00:00
Philip Withnall 6eac575f21 libostree: Make OstreeRemote a public and internal API
Previously it was static to ostree-repo.c. Make it usable throughout
libostree so it can be used by an upcoming commit, but also expose the
typedef and reference counting functions so that opaque OstreeRemote
pointers can be used by user code, in anticipation of exposing more of
its API publicly in future.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #832
Approved by: cgwalters
2017-05-08 18:48:07 +00:00
Philip Withnall eeee5a0a1e libostree: Expose $OSTREE_FEATURES in the pkg-config file
This allows consumers of libostree to check at configure time whether it
supports the feature they want.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #832
Approved by: cgwalters
2017-05-08 18:48:07 +00:00
Philip Withnall c9244b1bb2 build: Add --enable-experimental-api configure option for unstable APIs
There are currently no unstable APIs, but some will be added in
following commits. They will be built and exposed in the libostree
global symbol list iff configured with --enable-experimental-api.

Distributions should not package OSTree with --enable-experimental-api.
This is designed for previewing new APIs on controlled platforms; any of
the APIs hidden behind this option may be changed or removed at any
point.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #832
Approved by: cgwalters
2017-05-08 18:48:07 +00:00
Philip Withnall 9690a54e47 tests: Fix regex escaping in test-summary-view.sh
There were some regex special characters in the pattern strings, which I
think were causing the test to fail on some Travis builds due to using
an invalid regex.

Fix that by matching using fixed strings instead. We don’t need regexes
here. Use a new assert_file_has_content_literal to do that for us.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #838
Approved by: cgwalters
2017-05-08 17:36:33 +00:00
Philip Withnall f3cc0eb25a tests: Add a test for ostree summary --view
This includes a test of the new human-readable key names.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #826
Approved by: cgwalters
2017-05-08 00:55:25 +00:00
Philip Withnall c1290177a3 ostree: Use #defines for well-known metadata key names
Rather than hard-coding the names as strings. This makes the code a
little more maintainable.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #826
Approved by: cgwalters
2017-05-08 00:55:24 +00:00
Philip Withnall a946c3d423 ostree: Improve formatting for well-known summary metadata keys
If a summary metadata key is well-known, like ostree.static-deltas, or
ostree.summary.last-modified, format it a little more nicely.

This is especially important for timestamps like last-modified, since
otherwise they’re formatted as a big-endian uint64, which is basically
unusable for the user.

Non-formatted output can still be retrieved using the OSTREE_DUMP_RAW
flag, and the non-formatted key name is always printed for clarity.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #826
Approved by: cgwalters
2017-05-08 00:55:24 +00:00
Philip Withnall 8ea654251a ostree: Add --view mode to ostree summary
This allows a locally generated summary file to be viewed. It accepts
the same arguments as `ostree remote summary` (i.e. --raw).

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #826
Approved by: cgwalters
2017-05-08 00:55:24 +00:00
Philip Withnall 015ce7520b libostree: Document endianness of GVariant metadata types
Endianness strikes again.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #826
Approved by: cgwalters
2017-05-08 00:55:24 +00:00
Philip Withnall 9aa8d420cf libostree: Add some additional metadata to the summary file
• Commit timestamps, so it’s easy to work out whether a given commit is
   newer than the one we have locally
 • Summary file timestamp, so it’s easy to work out whether the summary
   file is more up to date than another summary file
 • Summary file expiry time, so clients can work out when they should
   expect the summary file to next be updated, and hence can query for
   it at roughly the right time

The expiry time requires input from the user, so is currently never set
automatically. Programs using libostree can set it if they wish.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #826
Approved by: cgwalters
2017-05-08 00:55:24 +00:00
Philip Withnall bf9772f231 libostree: Ensure progress keys are all always set
If one of the progress keys is set in a pull operation, a ::changed
signal is emitted on the progress object, and the callback for that
could query any of the progress keys — so they all need to be set,
otherwise we get an assertion failure in ostree_async_progress_get() due
to a named key not existing.

Spotted by Dan Nicholson in PR #819.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #835
Approved by: cgwalters
2017-05-05 14:58:03 +00:00
Philip Withnall f1da7ec300 libostree: Fix potential use of uninitialised memory in progress API
Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #835
Approved by: cgwalters
2017-05-05 14:58:03 +00:00
Philip Withnall 4c731165bb libostree: Add missing checks for invalid timestamps
g_date_time_new_from_unix_utc() will not always return a valid GDateTime
— if the input timestamp is too big, GDateTime cannot represent it, and
the constructor returns NULL.

Add some missing checks for these situations. We don’t ever expect
timestamps to be this big, but they could be as a result of corruption
or a malicious repository.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #825
Approved by: cgwalters
2017-05-03 15:23:15 +00:00
Philip Withnall a1c866ed52 libostree: Fix a typo in docs for ostree_repo_pull_with_options()
Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #828
Approved by: cgwalters
2017-05-03 14:16:23 +00:00
Philip Withnall cbe3989b2b libostree: Get and set OstreeAsyncProgress:status atomically
Use the new well-known `status` key for OstreeAsyncProgress to get and
set the status atomically with other keys in an OstreeAsyncProgress
instance.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #819
Approved by: cgwalters
2017-04-29 11:50:15 +00:00
Philip Withnall ce83abb868 libostree: Allow OstreeAsyncProgress:status to be set atomically
Rework how the status is handled in OstreeAsyncProgress so that it’s now
a well-known key in the hash table. This means that it can be retrieved
and set atomically with other keys using
ostree_async_progress_[get|set]().

The behaviour of ostree_async_progress_[get|set]_status() is preserved,
with the caveat that `status` can now also be accessed using the other
API on OstreeAsyncProgress, and has to be accessed with the right
GVariant type.

Internally, a NULL status is represented by an empty status string
(since ostree_async_progress_[get|set]_variant() deliberately don’t
allow NULL variants to be set against keys, since that would break the
ostree_async_progress_get() API).

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #819
Approved by: cgwalters
2017-04-29 11:50:15 +00:00
Philip Withnall cdf876101b src: Port to new OstreeAsyncProgress atomic API
This will eliminate most of the potential races in progress reporting.
ostree_repo_pull_default_console_progress_changed() still calls three
getters, so there may still be races there, however.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #819
Approved by: cgwalters
2017-04-29 11:50:15 +00:00
Philip Withnall c27b66de80 libostree: Add multiple getter/setter support to OstreeAsyncProgress
OstreeAsyncProgress is thread-safe: it can have keys changed by one
thread while another is getting the same keys (modulo some locking
contention). However, the thread safety is done at the function call
level: if some code calls an OstreeAsyncProgress getter several times,
the key fetches are not atomic with respect to each other.

In the case of contention on the lock, this can result in consumers of
OstreeAsyncProgress data seeing an inconsistent state between the
properties they query, which could result in progress reporting
inaccuracies.

In the uncontested case, this results in the OstreeAsyncProgress lock
being locked and unlocked many times more than necessary.

Try to improve this by adding new API, which supports getting and
setting multiple keys atomically:
 • ostree_async_progress_get()
 • ostree_async_progress_set()

The new API uses GVariants and varargs: keys are passed as a
GVariantType string followed by arguments as for g_variant_new() or
g_variant_get(), followed by the next key, etc.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #819
Approved by: cgwalters
2017-04-29 11:50:15 +00:00
Philip Withnall f74e52a3a0 libostree: Rework OstreeAsyncProgress to use GVariants internally
OstreeAsyncProgress currently does some contortions to try and avoid
allocating space for guints and guint64s (on 64-bit platforms), but this
means it uses two GHashTables. A GHashTable allocates 8 buckets even
when empty. Given that the largest usage of OstreeAsyncProgress in
libostree puts 13 uints and 5 uint64s in it, this optimisation does not
save significant (if any) memory.

Instead, change OstreeAsyncProgress to store values internally as
GVariants, and expose this with some new API:
 • ostree_async_progress_get_variant()
 • ostree_async_progress_set_variant()
Each GVariant is allocated on the heap. As they are immutable, they are
thread-safe once returned by a getter.

The existing API continues to work as before, except in the case where a
key is set/got as both a uint and a uint64 — there will now be a
collision (and a GVariant type checking failure) whereas previously
there was no collision. Nothing in OSTree uses OstreeAsyncProgress this
way though.

The new API can be used to share more complex data via the progress API.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #819
Approved by: cgwalters
2017-04-29 11:50:15 +00:00
Philip Withnall d37acd3007 tests: Ignore some standard automake check output files
Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #810
Approved by: cgwalters
2017-04-27 13:43:49 +00:00
Philip Withnall c418600594 ostree: Use G_OPTION_ARG_FILENAME where appropriate
Instead of using G_OPTION_ARG_STRING, use G_OPTION_ARG_FILENAME, which
handles filename encoding conversion differently from the locale
conversion which G_OPTION_ARG_STRING. This will fix argument handling on
systems where the filename encoding is not the same as the locale
encoding (which is fairly unlikely since most systems use UTF-8).

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #810
Approved by: cgwalters
2017-04-27 13:43:49 +00:00
Philip Withnall 574c3ea6f9 libostree: Allow compression level to be set for archive-z2 stream
Add a ostree_raw_file_to_archive_z2_stream_with_options() variant of
ostree_raw_file_to_archive_z2_stream(), to allow a compression-level
option to be passed in and passed through to zlib.

This is useful when building archive-z2 files on the fly for
transmission over a non-bandwidth-limited channel, such as a local
network. In this case, CPU time is more valuable than bandwidth, so we
want a low compression level.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #721
Approved by: cgwalters
2017-03-06 18:19:45 +00:00
Philip Withnall f667a82fc1 build: Fix disabling --enable-man if xsltproc is not available
If --enable-man=maybe (the default), and xsltproc is not available, the
configuration code would check for it, correctly set enable_man=no, then
incorrectly overwrite that with enable_man=yes, which would result in
later trying to execute $(XSLTPROC) when it’s empty.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #720
Approved by: cgwalters
2017-03-06 13:48:17 +00:00
Philip Withnall e6a8979e05 ostree-repo: Clarify error behaviour of remote option getters
Clarify the documentation for functions like
ostree_repo_get_remote_boolean_option(), stating what out_value will be
set to on error.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #676
Approved by: cgwalters
2017-02-10 14:26:00 +00:00