Commit Graph

271 Commits

Author SHA1 Message Date
Philip Withnall 3cf53f7c58 lib/repo: Add gtk-doc comment to OstreeRepoCommitState
Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1335
Approved by: cgwalters
2017-11-08 15:47:11 +00:00
Philip Withnall 9c4870b5e1 lib/repo: Add OSTREE_REPO_COMMIT_STATE_NORMAL to represent most commits
This allows more explicit handling of commit state in code using
libostree, rather than hard-coding a commit state of 0 for ‘normal’.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1335
Approved by: cgwalters
2017-11-08 15:47:11 +00:00
Colin Walters 1222c2271b repo: Add wrapper function for setting devino cache on checkout opts
I was trying to use this with pygobject for an OCI+ostree project, and pygobject
rejected simply assigning to the field (understandably, since it can't bind the
lifetime together).

Add a wrapper function, which is still unsafe, but hides that unsafety
where most people shouldn't find it.  And if they do...well, sorry,
Rust wasn't invented when ostree was started.

Closes: #1295
Approved by: pwithnall
2017-10-20 18:20:19 +00:00
Colin Walters b8251d26bd lib/checkout: For "process whiteouts" mode, replace directories too
I'm playing around with some ostree ⇔ OCI/Docker bits, and ran
into this while importing an OCI image that built from the Fedora
base image where `/home` is a regular directory, and I added a layer
that did the ostree bits of moving it to `/var` and leaving a symlink.

OCI/Docker supports this.  Now since "process whiteouts" is really the
"enable OCI/Docker" mode, let's only replace dirs if that's enabled.
This leaves the `UNION_FILES` targeted for its original use case
which is unioning components/packages.  (Although that use case itself
is now a bit superceded by `UNION_IDENTICAL`, but eh).

Closes: #1294
Approved by: jlebon
2017-10-20 13:20:27 +00:00
Colin Walters bba7eb8069 commit: Add _CONSUME modifier flag
For many cases of commit, we can actually optimize things by simply "adopting"
the object rather than writing a new copy. For example, in rpm-ostree package
layering.

We can only make that optimization though if we take ownership of the file. This
commit hence adds an API where a caller tells us to do so. For now, that just
means we `unlink()` the files/dirs as we go, but we can now later add the
"adopt" optimization.

Closes: #1255
Approved by: jlebon
2017-10-10 13:02:08 +00:00
Philip Withnall 22c1fdfbd3 lib/repo: Change resolve_keyring_for_collection() to return a remote
Instead of returning just the keyring filename, return the entire
OstreeRemote, which has the keyring filename as one of its members. This
will simplify some upcoming changes, and allows slightly improved debug
logging.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1202
Approved by: cgwalters
2017-09-27 16:38:07 +00:00
Philip Withnall 149aec1099 lib/repo-refs: Add first version of ostree_repo_resolve_collection_ref()
This is a parallel for ostree_repo_resolve_rev_ext() which works on
collection–refs. At the moment, the implementation is simple and uses
ostree_repo_list_collection_refs(). In future, it could be rewritten to
check the checksum directly rather than enumerating all
potentially-relevant checksums.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1182
Approved by: cgwalters
2017-09-27 14:44:00 +00:00
Colin Walters 25a7c4bd4e lib/pull: Default checksum for archive mirror, add TRUSTED_HTTP flag
I now think commit fab1e113db was a mistake;
because it breaks the mental model that at least I'd built up that "local repos
don't have checksums verified, HTTP does".

For example, a problem with this is (with that mental model in place) it's easy
for people who set up mirrors like this to then do local pulls, and at that
point we've done a deployment with no checksum verification.

Further, since then we did PR #671 AKA commit 3d38f03 which is really most of
the speed hit.

So let's switch the default even for this case to doing checksum verification,
and add `ostree pull --http-trusted`. People who are in situations where they
know they want this can find it and turn it on.

Closes: https://github.com/ostreedev/ostree/issues/1211

Closes: #1212
Approved by: jlebon
2017-09-26 18:07:43 +00:00
Colin Walters 6e4146a354 tree-wide: Remove Emacs modelines
We added a `.dir-locals.el` in commit: 9a77017d87
There's no need to have it per-file, with that people might think
to add other editors, which is the wrong direction.

Closes: #1206
Approved by: jlebon
2017-09-21 21:38:34 +00:00
Philip Withnall 64b23fd089 lib/repo: Add ostree_repo_hash() and tests
Add a hash function for OstreeRepo instances, which relies on the repo
being open, and hence being able to hash the device and inode of its
root directory.

Add unit tests for this and ostree_repo_equal().

Signed-off-by: Philip Withnall <withnall@endlessm.com>

https://github.com/ostreedev/ostree/issues/1191

Closes: #1205
Approved by: cgwalters
2017-09-21 21:25:58 +00:00
Philip Withnall c62b36046b lib/repo: Drop outdated FIXME comment from ostree-repo.h
This was some incomplete planning from while the find_remotes() API was
being designed; now totally outdated.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1179
Approved by: cgwalters
2017-09-19 14:51:09 +00:00
Philip Withnall 981eb6c226 lib/repo: Add ostree_repo_equal() for comparing repos
This will compare their root directory inodes to see if they are the
same repository on disk. A convenience method for the users of the
public API who can’t access OstreeRepo.inode.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1179
Approved by: cgwalters
2017-09-19 14:51:09 +00:00
Colin Walters 051cdf396c lib/checkout: Rename disjoint union, change to merge identical files
It turns out that librpm automatically merges identical files between
distinct packages, and this occurs in practice with Fedora today between
`chkconfig` and `initscripts` for exmaple.

Since we added this for rpm-ostree, we basically want to do what librpm does,
let's change the semantics to do a merge.  While we're here rename
to `UNION_IDENTICAL`.

Closes: #1156
Approved by: jlebon
2017-09-13 19:19:33 +00:00
Ruixin f07432d4ce checkout: add an extra checkout_overwrite mode
This is for issue projectatomic/rpm-ostree#365,
an extra option of overwrite mode is added to the checkout command
so that when there is "non-directory" file already exist
during checkout, the error will be handled.

Some tests are added for regression

Closes: #1116
Approved by: cgwalters
2017-09-01 15:42:50 +00:00
Colin Walters 138c4d7aae libarchive: Add support for translating paths during commit
For rpm-ostree, I want to move RPM files in `/boot` to `/usr/lib/ostree-boot`.
This is currently impossible without forking the libarchive code.  Supporting
this is pretty straightforward; we already had pathname translation in
the libarchive code, we just need to expose it as an option.

On the command line side, I chose to wrap this as a regexp. That should be good
enough for a lot of use cases; sophisticated users should as always be making
use of the API. Note that this required some new `#ifdef LIBARCHIVE` bits to use
the new API. Following previous patterns here, we use the new API only if a
relevant option is enabled, ensuring unit test coverage of both paths.

For the test cases, I ended up changing the accounting to avoid having to
multiply the test count.

Closes: #1105
Approved by: jlebon
2017-08-30 14:30:30 +00:00
Matthew Leeds 7ed881baa7 lib/repo-refs: Include remote refs when using collections
When working with collections it can be useful to see remote refs rather
than just local and mirrored ones. This commit changes the "ostree refs
-c" output to include remote refs, and includes remote refs with
collection IDs in summary file generation as well. The former behavior
is consistent with how "ostree refs" works, and the latter behavior is
useful in facilitating P2P updates even when mirrors haven't been
configured.

To accomplish this, OstreeRepoListRefsExtFlags was extended with an
EXCLUDE_REMOTES flag. This was done rather than an INCLUDE_REMOTES flag
so that existing calls to ostree_repo_list_refs_ext continue to have the
same behavior. This flag was added to ostree_repo_list_collection_refs
(which is an experimental API break).

Also, add unit tests for the "refs -c" and summary file behavior, and
update relevant tests.

Closes: #1069
Approved by: cgwalters
2017-08-24 19:57:33 +00:00
Colin Walters 95bac299e5 lib/repo: Add some assertions for ABI sizes
Things like https://sourceware.org/libabigail/manual/abidiff.html
look interesting but in a brief look I couldn't work out
how to conveniently use them for quick ABI sanity checking without
doing a diff from a previous build (which we could do but would be
more involved).

This way will at least catch struct ABI breaks on x86_64 which
I think we'd be most likely to do accidentally when trying
to use one of the previous unused values.

I found the hole values via gdb's `pahole` command.

Closes: #1108
Approved by: jlebon
2017-08-24 14:58:51 +00:00
Jonathan Lebon 11fdca2919 pull: better description for --mirror
Describe the behaviour of --mirror a bit better.

Closes: #1100

Closes: #1099
Approved by: dustymabe
2017-08-23 01:23:38 +00:00
Colin Walters fd98bda3c7 repo: Introduce ostree_repo_open_at() and ostree_repo_create_at()
This essentially completes our fd-relative conversion.

While here, I cleaned up the semantics of `ostree_repo_create()` and
`ostree_repo_create_at()` to be more atomic - basically various scripts were
testing for the `objects` subdirectory, so let's formalize that.

Closes: #820
Approved by: jlebon
2017-08-15 12:35:10 +00:00
Philip Withnall f35b409077 lib/repo-refs: Add ostree_repo_remote_list_collection_refs() API
This parallels ostree_repo_remote_list_refs(), but returns a map of
OstreeCollectionRef → checksum, and includes refs from collection IDs
other than the remote repository’s main collection ID.

Use this in OstreeRepoFinderConfig to ensure that refs are matched
against even if they’re stored in the repository summary file’s
collection map, rather than its main ref map. This fixes false negatives
when searching for refs in some situations.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1058
Approved by: cgwalters
2017-08-08 13:59:58 +00:00
Colin Walters d5273b34d0 lib/repo: Add API to create and list ref aliases
There are multiple use cases where we'd like to alias refs.

First, having a "stable" alias which gets swapped across major
versions: https://pagure.io/atomic-wg/issue/228

Another case is when a ref is obsoleted;
<https://pagure.io/atomic-wg/issue/303>
This second one could be done with endoflife rebase, but I think
this case is better on the server side, as we might later change
our minds and do actual releases there.

I initially just added some test cases for symlinks in the `refs/heads` dir to
ensure this actually works (and it did), but I think it's worth having APIs.

Closes: #1033
Approved by: jlebon
2017-08-02 17:33:10 +00:00
Philip Withnall acb14648d7 lib/repo: Add OSTREE_REPO_METADATA_REF as a well-known metadata store
As discussed in https://github.com/ostreedev/ostree/pull/946, the
summary file is becoming an unsigned cache of ref information; any
additional metadata for the repository needs to move elsewhere in order
to remain signed. Introduce OSTREE_REPO_METADATA_REF as the well-known
name of a ref where such metadata can live, as the metadata on
contentless commits.

Don’t yet update the documentation for summary-related methods to
mention this, since it’s still hidden behind the
--enable-experimental-api configure option.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #946
Approved by: cgwalters
2017-07-11 19:50:32 +00:00
Philip Withnall 292230301d lib/repo-finder: Add basic support for finding remote URIs by ref name
Add an initial OstreeRepoFinder interface (but no implementations),
which will find remote URIs by ref names and collection IDs, the
combination of which is globally unique.

The new API is used in a new ostree_repo_find_updates() function, which
resolves a list of ref names to update into a set of remote URIs to pull
them from, which can be treated as mirrors. It is an attempt to
generalise resolution of the URIs to pull from, and to generalise
determination of the order and parallelisation which they should be
downloaded from in.

Includes fixes by Krzesimir Nowak <krzesimir@kinvolk.io>.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall fbf8df8829 lib/refs: Add methods for setting/listing collection–refs
These are tuples of (collection ID, ref name) which are a globally-unique
form of local ref. They use OstreeCollectionRef as an identifier, and hence
need to be accessed using new API, as the existing API uses string
identifiers and sometimes accepts refspecs. Remote names are not
supported as part an OstreeCollectionRef.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall 4de736fdfa lib/repo: Add collection ID support to OstreeRepo
Add {get,set}_collection_id() methods to OstreeRepo and some documentation
about the concept of a collection ID which globally identifies an
upstream repository. See the documentation for more details.

This will be used in future commits. For now, the new API is marked as
experimental (--enable-experimental-api).

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall 0a20e7d43c lib/ref: Add OstreeCollectionRef type for globally unique refs
This is a type representing the tuple (collection ID, ref name), which is
guaranteed to be globally unique. It will be used in upcoming commits.

It introduces the concept of a ‘collection’ which is a unique, curated
set of refs which lie in the same trust domain (i.e. all signed by the
same key and validated by the same developer). Flathub might be a
collection, for example; or the set of OS refs coming from a particular
OS vendor.

It includes a function for validating collection IDs.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Colin Walters 0635fcbfd9 lib/checkout: Add bareuseronly_dirs option
This is a continuation of https://github.com/ostreedev/ostree/pull/926
for directories instead of files.

See: https://github.com/flatpak/flatpak/issues/845

This option suppresses mode bits outside of `0775` for directory
checkouts.  I think most people should start doing this by default,
and use explicit overrides for e.g. `/tmp` if doing a recommit based
on a checkout.

Closes: #927
Approved by: alexlarsson
2017-06-13 20:05:31 +00:00
Colin Walters 6ed824bf00 lib/pull: Add OSTREE_REPO_PULL_FLAGS_BAREUSERONLY_FILES
This is an option which is intended mostly for flatpak;
see: https://github.com/flatpak/flatpak/issues/845

We're adding an option for pulling into *all*
repo modes that has an effect similar to the `bare-user-only`
change from https://github.com/ostreedev/ostree/pull/909

This way one can pull content into e.g. a root-owned `bare` repository and
ensure that there aren't any setuid or world-writable files.

Closes: #926
Approved by: alexlarsson
2017-06-13 18:44:28 +00:00
Colin Walters e8efd1c8dc checkout: Add SELinux labeling for checkout, use in deploy
This is a variant of the efforts in https://github.com/ostreedev/ostree/pull/741
Working on `rpm-ostree livefs`, I realized though I needed to just
check out *new* files directly into the live `/etc` (and possibly
delete obsolete files).

The way the current `/etc` merge works is fundamentally different from
that.  So my plan currently is to probably do something like:

 - Compute diff
 - Check out each *new* file individually (as a copy)
 - Optionally delete obsolete files

Also, a few other things become more important - in the current deploy code, we
copy all of the files, then relabel them. But we shouldn't expose to *live*
systems the race conditions of doing that, plus we should only relabel files we
checked out.

By converting the deploy's /etc code to use this, we fix the same TODO item
there around atomically having the label set up as we create files. And further,
if we kill the `/var` relabeling which I think is unnecessary since Anaconda
does it, we could delete large chunks of code there.

In the implementation, there are two types of things: regular files, and
symlinks. For regular files, in the `O_TMPFILE` case, we have the ability to
do *everything* atomically (including SELinux labeling) before linking it into
place. So let's just use that. For symlinks, we use `setfscreatecon()`.

Closes: #797
Approved by: jlebon
2017-04-25 16:52:33 +00:00
Colin Walters 6060abbb4b repo: Add a "force copy" flag to checkout
This is intended to be used for copying `/usr/etc` → `/etc` for
deployments.

A TODO here is to use `glnx_file_copy_at()` if the repo mode allows
it - then we'd use reflinks if available.

Closes: #804
Approved by: jlebon
2017-04-24 15:26:11 +00:00
Colin Walters 6a7ee4860f Fix a few gtk-doc warnings
Just continuing to chip away at this.

Closes: #788
Approved by: jlebon
2017-04-12 15:36:46 +00:00
Colin Walters 9016e9e8be Add flag to make SELinux label failure fatal, add hack for /proc
I was working on `rpm-ostree livefs` which does some ostree-based
filesystem diffs, and noticed that we were ending up with `/proc`
not being labeled in our base trees.

Reading the selinux-policy source, indeed we have:

```
/proc			-d	<<none>>
/proc/.*			<<none>>
```

This dates pretty far back.  We really don't want unlabeled
content in ostree.  In this case it's mostly OK since the kernel
will assign a label, but again *everything* should be labeled via
OSTree so that it's all consistent, which will fix `ostree diff`.

Notably, `/proc` is the *only* file path that isn't covered when composing a
Fedora Atomic Host. So I added a hack here to hardcode it (although I'm a bit
uncertain about whether it should really be `proc_t` on disk before systemd
mounts or not).

Out of conservatism, I made this a flag, so if we hit issues down the line, we
could easily change rpm-ostree to stumble on as it did before.

Closes: #768
Approved by: jlebon
2017-04-04 15:31:49 +00:00
Alexander Larsson b2d10dcaaa commit: Add --canonical-permissions argument
This adds to file permission masks the same bitmask that will
be applied to file objects in bare-user* repos. This will be
needed in the testsuite to ensure that the things we commit
will be expressable in bare-user-only repos.

Closes: #750
Approved by: cgwalters
2017-03-27 13:48:41 +00:00
Colin Walters 94948e3522 checkout: Support a "pure addition" mode
I plan to use this for `rpm-ostree livefs`.
https://github.com/projectatomic/rpm-ostree/issues/639

Closes: #714
Approved by: jlebon
2017-03-06 20:58:04 +00:00
Colin Walters 9c0af41710 lib: Add ostree_repo_reload_config()
For a long time we've cached the remote configs in the repo, which
mostly makes sense for the `repo/config` file, but less sense
for `/etc/ostree/remotes.d`, because we want to support admins
interactively editing them.

One can delete the repo instance and create a new one, but that's a bit ugly.
Let's introduce an API for this so rpm-ostree can reload remotes after
admins/scripts edit them in `/etc`.  We also might as well reload
any other entries in the config.

Structurually now, `ostree_repo_open()` deals with file descriptors, and then
calls `ostree_repo_reload_config()`. Except for the uncompressed cache, which is
the only thing that deals with FDs that can be configured. But we want to delete
that anyways.

No tests, since...we don't have a daemon in this codebase, don't want to shave
that yak just today.

Closes: #662
Approved by: jlebon
2017-02-07 16:12:58 +00:00
Colin Walters 5c940987e7 Add support for more selective pruning
There are use cases for having a single repo with branches
with different lifecycles; a simple example of what I was
trying to do in CentOS Atomic Host work is have "stable"
and "devel" branches, were we want to prune devel, but
retain *all* of stable.

This patch is split into two parts - first we add a low level "delete all
objects not in this set" API, and change the current prune API
to use this.

Next, we move more logic into the "ostree prune" command. This paves the way for
demonstrating how more sophisticated algorithms/logic could be developed outside
of the ostree core.

Also, the --keep-younger-than logic already lived in the commandline, so it
makes sense to keep extending it there.

Closes: https://github.com/ostreedev/ostree/issues/604

Closes: #646
Approved by: jlebon
2017-01-19 16:28:00 +00:00
Colin Walters 24bf257ee9 lib: Add an API to GPG verify a commit given a remote
Conceptually we've been moving towards having our GPG verification
paths be per-remote.  The code internally supports this, but we
didn't expose an API to use it conveniently.

This came up when trying to add a new `gpgkeypath` option, since
right now rpm-ostree manually finds keyrings for the remote, and
hence it wasn't looking at the keypath, and said "Unknown key"
in status.

Adding an API fixes this nicely.

Closes: #576
Approved by: giuseppe
2016-11-17 11:33:41 +00:00
Giuseppe Scrivano 47b0b4120a pull: use same name for parameter and documentation comment
Fixes this warning:

src/libostree/ostree-repo-pull.c:2162: Warning: OSTree: ostree_repo_pull_with_options: unknown parameter 'remote_name_or_baseurl' in documentation comment, should be 'remote_name'

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

Closes: #472
Approved by: jlebon
2016-08-27 12:40:46 +00:00
Colin Walters 3ef4cc2e5b lib: Add an API to list only "our" objects, fix prune to use it
When doing a prune, we should not try to delete objects in parent
repos, since it'll fail.  There is a bigger discussion about the
semantics of `parent=` to be had, but this will fix trying to use
`ostree prune --repo=/ostree/repo/extensions/rpmostree/pkgcache`.

Closes: https://github.com/ostreedev/ostree/issues/467

Closes: #471
Approved by: jlebon
2016-08-25 20:01:37 +00:00
Colin Walters 593effe2fb lib: Add padding booleans to OstreeRepoCheckoutAtOptions
Since this is a new API, and adding booleans is the most likely thing
we'll do, let's stick some explicit padding for them in here now.

We could use the `unused_ints[]` but it'd be out of order, and this
will more clearly remind people about the padding.  The efficiency hit
versus bitfields is annoying, but oh well, not a real world problem.

Closes: #427
Approved by: giuseppe
2016-08-04 12:18:28 +00:00
Colin Walters 18d826e3a5 repo: Flip the fsync default to off for new checkout API
Since we're adding a new API, we have the opportunity to fix
the defaults.  We expect clients to do a `syncfs()` or equivalent
on their own now, since it's way more efficient.

Flip the checkout fsync default to off.

Closes: #425
Approved by: giuseppe
2016-08-04 07:33:31 +00:00
Giuseppe Scrivano 0bac88e132 libostree: skip introspection for two functions
These two functions are not safe for gobject introspection, so annotate
them to be skipped:

1) ostree_repo_import_archive_to_mtree
2) ostree_repo_export_tree_to_archive

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

Closes: #417
Approved by: cgwalters
2016-07-30 11:24:52 +00:00
Giuseppe Scrivano 8867693240 libostree: mark ostree_repo_checkout_tree_at as deprecated
and move its definition to a separate file.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

Closes: #417
Approved by: cgwalters
2016-07-30 11:24:52 +00:00
Giuseppe Scrivano 30963766c7 libostree: new function ostree_repo_checkout_at
Provide a gobject introspection safe version for
`ostree_repo_checkout_tree_at'.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

Closes: #417
Approved by: cgwalters
2016-07-30 11:24:52 +00:00
Colin Walters 439069b2bb checkout: Add an option to require hardlinks
I've seen a few people hit this and wonder why checkouts are slow/take
space.  Really, ensuring this happens is the *point* of OSTree.
Physical copies should be a last resort fallback for very unusual
situations (one of those is rpm-ostree checking out the db since
librpm doesn't know how to read from libostree).

Even I hit the fact that `/var` is a mountpoint disallowing hardlinks
with `/ostree` once and was confused.  =)

Add this to the rofiles-fuse test case because it creates a mount
point.

Closes: #368
Approved by: jlebon
2016-06-27 13:08:46 +00:00
Yu Qi Zhang 02a2b689dd refs: resolve conflict between local/remote repos
Add the functionality to use the same name for refs in local and remote
repos. This helps users keep track of local refs of remote origin, much
like local and remote git branches.

Previously, when a local ref is specified, resolve_refspec would fall
back to searching through remote repos if the ref is not found locally.
This function now takes an extra flag to specify whether it should
search through remote repos. Additionally, ostree_repo_resove_rev_ext
was added to call resolve_refspec with fallback_remote being false, so
refs --create would no longer complain when trying to create a local
ref of the same name as a remote one.

Fix remote repo parsing not being handled correctly on refs --create.

Closes: #363
Approved by: jlebon
2016-06-23 19:52:26 +00:00
Mathnerd314 4cb77c51db core: Use OSTREE_SHA256_STRING_LEN instead of 64
Closes: #359
Approved by: cgwalters
2016-06-22 16:10:01 +00:00
Mathnerd314 1b88dc7f90 docs: Get API docs working again
This changes around a few things that didn't work for me:
* Section names seem to be ostree-* instead of libostree-*
* Also XML files are ostree-* (they didn't show up at all)
- gtk-doc doesn't seem to parse const _OSTREE_PUBLIC correctly
* pull documentation is now on the actual functions rather than stubs
* Update gitignore with some more files

And there some changes to make gtk-doc give fewer warnings (not finished)

Closes: #327
Approved by: cgwalters
2016-06-09 18:15:49 +00:00
Krzesimir Nowak 4929ab4033 repo: Add functions for verifying any data in repository
This can be useful for validating the 3rd party data that is put in
the extensions directory and is signed with the same keys as commits
or the summary file.

Closes: #310
Approved by: cgwalters
2016-05-27 11:20:00 +00:00
Krzesimir Nowak aa946cc136 repo: Allow using options for fetching summary
This adds a _with_options variant of the
ostree_repo_remote_fetch_summary function, so we can tell the fetcher
to use a specific URL instead taking it from the remote config.

Closes: #290
Approved by: cgwalters
2016-05-10 13:47:36 +00:00
Jonathan Lebon b1d3dd151c ostree-repo-libarchive.c: major refactor
- Make hardlink handling more generic. The previous strategy worked for
  tar archives, but not for cpio. It now works for both.
- Add support for SEL labeling (through the OstreeRepoCommitModifier)
- Add support for xattr_callback (through the OstreeRepoCommitModifier)
- Add support for filter (through the OstreeRepoCommitModifier)
- Add a use_ostree_convention option

Closes: #275
Approved by: cgwalters
2016-05-06 14:44:55 +00:00
Alexander Larsson 5595664e47 ostree export: Add --prefix option
This lets you set a prefix for the resulting archive patsh.
Especially useful in combination with --subpath, for instance
--subpath=subdir --prefix=subdir to extract just subdir.

Closes: #265
Approved by: cgwalters
2016-04-19 12:28:06 +00:00
Alexander Larsson 9e7e594907 Add OstreeRepo option for an out-of-band cache dir
This allows you to have a writable cache dir even for a system-owned
repository.

Closes: #250
Approved by: cgwalters
2016-04-14 15:55:08 +00:00
Alexander Larsson 37382590dc Export ostree_repo_get_remote_option* functions
These are useful for ostree users (like xdg-app) that have custom
options for remotes. In particular they are useful when we later make them
all respect self->parent_repo.

Closes: #236
Approved by: cgwalters
2016-04-07 20:02:46 +00:00
Alexander Larsson ed1e0c6d04 pull: Add OSTREE_REPO_PULL_FLAGS_UNTRUSTED flag
If this is set we verify all objects we pull, even for local remotes,
and we avoid hard-linking into local source repos.

https://bugzilla.gnome.org/show_bug.cgi?id=764125

Closes: #221
Approved by: cgwalters
2016-03-25 12:56:55 +00:00
Giuseppe Scrivano 898c7b6577 ostree-repo: new public function `ostree_repo_list_refs_ext`
It accepts a `flags` argument to control its behavior.  Differently
from `ostree_repo_list_refs`, the `refspec_prefix` is not removed from
the results.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2016-03-02 14:52:02 -05:00
Colin Walters 3efdcb54b7 repo: Add ostree_repo_get_dfd()
I plan to use this in rpm-ostree at least for two reasons:
 - To find the mtime on the repo
 - To use the tmp/ directory to stage content (but we should eventually
   add a better API)
2016-03-01 21:52:22 -05:00
Colin Walters 16979cc5ed lib: Introduce versioned symbols
As rpm-ostree evolves, it keeps driving API additions to libostree.
This creates a relatively tight coupling.

However, if delivering via e.g. RPM, unless one manually remembers to
increment the `Requires:` in the spec file, it's possible for the two
to become desynchronized.

RPM handles versioned symbols and will ensure a dependency if the
application starts using a newer version.

To implement this, switch to `-fvisibility=hidden`, along with an
annotation in the header, and finally add a `.sym` file.

This matches what other projects like systemd and libvirt do.

Although rather than attempting to retroactively version symbols, glom
them all onto the current one.
2016-03-01 21:45:26 -05:00
Colin Walters 3a555114bc repo: Add ostree_repo_import_archive_to_mtree
This is a more flexible version of the previous
ostree_repo_write_archive_to_mtree() which took a file reference.
This has an extensible options structure, and in particular
now supports `ignore_unsupported_content`.

I plan to use this for importing Docker images which contain device
nodes.  (There's no reason for container images to have those, so
we'll just ignore them).

Also here, just like the export variant, the caller is responsible for
setting up libarchive.
2016-02-19 08:54:04 -05:00
Colin Walters b08b1abccd Rename libarchive write API to "export", matching command line
I was going to add new API for importing, and it was really confusing
that what I think of now as import and export both had "write" in the
name.  It's just clearer to talk about the direction.

At the same time, include `Export` in the options structure.

This isn't an ABI break as the API isn't in a release.
2016-02-19 08:54:04 -05:00
Colin Walters baaf7450da Support Docker-style whiteouts
This is to enable importing Docker layers as ostree commits, then
checking them out in a union.

The prototype work for this is in:
https://github.com/cgwalters/dlayer-ostree

Though it will likely ultimately end up in:
https://github.com/projectatomic/atomic
2016-02-16 10:54:59 -05:00
Colin Walters 355f8438ef Add an `export` builtin, and API to write to libarchive
At the moment I'm looking at using rpm-ostree to manage RPM inputs
which can then be converted into Docker images.  It's most convenient
if we can stream directly out of libostree rather than doing a
checkout + tar combination.

There are also backup/debugging etc. reasons to implement `export` as
well.
2016-02-14 09:53:01 -05:00
Colin Walters cd0a9d3435 Add a checkout option to skip fsync
This is a better followup to dc9239dd7b
since I wanted to do fsync-less checkouts in rpm-ostree too, and
replicating the "turn off fsync temporarily" was in retrospect just a
hack.

We can simply add a boolean to the checkout options.

https://github.com/GNOME/ostree/pull/172
2016-01-26 15:39:46 -05:00
Colin Walters dc9239dd7b sysroot: Don't individually fsync dirs in checkout, rely on syncfs
Originally, a lot of the `fsync()` calls here were added for the
wrong reason - I was chasing a bug that ended up being the extlinux
bootloader not parsing 64 bit ext4 filesystems.  But since it looked
like corruption, I tried adding a lot more `fsync()` calls.

All we should have to do is use `syncfs()`.  If that doesn't work,
it's a kernel bug.

I'm making this change because skipping the individual fsyncs can be a
major performance win - it's easier for the FS to optimize, we do more
in parallel, etc.

https://bugzilla.gnome.org/show_bug.cgi?id=757117
2016-01-13 13:15:08 -05:00
Colin Walters 5929ce9e0e repo: Add APIs for devino optimization between checkout -> commit
A fast way to generate new OSTree content using an existing
tree is to checkout (as hard links), add/replace files, then
call `ostree_repo_scan_hardlinks()`, then commit.

But `ostree_repo_scan_hardlinks()` scans the entire repo, which
can be slow if you have a lot of content.

All we really need is a mapping of (device,inode) -> checksum
just for the objects we checked out, then use that mapping
for commits.

This patch adds API so that callers can create a mapping via
`ostree_repo_devino_cache_new()`, then pass it to
`ostree_repo_checkout_tree_at()` which will populate it, and then
`ostree_repo_write_directory_to_mtree()` can consume it.

I plan to use this in rpm-ostree for package layering work.

Notes:
 - The old `ostree_repo_scan_hardlinks()` API still works.
 - I tweaked the cache to be a set with the checksum colocated with
   the key, to avoid a separate malloc block per entry.

https://github.com/GNOME/ostree/pull/167
2016-01-07 14:19:12 -05:00
Colin Walters 21fbc16bc3 repo: Expose dfd-relative mtree writes as public API
For use in rpm-ostree.  We already had all of the internals for this.
2016-01-05 17:43:08 -05:00
Giuseppe Scrivano 6b1e495a7a repo: new function ostree_repo_prune_static_deltas
Extract existing code from ostree_repo_prune and add an argument COMMIT,
that controls which commit purge.  If not set, the old behavior is kept.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-12-18 11:21:57 +01:00
Matthew Barnes 1df16a7675 repo: Add ostree_repo_verify_summary()
Verifies signatures on a summary -- both taken as GBytes inputs -- and
returns an OstreeGpgVerifyResult.
2015-12-17 15:49:44 -05:00
Giuseppe Scrivano 14ffd7022a libostree: add new API ostree_repo_write_commit_with_time
It extends ostree_repo_write_commit as it permits to override the
commit timestamp.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-11-16 11:07:55 +01:00
Alexander Larsson ec56fea821 deltas: Support passing filename to ostree_repo_static_delta_execute_offline
If you pass a diriectory it will look for the "superblock" child, otherwise
it will use the file as the superblock. I need this in xdg-app to be able
to install any filename as a bundle.
2015-11-10 08:56:15 +01:00
Giuseppe Scrivano 11ba9056ea pull: new option --commit-metadata-only
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-07-24 12:43:35 -04:00
Giuseppe Scrivano 133fb5ffdc libostree: new API ostree_repo_remote_list_refs
The new API permits to query a remote repository summary file and
retrieve the list of available refs.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-07-24 12:37:42 -04:00
Matthew Barnes 104b8bf6fb repo: Add a "sysroot-path" property
Adds ostree_repo_new_for_sysroot_path() to specify a system root path.
OstreeRepo otherwise uses _ostree_get_default_sysroot_path().
2015-07-16 12:48:56 -04:00
Giuseppe Scrivano b09c9b7005 repo: new function ostree_repo_remote_get_gpg_verify_summary
It allows to specify whether GPG verification for the summary file is
enabled for a specific repository.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-06-26 11:02:25 +02:00
Matthew Barnes 0dbf91484b repo: Add ostree_repo_remote_fetch_summary()
Reusable method for fetching a summary file and signatures.
2015-06-26 11:02:24 +02:00
Giuseppe Scrivano 96baf34edf ostree-repo: document OSTREE_REPO_COMMIT_MODIFIER_FLAGS_GENERATE_SIZES
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-05-19 11:13:34 +02:00
Matthew Barnes 4d7e73ede1 repo: Add ostree_repo_remote_gpg_import()
Imports one or more GPG keys from a source stream or from the user's
personal keyring into a remote-specific keyring.  The keys to import
can optionally be restricted by a list of key IDs.

The imported keys are used to conduct GPG verification when pulling
from the given remote.
2015-05-13 13:08:49 -04:00
Giuseppe Scrivano fee785a72e ostree-repo: add new API to sign the summary file
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-05-07 21:58:04 +02:00
Colin Walters 5becd5ccad Teach fsck about partial commits
An OSTree user noticed that `ostree fsck` would produce `missing
object` errors in the case of interrupted pulls.

It's possible to do e.g. `ostree pull --subpath=/usr/share/rpm ...`,
which gets you just that portion of the commit.  The use case for this
was being able to see what changes would appear in an update before
actually downloading all of it.

(I think this would be better covered by static deltas, but those
 aren't final yet, and `--subpath` predates it)

Further, `.commitpartial` is used as a successor to the `transaction`
symlink for more precise knowledge in the case where a pull was
interrupted that we needed to resume scanning.

So it makes sense for `ostree fsck` to be aware of it.
2015-05-06 08:07:20 -04:00
Matthew Barnes 54bf665521 repo: Add ostree_repo_remote_get_gpg_verify()
Trivial function, but it does at least centralize the default value.
2015-04-16 18:13:13 -04:00
Colin Walters c2aabcac3b ostree_repo_checkout_tree_at: New API for checkouts
rpm-ostree currently uses ostree_repo_checkout_tree(), which as a side
effect will use the uncompressed objects cache by default.  This is
rather annoying if you're using rpm-ostree on a server-side
repository, because if you then rsync the repo, you'll be syncing out
the uncompressed objects unless you exclude them.

We added the ability to disable the uncompressed cache in the
repository config to fix this, but it's better to allow application
control over this.  The uncompressed cache will in some future version
become opt in as well.

This new API further:
 - Drops the `GFile` usage in favor of `openat` APIs
 - Improves ergonomics by avoiding callers having to query the source
   `GFileInfo` (and carry around a copy of `OSTREE_GIO_FAST_QUERYINFO`)
 - Has a more extensible options structure

Per the comment, I rather crudely have the `ostree checkout` builtin
call both APIs to ensure some testing coverage.

However, I'd like to in the future have easier-to-set-up testing code
that calls `libtest.sh` to set up dummy data.
2015-04-07 15:12:16 -04:00
Matthew Barnes a25c7fab12 Add ostree_repo_is_writable() 2015-03-25 17:24:05 -04:00
Matthew Barnes 8d127b9dcb repo: Add ostree_repo_verify_commit_ext()
Similar to ostree_repo_verify_commit(), but returns more verification
details by way of an OstreeGpgVerifyResult object instead of a boolean.
2015-03-18 11:52:47 -04:00
Colin Walters 0f74ed62b7 repo: Add a new iterator traversal API for commits
This is a more optimized version of the GFile * APIs, and is now used
internally by the previous ostree_repo_traverse().
2015-02-16 10:10:35 -05:00
Giuseppe Scrivano 6a3959c895 syntax-check: Remove empty lines at the end of file
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-02-02 15:07:56 +01:00
Colin Walters 1bcc7a8e3a Merge branch 'giuseppe/staticdeltas' of https://github.com/giuseppe/ostree 2014-12-19 16:31:31 -05:00
Matthew Barnes 880328ba03 Add ostree_repo_pull_default_console_progress_changed()
Replaces ot_common_pull_progress() in ostree binary, so it can be shared
with rpm-ostree.
2014-12-18 21:31:53 -05:00
Colin Walters ca678224be Static deltas support
https://bugzilla.gnome.org/show_bug.cgi?id=721799
2014-12-18 12:48:47 +01:00
Matthew Barnes 880940f93b doc: Update gtk-docs 2014-12-17 19:34:17 -05:00
Matthew Barnes 39be27fc93 Add ostree_repo_remote_list()
Lists available remote names.
2014-12-17 16:05:37 -05:00
Colin Walters f6a6e68412 Add more flexible _remote_change() API , expose via 'ostree remote'
For Anaconda, I needed OSTREE_REPO_REMOTE_CHANGE_ADD_IF_NOT_EXISTS,
with the GFile *sysroot argument to avoid ugly hacks.  We want to
write the content provided via "ostreesetup" as a remote to the target
chroot only in the case where it isn't provided as part of the tree
content itself.

This is also potentially useful in idempotent systems management tools
like Ansible.

https://bugzilla.gnome.org/show_bug.cgi?id=741577
2014-12-15 21:28:09 -05:00
Matthew Barnes f79896693e repo: Add ostree_repo_remote_get_url()
Peeking at remote details by way of ostree_repo_copy_config() doesn't
work anymore.
2014-12-08 12:49:23 -05:00
Colin Walters 64dec0add8 libostree: Add ostree_repo_pull_with_options()
We potentially need a lot of argument types for pull.  Rather than
have a C function with tons of arguments, let's use a GVariant a{sv}
as a handy extensible (and immutable) bag of properties.

This is prepratory work for adding an option to pull to traverse
history.

https://bugzilla.gnome.org/show_bug.cgi?id=737844
2014-10-28 11:16:51 -04:00
Colin Walters cdfcf09316 Add "ostree remote delete" and corresponding API
For Anaconda, we have an ugly bootstrapping problem where we need to
add the remote to the repository's config, then do a pull+deploy, then
remove and re-add the config, because /etc/ostree/remotes.d doesn't
exist yet in the target system.

https://bugzilla.gnome.org/show_bug.cgi?id=738698
2014-10-17 19:09:42 -04:00
Giuseppe Scrivano 68b37dc001 libostree: fix a gobject-introspection warning
src/libostree/ostree-repo.c:1759: Warning: OSTree:
  ostree_repo_import_object_from: unknown parameter 'checksum' in
  documentation comment, should be 'sha256'

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2014-10-06 19:03:34 +02:00
Colin Walters 7ab1fb2369 Add API to directly link() objects between repositories
And use it in pull-local.  As one might expect, this is blazingly fast
if they're on the same filesystem.

I'll be using this to "promote" builds between different repositories.
2014-10-01 23:20:35 -04:00
Colin Walters f8f5da219e Add repository "summary" file and metalink support
For Fedora and potentially other distributions which use globally
distributed mirrors, metalink is a popular solution to redirect
clients to a dynamic set of mirrors.

In order to make metalink work though, it needs *one* file which can
be checksummed.  (Well, potentially we could explode all refs into the
metalink.xml, but that would be a lot more invasive, and a bit weird
as we'd end up checksumming the checksum file).

This commit adds a new command:

$ ostree summary -u

To regenerate the summary file.  Can only be run by one process at a
time.

After that's done, the metalink can be generated based on it, and the
client fetch code will parse and load it.

https://bugzilla.gnome.org/show_bug.cgi?id=729585
2014-09-03 13:21:52 -04:00
Anne LoVerso 3742c32945 repo-pull: Allow pulling only one directory
Changes the pull API to allow pulling only a single directory instead
of the whole deployment.  This option is utilized by the check-diff
option in rpm-ostree.

Add a new state directory to hold <checksum>.commitpartial files, so
we know that we've only downloaded partial state.
2014-08-20 15:09:32 -04:00
Anne LoVerso fd56952d6b ostree-repo-resolve-rev: Add function to accept a partial checksum
This patch adds a function that will parse a partial checksum when
resolving a refspec.  If the inputted refspec matches a truncated
existing checksum, it will return that checksum to be parsed.  If
multiple truncated checksums match the partial refspec, it is not
unique and will return false.  This addition is inspired by the same
functionality in Docker, which allows a user to reference a specific
commit without typing the entire checksum.

partial checksums: Add function to abstract comparison

This modifies the list_objects and list_objects_at functions
to take an additional argument for the string that a commit starts
with.  If this string arg is not null, it will only list commit
objects beginning with that string.  This allows for a new function
ostree_repo_list_commit_objects_starting_with to pass a partial string
and return a list of all matching commits.  This improves on the
previous strategy of listing refs because it will list all commit objects,
even ones in past history.  This update also includes bugfixes on
error handling and string comparison, and changes the output structure
of resolve_partial_checksum. The new strcuture will no longer return FALSE
without error.  Also, the hashtable foreach now uses iter.  Also
includes modified test file
2014-07-18 15:15:44 -04:00
Colin Walters 2265ccdeb3 libostree: Add ostree_repo_remote_add() API, port "ostree remote add"
At least one external tool is using the API, and wants to add a
remote, but all of the logic right now is in the tool.  Move it to the
library.

https://bugzilla.gnome.org/show_bug.cgi?id=731984
2014-06-20 15:36:46 -04:00