Commit Graph

238 Commits

Author SHA1 Message Date
Matthew Leeds ed41822b45 Rename core.repo-finders to core.default-repo-finders
This renames a config key to make its semantics more obvious. Despite
what the commit message says, it only applies when a set of repo finders
is not specified (either on the command line or in a library API call).
This also renames the corresponding ostree_repo_get function. We can do
this since it hasn't been released yet.

Closes: #1763
Approved by: pwithnall
2018-10-23 14:26:50 +00:00
Matthew Leeds 3956fc885b Allow disabling pulling from LAN/USB/Internet
Currently libostree essentially has two modes when it's pulling refs:
the "legacy" code paths pull only from the Internet, and the code paths
that are aware of collection IDs try to pull from the Internet, the
local network, and mounted filesystems (such as USB drives). The problem
is that while we eventually want to migrate everyone to using collection
IDs, we don't want to force checking LAN and USB sources if the user
just wants to pull from the Internet, since the LAN/USB code paths can
have privacy[1], security[2], and performance[3] implications.

So this commit implements a new repo config option called "repo-finders"
which can be configured to, for example, "config;lan;mount;" to check
all three sources or "config;mount;" to disable searching the LAN. The
set of values mirror those used for the --finders option of the
find-remotes command. This configuration affects pulls in three places:
1. the ostree_repo_find_remotes_async() API, regardless of whether or
not the user of the API provided a list of OstreeRepoFinders
2. the ostree_repo_finder_resolve_async() /
ostree_repo_finder_resolve_all_async() API
3. the find-remotes command

This feature is especially important right now since we soon want to
have Flathub publish a metadata key which will have Flatpak clients
update the remote config to add a collection ID.[4]

This effectively fixes https://github.com/flatpak/flatpak/issues/1863
but I'll patch Flatpak too, so it doesn't pass finders to libostree only
to then have them be removed.

[1] https://github.com/flatpak/flatpak/issues/1863#issuecomment-404128824
[2] https://github.com/ostreedev/ostree/issues/1527
[3] Based on how long the "ostree find-remotes" command takes to
  complete, having the LAN finder enabled slows down that step of the
  pull process by about 40%. See also
  https://github.com/flatpak/flatpak/issues/1862
[4] https://github.com/flathub/flathub/issues/676

Closes: #1758
Approved by: cgwalters
2018-10-21 19:11:43 +00:00
Colin Walters 673cacd633 repo: Add a checkout option to not hardlink zero-sized files
In rpm-ostree we've hit a few cases where hardlinking zero-sized
files causes us problems.  The most prominent is lock files in
`/usr/etc`, such as `/usr/etc/selinux/semanage.LOCK`.  If there
are two zero-sized lock files to grab, but they're hardlinked,
then locking will fail.

Another case here is if one is using ostree inside a container
and don't have access to FUSE (i.e. `rofiles-fuse`), then the
ostree hardlinking can cause files that aren't ordinarily hardlinked
to become so, and mutation of one mutates all.  An example where
this is concerning is Python `__init__.py` files.

Now, these lock files should clearly not be in the tree to begin
with, but - we're not gaining a huge amount by hardlinking these
files either, so let's add an option to disable it.

Closes: #1752
Approved by: jlebon
2018-10-11 16:32:25 +00:00
Umang Jain a0937b6cf0 lib/repo: Separate min-free-space-* calculation from transaction codepath
Earlier, the actual reserved space (in blocks) were calculated inside the
transaction codepath ostree_repo_prepare_transaction(). However, while
reworking on ostree_repo_get_min_free_space_bytes() API, it was realized that
this calculation can be done independently from the transaction's codepaths, hence
enabling the usage for ostree_repo_get_min_free_space_bytes() API irrespective
of whether there is an ongoing transaction or not.

https://github.com/ostreedev/ostree/issues/1720

Closes: #1722
Approved by: pwithnall
2018-09-21 15:09:12 +00:00
Matthew Leeds fc84fb402c lib/repo: Define a metadata key, ostree.deploy-collection-id
This commit defines a metadata key that tells clients to update their
remote config to add a collection ID. This functionality is currently
implemented in Flatpak for the key "xa.collection-id", but there are two
good reasons for moving the key to OSTree:

1) Servers such as Flathub shouldn't set xa.collection-id in their
metadata now or in the medium term future, because many users are still
using old versions of Flatpak and OSTree[1] which would hit various
bugs[2][3][4] on the P2P code paths that are enabled by collection IDs.
Defining a new key means that only clients running recent
(as-yet-unreleased) versions of Flatpak and OSTree will pay attention to
it and deploy the collection ID, leaving the users on old versions
unaffected.

2) OSTree is as "invested" in collection IDs as Flatpak, so there's no
reason the key should be defined in Flatpak rather than here. According
to Philip Withnall, the reason the key was put in Flatpak originally was
that at the time there was uncertainty about tying OSTree to collection
IDs.

[1] https://ahayzen.com/direct/flathub.html#downloadsbyflatpakstacked
[2] https://github.com/ostreedev/ostree/commit/e4e6d85ea
[3] https://github.com/flatpak/flatpak/commit/5813639f
[4] https://github.com/flatpak/flatpak/commit/5b21a5b7

Closes: #1726
Approved by: pwithnall
2018-09-21 13:04:51 +00:00
Umang Jain 68420f70bb lib/repo: Add an API to get min-free-space-* reserved bytes
https://phabricator.endlessm.com/T23694

Closes: #1715
Approved by: cgwalters
2018-09-04 21:31:34 +00:00
Matthew Leeds 8fbf19c9f5 Make P2P API public (no longer experimental)
Currently the API that allows P2P operations (e.g. pulling an ostree ref
from a LAN or USB source) is hidden behind the configure flag
--enable-experimental-api. This commit makes the API public and makes
that flag essentially a no-op (leaving it in place in case we want to
use it again in the future). The P2P API has been tested over the last
several months and proven to work.

This means that since we're no longer using the "experimental" feature
flag, P2P builds of Flatpak will fail when using versions of OSTree from
this commit onwards, until Flatpak is patched in the near future. If you
want to build Flatpak < 0.11.8 with P2P enabled and link against OSTree
2018.6, you'll have to patch Flatpak.  However, since Flatpak won't yet
have a hard dependency on OSTree 2018.6, it needs a new way to determine
if the P2P API in OSTree is available, so this commit adds a "p2p"
feature flag. This way the feature set is more semantically correct than
if we had continued to use the "experimental" feature flag.

In addition to making the P2P API public, this commit makes the P2P unit
tests run by default, removes the f27-experimental CI instance that's no
longer needed, changes a few man pages to reflect the changes, and
updates the bash completion script to accept the new commands and
options.

Closes: #1596
Approved by: cgwalters
2018-06-04 19:20:10 +00:00
Colin Walters 371081d123 lib: Add a public helper method for pruning to find all ref'd commits
Prep for reworking how we do sysroot cleanup.  We're going to
start doing more lowlevel pruning work there, and I wanted to avoid
duplicating the ref enumeration.

Closes: #1566
Approved by: jlebon
2018-05-24 12:56:11 +00:00
Colin Walters 8c1542134c lib/repo: Enable locking by default, but drop external API
The code has been sitting around for a while but since I disabled
it by default, I doubt anyone is really using it or relying on it.

This patch and turns on locking by default, and also drops the
API which was only public in the experimental API builds.
Conceptually these are two distinct things, and we
may actually want to split up the patches.

I don't think this will break anyone, but it's hard to say for sure.
It's also going to be hard to find out until we actually release
I suspect...

But anyone who is broken should be able to add `locking=false` into
their repo config.  On the flip side Endless has been shipping with
this enabled and it is reported to help.

The reason to drop the APIs: I'm a bit concerned about the interactions over time
between libostree's use of the API and any apps that start using it.
For example, if an app specifies a SHARED lock in their code, then
later internally we decide to temporarily grab an `EXCLUSIVE`, but the
app had a second thread/process that was `EXCLUSIVE` already, and
that process was waiting on the first bit of code, then we could
deadlock. I can't think of a real world situation where this would happen
yet though.

We are likely to in the future have say `fsck` take an external lock,
`checkout` grab a shared one, etc.

Closes: #1555
Approved by: jlebon
2018-04-30 17:24:51 +00:00
Alexander Larsson f258e9e5ff lib/repo: Add ostree_repo_traverse_commit_union_with_parents
This is a version of ostree_repo_traverse_commit_union that also
remembers where the objects came from, by recording the parent
relationships in a hashtable. This can be used to later find which
commits each object was from, which we want to use in fsck.

Closes: #1533
Approved by: cgwalters
2018-04-14 15:36:21 +00:00
Jonathan Lebon 2e95e06616 lib/checkout: add filter API to skip over files
This is analogous to the filtering support for the commit API: we allow
library users to skip over checking out specific files. This is useful
in some tricky situations where we *know* that the files to be checked
out will conflict with existing files in subtle ways.

One such example is in rpm-ostree support for multilib. There, we want
to allow checking out a package onto an existing tree, but skipping over
files that are not coloured to our preferred value (e.g. not overwriting
an i686 version of `ldconfig` if we already have the `x86_64` version).
See https://github.com/projectatomic/rpm-ostree/pull/1227 for details.

Closes: #1441
Approved by: cgwalters
2018-02-06 15:38:20 +00:00
Marcus Folkesson 6bf4b3e1d8 Add SPDX-License-Identifier to source files
SPDX License List is a list of (common) open source
licenses that can be referred to by a “short identifier”.
It has several advantages compared to the common "license header texts"
usually found in source files.

Some of the advantages:
* It is precise; there is no ambiguity due to variations in license header
  text
* It is language neutral
* It is easy to machine process
* It is concise
* It is simple and can be used without much cost in interpreted
  environments like java Script, etc.
* An SPDX license identifier is immutable.
* It provides simple guidance for developers who want to make sure the
  license for their code is respected

See http://spdx.org for further reading.

Signed-off-by: Marcus Folkesson <marcus.folkesson@gmail.com>

Closes: #1439
Approved by: cgwalters
2018-01-30 20:03:42 +00:00
Colin Walters 7935b881bf lib/repo: Add an API to mark a commit as partial
For the [rpm-ostree jigdo ♲📦](https://github.com/projectatomic/rpm-ostree/issues/1081) work.
We're basically doing "pull" via a non-libostree mechanism, and this
should be fully supported.  As I mentioned earlier we should try to
have `ostree-repo-pull.c` only use public APIs; this gets us closer
to that.

Closes: #1376
Approved by: jlebon
2017-12-14 15:51:07 +00:00
Colin Walters 73d910e82e Add public API for fsck, use it before loading metadata
A while ago I did `truncate -s 0 /path/to/repo/00/123.commit`, and expected a
checksum error, but I actually got a validation error due to us loading the
commit into a variant and trying to parse out the parent checksum, etc.

I first started by changing the `load_and_fsck_one_object()` function to
checksum before loading, but the problem is that we do a traverse of all objects
first. Fixing this is going to require an `OSTREE_REPO_COMMIT_TRAVER_FLAG_FSCK`
or something.

In the meantime at least though, let's add a public API to fsck a single object
which *does* checksum cleanly before parsing the object, and change the `fsck`
command to use it.

We then change the fsck binary to do this while iterating over the refs
and finding the commit object.  This way we'll at least get a checksum
first for commit objects, even if not dirtree/dirmeta.

Closes: #1364
Approved by: jlebon
2017-12-12 14:03:09 +00:00
Dan Nicholson 7d863ed9e4 lib/repo: Add locking auto cleanup handler
Define an auto cleanup handler for use with repo locking. This is based
on the existing auto transaction cleanup. A wrapper for
ostree_repo_lock_push() is added with it. The intended usage is like so:

  g_autoptr(OstreeRepoAutoLock) lock = NULL;
  lock = ostree_repo_auto_lock_push (repo, lock_type, cancellable, error);
  if (!lock)
    return FALSE;

The functions and type are marked to be skipped by introspection since I
can't see them being usable from bindings.

Closes: #1343
Approved by: cgwalters
2017-12-05 02:32:47 +00:00
Dan Nicholson 4e78ddd2da lib/repo: Add repo locking mechanism
Currently ostree has no method of guarding against concurrent pruning.
When there are multiple repo writers, it's possible to have a pull or
commit race against a prune and end up with missing objects.

This adds a file based repo locking mechanism. The intention is to take
a shared lock when writing objects and an exclusive lock when deleting
them. In order to make use of the locking throughout the library in a
fine grained fashion, the lock acts recursively with a stack of lock
states. If the lock becomes exclusive, it will stay in that state until
the stack is unwound past the initial exclusive push. The file locking
is similar to GLnxLockFile in that it uses open file descriptor locks
but falls back to flock when needed.

The lock also attempts to be thread safe by storing the lock state in
thread local storage with GPrivate. This means that each thread will
have an independent lock for each repository it opens. There are some
drawbacks to that, but it seemed impossible to manage the lock state
coherently in the face of multithreaded access.

The API is a push/pop interface in accordance with the recursive nature
of the locking. The push interface uses an enum that's translated to
LOCK_SH or LOCK_EX as needed. Both interfaces use an internal timeout
field to decide whether to manage the lock in a blocking or non-blocking
fashion. The intention is to allow ostree applications as well as
administrators to control this timeout. For now, the default is a 30
second timeout.

Note that the timeout is handled synchronously in thread since the lock
is maintained in thread local storage. I.e., the thread that acquires
the lock needs to be the same thread that runs the operation. There may
be a way to offer an asynchronous version, but it's not clear exactly
how that would work since it would likely involve a separate thread that
invokes a callback when the locking operation completes.

https://bugzilla.gnome.org/show_bug.cgi?id=759442

Closes: #1343
Approved by: cgwalters
2017-12-05 02:32:47 +00:00
Colin Walters 7c8ea25306 lib/repo: Add a DEVINO_CANONICAL commit modifier flag
I was seeing the `Writing OSTree commit...` phase of rpm-ostree
being very slow lately.  This turns out to be more fallout from
https://github.com/ostreedev/ostree/pull/1170
AKA commit: 8fe4536

Loading the xattrs is slow on my system (F27AW, XFS+LVM, NVMe). I haven't fully
traced through why, but AIUI at least on XFS the xattrs are often stored outside
of the inode so it's a little bit like doing an `open()+read()`. Plus there's
the LSM overhead, etc.

The thing is that for rpm-ostree's package layering use case, we
basically always want to treat the on-disk state as canonical.  (There's
a subtle case here if one does overrides for something that contains
policy but we'll fix that).

Anyways, so we're in a state now where we do the slow but correct thing by
default, which seems sane. But let's allow the app to opt-in to telling us
"really trust devino". The difference between a `stat()` + hash table lookup
versus the full xattr load on my test case of `rpm-ostree install
./tree-1.7.0-10.fc27.x86_64.rpm` is absolutely dramatic; consistently on the
order of 10s without this support, and <1s with (800ms).

Closes: #1357
Approved by: jlebon
2017-12-04 14:42:37 +00:00
Philip Withnall 3cf53f7c58 lib/repo: Add gtk-doc comment to OstreeRepoCommitState
Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1335
Approved by: cgwalters
2017-11-08 15:47:11 +00:00
Philip Withnall 9c4870b5e1 lib/repo: Add OSTREE_REPO_COMMIT_STATE_NORMAL to represent most commits
This allows more explicit handling of commit state in code using
libostree, rather than hard-coding a commit state of 0 for ‘normal’.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1335
Approved by: cgwalters
2017-11-08 15:47:11 +00:00
Colin Walters 1222c2271b repo: Add wrapper function for setting devino cache on checkout opts
I was trying to use this with pygobject for an OCI+ostree project, and pygobject
rejected simply assigning to the field (understandably, since it can't bind the
lifetime together).

Add a wrapper function, which is still unsafe, but hides that unsafety
where most people shouldn't find it.  And if they do...well, sorry,
Rust wasn't invented when ostree was started.

Closes: #1295
Approved by: pwithnall
2017-10-20 18:20:19 +00:00
Colin Walters b8251d26bd lib/checkout: For "process whiteouts" mode, replace directories too
I'm playing around with some ostree ⇔ OCI/Docker bits, and ran
into this while importing an OCI image that built from the Fedora
base image where `/home` is a regular directory, and I added a layer
that did the ostree bits of moving it to `/var` and leaving a symlink.

OCI/Docker supports this.  Now since "process whiteouts" is really the
"enable OCI/Docker" mode, let's only replace dirs if that's enabled.
This leaves the `UNION_FILES` targeted for its original use case
which is unioning components/packages.  (Although that use case itself
is now a bit superceded by `UNION_IDENTICAL`, but eh).

Closes: #1294
Approved by: jlebon
2017-10-20 13:20:27 +00:00
Colin Walters bba7eb8069 commit: Add _CONSUME modifier flag
For many cases of commit, we can actually optimize things by simply "adopting"
the object rather than writing a new copy. For example, in rpm-ostree package
layering.

We can only make that optimization though if we take ownership of the file. This
commit hence adds an API where a caller tells us to do so. For now, that just
means we `unlink()` the files/dirs as we go, but we can now later add the
"adopt" optimization.

Closes: #1255
Approved by: jlebon
2017-10-10 13:02:08 +00:00
Philip Withnall 22c1fdfbd3 lib/repo: Change resolve_keyring_for_collection() to return a remote
Instead of returning just the keyring filename, return the entire
OstreeRemote, which has the keyring filename as one of its members. This
will simplify some upcoming changes, and allows slightly improved debug
logging.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1202
Approved by: cgwalters
2017-09-27 16:38:07 +00:00
Philip Withnall 149aec1099 lib/repo-refs: Add first version of ostree_repo_resolve_collection_ref()
This is a parallel for ostree_repo_resolve_rev_ext() which works on
collection–refs. At the moment, the implementation is simple and uses
ostree_repo_list_collection_refs(). In future, it could be rewritten to
check the checksum directly rather than enumerating all
potentially-relevant checksums.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1182
Approved by: cgwalters
2017-09-27 14:44:00 +00:00
Colin Walters 25a7c4bd4e lib/pull: Default checksum for archive mirror, add TRUSTED_HTTP flag
I now think commit fab1e113db was a mistake;
because it breaks the mental model that at least I'd built up that "local repos
don't have checksums verified, HTTP does".

For example, a problem with this is (with that mental model in place) it's easy
for people who set up mirrors like this to then do local pulls, and at that
point we've done a deployment with no checksum verification.

Further, since then we did PR #671 AKA commit 3d38f03 which is really most of
the speed hit.

So let's switch the default even for this case to doing checksum verification,
and add `ostree pull --http-trusted`. People who are in situations where they
know they want this can find it and turn it on.

Closes: https://github.com/ostreedev/ostree/issues/1211

Closes: #1212
Approved by: jlebon
2017-09-26 18:07:43 +00:00
Colin Walters 6e4146a354 tree-wide: Remove Emacs modelines
We added a `.dir-locals.el` in commit: 9a77017d87
There's no need to have it per-file, with that people might think
to add other editors, which is the wrong direction.

Closes: #1206
Approved by: jlebon
2017-09-21 21:38:34 +00:00
Philip Withnall 64b23fd089 lib/repo: Add ostree_repo_hash() and tests
Add a hash function for OstreeRepo instances, which relies on the repo
being open, and hence being able to hash the device and inode of its
root directory.

Add unit tests for this and ostree_repo_equal().

Signed-off-by: Philip Withnall <withnall@endlessm.com>

https://github.com/ostreedev/ostree/issues/1191

Closes: #1205
Approved by: cgwalters
2017-09-21 21:25:58 +00:00
Philip Withnall c62b36046b lib/repo: Drop outdated FIXME comment from ostree-repo.h
This was some incomplete planning from while the find_remotes() API was
being designed; now totally outdated.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1179
Approved by: cgwalters
2017-09-19 14:51:09 +00:00
Philip Withnall 981eb6c226 lib/repo: Add ostree_repo_equal() for comparing repos
This will compare their root directory inodes to see if they are the
same repository on disk. A convenience method for the users of the
public API who can’t access OstreeRepo.inode.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1179
Approved by: cgwalters
2017-09-19 14:51:09 +00:00
Colin Walters 051cdf396c lib/checkout: Rename disjoint union, change to merge identical files
It turns out that librpm automatically merges identical files between
distinct packages, and this occurs in practice with Fedora today between
`chkconfig` and `initscripts` for exmaple.

Since we added this for rpm-ostree, we basically want to do what librpm does,
let's change the semantics to do a merge.  While we're here rename
to `UNION_IDENTICAL`.

Closes: #1156
Approved by: jlebon
2017-09-13 19:19:33 +00:00
Ruixin f07432d4ce checkout: add an extra checkout_overwrite mode
This is for issue projectatomic/rpm-ostree#365,
an extra option of overwrite mode is added to the checkout command
so that when there is "non-directory" file already exist
during checkout, the error will be handled.

Some tests are added for regression

Closes: #1116
Approved by: cgwalters
2017-09-01 15:42:50 +00:00
Colin Walters 138c4d7aae libarchive: Add support for translating paths during commit
For rpm-ostree, I want to move RPM files in `/boot` to `/usr/lib/ostree-boot`.
This is currently impossible without forking the libarchive code.  Supporting
this is pretty straightforward; we already had pathname translation in
the libarchive code, we just need to expose it as an option.

On the command line side, I chose to wrap this as a regexp. That should be good
enough for a lot of use cases; sophisticated users should as always be making
use of the API. Note that this required some new `#ifdef LIBARCHIVE` bits to use
the new API. Following previous patterns here, we use the new API only if a
relevant option is enabled, ensuring unit test coverage of both paths.

For the test cases, I ended up changing the accounting to avoid having to
multiply the test count.

Closes: #1105
Approved by: jlebon
2017-08-30 14:30:30 +00:00
Matthew Leeds 7ed881baa7 lib/repo-refs: Include remote refs when using collections
When working with collections it can be useful to see remote refs rather
than just local and mirrored ones. This commit changes the "ostree refs
-c" output to include remote refs, and includes remote refs with
collection IDs in summary file generation as well. The former behavior
is consistent with how "ostree refs" works, and the latter behavior is
useful in facilitating P2P updates even when mirrors haven't been
configured.

To accomplish this, OstreeRepoListRefsExtFlags was extended with an
EXCLUDE_REMOTES flag. This was done rather than an INCLUDE_REMOTES flag
so that existing calls to ostree_repo_list_refs_ext continue to have the
same behavior. This flag was added to ostree_repo_list_collection_refs
(which is an experimental API break).

Also, add unit tests for the "refs -c" and summary file behavior, and
update relevant tests.

Closes: #1069
Approved by: cgwalters
2017-08-24 19:57:33 +00:00
Colin Walters 95bac299e5 lib/repo: Add some assertions for ABI sizes
Things like https://sourceware.org/libabigail/manual/abidiff.html
look interesting but in a brief look I couldn't work out
how to conveniently use them for quick ABI sanity checking without
doing a diff from a previous build (which we could do but would be
more involved).

This way will at least catch struct ABI breaks on x86_64 which
I think we'd be most likely to do accidentally when trying
to use one of the previous unused values.

I found the hole values via gdb's `pahole` command.

Closes: #1108
Approved by: jlebon
2017-08-24 14:58:51 +00:00
Jonathan Lebon 11fdca2919 pull: better description for --mirror
Describe the behaviour of --mirror a bit better.

Closes: #1100

Closes: #1099
Approved by: dustymabe
2017-08-23 01:23:38 +00:00
Colin Walters fd98bda3c7 repo: Introduce ostree_repo_open_at() and ostree_repo_create_at()
This essentially completes our fd-relative conversion.

While here, I cleaned up the semantics of `ostree_repo_create()` and
`ostree_repo_create_at()` to be more atomic - basically various scripts were
testing for the `objects` subdirectory, so let's formalize that.

Closes: #820
Approved by: jlebon
2017-08-15 12:35:10 +00:00
Philip Withnall f35b409077 lib/repo-refs: Add ostree_repo_remote_list_collection_refs() API
This parallels ostree_repo_remote_list_refs(), but returns a map of
OstreeCollectionRef → checksum, and includes refs from collection IDs
other than the remote repository’s main collection ID.

Use this in OstreeRepoFinderConfig to ensure that refs are matched
against even if they’re stored in the repository summary file’s
collection map, rather than its main ref map. This fixes false negatives
when searching for refs in some situations.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #1058
Approved by: cgwalters
2017-08-08 13:59:58 +00:00
Colin Walters d5273b34d0 lib/repo: Add API to create and list ref aliases
There are multiple use cases where we'd like to alias refs.

First, having a "stable" alias which gets swapped across major
versions: https://pagure.io/atomic-wg/issue/228

Another case is when a ref is obsoleted;
<https://pagure.io/atomic-wg/issue/303>
This second one could be done with endoflife rebase, but I think
this case is better on the server side, as we might later change
our minds and do actual releases there.

I initially just added some test cases for symlinks in the `refs/heads` dir to
ensure this actually works (and it did), but I think it's worth having APIs.

Closes: #1033
Approved by: jlebon
2017-08-02 17:33:10 +00:00
Philip Withnall acb14648d7 lib/repo: Add OSTREE_REPO_METADATA_REF as a well-known metadata store
As discussed in https://github.com/ostreedev/ostree/pull/946, the
summary file is becoming an unsigned cache of ref information; any
additional metadata for the repository needs to move elsewhere in order
to remain signed. Introduce OSTREE_REPO_METADATA_REF as the well-known
name of a ref where such metadata can live, as the metadata on
contentless commits.

Don’t yet update the documentation for summary-related methods to
mention this, since it’s still hidden behind the
--enable-experimental-api configure option.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #946
Approved by: cgwalters
2017-07-11 19:50:32 +00:00
Philip Withnall 292230301d lib/repo-finder: Add basic support for finding remote URIs by ref name
Add an initial OstreeRepoFinder interface (but no implementations),
which will find remote URIs by ref names and collection IDs, the
combination of which is globally unique.

The new API is used in a new ostree_repo_find_updates() function, which
resolves a list of ref names to update into a set of remote URIs to pull
them from, which can be treated as mirrors. It is an attempt to
generalise resolution of the URIs to pull from, and to generalise
determination of the order and parallelisation which they should be
downloaded from in.

Includes fixes by Krzesimir Nowak <krzesimir@kinvolk.io>.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall fbf8df8829 lib/refs: Add methods for setting/listing collection–refs
These are tuples of (collection ID, ref name) which are a globally-unique
form of local ref. They use OstreeCollectionRef as an identifier, and hence
need to be accessed using new API, as the existing API uses string
identifiers and sometimes accepts refspecs. Remote names are not
supported as part an OstreeCollectionRef.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall 4de736fdfa lib/repo: Add collection ID support to OstreeRepo
Add {get,set}_collection_id() methods to OstreeRepo and some documentation
about the concept of a collection ID which globally identifies an
upstream repository. See the documentation for more details.

This will be used in future commits. For now, the new API is marked as
experimental (--enable-experimental-api).

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall 0a20e7d43c lib/ref: Add OstreeCollectionRef type for globally unique refs
This is a type representing the tuple (collection ID, ref name), which is
guaranteed to be globally unique. It will be used in upcoming commits.

It introduces the concept of a ‘collection’ which is a unique, curated
set of refs which lie in the same trust domain (i.e. all signed by the
same key and validated by the same developer). Flathub might be a
collection, for example; or the set of OS refs coming from a particular
OS vendor.

It includes a function for validating collection IDs.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Colin Walters 0635fcbfd9 lib/checkout: Add bareuseronly_dirs option
This is a continuation of https://github.com/ostreedev/ostree/pull/926
for directories instead of files.

See: https://github.com/flatpak/flatpak/issues/845

This option suppresses mode bits outside of `0775` for directory
checkouts.  I think most people should start doing this by default,
and use explicit overrides for e.g. `/tmp` if doing a recommit based
on a checkout.

Closes: #927
Approved by: alexlarsson
2017-06-13 20:05:31 +00:00
Colin Walters 6ed824bf00 lib/pull: Add OSTREE_REPO_PULL_FLAGS_BAREUSERONLY_FILES
This is an option which is intended mostly for flatpak;
see: https://github.com/flatpak/flatpak/issues/845

We're adding an option for pulling into *all*
repo modes that has an effect similar to the `bare-user-only`
change from https://github.com/ostreedev/ostree/pull/909

This way one can pull content into e.g. a root-owned `bare` repository and
ensure that there aren't any setuid or world-writable files.

Closes: #926
Approved by: alexlarsson
2017-06-13 18:44:28 +00:00
Colin Walters e8efd1c8dc checkout: Add SELinux labeling for checkout, use in deploy
This is a variant of the efforts in https://github.com/ostreedev/ostree/pull/741
Working on `rpm-ostree livefs`, I realized though I needed to just
check out *new* files directly into the live `/etc` (and possibly
delete obsolete files).

The way the current `/etc` merge works is fundamentally different from
that.  So my plan currently is to probably do something like:

 - Compute diff
 - Check out each *new* file individually (as a copy)
 - Optionally delete obsolete files

Also, a few other things become more important - in the current deploy code, we
copy all of the files, then relabel them. But we shouldn't expose to *live*
systems the race conditions of doing that, plus we should only relabel files we
checked out.

By converting the deploy's /etc code to use this, we fix the same TODO item
there around atomically having the label set up as we create files. And further,
if we kill the `/var` relabeling which I think is unnecessary since Anaconda
does it, we could delete large chunks of code there.

In the implementation, there are two types of things: regular files, and
symlinks. For regular files, in the `O_TMPFILE` case, we have the ability to
do *everything* atomically (including SELinux labeling) before linking it into
place. So let's just use that. For symlinks, we use `setfscreatecon()`.

Closes: #797
Approved by: jlebon
2017-04-25 16:52:33 +00:00
Colin Walters 6060abbb4b repo: Add a "force copy" flag to checkout
This is intended to be used for copying `/usr/etc` → `/etc` for
deployments.

A TODO here is to use `glnx_file_copy_at()` if the repo mode allows
it - then we'd use reflinks if available.

Closes: #804
Approved by: jlebon
2017-04-24 15:26:11 +00:00
Colin Walters 6a7ee4860f Fix a few gtk-doc warnings
Just continuing to chip away at this.

Closes: #788
Approved by: jlebon
2017-04-12 15:36:46 +00:00
Colin Walters 9016e9e8be Add flag to make SELinux label failure fatal, add hack for /proc
I was working on `rpm-ostree livefs` which does some ostree-based
filesystem diffs, and noticed that we were ending up with `/proc`
not being labeled in our base trees.

Reading the selinux-policy source, indeed we have:

```
/proc			-d	<<none>>
/proc/.*			<<none>>
```

This dates pretty far back.  We really don't want unlabeled
content in ostree.  In this case it's mostly OK since the kernel
will assign a label, but again *everything* should be labeled via
OSTree so that it's all consistent, which will fix `ostree diff`.

Notably, `/proc` is the *only* file path that isn't covered when composing a
Fedora Atomic Host. So I added a hack here to hardcode it (although I'm a bit
uncertain about whether it should really be `proc_t` on disk before systemd
mounts or not).

Out of conservatism, I made this a flag, so if we hit issues down the line, we
could easily change rpm-ostree to stumble on as it did before.

Closes: #768
Approved by: jlebon
2017-04-04 15:31:49 +00:00
Alexander Larsson b2d10dcaaa commit: Add --canonical-permissions argument
This adds to file permission masks the same bitmask that will
be applied to file objects in bare-user* repos. This will be
needed in the testsuite to ensure that the things we commit
will be expressable in bare-user-only repos.

Closes: #750
Approved by: cgwalters
2017-03-27 13:48:41 +00:00