For a long time we've cached the remote configs in the repo, which
mostly makes sense for the `repo/config` file, but less sense
for `/etc/ostree/remotes.d`, because we want to support admins
interactively editing them.
One can delete the repo instance and create a new one, but that's a bit ugly.
Let's introduce an API for this so rpm-ostree can reload remotes after
admins/scripts edit them in `/etc`. We also might as well reload
any other entries in the config.
Structurually now, `ostree_repo_open()` deals with file descriptors, and then
calls `ostree_repo_reload_config()`. Except for the uncompressed cache, which is
the only thing that deals with FDs that can be configured. But we want to delete
that anyways.
No tests, since...we don't have a daemon in this codebase, don't want to shave
that yak just today.
Closes: #662
Approved by: jlebon
There are use cases for having a single repo with branches
with different lifecycles; a simple example of what I was
trying to do in CentOS Atomic Host work is have "stable"
and "devel" branches, were we want to prune devel, but
retain *all* of stable.
This patch is split into two parts - first we add a low level "delete all
objects not in this set" API, and change the current prune API
to use this.
Next, we move more logic into the "ostree prune" command. This paves the way for
demonstrating how more sophisticated algorithms/logic could be developed outside
of the ostree core.
Also, the --keep-younger-than logic already lived in the commandline, so it
makes sense to keep extending it there.
Closes: https://github.com/ostreedev/ostree/issues/604Closes: #646
Approved by: jlebon
Conceptually we've been moving towards having our GPG verification
paths be per-remote. The code internally supports this, but we
didn't expose an API to use it conveniently.
This came up when trying to add a new `gpgkeypath` option, since
right now rpm-ostree manually finds keyrings for the remote, and
hence it wasn't looking at the keypath, and said "Unknown key"
in status.
Adding an API fixes this nicely.
Closes: #576
Approved by: giuseppe
Fixes this warning:
src/libostree/ostree-repo-pull.c:2162: Warning: OSTree: ostree_repo_pull_with_options: unknown parameter 'remote_name_or_baseurl' in documentation comment, should be 'remote_name'
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Closes: #472
Approved by: jlebon
When doing a prune, we should not try to delete objects in parent
repos, since it'll fail. There is a bigger discussion about the
semantics of `parent=` to be had, but this will fix trying to use
`ostree prune --repo=/ostree/repo/extensions/rpmostree/pkgcache`.
Closes: https://github.com/ostreedev/ostree/issues/467Closes: #471
Approved by: jlebon
Since this is a new API, and adding booleans is the most likely thing
we'll do, let's stick some explicit padding for them in here now.
We could use the `unused_ints[]` but it'd be out of order, and this
will more clearly remind people about the padding. The efficiency hit
versus bitfields is annoying, but oh well, not a real world problem.
Closes: #427
Approved by: giuseppe
Since we're adding a new API, we have the opportunity to fix
the defaults. We expect clients to do a `syncfs()` or equivalent
on their own now, since it's way more efficient.
Flip the checkout fsync default to off.
Closes: #425
Approved by: giuseppe
These two functions are not safe for gobject introspection, so annotate
them to be skipped:
1) ostree_repo_import_archive_to_mtree
2) ostree_repo_export_tree_to_archive
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Closes: #417
Approved by: cgwalters
Provide a gobject introspection safe version for
`ostree_repo_checkout_tree_at'.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Closes: #417
Approved by: cgwalters
I've seen a few people hit this and wonder why checkouts are slow/take
space. Really, ensuring this happens is the *point* of OSTree.
Physical copies should be a last resort fallback for very unusual
situations (one of those is rpm-ostree checking out the db since
librpm doesn't know how to read from libostree).
Even I hit the fact that `/var` is a mountpoint disallowing hardlinks
with `/ostree` once and was confused. =)
Add this to the rofiles-fuse test case because it creates a mount
point.
Closes: #368
Approved by: jlebon
Add the functionality to use the same name for refs in local and remote
repos. This helps users keep track of local refs of remote origin, much
like local and remote git branches.
Previously, when a local ref is specified, resolve_refspec would fall
back to searching through remote repos if the ref is not found locally.
This function now takes an extra flag to specify whether it should
search through remote repos. Additionally, ostree_repo_resove_rev_ext
was added to call resolve_refspec with fallback_remote being false, so
refs --create would no longer complain when trying to create a local
ref of the same name as a remote one.
Fix remote repo parsing not being handled correctly on refs --create.
Closes: #363
Approved by: jlebon
This changes around a few things that didn't work for me:
* Section names seem to be ostree-* instead of libostree-*
* Also XML files are ostree-* (they didn't show up at all)
- gtk-doc doesn't seem to parse const _OSTREE_PUBLIC correctly
* pull documentation is now on the actual functions rather than stubs
* Update gitignore with some more files
And there some changes to make gtk-doc give fewer warnings (not finished)
Closes: #327
Approved by: cgwalters
This can be useful for validating the 3rd party data that is put in
the extensions directory and is signed with the same keys as commits
or the summary file.
Closes: #310
Approved by: cgwalters
This adds a _with_options variant of the
ostree_repo_remote_fetch_summary function, so we can tell the fetcher
to use a specific URL instead taking it from the remote config.
Closes: #290
Approved by: cgwalters
- Make hardlink handling more generic. The previous strategy worked for
tar archives, but not for cpio. It now works for both.
- Add support for SEL labeling (through the OstreeRepoCommitModifier)
- Add support for xattr_callback (through the OstreeRepoCommitModifier)
- Add support for filter (through the OstreeRepoCommitModifier)
- Add a use_ostree_convention option
Closes: #275
Approved by: cgwalters
This lets you set a prefix for the resulting archive patsh.
Especially useful in combination with --subpath, for instance
--subpath=subdir --prefix=subdir to extract just subdir.
Closes: #265
Approved by: cgwalters
These are useful for ostree users (like xdg-app) that have custom
options for remotes. In particular they are useful when we later make them
all respect self->parent_repo.
Closes: #236
Approved by: cgwalters
It accepts a `flags` argument to control its behavior. Differently
from `ostree_repo_list_refs`, the `refspec_prefix` is not removed from
the results.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
I plan to use this in rpm-ostree at least for two reasons:
- To find the mtime on the repo
- To use the tmp/ directory to stage content (but we should eventually
add a better API)
As rpm-ostree evolves, it keeps driving API additions to libostree.
This creates a relatively tight coupling.
However, if delivering via e.g. RPM, unless one manually remembers to
increment the `Requires:` in the spec file, it's possible for the two
to become desynchronized.
RPM handles versioned symbols and will ensure a dependency if the
application starts using a newer version.
To implement this, switch to `-fvisibility=hidden`, along with an
annotation in the header, and finally add a `.sym` file.
This matches what other projects like systemd and libvirt do.
Although rather than attempting to retroactively version symbols, glom
them all onto the current one.
This is a more flexible version of the previous
ostree_repo_write_archive_to_mtree() which took a file reference.
This has an extensible options structure, and in particular
now supports `ignore_unsupported_content`.
I plan to use this for importing Docker images which contain device
nodes. (There's no reason for container images to have those, so
we'll just ignore them).
Also here, just like the export variant, the caller is responsible for
setting up libarchive.
I was going to add new API for importing, and it was really confusing
that what I think of now as import and export both had "write" in the
name. It's just clearer to talk about the direction.
At the same time, include `Export` in the options structure.
This isn't an ABI break as the API isn't in a release.
At the moment I'm looking at using rpm-ostree to manage RPM inputs
which can then be converted into Docker images. It's most convenient
if we can stream directly out of libostree rather than doing a
checkout + tar combination.
There are also backup/debugging etc. reasons to implement `export` as
well.
This is a better followup to dc9239dd7b
since I wanted to do fsync-less checkouts in rpm-ostree too, and
replicating the "turn off fsync temporarily" was in retrospect just a
hack.
We can simply add a boolean to the checkout options.
https://github.com/GNOME/ostree/pull/172
Originally, a lot of the `fsync()` calls here were added for the
wrong reason - I was chasing a bug that ended up being the extlinux
bootloader not parsing 64 bit ext4 filesystems. But since it looked
like corruption, I tried adding a lot more `fsync()` calls.
All we should have to do is use `syncfs()`. If that doesn't work,
it's a kernel bug.
I'm making this change because skipping the individual fsyncs can be a
major performance win - it's easier for the FS to optimize, we do more
in parallel, etc.
https://bugzilla.gnome.org/show_bug.cgi?id=757117
A fast way to generate new OSTree content using an existing
tree is to checkout (as hard links), add/replace files, then
call `ostree_repo_scan_hardlinks()`, then commit.
But `ostree_repo_scan_hardlinks()` scans the entire repo, which
can be slow if you have a lot of content.
All we really need is a mapping of (device,inode) -> checksum
just for the objects we checked out, then use that mapping
for commits.
This patch adds API so that callers can create a mapping via
`ostree_repo_devino_cache_new()`, then pass it to
`ostree_repo_checkout_tree_at()` which will populate it, and then
`ostree_repo_write_directory_to_mtree()` can consume it.
I plan to use this in rpm-ostree for package layering work.
Notes:
- The old `ostree_repo_scan_hardlinks()` API still works.
- I tweaked the cache to be a set with the checksum colocated with
the key, to avoid a separate malloc block per entry.
https://github.com/GNOME/ostree/pull/167
Extract existing code from ostree_repo_prune and add an argument COMMIT,
that controls which commit purge. If not set, the old behavior is kept.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
If you pass a diriectory it will look for the "superblock" child, otherwise
it will use the file as the superblock. I need this in xdg-app to be able
to install any filename as a bundle.
The new API permits to query a remote repository summary file and
retrieve the list of available refs.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
It allows to specify whether GPG verification for the summary file is
enabled for a specific repository.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Imports one or more GPG keys from a source stream or from the user's
personal keyring into a remote-specific keyring. The keys to import
can optionally be restricted by a list of key IDs.
The imported keys are used to conduct GPG verification when pulling
from the given remote.
An OSTree user noticed that `ostree fsck` would produce `missing
object` errors in the case of interrupted pulls.
It's possible to do e.g. `ostree pull --subpath=/usr/share/rpm ...`,
which gets you just that portion of the commit. The use case for this
was being able to see what changes would appear in an update before
actually downloading all of it.
(I think this would be better covered by static deltas, but those
aren't final yet, and `--subpath` predates it)
Further, `.commitpartial` is used as a successor to the `transaction`
symlink for more precise knowledge in the case where a pull was
interrupted that we needed to resume scanning.
So it makes sense for `ostree fsck` to be aware of it.
rpm-ostree currently uses ostree_repo_checkout_tree(), which as a side
effect will use the uncompressed objects cache by default. This is
rather annoying if you're using rpm-ostree on a server-side
repository, because if you then rsync the repo, you'll be syncing out
the uncompressed objects unless you exclude them.
We added the ability to disable the uncompressed cache in the
repository config to fix this, but it's better to allow application
control over this. The uncompressed cache will in some future version
become opt in as well.
This new API further:
- Drops the `GFile` usage in favor of `openat` APIs
- Improves ergonomics by avoiding callers having to query the source
`GFileInfo` (and carry around a copy of `OSTREE_GIO_FAST_QUERYINFO`)
- Has a more extensible options structure
Per the comment, I rather crudely have the `ostree checkout` builtin
call both APIs to ensure some testing coverage.
However, I'd like to in the future have easier-to-set-up testing code
that calls `libtest.sh` to set up dummy data.
For Anaconda, I needed OSTREE_REPO_REMOTE_CHANGE_ADD_IF_NOT_EXISTS,
with the GFile *sysroot argument to avoid ugly hacks. We want to
write the content provided via "ostreesetup" as a remote to the target
chroot only in the case where it isn't provided as part of the tree
content itself.
This is also potentially useful in idempotent systems management tools
like Ansible.
https://bugzilla.gnome.org/show_bug.cgi?id=741577
We potentially need a lot of argument types for pull. Rather than
have a C function with tons of arguments, let's use a GVariant a{sv}
as a handy extensible (and immutable) bag of properties.
This is prepratory work for adding an option to pull to traverse
history.
https://bugzilla.gnome.org/show_bug.cgi?id=737844
For Anaconda, we have an ugly bootstrapping problem where we need to
add the remote to the repository's config, then do a pull+deploy, then
remove and re-add the config, because /etc/ostree/remotes.d doesn't
exist yet in the target system.
https://bugzilla.gnome.org/show_bug.cgi?id=738698
src/libostree/ostree-repo.c:1759: Warning: OSTree:
ostree_repo_import_object_from: unknown parameter 'checksum' in
documentation comment, should be 'sha256'
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
And use it in pull-local. As one might expect, this is blazingly fast
if they're on the same filesystem.
I'll be using this to "promote" builds between different repositories.
For Fedora and potentially other distributions which use globally
distributed mirrors, metalink is a popular solution to redirect
clients to a dynamic set of mirrors.
In order to make metalink work though, it needs *one* file which can
be checksummed. (Well, potentially we could explode all refs into the
metalink.xml, but that would be a lot more invasive, and a bit weird
as we'd end up checksumming the checksum file).
This commit adds a new command:
$ ostree summary -u
To regenerate the summary file. Can only be run by one process at a
time.
After that's done, the metalink can be generated based on it, and the
client fetch code will parse and load it.
https://bugzilla.gnome.org/show_bug.cgi?id=729585
Changes the pull API to allow pulling only a single directory instead
of the whole deployment. This option is utilized by the check-diff
option in rpm-ostree.
Add a new state directory to hold <checksum>.commitpartial files, so
we know that we've only downloaded partial state.
This patch adds a function that will parse a partial checksum when
resolving a refspec. If the inputted refspec matches a truncated
existing checksum, it will return that checksum to be parsed. If
multiple truncated checksums match the partial refspec, it is not
unique and will return false. This addition is inspired by the same
functionality in Docker, which allows a user to reference a specific
commit without typing the entire checksum.
partial checksums: Add function to abstract comparison
This modifies the list_objects and list_objects_at functions
to take an additional argument for the string that a commit starts
with. If this string arg is not null, it will only list commit
objects beginning with that string. This allows for a new function
ostree_repo_list_commit_objects_starting_with to pass a partial string
and return a list of all matching commits. This improves on the
previous strategy of listing refs because it will list all commit objects,
even ones in past history. This update also includes bugfixes on
error handling and string comparison, and changes the output structure
of resolve_partial_checksum. The new strcuture will no longer return FALSE
without error. Also, the hashtable foreach now uses iter. Also
includes modified test file
There's several use cases for calling into ostree itself to do
mirroring, instead of using bare rsync. For example, it's a bit more
efficient as it doesn't require syncing the objects/ directory.
https://bugzilla.gnome.org/show_bug.cgi?id=728351
The current "transaction" symlink was introduced to fix issues with
interrupted pulls; normally we assume that if we have a metadata
object, we also have all objects to which it refers.
There used to be a "summary" which had all the available refs, but I
deleted it because it wasn't really used, and was still racy despite
the transaction bits.
We still want the pull process to use the transaction link, so don't
delete the APIs, just relax the restriction on object writing, and
introduce a new ostree_repo_set_ref_immediate().
For many OS install scenarios, one runs through an installer which may
come with embedded data, and then the OS is configured post-install to
receive updates.
In this model, it'd be nice to avoid the post-install having to rewrite
the /ostree/repo/config file.
Additionally, it feels weird for admins to interact with "/ostree" -
let's make the system feel more like Unix and have our important
configuration in /etc.
https://bugzilla.gnome.org/show_bug.cgi?id=729343
It's better if this is independent from the OstreeSysroot; for
example, a policy is active in a given deployment root at once, not
for a sysroot globally.
We can also collect SELinux-related API in one place.
Unfortunately at the moment there can be only one instance of this
class per process.
This has a very basic level of functionality (deltas can be generated,
and applied offline). There is only some stubbed out pull code to
fetch them via HTTP.
But, better to commit this now and improve it from a known starting
point, rather than have it languish in a branch.
This will be used by guestmount - it's WAY faster. We only take disks
as a unit, so it's safe. If the process fails halfway through, we
just start over from scratch the next time anyways.
The trees as shipped come with /usr/etc, which should just be labeled
as usr_t. When we do a deployment, we need to relabel the copies of
the files we're making in /etc.
SELinux support is compile and runtime optional.
We can't use #ifdef in the headers, since then g-ir-scanner won't pick
up the functions (unless we included config.h). Let's instead always
have the symbols, but just set an error if we were built without
support for it, just like how pull works.
Several APIs in libostree were moved there from the commandline code,
and have hardcoded g_print() for progress and notifications. This
isn't useful for people who want to write PackageKit backends, custom
GUIs and the like.
From what I can tell, there isn't really a winning precedent in GLib
for progress notifications.
PackageKit has the model where the source has GObject properties that
change as async ops execute, which isn't bad...but I'd like something
a bit more general where say you can have multiple outstanding async
ops and sensibly track their state.
So, OstreeAsyncProgress is basically a threadsafe property bag with a
change notification signal.
Use this new API to move the GSConsole usage (i.e. g_print()) out from
libostree/ and into ostree/.
Add a --generate-sizes option to commit to add size information to the
commit metadata. This will be used by higher level code which wants
to determine the total size necessary for downloading.
This uses gpgv for verification against DATADIR/ostree/pubring.gpg by
default. The keyring can be overridden by specifying OSTREE_GPG_HOME.
Add a unit test for commit signing with gpg key and verifying on pull;
to implement this we ship a test GPG key generated with no password
for Ostree Tester <test@test.com>.
Change all of the existing tests to disable GPG verification.
Add an optional dependency on gpgme to add GPG signatures into the
detached metadata, with the key "ostree.gpgsigs", as an "aay", an
array of signatures (treated as binary data).
The commit command gains a --gpg-sign=<key-id> argument. Also add an
argument --gpg-homedir to set the GPG homedir where we look for
keyrings.
read_commit resolves the ref to a commit, and a lot of consumers want
the resolved commit for their own purposes; this prevents them from
calling resolve_rev themselves.
https://bugzilla.gnome.org/show_bug.cgi?id=707727
We want an OstreeRepoFile to be the way to reference a "filesystem
tree" that's stored in the repo, which is a combination of a DIR_TREE
and a DIR_META. The idea is that once you write an mtree to the repo
using ostree_repo_write_mtree, it becomes serialized and you get an
OstreeRepoFile in return.
Change any APIs that care about DIR_TREE / DIR_META checksums to care
about OstreeRepoFiles instead, which right now is mostly is
ostree_repo_write_commit.
https://bugzilla.gnome.org/show_bug.cgi?id=707727
Previously I thought we'd have to ditch the current commit
format to avoid a{sv} due to
See https://bugzilla.gnome.org/show_bug.cgi?id=673012
But I realized that we don't really have to care about
unpacking/repacking commit objects, so let's just re-expose the
existing metadata a{sv} in commits in the API.
Also, add support for "detached" metadata that can be updated at any
time post-commit. This is specifically designed for GPG signatures.
https://bugzilla.gnome.org/show_bug.cgi?id=707379
Rather than having separate write_ref calls, make clients start a
transaction, add some refs, and then commit it. While this doesn't
make it 100% atomic, it makes it easier for us to use an atomic
model, and it means we don't do as much I/O updating the summary
file and such.
https://bugzilla.gnome.org/show_bug.cgi?id=707644
An earlier version of this API acted like git in that some objects
would be staged in a temporary directory which would be then committed
in one go by moving files around. The API doesn't match most users
expectations though, as while the stage is nice as a high-level API
it isn't really suited for low-level APIs.
While the stage was removed, the APIs were never renamed. Rename
them now so that they match expectations.
https://bugzilla.gnome.org/show_bug.cgi?id=707644
It turns out every builtin (with one special exception) that takes a
repo argument did the same thing; let's just centralize it. The
special exception was "ostree init --repo=foo" where foo is expected
to *not* actually be a repo. In that case, simply skip the
ostree_repo_check() invocation.
https://bugzilla.gnome.org/show_bug.cgi?id=706762
Colin Walters
Giuseppe Scrivano
Yu Qi Zhang
Mathnerd314
Krzesimir Nowak
Jonathan Lebon
Alexander Larsson
Matthew Barnes
Anne LoVerso
Jeremy Whiting
Jasper St. Pierre