The fact that we weren't doing this is at best an oversight, and
for some deployment models a security vulnerability. Having both
`gpg-verify` and `gpg-verify-summary` shows that we were intending
them to be orthogonal/independent.
Lately I've been advocating moving towards pinned TLS instead of
gpg-signed summaries, and if we follow that path, performing GPG
verification of commit objects even if using deltas is more important,
as it provides an at-rest verifiable authenticity and integrity
mechanism.
Content providers which are signing their summary files and/or using
TLS (particularly pinned TLS) for transport should treat this as a
nice-to-have. However, for providers which are serving content over
plain HTTP and relying on GPG, this is a critical update.
Closes: https://github.com/ostreedev/ostree/issues/517Closes: #589
Approved by: jlebon
What in the code is called "scanning" is ensuring (potentially
recursively) have an object, and if not, fetching it. And then if
it's metadata, parsing it and finding new objects to fetch.
This logic has grown fairly complex. What I'm trying to fix
right now is that if we're doing a pull-local to a remote repository
via `sshfs` (FUSE) we still end up scanning, which is inefficient.
We can take advantage of the "commitpartial" logic here - if a commit
isn't partial, it's complete, hence we don't need to scan it.
At the same time, I'm changing the logic here to *always* do scans for
dirtree objects. This will fix cases where multiple commits share
dirtree objects. We have "commitpartial" metadata, but no such concept
of partial/complete for dirtrees.
But, we'll only ever scan dirtrees if we scan commits, which is
what the section above fixes.
Closes: https://github.com/ostreedev/ostree/issues/543Closes: #564
Approved by: alexlarsson
Some deployments may want to gate access to content based on things
like OAuth. In this model, the client system would normally compute a
token and pass it to the server via an API.
We could theoretically support this in the remote config too, but
that'd be a bit weird for OAuth as the information is dynamic.
Therefore this cleans up the code a little bit to more clearly handle
the case that the fetcher is initialized from both remote config
data plus pull options.
Closes: #574
Approved by: giuseppe
Optionally read cookie jars for a remote to be used when downloading
data. This can be used for private repositories which require specific
cookies to be present, e.g. repositories hosted on Amazon cloudfront
using signed cookies.
Closes: #531
Approved by: cgwalters
We should just download the commit objects directly, as it's
obviously a lot more efficient than deltas.
I had to generate a summary file in more places in the tests,
since once created, it needs to be updated.
Closes: https://github.com/ostreedev/ostree/issues/528Closes: #566
Approved by: jlebon
If this is true, don't initiate, abort of commit a transaction, instead
it is assumed that the caller initiated the transaction, and that it
will eventually be commited.
This allows you to do multiple pulls or a combination of pulls and
commits in a single transaction.
Closes: #525
Approved by: cgwalters
In https://github.com/ostreedev/ostree/pull/408 we fixed a
bug where we would crash when trying to execute deltas into
an archive repo (which isn't presently supported).
But that was overly aggressive - we obviously *can* execute deltas
when mirroring into a bare repo. This should fix a regression with
the way flatpak uses mirroring to pull from a user repo into the
system.
Closes: #506
Approved by: alexlarsson
While converting the mirrorlist code from using GSList to GPtrArray, I
completely missed the fact that there is now a much cleaner way to do
this.
Closes: #484
Approved by: cgwalters
This commit adds mirrorlist support to the fetcher. Users can now
prepend url or/and contenturl by mirrorlist= to interpret the link as a
mirrorlist.
If an object is not found, the fetcher will automatically try the next
mirror in the order given in the list (assuming the order returned by
the server is significant).
Closes: #469
Approved by: cgwalters
This made sense back when we used a main loop even when we needed to
fetch objects synchronously. Nowadays, we no longer actually update
progress before the FETCHING_OBJECTS phase, which is only for async
requests.
This allows us to get rid of fetch_uri_contents_membuf_sync() and to
generalize fetch_uri_contents_utf8_sync() so that it only requires a
fetcher. This will be needed later.
Closes: #469
Approved by: cgwalters
Allow users to pass a --contenturl during `remote add` and store it in
the remote config.
Fish out the contenturl setting from the remote config and use it when
downloading static deltas and objects (except for commit signatures).
The idea here is that items in the trust chain (summary & sigs) can be
fetched from a more secure e.g. TLS-pinned location, while objects
themselves are fetched from another location. Once mirrorlist support is
added, this use-case will become even more advantageous.
Closes: #469
Approved by: cgwalters
OSTree function ostree_repo_pull_with_options starts a
series of operations that makes heavy use of the PullData's
cancellable.
This isn't effective, however, since nowhere in the code
the OtPullData.cancellable field is set. This is visible,
for example, when trying to cancel a Flatpak pull and nothing
happens, because the cancellable is not properly passed
to the pull data.
Fix that by setting the cancellable field of the pull data. It
owns a reference for safety reasons, and unreferences it at the
end of the operation.
ostreedev/ostree#482
Closes: #483
Approved by: cgwalters
Fixes this warning:
src/libostree/ostree-repo-pull.c:2162: Warning: OSTree: ostree_repo_pull_with_options: unknown parameter 'remote_name_or_baseurl' in documentation comment, should be 'remote_name'
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Closes: #472
Approved by: jlebon
In CentOS, these happened to appear in a repo that is served
via rsync, and having them not be world-readable caused mirroring
tools to fail.
They aren't secret, so don't make them so.
Closes: #468
Approved by: giuseppe
We used to only look at the "gpg-verify" and "gpg-verify-summary"
options when we're passed a local URL. Make these options also have an
effect in the configured remote case.
Closes: #465
Approved by: cgwalters
For local pulls there's no benefit pulling the static delta over the
individual object files since there's no HTTP overhead. Furthermore,
processing deltas always generates the objects whereas a standard pull
ensures that the exact object files are copied. Using deltas also
prevents hardlinking the objects if the repos exist on the same
filesystem.
Closes: #447Closes: #448
Approved by: cgwalters
We have a better API now, drop use of the internal helper, which also
depended on libgsystem.
This required bumping libglnx to pull in a fix.
Closes: #429
Approved by: giuseppe
In general this is even cleaner now, though it was better after I
extracted a helper function for the "write tempfile with contents"
bits that were shared between metadata and regular file codepaths.
Closes: #369
Approved by: jlebon
We don't presently support this, since the static delta code assumes
it can just `mmap()` file objects. We could at some point implement
this, but for now just skip executing deltas when doing
`archive -> archive` mirroring.
I noticed this when trying to mirror a repo in Jenkins in
[CAHC](https://wiki.centos.org/SpecialInterestGroup/Atomic/Devel).
Closes: #408
Approved by: giuseppe
This should fix the memory leaks in #352
This is a subset of the changes, the other part is in my pull code rewrite
Closes: #382
Approved by: cgwalters
G_DEFINE_AUTOPTR_CLEANUP_FUNC is a new function in GLib 2.44, but
libglnx contains a backported version of it. A few source files were
however using G_DEFINE_AUTOPTR_CLEANUP_FUNC either without including
libglnx.h, or without including it early enough.
This fix is similar to the one in commit d368624.
Closes#376Closes: #377
Approved by: smcv
This changes around a few things that didn't work for me:
* Section names seem to be ostree-* instead of libostree-*
* Also XML files are ostree-* (they didn't show up at all)
- gtk-doc doesn't seem to parse const _OSTREE_PUBLIC correctly
* pull documentation is now on the actual functions rather than stubs
* Update gitignore with some more files
And there some changes to make gtk-doc give fewer warnings (not finished)
Closes: #327
Approved by: cgwalters
This centralizes the ifdef's in one file, which will make it
easier to write new pull backends.
ostree-repo-pull.c is now built unconditionally
Closes: #327
Approved by: cgwalters
Various places need to include libglnx.h for the autoptr backport
fallbacks to be there before ostree-autocleanups.h is included.
This fixes the build on centos7·
Closes: #309
Approved by: giuseppe
This tries to avoid leaking GVariantBuilders and GVariants in some
situations. The leaks were usually happening when some error occurred
or because of unclear variant ownership situation.
The former is mostly about making sure that g_variant_builder_clear is
called on builders that didn't finish their variant building process.
The latter is surely more work - sometimes the result of
g_variant_builder_end() should not be passed directly to a function,
but rather stored in a g_autoptr(GVariant), sunk and then passed to a
function. IMO, with an advent of g_autoptr, GVariants should be always
sunk instead of relying on some receiver function sinking it. This
would make an easy-to-follow policy of always sinking your
variants. Functions could then assume that the passed variant is
already sunk. These leaks are still happenning in commands, but they
are less harmful, since that code will not be used by some daemon as a
library routine.
Closes: #291
Approved by: cgwalters
I think it's cleaner if we use `remote_repo_local` to know
that we have a local repo. In reality, it might be nicest
if we didn't even create an `OstreeFetcher` for this case,
but untangling the code is tricky.
Closes: #239
Approved by: alexlarsson
Otherwise we get undefined behaviour if the client didn't explicitly set
any flags.
Also, add documentation for all the other options supported by
ostree_repo_pull_with_options().
Closes: #252
Approved by: cgwalters
These are useful for ostree users (like xdg-app) that have custom
options for remotes. In particular they are useful when we later make them
all respect self->parent_repo.
Closes: #236
Approved by: cgwalters
Alexander Larsson
Colin Walters
Sjoerd Simons
Simon McVittie
Jonathan Lebon
Georges Basile Stavracas Neto
Giuseppe Scrivano
Dan Nicholson
Mathnerd314
Bastien Nocera
Krzesimir Nowak