⚠️ There is a notable spiked pit trap here around
`posix_fallocate()` and `errno`. This has bit other projects,
see e.g.
7bb87460e6
Otherwise the port was straightforward.
It used to be allowed to run something like "ostree remote refs" on
a read-only (e.g. system) repo. However, the summary cache caused that to
break. This commit just makes it not save the cache if we get some kind
of permission error when writing it. It'll still work, even without the
cache.
https://bugzilla.gnome.org/show_bug.cgi?id=763855
It allows an optimization to skip the download of the summary file
if its .sig file is unchanged.
Downloading the .sig file is much cheaper than downloading the summary
file from repositories with many branches.
https://bugzilla.gnome.org/show_bug.cgi?id=762973
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
I plan to use this in rpm-ostree at least for two reasons:
- To find the mtime on the repo
- To use the tmp/ directory to stage content (but we should eventually
add a better API)
I don't know why we didn't do this a long time ago. This extends the
pull API to allow grabbing a specific commit, and will set the branch
to it. There's some support for this in the deploy engine, but there
are a lot of reasons to support it for raw pulls (such as subset
mirroring cases).
In fact I'm thinking we should also have the override-version logic
here too.
NOTE: One thing I debated here is inventing a new syntax on the
command line. Git doesn't seem to have this functionality (probably
because it'd be rarely used). The '@' character at least doesn't
conflict with anything.
Anyways, I wanted this for some other test cases. Without this,
writing tests that go between different commits is more awkward as one
must generate the content in one repo, then pull downstream, then
generate more content, then pull again. But now I can just keep track
of commit IDs and do exactly what I want without synchronizing the
tests.
I'd like to incrementally convert all of `ostree-repo*.c` to
fd-relative usage, so that we can sanely introduce
`ostree_repo_new_at()` which doesn't involve GFile.
This one is medium risk, but passes the test suite.
Originally, a lot of the `fsync()` calls here were added for the
wrong reason - I was chasing a bug that ended up being the extlinux
bootloader not parsing 64 bit ext4 filesystems. But since it looked
like corruption, I tried adding a lot more `fsync()` calls.
All we should have to do is use `syncfs()`. If that doesn't work,
it's a kernel bug.
I'm making this change because skipping the individual fsyncs can be a
major performance win - it's easier for the FS to optimize, we do more
in parallel, etc.
https://bugzilla.gnome.org/show_bug.cgi?id=757117
The tmp directory is lazily created for each fetcher instance, since
it may require superuser permissions and some instances only need
_ostree_fetcher_request_uri_to_membuf() which keeps everything in
memory buffers.
This way two pulls will not use the same tmpdir and accidentally
overwrite each other. However, consecutive OstreeFetchers will reuse
the tmpdirs, so that we can properly resume downloading large objects.
https://bugzilla.gnome.org/show_bug.cgi?id=757611
Concurrent pulls break since we're sharing the staging directory for
all transactions in the repo. This makes us use a per-transaction directory.
However, in order for resumes to work we first look for existing
staging directories and try to aquire an exclusive lock for them. If
we can't find any staging directory or they are all already locked,
then we create a new one.
https://bugzilla.gnome.org/show_bug.cgi?id=757611
This creates a subdirectory of the tmp dir with a selected prefix,
and takes a lockfile to ensure that nobody else is using the same directory.
However, if a directory with the same prefix already exists and is
not locked that is used instead.
The later is useful if you want to support some kind of resumed operation
on the tmpdir.
touch reused dirs
https://bugzilla.gnome.org/show_bug.cgi?id=757611
When a commit is deleted and the repo is configured to use tombstone
commits, create one. Delete the tombstone file only if the commit is
pulled again.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Had a rare situation where I had no libsoup development files, so I
took the opportunity to fix the build errors. Ugly, but works now.
Would be nice if libsoup could be a hard dependency since we rarely
ever test a configuration without it.
xdg-app was hanging for me with v2015.8, but worked with v2015.7.
I narrowed things down to the GMainLoop/context commit, in which
we started pushing a temporary main context for synchronous
requests internally.
That's never really going to work with libsoup - there needs
to be a single main context which works on the socket. Furthermore,
clients couldn't get progress messages that way.
For *other* internal uses where we added APIs that talk to the remote
repo, we cleanly push a temporary main context.
(Note that I kind of snuck in a change here around the GError handling
in pulls that isn't strictly related but came up in testing)
First of all, what we were doing with having GMainLoop in the internal
APIs is wrong. Synchronous APIs should always create their own main
context and not iterate the caller's. Doing the latter creates
potential for evil reentrancy issues. Sync API should block, async
API is for not blocking.
Now that's out of the way, fix the pull code to do the clean
```
while (termination_condition (state))
g_main_context_iteration (mainctx, TRUE);
```
model for looping. This is a lot easier to understand and ultimately
more reliable than having other code call `g_main_loop_quit()`, as the
loop condition is in exactly one place.
We can also remove the idle source which only fired once.
Note we have to add a hack here to discard the synchronous session and
create a new one which we only use async.
https://bugzilla.gnome.org/show_bug.cgi?id=753336
It allows to specify whether GPG verification for the summary file is
enabled for a specific repository.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
The global keyring directory (trusted.gpg.d) is deprecated. Only use it
when a specified remote does NOT have its own keyring, or when verifying
local repository objects.
Note, because mixing in the global keyring directory is now an explicit
choice, OstreeGpgVerifier no longer needs to implement GInitableIface.
If a remote keyring does not already exist, create an empty pubring.gpg
file in the temporary directory prior to importing keys. This prevents
gpg2 from creating a pubring.kbx file in the new keybox format [1]. We
want to stay with the older keyring format since its performances issues
are not relevant here.
[1] https://gnupg.org/faq/whats-new-in-2.1.html#keybox
External daemons like rpm-ostree want push notification any time a
change is made by an external entity. inotify provides notification,
but a problem is there's no easy way to monitor all of the refs.
In the past, there has been discussion of opt-in recursive timestamps:
https://lkml.org/lkml/2013/4/5/307
But in today's world, let's just bump the mtime on the repo itself, as
a central inotify point.
Closes: https://github.com/GNOME/ostree/pull/111
Imports one or more GPG keys from a source stream or from the user's
personal keyring into a remote-specific keyring. The keys to import
can optionally be restricted by a list of key IDs.
The imported keys are used to conduct GPG verification when pulling
from the given remote.
Colin Walters
Alexander Larsson
Giuseppe Scrivano
Matthew Barnes
John Hiesey
Jeff Ortel