This way two pulls will not use the same tmpdir and accidentally
overwrite each other. However, consecutive OstreeFetchers will reuse
the tmpdirs, so that we can properly resume downloading large objects.
https://bugzilla.gnome.org/show_bug.cgi?id=757611
ostree_checksum_bytes_peek() can return NULL if the checksum has an
incorrect length (most likely from disk corruption) but most callers
are not prepared to handle this and would likely crash.
Use ostree_checksum_bytes_peek_validate() instead, which sets a
GError on an invalid checksum.
This is very useful for the inline-parts case, as you can then include
detached signatures in a single file representing the commit.
It is not as important for the generic pull case, as the detached
metadata is only a single small file. Additionally the detached
metadata is not content referenced and may change after the static
delta file was created, so we need to pull the latest version anyway.
In this mode the parts are stored in the metadata of the main delta
superblock file. This can be useful if you want a single-file delta
for easy transport, or for http in the case the delta is very small.
Add a new object type: OSTREE_OBJECT_TYPE_TOMBSTONE_COMMIT that is
used when a commit was intentionally removed.
If the remote repository doesn't use tombstone commits, do not fail on
a missing commit (change 0b795785dd).
When the remote repository uses tombstones, if a commit cannot be
found, check if the tombstone file is present and fail if it is not
present.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
xdg-app was hanging for me with v2015.8, but worked with v2015.7.
I narrowed things down to the GMainLoop/context commit, in which
we started pushing a temporary main context for synchronous
requests internally.
That's never really going to work with libsoup - there needs
to be a single main context which works on the socket. Furthermore,
clients couldn't get progress messages that way.
For *other* internal uses where we added APIs that talk to the remote
repo, we cleanly push a temporary main context.
(Note that I kind of snuck in a change here around the GError handling
in pulls that isn't strictly related but came up in testing)
I noticed xdg-app was looping trying to fetch 1427 refs. We
don't want to do that unless asked to.
(And also, we need to make static delta requests async)
On systems with slow disks, the recursive scanning of directories can
be expensive -- it takes upwards of 2 minutes on our systems. This can
block the main loop for such a long time that it allows the download to
time out...
As such, move all the scanning of objects to a queue, processed from
an idle, to make sure that we don't block the main loop when scanning.
https://bugzilla.gnome.org/show_bug.cgi?id=753336
First of all, what we were doing with having GMainLoop in the internal
APIs is wrong. Synchronous APIs should always create their own main
context and not iterate the caller's. Doing the latter creates
potential for evil reentrancy issues. Sync API should block, async
API is for not blocking.
Now that's out of the way, fix the pull code to do the clean
```
while (termination_condition (state))
g_main_context_iteration (mainctx, TRUE);
```
model for looping. This is a lot easier to understand and ultimately
more reliable than having other code call `g_main_loop_quit()`, as the
loop condition is in exactly one place.
We can also remove the idle source which only fired once.
Note we have to add a hack here to discard the synchronous session and
create a new one which we only use async.
https://bugzilla.gnome.org/show_bug.cgi?id=753336
ostree_repo_prepare_transaction() should always be matched with a call
to either ostree_repo_commit_transaction() or
ostree_repo_abort_transaction().
Since ostree_repo_pull_with_options() does not call
ostree_repo_abort_transaction() on errors, the OstreeRepo instance will
hit an assertion when it's re-used later for another attempt, such as
when the update is driven by an external component through libostree and
network temporarily goes down.
This commit simply always calls ostree_repo_abort_transaction() in the
exit path of ostree_repo_pull_with_options(), since the function is safe
to call even when we're not in a transaction, and that matches e.g. what
ostree-sysroot-cleanup.c does.
I did a quick audit pass through the pull code. What I focused on the
most is the case where `gpg-verify-summary=true`, and in particular
where `gpg-verify=false` too. This should be a valid and secure
configuration.
The primary change here is to error out very quickly if either
`summary` or `summary.sig` are 404. Previously, we'd only error out
if we were processing deltas.
Expand the existing test case to cover this, plus invalid summary and
invalid sig. (The test case was failing with current git master too).
This may not be the best idea for general usage, but the only use case
for metalinks currently is fetching a summary file and those are pretty
small. Far more convenient to return the file content in a GBytes.
We need to check that it's 'ay'. Also reuse the existing validation
function to check it's 32 bytes rather than potentially crashing with
assertion.
Just noticed this during a code review.
An OSTree user noticed that `ostree fsck` would produce `missing
object` errors in the case of interrupted pulls.
It's possible to do e.g. `ostree pull --subpath=/usr/share/rpm ...`,
which gets you just that portion of the commit. The use case for this
was being able to see what changes would appear in an update before
actually downloading all of it.
(I think this would be better covered by static deltas, but those
aren't final yet, and `--subpath` predates it)
Further, `.commitpartial` is used as a successor to the `transaction`
symlink for more precise knowledge in the case where a pull was
interrupted that we needed to resume scanning.
So it makes sense for `ostree fsck` to be aware of it.
First, this is just a general continuation of the `GFile -> openat`
transition.
Second, it's preparatory work for fsck to gain awareness of partial
commits.
Always request detached metadata for commit objects, even if we already
have the commit object. This ensures we fetch any post facto detached
metadata updates such as new GPG signatures.
https://bugzilla.gnome.org/748220
Emitted during a pull operation upon GPG verification (if enabled).
Applications can connect to this signal to output the verification
results if desired.
Similar to c2b01ad. For some reason I was thinking the commit data
still needed to be written to disk prior to verifying, but it's just
another artifact of spawning gpgv2 (predates using GPGME).
Makes for a nice cleanup in fetch_metadata_to_verify_delta_superblock()
as well.
In anticipation of API enhancements for GPG signature verification, which
would otherwise require a non-functional stub version were GPGME excluded.
GPGME is a pretty lightweight dependency, and the motivation to exclude
it is not clear.
Alexander Larsson
Matthew Barnes
Giuseppe Scrivano
Colin Walters
Jasper St. Pierre
Cosimo Cecchi
John Hiesey