The tmp directory is lazily created for each fetcher instance, since
it may require superuser permissions and some instances only need
_ostree_fetcher_request_uri_to_membuf() which keeps everything in
memory buffers.
This is a continuation of earlier work to drop the individual fsync on
files/directories in favor of relying on `syncfs()` for speed.
As part of that cleanup, I'm porting it to be fd-relative.
I feel relatively confident about this change given that this area of
the code has notable test suite coverage, although that code runs as
non-root.
I'm porting the deployment code to be fd-relative, but part of the
logic was using `GFile` to talk to `OstreeRepoFile` to determine the
"bootcsum" (boot config checksum) before checking out the file tree.
We can avoid having both code paths by checking out the tree first,
then looking at it on the filesystem.
Extract existing code from ostree_repo_prune and add an argument COMMIT,
that controls which commit purge. If not set, the old behavior is kept.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Given the previous commit, which isolates SoupSession in a separate
thread, it should be safe to start pushing a temporary main context
for synchronous requests again.
This partially reverts 84fe2ff, which partially reverted 9f3d586.
Related to https://bugzilla.gnome.org/show_bug.cgi?id=753336
Move the SoupSession to a separate thread with its own isolated main
context and main loop. All interaction with the SoupSession occurs
by way of idle sources attached to the session's main context, which
execute on the session's thread.
This should solve the problem of running an asynchronous fetch request
synchronously by pushing a new thread-default main context and iterating
a main loop until the request completes. Prior to this, the new thread-
default main context would interfere with the SoupSession's own async
processing.
A lot of effort here just to avoid touching SoupSession directly in
ostree_fetcher_new(). The reason will become apparent in subsequent
commits.
Note this introduces generated enum/flags GTypes using glib-mkenums.
I could have just made the property type as plain integer, but doing
properties right will henceforth be easier now that the automake-fu
is established.
This way two pulls will not use the same tmpdir and accidentally
overwrite each other. However, consecutive OstreeFetchers will reuse
the tmpdirs, so that we can properly resume downloading large objects.
https://bugzilla.gnome.org/show_bug.cgi?id=757611
Concurrent pulls break since we're sharing the staging directory for
all transactions in the repo. This makes us use a per-transaction directory.
However, in order for resumes to work we first look for existing
staging directories and try to aquire an exclusive lock for them. If
we can't find any staging directory or they are all already locked,
then we create a new one.
https://bugzilla.gnome.org/show_bug.cgi?id=757611
This creates a subdirectory of the tmp dir with a selected prefix,
and takes a lockfile to ensure that nobody else is using the same directory.
However, if a directory with the same prefix already exists and is
not locked that is used instead.
The later is useful if you want to support some kind of resumed operation
on the tmpdir.
touch reused dirs
https://bugzilla.gnome.org/show_bug.cgi?id=757611
ostree_repo_write_commit_with_time() converts the timestamp to
big-endian byte order.
ostree_repo_write_commit() was also doing this when calling
ostree_repo_write_commit_with_time(), resulting in a corrupted
commit object (timestamp bytes were backwards).
Recent regression in 14ffd7022a
Do not delete a .commitmeta file after removing the last metadata entry.
This way a client will pull the empty .commitmeta file and overwrite old
metadata as expected.
https://bugzilla.gnome.org/750459
ostree_checksum_bytes_peek() can return NULL if the checksum has an
incorrect length (most likely from disk corruption) but most callers
are not prepared to handle this and would likely crash.
Use ostree_checksum_bytes_peek_validate() instead, which sets a
GError on an invalid checksum.
This is very useful for the inline-parts case, as you can then include
detached signatures in a single file representing the commit.
It is not as important for the generic pull case, as the detached
metadata is only a single small file. Additionally the detached
metadata is not content referenced and may change after the static
delta file was created, so we need to pull the latest version anyway.
If you pass a diriectory it will look for the "superblock" child, otherwise
it will use the file as the superblock. I need this in xdg-app to be able
to install any filename as a bundle.
Also renames OstreeRepoTrustedContentBareCommit to
OstreeRepoContentBareCommit so that it can be used by both.
This will be needed when we introduce checksum verification of objects
in static deltas.
In this mode the parts are stored in the metadata of the main delta
superblock file. This can be useful if you want a single-file delta
for easy transport, or for http in the case the delta is very small.
When a commit is deleted and the repo is configured to use tombstone
commits, create one. Delete the tombstone file only if the commit is
pulled again.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Add a new object type: OSTREE_OBJECT_TYPE_TOMBSTONE_COMMIT that is
used when a commit was intentionally removed.
If the remote repository doesn't use tombstone commits, do not fail on
a missing commit (change 0b795785dd).
When the remote repository uses tombstones, if a commit cannot be
found, check if the tombstone file is present and fail if it is not
present.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
There might be a race here in that we create new symlink files *after*
calling `syncfs`, and they are not guaranteed to end up on disk.
Rework the code so that we create symlinks before, and then only
rename them after (and `fsync()` the directory for good measure).
Additional-fixes-by: Giuseppe Scrivano <gscrivan@redhat.com>
Tested-by: Giuseppe Scrivano <gscrivan@redhat.com>
This still needs verification that we're fixing a real bug; but I'm
fairly confident this won't make the fsync situation worse.
https://bugzilla.gnome.org/show_bug.cgi?id=755595
Upgrader now looks for an "override-commit" key in the origin file
with a commit checksum, which causes the upgrader to pull and deploy
the specified commit rather than the latest available commit on the
origin refspec.
The current code checks if /boot/uEnv.txt is a symlink to
decice if sysroot requires u-boot support. Why this is bad:
There are 2 ways to provide a custom env to u-boot from user space:
1) A compiled binary that is sourced from u-boot.
2) A text file (usually /uEnv.txt) that is imported into env from u-boot.
The current OSTree u-boot integration code was designed with the 1st
case in mind.
Many bootscripts provided by an embedded device vendors expect
to find uEnv.txt in the top level directory, it is often hardcoded
when building u-boot and is difficult to change later on. Or in other
cases it is stored in read-only memory so changing it would require
re-flushing boot loader with a new env. So the issue here is that
OSTree's and vendor uEnv.txt want to exist and the same path and OSTree
would throw away any changes added to /uEnv.txt by user on the next
upgrade/deploy.
This patch "hides" away the OSTree's env file loader/uEnv.txt from users
who are used to edditing uEnv.txt at the top level directory. Now to add
OSTree support on such boards you can simply add a custom logic in uEnv.txt
that loads ostree env from /loader/uEnv.txt
This change is backward compatible with the previous ostree releases and
solves the issue described in:
https://bugzilla.gnome.org/show_bug.cgi?id=755787
There is already a check that the destination object does not
exist in all other cases when processing an incoming static delta.
However, the bspatch case would still try to run and fail. Add
an analogous check to that case as well.
https://bugzilla.gnome.org/show_bug.cgi?id=756260
zlib can return LZMA_BUF_ERROR, which indicates that either
the input or output buffer has size zero. This case should cause
the correct error to be passed back from g_converter_convert
to expand the relevant buffer. Since this error is ambiguous
as to which buffer is too small, an explicit check on the
output buffer size is added as well.
https://bugzilla.gnome.org/show_bug.cgi?id=756260
I was working on a different test, and ended up being very confused at
the behavior where removing the last deployment didn't remove the last
`ostree/X/X/X` ref pointing to its commit.
There's no reason to special case the last undeployment AFAIK, and the
existing code handles this.
Track outstanding HTTP requests in a table for easier debugging.
Also fixes a bug discussed in https://bugzilla.gnome.org/755224
where the outstanding request counter was not decremented in the
event of an error, which could result in the fetcher hitting its
max request limit and locking up.
The bug is fixed by removing the request struct from the table in
pending_uri_free(), which is always called regardless of error,
so the outstanding request count is always accurate.
Have OstreeFetcherPendingURI be the GTask's task_data and pass the GTask
around in queues and callback closures. The reference counting before
was a little confusing and this helps clarify it, at least to me.
OstreeFetcherPendingURI no longer needs its own reference count.
Matthew Barnes
Colin Walters
Giuseppe Scrivano
Alexander Larsson
John Hiesey
Gatis Paeglis
Leandro Santiago