[Originally from bubblewrap commits c5c999a7 "tests: test --userns"
and 3e5fe1bf "tests: Better error message if assert_files_equal fails";
separated into this commit by Simon McVittie.]
The cache shouldn't be affected by the user passing in some other
summary as it may not be the "official one".
I ran into this in flatpak where the passed summary was correct, but
the re-saving of the cache updated the mtime of the cached file which
led to later http If-Modified-Since calls failing to update.
I got:
src/libostree/ostree-repo.c:5232: Warning: OSTree: ostree_repo_gpg_sign_data: unknown parameter 'out_signature' in documentation comment, should be 'out_signatures'
If we have a commit id for all the refs we're pulling, and if we
don't need the summary to list all the refs when mirroring then the
only reason to download the summary is for the list of deltas.
With the new "indexed-deltas" property in the config file (and mirrored
to the summary file) we can detect when we don't need the summary for
deltas and completely avoid downloading it then.
Clients can use these during pull and avoid downloading the summary if
needed, or use the indexed-deltas instead of relying on the ones in
the summary which may be left out.
It is useful to be able to trigger this without having to regenerate
the summary. For example, if you are not using summaries, or ar generating
the summaries yourself.
When we update the summary file (and its list of deltas) we also update
all delta indexes. The index format is a single `a{sv}` variant identical
to the metadata-part of the summary with (currently) only the
`ostree.static-deltas` key.
Since we expect most delta indexes to change rarely, we avoid
unnecessary writes when reindexing. New indexes are compared to
existing ones and only the changed ones are written to disk. This
avoids unnecessary write load and mtime changes on the repo server.
This gets the subpath for a delta index file, which is of the form
"delta-indexes/$commit.index", that contains all the deltas going
to the particular commit.
This was only used in one place, and (especially with the simplification
with GMainContextPopDefault) and the one caller doesn't really do
much more than call the helper. Additionally, what little it does (saving
the result in the cache) is inherently tied to how the helper work,
and will become even more so when we support summary indexes.
This is a preparatory cleanup for supporting summary indexes. It
doesn't change any behaviour and passes make check on its own.
This loads and makes a digest for a delta superblock. The previous
code was used when generating the deltas section in the summary
file. This changes nothing, but is in preparation for using similar
formats in a separate delta index file.
The change in cbf1aca1d5c08d2f40832d16670484ba878d95fb actually
only mmaps the signature file, not the summary. This change makes
use mmap both, as well as extract the cache loading into a helper
function that we will later use in more places.
ostree_repo_list_static_delta_names() tried to validate that
any second-level directory element was a directory, but there was
a cut-and-paste issue, and it used `dent->d_type` instead
of `sub_dent->d_type`.
This fixes the code, but all old ostree versions will break if
there are non-directories in a subdirectory of the deltas directory
in the repo, so be wary.
copy_option() unnecessarily passed ownership of the value
to g_variant_dict_insert_value, but that already refs, so it was leaked.
Closes: #1702
Approved by: cgwalters
We need to have the g_auto(GLnxDirFdIterator) inside the loop, or
we don't correctly clean up when iterating several times.
Closes: #1700
Approved by: cgwalters
Stopping on the first error is nice if you just want to know if everything is ok, but
if you want to figure out all that is wrong its nice to be able to continue and
print all corruptions.
Closes: #1591
Approved by: cgwalters
This means we can later use various operations to heal the repository
because ostree does not assume all objects are there.
This the begining of a fix for https://github.com/ostreedev/ostree/pull/345Closes: #1533
Approved by: cgwalters
This is a version of ostree_repo_traverse_commit_union that also
remembers where the objects came from, by recording the parent
relationships in a hashtable. This can be used to later find which
commits each object was from, which we want to use in fsck.
Closes: #1533
Approved by: cgwalters
In case of some kind of race or other weirdness we might be getting
non-matching versions of summary.sig and summary, where summary.sig
is the latest version. Currently we're saving them to the cache
directly after downloading them successfully, but they will then fail
to gpg validate. Then on the next run we'll keep using the cached files
even if they are incorrect, until summary.sig changes upstream.
This changes the order so that we verify the signatures before saving
to the cache, thus ensuring that we don't end up in a stuck state.
Fixes https://github.com/ostreedev/ostree/issues/1523Closes: #1529
Approved by: cgwalters
In _try_clone_from_payload_link, don't try to do the clone in the
parent repo, because we don't want to modify that. parent repos are
typically used when you want a shared, immutable base.
For example in flatpak, the parent repo is the system repo which you
don't have write access to, so any modification to it will fail with
EACCES, making it impossible to install via the system helper.
Closes: #1524
Approved by: cgwalters
We use utimens instead of utime, thus allowing nanosecond timestamps,
and also fixes a bug where we used to passed UTIME_OMIT to tv_nsec
which made the entire operation a no-op.
Closes: #1351
Approved by: cgwalters
Directly when we allocate a new part we finish the old one,
writing the compressed data to a temporary file and generating
the delta header for it.
When all these are done we loop over them and collect the headers,
sizes and either copy the tempfile data into the inlined superblock
or link the tempfiles to disk with the proper names.
Closes: #1309
Approved by: cgwalters
This allows us to create the final delta desciptor directly on disk
rather than having it all in memory. This is nice because it can
become quite large if inlined parts are used.
Note however, that we currently generate all the delta parts in
memory before adding them to the delta, so we still keep all individual
parts in memory. Fixing that is the next step.
Closes: #1309
Approved by: cgwalters
This is similar to GVariantBuilder in that it constructs variant
containers, but it writes it directly to a file descriptor rather
than keep the entier thing in memory. This is useful to create large
variants without using a lot of memory.
Closes: #1309
Approved by: cgwalters
If you lchown("symlink") then we were incorrectly trying to chown the
symlink target, rather than the symlink itself. In particular, this cause
cp -a to fail for a broken symlink. Additionally, it was using the
symlink target when verifying writability, rather than the symlink
itself.
To fix this, we need pass AT_SYMLINK_NOFOLLOW in these cases.
In general, the kernel itself will always resolve any symlinks for us
before calling into the fuse backend, so we should really never do any
symlink following in the fuse fs itself. So, we pro-actively add
NOFOLLOW flags to a few other places:
truncate:
In reality this will never be hit, because
the kernel will resolve symlinks before calling us.
access:
It seems the current fuse implementation never calls this
(faccessat w/AT_SYMLINK_NOFOLLOW never reaches the fuse fs)
but if this ever is implemented this is the correct behaviour.
We would ideally do `chmod` but this is not implemented on current kernels.
Because we're not multi-threaded, this is OK anyways.
Further, our write verification wasn't correctly handling the case of hardlinked
symlinks, which can occur for `bare` checkouts but *not* `bare-user` which the
tests were using. Change to `bare` mode to verify that.
Closes: #1137
Approved by: alexlarsson
Flatpak make check is failing when applying a static delta
to a bare-user-only repo due to an assert. The fix is to add
bare-user-only to the assert check.
Closes: #940
Approved by: giuseppe
We're hitting this in flathub, where we have a bunch of local builds,
but we also mirror a few refs from the gnome runtime repo into it.
Its fixable by re-doing the summary, but for a short time the
wrong version is visible.
Fixes https://github.com/ostreedev/ostree/issues/846Closes: #935
Approved by: cgwalters
Its often the case that we want to look at objects inside a commit,
before the objects the transaction is finished. For instance:
https://github.com/flatpak/flatpak/pull/837
Which tries to verify the file permissions before committing the
transaction.
And:
https://github.com/flatpak/flatpak/commit/1e5ffa926a25acb655af7889b679b140bf44870b
Which collects the storage size of the objects so that we can
put the total download size in the commit metadata.
I tried to find all the places where we did reads from the
object directories, and in particular this fixes:
- `ostree_repo_load_file()` for `bare` repos (`archive` was already working).
- `ostree_repo_query_object_storage_size()`
- Applying deltas that reference not-yet-commited objects
Closes: #916
Approved by: cgwalters
When a transaction is finished and we have moved all the staged loose
objects into the repo we fsync all the object directory, to ensure the
filenames are stable before we update the refs files to point to the
new commits.
With out this an unclean shutdown after the transaction is finished
could result in a refs file that points to an incomplete commit.
https://bugzilla.gnome.org/show_bug.cgi?id=759442Closes: #918
Approved by: cgwalters
The summary file can get large, but it compresses well (something
which is not true of other files in the ostree repo which are
already compressed). By sending Accept-Encoding: gzip (and
handling the compressed results) we send a lot less data.
I set up the flathub repo (http://flathub.org/repo) to enable
gzip for the summary file (only), and the result is that the
331514 byte large summary was transferred in 122889 bytes.
On my (fast) network this decreased the time i took to do
"flatpak remote-ls flathub" by about 100msec.
This fixes https://github.com/ostreedev/ostree/issues/802Closes: #882
Approved by: cgwalters
This is somewhat complicated by such repos only properly supporting
some subset of file metadata (uid/gid 0, etc). We fix this by
always commiting with filters that make it work.
Closes: #750
Approved by: cgwalters
This adds to file permission masks the same bitmask that will
be applied to file objects in bare-user* repos. This will be
needed in the testsuite to ensure that the things we commit
will be expressable in bare-user-only repos.
Closes: #750
Approved by: cgwalters
This mode is similar to bare-user, but does not store the permission,
ownership (uid/gid) and xattrs in an xattr on the file objects in the
repo. Additionally it stores symlinks as symlinks rather than as
regular files+xattrs, like the bare mode. The later is needed because
we can't store the is-symlink in the xattr.
This means that some metadata is lost, such as the uid. When reading a
repo like this we always report uid, gid as 0, and no xattrs, so
unless this is true in the commit the resulting repository will
not fsck correctly.
However, it the main usecase of the repository is to check out with
--user-mode, then no information is lost, and the repository can
work on filesystems without xattrs (such as tmpfs).
Closes: #750
Approved by: cgwalters
These allow us to avoid copying a lot of data around
in userspace. Instead we splice the data directly from
the fd to the destination fd.
Closes: #684
Approved by: cgwalters
When doing commit --tree=ref=XXX while at the same time applying some
form of modifier, ostree dies trying to read the xattrs using the
raw syscalls. We fix this by falling back to ostree_repo_file_get_xattrs()
in this case.
Also adds a testcase for this.
Closes: #577
Approved by: cgwalters
If we have a partial commit it is not an error for a dirmeta to be
missing (in fact, that is likely), so instead of returning a not-found
error from ostree_repo_traverse_commit() we ignore the error and
continue.
In particular, this means we don't stop early at the first
missing dirmeta, which previously caused ostree_repo_prune() to
thing the dirmetas after that to be unreached and thus purged.
Also, we remove the special casing in ostree_repo_prune() to
not report errors for commitpartial, because these should not
be reported anymore.
This fixes https://github.com/ostreedev/ostree/issues/541Closes: #542
Approved by: cgwalters
If there is a transaction active, then we put writes to detached
metadata into the staging dir, and when reading it we look there
first. This allows transactions to be aborted half-way without
writing the detached metadata into the repository (possibly
overwriting any old metadata from there).
This fixes https://github.com/ostreedev/ostree/issues/526Closes: #539
Approved by: giuseppe
If the detached metadata is not in the repo, try in the parent
repo if that is set.
Without this a commit will not gpg validate in the child repo
Closes: #539
Approved by: giuseppe
Commit 1d4f1b8878 started using hardlinks
checkouts of symlinks. However, symlinks are not stored as symlink in the
repo for bare-user repos, so this breaks user-mode checkouts of such repos.
We fix this by checking for !is_symlink in the bare-user case.
This fixes:
https://github.com/ostreedev/ostree/issues/537Closes: #538
Approved by: giuseppe
If this is true, don't initiate, abort of commit a transaction, instead
it is assumed that the caller initiated the transaction, and that it
will eventually be commited.
This allows you to do multiple pulls or a combination of pulls and
commits in a single transaction.
Closes: #525
Approved by: cgwalters
In flatpak i was using a parent repo, and it failed to update
with ENOENT when dispatching an set-read-source opcode, because the
object it referenced was in the parent repo.
This fixes that by making _ostree_repo_read_bare_fd look
at parent_repo.
Closes: #362
Approved by: cgwalters
Various places need to include libglnx.h for the autoptr backport
fallbacks to be there before ostree-autocleanups.h is included.
This fixes the build on centos7·
Closes: #309
Approved by: giuseppe
In the case we have a repo with a parent, and the child repo has a
remote called "foo", but some option is unset. Then when we look up
the parent repo for a value before using the default we will fail due
to the parent not having the "foo" remote. As soon as we find the
requested remote at some point in the hierarchy we need to ignore further
errors and use the default value.
Closes: #274
Approved by: giuseppe
We looked for and locked old temporary directories so we can
reuse them if not in use. However, once we found one that
we can reuse we didn't stop iterating, and eventually we
reached the end. This means we can lock multiple dirs.
Closes: #273
Approved by: giuseppe
This lets you set a prefix for the resulting archive patsh.
Especially useful in combination with --subpath, for instance
--subpath=subdir --prefix=subdir to extract just subdir.
Closes: #265
Approved by: cgwalters
This allows you to replace the default
$sysroot/$sysconfdir/ostree/repos.d string value, and to use a similar
feature for repos that are not the system repo.
In particular, this allows us to support /etc/xdg-app/remotes.d for
xdg-app.
Closes: #247
Approved by: cgwalters
in write_directory_content_to_mtree_internal dfd_iter can be NULL,
for instance if commiting from --tree=ref=FOO. Don't blindly de-ref
it to avoid crashing.
Closes: #256
Approved by: cgwalters
ostree pull-local crashed for me in thread_closure_unref () doing:
g_mutex_clear (&thread_closure->output_stream_set_lock);
Seems like we never initialize this mutex.
Closes: #254
Approved by: cgwalters
Alexander Larsson