If pull is interrupted, we may have downloaded an arbitrary subset of
the requested objects. Previously, we handled this by scanning for
all objects each time.
However, there's an easy optimization - this patch creates a lock file
in the repo. If we don't see that file when starting a pull, we know
we don't need to stat() every file; presence of a dirtree object for
example implies the existence of everything it references.
This is an incompatible change to archive-z, thus it is now renamed to
archive-z2 and ostree will no longer parse archive-z.
I noticed in perf that we were spending some time zlib-decompressing
file headers, which is just inefficient. Rather than do this, keep
the headers uncompressed, and just zlib-compress content.
This gives us something closer to the advantages of archive and
archive-z when using the latter. Concretely we get deduplication
among multiple checkouts, along with the "devino" hash table trick
during commits to avoid checksumming content again.
This is enabled by default.
For similar reasons as metadata, this avoids having the main thread
blocked in fdatasync(), and even better - we can achieve much higher
parallelism if we have multiple threads blocked on fdatasync().
This is where loose content objects are stored as one compressed file,
instead of the two separate ones for regular archive mode. This mode
would be suitable for HTTP servers, beause only one HTTP request is
necessary, and the result would be compressed.
Cleanly separate metadata/content APIs, rather than defaulting to
raw streams. This helps most use cases.
Also, drop support for staging content without knowing the total
length. This complicated the code, and for things like streaming
HTTP, we should be able to figure this out from Content-Length.
They're not a large efficiency win at the moment, because we don't
do any delta compression.
At the moment, they simply served to compress data, but we will change
the archive mode to do that by default.
This can be a large performance win in certain circumstances:
* Cold buffer cache (we don't block the whole process)
* Requiring a copy instead of hardlink
This will allow us to use hard links again for user-mode checkouts,
rather than the hackish link cache. It was pretty silly anyways to
have file objects be stored with just a small metadata header
prepended, but uncompressed.
Either they should be hardlinkable, or compressed (in pack files).
Rather than passing xattr/file_info for all objects, change the API to
assume we're passing the defined object stream for each type. Namely,
for OSTREE_OBJECT_TYPE_FILE, we're now giving the "archive file" data.
This significantly cleans up the code for committing to archive mode
repositories, at the cost of having to (at present) create an
intermediate temporary file when committing to raw repositories.
This will be useful for ostbuild; a user can create their own archive
mode repository which transparently inherits objects from the
root-owned one in /ostree.
This is a convenient way to have a lookaside directory of hard links,
which can greatly speed up checkouts. In the future we probably want
to push this down into the repository.
Previously we had the "staged" state to ensure we didn't add a commit
object without the associated dirtree, etc. However it's
easier/better to just ensure in the pull command that we have all
referenced objects.
Also change pull to download metadata first. This will allow adding
a progress bar later.
Expose the lower-level functionality in libostree, change checkout
builtin to be a higher level driver. This will allow us to more
easily improve the "checkout" builtin..
Before we were creating randomly-named temporary files in repo/tmp
when downloading via pull, but that means if the download process is
interrupted, we have to redownload everything again.
Let's still keep the concept of a "transaction" where files are
stored in the repository as atomically as possible (i.e. we
do a bunch of rename() calls), but now we also have an explicit
"tmp/pending/objects" directory that contains named objects.
This allows us to then skip redownloading things that are pending.
The builder wants the ability to mark a given file as e.g. setuid. To
implement this, the repo now has a callback-based API when importing a
directory to modify or remove items.
The commit tool accepts a "statoverride" file as input which looks like:
+mode /path/to/file
The tar files we're making of artifacts don't include parent
directories. Now we could change the builder to make them, but we can
also just autocreate them on import. Mode 0755 with no xattrs seems
OK here.
Rather than offering high level "commit directory", instead perform
operations on a mtree. Commits are treated more like regular objects.
Change the commit builtin to drive this all at a lower level.
ostbuild will generate two artifacts: foo-runtime.tar.gz and
foo-devel.tar.gz in the general case. When committing to the devel
tree, it'd be lame (i.e. slower and not atomic) to have to commit
twice.
This will allow us to have hardlink checkouts of archives. A key use
case here is an archive repo of an OS (with root-owned files etc.)
where we want to do builds in a user tree.
A positive side effect of doing things this way is that now the SHA256
checksums for a given file should be identical regardless of whether
it's stored in an archive or bare repository.
Colin Walters