This is a noticeable cleanup, and fixes another big user of GFile* in
performance/security sensitive codepaths.
I'm specifically making this change because the static deltas code was
leaking temporary files, and cleaning that up nicely would be best if
we were fd relative.
You create these with something like:
ostree static-delta generate --empty --to=master
These will be automatically used during pull if no previous revision
exists in the target repo.
These work very much like the normal static deltas except they
are named just by the "to" revision. I.e:
deltas/94/f7d2dc23759dd21f9bd01e6705a8fdf98f90cad3e0109ba3f6c091c1a3774d
for a from-scratch to 94f7d2dc23759dd21f9bd01e6705a8fdf98f90cad3e0109ba3f6c091c1a3774d delta.
https://bugzilla.gnome.org/show_bug.cgi?id=721799
ostree_repo_pull_with_options() needs this, and I'd rather keep the
OstreeRemote struct definition tucked away in ostree-repo.c with its
own internal API.
Rename _ostree_fetcher_contents_membuf_sync to
ostree_fetcher_request_uri_to_membuf and drop unused argument
user_data.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
src/libostree/ostree-repo-pull.c:1676:22: warning: 'flags' may be used uninitialized in this function [-Wmaybe-uninitialized]
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
We potentially need a lot of argument types for pull. Rather than
have a C function with tons of arguments, let's use a GVariant a{sv}
as a handy extensible (and immutable) bag of properties.
This is prepratory work for adding an option to pull to traverse
history.
https://bugzilla.gnome.org/show_bug.cgi?id=737844
Now that we have a summary file, we can use it to allow a simple:
ostree pull --mirror
To download the latest commit on every branch. Also, for a case I'm
dealing with there's only one branch, but I don't want mirror users to
have to hardcode it.
https://bugzilla.gnome.org/show_bug.cgi?id=737807
For Fedora and potentially other distributions which use globally
distributed mirrors, metalink is a popular solution to redirect
clients to a dynamic set of mirrors.
In order to make metalink work though, it needs *one* file which can
be checksummed. (Well, potentially we could explode all refs into the
metalink.xml, but that would be a lot more invasive, and a bit weird
as we'd end up checksumming the checksum file).
This commit adds a new command:
$ ostree summary -u
To regenerate the summary file. Can only be run by one process at a
time.
After that's done, the metalink can be generated based on it, and the
client fetch code will parse and load it.
https://bugzilla.gnome.org/show_bug.cgi?id=729585
Changes the pull API to allow pulling only a single directory instead
of the whole deployment. This option is utilized by the check-diff
option in rpm-ostree.
Add a new state directory to hold <checksum>.commitpartial files, so
we know that we've only downloaded partial state.
Some organizations will want to use private Certificate Authorities to
serve content to their clients. While it's possible to add the CA
to the system-wide CA store, that has two drawbacks:
1) Compromise of that cert means it can be used for other web traffic
2) All of ca-certificates is trusted
This patch allows a much stronger scenario where *only* the CAs in
tls-ca-path are used for verification from the given repository.
https://bugzilla.gnome.org/show_bug.cgi?id=726256
The generic GKeyFile error isn't quite informative enough here.
I hit this with the new compose process where we don't automatically
inject a configured remote into the generated disk images; we expect
people to add them.
https://bugzilla.gnome.org/show_bug.cgi?id=731346
There's several use cases for calling into ostree itself to do
mirroring, instead of using bare rsync. For example, it's a bit more
efficient as it doesn't require syncing the objects/ directory.
https://bugzilla.gnome.org/show_bug.cgi?id=728351
If fetching GPG-signed commits over plain HTTP, a MitM attacker can
fill up the drive of targets by simply returning an enormous stream
for the commit object.
Related to this, an attacker can also cause OSTree to perform large
memory allocations by returning enormous GVariants in the metadata.
This helps close that attack by limiting all metadata objects to 10
MiB, so the initial fetch will be truncated.
But now the attack is only slightly more difficult as the attacker
will have to return a correctly formed commit object, then return a
large stream of < 10 MiB dirmeta/dirtree objects.
https://bugzilla.gnome.org/show_bug.cgi?id=725921
For the static deltas work, we're using the already-extant internal
API to perform a HTTP fetch for optional data - static deltas are
optional.
Except that we didn't correctly unset the error if we were doing an
optional fetch and the data wasn't found.
We shouldn't g_print() from a library, particularly when the
expectation is that the client has an async progress set up.
This should fix the pull output extending the status line.
If a MITM attacker (or just network corruption) causes a temporary
downloaded object in tmp/ to be corrupted, we'll end up
continually trying to commit it, and fail.
Fix this unlinking the temp file immediately after opening it. This
will ensure that if we exit due to an error (or crash), the kernel
will clean up the space for us.
https://bugzilla.gnome.org/show_bug.cgi?id=725924
Mixing async and threads has proved to be too much for my little mind.
It has race conditions that I've tried repeatedly to fix, but failed.
The threading here was scanning metadata objects - and there are
two parts to that:
1) Physically loading them from disk
2) Parsing them
Now #1 has been partially addressed by avoiding a storm of lstat() if
we're starting from a known working state. If pull gets interrupted,
then we do need to rescan all objects. Also, we can address this with
local metadata packfiles.
The other potentially slow bit is that we recurse across the metadata,
blocking the main thread. We could ameliorate that in the future by
scheduling metadata parsing as idle "chunks".
Anyways, let's move the needle back to reliability, and readd speed
more carefully.
https://bugzilla.gnome.org/show_bug.cgi?id=706456
This has a very basic level of functionality (deltas can be generated,
and applied offline). There is only some stubbed out pull code to
fetch them via HTTP.
But, better to commit this now and improve it from a known starting
point, rather than have it languish in a branch.
Colin Walters
Alexander Larsson
Matthew Barnes
Giuseppe Scrivano
Anne LoVerso
James Antill