It allows an optimization to skip the download of the summary file
if its .sig file is unchanged.
Downloading the .sig file is much cheaper than downloading the summary
file from repositories with many branches.
https://bugzilla.gnome.org/show_bug.cgi?id=762973
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
This will allow daemons like rpm-ostree to detect if there are any new
deployments efficiently, in combination with using inotify. If there
are any changes, rpm-ostree wants publish them on DBus.
While we're here, add some changes to start doing unit C testing of
the sysroot API.
And change the command line to use it. rpm-ostree had a copy
of this code, and thus there's a clear reason to have an API.
While we're moving this into API, ensure the mtime on deploy is bumped
after an osname is created, so that daemons like rpm-ostree can notice
changes. (In reality, creating the directory should do this, but
let's be double sure)
This allows other processes (e.g. rpm-ostreed) to monitor for external
changes (e.g. if someone does `ostree admin undeploy`) in a relatively
sane fashion.
Specifically, I'm trying to fix:
https://github.com/projectatomic/rpm-ostree/issues/220
It accepts a `flags` argument to control its behavior. Differently
from `ostree_repo_list_refs`, the `refspec_prefix` is not removed from
the results.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
I plan to use this in rpm-ostree at least for two reasons:
- To find the mtime on the repo
- To use the tmp/ directory to stage content (but we should eventually
add a better API)
As rpm-ostree evolves, it keeps driving API additions to libostree.
This creates a relatively tight coupling.
However, if delivering via e.g. RPM, unless one manually remembers to
increment the `Requires:` in the spec file, it's possible for the two
to become desynchronized.
RPM handles versioned symbols and will ensure a dependency if the
application starts using a newer version.
To implement this, switch to `-fvisibility=hidden`, along with an
annotation in the header, and finally add a `.sym` file.
This matches what other projects like systemd and libvirt do.
Although rather than attempting to retroactively version symbols, glom
them all onto the current one.
I see when analyzing a delta here that due to byteswapping a negative
compression ratio of 540%, 66%, and 28%. Let's arbitrarily pick 20%
as a threshold for detecting byetswapping.
If the average object size is greater than 4GiB, let's assume we're
dealing with opposite endianness. I'm fairly confident no one is
going to be shipping peta- or exa- byte size ostree deltas, period.
Past the gigabyte scale you really want bittorrent or something.
xdg-app passed this a filename directly, and in this case it should be
used as is. This regressed to always look for "superblock" in the same
directory as the passed in filename.
https://bugzilla.gnome.org/show_bug.cgi?id=762617
Some Docker layers are just metadata in the `layer.json`. If one is
mapping Docker layers to OSTree commits, one needs to create a dummy
root directory, because OSTree doesn't support metadata-only commits.
Let's just push that logic down here because it's easier than special
casing it in higher levels.
One of the design goals with deltas was not just wire efficiency,
but also having all the data up front about how much data would
be transferred before starting.
Let's expose that better by adding a `dry-run` option to the pull API.
This requires static deltas to be useful. Basically we simply call
the progress callback once with the data from the superblock.
For a production release repository, most OS vendors would want
to just always use static deltas. Add the ability for the pulls to
require it.
(I think I'll also add a summary key for this actually in addition,
so the repo manager can force it too)
If ostree is run in a test setup where it operates as root in a tmp
directory, it might cause issues to flag the deployments as immutable.
The test harness might simply be doing an `rm -rf` (effectively the case
for gnome-desktop-testing-runner), which will then fail.
We add a new debug option to the ostree_sysroot object using GLib's
GDebugKey functionality to allow our tests to communicate to ostree that
we don't want immutable deployments.
This is a more flexible version of the previous
ostree_repo_write_archive_to_mtree() which took a file reference.
This has an extensible options structure, and in particular
now supports `ignore_unsupported_content`.
I plan to use this for importing Docker images which contain device
nodes. (There's no reason for container images to have those, so
we'll just ignore them).
Also here, just like the export variant, the caller is responsible for
setting up libarchive.
I was going to add new API for importing, and it was really confusing
that what I think of now as import and export both had "write" in the
name. It's just clearer to talk about the direction.
At the same time, include `Export` in the options structure.
This isn't an ABI break as the API isn't in a release.
I was getting a compilation error with the GCC hardening flags which
look for a missing mode with `O_CREAT`. The right fix here is to drop
`O_CREAT`, as truncate() should throw `ENOENT` if the file doesn't
exist.
I don't know why we didn't do this a long time ago. This extends the
pull API to allow grabbing a specific commit, and will set the branch
to it. There's some support for this in the deploy engine, but there
are a lot of reasons to support it for raw pulls (such as subset
mirroring cases).
In fact I'm thinking we should also have the override-version logic
here too.
NOTE: One thing I debated here is inventing a new syntax on the
command line. Git doesn't seem to have this functionality (probably
because it'd be rarely used). The '@' character at least doesn't
conflict with anything.
Anyways, I wanted this for some other test cases. Without this,
writing tests that go between different commits is more awkward as one
must generate the content in one repo, then pull downstream, then
generate more content, then pull again. But now I can just keep track
of commit IDs and do exactly what I want without synchronizing the
tests.
At the moment I'm looking at using rpm-ostree to manage RPM inputs
which can then be converted into Docker images. It's most convenient
if we can stream directly out of libostree rather than doing a
checkout + tar combination.
There are also backup/debugging etc. reasons to implement `export` as
well.
While it's not strictly tied to OSTree, let's move
https://github.com/cgwalters/rofiles-fuse in here because:
- It's *very* useful in concert with OSTree
- It's tiny
- We can reuse OSTree's test, documentation, etc. infrastructure
One thing to consider also is that at some point we could experiment
with writing a FUSE filesystem for OSTree. This could internalize a
better equivalent of `--link-checkout-speedup`, but on the other hand,
the cost of walking filesystem trees for these types of operations is
really quite small.
But if we did decide to do more FUSE things in OSTree, this is a step
towards that too.
Now we display stats on the individual parts, such as the blob size
and the number of each type of opcode. Most interesting to me is
things like how many bsdiff opcodes there are vs new objects, etc.
We had code to deal with opening/checksumming/decompressing static
deltas in a few places. I'd like to teach `ostree static-delta show`
how to display more information, and this will allow it to just use
`_ostree_static_delta_part_open()` too.
Right now though, almost all of the details of deltas are private, so
we can't do the "honest thing" and have the command line just use the
shared library.
Eventually some of this should appear in the API, but for now add
command line which is useful for debugging.
I'd like to incrementally convert all of `ostree-repo*.c` to
fd-relative usage, so that we can sanely introduce
`ostree_repo_new_at()` which doesn't involve GFile.
This one is medium risk, but passes the test suite.
The original intention here was that we'd keey around a copy of the
file so that grub2 could eventually learn how to do atomic updates by
checking for a "fully written" marker in the *new* file, and if it
didn't exist, falling back to grub2.cfg.old.
I haven't yet proposed that upstream, but we might as well stop
deleting the file since it's useful as a backup at least.
Reported-by: Gatis Paeglis
This is a better followup to dc9239dd7b
since I wanted to do fsync-less checkouts in rpm-ostree too, and
replicating the "turn off fsync temporarily" was in retrospect just a
hack.
We can simply add a boolean to the checkout options.
https://github.com/GNOME/ostree/pull/172
Originally, a lot of the `fsync()` calls here were added for the
wrong reason - I was chasing a bug that ended up being the extlinux
bootloader not parsing 64 bit ext4 filesystems. But since it looked
like corruption, I tried adding a lot more `fsync()` calls.
All we should have to do is use `syncfs()`. If that doesn't work,
it's a kernel bug.
I'm making this change because skipping the individual fsyncs can be a
major performance win - it's easier for the FS to optimize, we do more
in parallel, etc.
https://bugzilla.gnome.org/show_bug.cgi?id=757117
A fast way to generate new OSTree content using an existing
tree is to checkout (as hard links), add/replace files, then
call `ostree_repo_scan_hardlinks()`, then commit.
But `ostree_repo_scan_hardlinks()` scans the entire repo, which
can be slow if you have a lot of content.
All we really need is a mapping of (device,inode) -> checksum
just for the objects we checked out, then use that mapping
for commits.
This patch adds API so that callers can create a mapping via
`ostree_repo_devino_cache_new()`, then pass it to
`ostree_repo_checkout_tree_at()` which will populate it, and then
`ostree_repo_write_directory_to_mtree()` can consume it.
I plan to use this in rpm-ostree for package layering work.
Notes:
- The old `ostree_repo_scan_hardlinks()` API still works.
- I tweaked the cache to be a set with the checksum colocated with
the key, to avoid a separate malloc block per entry.
https://github.com/GNOME/ostree/pull/167
The logic for checking which bootversion to use tries to access
sysroot->bootversion if the user didn't specify an explicit bootversion
on the command-line nor through the env var. However, at that point, the
sysroot object is not yet initialized, so it will always return 0, even
when it's 1.
This would cause e.g. `grub2-mkconfig` to have no output for the BLS
entries whenever the entries were under `/boot/loader.1`.
Related: RHBZ1293986
The tmp directory is lazily created for each fetcher instance, since
it may require superuser permissions and some instances only need
_ostree_fetcher_request_uri_to_membuf() which keeps everything in
memory buffers.
This is a continuation of earlier work to drop the individual fsync on
files/directories in favor of relying on `syncfs()` for speed.
As part of that cleanup, I'm porting it to be fd-relative.
I feel relatively confident about this change given that this area of
the code has notable test suite coverage, although that code runs as
non-root.
I'm porting the deployment code to be fd-relative, but part of the
logic was using `GFile` to talk to `OstreeRepoFile` to determine the
"bootcsum" (boot config checksum) before checking out the file tree.
We can avoid having both code paths by checking out the tree first,
then looking at it on the filesystem.
Extract existing code from ostree_repo_prune and add an argument COMMIT,
that controls which commit purge. If not set, the old behavior is kept.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Downloads and prints a remote summary file and any signatures in an
easy-to-read format, or alternatively with the --raw option, prints
the summary GVariant data directly.
https://bugzilla.gnome.org/show_bug.cgi?id=759250
Given the previous commit, which isolates SoupSession in a separate
thread, it should be safe to start pushing a temporary main context
for synchronous requests again.
This partially reverts 84fe2ff, which partially reverted 9f3d586.
Related to https://bugzilla.gnome.org/show_bug.cgi?id=753336
Move the SoupSession to a separate thread with its own isolated main
context and main loop. All interaction with the SoupSession occurs
by way of idle sources attached to the session's main context, which
execute on the session's thread.
This should solve the problem of running an asynchronous fetch request
synchronously by pushing a new thread-default main context and iterating
a main loop until the request completes. Prior to this, the new thread-
default main context would interfere with the SoupSession's own async
processing.
A lot of effort here just to avoid touching SoupSession directly in
ostree_fetcher_new(). The reason will become apparent in subsequent
commits.
Note this introduces generated enum/flags GTypes using glib-mkenums.
I could have just made the property type as plain integer, but doing
properties right will henceforth be easier now that the automake-fu
is established.
This way two pulls will not use the same tmpdir and accidentally
overwrite each other. However, consecutive OstreeFetchers will reuse
the tmpdirs, so that we can properly resume downloading large objects.
https://bugzilla.gnome.org/show_bug.cgi?id=757611
Concurrent pulls break since we're sharing the staging directory for
all transactions in the repo. This makes us use a per-transaction directory.
However, in order for resumes to work we first look for existing
staging directories and try to aquire an exclusive lock for them. If
we can't find any staging directory or they are all already locked,
then we create a new one.
https://bugzilla.gnome.org/show_bug.cgi?id=757611
This creates a subdirectory of the tmp dir with a selected prefix,
and takes a lockfile to ensure that nobody else is using the same directory.
However, if a directory with the same prefix already exists and is
not locked that is used instead.
The later is useful if you want to support some kind of resumed operation
on the tmpdir.
touch reused dirs
https://bugzilla.gnome.org/show_bug.cgi?id=757611
Bison is a well known external dependency, so just require it.
Including the generated content in git means it may or may not
be regenerated based randomly on timestamps, etc.
Also use `$(AM_V_GEN)` so we get prettier output.
Previously we were just ignoring this, which hid a bug in
an earlier commit that generated them.
Also change the `commit` program to use both APIs - this
involves extra code, but not too much.
This way, reverting the fix with this on top caused the test suite to
fail. Adding an active test for this would need a custom test program
using the C API, or adding a cmdline flag to the client, neither of
which quite seemed worth it.
Use the parse-datetime module from gnulib, and adapt it to not require
other modules as portability is not really an issue for us.
DATE can be specified in different formats, such as: "-1 week", "last
monday", "1 week ago".
Include the generated .c file in the repository so to not add another
dependency to Bison.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
ostree_repo_write_commit_with_time() converts the timestamp to
big-endian byte order.
ostree_repo_write_commit() was also doing this when calling
ostree_repo_write_commit_with_time(), resulting in a corrupted
commit object (timestamp bytes were backwards).
Recent regression in 14ffd7022a
Just to make copy-and-paste a little easier, as I often use this command
immediately before rebasing.
e.g.
# ostree remote refs fedora-atomic
fedora-atomic:fedora-atomic/f23/x86_64/docker-host
fedora-atomic:fedora-atomic/f23/x86_64/testing/docker-host
^^^^^^^^^^^^^^ (this part is new)
# rpm-ostree rebase fedora-atomic:fedora-atomic/f23/x86_64/testing/docker-host
Do not delete a .commitmeta file after removing the last metadata entry.
This way a client will pull the empty .commitmeta file and overwrite old
metadata as expected.
https://bugzilla.gnome.org/750459
ostree_checksum_bytes_peek() can return NULL if the checksum has an
incorrect length (most likely from disk corruption) but most callers
are not prepared to handle this and would likely crash.
Use ostree_checksum_bytes_peek_validate() instead, which sets a
GError on an invalid checksum.
It may have a different meaning, and the usage screen is not helpful.
Print the usage screen only when the command is not found.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
This is very useful for the inline-parts case, as you can then include
detached signatures in a single file representing the commit.
It is not as important for the generic pull case, as the detached
metadata is only a single small file. Additionally the detached
metadata is not content referenced and may change after the static
delta file was created, so we need to pull the latest version anyway.
If you pass a diriectory it will look for the "superblock" child, otherwise
it will use the file as the superblock. I need this in xdg-app to be able
to install any filename as a bundle.
Also renames OstreeRepoTrustedContentBareCommit to
OstreeRepoContentBareCommit so that it can be used by both.
This will be needed when we introduce checksum verification of objects
in static deltas.
In this mode the parts are stored in the metadata of the main delta
superblock file. This can be useful if you want a single-file delta
for easy transport, or for http in the case the delta is very small.
Giuseppe Scrivano
Colin Walters
Alexander Larsson
Jonathan Lebon
Matthew Barnes