ostree_repo_write_commit_with_time() converts the timestamp to
big-endian byte order.
ostree_repo_write_commit() was also doing this when calling
ostree_repo_write_commit_with_time(), resulting in a corrupted
commit object (timestamp bytes were backwards).
Recent regression in 14ffd7022a
Just to make copy-and-paste a little easier, as I often use this command
immediately before rebasing.
e.g.
# ostree remote refs fedora-atomic
fedora-atomic:fedora-atomic/f23/x86_64/docker-host
fedora-atomic:fedora-atomic/f23/x86_64/testing/docker-host
^^^^^^^^^^^^^^ (this part is new)
# rpm-ostree rebase fedora-atomic:fedora-atomic/f23/x86_64/testing/docker-host
Do not delete a .commitmeta file after removing the last metadata entry.
This way a client will pull the empty .commitmeta file and overwrite old
metadata as expected.
https://bugzilla.gnome.org/750459
ostree_checksum_bytes_peek() can return NULL if the checksum has an
incorrect length (most likely from disk corruption) but most callers
are not prepared to handle this and would likely crash.
Use ostree_checksum_bytes_peek_validate() instead, which sets a
GError on an invalid checksum.
It may have a different meaning, and the usage screen is not helpful.
Print the usage screen only when the command is not found.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
This is very useful for the inline-parts case, as you can then include
detached signatures in a single file representing the commit.
It is not as important for the generic pull case, as the detached
metadata is only a single small file. Additionally the detached
metadata is not content referenced and may change after the static
delta file was created, so we need to pull the latest version anyway.
If you pass a diriectory it will look for the "superblock" child, otherwise
it will use the file as the superblock. I need this in xdg-app to be able
to install any filename as a bundle.
Also renames OstreeRepoTrustedContentBareCommit to
OstreeRepoContentBareCommit so that it can be used by both.
This will be needed when we introduce checksum verification of objects
in static deltas.
In this mode the parts are stored in the metadata of the main delta
superblock file. This can be useful if you want a single-file delta
for easy transport, or for http in the case the delta is very small.
When a commit is deleted and the repo is configured to use tombstone
commits, create one. Delete the tombstone file only if the commit is
pulled again.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Add a new object type: OSTREE_OBJECT_TYPE_TOMBSTONE_COMMIT that is
used when a commit was intentionally removed.
If the remote repository doesn't use tombstone commits, do not fail on
a missing commit (change 0b795785dd).
When the remote repository uses tombstones, if a commit cannot be
found, check if the tombstone file is present and fail if it is not
present.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
There might be a race here in that we create new symlink files *after*
calling `syncfs`, and they are not guaranteed to end up on disk.
Rework the code so that we create symlinks before, and then only
rename them after (and `fsync()` the directory for good measure).
Additional-fixes-by: Giuseppe Scrivano <gscrivan@redhat.com>
Tested-by: Giuseppe Scrivano <gscrivan@redhat.com>
This still needs verification that we're fixing a real bug; but I'm
fairly confident this won't make the fsync situation worse.
https://bugzilla.gnome.org/show_bug.cgi?id=755595
Adds an entry to the origin file to force the OstreeSysrootUpgrader to
pull and deploy the given checksum.
[origin]
override-commit=CHECKSUM
If the option is not given, any such entry is explicitly removed from
the origin file to ensure we upgrade to the latest available commit.
Upgrader now looks for an "override-commit" key in the origin file
with a commit checksum, which causes the upgrader to pull and deploy
the specified commit rather than the latest available commit on the
origin refspec.
The current code checks if /boot/uEnv.txt is a symlink to
decice if sysroot requires u-boot support. Why this is bad:
There are 2 ways to provide a custom env to u-boot from user space:
1) A compiled binary that is sourced from u-boot.
2) A text file (usually /uEnv.txt) that is imported into env from u-boot.
The current OSTree u-boot integration code was designed with the 1st
case in mind.
Many bootscripts provided by an embedded device vendors expect
to find uEnv.txt in the top level directory, it is often hardcoded
when building u-boot and is difficult to change later on. Or in other
cases it is stored in read-only memory so changing it would require
re-flushing boot loader with a new env. So the issue here is that
OSTree's and vendor uEnv.txt want to exist and the same path and OSTree
would throw away any changes added to /uEnv.txt by user on the next
upgrade/deploy.
This patch "hides" away the OSTree's env file loader/uEnv.txt from users
who are used to edditing uEnv.txt at the top level directory. Now to add
OSTree support on such boards you can simply add a custom logic in uEnv.txt
that loads ostree env from /loader/uEnv.txt
This change is backward compatible with the previous ostree releases and
solves the issue described in:
https://bugzilla.gnome.org/show_bug.cgi?id=755787
Using `commit_subject` instead of `arg` is clearer as it can refer to
a directory, archive or ref.
This is just an aesthetic change in the source code, having no impact
anywhere else.
There is already a check that the destination object does not
exist in all other cases when processing an incoming static delta.
However, the bspatch case would still try to run and fail. Add
an analogous check to that case as well.
https://bugzilla.gnome.org/show_bug.cgi?id=756260
zlib can return LZMA_BUF_ERROR, which indicates that either
the input or output buffer has size zero. This case should cause
the correct error to be passed back from g_converter_convert
to expand the relevant buffer. Since this error is ambiguous
as to which buffer is too small, an explicit check on the
output buffer size is added as well.
https://bugzilla.gnome.org/show_bug.cgi?id=756260
I was working on a different test, and ended up being very confused at
the behavior where removing the last deployment didn't remove the last
`ostree/X/X/X` ref pointing to its commit.
There's no reason to special case the last undeployment AFAIK, and the
existing code handles this.
Track outstanding HTTP requests in a table for easier debugging.
Also fixes a bug discussed in https://bugzilla.gnome.org/755224
where the outstanding request counter was not decremented in the
event of an error, which could result in the fetcher hitting its
max request limit and locking up.
The bug is fixed by removing the request struct from the table in
pending_uri_free(), which is always called regardless of error,
so the outstanding request count is always accurate.
Have OstreeFetcherPendingURI be the GTask's task_data and pass the GTask
around in queues and callback closures. The reference counting before
was a little confusing and this helps clarify it, at least to me.
OstreeFetcherPendingURI no longer needs its own reference count.
Had a rare situation where I had no libsoup development files, so I
took the opportunity to fix the build errors. Ugly, but works now.
Would be nice if libsoup could be a hard dependency since we rarely
ever test a configuration without it.
To support deploying older commits:
ostree pull <remote> <checksum>
ostree admin deploy <checksum>
Prior to this, the deploy command garbage collected <checksum> since
there's no ref pointing to it, and then ostree_sysroot_deploy_tree()
fails because it can't find the <checksum> commit.
https://bugzilla.gnome.org/732526
New public function works like ostree_sysroot_cleanup() EXCEPT FOR
pruning the repository.
Under the hood, add _ostree_sysroot_piecemeal_cleanup() which takes
flags to better control what files are cleaned up. Both public cleanup
functions are now wrappers for _ostree_sysroot_piecemeal_cleanup() with
different flags.
xdg-app was hanging for me with v2015.8, but worked with v2015.7.
I narrowed things down to the GMainLoop/context commit, in which
we started pushing a temporary main context for synchronous
requests internally.
That's never really going to work with libsoup - there needs
to be a single main context which works on the socket. Furthermore,
clients couldn't get progress messages that way.
For *other* internal uses where we added APIs that talk to the remote
repo, we cleanly push a temporary main context.
(Note that I kind of snuck in a change here around the GError handling
in pulls that isn't strictly related but came up in testing)
I noticed xdg-app was looping trying to fetch 1427 refs. We
don't want to do that unless asked to.
(And also, we need to make static delta requests async)
There's no reason to keep them hidden. I have a hard policy that
OSTree should *not* be used to carry secrets. Things like host ssh
private keys should be set up out of band by an OS-external
configuration mechanism such as kickstart, cloud-init, etc.
We also assume that hiding binaries is not very useful as most
attackers would be able to find them on the Internet or (for
subscribed content) acting as a customer.
This fixes a bug with mirroring because we changed to take the
unmodified upstream objects rather than uncompress <-> recompress.
https://bugzilla.gnome.org/show_bug.cgi?id=748959
Now that the computed similar objects are all regular files,
get_unpacked_unlinked_content should never be called on any other
object type. Assert that this is true instead of silently succeeding.
_ostree_delta_compute_similar_objects should not output symlinks.
Previously, a symlink in the "from" commit could be matched to a
real file in the "to" commit, since nothing was filtering symlinks
on the "from" side. This led to failures running the bzdiff
algorithm.
On systems with slow disks, the recursive scanning of directories can
be expensive -- it takes upwards of 2 minutes on our systems. This can
block the main loop for such a long time that it allows the download to
time out...
As such, move all the scanning of objects to a queue, processed from
an idle, to make sure that we don't block the main loop when scanning.
https://bugzilla.gnome.org/show_bug.cgi?id=753336
First of all, what we were doing with having GMainLoop in the internal
APIs is wrong. Synchronous APIs should always create their own main
context and not iterate the caller's. Doing the latter creates
potential for evil reentrancy issues. Sync API should block, async
API is for not blocking.
Now that's out of the way, fix the pull code to do the clean
```
while (termination_condition (state))
g_main_context_iteration (mainctx, TRUE);
```
model for looping. This is a lot easier to understand and ultimately
more reliable than having other code call `g_main_loop_quit()`, as the
loop condition is in exactly one place.
We can also remove the idle source which only fired once.
Note we have to add a hack here to discard the synchronous session and
create a new one which we only use async.
https://bugzilla.gnome.org/show_bug.cgi?id=753336
ostree_repo_prepare_transaction() should always be matched with a call
to either ostree_repo_commit_transaction() or
ostree_repo_abort_transaction().
Since ostree_repo_pull_with_options() does not call
ostree_repo_abort_transaction() on errors, the OstreeRepo instance will
hit an assertion when it's re-used later for another attempt, such as
when the update is driven by an external component through libostree and
network temporarily goes down.
This commit simply always calls ostree_repo_abort_transaction() in the
exit path of ostree_repo_pull_with_options(), since the function is safe
to call even when we're not in a transaction, and that matches e.g. what
ostree-sysroot-cleanup.c does.
The new API permits to query a remote repository summary file and
retrieve the list of available refs.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Returns a GFile for the default system root, which is usually the root
directory unless overridden by the OSTREE_SYSROOT environment variable
(which is mainly intended for testing).
libsoup will cache sessions, so it might be the case that we get a
reused session when pulling from the same repo multiple times in one
process.
In this case we were leaking signal connections, which caused
callbacks into freed memory with bad consequences.
Fix it by tying the signal connection to the object lifetime.
I did a quick audit pass through the pull code. What I focused on the
most is the case where `gpg-verify-summary=true`, and in particular
where `gpg-verify=false` too. This should be a valid and secure
configuration.
The primary change here is to error out very quickly if either
`summary` or `summary.sig` are 404. Previously, we'd only error out
if we were processing deltas.
Expand the existing test case to cover this, plus invalid summary and
invalid sig. (The test case was failing with current git master too).
It allows to specify whether GPG verification for the summary file is
enabled for a specific repository.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Works like "ostree refs" but fetches refs from a remote repo.
This depends on the remote repo having a summary file, but any repo
being served over HTTP *ought* to have one.
This may not be the best idea for general usage, but the only use case
for metalinks currently is fetching a summary file and those are pretty
small. Far more convenient to return the file content in a GBytes.
Matthew Barnes
Giuseppe Scrivano
Alexander Larsson
Colin Walters
John Hiesey
Daniel Drake
Gatis Paeglis
Leandro Santiago
Jasper St. Pierre
Cosimo Cecchi