At least in all Linux kernels up to today, one can never `link()` across
devices, so we might as well verify that up front. This will help for a future
patch to add a new type of union-add checkout, since Linux checks for `EEXIST`
before `EXDEV`.
Closes: #714
Approved by: jlebon
The previous logic for static deltas was to use as a FROM
revision the current branch tip. However, we want
to support deltas between branches in an automatic
fashion.
If a summary file is available, we already have an
enumerated list of deltas - so the logic introduced
here is to search it, and find the newest commit
we have locally that matches the TO revision target.
This builds on some thoughts from
https://github.com/ostreedev/ostree/pull/151#issuecomment-232390232
Closes: https://github.com/ostreedev/ostree/pull/151Closes: #710
Approved by: giuseppe
In https://github.com/ostreedev/ostree/pull/408, we disabled the use of
static deltas when mirroring. Later,
https://github.com/ostreedev/ostree/pull/506 loosened this up again so
that we could use static deltas when mirroring into bare{-user} repos.
However, the issue which originally spurrred #408 is even more generic
than that: we want to avoid static deltas for any archive repo, not just
when doing a mirror pull. This patch tightens this up, and also
relocates the decision code to make it easier to read.
Closes: #715
Approved by: cgwalters
The C API (ostree_repo_static_delta_generate) knows what to do
with it, but this parameter was never exposed via command line
tool.
Closes: https://github.com/ostreedev/ostree/issues/695Closes: #703
Approved by: jlebon
This makes it easier to script downloading updates in the background,
and only do deployments just before rebooting.
Partially addresses https://github.com/ostreedev/ostree/issues/640Closes: #642
Approved by: jlebon
I learned today that `docker version` does this and I really like
the idea. While we have the patient open, also add the gitrev
with code taken from https://github.com/projectatomic/rpm-ostree/pull/584Closes: #691
Approved by: giuseppe
I think I commented this out while debugging something, and forgot to re-enable
it. Reading the log files should be a better again after this.
Closes: #699
Approved by: giuseppe
It's just simpler, and I'm not sure people are going to care
much about the difference by default.
We already folded in the fallback sizes into the download totals, so folding in
the count makes things consistent; previously you could see e.g.
`3/3 parts, 100MB/150MB` and be confused.
Closes: #678
Approved by: giuseppe
For rpm-ostree, we already link to libcurl indirectly via librepo, and
only having one HTTP library in process makes sense.
Further, libcurl is (I think) more popular in the embedded space. It
also supports HTTP/2.0 today, which is a *very* nice to have for OSTree.
This seems to be working fairly well for me in my local testing, but it's
obviously brand new nontrivial code, so it's going to need some soak time.
The ugliest part of this is having to vendor in the soup-url code. With
Oxidation we could follow the path of Firefox and use the
[Servo URL parser](https://github.com/servo/rust-url). Having to redo
cookie parsing also sucked, and that would also be a good oxidation target.
But that's for the future.
Closes: #641
Approved by: jlebon
As OSTree has evolved over time, the tests grew with it. We
didn't start out with static deltas or a summary file, and the
tests reflect this.
What I really want to do is change more of the pull tests, from
corruption/proxying/mirroring etc. to use this more realistic
repo rather than the tiny one the other test creates.
We start by using some of the code from `test-pull-many.sh`, and change that
test to use `--disable-static-deltas` for pull, since the point of that test is
to *not* test deltas.
Still TODO is investigate changing other tests to use this.
Closes: #658
Approved by: jlebon
We weren't running it before. Also I switched it to use GLib. Preparation for
some oxidation work (having an implementation of bupsplit in Rust).
I exported another function to do the raw rollsum operation which is what this
test suite uses.
Closes: #655
Approved by: jlebon
This would be more likely to tickle things like
https://github.com/ostreedev/ostree/issues/601
reliably.
Also, while working on the curl backend, I hit on the fact that curl doesn't
queue (by default, you can enable) and will happily create 20000+ concurrent TCP
connections if you try. Having this test would have made that more likely to
fail.
Closes: #650
Approved by: giuseppe
There are use cases for having a single repo with branches
with different lifecycles; a simple example of what I was
trying to do in CentOS Atomic Host work is have "stable"
and "devel" branches, were we want to prune devel, but
retain *all* of stable.
This patch is split into two parts - first we add a low level "delete all
objects not in this set" API, and change the current prune API
to use this.
Next, we move more logic into the "ostree prune" command. This paves the way for
demonstrating how more sophisticated algorithms/logic could be developed outside
of the ostree core.
Also, the --keep-younger-than logic already lived in the commandline, so it
makes sense to keep extending it there.
Closes: https://github.com/ostreedev/ostree/issues/604Closes: #646
Approved by: jlebon
Debian's Lintian packaging consistency check complains that it isn't
executable but has a #! line. In fact it's reasonable to run this
script directly, so make it executable, and put it in a _scripts
variable so it will be installed executable.
Closes: #652
Approved by: cgwalters
They are installed non-executable, which makes Debian's Lintian
packaging consistency check complain that #! is only useful
in executable scripts. But in fact they are not useful to execute
directly (they rely on setup being done in the script that sources
them), so just chmod them -x.
Closes: #652
Approved by: cgwalters
In its initial commit, Alexander Larsson wrote
This works standalone, but unfortunately it breaks in
gnome-desktop-testing-runner as /tmp doesn't support
xattrs, so it is not installed atm.
but we now (a) use /var/tmp, and (b) explicitly skip the test if
xattr support is unavailable. So it should be OK to run now.
Closes: #652
Approved by: cgwalters
libcurl AFAICS doesn't have an API to convert HTTP code ➡️ error
string, so let's make the test regexp operate on both.
Closes: #651
Approved by: giuseppe
It turns out libsoup strips all whitespace even *inside* a URL. We could do that
for libcurl too but...really, people shouldn't do that. In this test we were
adding the trailing newline into the URL. If someone complains who is using the
libcurl code we can deal with it then.
Closes: #651
Approved by: giuseppe
We had a lot of copies of the "echo something 1>&2; exit 1" code even though
`assert_not_reached()` was it. Hence, I think we need a shorter alias for that.
Doing this particularly since I noticed a missing `1` in an `exit 1` call in the
rpm-ostree copy of this.
Closes: #648
Approved by: jlebon
Working on the libcurl backend, I hit the issue that the trivial-httpd program
depends on libsoup. I briefly considered having two versions, but libcurl is
client only, and moreover trivial-httpd is no longer trivial - it has various
features which are used by the test suite extensively.
Hence, what we'll do is build it as a separate binary which links to libsoup,
and use it during the tests. We *also* currently still provide `ostree
trivial-httpd` since some things use it like `rpm-ostree-toolbox` and the
Cockpit tests.
After those are ported to use some other webserver, I plan to add a build-time
option to drop it.
Closes: #636
Approved by: jlebon
This is a migration from the origin version. It's
nicer to have it in the remote, since that's what one
needs to change. Then tools don't need to mess with
the origin file.o
In fact in this scenario one can keep the "media source" like
`file:///install/repo` or whatever, since conceptually that's where it
came from. We're just providing a better error.
Closes: https://github.com/ostreedev/ostree/issues/626Closes: #627
Approved by: jlebon
The spam in stderr was bothering me, and further at some eventual
point in the future we want to annotate the functions with
`__attribute__((nonnull))` which would then cause tests like these to
become undefined behavior.
The coverage of this isn't worth the log spam basically.
Closes: #611
Approved by: jlebon
This reduces the diff when comparing these scripts with similar glue
in dbus or elsewhere.
Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>
Closes: #600
Approved by: cgwalters
This is deliberately permissive: a lot of it is generic, and I'm
using similar scripts in dbus.
Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>
Closes: #600
Approved by: cgwalters
Otherwise, we'll fail (due to set -u) if this parameter variable isn't
passed.
Signed-off-by: Simon McVittie <simon.mcvittie@collabora.co.uk>
Closes: #600
Approved by: cgwalters
glnx_make_lock_file requires that the dfd passed in survives the
lifetime of the lock. Since dfd_iter.fd gets cleaned up after the
function returns, this isn't the case. dfd_iter.fd should be equivalent
to tmpdir_dfd, since we iter on ".", and that survives past the
function, so just use that instead.
Closes: #591
Approved by: cgwalters
The fact that we weren't doing this is at best an oversight, and
for some deployment models a security vulnerability. Having both
`gpg-verify` and `gpg-verify-summary` shows that we were intending
them to be orthogonal/independent.
Lately I've been advocating moving towards pinned TLS instead of
gpg-signed summaries, and if we follow that path, performing GPG
verification of commit objects even if using deltas is more important,
as it provides an at-rest verifiable authenticity and integrity
mechanism.
Content providers which are signing their summary files and/or using
TLS (particularly pinned TLS) for transport should treat this as a
nice-to-have. However, for providers which are serving content over
plain HTTP and relying on GPG, this is a critical update.
Closes: https://github.com/ostreedev/ostree/issues/517Closes: #589
Approved by: jlebon
You'd expect
ostree commit --tree=ref=A --tree=ref=B
to produce a commit with the union of the trees given. Instead you'd get
a commit with the contents of just the latter commit. This was due to an
optimisation where we'd skip filling out the `files` and `subdirs`
members of the mtree, just filling in the metadata instead. This backfires
becuase this same code relies on checking the `files` and `subdirs` members
itself to work out whether the mtree is empty.
This commit removes the optimisation, fixing the bug. Maybe there's a way
to keep the optimisation and still fix the bug but it's not obvious to
me.
Closes: #581
Approved by: cgwalters
Conceptually we've been moving towards having our GPG verification
paths be per-remote. The code internally supports this, but we
didn't expose an API to use it conveniently.
This came up when trying to add a new `gpgkeypath` option, since
right now rpm-ostree manually finds keyrings for the remote, and
hence it wasn't looking at the keypath, and said "Unknown key"
in status.
Adding an API fixes this nicely.
Closes: #576
Approved by: giuseppe
For Project Atomic, we already have RPM signatures which use files in
`/etc/pki/rpm-gpg`. It's convenient to simply bind the OSTree remote
configuration to those file paths, rather than having duplicate key
data.
This does mean that we need to parse the files for verification, so we
end up importing them into the verifier's temporary keyring, which is
a bit ugly, but it's what other projects do.
Closes: https://github.com/ostreedev/ostree/issues/573Closes: #575
Approved by: giuseppe
When doing commit --tree=ref=XXX while at the same time applying some
form of modifier, ostree dies trying to read the xattrs using the
raw syscalls. We fix this by falling back to ostree_repo_file_get_xattrs()
in this case.
Also adds a testcase for this.
Closes: #577
Approved by: cgwalters
Some deployments may want to gate access to content based on things
like OAuth. In this model, the client system would normally compute a
token and pass it to the server via an API.
We could theoretically support this in the remote config too, but
that'd be a bit weird for OAuth as the information is dynamic.
Therefore this cleans up the code a little bit to more clearly handle
the case that the fetcher is initialized from both remote config
data plus pull options.
Closes: #574
Approved by: giuseppe
We should just download the commit objects directly, as it's
obviously a lot more efficient than deltas.
I had to generate a summary file in more places in the tests,
since once created, it needs to be updated.
Closes: https://github.com/ostreedev/ostree/issues/528Closes: #566
Approved by: jlebon
Various bootloader add kernel commandline options dynamically, filter
these out when grabbing boot options from /proc/cmdline. Specifically
grub adds BOOT_IMAGE and systemd-boot adds initrd.
Closes: #560
Approved by: cgwalters
Otherwise several tests fail, for example in this build done in a French
locale by Debian's reproducible builds initiative, to check whether
the resulting binaries are identical to what was produced in an
English locale:
<https://tests.reproducible-builds.org/debian/logs/unstable/amd64/ostree_2016.11-1.build2.log.gz>
(test-basic)
# error: Cannot write to repository: Permission non accordée
...
File 'error-message' doesn't match regexp 'Permission denied'
(test-help)
# Utilisation :
# ostree [OPTION...] COMMAND
...
File 'out' doesn't match regexp '[Uu]sage'
(test-pull-metalink)
# error: Erreur à la ligne 1, caractère 1 : Le document doit commencer avec un élément (par ex. <book>)
...
File 'err.txt' doesn't match regexp 'Document must begin with an element'
Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #558
Approved by: cgwalters
I'm doing builds and `make check` inside a Docker container, with
selinux on as a build-time option, but no policy in the container.
This currently aborts. Let's not do that.
(This type of thing is why installed tests are a better model)
Closes: #546
Approved by: jlebon
If we have a partial commit it is not an error for a dirmeta to be
missing (in fact, that is likely), so instead of returning a not-found
error from ostree_repo_traverse_commit() we ignore the error and
continue.
In particular, this means we don't stop early at the first
missing dirmeta, which previously caused ostree_repo_prune() to
thing the dirmetas after that to be unreached and thus purged.
Also, we remove the special casing in ostree_repo_prune() to
not report errors for commitpartial, because these should not
be reported anymore.
This fixes https://github.com/ostreedev/ostree/issues/541Closes: #542
Approved by: cgwalters
Quoting Dan Nicholson in
<https://github.com/ostreedev/ostree/pull/330#issuecomment-245499099>
mtime of 0 has been the semantics of ostree deployments from basically
the beginning of the project. We (and others, see
flatpak/flatpak@b5204c9) rely on that fact when generating trees.
In particular, this affects caches that use the mtime of the
associated file or directory to determine if the cache is valid. By
arbitrarily changing the mtime of the files to something else, all
the caches we setup in the build are now invalidated. Preseeding
caches is really important to the user experience as it avoids
having the user wait while they're regenerated on first run.
Now, we could change our build infrastructure to preset all the
mtimes to 1 to match this change, but what does that do for our
existing users who are on an ostree that deploys with mtimes of 0?
We could just revert this change at Endless (and the associated one
in Flatpak), and that would be fine for our users. However, if we
point non-Endless users to our apps, they'll have the great
experience of waiting 10 seconds the first time they launch it while
the fontconfig cache is rebuilt unnecessarily.
Closes: #495
Approved by: jlebon
musl libc's implementation of `realpath` works by opening the path and then
doing a lookup in `/proc/self/fd` to find the canonical path. This fails
if `/proc` is not mounted. This causes problems for us if
`ostree-prepare-root` is `init` as `/proc` won't be mounted.
We have to mount `/proc` anyway for `/proc/cmdline` so this fix just
expands the scope over which `/proc` is mounted to include both our
`realpath` calls.
See also:
* http://www.openwall.com/lists/musl/2016/06/08/2 and
* http://git.musl-libc.org/cgit/musl/tree/src/misc/realpath.c?id=e738b8cbe64b6dd3ed9f47b6d4cd7eb2c422b38dCloses: #485
Approved by: cgwalters
This test previously depended on manually building ostree-prepare-root.
Since 42dab85 we've been able to build static binaries with the usual
autotools build-system. This change reflects the fact that
`ostree-prepare-root` is built into $srcdir rather than `src/switchroot`
where I was building manually.
This test now passes with `./configure --with-static-compiler=gcc` (glibc)
but still fails with `./configure --with-static-compiler=musl-gcc` (musl).
Closes: #485
Approved by: cgwalters
This is a proper fix for:
https://bugzilla.gnome.org/show_bug.cgi?id=755787
With this patch, an admin (system builder) can now:
1) Edit /usr/lib/ostree-boot/uEnv.txt
2) Deploy the new tree. OSTree will append system's uEnv.txt
to the OSTree's managed uEnv.txt (loader/uEnv.txt).
It is common for u-boot systems to read in an extra env
from external /uEnv.txt. The same file OSTree uses to pass
in its env. With this patch /uEnv.txt now contains OSTree's
env + custom env added by system builders.
Closes: #466
Approved by: cgwalters
When doing a prune, we should not try to delete objects in parent
repos, since it'll fail. There is a bigger discussion about the
semantics of `parent=` to be had, but this will fix trying to use
`ostree prune --repo=/ostree/repo/extensions/rpmostree/pkgcache`.
Closes: https://github.com/ostreedev/ostree/issues/467Closes: #471
Approved by: jlebon
The test suite was actually doing something before, just
not quite what I intended. Without `-U` for bare-user checkouts
we end up doing a copy.
Now, a future commit will change how rofiles work, which would cause
the test suite to permit inplace mutation for non-hardlinked files.
So let's ensure they *are* hardlinked.
Closes: #462
Approved by: jlebon
Pulls from local repos now default to disabling static deltas so that
objects are copied. To check this is the case, see if the object files
are hardlinked after pulling.
Closes: #447Closes: #448
Approved by: cgwalters
When testing pulling of deltas, use the new --require-static-deltas
option to pull-local to ensure that deltas are actually used. To support
the require-static-deltas mode, the summary in the remote repo must be
generated.
Closes: #447Closes: #448
Approved by: cgwalters
Lots and lots of preparation led to this moment - when nothing
apparent changes for users! Woo!
But seriously, having the extra dependency is a minor annoyance, and
in the big picture I think the libgsystem idea was wrong - we need to
land things in GLib, and use git submodules for API-unstable or
Linux-specific sharing. For a lot of OSTree, the libgsystem `GFile*`
orientation was also wrong, we really want fd-relative.
Closes: #444
Approved by: jlebon
I hit an error with [CAHC](https://wiki.centos.org/SpecialInterestGroup/Atomic/Devel)
where we were doing time-based pruning. `ostree summary -u` started failing,
and it took me a bit to realize it was because we were pruning
even the tip of old branches, which I was not at all expecting,
and I don't think users will too.
Perhaps in the future we could add some sort of --prune-ref-tips or
something if people wanted it, but I doubt it.
Closes: #445
Approved by: jlebon
.travis.yml is obviously still Travis-specific, but tests/ci-* are
designed to be shareable with other CI environments if there is interest
in doing so.
At the moment I'm only testing on Debian and Ubuntu. In principle we
could try a non-Debian-derived Docker container such as Fedora or CentOS
inside travis-ci's Ubuntu environment, similar to what I'm doing
for Debian, but I don't know the correct setup commands to use there.
Closes: #438
Approved by: cgwalters
ostree's naming convention for whiteouts is similar to what is
done in aufs, which means we can't compose the trees to test this
feature under Docker with the aufs storage driver, as used on
travis-ci.
Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #437
Approved by: cgwalters
This allows ostree-prepare-root outside of the initramfs context where the
real rootfs is already mounted at /. We can't use `mount --move` in this
case because we would be trying to move / into a subdirectory of itself.
Closes: #403
Approved by: cgwalters
These tests use unshare and mount to prepare a fake initrd/early boot
directory structure so we can then test ostree-prepare-root.
Things that are tested:
* Running in an initrd environment
* Running without initrd
* /var and /sysroot being mounted correctly
* /usr being mounted read-only
Things not tested (yet):
* Running as init - this could be accomplished by unsharing the pid
namespace too.
* mounting/unmounting `/proc` if `/proc/cmdline` isn't available
* Persistent overlayfs for `/usr`
* Probably other things
The tests are basic but can be extended in the future as we do more work
on `ostree-prepare-root`.
These tests must be run as root as they require the ability to `mount`
and to `unshare` the mount namespace. Perhaps in the future we can use
user namespaces for this test once they are more widely available.
Closes: #403
Approved by: cgwalters
In general we want to support "idempotentcy" or "state
synchronization" across interruption. If a repo is only partially
created due to a crash or whatever, it's hard for a user to know that.
Let's just make `ostree_repo_create()` idempotent. Since all we're
doing is a set of `mkdirat()` invocations, it's quite simple.
This also involved porting to fd-relative, which IMO makes the
code a lot clearer.
Closes: #422
Approved by: 14rcole
There are variants of this, but right now because `GTask` has its
own pool which isn't necessarily cleaned up on exit, we need
to suppress all of this.
Closes: #410
Approved by: giuseppe
We don't presently support this, since the static delta code assumes
it can just `mmap()` file objects. We could at some point implement
this, but for now just skip executing deltas when doing
`archive -> archive` mirroring.
I noticed this when trying to mirror a repo in Jenkins in
[CAHC](https://wiki.centos.org/SpecialInterestGroup/Atomic/Devel).
Closes: #408
Approved by: giuseppe
This reverts commit 71301d1824.
I was confused by the fact that the non-matching content was
empty and forgot we already did print it.
Reported-by: smcv
Closes: #390
Approved by: jlebon
https://github.com/ostreedev/ostree/pull/372 caused these tests to
start failing when the host system is managed using ostree - since the
tests *do* replace the `ostree=` kernel argument.
Closes: #384
Approved by: cgwalters
Systems like pulp may want to keep retrying in a loop if the server
throws a (hopefully transient) 500, and we need test coverage of
handling these errors versus our existing 404 and 206 coverage.
Closes: #383
Approved by: mbarnes
We clean up the temporary directory on failure, which means it's hard
to know *why* a regex didn't match. Print it when we hit an error.
Closes: #383
Approved by: mbarnes
I often want to have "idempotent" systems that iterate to a known
state. If after generating a commit, the system is interrupted, I'd
like the next run to still generate a delta. But we don't want to
regenerate if one exists, hence this option.
Closes: #375
Approved by: jlebon
On some systems there may be no root= argument, so the tests for
appending /proc/cmdline arguments will fail. Instead, loop over each of
the arguments in the host's /proc/cmdline and test that they're in the
constructed config file. That will actually test if ostree added all of
the system's /proc/cmdline args correctly. The regex isn't perfect here,
but it's probably good enough for this test.
Closes: #372
Approved by: cgwalters
When creating sysroots with libostreetest, we don't get the benefit of
the OSTREE_SYSROOT_DEBUG setting in libtest.sh. That means we'll get
immutable deployments that can't be easily cleaned up.
Ensure the environment variable is set before creating new sysroots. It
would be nice to set the debug flags directly, but that's private API
that isn't currently pulled into libostreetest.
Closes: #372
Approved by: cgwalters
I've seen a few people hit this and wonder why checkouts are slow/take
space. Really, ensuring this happens is the *point* of OSTree.
Physical copies should be a last resort fallback for very unusual
situations (one of those is rpm-ostree checking out the db since
librpm doesn't know how to read from libostree).
Even I hit the fact that `/var` is a mountpoint disallowing hardlinks
with `/ostree` once and was confused. =)
Add this to the rofiles-fuse test case because it creates a mount
point.
Closes: #368
Approved by: jlebon
Add the functionality to use the same name for refs in local and remote
repos. This helps users keep track of local refs of remote origin, much
like local and remote git branches.
Previously, when a local ref is specified, resolve_refspec would fall
back to searching through remote repos if the ref is not found locally.
This function now takes an extra flag to specify whether it should
search through remote repos. Additionally, ostree_repo_resove_rev_ext
was added to call resolve_refspec with fallback_remote being false, so
refs --create would no longer complain when trying to create a local
ref of the same name as a remote one.
Fix remote repo parsing not being handled correctly on refs --create.
Closes: #363
Approved by: jlebon
We noticed that once a ref folder is created, there is no existing
command that can remove it. For example, once "foo/bar" is created,
even if the user deletes foo or all the refs under foo, the folder
will persist.
Now when the user attempts to create a ref "foo" either through commit
or refs --create, if a folder "foo" exists but is empty of refs, the
folder is removed and the new ref "foo" is created.
New unit tests in tests-ref.sh verify this functionality.
Closes: #354
Approved by: cgwalters
For some reason I'm really not inclined to debug right now, `libtool`
is eating the `-fsanitize=address` argument when linking
`libreaddir-rand.so`, which causes failures since we're trying to
`LD_PRELOAD`.
Closes: #351
Approved by: jlebon
It's unfortunate that in automake one has to explicitly include the globa
`$(AM_CFLAGS)` if one sets `CFLAGS`, and similarly for other variables.
I'm trying to use `-fsanitize=address`, and not including it was
causing linker failures.
We also weren't inheriting the global warnings etc., so I had to fix a
decl-after-statement.
Closes: #351
Approved by: jlebon
I find the "-z2" is really a long ago relic of the past when I changed
the format. We no longer have anything to do with the original
`archive`, so let's start allowing people to type `--mode=archive`
which just looks saner.
At some point later I'll update the docs too, but it'll be an annoying
transition period as we'll have to say "On older OSTree, use -z2" etc.
Closes: #346
Approved by: giuseppe
Added the ability to create a ref (much like a git tag) for an
existing commit through "ostree refs EXISTING --create=NEWREF".
Previously the only way to create a new ref was by creating a new commit,
but refs --create allows multiple refs to point to the same commit.
The command will fail if:
- None/more than one existing ref is specified
- The specified EXISTING tag does not exist, or was not specified
- The specified NEWREF already exists, or is the name of a folder
Add unit tests in tests-ref.sh to verify above functionality
Closes: #340
Approved by: jlebon
Otherwise, during "make distcheck" it will be read-only (because it
is a copy of the read-only ${srcdir} with permissions preserved), and
deletion will fail during cleanup.
Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #337
Approved by: cgwalters
As with the C tests in commit 08580118, this makes sure the test
fails as soon as something goes wrong.
Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #335
Approved by: giuseppe
The recent memleak fixes motivated me to look at the bitrotted code to
run invocations of `ostree` in the test suite underneath valgrind.
There are a few things here. First, update suppressions file from
libhif, since I recently worked on it.
When running *uninstalled* as we now support, we need
`libtool --mode=execute` in the mix so it expands out to
the uninstalled binary and we don't valgrind the intermediate shell.
However, it's harder than that because we chdir into a tmpdir,
which defeats the libtool logic. AFAICS, the only fix for this
is to determine the realbin path before we chdir, and then unfortunately
we need to change every use of `ostree` to `${OSTREE}` =(
Then this immediately breaks for me on RHEL7 because my ancient
copy of `valgrind-3.10.0-16.el7.x86_64` is unaware of syscall 306, i.e.
`syncfs`.
But let's do this first before I dive into that.
Closes: #292
Approved by: krnowak
1 is a better choice than 0 because some programs use 0
as a special value; for example, GNU Tar warns of an
"implausibly old timestamp" with 0.
Closes: #330
Approved by: cgwalters
We have had in the past issues with running `ostree_repo_pull()`
multiple times in the same process, embarassingly enough. Nothing in
the current test suite covers this, so let's start.
Closes: #322
Approved by: jlebon
I was extending the C-based test suite to cover more, and noticed that
we weren't aborting if a command failed. That made the tests somewhat
less useful, so let's fix it.
Closes: #321
Approved by: jlebon
It is quite similar to the already existing
ostree_raw_file_to_content_stream function, so I factored the common
part to a separate function. The difference is that we cannot report
the size of the resulting stream.
Can be useful for serving a "bare" repository as a faked "archive-z2"
repository.
Closes: #308
Approved by: cgwalters
In practice, a lot of subjects are empty, because the commit date
and branch are sufficient identification. For example, rpm-ostree
does not use subjects. It also doesn't use the command-line ostree
commit tool, so this was not a problem there, but this makes the
behavior consistent.
Also adds a test that empty subjects and omitting the subject
are equivalent. The --timestamp is so that the commits do not
have different timestamps.
Closes: #305
Approved by: cgwalters
It's very useful for third-party applications to have someplace to store
their data guaranteed to be on the same device as the repo (thus
ensuring hardlinks) while still being shielded away from any of OSTree's
timely garbage collections.
We create a new "extensions/" subdirectory where apps can include
whatever they wish in "extensions/myapp/". This subdirectory is
completely unmanaged by ostree.
NB: I didn't bother making it a member of the OstreeRepo proper since we
don't really use it for anything else yet.
Closes: #286
Approved by: cgwalters
- Do a bit of refactoring
- Add test for use_ostree_convention
- Add test for xattr_callback
- Add test for SELinux labeling
Closes: #275
Approved by: cgwalters
- Test both tar and cpio archives
- Test more hardlink corner cases
- Test symlinks more rigorously
- Test stat override
- Test skip list
Closes: #275
Approved by: cgwalters
This was already supported by the commit modifier API, just needed to
expose it. This will also be used to test the libarchive API in a future
test.
Closes: #275
Approved by: cgwalters
We had a policy of cleaning up all files in `$repo/tmp` older
than one day, but we should really clean up previous bootid staging
directories too, as they can potentially take up a lot of disk space.
https://bugzilla.gnome.org/show_bug.cgi?id=760531Closes: #170
Approved by: jlebon
Setting this causes commit to error out. There are other ways we
could do this in a more sophisticated fashion, such as via SystemTap
etc. But this has low-tech applicablity, works as non-root.
The reason I'm adding this is so that we can add test cases for
cleanup of the `tmp/staging-` directory.
Closes: #170
Approved by: jlebon
This lets you set a prefix for the resulting archive patsh.
Especially useful in combination with --subpath, for instance
--subpath=subdir --prefix=subdir to extract just subdir.
Closes: #265
Approved by: cgwalters
I'm not sure if something changed or there was a race condition , or
if extraterrestrials intervened to make the tests pass before, but
in the Docker container `/dev/fuse` does exist and since we appear
to be running as uid 0 it'll be writable.
We previously discussed that another requirement for FUSE (and most
setuid binaries) is having `CAP_SYS_ADMIN`, so let's check for that in
the bounding set too.
Closes: #266
Approved by: jlebon
I've seen an intermittent test failure in an autobuilder (sbuild)
environment where logs from failed builds cannot be retrieved,
but I can no longer reproduce it. Put the contents of the offending
file in the test's failing output so that if the failure comes back,
it can be debugged.
Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #264
Approved by: cgwalters
We keep forgetting to update `apidoc/ostree-sections.txt`, so let's
start enforcing it. Of course it turns out we had some bugs here
like symbols marked as public but never implemented, etc. Those
are fixed in the prior commits.
Closes: #263
Approved by: giuseppe
Force the otherwise disabled gpg verifications on.
Note: You need to pass --remote=foo so we know what gpg keys to verify
against.
Closes: #237
Approved by: cgwalters
ostree-grub-generator can be used to customize
the generated grub.cfg file. Compile time
decision ostree-grub-generator vs grub2-mkconfig
can be overwritten with the OSTREE_GRUB2_EXEC
envvar - useful for auto tests and OS installers.
Why this alternative approach:
1) The current approach is less flexible than using a
custom 'ostree-grub-generator' script. Each system can
adjust this script for its needs, instead of using the
hardcoded values from ostree-bootloader-grub2.c.
2) Too much overhead on embedded to generate grub.cfg
via /etc/grub.d/ configuration files. It is still
possible to do so, even with this patch applied.
No need to install grub2 package on a target device.
3) The grub2-mkconfig code path has other issues:
https://bugzilla.gnome.org/show_bug.cgi?id=761180
Task: https://bugzilla.gnome.org/show_bug.cgi?id=762220Closes: #228
Approved by: cgwalters
test-sysroot.js runs libtestExec() twice, one of which is after
creating non-hidden directories in $(pwd), so this check needs to be
skipped the second time.
Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #234
Approved by: cgwalters
If we touch test-xattrs before sourcing libtest.sh, we get:
test tmpdir=/tmp/test-tmp-ostree_test-xattrs.sh.test-HSEXEY is not
empty; run this test via `make check TESTS=`, not directly
Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #234
Approved by: cgwalters
This fixes the bug that in installed-tests that run testlib.sh under
"bash -c" (i.e. the C and JS tests), $(dirname $0) is "." and we can't do
the LD_PRELOAD correctly:
ERROR: ld.so: object './libreaddir-rand.so' from LD_PRELOAD cannot be
preloaded (cannot open shared object file): ignored.
Similarly, those tests can't copy gpghome correctly.
This also removes the confusing situation that SRCDIR in libtest.sh
(which is ${top_srcdir}/tests) does not mean the same thing as SRCDIR
in test-abi.sh (which is just ${top_srcdir}).
Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #234
Approved by: cgwalters
It's not working for me in `make check` on a RHEL 7 Workstation,
apparently because no GPG agent is spawned. I'm guessing this has
something to do with the GPG version?
The downside of this is we will be less likely to notice if GPG
changes again and we start leaking agents like we're in The Matrix
Reloaded. But the real solution to that is containers anyways.
Closes: #233
Approved by: smcv
moreutils parallel isn't compatible with the command-line syntax
used here. Because it doesn't implement GNU-style --help and exits 1
when that option is given, this test was correctly skipped when
using moreutils parallel, but only by mistake.
moreutils parallel might conceivably gain --help and --version in
future, but hopefully nothing incompatible with GNU parallel is going
to gain a --gnu option.
Also use the --gnu option to force the new command-line semantics;
some versions optionally supported an incompatible command-line syntax
taken from moreutils parallel.
Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #232
Approved by: cgwalters
[smcv: split out from a larger commit, part of PR #231; add commit message]
Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #232
Approved by: cgwalters
test-pull-untrusted.sh would pass when run as root, but fail when run
as testuser. It turned out that the way the files were stored in the
repo when running as a testuser were different, which meant that a
different .file object was chosen for corruption. Except that file
turned out to be a symlink, so the echo "broke" actually just wrote
to the no_such_file symlink target, thus keeping the actual symlink
file's checksum the same and causing the pull-local to pass when it
should have failed.
[smcv: split this out of a larger commit, part of PR #231]
Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #232
Approved by: cgwalters
The "function foo()" syntax is bash-specific, and Colin indicated in
PR #226 that he prefers to require bash rather than trying to support
every POSIX shell.
Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #232
Approved by: cgwalters
Some autobuilder environments place the entire build chroot on tmpfs, so
even /var/tmp might not have this.
Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #232
Approved by: cgwalters
The API supports this, and it's not hard for us to do in the command
line as well. One possible use case is separating "content
generation" in a separate server.
Related: https://github.com/ostreedev/ostree/pull/223Closes: #227
Approved by: jlebon
When I'm doing local development builds, it's quite common for me not
to want to accumulate history. There are also use cases for this on
build servers as well.
In particular, using this, one could write a build system that didn't
necessarily need to have access to (a copy of) the OSTree repository.
Instead, the build system would determine the last commit ID on the
branch, and pass that to a worker node, then sync the generated
content back.
The API supported generating custom commits that don't necessarily
reference the previous commit on the same branch, let's just expose
this in the command line for convenience.
I plan to also support this rpm-ostree.
Closes: #223
Approved by: jlebon
The test tries to get a filesystem that supports xattrs by writing
to /var/tmp, but in some automated build environments the entire
build chroot is on a tmpfs.
Signed-off-by: Simon McVittie <smcv@debian.org>
GNOME Continuous uses ginstest-runner --report-directory, which causes
the tests to save their tmpdirs persistently. This also means the
result directories didn't match the `/(var/)?tmp` regexp, which broke
the ostree tests in GContinuous.
Fix this by simply asserting that the tmpdir either has `.tmpdir` or
nothing.
%Z only uses seconds, so it's possible that we did the commit
in the same second, which made this test racy.
- Switch to full nanosecond precision using '%.Y' so it always differs
- Fix the inverted `cmp` usage
- Add a missing `ok`
This will allow daemons like rpm-ostree to detect if there are any new
deployments efficiently, in combination with using inotify. If there
are any changes, rpm-ostree wants publish them on DBus.
While we're here, add some changes to start doing unit C testing of
the sysroot API.
I want to be able to easily test the C API on actual data in an OSTree
repo. The shell `libtest.sh` has code to generate it. Bridge the two
worlds by introducing a little `libostreetest` library which has a C
API which spawns a shell that runs things in `libtest.sh`.
Yes, this is about as beautiful as it sounds, which is to say, it's
not. But it works!
Note while we were here, I realized we were actually now creating
*two* tmpdirs per test in `make check` because the tap driver was
already doing that. Unify it so we know the C code can rely on it.
This allows other processes (e.g. rpm-ostreed) to monitor for external
changes (e.g. if someone does `ostree admin undeploy`) in a relatively
sane fashion.
Specifically, I'm trying to fix:
https://github.com/projectatomic/rpm-ostree/issues/220
OSTree's code for testing predates the `glib-tap.mk` making its
way into GLib. Let's switch to it, as it provides a number
of advantages.
By far the biggest advantage is that `make check` can start to run
most of the tests *in addition* to having them work installed.
This commit keeps the installed tests working, but `make check` turns
out to be really broken because...our TAP usage has bitrotted to say
the least. Fix that all up.
Do some hacks so that the tests work uninstalled as well - in
particular, `glib-tap.mk` and the bits encoded into
`g_test_build_filename()` assume *recursive* Automake (blah). Work
around that by creating a symlink when installed to loop back.
If the average object size is greater than 4GiB, let's assume we're
dealing with opposite endianness. I'm fairly confident no one is
going to be shipping peta- or exa- byte size ostree deltas, period.
Past the gigabyte scale you really want bittorrent or something.
Some Docker layers are just metadata in the `layer.json`. If one is
mapping Docker layers to OSTree commits, one needs to create a dummy
root directory, because OSTree doesn't support metadata-only commits.
Let's just push that logic down here because it's easier than special
casing it in higher levels.
One of the design goals with deltas was not just wire efficiency,
but also having all the data up front about how much data would
be transferred before starting.
Let's expose that better by adding a `dry-run` option to the pull API.
This requires static deltas to be useful. Basically we simply call
the progress callback once with the data from the superblock.
For a production release repository, most OS vendors would want
to just always use static deltas. Add the ability for the pulls to
require it.
(I think I'll also add a summary key for this actually in addition,
so the repo manager can force it too)
If ostree is run in a test setup where it operates as root in a tmp
directory, it might cause issues to flag the deployments as immutable.
The test harness might simply be doing an `rm -rf` (effectively the case
for gnome-desktop-testing-runner), which will then fail.
We add a new debug option to the ostree_sysroot object using GLib's
GDebugKey functionality to allow our tests to communicate to ostree that
we don't want immutable deployments.
This is a more flexible version of the previous
ostree_repo_write_archive_to_mtree() which took a file reference.
This has an extensible options structure, and in particular
now supports `ignore_unsupported_content`.
I plan to use this for importing Docker images which contain device
nodes. (There's no reason for container images to have those, so
we'll just ignore them).
Also here, just like the export variant, the caller is responsible for
setting up libarchive.
I don't know why we didn't do this a long time ago. This extends the
pull API to allow grabbing a specific commit, and will set the branch
to it. There's some support for this in the deploy engine, but there
are a lot of reasons to support it for raw pulls (such as subset
mirroring cases).
In fact I'm thinking we should also have the override-version logic
here too.
NOTE: One thing I debated here is inventing a new syntax on the
command line. Git doesn't seem to have this functionality (probably
because it'd be rarely used). The '@' character at least doesn't
conflict with anything.
Anyways, I wanted this for some other test cases. Without this,
writing tests that go between different commits is more awkward as one
must generate the content in one repo, then pull downstream, then
generate more content, then pull again. But now I can just keep track
of commit IDs and do exactly what I want without synchronizing the
tests.
At the moment I'm looking at using rpm-ostree to manage RPM inputs
which can then be converted into Docker images. It's most convenient
if we can stream directly out of libostree rather than doing a
checkout + tar combination.
There are also backup/debugging etc. reasons to implement `export` as
well.
While it's not strictly tied to OSTree, let's move
https://github.com/cgwalters/rofiles-fuse in here because:
- It's *very* useful in concert with OSTree
- It's tiny
- We can reuse OSTree's test, documentation, etc. infrastructure
One thing to consider also is that at some point we could experiment
with writing a FUSE filesystem for OSTree. This could internalize a
better equivalent of `--link-checkout-speedup`, but on the other hand,
the cost of walking filesystem trees for these types of operations is
really quite small.
But if we did decide to do more FUSE things in OSTree, this is a step
towards that too.
I'd like to incrementally convert all of `ostree-repo*.c` to
fd-relative usage, so that we can sanely introduce
`ostree_repo_new_at()` which doesn't involve GFile.
This one is medium risk, but passes the test suite.
I noticed in the static deltas tests, there were some tests that
should have been under `-o pipefail` to ensure we properly propagate
errors.
There were a few places where we were referencing undefined variables.
Overall, this is clearly a good idea IMO.
Downloads and prints a remote summary file and any signatures in an
easy-to-read format, or alternatively with the --raw option, prints
the summary GVariant data directly.
https://bugzilla.gnome.org/show_bug.cgi?id=759250
Use the parse-datetime module from gnulib, and adapt it to not require
other modules as portability is not really an issue for us.
DATE can be specified in different formats, such as: "-1 week", "last
monday", "1 week ago".
Include the generated .c file in the repository so to not add another
dependency to Bison.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Do not delete a .commitmeta file after removing the last metadata entry.
This way a client will pull the empty .commitmeta file and overwrite old
metadata as expected.
https://bugzilla.gnome.org/750459
And add a syntax rule to avoid this in future.
Fixed by:
sed -i -e 's|^ostree |${CMD_PREFIX} ostree |g' tests/*.sh
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Adds an entry to the origin file to force the OstreeSysrootUpgrader to
pull and deploy the given checksum.
[origin]
override-commit=CHECKSUM
If the option is not given, any such entry is explicitly removed from
the origin file to ensure we upgrade to the latest available commit.
I was working on a different test, and ended up being very confused at
the behavior where removing the last deployment didn't remove the last
`ostree/X/X/X` ref pointing to its commit.
There's no reason to special case the last undeployment AFAIK, and the
existing code handles this.
There's no reason to keep them hidden. I have a hard policy that
OSTree should *not* be used to carry secrets. Things like host ssh
private keys should be set up out of band by an OS-external
configuration mechanism such as kickstart, cloud-init, etc.
We also assume that hiding binaries is not very useful as most
attackers would be able to find them on the Internet or (for
subscribed content) acting as a customer.
This fixes a bug with mirroring because we changed to take the
unmodified upstream objects rather than uncompress <-> recompress.
https://bugzilla.gnome.org/show_bug.cgi?id=748959
Eliminates the need for constantly passing --sysroot=sysroot, but
also makes ostree place remote configs for sysroot/ostree/repo in
sysroot/etc/ostree/remotes.d where they should have been all along.
I did a quick audit pass through the pull code. What I focused on the
most is the case where `gpg-verify-summary=true`, and in particular
where `gpg-verify=false` too. This should be a valid and secure
configuration.
The primary change here is to error out very quickly if either
`summary` or `summary.sig` are 404. Previously, we'd only error out
if we were processing deltas.
Expand the existing test case to cover this, plus invalid summary and
invalid sig. (The test case was failing with current git master too).
Works like "ostree refs" but fetches refs from a remote repo.
This depends on the remote repo having a summary file, but any repo
being served over HTTP *ought* to have one.
libtest always makes a copy of the gpghome directory to the test
directory, so there's no need to operate on the installed copy. This
allows test-remote-gpg-import to pass as an unprivileged user since it
otherwise couldn't create the temp files gpgme creates.
On 32 bit systems, _FILE_OFFSET_BITS will be set to 64 by
AC_SYS_LARGEFILE. This causes the glibc headers to use readdir64 rather
than readdir. Emulate that behavior in the preloader or the tests will
all fail with "No such file or directory".
Just noticed this while debugging something else. We don't want to
commit the whole test dir, just the workdir.
Trying to commit the repo itself is potentially subject to race
conditions at least.
Having undefined (but in practice rarely changing) ordering for
`readdir()` ended up screwing us over for bootloader config
generation; see https://bugzilla.redhat.com/show_bug.cgi?id=1226520
Let's make things significantly more likely to fail more quickly in
the future if similar bugs are introduced. We accomplish this by
introducing a little `LD_PRELOAD` library that randomizes the results
of `readdir()`.
Having undefined ordering (but in practice rarely changing)
ordering for `readdir()` ended up screwing us over with respect
to bootloader config file read ordering.
Let's make things significantly more likely to fail more quickly in
the future if similar bugs are introduced. We accomplish this by
introducing a little `LD_PRELOAD` library that randomizes the results
of `readdir()`.
One can run this on a machine to validate things. I'd like to
get this plugged into the actual OSTree tests as soon as we can
figure out how to sanely run grub2-generate as non-root in
our test suite.
Alternatively, this script can easily be run on a real install.
External daemons like rpm-ostree want push notification any time a
change is made by an external entity. inotify provides notification,
but a problem is there's no easy way to monitor all of the refs.
In the past, there has been discussion of opt-in recursive timestamps:
https://lkml.org/lkml/2013/4/5/307
But in today's world, let's just bump the mtime on the repo itself, as
a central inotify point.
Closes: https://github.com/GNOME/ostree/pull/111
When I removed the `transaction` symlink, that made this test start
failing. Fix it by doing `chmod` on `repo/objects`, which is what the
core `ostree_repo_is_writable()` looks at.
I use the trivial httpd server locally. Each time I restart the
server, I end up modifying manually the config file for other repos so
to point to the correct port. In this way I can just re-use the same
port.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
An OSTree user noticed that `ostree fsck` would produce `missing
object` errors in the case of interrupted pulls.
It's possible to do e.g. `ostree pull --subpath=/usr/share/rpm ...`,
which gets you just that portion of the commit. The use case for this
was being able to see what changes would appear in an update before
actually downloading all of it.
(I think this would be better covered by static deltas, but those
aren't final yet, and `--subpath` predates it)
Further, `.commitpartial` is used as a successor to the `transaction`
symlink for more precise knowledge in the case where a pull was
interrupted that we needed to resume scanning.
So it makes sense for `ostree fsck` to be aware of it.
If a system administrator happens to type `ostree admin upgrade`
multiple times, currently that will lead to a potentially corrupted
system.
I originally attempted to do locking *internally* in `libostree`, but
that didn't work out because currently a number of the commands
perform multi-step operations that all need to be serialized. All of
the current code in `ostree admin deploy` is an example.
Therefore, allow callers to perform locking, as most of the higher
level logic is presently implemented there.
At some point, we can revisit having internal locking, but it will be
difficult. A more likely approach would be similar to Java's approach
with concurrency on iterators - a "fail fast" method.
Always request detached metadata for commit objects, even if we already
have the commit object. This ensures we fetch any post facto detached
metadata updates such as new GPG signatures.
https://bugzilla.gnome.org/748220
First, git doesn't do this, and whatever Linus thinks is right or
something.
Second specifically to OSTree, it's quite common to not have
intermediate commits. If one wants to reset a ref in order to prune
data after a deployment, the parentage check will fail.
Closes: https://github.com/GNOME/ostree/pull/87
rpm-ostree currently uses ostree_repo_checkout_tree(), which as a side
effect will use the uncompressed objects cache by default. This is
rather annoying if you're using rpm-ostree on a server-side
repository, because if you then rsync the repo, you'll be syncing out
the uncompressed objects unless you exclude them.
We added the ability to disable the uncompressed cache in the
repository config to fix this, but it's better to allow application
control over this. The uncompressed cache will in some future version
become opt in as well.
This new API further:
- Drops the `GFile` usage in favor of `openat` APIs
- Improves ergonomics by avoiding callers having to query the source
`GFileInfo` (and carry around a copy of `OSTREE_GIO_FAST_QUERYINFO`)
- Has a more extensible options structure
Per the comment, I rather crudely have the `ostree checkout` builtin
call both APIs to ensure some testing coverage.
However, I'd like to in the future have easier-to-set-up testing code
that calls `libtest.sh` to set up dummy data.
If the starting index is beyond the end of the list, it's a programming
error. Previously, the code was trying to raise a runtime error, but
actually causing a segfault.
This was detected by test code in test-mutable-tree.c, which is removed
in this commit because it should now not be possible to crash here.
https://bugzilla.gnome.org/747032
Turns out the expired signature case was failing because the signature
itself was corrupted. Reconstructed the test data and updated the test
code. Note, an expired signature is still counted as valid.
Also, handy debugging trick for setting a key or signature expiry: the
CLI makes it appear the shortest expiry is 1 day, but it also secretly
recognizes "seconds=N".
Wrappers a referenced gpgme_verify_result_t so detailed verify results
can be examined independently of executing a verify operation.
_ostree_gpg_verifier_check_signature() now returns this object instead
of a single valid/invalid boolean, but the idea is for OstreeRepo to also
return this object for commit signature verification so it can be utilized
at the CLI layer (and possibly by other programs).
We already set all file mtimes to 0 so that they are constant
over all checkouts, and can be made constant with a known value from
the system where the ostree was created.
However, this was not happening for directories. Zero their mtimes too.
This is important for shipping a fontconfig cache in the ostree;
the fontconfig cache files embed a directory mtime.
This does an rsync-style prepared delta basically. On my test data,
it shaves ~6MB of uncompressed data. Not a huge amount, but I expect
this to be more useful for things like binaries which embed data, etc.
It's always been suboptimal to have both pull and pull-local; as we go
beyond the raw object data into things like deltas and summary files,
the logic to perform e.g. mirroring should only be in one place.
This will be used by Pulp's OSTree content plugin at least to perform
promotions.
When doing a pull --mirror from an archive-z2 repository into another
archive-z2 repository, currently we gunzip/checksum/gzip each content
object. The re-gzip process in particular is fairly expensive.
This does assume that the upstream content is trusted and correct.
It'd be nice in the future to do at least a CRC check, if not the full
checksum. (Could we append CRC data to the end of filez objects?)
We could also choose to only do this optimization if fetching over
TLS.
before: 1626 metadata, 20320 content objects fetched; 299634 KiB transferred in 62 seconds
after : 1626 metadata, 20320 content objects fetched; 299634 KiB transferred in 11 seconds
See projectatomic/rpm-ostree#42 for rationale. There are two high
level use cases:
- If the OS comes unconfigured, this is a way to point it at a repo of your choice.
- To switch between repositories while keeping the same branch easily.
You create these with something like:
ostree static-delta generate --empty --to=master
These will be automatically used during pull if no previous revision
exists in the target repo.
These work very much like the normal static deltas except they
are named just by the "to" revision. I.e:
deltas/94/f7d2dc23759dd21f9bd01e6705a8fdf98f90cad3e0109ba3f6c091c1a3774d
for a from-scratch to 94f7d2dc23759dd21f9bd01e6705a8fdf98f90cad3e0109ba3f6c091c1a3774d delta.
https://bugzilla.gnome.org/show_bug.cgi?id=721799
For Anaconda, I needed OSTREE_REPO_REMOTE_CHANGE_ADD_IF_NOT_EXISTS,
with the GFile *sysroot argument to avoid ugly hacks. We want to
write the content provided via "ostreesetup" as a remote to the target
chroot only in the case where it isn't provided as part of the tree
content itself.
This is also potentially useful in idempotent systems management tools
like Ansible.
https://bugzilla.gnome.org/show_bug.cgi?id=741577
This creates a archive-z2 repo, pull-locals it to bare-user and then
again back to archive-z2 making sure things fsck along the way.
Then it checks out all repos and makes sure each one reproduces
the same result.
Unfortunately we can't install this as a real test because
it doesn't work in the test-runner because tmpfs doesn't support
user xattrs.
https://bugzilla.gnome.org/show_bug.cgi?id=741125
This just does whatever test-basic.sh does, but on a bare-user
repo.
This works standalone, but unfortunately it breaks in
gnome-desktop-testing-runner as /tmp doesn't support
xattrs, so it is not installed atm.
https://bugzilla.gnome.org/show_bug.cgi?id=741125
Use the pattern:
$PRETTY_NAME [$COMMIT_VERSION] (ostree[:$OSNAME][:$DEPLOYMENT_INDEX])
$OSNAME is only shown if there are multiple values.
$COMMIT_VERSION refers to the version tag in the commit's metadata.
$DEPLOYMENT_INDEX is only shown if no $COMMIT_VERSION is available.
https://bugzilla.gnome.org/show_bug.cgi?id=739416
fixes a coredump when using a command like:
$ ostree --repo=repo checkout -U --subpath=/usr/lib/passwd \
fedora-atomic/rawhide/x86_64/docker-host usrlib-new
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
For Anaconda, we have an ugly bootstrapping problem where we need to
add the remote to the repository's config, then do a pull+deploy, then
remove and re-add the config, because /etc/ostree/remotes.d doesn't
exist yet in the target system.
https://bugzilla.gnome.org/show_bug.cgi?id=738698
In this approach, we drop a /etc/grub.d/15_ostree file which is a
hybrid of shell/C that picks up bits from the GRUB2 library (e.g. the
block device script generation), and then calls into libostree's
GRUB2 code which knows about the BLS entries.
This is admittedly ugly. There exists another approach for GRUB2 to
learn the BLS specification. However, the spec has a few issues:
https://www.redhat.com/archives/anaconda-devel-list/2014-July/msg00002.html
This approach also gives a bit more control to the admin via the
naming of the 15_ostree symlink; they can easily disable it:
Or reorder the ostree entries ahead of 10_linux:
Also, this approach doesn't require patches for grub2, which is an
issue with the pressure to backport (rpm-)OSTree to EL7.
Some operating systems may come with external tools for subscription
management that drive access to the content. In that case, the origin
file may not be useful (for example, it could refer to an installer
ISO).
This patch will allow OS installers to inject that state, with a
useful error message, directing the system administrator to an
external tool.
See: https://github.com/projectatomic/rpm-ostree/issues/31https://bugzilla.gnome.org/show_bug.cgi?id=737686
Recursive over ostree and all subcommands, and check that --help
is supported, properly outputs to standard out, and exits
with a 0 exit status. Check that for commands with subcommands,
they produce the help output to standard error when run with no arguments.
https://bugzilla.gnome.org/show_bug.cgi?id=737194
Previously, in the case where a parent directory of a modified config
file was removed, we would throw an exception. This happens when
switching from a tree that has some software (e.g. firewalld), to one
that does not.
While it's nice to have this warning that your config file probably no
longer applies, there's no need to make it so...fatal.
It's particularly problematic that the only easy workaround is to
remove the config files from your current tree - which breaks
rollback.
The solution then is for for us to take ownership of the parent
directories too into the new /etc. Admins can clean up these files
afterwards at any time.
https://bugzilla.gnome.org/show_bug.cgi?id=734293
While looking to fix a different bug here, I found the current
state of things where we had a mix of fd-relative API versus not
frustrating.
Change the code around to consistently use *at, and also add some more
tests.
For Fedora and potentially other distributions which use globally
distributed mirrors, metalink is a popular solution to redirect
clients to a dynamic set of mirrors.
In order to make metalink work though, it needs *one* file which can
be checksummed. (Well, potentially we could explode all refs into the
metalink.xml, but that would be a lot more invasive, and a bit weird
as we'd end up checksumming the checksum file).
This commit adds a new command:
$ ostree summary -u
To regenerate the summary file. Can only be run by one process at a
time.
After that's done, the metalink can be generated based on it, and the
client fetch code will parse and load it.
https://bugzilla.gnome.org/show_bug.cgi?id=729585
Changes the pull API to allow pulling only a single directory instead
of the whole deployment. This option is utilized by the check-diff
option in rpm-ostree.
Add a new state directory to hold <checksum>.commitpartial files, so
we know that we've only downloaded partial state.
If we deployed a new tree inside an existing OS, inheriting kernel
args, we need to use append() instead of replace() to avoid collapsing
multiply specified kernel arguments like console=/dev/foo
console=/dev/bar.
Reported-by: Dusty Mabe <dusty@dustymabe.com>
The user might "ostree ls /usr/bin/bash/blah", which previously would
segfault.
A somewhat related future enhancement here would be for "ostree ls" to
follow symbolic links.
Reported-by: Dusty Mabe <dustymabe@gmail.com>
https://bugzilla.gnome.org/show_bug.cgi?id=733476
This patch adds a function that will parse a partial checksum when
resolving a refspec. If the inputted refspec matches a truncated
existing checksum, it will return that checksum to be parsed. If
multiple truncated checksums match the partial refspec, it is not
unique and will return false. This addition is inspired by the same
functionality in Docker, which allows a user to reference a specific
commit without typing the entire checksum.
partial checksums: Add function to abstract comparison
This modifies the list_objects and list_objects_at functions
to take an additional argument for the string that a commit starts
with. If this string arg is not null, it will only list commit
objects beginning with that string. This allows for a new function
ostree_repo_list_commit_objects_starting_with to pass a partial string
and return a list of all matching commits. This improves on the
previous strategy of listing refs because it will list all commit objects,
even ones in past history. This update also includes bugfixes on
error handling and string comparison, and changes the output structure
of resolve_partial_checksum. The new strcuture will no longer return FALSE
without error. Also, the hashtable foreach now uses iter. Also
includes modified test file
There's several use cases for calling into ostree itself to do
mirroring, instead of using bare rsync. For example, it's a bit more
efficient as it doesn't require syncing the objects/ directory.
https://bugzilla.gnome.org/show_bug.cgi?id=728351
If fetching GPG-signed commits over plain HTTP, a MitM attacker can
fill up the drive of targets by simply returning an enormous stream
for the commit object.
Related to this, an attacker can also cause OSTree to perform large
memory allocations by returning enormous GVariants in the metadata.
This helps close that attack by limiting all metadata objects to 10
MiB, so the initial fetch will be truncated.
But now the attack is only slightly more difficult as the attacker
will have to return a correctly formed commit object, then return a
large stream of < 10 MiB dirmeta/dirtree objects.
https://bugzilla.gnome.org/show_bug.cgi?id=725921
I was getting a weird hang in the installed tests with the shell as a
zombie process, not reaped by the parent, which was just stuck in
select() on the output pipes. The thing is we don't actually want to
capture stdout/stderr, we just want to inherit.
GSystem.Subprocess makes that possible, so let's just use it now that
it's a proper installed library.
We don't want to allow MITM attackers to intercept upgrade requests
and provide clients with older OS versions vulnerable to security
flaws.
Only "ostree admin upgrade" gets this behavior for now - whether we
want to do it for "ostree admin switch" is another question.
These GPG tests were failing for me on EL7 - it appears to be because
we had only one directory for both private and public keys, and we
were giving that to ostree for verification, which passed them onto
gpgv.
In EL7 beta at least, gpgv now barfs if it finds a private key where
it is just expecting to find public keys.
Fix this by splitting out the public trusted directory from the
private key directory. Except now for signing, we still need the
public key there, so symlink it. Whee!
This has a very basic level of functionality (deltas can be generated,
and applied offline). There is only some stubbed out pull code to
fetch them via HTTP.
But, better to commit this now and improve it from a known starting
point, rather than have it languish in a branch.
This test had some nondeterminism because we chose a random
object to corrupt, but because there were multiple commits, it
was possible that we chose an object that was not being pulled.
Fix this by writing some custom GJS code to find an explicitly random
object that exists in a given ref, an change a random byte offset.
This adds a lot more randomness to the testing too.
The "ordered hash" code was really just for kernel arguments. And it
turns out it needs to be a multihash (for e.g. multiple console=
arguments).
So turn the OstreeOrderedHash into OstreeKernelArgs, and move the bits
to split key=value and such into there.
Now we're not making this public API yet - the public OstreeSysroot
just takes char **kargs. To facilitate code reuse between ostree/ and
libostree/, make it a noinst libtool library. It'll be duplicated in
the binary and library, but that's OK for now. We can investigate
making OstreeKernelArgs public later.
https://bugzilla.gnome.org/show_bug.cgi?id=721136
The test-admin-deploy-1 was copied into -uboot at some point. But
really they should be testing exactly the same thing, except for
the bootloader backend.
Unify these tests by extracting a common test core.
The official way to add bootloader arguments to the current deployment
is to redeploy with --karg. However, doing so tripped up an
optimization made inside the deployment code to just swap the
bootlinks if we're keeping the same "bootcsum".
Change this optimization to look at the pair of (bootcsum, options).
This large patch moves the core xattr logic down into libgsystem,
which allows the gs_shutil_cp_a() API to copy them. In turn, this
allows us to just use that API instead of rolling our own recursive
copy here.
As noted in the new comment though, one case that we are explicitly
regressing is where the new /etc removes a parent directory that's
needed by a modified file. This seems unlikely for most vendors now,
but let's do that as a separate bug.
https://bugzilla.gnome.org/show_bug.cgi?id=711058
Several APIs in libostree were moved there from the commandline code,
and have hardcoded g_print() for progress and notifications. This
isn't useful for people who want to write PackageKit backends, custom
GUIs and the like.
From what I can tell, there isn't really a winning precedent in GLib
for progress notifications.
PackageKit has the model where the source has GObject properties that
change as async ops execute, which isn't bad...but I'd like something
a bit more general where say you can have multiple outstanding async
ops and sensibly track their state.
So, OstreeAsyncProgress is basically a threadsafe property bag with a
change notification signal.
Use this new API to move the GSConsole usage (i.e. g_print()) out from
libostree/ and into ostree/.
Add a --generate-sizes option to commit to add size information to the
commit metadata. This will be used by higher level code which wants
to determine the total size necessary for downloading.
We need to use the full shutil_rm_rf() in order to actually delete
complete directories.
Test suite code based on a patch from Sjoerd Simons <sjored@luon.net>
https://bugzilla.gnome.org/show_bug.cgi?id=710097
Adapted from Google protobufs. For several cases, we want to support
e.g. file sizes up to guint64, but paying the cost of 8 bytes for each
number is too high.
This will be used for static deltas and sizes metadata.
Colin Walters
Jonathan Lebon
Gatis Paeglis
Simon McVittie
Simon McVittie
Jasper St. Pierre
William Manley
Alexander Larsson
Sjoerd Simons
Dan Nicholson
Yu Qi Zhang
Mathnerd314
Krzesimir Nowak
Giuseppe Scrivano
Gatis Paeglis
Matthew Barnes
Sam Thursfield
Daniel Drake
Owen W. Taylor
Anne LoVerso
Roddy Shuler
Daniel Narvaez
Jeremy Whiting