I'm regretting a bit having the `guint8*csum` variant of checksums
except for the serialized form. Once we start doing processing
it's easier to just have it remain hex.
Do an on-stack conversion for the metadata scanning function; this
drops a malloc and also just looks nicer.
Also add some long-awaited function comments to the two.
Closes: #1240
Approved by: jlebon
I was reading the pull code for the last release, and spotted
a bug in commit f923c2e1ea - in
the case where the ref doesn't exist, we don't set an error,
tripping an assertion in the main code.
The previous code wanted the ref to always exist, so just flip back the boolean
for "ignore noent". I moved the `g_strchomp()` just into the HTTP path - if a
local repo is corrupted in this way it's something to fix in that repo.
Closes: #1238
Approved by: pwithnall
I was trying to do a change for rpm-ostree to use
`OSTREE_REPO_COMMIT_MODIFIER_FLAGS_CANONICAL_PERMISSIONS`
for container builds with `bare-user-only,` but hit an assertion here
ultimtely because we weren't setting `standard::type`.
Rather than hand-rolling `GFileInfo` creation, use the stat buffer conversion
code which is more robust and used in multiple places already.
Closes: #1227
Approved by: jlebon
In particular I'd like to get the copy fix in, since it might affect users for
the keyring bits.
Update submodule: libglnx
Closes: #1225
Approved by: jlebon
I noticed this while debugging why I was seeing "2 metadata objects" fetched for
a different PR. I knew 1 was detached meta, but the other turned out to be this.
There's no reason to request a delta if the ref is unchanged.
Closes: #1220
Approved by: jlebon
I was trying to debug this while working on another PR. Add `echo ok` lines and
also use `assert_file_has_content` more directly; there's no reason to use an
intermediate `grep` since if it fails it won't print the original file.
Closes: #1220
Approved by: jlebon
So the output isn't all intermingled. I just pushed a commit
to add `--log-directory`, so we need to build it from git master
for now.
Closes: #1218
Approved by: jlebon
Followup for recent work in commits:
- 8a7a359709
- 1a9a473580
Keep track of how many objects we imported, and print that for `ostree
pull-local` (also do this even if noninteractive, like we did for `pull`).
In implementing this at first I used separate variables for import
from repo vs import from localcache, but that broke some of the
tests that checked those values.
It's easier to just merge them; we know from looking at whether or not
`remote_repo_local` is set whether or not we were doing a "HTTP pull with
localcache" versus a true `pull-local` and can use that when rendering status.
Closes: #1219
Approved by: jlebon
Our CI uses default Docker, which has SELinux labeling but is rather
evil in returning `EOPNOTSUPP` to any attempts to set `security.selinux`,
even if to the same value.
The previous fire 🔥 for this was: https://github.com/ostreedev/ostree/pull/759
The `bare` repo mode really only makes sense as uid 0, so our installed
test framework is a good match for this. However, the unit tests *do*
work in a privileged container even as non-root, and *also* should
work on SELinux-disabled systems. So let's teach the test framework
how to skip in those situations.
I tested this both in a priv container (my default builder) and an unpriv
container (like our CI).
At the same time, start executing the `test-basic.sh` from an installed test,
so we get better coverage than before.
This is just the start - all of the sysroot tests really need the
same treatment.
Closes: #1217
Approved by: jlebon
I now think commit fab1e113db was a mistake;
because it breaks the mental model that at least I'd built up that "local repos
don't have checksums verified, HTTP does".
For example, a problem with this is (with that mental model in place) it's easy
for people who set up mirrors like this to then do local pulls, and at that
point we've done a deployment with no checksum verification.
Further, since then we did PR #671 AKA commit 3d38f03 which is really most of
the speed hit.
So let's switch the default even for this case to doing checksum verification,
and add `ostree pull --http-trusted`. People who are in situations where they
know they want this can find it and turn it on.
Closes: https://github.com/ostreedev/ostree/issues/1211Closes: #1212
Approved by: jlebon
Rather than carrying two booleans, just convert `OstreeRepoPullFlags`
into `OstreeRepoImportFlags`. This allows us to drop an internal
wrapper function and just directly call `_ostree_repo_import_object()`.
This though reveals that our mirroring import path doesn't check the
`OSTREE_REPO_PULL_FLAGS_UNTRUSTED` flag...it probably should.
Prep for further work.
Closes: #1212
Approved by: jlebon
Make the "local repo" processing conditional the same as the "localcache" bits;
this is really just a de-indent. Also add some comments. Prep for further work.
Closes: #1212
Approved by: jlebon
This fixes up the last of the embarassing bits I saw from
the stack trace in:
https://github.com/ostreedev/ostree/issues/1184
We had a hardlink fast path, but that doesn't apply across
devices, which occurs in two notable cases:
- Installer ISO with local repo
- Tools like pungi that copy the repo to a local snapshot
Obviously there are a lot of subtleties here around things like the
bare-user-only conversions as well as exactly what data we copy. I think to get
better test coverage we may want to add `pull-local --no-hardlink` or so.
Closes: #1197
Approved by: jlebon
For the old `OSTREE_REPO_MODE_ARCHIVE_Z2`. Use it mostly tree
wide except for the repo finder tests (to avoid conflicting with
some outstanding PRs).
Just noted another user coming in some of those tests and wanted to do a
cleanup.
Closes: #1209
Approved by: jlebon
We added a `.dir-locals.el` in commit: 9a77017d87
There's no need to have it per-file, with that people might think
to add other editors, which is the wrong direction.
Closes: #1206
Approved by: jlebon
Such an evil bug 🙈. I was just reading an strace trying to figure out what was
going on, and noticed we had the `XXXXXX` in the lockfile name. It was only
after that I realized that that this might *be* the cause of the skopeo issue.
This is another case where we definitely need more test coverage of things that
actually use the API multiple times in process; might look at dusting off the
work for the rpm-ostree test.
Closes: https://github.com/ostreedev/ostree/issues/1196Closes: #1204
Approved by: jlebon
We keep occasionally regressing this so let's start covering it now. I'm
intentionally not running the tests since that would likely entail a lot more
conditionalizing pain.
Closes: #1194
Approved by: jlebon
While opening a repo we've recorded the device/inode for a while; use it to
avoid calling `linkat()` during object import if we know it's going to fail.
Closes: #1193
Approved by: jlebon
Conceptually `ostree-repo-pull.c` should be be written using
just public APIs; we theoretically support building without HTTP
for people who just want to use the object store portion and
do their own fetching.
We have some nontrivial behaviors in the pull layer though; one
of those is the "bareuseronly" verification. Make a new internal
API that accepts flags, move it into `commit.c`. This
is prep for further work in changing object import to support
reflinks.
Closes: #1193
Approved by: jlebon
There are use cases for not syncing at all; think build cache repos, etc. Let's
be consistent here and make sure if fsync is disabled we do no sync at all.
I chose this opportunity to add tests using the shiny new strace fault
injection. I can forsee using this for a lot more things, so I made
the support for detecting things generic.
Related: https://github.com/ostreedev/ostree/issues/1184Closes: #1186
Approved by: jlebon
Make the equivalent of a `GOnce` or Rust `lazy_static!` detecting this and share
it between the two callers.
Prep for a future similar patch for strace fault injection.
Closes: #1186
Approved by: jlebon
I saw in a stack trace that the main thread was calling `exit()` even while
worker threads were alive and doing sha256/write/fsync etc. for objects.
The stack trace was a SEGV as the main thread was calling into library
`atexit()` handlers and we were a liblz4 destructor:
```
#0 0x00007f2db790f8d4 _fini (liblz4.so.1)
#1 0x00007f2dbbae1c68 __run_exit_handlers (libc.so.6)
```
(Why that library has a destructor I don't know offhand, can't find
it in the source in a quick look)
Anyways, global library destructors and worker threads continuing simply don't
mix. Let's wait for our outstanding operations before we exit. This is also a
good idea for projects using libostree as a shared library, as we don't want
worker threads outliving operations.
Our existing pull corruption tests exercise coverage here.
I added a new `caught-error` status boolean to the progress API, and use it the
commandline to tell the user that we're waiting for outstanding ops.
Closes: #1185
Approved by: jlebon
We have a lot of layers of abstraction here; let's fold in the `trusted`
conditional into the call, since that's all the public API we're using does
anyways.
Prep for a future patch around object copying during imports.
Closes: #1187
Approved by: jlebon
In 5c940987e7 / #646 we
added `--retain-branch-depth`; this adds a symmetric
`--only-branch` for the case where a repo owner just
wants to prune a specific branch.
The implementation here is pretty straightforward; we
just walk all refs and inject the equivalent of
`--retain-branch-depth=$ref=-1` if they're *not* in
`--only-branch`.
Closes: https://github.com/ostreedev/ostree/issues/1115Closes: #1127
Approved by: jlebon
Update libglnx, which is mostly port the repo stagedir code
to the new tmpdir API. This turned out to require some
libglnx changes to support de-allocating the tmpdir ref while
still maintaining the on-disk dir.
Update submodule: libglnx
Closes: #1172
Approved by: jlebon
Doing this in prep for libglnx tmpdir porting, but I think we should also do
this because the partial fetch code IMO was never fully baked; among other
things it was never integrated into the scheme we came up with for "boot id
sync" that we use for complete/staged objects.
There's a lot of complexity here that while we have some coverage for, I think
we need to refocus on the core functionality. The libcurl backend doesn't have
an equivalent to this today.
In particular for small objects, this is simply overly complex. The downside is
clearly for large objects like FAH's 61MB initramfs; not being able to resume
fetches of those is unfortunate.
In practice though, I think most people should be using deltas, and we need to
make sure deltas work for large objects anyways.
Further ultimately the peer-to-peer work should help a lot for people
with truly unreliable connections.
Closes: #1176
Approved by: jlebon
I was looking at fixing an `rpm-ostree livefs` bug where we need to replace
`/usr/lib/passwd`. It's obviously bad if that temporarily disappears 😉. My plan
is to do a subpath checkout of just `/usr/lib/{passwd,group}`.
Make this atomic (i.e. file always exists) by changing the logic to create a
temporary link in repo/tmp, then rename() it into place.
A bonus here is we kill one of the very few (only?) non-error-cleanup i.e.
"non-linear" `goto`s in the ostree codebase.
Closes: #1171
Approved by: jlebon
It turns out that librpm automatically merges identical files between
distinct packages, and this occurs in practice with Fedora today between
`chkconfig` and `initscripts` for exmaple.
Since we added this for rpm-ostree, we basically want to do what librpm does,
let's change the semantics to do a merge. While we're here rename
to `UNION_IDENTICAL`.
Closes: #1156
Approved by: jlebon
We use the "exhaustive enum" pattern (i.e. no `default:`) in some places so
we're forced to touch all users when adding cases.
Closes: #1167
Approved by: peterbaouoft
The new libglnx `glnx_mkdtempat()` uses autocleanups, which
is inconvenient for this use case where we *don't* want autocleanups.
Since we don't need it to be fd-relative, just directly invoke
`g_mkdtemp_full()` which is fine for this use case.
Prep for updating libglnx.
Closes: #1161
Approved by: jlebon
There was only one tricky bit here around the ownership of the lines; I made use
of `g_steal_pointer()` to consistently track ownership, and converted to a `for`
loop while still preserving the loop logic around the last entry.
Closes: #1154
Approved by: jlebon
Steal some code from flatpak for this, which allows porting a few more things to
new style. I started on a public API version of this but was trying to roll some
other things into it and it snowballed. Let's do this version since it's easy
for now.
While here I changed things so that `generate_deployment_refs()` now just uses
`_set_ref_immediate()` rather than requring a txn.
Also, AFAICS there was no test coverage of `generate_deployment_refs()`; I tried
commenting it out and at least `admin-test.sh` passed. Add some coverage of this
- I verified that with this commenting out bits of that function cause the test
to fail.
Closes: #1132
Approved by: jlebon
I resisted trying to do anything invasive here like fd-relative porting as our
coverage is weak. But this was all straightforward porting to decl-after-stmt
style.
Closes: #1153
Approved by: jlebon
There were some important ones there like a random `syncfs()`. The remaining
users are mostly blocked on the "fstatat enoent" case, I'll wait to port those.
Closes: #1150
Approved by: jlebon
A lof of the functions here are async and have nontrivial exits, but these ones
are all sync were straightforward ports.
Not prep for anything, just chipping away at porting.
Closes: #1146
Approved by: jlebon
We have `ot_ensure_unlinked_at()` for the "ignore ENOENT" case, and
`glnx_unlinkat()` otherwise. Port all in-tree callers to one or the other as
appropriate.
Just noticed an unprefixed error in the refs case and decided to do a tree-wide
check.
Closes: #1142
Approved by: jlebon
I'd mostly been skipping the GPG functions due to lack of autoptr for a few
things, but I noticed these bits were straightforward.
Closes: #1136
Approved by: jlebon
The vast majority of invocations of `ot_gpgme_error_to_gio_error()` were paired
with `g_prefix_error()`; let's combine them for the same reason we do
`glnx_throw_errno_prefix()`. For the few cases that don't we might as well add
some prefix.
I also changed it to `return FALSE` in prep for more style porting.
Closes: #1135
Approved by: jlebon
I was reading the pull-local command docs and realized it was somewhat unclear
that `--untrusted` *only* applied to local repo pulls; in other words that we
always treat non-local pulls as untrusted.
Tweak the docstring, and add tests that verify this explicitly.
Closes: #1130
Approved by: jlebon
However, they weren't showing up in the output HTML and I have
no idea why; I looked at what we're doing and it looks close enough
to what's going on in `GDBusConnection` that I was using as a reference.
I'm not going to spend a lot of time to debug it right now.
Closes: #1140
Approved by: jlebon
Followup from previous patch - we can now centralize the sysroot loading.
Besides the obvious cleanup value, this is also prep for dropping an
`ostree_sysroot_get_path()` user.
Closes: #1123
Approved by: jlebon
This is exactly analogous to the `ostree init` case where
we have `OSTREE_BUILTIN_FLAG_NO_REPO` to avoid trying to load
a repo when we're creating one.
Let's avoid the pointless sysroot for `init-fs`; among other
things this will then let us do `ostree_sysroot_load()` inside
the argument parsing, and drop it from every other user.
Closes: #1123
Approved by: jlebon
Rather than calling `ostree_sysroot_get_path()`, which I'd like to deprecate for
the same reason as `ostree_repo_get_path()`.
Closes: #1123
Approved by: jlebon
In almost all places. There are just a few exceptions; one tricky bit for
example is that the repo config must still have `mode=archive-z2`, since
`archive` used to mean something else. (We could very likely just get rid of
that check, but eh, later).
I also added a test that one can still do `ostree repo init --mode=archive-z2`.
Closes: #1125
Approved by: jlebon
For rpm-ostree, I want to move RPM files in `/boot` to `/usr/lib/ostree-boot`.
This is currently impossible without forking the libarchive code. Supporting
this is pretty straightforward; we already had pathname translation in
the libarchive code, we just need to expose it as an option.
On the command line side, I chose to wrap this as a regexp. That should be good
enough for a lot of use cases; sophisticated users should as always be making
use of the API. Note that this required some new `#ifdef LIBARCHIVE` bits to use
the new API. Following previous patterns here, we use the new API only if a
relevant option is enabled, ensuring unit test coverage of both paths.
For the test cases, I ended up changing the accounting to avoid having to
multiply the test count.
Closes: #1105
Approved by: jlebon
This was really straightforward to implement, and is useful
for dev/test scenarios mainly like we have in rpm-ostree at least.
Closes: https://github.com/ostreedev/ostree/issues/1113Closes: #1114
Approved by: jlebon
Things like https://sourceware.org/libabigail/manual/abidiff.html
look interesting but in a brief look I couldn't work out
how to conveniently use them for quick ABI sanity checking without
doing a diff from a previous build (which we could do but would be
more involved).
This way will at least catch struct ABI breaks on x86_64 which
I think we'd be most likely to do accidentally when trying
to use one of the previous unused values.
I found the hole values via gdb's `pahole` command.
Closes: #1108
Approved by: jlebon
It's been this way for a long time, though not forever; I went
back to v2014.11 as a random choice and it worked then. Not
going to bother doing a full archive search for this though.
Anyone who wants more context can help themselves.
Closes: https://github.com/ostreedev/ostree/issues/1096Closes: #1106
Approved by: jlebon
This triggers obscure bugs; really we shouldn't be overriding
the global locale here. In practice, production build systems
will be using fixed locales anyways.
Also, we only use a small subset of this file (`make syntax-check`),
which appears to work OK without this.
I will probably try to work out where to submit this as at
least an issue report for upstream gnulib.
Closes: https://github.com/ostreedev/ostree/issues/1101Closes: #1107
Approved by: heftig
This makes `ostree commit --tree=tar` honor `--owner-uid` and `--owner-gid`
for the root directory.
Prep for further commit filtering work, although mostly for the unit test cases;
this ensures we can use `ostree checkout` after autocreating a root directory.
Closes: #1104
Approved by: jlebon
The wrapping here is unnecessary, since `_ostree_repo_commit_modifier_apply()`
already does what this function did. Further, the return type was wrong.
Saw this while doing some libarchive work.
Closes: #1104
Approved by: jlebon
For both flatpak and ostree-as-host, we really want to verify up front during
pulls that we're not being downgraded. Currently both flatpak and
`OstreeSysrootUpgrader` do this before deployments, but at that point we've
already downloaded all the data, which is annoying.
Closes: https://github.com/ostreedev/ostree/issues/687Closes: #1055
Approved by: jlebon
Follow up to <https://github.com/ostreedev/ostree/pull/1079>; I was working on
the rpm-ostree updates for this, and I think it's more consistent if we have
`.img` here, since that's a closer match to the "remove $kver" that results in
`vmlinuz`. Also just best practice to have file suffix types where they make
sense.
The astute reader might notice this sneaks in a change where we'd crash if the
legacy bootdir didn't have an initramfs...yeah, should probably have test
coverage of that.
Closes: #1095
Approved by: jlebon
I find "libOSTree" awkward to type and really to look at. Let's be nicer on
people's pinky fingers and eyes and drop it all down to lowercase.
Closes: #1093
Approved by: jlebon
This is the new Fedora kernel standard layout; it has the advantage
of being in `/usr` like `/usr/lib/ostree-boot`, but it's not OSTree
specific.
Further, I think in practice forcing tree builders to compute the checksum is an
annoying stumbling block; since we already switched to e.g. computing checksums
always when doing pulls, the cost of doing another checksum for the
kernel/initramfs is tiny. The "bootcsum" becomes more of an internal
implementation detail.
Now, there is a transition; my current thought for this is that rpm-ostree will
change to default to injecting into both `/usr/lib/ostree-boot` and
`/usr/lib/modules`, and stop doing `/boot`, then maybe next year say we drop the
`/usr/lib/ostree-boot` by default.
A twist here is that the default Fedora kernel RPM layout (and what's in
rpm-ostree today) includes a kernel but *not* an initramfs in
`/usr/lib/modules`. If we looked only there, we'd just find the kernel. So we
need to look in both, and then special case this - pick the legacy layout if we
have `/usr/lib/modules` but not an initramfs.
While here, rework the code to have an `OstreeKernelLayout` struct which makes
dealing with all of the variables nicer.
Closes: #1079
Approved by: jlebon
I noticed this with a local build of an RPM:
```
/usr/include/glib-2.0/glib/glib-autocleanups.h:28:3: warning: 'help' may be used uninitialized in this function [-Wmaybe-uninitialized]
g_free (*pp);
^~~~~~~~~~~~
src/ostree/ot-main.c:82:20: note: 'help' was declared here
g_autofree char *help;
^~~~
```
Closes: #1091
Approved by: jlebon
This is simplest for now. Compare with similar logic from
`/usr/lib/tmpfiles.d/tmp.conf`:
```
R! /tmp/systemd-private-*
```
Closes: https://github.com/ostreedev/ostree/issues/393Closes: #1090
Approved by: jlebon
I actually don't quite know what the version inheritance really does, but let's
be safe and fix this. I'm being conservative here and fixing it to inherit from
2017.8, skipping .9 since that doesn't have a parent.
Related: https://github.com/ostreedev/ostree/issues/1087Closes: #1088
Approved by: jlebon
In the production case since we used `daemon()` our stderr is `/dev/null`¹
there's not much use in logging errors from `FITHAW` or `exit(1)`, and doing so
breaks the test suite which checks the return from `waitpid()`. There's nothing
we can really do if `FITHAW` fails, and in most of those cases `EINVAL`,
`EOPNOTSUPP`, we *shouldn't* do anything anyways.
¹ Though perhaps we should set up the systemd journal, but let's not
go there right now.
Closes: #1084
Approved by: jlebon
I added `waitpid()`, but that didn't actually help because we were
`daemon()`izing. Don't daemonize if we're testing so that we can `waitpid()`.
Note I still haven't reproduced this race locally, but I'm pretty sure this will
fix it.
While here, actually check the return value from `waitpid()` just in case
something goes wrong there.
Closes: #1084
Approved by: jlebon
This is in line with the "/etc is for sysadmins", "/usr is OS" model;
e.g. systemd's bash completions go there.
Making this change since I was looking at the required spec file changes.
Closes: #1083
Approved by: mbarnes
This is mostly the `copy_file_range` changes plus the Coverity files.
```
Colin Walters (4):
localalloc: Abort on EBADF from close() by default
local-alloc: Remove almost all macros like glnx_free, glnx_unref_variant
console: Fix Coverity NULL deref warning
fdio: Merge systemd code to use copy_file_range(), use FICLONE
Jonathan Lebon (1):
console: trim useless check
Matthew Leeds (1):
dirfd: Fix typo in comment
Philip Withnall (1):
glnx-console: Add missing NULL check before writing out text
```
Update submodule: libglnx
Closes: #1081
Approved by: jlebon
Colin Walters