Commit Graph
100 Commits
Author SHA1 Message Date
Colin Walters 9d82d258ba tests: Add a "pull-test2" that uses slightly more realistic content
This one actually exercises the delta paths more; I pulled the
data from some local F26AH builds.  One thing I noticed while doing
that is that we try bsdiff on xz'd data, but that won't be useful.
https://github.com/projectatomic/rpm-ostree/issues/470
https://bugzilla.redhat.com/show_bug.cgi?id=1367496

Anyways I verified this test fails without the fixup for `pread()`
in https://github.com/ostreedev/ostree/pull/1312

Closes: #1314
Approved by: jlebon
2017-11-01 17:38:26 +00:00
Colin Walters aec1191fa9 deltas: Add an if0'd bit to print bsdiff objects
I used this to find the bsdiff'd objects.

Closes: #1314
Approved by: jlebon
2017-11-01 17:38:26 +00:00
Colin Walters e885d029aa bin/delta-compilation: Add a "progress bar" for bsdiff generation
It's the slowest part, let's show admins something. This "update every 10%" code
was copied from the fsck command; obviously a better approach would be "progress
every N seconds" but doing that somewhat accurately requires making things
async; not worth it here yet.

Closes: #1314
Approved by: jlebon
2017-11-01 17:38:26 +00:00
Colin Walters 62445166fc lib/pull: Delete unused "exampleos" code
We never ended up using this, and I'm going to revisit this in another patch
with a different approach that has useful *content* and not just a lot of files.

Closes: #1314
Approved by: jlebon
2017-11-01 17:38:26 +00:00
Colin Walters 80ff73ba26 lib/pull: When --require-static-deltas, use them even for file:/// repos
I didn't fully spelunk this, but from what `static-delta-generate-crosscheck.sh`
had, we appeared to be doing this before, and it's clearly useful for local
testing rather than needing to spin up a HTTP server.

Closes: #1313
Approved by: jlebon
2017-10-31 14:20:20 +00:00
Colin Walters 7f8ff5756e lib: Minor static delta fixes
First, the manual crosscheck script bitrotted; it got caught up
in the "use libtest repo creation wrapper" bit, and also it
seems like at some point `pull --require-static-deltas` changed
meaning when dealing with `file:///` repos.  I have more work to
unwind that.

Next, I'm seeing a delta failure which looks like a static delta
miscompilation with rollsums; change the compiler to print out
the source object too, which helped me debug this.

And finally in the processing code, fix incorrect error prefixing, which was
misleading.

Closes: #1311
Approved by: ashcrow
2017-10-30 22:52:02 +00:00
Colin Walters 0d259ac401 lib/deltas: Fix change to use pread() in write opcode
Fixes: 93457071cb "lib/deltas: Use pread() instead of lseek()+read()"

Caught this when trying to test alex's patch locally. I am going to review our
static delta pulls and try to get something more comprehensive locally. But in
the meantime this patch is clearly right.

Closes: #1312
Approved by: jlebon
2017-10-30 19:03:16 +00:00
Colin Walters 90ebd48f6a lib/repo: Fix loading commitstate with parent repos
This makes the code nicer too. Properly unit testing this though really wants
like a whole set of stuff around parent repos...but we do have coverage of the
non-parent path in the current pull tests.

Closes: https://github.com/ostreedev/ostree/issues/1306

Closes: #1308
Approved by: alexlarsson
2017-10-26 07:06:50 +00:00
Colin Walters d76840d0c1 tree-wide: Use autoptr for OstreeKernelArgs
Much nicer looking.  Prep for more cleanup from
https://github.com/projectatomic/rpm-ostree/pull/1013

Closes: #1302
Approved by: peterbaouoft
2017-10-25 01:48:56 +00:00
Colin Walters b11d1aa3f2 ci: Update centos container to current again
Repo sync issue should be cleared now, and for some reason the previous
container is already GC'd.

Closes: #1304
Approved by: peterbaouoft
2017-10-24 14:59:21 +00:00
Colin Walters ed15723cd1 lib/commit: Fix hardlink checkout commit with bare-user + mod xattrs
This is more subtle fallout from:
https://github.com/ostreedev/ostree/pull/1170
AKA commit: 8fe4536257

Before, if we found a devino cache hit, we'd use it unconditionally.

Recall that `bare-user` repositories are very special in that they're the only
mode where the on disk state ("physical state") is not the "real" state. The
latter is stored in the `user.ostreemeta` xattr. (`bare-user` repos are also
highly special in that symlinks are regular files physically, but that's not
immediately relevant here).

Since we now have `bare-user-only` for the "pure unprivileged container" case,
`bare-user` should just be used for "OS builds" which have nonzero uids (and
possibly SELinux labels etc.)

In an experimental tool I'm writing "skopeo2ostree" which imports OCI images
into refs, then squashes them together into a single final commit, we lost the
the `81` group ID for `/usr/libexec/dbus-1/dbus-daemon-launch-helper`.

This happened because the commit code was loading the "physical" disk state,
where the uid/gid are zero because that's the uid I happened to be using. We
didn't just directly do the link speedup because I was using `--selinux-policy`
which caused the xattrs to change, which caused us to re-commit objects from the
physical state.

The unit test I added actually doesn't quite trigger this, but I left
it because "why not".  Really testing this requires the installed test
which uses SELinux policy from `/`.

The behavior without this fix looks like:

```
-00755 0 0     12 { [(b'user.ostreemeta', [byte 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x51, 0x00, 0x00, 0x81, 0xed]), (b'security.selinux', b'system_u:object_r:lib_t:s0')] } /usr/lib/dbus-daemon-helper
```

which was obviously totally broken - we shouldn't be picking up the
`user.ostreemeta` xattr and actually committing it of course.

Closes: #1297
Approved by: jlebon
2017-10-23 17:02:28 +00:00
Colin Walters 795a953a36 core: Add standard SOURCE_TITLE metadata key
This is a freeform string useful to track/display when a commit is "derived"
from some other format.  For example, in the rpm-ostree test we make a
`vmcheck` ref that conceptually overlays the default ref like
`fedora-atomic:fedora/26/x86_64/atomic-host`.

My current patch sets the source title to e.g.
"Dev overlay on fedora-atomic:fedora/26/x86_64/atomic-host".

Another case I'm working on now is importing OCI images to use
as host images.  For that case, the source title is
With this patch we could then set the original OCI image name + tag
as the source name, like:
"oci:cgwalters/demo-custom-fedora-atomic-host:26".

Closes: #1296
Approved by: jlebon
2017-10-23 14:19:41 +00:00
Colin Walters 0e0fe85922 bin/status: Inline metadata parsing for versions
Pull out the commit metadata explicitly; still just rendering the version, but
this is prep for rendering other metadata keys.

Closes: #1296
Approved by: jlebon
2017-10-23 14:19:41 +00:00
Colin Walters 32357cef4d bin/status: Port to new style
No functional changes; prep for more work.

Closes: #1296
Approved by: jlebon
2017-10-23 14:19:41 +00:00
Colin Walters 4cda9f14fb docs/related-projects: Add Google Usenix link
Closes: https://github.com/ostreedev/ostree/issues/1284

Closes: #1299
Approved by: cgwalters
2017-10-21 14:12:02 +00:00
Colin Walters 1222c2271b repo: Add wrapper function for setting devino cache on checkout opts
I was trying to use this with pygobject for an OCI+ostree project, and pygobject
rejected simply assigning to the field (understandably, since it can't bind the
lifetime together).

Add a wrapper function, which is still unsafe, but hides that unsafety
where most people shouldn't find it.  And if they do...well, sorry,
Rust wasn't invented when ostree was started.

Closes: #1295
Approved by: pwithnall
2017-10-20 18:20:19 +00:00
Colin Walters b8251d26bd lib/checkout: For "process whiteouts" mode, replace directories too
I'm playing around with some ostree ⇔ OCI/Docker bits, and ran
into this while importing an OCI image that built from the Fedora
base image where `/home` is a regular directory, and I added a layer
that did the ostree bits of moving it to `/var` and leaving a symlink.

OCI/Docker supports this.  Now since "process whiteouts" is really the
"enable OCI/Docker" mode, let's only replace dirs if that's enabled.
This leaves the `UNION_FILES` targeted for its original use case
which is unioning components/packages.  (Although that use case itself
is now a bit superceded by `UNION_IDENTICAL`, but eh).

Closes: #1294
Approved by: jlebon
2017-10-20 13:20:27 +00:00
Colin Walters e466e482b1 Disallow refs starting with a non-letter or digit
Change the regexp for validating refs to require at least one letter or digit
before allowing the other special chars in the set `[.-_]`. Names that start
with `.` are traditionally Unix hidden files; let's ignore them under the
assumption they're metadata for some other tool, and we don't want to
potentially conflict with the special `.` and `..` Unix directory entries.
Further, names starting with `-` are problematic for Unix cmdline option
processing; there's no good reason to support that. Finally, disallow `_` just
on general principle - it's simpler to say that ref identifiers must start with
a letter or digit.

We also ignore any existing files (that might be previously created refs) that
start with `.` in the `refs/` directory - there's a Red Hat tool for content
management that injects `.rsync` files, which is why this patch was first
written.

V1: Update to ban all refs starting with a non-letter/digit, and
    also add another call to `ostree_validate_rev` in the pull
    code.

Closes: https://github.com/ostreedev/ostree/issues/1285

Closes: #1286
Approved by: jlebon
2017-10-18 20:55:43 +00:00
Colin Walters a2f8315eae lib/commit: (refactor) Clean up delta bare write API
The way `_ostree_repo_open_content_bare()` did both looking for the object and
possibly creating a new fd was just weird and inconsistent with e.g. the pull
code where we always call `has_object()` first.

Just call `has_object()` in the delta paths that used this too, making the
implementation right now a thin wrapper around
`glnx_open_tmpfile_linkable_at()`, but this is prep for a later patch which does
more.

Closes: #1283
Approved by: jlebon
2017-10-18 14:07:55 +00:00
Colin Walters 40a0b9fb73 lib/repo: Update summary code to use newer hashing API
And drop the unnecessary wrapper.

Closes: #1287
Approved by: jlebon
2017-10-18 13:27:11 +00:00
Colin Walters de5f781089 lib/pull: Update summary checksum code to use new checksum API
This is another case where making an input stream out of a memory buffer is a
bit silly; just hash the `GBytes` directly.

Closes: #1287
Approved by: jlebon
2017-10-18 13:27:11 +00:00
Colin Walters 9955695da3 syntax-check: Add a rule to enforce glnx_autofd over glnx_fd_close
And fix the one final use.

Closes: #1280
Approved by: jlebon
2017-10-17 16:43:02 +00:00
Colin Walters 3577b4a6c6 lib/commit: Use direct fd xattr operations again on regular files
A side effect of commit 8fe4536257 is that
we started listing all xattrs even for files with device/inode matches;
further, we did that using the dfd/name which means we went through
the `/proc` path, which is slower and uglier.

Noticed this in strace while looking at adoption code.

Closes: #1280
Approved by: jlebon
2017-10-17 16:43:02 +00:00
Colin Walters eeabd4baf7 lib/commit: Fix indentation in file commit code
No functional changes; the indentation was off here and it was
confusing me working on another patch.

Closes: #1280
Approved by: jlebon
2017-10-17 16:43:02 +00:00
Colin Walters bc7ff2cd1d lib/commit: Avoid trying to delete . with _CONSUME flag
This helps port rpm-ostree.

Closes: #1278
Approved by: jlebon
2017-10-17 16:24:13 +00:00
Colin Walters 93457071cb lib/deltas: Use pread() instead of lseek()+read()
That's why the syscall was invented, so let's use it. Just noticed while reading
the code while working on another patch.

Closes: #1270
Approved by: jlebon
2017-10-17 15:20:08 +00:00
Colin Walters bb05b187b6 lib/deltas: Use fastpath for regfile writes for bare-user-only
Noticed this omission while looking at the code while working on another patch.

Closes: #1270
Approved by: jlebon
2017-10-17 15:20:08 +00:00
Colin Walters 95afe2848d tests: Add test-pull-bareuseronly
I was going to fix a bug in the static deltas code and I noticed
we were missing `pull-test.sh` coverage for bareuseronly 🙈.

Obviously fixing this requires duplicating some of the bits we have in
`basic-test.sh`; need to hoist that into `libtest.sh`. For now though let's get
the coverage.

Closes: #1270
Approved by: jlebon
2017-10-17 15:20:08 +00:00
Colin Walters 464965e6b4 lib/sysroot: Fix error handling when mounting overlayfs fails
This isn't perfect, but at least we fix an error-overwrite error, and in
practice `ostree admin unlock` isn't wrapped by `rpm-ostree` yet, so spew to
stderr is OK.

Closes: https://github.com/ostreedev/ostree/issues/1273

Closes: #1279
Approved by: guyshapiro
2017-10-17 06:58:22 +00:00
Colin Walters e744f2ad6f lib: Use a common helper function to compare checksums
So we get a consistent error message; came up in a PR review.

Closes: #1277
Approved by: jlebon
2017-10-17 05:06:07 +00:00
Colin Walters 16c31a9b58 lib/commit: Implement "adoption" with CONSUME flag
For checkouts that are on the same device, for regular files we can simply
"adopt" existing files. This is useful in the "build from subtrees" pattern that
happens with e.g. `rpm-ostree install` as well as flatpak and gnome-continuous.

New files are things like an updated `ldconfig` cache, etc. And particularly for
`rpm-ostree` we always regenerate the rpmdb, which for e.g. this workstation is
`61MB`.

We probably should have done this from the start, and instead had a `--copy`
flag to commit, but obviously we have to be backwards compatible.

There's more to do here - the biggest gap is probably for `bare-user` repos,
which are often used with things like `rpm-ostree compose tree` for host
systems. But we can do that later.

Closes: #1272
Approved by: jlebon
2017-10-16 18:22:09 +00:00
Colin Walters 729790bedc tests/basic: Add missing ${COMMIT_ARGS} for bare-user-only
I was working on "adopt" and hit corruption; turns out we were missing the
`--canonical-permissions` arg in this existing test.

(Need to abstract all of this more)

Closes: #1272
Approved by: jlebon
2017-10-16 18:22:09 +00:00
Colin Walters 1c108d1b68 lib/pull: Also do commit GPG verification before writing
I was working on a patch to do build on the work done to
import content objects async to do the same for metadata, but right
now we basically rely on writing them first to do the GPG verification
when scanning.

Things will be cleaner for that if we can pass the commit object directly into
`scan_commit_object()` and consistently use `gpg_verify_unwritten_commit()`.

We're careful here to continue to do it both ways (but at most one time), to
account for the case where a bad commit has been pulled and written - we need to
keep failing GPG verification there.

Closes: #1269
Approved by: jlebon
2017-10-16 15:32:39 +00:00
Colin Walters 22029d71c9 lib/pull: Drop duplicate/different error for GPG but no detached meta
Prep for a later patch to do GPG verification before writing commit objects;
`_ostree_repo_gpg_verify_with_metadata()` already handles this, and so dropping
this gives us consistent error messages.

Closes: #1269
Approved by: jlebon
2017-10-16 15:32:39 +00:00
Colin Walters a1986b1a80 lib/deltas: Check cancellable during processing
Let's react to `Ctrl-C` faster here. Noticed while I was doing an update on my
desktop and playing with cancellation.

Closes: #1266
Approved by: jlebon
2017-10-12 15:04:02 +00:00
Colin Walters 8f6ec62bfb lib/deploy: Use _exit() for FIFREEZE watchdog
This works around an (IMO) SpiderMonkey bug - it tries to
clean up in a shared library destructor, but doesn't install a
`pthread_atfork()` handler to unset its state.

Closes: https://github.com/ostreedev/ostree/issues/1262

Closes: #1264
Approved by: dbnicholson
2017-10-11 21:24:39 +00:00
Colin Walters 1825f03fe7 tree-wide: Update to new libglnx fd APIs
This ends up a lot better IMO.  This commit is *mostly* just
`s/glnx_close_fd/glnx_autofd`, but there's also a number of hunks like:

```
-  if (self->sysroot_fd != -1)
-    {
-      (void) close (self->sysroot_fd);
-      self->sysroot_fd = -1;
-    }
+  glnx_close_fd (&self->sysroot_fd);
```

Update submodule: libglnx

Closes: #1259
Approved by: jlebon
2017-10-11 19:26:10 +00:00
Colin Walters 3e3d28632d lib/commit: Make -path commit helper API private
It's no longer called directly by the pull code, so make it static.

The goal here is to have the pull and local-fs commit paths use higher level
more efficient APIs, and eventually make those APIs public.

Closes: #1257
Approved by: jlebon
2017-10-11 19:04:46 +00:00
Colin Walters bb51a43d81 lib/core: Use GBytes for file headers
This simplifies a lot of code; the header function was structured
to write to an input stream, but many callers only wanted the checksum,
so it's simpler (and error-free) to simply allocate a whole buffer
and checksum that.

For the callers that want to write it, it's also still simpler to allocate the
buffer and write the whole thing rather than having this function do the
writing.

A lot of the complexity here again is a legacy of the packfile code, which is
dead.

This is prep for faster regfile commits where we can avoid `G{In,Out}putStream`.

Closes: #1257
Approved by: jlebon
2017-10-11 19:04:46 +00:00
Colin Walters d10593e65d lib/core: Port a few functions to decl-after-stmt
No functional changes, just prep for more work.

Closes: #1257
Approved by: jlebon
2017-10-11 19:04:46 +00:00
Colin Walters cd8fc8e37a lib/core: (refactor) Drop wrapper and unneeded args for variant writing
Nothing was using the `bytes_written` data (we always discard partially written
tmpfiles), so simplify everything by dropping it. Further, we always passed an
offset of `0`, so drop that argument too. (I believe that this was previously
used by the "pack files" code that we deleted long ago)

Second, we had an unnecessary internal wrapper for this function; drop that too.

Closes: #1257
Approved by: jlebon
2017-10-11 19:04:46 +00:00
Colin Walters 1c9975cbd1 lib: Add a lighter weight internal checksum wrapper
The faster (OpenSSL/GnuTLS) code lived in a `GInputStream` wrapper, and that
adds a lot of weight (GObject + vtable calls). Move it into a simple
autoptr-struct wrapper, and use it in the metadata path, so we're
now using the faster checksums there too.

This also drops a malloc there as the new API does hexdigest in place to a
buffer.

Prep for more work in the commit path to avoid `GInputStream` for local file
commits, and ["adopting" files](https://github.com/ostreedev/ostree/pull/1255).

Closes: #1256
Approved by: jlebon
2017-10-10 21:25:40 +00:00
Colin Walters bba7eb8069 commit: Add _CONSUME modifier flag
For many cases of commit, we can actually optimize things by simply "adopting"
the object rather than writing a new copy. For example, in rpm-ostree package
layering.

We can only make that optimization though if we take ownership of the file. This
commit hence adds an API where a caller tells us to do so. For now, that just
means we `unlink()` the files/dirs as we go, but we can now later add the
"adopt" optimization.

Closes: #1255
Approved by: jlebon
2017-10-10 13:02:08 +00:00
Colin Walters fc33ae018d lib/deltas: Move variant read helper into the deltas code
What the deltas code is doing is weird/unfortunate.  The name
`ot_variant_read()` conflicts too much with `ot_variant_read_fd()`.
Since nothing else uses it, move it into the deltas code.

Closes: #1254
Approved by: jlebon
2017-10-05 15:41:00 +00:00
Colin Walters a903f96673 lib/util: Delete some unused functions
Hooray, dead code.

Closes: #1254
Approved by: jlebon
2017-10-05 15:41:00 +00:00
Colin Walters 2e3889a4eb lib/pull: Change fetcher to return O_TMPFILE
A lot of the libostree code is honestly too complex for its
own good (this is mostly my fault).  The way we do HTTP writes
is still one of those.  The way the fetcher writes tempfiles,
then reads them back in is definitely one of those.

Now that we've dropped the "partial object" bits in:
https://github.com/ostreedev/ostree/pull/1176 i.e. commit
https://github.com/ostreedev/ostree/commit/0488b4870e80ef575d8b0edf6f2a9e5ad54bf4df
we can simplify things a lot more by having the fetcher
return an `O_TMPFILE` rather than a filename.

For trusted archive mirroring, we need to enable linking
in the tmpfiles directly.

Otherwise for at least content objects they're compressed, so we couldn't link
them in. For metadata, we need to do similar logic to what we have around
`mmap()` to only grab a tmpfile if the size is large enough.

Closes: #1252
Approved by: jlebon
2017-10-05 14:58:20 +00:00
Colin Walters 7f6af94c5a lib/utils: Port a bit to decl-after-stmt style
Add add some more comments.

Closes: #1247
Approved by: jlebon
2017-10-05 13:28:59 +00:00
Colin Walters 5c7d2dd8be Deduplicate and fix up our use of mmap()
Buried in this large patch is a logical fix:

```
-  if (!map)
-    return glnx_throw_errno_prefix (error, "mmap");
+  if (map == (void*)-1)
+    return glnx_null_throw_errno_prefix (error, "mmap");
```

Which would have helped me debug another patch I was working
on.  But it turns out that actually correctly checking for
errors from `mmap()` triggers lots of other bugs - basically
because we sometimes handle zero-length variants (in detached
metadata).  When we start actually returning errors due to
this, things break.  (It wasn't a problem in practice before
because most things looked at the zero size, not the data).

Anyways there's a bigger picture issue here - a while ago
we made a fix to only use `mmap()` for reading metadata from disk
only if it was large enough (i.e. `>16k`).  But that didn't
help various other paths in the pull code and others that were
directly doing the `mmap()`.

Fix this by having a proper low level fs helper that does "read all data from
fd+offset into GBytes", which handles the size check. Then the `GVariant` bits
are just a clean layer on top of this. (At the small cost of an additional
allocation)

Side note: I had to remind myself, but the reason we can't just use
`GMappedFile` here is it doesn't support passing an offset into `mmap()`.

Closes: #1251
Approved by: jlebon
2017-10-04 20:42:39 +00:00
Colin Walters 351ffdb977 lib/sysroot: Fix pointer going out of scope in unlock code
Fixes Coverity CID #1457317

Closes: #1249
Approved by: jlebon
2017-10-04 14:56:31 +00:00
Colin Walters e80efe0b06 lib/core: Init struct stat buffer
Regression from d57410a7e6

Fixes Coverity CID #1457316

Closes: #1249
Approved by: jlebon
2017-10-04 14:56:31 +00:00
Colin Walters be100e0ee2 lib/pull: Minor cleanup to metadata scanning function, add docs
I'm regretting a bit having the `guint8*csum` variant of checksums
except for the serialized form.  Once we start doing processing
it's easier to just have it remain hex.

Do an on-stack conversion for the metadata scanning function; this
drops a malloc and also just looks nicer.

Also add some long-awaited function comments to the two.

Closes: #1240
Approved by: jlebon
2017-10-02 19:55:54 +00:00
Colin Walters 9fffc27cfb lib/uboot: Port to fd-relative
Prep for `ostree_sysroot_new_at()`.

Closes: #1237
Approved by: jlebon
2017-10-02 16:19:59 +00:00
Colin Walters fcb1e87f82 lib/syslinux: Port to fd-relative
Prep for `ostree_sysroot_new_at()`.  Also just generally nicer.

Closes: #1237
Approved by: jlebon
2017-10-02 16:19:59 +00:00
Colin Walters b8c15ae859 lib/pull: Fix regression with pull-local for nonexistent refs
I was reading the pull code for the last release, and spotted
a bug in commit f923c2e1ea - in
the case where the ref doesn't exist, we don't set an error,
tripping an assertion in the main code.

The previous code wanted the ref to always exist, so just flip back the boolean
for "ignore noent". I moved the `g_strchomp()` just into the HTTP path - if a
local repo is corrupted in this way it's something to fix in that repo.

Closes: #1238
Approved by: pwithnall
2017-10-02 16:02:28 +00:00
Colin Walters 06c731bfa3 lib/sysroot: Add some more comments
Also avoid gtk-doc style for private functions, as it tries to parse them and
complains since they aren't public.

Closes: #1230
Approved by: jlebon
2017-10-02 15:12:09 +00:00
Colin Walters 7da4c2162d lib/commit: Add some gtk-doc and internal doc comments
Just making more of an effort for this for obvious reasons. We had a few public
APIs not documented too.

Closes: #1230
Approved by: jlebon
2017-10-02 15:12:09 +00:00
Colin Walters 64f3257d88 Release 2017.12
Closes: #1234
Approved by: jlebon
2017-10-02 14:44:34 +00:00
Colin Walters a7f04a14b5 libarchive: Fix assertion error if used with CANONICAL_PERMISSIONS
I was trying to do a change for rpm-ostree to use
`OSTREE_REPO_COMMIT_MODIFIER_FLAGS_CANONICAL_PERMISSIONS`
for container builds with `bare-user-only,` but hit an assertion here
ultimtely because we weren't setting `standard::type`.

Rather than hand-rolling `GFileInfo` creation, use the stat buffer conversion
code which is more robust and used in multiple places already.

Closes: #1227
Approved by: jlebon
2017-09-28 19:59:24 +00:00
Colin Walters 15096ac49e tests: Add 404 tests for dirtree objects too
I now believe the flatpak issue we were hitting was
https://github.com/ostreedev/ostree/pull/1185
but let's add these tests anyways for more coverage.

Closes: #888
Approved by: jlebon
2017-09-28 13:05:55 +00:00
Colin Walters aa067aeafa tree-wide: Bump libglnx, port to new lockfile init
In particular I'd like to get the copy fix in, since it might affect users for
the keyring bits.

Update submodule: libglnx

Closes: #1225
Approved by: jlebon
2017-09-27 20:08:34 +00:00
Colin Walters 5b860fba05 lib/pull: Don't request deltas for unchanged commits
I noticed this while debugging why I was seeing "2 metadata objects" fetched for
a different PR. I knew 1 was detached meta, but the other turned out to be this.

There's no reason to request a delta if the ref is unchanged.

Closes: #1220
Approved by: jlebon
2017-09-27 19:04:11 +00:00
Colin Walters 3314140415 tests/commit-sign: Update a bit to more modern style
I was trying to debug this while working on another PR. Add `echo ok` lines and
also use `assert_file_has_content` more directly; there's no reason to use an
intermediate `grep` since if it fails it won't print the original file.

Closes: #1220
Approved by: jlebon
2017-09-27 19:04:11 +00:00
Colin Walters d319e75982 lib/diff: Add compile-time ABI check on 64 bit arches
Like what was done for most of the `ostree-repo.h` values.  Prep
for adding a new option.

Closes: #1223
Approved by: jlebon
2017-09-27 18:20:10 +00:00
Colin Walters b71a913e21 ci: Record primary context results using g-d-t-r --log-directory
So the output isn't all intermingled.  I just pushed a commit
to add `--log-directory`, so we need to build it from git master
for now.

Closes: #1218
Approved by: jlebon
2017-09-27 18:08:06 +00:00
Colin Walters c6f972406e lib/pull: Add status for imported objects
Followup for recent work in commits:

 - 8a7a359709
 - 1a9a473580

Keep track of how many objects we imported, and print that for `ostree
pull-local` (also do this even if noninteractive, like we did for `pull`).

In implementing this at first I used separate variables for import
from repo vs import from localcache, but that broke some of the
tests that checked those values.

It's easier to just merge them; we know from looking at whether or not
`remote_repo_local` is set whether or not we were doing a "HTTP pull with
localcache" versus a true `pull-local` and can use that when rendering status.

Closes: #1219
Approved by: jlebon
2017-09-27 15:35:11 +00:00
Colin Walters 5963d5a2a9 tests,ci: Move "test-basic" (bare mode) to installed test
Our CI uses default Docker, which has SELinux labeling but is rather
evil in returning `EOPNOTSUPP` to any attempts to set `security.selinux`,
even if to the same value.

The previous fire 🔥 for this was: https://github.com/ostreedev/ostree/pull/759

The `bare` repo mode really only makes sense as uid 0, so our installed
test framework is a good match for this.  However, the unit tests *do*
work in a privileged container even as non-root, and *also* should
work on SELinux-disabled systems.  So let's teach the test framework
how to skip in those situations.

I tested this both in a priv container (my default builder) and an unpriv
container (like our CI).

At the same time, start executing the `test-basic.sh` from an installed test,
so we get better coverage than before.

This is just the start - all of the sysroot tests really need the
same treatment.

Closes: #1217
Approved by: jlebon
2017-09-27 13:13:14 +00:00
Colin Walters 25a7c4bd4e lib/pull: Default checksum for archive mirror, add TRUSTED_HTTP flag
I now think commit fab1e113db was a mistake;
because it breaks the mental model that at least I'd built up that "local repos
don't have checksums verified, HTTP does".

For example, a problem with this is (with that mental model in place) it's easy
for people who set up mirrors like this to then do local pulls, and at that
point we've done a deployment with no checksum verification.

Further, since then we did PR #671 AKA commit 3d38f03 which is really most of
the speed hit.

So let's switch the default even for this case to doing checksum verification,
and add `ostree pull --http-trusted`. People who are in situations where they
know they want this can find it and turn it on.

Closes: https://github.com/ostreedev/ostree/issues/1211

Closes: #1212
Approved by: jlebon
2017-09-26 18:07:43 +00:00
Colin Walters b6507a930e lib/pull: Refactor to use OstreeRepoImportFlags internally
Rather than carrying two booleans, just convert `OstreeRepoPullFlags`
into `OstreeRepoImportFlags`.  This allows us to drop an internal
wrapper function and just directly call `_ostree_repo_import_object()`.

This though reveals that our mirroring import path doesn't check the
`OSTREE_REPO_PULL_FLAGS_UNTRUSTED` flag...it probably should.

Prep for further work.

Closes: #1212
Approved by: jlebon
2017-09-26 18:07:43 +00:00
Colin Walters 72ddd4f419 lib/pull: Minor refactoring of metadata scanning function
Make the "local repo" processing conditional the same as the "localcache" bits;
this is really just a de-indent. Also add some comments. Prep for further work.

Closes: #1212
Approved by: jlebon
2017-09-26 18:07:43 +00:00
Colin Walters d0d27288da bin/pull: Fix description of --untrusted
It means *do* verify for local.

Closes: #1212
Approved by: jlebon
2017-09-26 18:07:43 +00:00
Colin Walters 8a7a359709 lib/commit: Add a copy fastpath for imports
This fixes up the last of the embarassing bits I saw from
the stack trace in:
https://github.com/ostreedev/ostree/issues/1184

We had a hardlink fast path, but that doesn't apply across
devices, which occurs in two notable cases:

 - Installer ISO with local repo
 - Tools like pungi that copy the repo to a local snapshot

Obviously there are a lot of subtleties here around things like the
bare-user-only conversions as well as exactly what data we copy. I think to get
better test coverage we may want to add `pull-local --no-hardlink` or so.

Closes: #1197
Approved by: jlebon
2017-09-26 16:50:41 +00:00
Colin Walters 3a08f7159d lib/commit: Some misc porting to decl-after-stmnt
Just happened to have this file open.

Closes: #1214
Approved by: jlebon
2017-09-26 13:31:05 +00:00
Colin Walters ee5ecf33a5 lib: Define an alias OSTREE_REPO_MODE_ARCHIVE
For the old `OSTREE_REPO_MODE_ARCHIVE_Z2`.  Use it mostly tree
wide except for the repo finder tests (to avoid conflicting with
some outstanding PRs).

Just noted another user coming in some of those tests and wanted to do a
cleanup.

Closes: #1209
Approved by: jlebon
2017-09-21 22:17:55 +00:00
Colin Walters 1797aff6a8 lib/core: Add static assertions for OstreeRepoMode enum values
Prep for adding an alias.

Closes: #1209
Approved by: jlebon
2017-09-21 22:17:55 +00:00
Colin Walters 6e4146a354 tree-wide: Remove Emacs modelines
We added a `.dir-locals.el` in commit: 9a77017d87
There's no need to have it per-file, with that people might think
to add other editors, which is the wrong direction.

Closes: #1206
Approved by: jlebon
2017-09-21 21:38:34 +00:00
Colin Walters ae075d23e3 lib/repo: Use correct name for tmpdir lockfile
Such an evil bug 🙈. I was just reading an strace trying to figure out what was
going on, and noticed we had the `XXXXXX` in the lockfile name. It was only
after that I realized that that this might *be* the cause of the skopeo issue.

This is another case where we definitely need more test coverage of things that
actually use the API multiple times in process; might look at dusting off the
work for the rpm-ostree test.

Closes: https://github.com/ostreedev/ostree/issues/1196

Closes: #1204
Approved by: jlebon
2017-09-21 21:10:34 +00:00
Colin Walters e689b8b4a5 ci: Add an "all options disabled" context
We keep occasionally regressing this so let's start covering it now. I'm
intentionally not running the tests since that would likely entail a lot more
conditionalizing pain.

Closes: #1194
Approved by: jlebon
2017-09-21 20:57:25 +00:00
Colin Walters d75316c907 lib/commit: Don't try to call linkat() for import on distinct devices
While opening a repo we've recorded the device/inode for a while; use it to
avoid calling `linkat()` during object import if we know it's going to fail.

Closes: #1193
Approved by: jlebon
2017-09-21 19:14:59 +00:00
Colin Walters 160864d557 lib: Move bareuseronly verification into commit/core
Conceptually `ostree-repo-pull.c` should be be written using
just public APIs; we theoretically support building without HTTP
for people who just want to use the object store portion and
do their own fetching.

We have some nontrivial behaviors in the pull layer though; one
of those is the "bareuseronly" verification.  Make a new internal
API that accepts flags, move it into `commit.c`.  This
is prep for further work in changing object import to support
reflinks.

Closes: #1193
Approved by: jlebon
2017-09-21 19:14:59 +00:00
Colin Walters 3f8f878fa3 tree-wide: Squash a few warnings when building with no features
Noticed these since I had a local configured build with no features for testing
the PAPR context for it.

Closes: #1201
Approved by: jlebon
2017-09-21 15:37:48 +00:00
Colin Walters f639f8f4c0 lib/checkout: Squash a gcc maybe-uninitialized warning
gcc doesn't know that our enum cases must be exhaustive.

Closes: #1201
Approved by: jlebon
2017-09-21 15:37:48 +00:00
Colin Walters 3767ac4ad8 lib/repo: Move alloca() outside of loop
Just noticed this while looking at the code for a different issue.

Closes: #1201
Approved by: jlebon
2017-09-21 15:37:48 +00:00
Colin Walters 75150fe04a lib/repo: Don't syncfs or fsync() dirs if fsync opt is disabled
There are use cases for not syncing at all; think build cache repos, etc. Let's
be consistent here and make sure if fsync is disabled we do no sync at all.

I chose this opportunity to add tests using the shiny new strace fault
injection.  I can forsee using this for a lot more things, so I made
the support for detecting things generic.

Related: https://github.com/ostreedev/ostree/issues/1184

Closes: #1186
Approved by: jlebon
2017-09-21 13:21:59 +00:00
Colin Walters 7a8511e0ca tests/libtest: Factor out user xattr detection
Make the equivalent of a `GOnce` or Rust `lazy_static!` detecting this and share
it between the two callers.

Prep for a future similar patch for strace fault injection.

Closes: #1186
Approved by: jlebon
2017-09-21 13:21:59 +00:00
Colin Walters 5c4f26bd65 lib/pull: Wait for pending ops to complete on error
I saw in a stack trace that the main thread was calling `exit()` even while
worker threads were alive and doing sha256/write/fsync etc. for objects.

The stack trace was a SEGV as the main thread was calling into library
`atexit()` handlers and we were a liblz4 destructor:

```
 #0  0x00007f2db790f8d4 _fini (liblz4.so.1)
 #1  0x00007f2dbbae1c68 __run_exit_handlers (libc.so.6)
```

(Why that library has a destructor I don't know offhand, can't find
 it in the source in a quick look)

Anyways, global library destructors and worker threads continuing simply don't
mix. Let's wait for our outstanding operations before we exit. This is also a
good idea for projects using libostree as a shared library, as we don't want
worker threads outliving operations.

Our existing pull corruption tests exercise coverage here.

I added a new `caught-error` status boolean to the progress API, and use it the
commandline to tell the user that we're waiting for outstanding ops.

Closes: #1185
Approved by: jlebon
2017-09-19 19:05:26 +00:00
Colin Walters 3e564116b2 lib/repo: Minor cleanup to object import function
We have a lot of layers of abstraction here; let's fold in the `trusted`
conditional into the call, since that's all the public API we're using does
anyways.

Prep for a future patch around object copying during imports.

Closes: #1187
Approved by: jlebon
2017-09-19 18:51:03 +00:00
Colin Walters 13c3898cc2 tree-wide: Some glnx_fstatat_allow_noent() porting
The new API is definitely nicer.

Closes: #1180
Approved by: jlebon
2017-09-19 15:03:05 +00:00
Colin Walters 1036cc8084 ci: Use distro-sync instead of upgrade
This should help ensure that if installing `glibc-devel` downgrades `glibc`
we'll run through our hackround for
<https://bugzilla.redhat.com/show_bug.cgi?id=1394862>.

Closes: #1190
Approved by: jlebon
2017-09-19 14:29:47 +00:00
Colin Walters ab200495a8 bin/prune: Add --only-branch
In 5c940987e7 / #646 we
added `--retain-branch-depth`; this adds a symmetric
`--only-branch` for the case where a repo owner just
wants to prune a specific branch.

The implementation here is pretty straightforward; we
just walk all refs and inject the equivalent of
`--retain-branch-depth=$ref=-1` if they're *not* in
`--only-branch`.

Closes: https://github.com/ostreedev/ostree/issues/1115

Closes: #1127
Approved by: jlebon
2017-09-18 17:20:38 +00:00
Colin Walters 58e4abe4bc tests/prune: Factor out a helper for counting commits in the repo
We had lots of duplicates; prep for adding more tests.

Closes: #1127
Approved by: jlebon
2017-09-18 17:20:38 +00:00
Colin Walters d0b0578cc1 Update libglnx
Update libglnx, which is mostly port the repo stagedir code
to the new tmpdir API.  This turned out to require some
libglnx changes to support de-allocating the tmpdir ref while
still maintaining the on-disk dir.

Update submodule: libglnx

Closes: #1172
Approved by: jlebon
2017-09-18 17:09:34 +00:00
Colin Walters 0488b4870e lib/pull: Drop partial fetch code from libsoup backend
Doing this in prep for libglnx tmpdir porting, but I think we should also do
this because the partial fetch code IMO was never fully baked; among other
things it was never integrated into the scheme we came up with for "boot id
sync" that we use for complete/staged objects.

There's a lot of complexity here that while we have some coverage for, I think
we need to refocus on the core functionality. The libcurl backend doesn't have
an equivalent to this today.

In particular for small objects, this is simply overly complex. The downside is
clearly for large objects like FAH's 61MB initramfs; not being able to resume
fetches of those is unfortunate.

In practice though, I think most people should be using deltas, and we need to
make sure deltas work for large objects anyways.

Further ultimately the peer-to-peer work should help a lot for people
with truly unreliable connections.

Closes: #1176
Approved by: jlebon
2017-09-15 17:01:51 +00:00
Colin Walters c32f234e9a lib/checkout: Do UNION_FILES via atomic renameat()
I was looking at fixing an `rpm-ostree livefs` bug where we need to replace
`/usr/lib/passwd`. It's obviously bad if that temporarily disappears 😉. My plan
is to do a subpath checkout of just `/usr/lib/{passwd,group}`.

Make this atomic (i.e. file always exists) by changing the logic to create a
temporary link in repo/tmp, then rename() it into place.

A bonus here is we kill one of the very few (only?) non-error-cleanup i.e.
"non-linear" `goto`s in the ostree codebase.

Closes: #1171
Approved by: jlebon
2017-09-15 16:44:00 +00:00
Colin Walters 7499620254 lib/repo: Port gpg signing function to new code style
We already had all of the autocleanups ready for this.

Closes: #1164
Approved by: jlebon
2017-09-15 01:43:16 +00:00
Colin Walters 6c0738a000 Release 2017.11
Closes: #1173
Approved by: jlebon
2017-09-14 15:04:42 +00:00
Colin Walters 1c2d344074 tests: Port some bits of C to new style
Where we can; perhaps after updating libglnx we should use the
new test error macro?

Closes: #1169
Approved by: jlebon
2017-09-13 19:32:36 +00:00
Colin Walters 051cdf396c lib/checkout: Rename disjoint union, change to merge identical files
It turns out that librpm automatically merges identical files between
distinct packages, and this occurs in practice with Fedora today between
`chkconfig` and `initscripts` for exmaple.

Since we added this for rpm-ostree, we basically want to do what librpm does,
let's change the semantics to do a merge.  While we're here rename
to `UNION_IDENTICAL`.

Closes: #1156
Approved by: jlebon
2017-09-13 19:19:33 +00:00
Colin Walters 8d3752a0d6 lib/repo: Port tmpdir locking func to new style
Prep for future work.

Closes: #1168
Approved by: jlebon
2017-09-13 19:02:31 +00:00
Colin Walters 93038bcf71 build-sys: Add -Werror=switch
We use the "exhaustive enum" pattern (i.e. no `default:`) in some places so
we're forced to touch all users when adding cases.

Closes: #1167
Approved by: peterbaouoft
2017-09-13 17:06:45 +00:00
Colin Walters 662ad5b171 lib/sysroot: Use direct g_mkdtemp() for overlay tmpdir
The new libglnx `glnx_mkdtempat()` uses autocleanups, which
is inconvenient for this use case where we *don't* want autocleanups.
Since we don't need it to be fd-relative, just directly invoke
`g_mkdtemp_full()` which is fine for this use case.

Prep for updating libglnx.

Closes: #1161
Approved by: jlebon
2017-09-12 20:09:12 +00:00