They don't play nicely currently with HTTP2 where we may
have lots of requests queued.
https://github.com/ostreedev/ostree/issues/878#issuecomment-347228854
In practice anyways I think issues here are better solved on a higher level -
e.g. apps today can use an overall timeout on pulls and if they exceed the limit
set the cancellable.
Closes: #1349
Approved by: jlebon
If a newly allocated tmpdir can't be locked, set initialized to FALSE so
that glnx_tmpdir_cleanup doesn't delete it when new_tmpdir goes out of
scope.
Closes: #1346
Approved by: cgwalters
Another tmpdir user may have deleted an existing tmpdir between the time
the current user called readdir and tried to open it.
Closes: #1346
Approved by: cgwalters
By default, unless it’s const, an (out) GHashTable will be assumed to be
(transfer full). That means the binding needs to free all the items in
the hash table, plus the table itself.
However, all the GHashTables we use have free functions set already, so
freeing the hash table will free its items. This results in a
double-free.
Fix that by ensuring we annotate such (out) hash tables as (transfer
container). Also annotate some other hash tables as (transfer none)
where appropriate, for clarity.
This fixes OSTree.Repo.list_collection_refs() in the Python bindings.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1341
Approved by: dbnicholson
This reverts commit 519b30b7e1. Now that
the experimental GIR is being built correctly and OstreeRemote is a real
boxed type, this can be exposed again.
Closes: #1337
Approved by: pwithnall
Now that g-ir-scanner is being told about ENABLE_EXPERIMENTAL_API, it
can include these types correctly. Drop the __GI_SCANNER__ guards in the
header files so that all the declarations are found.
After this, you can actually construct the types normally:
>>> OSTree.CollectionRef.new('com.example.Foo', 'bar')
<OSTree.CollectionRef object at 0x7f2bba4c7528 (OstreeCollectionRef at 0x55c033ff2f30)>
Closes: #1337
Approved by: pwithnall
Without this, you can't really use OstreeRemote as a GObject, which is a
requirement for bindings.
This was found when attempting to include OstreeRemote in the GIR, and
g-ir-scanner wasn't able to link it's temporary object due to an
"undefined reference to `ostree_remote_get_type'" error.
Closes: #1337
Approved by: pwithnall
I wanted to inspect a summary file the other day and was saddened to
find it was broken:
$ ostree summary --raw
error: No option specified; use -u to update summary
Fix the test to do the normal thing of passing just --raw without
--view. It's legal to pass --raw and --view, but it shouldn't be a
requirement.
Closes: #1336
Approved by: cgwalters
Fedora switched to 'xz' compress kernel modules, and recently
[RHEL7 did too](https://bugzilla.redhat.com/show_bug.cgi?id=1367496).
This compression defeats bsdiff.
While we have a "rollsum-able" test, we don't have a "bsdiff-able" test as it'd
be very expensive (we'd have to bsdiff, then apply it and compare the result).
Let's do the tactical quick fix here and just not try to delta files ending in
`.xz.`. This avoids us using bsdiff pointlessly for over 4000 files, which is
quite a notable speed increase for generating deltas.
Closes: #1333
Approved by: jlebon
This allows more explicit handling of commit state in code using
libostree, rather than hard-coding a commit state of 0 for ‘normal’.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1335
Approved by: cgwalters
Reorder cleanup functions so that curl_multi_cleanup() runs before
self->sockets is destroyed. This avoids an assert and invalid memory
access in sock_cb where self->sockets is dereferenced during
curl_multi_cleanup().
Closes: https://github.com/ostreedev/ostree/issues/1331Closes: #1332
Approved by: cgwalters
A tricky thing here that caused this to go past a lot of our tests
is that the code was mostly OK if there was an available delta from
an older commit. But this case broke if we e.g. had a new OS
deployment and did a `--require-static-deltas` pull, i.e. the initial
state.
I cleaned up our "find static delta state" function to return an enumeration,
and extended it with an "already have the commit" state. A problem
I then hit is that we've historically fetched detached metadata for
non-delta pulls, even if the commit hasn't changed. I decided not to
do that for `--require-static-deltas` pulls for now; otherwise the
code gets notably more complex.
Closes: https://github.com/ostreedev/ostree/issues/1321Closes: #1323
Approved by: jlebon
The main thing here is that a ton of stuff has happened in gnulib since we
imported `parse-datetime.y`. I cherry-picked a little bit of it, but that
upstream doesn't seem to build with `-Wundef`, so I just deleted some hunks.
(Note I reindented the warnings consistently)
Update submodule: libglnx
Closes: #1320
Approved by: jlebon
Since ostree_remote_get_type is not made available to g-ir-scanner, it
treats OstreeRemote as a bare struct. That's not kosher for bindings and
it issues the following warning:
src/libostree/ostree-repo-pull.c:5560: Warning: OSTree:
ostree_repo_resolve_keyring_for_collection: return value: Invalid
non-constant return of bare structure or union; register as boxed type
or (skip)
For now, just skip this API for bindings.
Closes: #1322
Approved by: pwithnall
g-ir-scanner was spitting this warning:
src/libostree/ostree-core.c:281: Warning: OSTree:
ostree_validate_collection_id: unknown parameter 'rev' in
documentation comment, should be 'collection_id'
Closes: #1322
Approved by: pwithnall
It's the slowest part, let's show admins something. This "update every 10%" code
was copied from the fsck command; obviously a better approach would be "progress
every N seconds" but doing that somewhat accurately requires making things
async; not worth it here yet.
Closes: #1314
Approved by: jlebon
I didn't fully spelunk this, but from what `static-delta-generate-crosscheck.sh`
had, we appeared to be doing this before, and it's clearly useful for local
testing rather than needing to spin up a HTTP server.
Closes: #1313
Approved by: jlebon
First, the manual crosscheck script bitrotted; it got caught up
in the "use libtest repo creation wrapper" bit, and also it
seems like at some point `pull --require-static-deltas` changed
meaning when dealing with `file:///` repos. I have more work to
unwind that.
Next, I'm seeing a delta failure which looks like a static delta
miscompilation with rollsums; change the compiler to print out
the source object too, which helped me debug this.
And finally in the processing code, fix incorrect error prefixing, which was
misleading.
Closes: #1311
Approved by: ashcrow
Fixes: 93457071cb "lib/deltas: Use pread() instead of lseek()+read()"
Caught this when trying to test alex's patch locally. I am going to review our
static delta pulls and try to get something more comprehensive locally. But in
the meantime this patch is clearly right.
Closes: #1312
Approved by: jlebon
Directly when we allocate a new part we finish the old one,
writing the compressed data to a temporary file and generating
the delta header for it.
When all these are done we loop over them and collect the headers,
sizes and either copy the tempfile data into the inlined superblock
or link the tempfiles to disk with the proper names.
Closes: #1309
Approved by: cgwalters
This allows us to create the final delta desciptor directly on disk
rather than having it all in memory. This is nice because it can
become quite large if inlined parts are used.
Note however, that we currently generate all the delta parts in
memory before adding them to the delta, so we still keep all individual
parts in memory. Fixing that is the next step.
Closes: #1309
Approved by: cgwalters
This is similar to GVariantBuilder in that it constructs variant
containers, but it writes it directly to a file descriptor rather
than keep the entier thing in memory. This is useful to create large
variants without using a lot of memory.
Closes: #1309
Approved by: cgwalters
g_autoptr was new in GLib 2.44, but we officially only require 2.40,
so we need to use the backport in libglnx.
Signed-off-by: Simon McVittie <smcv@collabora.com>
Closes: #1310
Approved by: cgwalters
This makes the code nicer too. Properly unit testing this though really wants
like a whole set of stuff around parent repos...but we do have coverage of the
non-parent path in the current pull tests.
Closes: https://github.com/ostreedev/ostree/issues/1306Closes: #1308
Approved by: alexlarsson
For example, tmpfs or a cgroup file system. This is basically an
optimisation of the list of file systems we check for repositories,
since we would never expect any of these file systems to be capable of
containing a repository.
Depends on the new API from
https://bugzilla.gnome.org/show_bug.cgi?id=788927.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1307
Approved by: cgwalters
These two code paths tried to propagate errors which had never been set.
Set new errors instead.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1303
Approved by: cgwalters
A GVariantIter* was being passed to a GVariant format string varargs,
rather than a GVariantIter**. This resulted in memory corruption.
So we can continue to reuse ref_map throughout the function, make it a
GVariantIter* rather than a stack-allocated GVariantIter.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1301
Approved by: cgwalters
This is more subtle fallout from:
https://github.com/ostreedev/ostree/pull/1170
AKA commit: 8fe4536257
Before, if we found a devino cache hit, we'd use it unconditionally.
Recall that `bare-user` repositories are very special in that they're the only
mode where the on disk state ("physical state") is not the "real" state. The
latter is stored in the `user.ostreemeta` xattr. (`bare-user` repos are also
highly special in that symlinks are regular files physically, but that's not
immediately relevant here).
Since we now have `bare-user-only` for the "pure unprivileged container" case,
`bare-user` should just be used for "OS builds" which have nonzero uids (and
possibly SELinux labels etc.)
In an experimental tool I'm writing "skopeo2ostree" which imports OCI images
into refs, then squashes them together into a single final commit, we lost the
the `81` group ID for `/usr/libexec/dbus-1/dbus-daemon-launch-helper`.
This happened because the commit code was loading the "physical" disk state,
where the uid/gid are zero because that's the uid I happened to be using. We
didn't just directly do the link speedup because I was using `--selinux-policy`
which caused the xattrs to change, which caused us to re-commit objects from the
physical state.
The unit test I added actually doesn't quite trigger this, but I left
it because "why not". Really testing this requires the installed test
which uses SELinux policy from `/`.
The behavior without this fix looks like:
```
-00755 0 0 12 { [(b'user.ostreemeta', [byte 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x51, 0x00, 0x00, 0x81, 0xed]), (b'security.selinux', b'system_u:object_r:lib_t:s0')] } /usr/lib/dbus-daemon-helper
```
which was obviously totally broken - we shouldn't be picking up the
`user.ostreemeta` xattr and actually committing it of course.
Closes: #1297
Approved by: jlebon
Add some missing annotations and clarify that it always returns an open
repository on success.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1300
Approved by: cgwalters
This is a freeform string useful to track/display when a commit is "derived"
from some other format. For example, in the rpm-ostree test we make a
`vmcheck` ref that conceptually overlays the default ref like
`fedora-atomic:fedora/26/x86_64/atomic-host`.
My current patch sets the source title to e.g.
"Dev overlay on fedora-atomic:fedora/26/x86_64/atomic-host".
Another case I'm working on now is importing OCI images to use
as host images. For that case, the source title is
With this patch we could then set the original OCI image name + tag
as the source name, like:
"oci:cgwalters/demo-custom-fedora-atomic-host:26".
Closes: #1296
Approved by: jlebon
Pull out the commit metadata explicitly; still just rendering the version, but
this is prep for rendering other metadata keys.
Closes: #1296
Approved by: jlebon
I was trying to use this with pygobject for an OCI+ostree project, and pygobject
rejected simply assigning to the field (understandably, since it can't bind the
lifetime together).
Add a wrapper function, which is still unsafe, but hides that unsafety
where most people shouldn't find it. And if they do...well, sorry,
Rust wasn't invented when ostree was started.
Closes: #1295
Approved by: pwithnall
I'm playing around with some ostree ⇔ OCI/Docker bits, and ran
into this while importing an OCI image that built from the Fedora
base image where `/home` is a regular directory, and I added a layer
that did the ostree bits of moving it to `/var` and leaving a symlink.
OCI/Docker supports this. Now since "process whiteouts" is really the
"enable OCI/Docker" mode, let's only replace dirs if that's enabled.
This leaves the `UNION_FILES` targeted for its original use case
which is unioning components/packages. (Although that use case itself
is now a bit superceded by `UNION_IDENTICAL`, but eh).
Closes: #1294
Approved by: jlebon
This is similar idea as
5c0bf88915,
The duplicated description is now removed, and the description
of the command is now displayed beneath the Usage.
For example:
ostree cat -h will output the following:
"Usage:
ostree cat [OPTION?] COMMIT PATH...
Concatenate contents of files"
Closes: #1267
Approved by: cgwalters
This is a similar approach as
12c34bb249.
One thing to note is when we parse the admin related functions,
we still keep the old admin related flags, and added a new parameter
to represent the command struct.
This allows us to identify the caller of the function, making it
easier for us to possibly deduplicate the subcommand handling in
the future. A similar approach is done in rpm-ostree:
83aeb018c1
This also makes it easier for us to change the prototype of the function.
If we want to add something new in the future, we won't need to touch every prototype.
Closes: #1267
Approved by: cgwalters
Added a description argument to all type
of commands. Now when we include -h or --help
for commands that contain subcommands, the description
for those subcommands are shown.
The added subcommands help will be provided to the following commands:
- ostree -h
- ostree admin -h
- ostree admin instutil -h
- ostree remote -h
- ostree static-delta -h
Closes: #1267
Approved by: cgwalters
This is another OstreeRepoFinder implementation; it returns results from
a given set of URIs. It’s designed to be used for implementing user
overrides to other repo-finders, or for implementing unit tests.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1281
Approved by: mwleeds
Use g_variant_iter_loop() rather than next(), since it automatically
handles freeing the child memory each iteration. Previously, we leaked
it for all but the last iteration.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1293
Approved by: cgwalters
Change the regexp for validating refs to require at least one letter or digit
before allowing the other special chars in the set `[.-_]`. Names that start
with `.` are traditionally Unix hidden files; let's ignore them under the
assumption they're metadata for some other tool, and we don't want to
potentially conflict with the special `.` and `..` Unix directory entries.
Further, names starting with `-` are problematic for Unix cmdline option
processing; there's no good reason to support that. Finally, disallow `_` just
on general principle - it's simpler to say that ref identifiers must start with
a letter or digit.
We also ignore any existing files (that might be previously created refs) that
start with `.` in the `refs/` directory - there's a Red Hat tool for content
management that injects `.rsync` files, which is why this patch was first
written.
V1: Update to ban all refs starting with a non-letter/digit, and
also add another call to `ostree_validate_rev` in the pull
code.
Closes: https://github.com/ostreedev/ostree/issues/1285Closes: #1286
Approved by: jlebon
The way `_ostree_repo_open_content_bare()` did both looking for the object and
possibly creating a new fd was just weird and inconsistent with e.g. the pull
code where we always call `has_object()` first.
Just call `has_object()` in the delta paths that used this too, making the
implementation right now a thin wrapper around
`glnx_open_tmpfile_linkable_at()`, but this is prep for a later patch which does
more.
Closes: #1283
Approved by: jlebon
This is another case where making an input stream out of a memory buffer is a
bit silly; just hash the `GBytes` directly.
Closes: #1287
Approved by: jlebon
A side effect of commit 8fe4536257 is that
we started listing all xattrs even for files with device/inode matches;
further, we did that using the dfd/name which means we went through
the `/proc` path, which is slower and uglier.
Noticed this in strace while looking at adoption code.
Closes: #1280
Approved by: jlebon
That's why the syscall was invented, so let's use it. Just noticed while reading
the code while working on another patch.
Closes: #1270
Approved by: jlebon
This isn't perfect, but at least we fix an error-overwrite error, and in
practice `ostree admin unlock` isn't wrapped by `rpm-ostree` yet, so spew to
stderr is OK.
Closes: https://github.com/ostreedev/ostree/issues/1273Closes: #1279
Approved by: guyshapiro
For checkouts that are on the same device, for regular files we can simply
"adopt" existing files. This is useful in the "build from subtrees" pattern that
happens with e.g. `rpm-ostree install` as well as flatpak and gnome-continuous.
New files are things like an updated `ldconfig` cache, etc. And particularly for
`rpm-ostree` we always regenerate the rpmdb, which for e.g. this workstation is
`61MB`.
We probably should have done this from the start, and instead had a `--copy`
flag to commit, but obviously we have to be backwards compatible.
There's more to do here - the biggest gap is probably for `bare-user` repos,
which are often used with things like `rpm-ostree compose tree` for host
systems. But we can do that later.
Closes: #1272
Approved by: jlebon
I was working on a patch to do build on the work done to
import content objects async to do the same for metadata, but right
now we basically rely on writing them first to do the GPG verification
when scanning.
Things will be cleaner for that if we can pass the commit object directly into
`scan_commit_object()` and consistently use `gpg_verify_unwritten_commit()`.
We're careful here to continue to do it both ways (but at most one time), to
account for the case where a bad commit has been pulled and written - we need to
keep failing GPG verification there.
Closes: #1269
Approved by: jlebon
Prep for a later patch to do GPG verification before writing commit objects;
`_ostree_repo_gpg_verify_with_metadata()` already handles this, and so dropping
this gives us consistent error messages.
Closes: #1269
Approved by: jlebon
ENOTSUP and EOPNOTSUPP are numerically equal on most Linux ports,
but inexplicably differ on PA-RISC (hppa) and possibly other
rare architectures.
Signed-off-by: Simon McVittie <smcv@collabora.com>
Closes: #1275
Approved by: cgwalters
There's a subtle issue going on with the way we use `UNION_IDENTICAL`
now in rpm-ostree. Basically, the crux of the issue is that we checkout
the whole tree from the system repo, but then overlay packages by
checking out from the pkgcache repo. This is an easy way to break the
assumption that we will be merging hardlinks from the same repo.
This ends up causing issues like:
https://github.com/projectatomic/rpm-ostree/issues/1047
There, `vim-minimal` is already part of the host and has an object for
`/usr/share/man/man1/ex.1.gz`. `vim-common` has that same file, but
because it's unpacked in the pkgcache repo first, the hardlinks are not
the same.
There are a few ways we *could* work around this in rpm-ostree itself,
e.g. by re-establishing hardlinks when we do the content pull into the
system repo, but it still felt somewhat hacky. Let's just do this the
proper way and fall back to checksumming the target file if needed,
which is what librpm does as well in this case. Note that we only
checksum if they're not hard links, but they're the same size.
Closes: #1258
Approved by: cgwalters
In case a filename contains invalid UTF-8 characters, libostree will
pass it to g_variant_builder_add() in create_tree_variant_from_hashes()
anyway, which leads to a critical warning from glib and an invalid
commit. This commit makes ostree print a useful error and exit instead.
Closes: #1271
Approved by: cgwalters
Let's react to `Ctrl-C` faster here. Noticed while I was doing an update on my
desktop and playing with cancellation.
Closes: #1266
Approved by: jlebon
This is like `ostree_checksum_file` but fd-relative. This will be used
by https://github.com/ostreedev/ostree/pull/1258.
AFAICT, we actually didn't have any tests that check the `checksum` CLI.
Add a basic one here to test the old code as well as the new code.
Closes: #1263
Approved by: cgwalters
No functional changes, prep for patch. (Well, I did add a new `success`
member in the async struct so that we return `FALSE` if we failed).
Closes: #1263
Approved by: cgwalters
This works around an (IMO) SpiderMonkey bug - it tries to
clean up in a shared library destructor, but doesn't install a
`pthread_atfork()` handler to unset its state.
Closes: https://github.com/ostreedev/ostree/issues/1262Closes: #1264
Approved by: dbnicholson
This ends up a lot better IMO. This commit is *mostly* just
`s/glnx_close_fd/glnx_autofd`, but there's also a number of hunks like:
```
- if (self->sysroot_fd != -1)
- {
- (void) close (self->sysroot_fd);
- self->sysroot_fd = -1;
- }
+ glnx_close_fd (&self->sysroot_fd);
```
Update submodule: libglnx
Closes: #1259
Approved by: jlebon
It's no longer called directly by the pull code, so make it static.
The goal here is to have the pull and local-fs commit paths use higher level
more efficient APIs, and eventually make those APIs public.
Closes: #1257
Approved by: jlebon
This simplifies a lot of code; the header function was structured
to write to an input stream, but many callers only wanted the checksum,
so it's simpler (and error-free) to simply allocate a whole buffer
and checksum that.
For the callers that want to write it, it's also still simpler to allocate the
buffer and write the whole thing rather than having this function do the
writing.
A lot of the complexity here again is a legacy of the packfile code, which is
dead.
This is prep for faster regfile commits where we can avoid `G{In,Out}putStream`.
Closes: #1257
Approved by: jlebon
Nothing was using the `bytes_written` data (we always discard partially written
tmpfiles), so simplify everything by dropping it. Further, we always passed an
offset of `0`, so drop that argument too. (I believe that this was previously
used by the "pack files" code that we deleted long ago)
Second, we had an unnecessary internal wrapper for this function; drop that too.
Closes: #1257
Approved by: jlebon
If the filesystem is already frozen, FIFREEZE returns EBUSY, and if the
filesystem is already thawed, FITHAW returns EINVAL. It's very unlikely
these issues would arise on a real ostree system since the sysroot would
be locked during the freeze/thaw cycle.
However, when multiple fake sysroots are used during the test suite (run
as root), the tests could race to run the freeze/thaw cycle without
locking. Furthermore, there's no reason why an independent process might
be trying to freeze the filesystem while ostree was deploying. Ignore
but warn for these errors since there's not much ostree can do about it,
anyways.
Closes: #1260
Approved by: cgwalters
The faster (OpenSSL/GnuTLS) code lived in a `GInputStream` wrapper, and that
adds a lot of weight (GObject + vtable calls). Move it into a simple
autoptr-struct wrapper, and use it in the metadata path, so we're
now using the faster checksums there too.
This also drops a malloc there as the new API does hexdigest in place to a
buffer.
Prep for more work in the commit path to avoid `GInputStream` for local file
commits, and ["adopting" files](https://github.com/ostreedev/ostree/pull/1255).
Closes: #1256
Approved by: jlebon
For many cases of commit, we can actually optimize things by simply "adopting"
the object rather than writing a new copy. For example, in rpm-ostree package
layering.
We can only make that optimization though if we take ownership of the file. This
commit hence adds an API where a caller tells us to do so. For now, that just
means we `unlink()` the files/dirs as we go, but we can now later add the
"adopt" optimization.
Closes: #1255
Approved by: jlebon
What the deltas code is doing is weird/unfortunate. The name
`ot_variant_read()` conflicts too much with `ot_variant_read_fd()`.
Since nothing else uses it, move it into the deltas code.
Closes: #1254
Approved by: jlebon
A lot of the libostree code is honestly too complex for its
own good (this is mostly my fault). The way we do HTTP writes
is still one of those. The way the fetcher writes tempfiles,
then reads them back in is definitely one of those.
Now that we've dropped the "partial object" bits in:
https://github.com/ostreedev/ostree/pull/1176 i.e. commit
0488b4870e
we can simplify things a lot more by having the fetcher
return an `O_TMPFILE` rather than a filename.
For trusted archive mirroring, we need to enable linking
in the tmpfiles directly.
Otherwise for at least content objects they're compressed, so we couldn't link
them in. For metadata, we need to do similar logic to what we have around
`mmap()` to only grab a tmpfile if the size is large enough.
Closes: #1252
Approved by: jlebon
Buried in this large patch is a logical fix:
```
- if (!map)
- return glnx_throw_errno_prefix (error, "mmap");
+ if (map == (void*)-1)
+ return glnx_null_throw_errno_prefix (error, "mmap");
```
Which would have helped me debug another patch I was working
on. But it turns out that actually correctly checking for
errors from `mmap()` triggers lots of other bugs - basically
because we sometimes handle zero-length variants (in detached
metadata). When we start actually returning errors due to
this, things break. (It wasn't a problem in practice before
because most things looked at the zero size, not the data).
Anyways there's a bigger picture issue here - a while ago
we made a fix to only use `mmap()` for reading metadata from disk
only if it was large enough (i.e. `>16k`). But that didn't
help various other paths in the pull code and others that were
directly doing the `mmap()`.
Fix this by having a proper low level fs helper that does "read all data from
fd+offset into GBytes", which handles the size check. Then the `GVariant` bits
are just a clean layer on top of this. (At the small cost of an additional
allocation)
Side note: I had to remind myself, but the reason we can't just use
`GMappedFile` here is it doesn't support passing an offset into `mmap()`.
Closes: #1251
Approved by: jlebon
Appease Coverity by using the same condition for both the ternary check
and the if-condition later on. It should be smart enough to figure out
that `dir_enum == NULL` implies that `dfd_iter != NULL` from the
assertion at the top of the function.
Coverity CID: #1457318Closes: #1250
Approved by: cgwalters
We want `pull` to be included as long as we have at least either
`libcurl` or `libsoup` to back it. Of course, this is a moot point for
now since `libsoup` is currently a build requirement.
Closes: #1244
Approved by: cgwalters
Spotted while reading through the code, it looks like the
copy_detached_metadata() call is accidentally omitted if a hardlink
already exists for the .commit object.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1242
Approved by: cgwalters
This commit adds debug output whenever libostree reads GPG keys, which
can come from different locations in the file system. This is especially
helpful in debugging "GPG signatures found, but none are in trusted
keyring" errors, which in my case was caused by OSTree looking in
/usr/local/share/ostree/trusted.gpg.d/ rather than
/usr/share/ostree/trusted.gpg.d/.
Closes: #1241
Approved by: cgwalters
I'm regretting a bit having the `guint8*csum` variant of checksums
except for the serialized form. Once we start doing processing
it's easier to just have it remain hex.
Do an on-stack conversion for the metadata scanning function; this
drops a malloc and also just looks nicer.
Also add some long-awaited function comments to the two.
Closes: #1240
Approved by: jlebon
These shouldn’t change the bloom filter’s behaviour at all, but make it
a bit more obvious what the programmatical limitations are on the sizes
it can deal with.
In reality, those sizes should never be reached because they won’t fit
in a DNS-SD record.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1239
Approved by: cgwalters
I was reading the pull code for the last release, and spotted
a bug in commit f923c2e1ea - in
the case where the ref doesn't exist, we don't set an error,
tripping an assertion in the main code.
The previous code wanted the ref to always exist, so just flip back the boolean
for "ignore noent". I moved the `g_strchomp()` just into the HTTP path - if a
local repo is corrupted in this way it's something to fix in that repo.
Closes: #1238
Approved by: pwithnall
This is the new way of publishing repository metadata, rather than as
additional-metadata in the summary file. The use of an ostree-metadata
ref means that the metadata from multiple upstream collections is not
conflated when doing P2P mirroring of many repositories.
The new ref is only generated if the repository has a collection ID set.
The old summary file continues to be generated for backwards
compatibility (and because it continues to be the canonical ref →
checksum map for the repository).
The new code is only used if configured with --enable-experimental-api.
Includes unit tests.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1158
Approved by: cgwalters
There is no error handling to do, so just return everywhere instead.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1158
Approved by: cgwalters
Compiling with -Wconversion warns on this line, as the conversion from
guint64 to guint8 is implicit (but safe: there is no bug here, since the
implicit cast is applied after the modulus arithmetic).
Make the cast explicit to silence -Wconversion.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1231
Approved by: cgwalters
There was an implicit cast from guint64 to gsize (which is 32-bit on
armhf, for example) before the modulus arithmetic which safely narrows
the index.
Fix that by using a guint64 intermediate variable and making the cast
explicit.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1231
Approved by: cgwalters
We can't use the cache if the file we want to commit has been modified
by the client through the file info or xattr modifiers. We would
prematurely look into the cache in `write_dfd_iter_to_mtree_internal`,
regardless of whether any filtering applied.
We remove that path there, and make sure that we only use the cache if
there were no modifications. We rename the `get_modified_xattrs` to
`get_final_xattrs` to reflect the fact that the xattrs may not be
modified.
One tricky bit that took me some time was that we now need to store the
st_dev & st_ino values in the GFileInfo because the cache lookup relies
on it. I'm guessing we regressed on this at some point.
This patch does slightly change the semantics of the xattr callback.
Previously, returning NULL from the cb meant no xattrs at all. Now, it
means to default to the on-disk state. We might want to consider putting
that behind a flag instead. Though it seems like a more useful behaviour
so that callers can only override the files they want to without losing
original on-disk state (and if they don't want that, just return an
empty GVariant).
Closes: #1165Closes: #1170
Approved by: cgwalters
I was trying to do a change for rpm-ostree to use
`OSTREE_REPO_COMMIT_MODIFIER_FLAGS_CANONICAL_PERMISSIONS`
for container builds with `bare-user-only,` but hit an assertion here
ultimtely because we weren't setting `standard::type`.
Rather than hand-rolling `GFileInfo` creation, use the stat buffer conversion
code which is more robust and used in multiple places already.
Closes: #1227
Approved by: jlebon
In particular I'd like to get the copy fix in, since it might affect users for
the keyring bits.
Update submodule: libglnx
Closes: #1225
Approved by: jlebon
I noticed this while debugging why I was seeing "2 metadata objects" fetched for
a different PR. I knew 1 was detached meta, but the other turned out to be this.
There's no reason to request a delta if the ref is unchanged.
Closes: #1220
Approved by: jlebon
Propagate the refspec_name from the OstreeRemote returned by an
OstreeRepoFinder through to the set_ref() call.
This changes ostree_repo_pull_with_options() to accept the
previously-disallowed combination of passing override-remote-name in
options and also setting a remote name in remote_name_or_baseurl.
ostree_repo_pull_with_options() will continue to pull using the remote
config named in remote_name_or_baseurl as before; but will now use the
remote name from override-remote-name when it’s setting the refs at the
end of the pull. This is consistent with the documentation for
override-remote-name.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1202
Approved by: cgwalters
When pulling from a dynamic (peer to peer) remote, the remote’s name is
set to a unique, generated string which doesn’t exist in repo/config. If
doing a non-mirror pull, however, we don’t want to use this name in the
refspecs for newly created or updated refs — we want to use the name of
the remote which provided the keyring for the pull (this will be a
remote from repo/config whose collection ID matches that being used for
the peer to peer pull).
Store both names in OstreeRemote. The name to use for refspecs is stored
as refspec_name, and is typically NULL unless it differs from name.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1202
Approved by: cgwalters
Instead of returning just the keyring filename, return the entire
OstreeRemote, which has the keyring filename as one of its members. This
will simplify some upcoming changes, and allows slightly improved debug
logging.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1202
Approved by: cgwalters
If override-remote-name is specified in the options to
ostree_repo_pull_with_options(), but the remote_name_or_baseurl argument
is also set to a remote name, the override-remote-name would be leaked.
Note that this is currently an invalid configuration, so this leak is
basically never hit.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1202
Approved by: cgwalters
Followup for recent work in commits:
- 8a7a359709
- 1a9a473580
Keep track of how many objects we imported, and print that for `ostree
pull-local` (also do this even if noninteractive, like we did for `pull`).
In implementing this at first I used separate variables for import
from repo vs import from localcache, but that broke some of the
tests that checked those values.
It's easier to just merge them; we know from looking at whether or not
`remote_repo_local` is set whether or not we were doing a "HTTP pull with
localcache" versus a true `pull-local` and can use that when rendering status.
Closes: #1219
Approved by: jlebon
This is more efficient in the non-collection case; in the collection
case, the implementation of ostree_repo_resolve_collection_ref() needs
to be rewritten to improve efficiency.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1182
Approved by: cgwalters
This is a parallel for ostree_repo_resolve_rev_ext() which works on
collection–refs. At the moment, the implementation is simple and uses
ostree_repo_list_collection_refs(). In future, it could be rewritten to
check the checksum directly rather than enumerating all
potentially-relevant checksums.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1182
Approved by: cgwalters
This can be used to put OSTree repositories on USB sticks in a format
recognised by OstreeRepoFinderMount.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1182
Approved by: cgwalters
Previously, collection–refs could only be pulled from a repository if it
had a summary file (which listed them). There was no way to pull from a
local repository which doesn’t have a summary file, and where the refs
were stored as refs/remotes/$remote/$ref, with a config section linking
that $remote to the queried collection ID.
Fix that by explicitly supporting pull_data->remote_repo_local in
fetch_ref_contents().
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1182
Approved by: cgwalters
I now think commit fab1e113db was a mistake;
because it breaks the mental model that at least I'd built up that "local repos
don't have checksums verified, HTTP does".
For example, a problem with this is (with that mental model in place) it's easy
for people who set up mirrors like this to then do local pulls, and at that
point we've done a deployment with no checksum verification.
Further, since then we did PR #671 AKA commit 3d38f03 which is really most of
the speed hit.
So let's switch the default even for this case to doing checksum verification,
and add `ostree pull --http-trusted`. People who are in situations where they
know they want this can find it and turn it on.
Closes: https://github.com/ostreedev/ostree/issues/1211Closes: #1212
Approved by: jlebon
Rather than carrying two booleans, just convert `OstreeRepoPullFlags`
into `OstreeRepoImportFlags`. This allows us to drop an internal
wrapper function and just directly call `_ostree_repo_import_object()`.
This though reveals that our mirroring import path doesn't check the
`OSTREE_REPO_PULL_FLAGS_UNTRUSTED` flag...it probably should.
Prep for further work.
Closes: #1212
Approved by: jlebon
Make the "local repo" processing conditional the same as the "localcache" bits;
this is really just a de-indent. Also add some comments. Prep for further work.
Closes: #1212
Approved by: jlebon
Noticed this while reading the code. The `child` var hasn't been
initialized yet at the time we throw this error (and even then, it's
only conditionally initialized). To be nice, let's just always calculate
the child path and pass that along.
Also do some minor style porting to decl near use.
Closes: #1216
Approved by: cgwalters
Add a few comments for each of the central functions used for committing
data from a directory. Took me a bit to understand the relationship
between those functions.
Closes: #1216
Approved by: cgwalters
This fixes up the last of the embarassing bits I saw from
the stack trace in:
https://github.com/ostreedev/ostree/issues/1184
We had a hardlink fast path, but that doesn't apply across
devices, which occurs in two notable cases:
- Installer ISO with local repo
- Tools like pungi that copy the repo to a local snapshot
Obviously there are a lot of subtleties here around things like the
bare-user-only conversions as well as exactly what data we copy. I think to get
better test coverage we may want to add `pull-local --no-hardlink` or so.
Closes: #1197
Approved by: jlebon
Add this as an additional well-known directory which is checked on
mounted removable drives to see if it contains OSTree repos we can pull
refs from.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
https://github.com/ostreedev/ostree/issues/1210Closes: #1213
Approved by: cgwalters
Introduce support for GnuTLS for computing cryptograpic
hashes, similar to the OpenSSL backend. A reason to do
this is some distributors want to avoid GPLv3, and GPG
pulls that in.
A possible extension of using GnuTLS would be replacing the GPG signing
with `PKCS#7` signatures and `X.509` keys.
We also support `--with-crypto=openssl`, which has the same effect
as `--with-openssl`, and continues to be supported.
Changes by Colin Walters <walters@verbum.org>:
- Drop libgcrypt option for now
- Unify buildsystem on --with-crypto
Link: https://mail.gnome.org/archives/ostree-list/2017-June/msg00002.html
Signed-off-by: Jussi Laako <jussi.laako@linux.intel.com>
Closes: #1189
Approved by: cgwalters
For the old `OSTREE_REPO_MODE_ARCHIVE_Z2`. Use it mostly tree
wide except for the repo finder tests (to avoid conflicting with
some outstanding PRs).
Just noted another user coming in some of those tests and wanted to do a
cleanup.
Closes: #1209
Approved by: jlebon
We added a `.dir-locals.el` in commit: 9a77017d87
There's no need to have it per-file, with that people might think
to add other editors, which is the wrong direction.
Closes: #1206
Approved by: jlebon
Add a hash function for OstreeRepo instances, which relies on the repo
being open, and hence being able to hash the device and inode of its
root directory.
Add unit tests for this and ostree_repo_equal().
Signed-off-by: Philip Withnall <withnall@endlessm.com>
https://github.com/ostreedev/ostree/issues/1191Closes: #1205
Approved by: cgwalters
Such an evil bug 🙈. I was just reading an strace trying to figure out what was
going on, and noticed we had the `XXXXXX` in the lockfile name. It was only
after that I realized that that this might *be* the cause of the skopeo issue.
This is another case where we definitely need more test coverage of things that
actually use the API multiple times in process; might look at dusting off the
work for the rpm-ostree test.
Closes: https://github.com/ostreedev/ostree/issues/1196Closes: #1204
Approved by: jlebon
While opening a repo we've recorded the device/inode for a while; use it to
avoid calling `linkat()` during object import if we know it's going to fail.
Closes: #1193
Approved by: jlebon
Conceptually `ostree-repo-pull.c` should be be written using
just public APIs; we theoretically support building without HTTP
for people who just want to use the object store portion and
do their own fetching.
We have some nontrivial behaviors in the pull layer though; one
of those is the "bareuseronly" verification. Make a new internal
API that accepts flags, move it into `commit.c`. This
is prep for further work in changing object import to support
reflinks.
Closes: #1193
Approved by: jlebon
In the `O_RDONLY` case, we were calling `openat` without a mode
argument. However, it's perfectly legal (albeit unusual) to do
`open(O_RDONLY|O_CREAT)`. One such application that makes use of this is
`flock(1)`.
This was actually caught by `_FORTIFY_SOURCE=2`, and once we run
`rofiles-fuse` with `-f`, the message is clear:
```
*** invalid openat64 call: O_CREAT or O_TMPFILE without mode ***:
rofiles-fuse terminated
======= Backtrace: =========
/lib64/libc.so.6(+0x7c8dc)[0x7f36d9f188dc]
/lib64/libc.so.6(__fortify_fail+0x37)[0x7f36d9fbfaa7]
/lib64/libc.so.6(+0x10019a)[0x7f36d9f9c19a]
rofiles-fuse[0x401768]
...
```
Without `_FORTIFY_SOURCE`, the file gets created, but its mode is
completely random.
I ran into this while investigating
https://github.com/projectatomic/rpm-ostree/pull/1003.
Closes: #1200
Approved by: cgwalters
There are use cases for not syncing at all; think build cache repos, etc. Let's
be consistent here and make sure if fsync is disabled we do no sync at all.
I chose this opportunity to add tests using the shiny new strace fault
injection. I can forsee using this for a lot more things, so I made
the support for detecting things generic.
Related: https://github.com/ostreedev/ostree/issues/1184Closes: #1186
Approved by: jlebon
Update the comments and remove an unneeded variable to make it clear
that the find_remotes_async() / pull_from_remotes_async() functions use
the unsigned summary support.
This is a follow-up of commit 8c148eb7e "lib/repo-finder: Emit
gpg-verify-summary=false in dynamic remote config".
Closes: #1195
Approved by: pwithnall
I saw in a stack trace that the main thread was calling `exit()` even while
worker threads were alive and doing sha256/write/fsync etc. for objects.
The stack trace was a SEGV as the main thread was calling into library
`atexit()` handlers and we were a liblz4 destructor:
```
#0 0x00007f2db790f8d4 _fini (liblz4.so.1)
#1 0x00007f2dbbae1c68 __run_exit_handlers (libc.so.6)
```
(Why that library has a destructor I don't know offhand, can't find
it in the source in a quick look)
Anyways, global library destructors and worker threads continuing simply don't
mix. Let's wait for our outstanding operations before we exit. This is also a
good idea for projects using libostree as a shared library, as we don't want
worker threads outliving operations.
Our existing pull corruption tests exercise coverage here.
I added a new `caught-error` status boolean to the progress API, and use it the
commandline to tell the user that we're waiting for outstanding ops.
Closes: #1185
Approved by: jlebon
We have a lot of layers of abstraction here; let's fold in the `trusted`
conditional into the call, since that's all the public API we're using does
anyways.
Prep for a future patch around object copying during imports.
Closes: #1187
Approved by: jlebon
This was some incomplete planning from while the find_remotes() API was
being designed; now totally outdated.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1179
Approved by: cgwalters
See issue #1174 for the rationale behind this. In summary:
• It required two lists of collection–refs to be maintained: one in the
repository, and one pointing to the repository.
• It didn’t automatically work for live USBs of OSs based on OSTree
(where there’s always a repository at /ostree/repo).
• It was unnecessarily complex.
The new scheme allows a list of repositories to be searched, but without
needing a layer of indirection through their collection–refs. It adds
/ostree/repo and /.ostree/repo as well-known repository locations which
are always checked on a mounted volume (if they exist).
Update the unit tests accordingly.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
https://github.com/ostreedev/ostree/issues/1174Closes: #1179
Approved by: cgwalters
This will compare their root directory inodes to see if they are the
same repository on disk. A convenience method for the users of the
public API who can’t access OstreeRepo.inode.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1179
Approved by: cgwalters
In 5c940987e7 / #646 we
added `--retain-branch-depth`; this adds a symmetric
`--only-branch` for the case where a repo owner just
wants to prune a specific branch.
The implementation here is pretty straightforward; we
just walk all refs and inject the equivalent of
`--retain-branch-depth=$ref=-1` if they're *not* in
`--only-branch`.
Closes: https://github.com/ostreedev/ostree/issues/1115Closes: #1127
Approved by: jlebon
Update libglnx, which is mostly port the repo stagedir code
to the new tmpdir API. This turned out to require some
libglnx changes to support de-allocating the tmpdir ref while
still maintaining the on-disk dir.
Update submodule: libglnx
Closes: #1172
Approved by: jlebon
A `"}"` at line 641 is missing when `HAVE_LIBARCHIVE` is not defined
(even though probably few will use ostree without libarchive).
Closes: #1181
Approved by: jlebon
Doing this in prep for libglnx tmpdir porting, but I think we should also do
this because the partial fetch code IMO was never fully baked; among other
things it was never integrated into the scheme we came up with for "boot id
sync" that we use for complete/staged objects.
There's a lot of complexity here that while we have some coverage for, I think
we need to refocus on the core functionality. The libcurl backend doesn't have
an equivalent to this today.
In particular for small objects, this is simply overly complex. The downside is
clearly for large objects like FAH's 61MB initramfs; not being able to resume
fetches of those is unfortunate.
In practice though, I think most people should be using deltas, and we need to
make sure deltas work for large objects anyways.
Further ultimately the peer-to-peer work should help a lot for people
with truly unreliable connections.
Closes: #1176
Approved by: jlebon
I was looking at fixing an `rpm-ostree livefs` bug where we need to replace
`/usr/lib/passwd`. It's obviously bad if that temporarily disappears 😉. My plan
is to do a subpath checkout of just `/usr/lib/{passwd,group}`.
Make this atomic (i.e. file always exists) by changing the logic to create a
temporary link in repo/tmp, then rename() it into place.
A bonus here is we kill one of the very few (only?) non-error-cleanup i.e.
"non-linear" `goto`s in the ostree codebase.
Closes: #1171
Approved by: jlebon
It turns out that librpm automatically merges identical files between
distinct packages, and this occurs in practice with Fedora today between
`chkconfig` and `initscripts` for exmaple.
Since we added this for rpm-ostree, we basically want to do what librpm does,
let's change the semantics to do a merge. While we're here rename
to `UNION_IDENTICAL`.
Closes: #1156
Approved by: jlebon
If the new configuration passed to ostree_write_config () tries to
update options for a remote defined in a separate config file, return an
error. Without this, the full configuration would contain duplicate
remote specifications, which would raise an error the next time the repo
is opened.
Closes: #1159
Approved by: cgwalters
Subcommands will demand a repo argument themselves. This allows one to
call `ostree remote` and get the "No subcommand" error rather than the
"Missing --repo" error.
Closes: #1126
Approved by: cgwalters
There's no need to load the sysroot for root commands which have
subcommands, like `ostree admin` and `ostree admin instutil`. Otherwise,
even just calling them without arguments will cause a failure. The
subcommands will have the appropriate flags set as needed.
Closes: #1126
Approved by: cgwalters
Convert the whole file to new style. Also tweak the help outputs to make
it similar enough to the other commands for tests to pass. Of course, we
should just centralize all subcommand handling the same way it was done
in rpm-ostree, though let's punt on that for now.
Closes: #1126
Approved by: cgwalters
Minor regression from https://github.com/ostreedev/ostree/pull/1106. We
want to print the usage text both when unknown commands are passed, as
well as when no commands are passed at all.
Closes: #1126
Approved by: cgwalters
The new libglnx `glnx_mkdtempat()` uses autocleanups, which
is inconvenient for this use case where we *don't* want autocleanups.
Since we don't need it to be fd-relative, just directly invoke
`g_mkdtemp_full()` which is fine for this use case.
Prep for updating libglnx.
Closes: #1161
Approved by: jlebon
This option allows a repo to explicitly opt out of adding new remotes in
a remotes configuration directory. This currently defaults to true for
system repos and false for non-system repos to maintain legacy behavior
that non-system repos don't add remotes in a configuration directory.
That would be problematic for flatpak, which specifies a remotes config
dir but adds remotes in ways that are incompatible with it.
So, what this really does is allow system repos to control whether they
want to add remotes in the config dir or not. That's important if your
flatpak repo is the system repo like at Endless.
Closes: #1134Closes: #1155
Approved by: cgwalters
If you lchown("symlink") then we were incorrectly trying to chown the
symlink target, rather than the symlink itself. In particular, this cause
cp -a to fail for a broken symlink. Additionally, it was using the
symlink target when verifying writability, rather than the symlink
itself.
To fix this, we need pass AT_SYMLINK_NOFOLLOW in these cases.
In general, the kernel itself will always resolve any symlinks for us
before calling into the fuse backend, so we should really never do any
symlink following in the fuse fs itself. So, we pro-actively add
NOFOLLOW flags to a few other places:
truncate:
In reality this will never be hit, because
the kernel will resolve symlinks before calling us.
access:
It seems the current fuse implementation never calls this
(faccessat w/AT_SYMLINK_NOFOLLOW never reaches the fuse fs)
but if this ever is implemented this is the correct behaviour.
We would ideally do `chmod` but this is not implemented on current kernels.
Because we're not multi-threaded, this is OK anyways.
Further, our write verification wasn't correctly handling the case of hardlinked
symlinks, which can occur for `bare` checkouts but *not* `bare-user` which the
tests were using. Change to `bare` mode to verify that.
Closes: #1137
Approved by: alexlarsson
There was only one tricky bit here around the ownership of the lines; I made use
of `g_steal_pointer()` to consistently track ownership, and converted to a `for`
loop while still preserving the loop logic around the last entry.
Closes: #1154
Approved by: jlebon
Steal some code from flatpak for this, which allows porting a few more things to
new style. I started on a public API version of this but was trying to roll some
other things into it and it snowballed. Let's do this version since it's easy
for now.
While here I changed things so that `generate_deployment_refs()` now just uses
`_set_ref_immediate()` rather than requring a txn.
Also, AFAICS there was no test coverage of `generate_deployment_refs()`; I tried
commenting it out and at least `admin-test.sh` passed. Add some coverage of this
- I verified that with this commenting out bits of that function cause the test
to fail.
Closes: #1132
Approved by: jlebon
I resisted trying to do anything invasive here like fd-relative porting as our
coverage is weak. But this was all straightforward porting to decl-after-stmt
style.
Closes: #1153
Approved by: jlebon
Before commit e0346c1, a non-system repo could specify
remotes-config-dir and have remotes read from there. However, adding
remotes would only be done in the config dir for a system repo. Restore
that by respecting remotes-config-dir when no sysroot is found and
adding back the ostree_repo_is_system() check when adding remotes.
Closes: #1133Closes: #1151
Approved by: cgwalters
Include non-default deployments in the uEnv.txt file imported by
U-Boot. All the configurations beside the defaults will have
numerical suffix E.G. "kernel_image2" or "bootargs2".
Those U-Boot environment variables may be used from interactive boot
prompt or from "altbootcmd" script.
Closes: #1138
Approved by: cgwalters
Split the code that merge the system uEnv to new function. While we're here,
clean up the logic to e.g. use `ot_openat_ignore_enoent()`.
Closes: #1138
Approved by: cgwalters
There were some important ones there like a random `syncfs()`. The remaining
users are mostly blocked on the "fstatat enoent" case, I'll wait to port those.
Closes: #1150
Approved by: jlebon
Add keys from the signing homedir to the GpgVerifier used to look
for duplicate signatures. This will allow signatures from subkeys
to be canonicalised and recognised as already signed despite the
differing key ID, avoiding duplicate signatures.
Closes: https://github.com/ostreedev/ostree/issues/608Closes: #1092
Approved by: cgwalters
In the case the signature time was bad, a line prefix was missing from the
result of `ostree_gpg_verify_result_describe_variant()`.
Closes: #1092
Approved by: cgwalters
Revert the switch of _FINGERPRINT to giving the primary key ID
rather than the signing key ID, and instead add the primary
key ID as a new attribute which is available if the key is not
missing.
Closes: https://github.com/ostreedev/ostree/issues/608Closes: #1092
Approved by: cgwalters
A lof of the functions here are async and have nontrivial exits, but these ones
are all sync were straightforward ports.
Not prep for anything, just chipping away at porting.
Closes: #1146
Approved by: jlebon
We have `ot_ensure_unlinked_at()` for the "ignore ENOENT" case, and
`glnx_unlinkat()` otherwise. Port all in-tree callers to one or the other as
appropriate.
Just noticed an unprefixed error in the refs case and decided to do a tree-wide
check.
Closes: #1142
Approved by: jlebon
I'd mostly been skipping the GPG functions due to lack of autoptr for a few
things, but I noticed these bits were straightforward.
Closes: #1136
Approved by: jlebon
The vast majority of invocations of `ot_gpgme_error_to_gio_error()` were paired
with `g_prefix_error()`; let's combine them for the same reason we do
`glnx_throw_errno_prefix()`. For the few cases that don't we might as well add
some prefix.
I also changed it to `return FALSE` in prep for more style porting.
Closes: #1135
Approved by: jlebon
I was reading the pull-local command docs and realized it was somewhat unclear
that `--untrusted` *only* applied to local repo pulls; in other words that we
always treat non-local pulls as untrusted.
Tweak the docstring, and add tests that verify this explicitly.
Closes: #1130
Approved by: jlebon
However, they weren't showing up in the output HTML and I have
no idea why; I looked at what we're doing and it looks close enough
to what's going on in `GDBusConnection` that I was using as a reference.
I'm not going to spend a lot of time to debug it right now.
Closes: #1140
Approved by: jlebon
Followup from previous patch - we can now centralize the sysroot loading.
Besides the obvious cleanup value, this is also prep for dropping an
`ostree_sysroot_get_path()` user.
Closes: #1123
Approved by: jlebon
This is exactly analogous to the `ostree init` case where
we have `OSTREE_BUILTIN_FLAG_NO_REPO` to avoid trying to load
a repo when we're creating one.
Let's avoid the pointless sysroot for `init-fs`; among other
things this will then let us do `ostree_sysroot_load()` inside
the argument parsing, and drop it from every other user.
Closes: #1123
Approved by: jlebon
Rather than calling `ostree_sysroot_get_path()`, which I'd like to deprecate for
the same reason as `ostree_repo_get_path()`.
Closes: #1123
Approved by: jlebon
In almost all places. There are just a few exceptions; one tricky bit for
example is that the repo config must still have `mode=archive-z2`, since
`archive` used to mean something else. (We could very likely just get rid of
that check, but eh, later).
I also added a test that one can still do `ostree repo init --mode=archive-z2`.
Closes: #1125
Approved by: jlebon
This is for issue projectatomic/rpm-ostree#365,
an extra option of overwrite mode is added to the checkout command
so that when there is "non-directory" file already exist
during checkout, the error will be handled.
Some tests are added for regression
Closes: #1116
Approved by: cgwalters
The new --selinux-policy added in [0] exposed a subtle issue in the way
we handle labeling during commit. The CI system in rpm-ostree hit this
when trying to make use of it[1].
Basically, because of the way we use a GVariant to represent xattrs, if
a file to be committed already has an SELinux label, the xattr object
ends up with *two* label entries. This of course throws off fsck later
on, since the checksum will have gone over both entries, even though the
on-disk file will only have a single label (in which the second entry
wins).
I confirmed that the `fsck` added in the installed test fails without
the rest of this patch.
[0] https://github.com/ostreedev/ostree/pull/1114
[1] https://github.com/projectatomic/rpm-ostree/pull/953Closes: #1121
Approved by: cgwalters
For rpm-ostree, I want to move RPM files in `/boot` to `/usr/lib/ostree-boot`.
This is currently impossible without forking the libarchive code. Supporting
this is pretty straightforward; we already had pathname translation in
the libarchive code, we just need to expose it as an option.
On the command line side, I chose to wrap this as a regexp. That should be good
enough for a lot of use cases; sophisticated users should as always be making
use of the API. Note that this required some new `#ifdef LIBARCHIVE` bits to use
the new API. Following previous patterns here, we use the new API only if a
relevant option is enabled, ensuring unit test coverage of both paths.
For the test cases, I ended up changing the accounting to avoid having to
multiply the test count.
Closes: #1105
Approved by: jlebon
This was really straightforward to implement, and is useful
for dev/test scenarios mainly like we have in rpm-ostree at least.
Closes: https://github.com/ostreedev/ostree/issues/1113Closes: #1114
Approved by: jlebon
This is a follow-up to https://github.com/ostreedev/ostree/pull/1097.
We make simple_write_deployment smart enough so that it can be used for
rpm-ostree's purposes. This is mostly an upstreaming of logic that
already existed there.
Notably we correctly append NOT_DEFAULT deployments *after* the booted
deployment and we now support RETAIN_PENDING and RETAIN_ROLLBACK flags
to have more granularity on deployment pruning.
Expose these new flags on the CLI using new options (as well as expose
the previously existing NOT_DEFAULT flag as --not-as-default).
I couldn't add tests for --retain-pending because the merge deployment
is always the topmost one. Though I did check that it worked in a VM.
Closes: #1110
Approved by: cgwalters
When working with collections it can be useful to see remote refs rather
than just local and mirrored ones. This commit changes the "ostree refs
-c" output to include remote refs, and includes remote refs with
collection IDs in summary file generation as well. The former behavior
is consistent with how "ostree refs" works, and the latter behavior is
useful in facilitating P2P updates even when mirrors haven't been
configured.
To accomplish this, OstreeRepoListRefsExtFlags was extended with an
EXCLUDE_REMOTES flag. This was done rather than an INCLUDE_REMOTES flag
so that existing calls to ostree_repo_list_refs_ext continue to have the
same behavior. This flag was added to ostree_repo_list_collection_refs
(which is an experimental API break).
Also, add unit tests for the "refs -c" and summary file behavior, and
update relevant tests.
Closes: #1069
Approved by: cgwalters
Things like https://sourceware.org/libabigail/manual/abidiff.html
look interesting but in a brief look I couldn't work out
how to conveniently use them for quick ABI sanity checking without
doing a diff from a previous build (which we could do but would be
more involved).
This way will at least catch struct ABI breaks on x86_64 which
I think we'd be most likely to do accidentally when trying
to use one of the previous unused values.
I found the hole values via gdb's `pahole` command.
Closes: #1108
Approved by: jlebon
It's been this way for a long time, though not forever; I went
back to v2014.11 as a random choice and it worked then. Not
going to bother doing a full archive search for this though.
Anyone who wants more context can help themselves.
Closes: https://github.com/ostreedev/ostree/issues/1096Closes: #1106
Approved by: jlebon
This makes `ostree commit --tree=tar` honor `--owner-uid` and `--owner-gid`
for the root directory.
Prep for further commit filtering work, although mostly for the unit test cases;
this ensures we can use `ostree checkout` after autocreating a root directory.
Closes: #1104
Approved by: jlebon
The wrapping here is unnecessary, since `_ostree_repo_commit_modifier_apply()`
already does what this function did. Further, the return type was wrong.
Saw this while doing some libarchive work.
Closes: #1104
Approved by: jlebon
For both flatpak and ostree-as-host, we really want to verify up front during
pulls that we're not being downgraded. Currently both flatpak and
`OstreeSysrootUpgrader` do this before deployments, but at that point we've
already downloaded all the data, which is annoying.
Closes: https://github.com/ostreedev/ostree/issues/687Closes: #1055
Approved by: jlebon
When using the
OSTREE_SYSROOT_SIMPLE_WRITE_DEPLOYMENT_FLAGS_NOT_DEFAULT flag, the
deployment is said to be added after the booted or merge deployment.
Fix the condition to do so instead of adding it in the second place.
Closes: #1097
Approved by: cgwalters
Follow up to <https://github.com/ostreedev/ostree/pull/1079>; I was working on
the rpm-ostree updates for this, and I think it's more consistent if we have
`.img` here, since that's a closer match to the "remove $kver" that results in
`vmlinuz`. Also just best practice to have file suffix types where they make
sense.
The astute reader might notice this sneaks in a change where we'd crash if the
legacy bootdir didn't have an initramfs...yeah, should probably have test
coverage of that.
Closes: #1095
Approved by: jlebon
This is the new Fedora kernel standard layout; it has the advantage
of being in `/usr` like `/usr/lib/ostree-boot`, but it's not OSTree
specific.
Further, I think in practice forcing tree builders to compute the checksum is an
annoying stumbling block; since we already switched to e.g. computing checksums
always when doing pulls, the cost of doing another checksum for the
kernel/initramfs is tiny. The "bootcsum" becomes more of an internal
implementation detail.
Now, there is a transition; my current thought for this is that rpm-ostree will
change to default to injecting into both `/usr/lib/ostree-boot` and
`/usr/lib/modules`, and stop doing `/boot`, then maybe next year say we drop the
`/usr/lib/ostree-boot` by default.
A twist here is that the default Fedora kernel RPM layout (and what's in
rpm-ostree today) includes a kernel but *not* an initramfs in
`/usr/lib/modules`. If we looked only there, we'd just find the kernel. So we
need to look in both, and then special case this - pick the legacy layout if we
have `/usr/lib/modules` but not an initramfs.
While here, rework the code to have an `OstreeKernelLayout` struct which makes
dealing with all of the variables nicer.
Closes: #1079
Approved by: jlebon
I noticed this with a local build of an RPM:
```
/usr/include/glib-2.0/glib/glib-autocleanups.h:28:3: warning: 'help' may be used uninitialized in this function [-Wmaybe-uninitialized]
g_free (*pp);
^~~~~~~~~~~~
src/ostree/ot-main.c:82:20: note: 'help' was declared here
g_autofree char *help;
^~~~
```
Closes: #1091
Approved by: jlebon
This is simplest for now. Compare with similar logic from
`/usr/lib/tmpfiles.d/tmp.conf`:
```
R! /tmp/systemd-private-*
```
Closes: https://github.com/ostreedev/ostree/issues/393Closes: #1090
Approved by: jlebon
I actually don't quite know what the version inheritance really does, but let's
be safe and fix this. I'm being conservative here and fixing it to inherit from
2017.8, skipping .9 since that doesn't have a parent.
Related: https://github.com/ostreedev/ostree/issues/1087Closes: #1088
Approved by: jlebon
In the production case since we used `daemon()` our stderr is `/dev/null`¹
there's not much use in logging errors from `FITHAW` or `exit(1)`, and doing so
breaks the test suite which checks the return from `waitpid()`. There's nothing
we can really do if `FITHAW` fails, and in most of those cases `EINVAL`,
`EOPNOTSUPP`, we *shouldn't* do anything anyways.
¹ Though perhaps we should set up the systemd journal, but let's not
go there right now.
Closes: #1084
Approved by: jlebon
I added `waitpid()`, but that didn't actually help because we were
`daemon()`izing. Don't daemonize if we're testing so that we can `waitpid()`.
Note I still haven't reproduced this race locally, but I'm pretty sure this will
fix it.
While here, actually check the return value from `waitpid()` just in case
something goes wrong there.
Closes: #1084
Approved by: jlebon
Just a random cozy patch I made while perusing the codebase. When
determining if two OstreeDeployment objects are the same, rather than
just checking for NULL, we can just directly check for equality of
pointers to also catch the trivial case.
Closes: #1082
Approved by: cgwalters
This is mostly the `copy_file_range` changes plus the Coverity files.
```
Colin Walters (4):
localalloc: Abort on EBADF from close() by default
local-alloc: Remove almost all macros like glnx_free, glnx_unref_variant
console: Fix Coverity NULL deref warning
fdio: Merge systemd code to use copy_file_range(), use FICLONE
Jonathan Lebon (1):
console: trim useless check
Matthew Leeds (1):
dirfd: Fix typo in comment
Philip Withnall (1):
glnx-console: Add missing NULL check before writing out text
```
Update submodule: libglnx
Closes: #1081
Approved by: jlebon
/* ATTENTION:
* Please remember to update the bash-completion script (bash/ostree) and
* man page (man/ostree-$COMMANDNAME.xml) when changing the option list.
*/
Closes: #1080
Approved by: cgwalters
This essentially completes our fd-relative conversion.
While here, I cleaned up the semantics of `ostree_repo_create()` and
`ostree_repo_create_at()` to be more atomic - basically various scripts were
testing for the `objects` subdirectory, so let's formalize that.
Closes: #820
Approved by: jlebon
If one of the localcache repos has the exact same commit we resolved
from the remote, then we need to make sure to mark it as partial so that
we download the full tree.
Closes: #1074Closes: #1076
Approved by: cgwalters
Saw this in a PR result; we need to wait for the child to have written its
result to stderr before we exit, otherwise the test suite may not read it in
time.
Closes: #1070
Approved by: jlebon
This will allow us to drop the awful hack in rpm-ostree where we watch our own
stdout. In general, libraries shouldn't write to stdout.
Also we can kill the systemd journal wrapper code. There's some duplication at
each call site now...but it's easier than trying to write a `sd_journal_send()`
wrapper.
I was originally going to have this emit all of the structured data too as a
`GVariant` but decided it wasn't worth it right now.
Closes: #1052
Approved by: jlebon
Add a new error domain for GPG signing/verification errors, and use it
throughout libostree for describing verification errors. This replaces
various uses of G_IO_ERROR_FAILED, and one instance of
G_IO_ERROR_NOT_FOUND (for which some code in ot-builtin-show.c had to be
changed to ensure it was still handled correctly).
The use of a separate error domain allows failures in GPG operations to
be handled separately from network failures (where the summary file
could not be found to be downloaded, for example) or timeouts.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1064Closes: #1071
Approved by: mbarnes
I'd like to move the new canonical kernel directory to `/usr/lib/modules/$kver`,
as Fedora has done. The `get_kernel_from_tree()` function now abstracts over
parsing the data (src vs destination filenames, as well as checksum) in
preparation for adding the new case.
In preparation for this, let's change the current test suite to use the
*current* directory of `/usr/lib/ostree-boot`, and also add coverage of `/boot`.
Closes: #1053
Approved by: jlebon
When returning results from finding repos, set gpg-verify-summary=false
in their configs, since any pulls from such remotes will necessarily
involve collection IDs, and hence should be using the unsigned summary
support. In the intended deployment mode for P2P transmission of OSTree
refs, summaries *cannot* be signed, so setting gpg-verify-summary=true
would cause all the pulls to fail.
The unsigned summary support is the move of repository metadata from
the summary file (not spliceable) to the well-known ostree-metadata ref
(spliceable, as it can exist for multiple collection IDs in the same
repository).
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1066
Approved by: cgwalters
See: http://marc.info/?l=linux-fsdevel&m=149520244919284&w=2
XFS doesn't flush the journal on `syncfs()`. GRUB doesn't know how to follow the
XFS journal, so if the filesystem is in a dirty state (possible with xfs
`/boot`, extremely likely with `/`, if the journaled data includes content for
`/boot`, the system may be unbootable if a system crash occurs.
Fix this by doing a `FIFREEZE`+`FITHAW` cycle. Now, most people
probably would have replaced the `syncfs()` invocation with those two
ioctls. But this would have become (I believe) the *only* place in
libostree where we weren't safe against interruption. The failure
mode would be ugly; nothing else would be able to write to the filesystem
until manual intervention.
The real fix here I think is to land an atomic `FIFREEZETHAW` ioctl
in the kernel. I might try a patch.
In the meantime though, let's jump through some hoops and set up
a "watchdog" child process that acts as a fallback unfreezer.
Closes: https://github.com/ostreedev/ostree/issues/876Closes: #1049
Approved by: jlebon
The API for downloading a summary file can legitimately return NULL for
the summary file contents when it returns TRUE (success). This indicates
an error 404 — the summary file was not found.
Two call sites were not handling that correctly, which was causing later
assertion failures.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1061Closes: #1065
Approved by: cgwalters
(remaining > 0) is asserted by the loop condition, and remaining is not
modified between that check and the G_UNLIKELY — so the condition in the
G_UNLIKELY will always be true.
Spotted by Coverity as issue #1452617.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1059
Approved by: cgwalters
summary_timestamp is checked for non-NULL-ness above, and the function
bails if it’s NULL.
Fixes Coverity issue #1452616.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1059
Approved by: cgwalters
This parallels ostree_repo_remote_list_refs(), but returns a map of
OstreeCollectionRef → checksum, and includes refs from collection IDs
other than the remote repository’s main collection ID.
Use this in OstreeRepoFinderConfig to ensure that refs are matched
against even if they’re stored in the repository summary file’s
collection map, rather than its main ref map. This fixes false negatives
when searching for refs in some situations.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1058
Approved by: cgwalters
This catches a few failure modes in the pull code a little earlier,
before the incorrectly-NULL repo makes its way into a closure and a
worker thread, where the cause of the problem is harder to track down.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1058
Approved by: cgwalters
As the comment explains, it’s possible for a result to be freed while
ref_to_checksum is NULL, even though normally the data structure
guarantees it’s non-NULL. This was causing crashes when results were
filtered out of a find-remotes call. Guard against that.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1058
Approved by: cgwalters
The intended behaviour of ostree_repo_find_remotes() is to return
results which have the latest version of at least one of the requested
refs. Results which have some of the requested refs, but don’t have the
latest version of any of them, should be ignored. The logic to do this
was broken in the case that a result contained a positive number of the
requested refs, but none of them were the latest version. (It previously
worked when the result contained none of the requested refs.)
Fix the counting to work correctly in both cases.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1058
Approved by: cgwalters
Coverity spotted an infloop here since we were incrementing `i++`
instead of `j++`. But adding a test revealed other bugs - we need
to keep the arrays in sync.
Coverity CID: 1452204
Closes: #1041
Approved by: pwithnall
If a delta happens to have zero objects, we could end up doing
a divide-by-zero when inferring endianness. In practice,
a zero-object delta isn't possible to generate I think, but
let's make sure the code is defensive all the same.
Spotted by Coverity.
Coverity CID: 1452208
Closes: #1041
Approved by: pwithnall
This commit sets prgname correctly so that the "ostree subcommand
--help" output prints the subcommand rather than just "ostree".
This was removed in commit f0519e541f because it tripped the thread
sanitizer, but it's being added back conditionally so most users who
don't compile with -fsanitize=adress see proper help output.
Closes: #1054
Approved by: cgwalters
Part of cleaning up our usage of libglnx; we want to use what's in GLib where we
can.
Had to change a few .c files to `#include ostree.h` early on to pick up
autoptrs for the core types.
Closes: #1040
Approved by: jlebon
There are multiple use cases where we'd like to alias refs.
First, having a "stable" alias which gets swapped across major
versions: https://pagure.io/atomic-wg/issue/228
Another case is when a ref is obsoleted;
<https://pagure.io/atomic-wg/issue/303>
This second one could be done with endoflife rebase, but I think
this case is better on the server side, as we might later change
our minds and do actual releases there.
I initially just added some test cases for symlinks in the `refs/heads` dir to
ensure this actually works (and it did), but I think it's worth having APIs.
Closes: #1033
Approved by: jlebon
I plan to at some point change rpm-ostree to read the journal messages from
libostree and render things like the time we spent in syncfs().
Closes: #1044
Approved by: jlebon
Define typedefs for read/write archives, and use the GLib
autocleanups for them. Prep for updating libglnx to drop its
custom autocleanup macros.
Closes: #1042
Approved by: jlebon
Prep for dropping `GLNX_DEFINE_CLEANUP_FUNCTION` from libglnx
in favor of using GLib's `G_DEFINE_AUTO_CLEANUP_FREE_FUNC()`.
Closes: #1042
Approved by: jlebon
It looks like `curl_multi_socket_action()` will return an error
if *one* of the requests has an error, but we already check
for that explicitly by iterating over each handle.
In libcurl, the "easy" layer doesn't really make use of this
return value. I did a bit of looking elsewhere; systemd
does check it as a runtime error, not an assertion. librepo
doesn't use the multi interface.
Closes: https://github.com/ostreedev/ostree/issues/1035Closes: #1038
Approved by: jlebon
Coverity complained that the `else if (bytes_read == 0)` was dead
code if we happened to find it was already false when testing
`else if (G_UNLIKELY (bytes_read == 0 ...`.
There was nothing wrong with the logic, but let's rework it to
only test the value once; I think it does end up nicer anyways.
Coverity CID: 1452186
Closes: #1037
Approved by: jlebon
No real problems here, but Coverity likes to see consistent checking of return
values, and I agree with it.
Coverity CID: 1452213
Coverity CID: 1452211
Closes: #1037
Approved by: jlebon
The fingerprint associated with each signature can be different to
the primary key ID (the normal one that people use to identify a
GPG key) if the signature is from a signing subkey. Try to find the
primary key and print this ID in preference to the subkey signature.
https://github.com/ostreedev/ostree/issues/608Closes: #1036
Approved by: cgwalters
Use gpgme_get_key to find the primary key for the key we are
looking for, and the primary key for each signature, and
compare these when looking up signatures.
The primary key is the first in the list of subkeys, which is
the normal key ID people use when referring to a GPG key as an
identity.
If the key has a signing subkey, signature->fpr will not match
the provided key_id, so looking up both keys and comparing the
primary key fingerprints ensures they are both canonicalised.
https://github.com/ostreedev/ostree/issues/608Closes: #1036
Approved by: cgwalters
This is a continuation of addition of journaling to libostree; see
e.g. <https://github.com/ostreedev/ostree/pull/708>.
I wanted more information at the end of fetches; in particular
some details about the delta execution (what opcodes etc.), but
this is a first step: we log things like the transferred data
as well as whether or not GPG was enabled, etc.
One awkward thing about this is how we map the fetcher options like
`tls-ca-path` back out into an enum for the code to log. But eh, hard to fix
without a bigger refactoring.
Closes: #1032
Approved by: jlebon
These were previously private, but since we expect people to use them, let's add
`#define`s like we did for some of the other commit metadata.
Closes: #1028
Approved by: jlebon
Mostly for the latest `-Wmaybe-uninitialized` fix, but while here also port some
places to newer APIs.
Update submodule: libglnx
Closes: #1027
Approved by: jlebon
Regression from previous tmpfile refactoring; unfortunately
the `OSTREE_REPO_COMMIT_MODIFIER_FLAGS_GENERATE_SIZES` option
only has coverage via gjs currently.
Might expose it via the cmdline in a later option, but in the big picture the
idea was that this data is better kept in static deltas.
Closes: https://github.com/ostreedev/ostree/issues/1014Closes: #1016
Approved by: jlebon
See: https://github.com/projectatomic/rpm-ostree/issues/885
If we get a successful Apache directory listing HTML when fetching what we
intend to be a ref, we'd dump the HTML into the error.
I did some scanning of the pull code, and this was the only case
I saw offhand where we were dumping text out into an error. Which
makes sense, since most of our formats are binary, the exeptions I
think are just `repo/config` and `repo/refs/`.
Closes: #1015
Approved by: mbarnes
Using the error prefixing in the delta processing allows us to
do new code style. Also strip trailing whitespace.
Use error prefixing in a few other random places. I didn't
hunt for all of them, just testing out the new API.
Use `glnx_fchmod()`. Also note I dropped one `fchmod (tmpf, 0600)`
which is no longer necessary.
Update submodule: libglnx
Closes: #1011
Approved by: jlebon
Prep for `ostree_repo_new_at()`. Down the line perhaps
we should extend libcurl to accept a file descriptor for cookies,
but this works OK for now.
Closes: #1010
Approved by: jlebon
Prep for `ostree_repo_new_at()`. These commands were directly accessing
`repo->repodir`, which it turns out was unnecessary since the the APIs they then
used were fd-relative. Except actually there were bugs there, so fix all of the
cookie util code to actually use the passed `dfd` and not just hardcode
`AT_FDCWD`.
Also, libsoup can't handle this (its APIs require fully qualifed paths), and
there's not a really good reason to have two implementations now; historically
it was useful to cross-check them, but I don't think we need that.
While I'm here, port to new style.
Closes: #1010
Approved by: jlebon
(Note this PR was reverted in <https://github.com/ostreedev/ostree/pull/902>;
this version should be better)
Using `${sysroot}` to mean the physical storage root: We don't want to write to
`${sysroot}/etc/ostree/remotes.d`, since nothing will read it, and really
`${sysroot}` should just have `/ostree` (ideally). Today the Anaconda rpmostree
code ends up writing there. Fix this by adding a notion of "physical" sysroot.
We determine whether the path is physical by checking for `/sysroot`, which
exists in deployment roots (and there shouldn't be a `${sysroot}/sysroot`).
In order to unit test this, I added a `--sysroot` argument to `remote add`.
However, doing this better would require reworking the command line parsing for
the `remote` argument to support specifying `--repo` or `--sysroot`, and I
didn't quite want to do that yet in this patch.
This second iteration of this patch fixes the bug we hit the first time;
embarassingly enough I broke `ostree remote list` finding system remotes.
The fix is to have `ostree_repo_open()` figure out whether it's the same
as `/ostree/repo` for now.
Down the line...we might consider having the `ostree remote` command line itself
instatiate an `OstreeSysroot` by default, but this maximizes compatibility; we
just have to pay a small cost that `ostree` usage outside of that case like
`ostree static-delta` in a releng Jenkins job or whatever will do this `stat()`
too.
Closes: https://github.com/ostreedev/ostree/issues/892Closes: #1008
Approved by: mbarnes
This came up in <https://github.com/ostreedev/ostree/pull/982>; when
we added more direct local importing, we did it synchronously.
This was actually quite a regression when doing local pulls between different
modes; in particular between a bare mode and `archive`, as we were suddenly
doing gzip {de,}compression in the main thread.
Down the line actually...a simpler fix is probably to change things so that the
local path is really only used when we know we can hardlink; everything else
would go though the fetcher codepath but with `file://`.
But this isn't a lot more code, and the speed/interactivity win is large.
Note we're only doing content async with this patch. We could do metadata as
well; we have the object already local. But the metadata code path is messier,
and metadata objects are smaller.
Another area where this comes up is that in e.g. Fedora releng, most operations
talk to a NetApp via NFS. So this has the classic network filesystem problem
that operations that are normally cheap like `stat()` can actually have
nontrivial latency. Doing as much as possible in threads is better there too.
Closes: #1006
Approved by: jlebon
There is no actual written guarantee in glib-mkenums that the template
line specified using --fhead will be added after the templates specified
inside the template file. Since the template file is only used once, we
can simply move the `#include` directive inside the template, so that it
is guaranteed to be in the right place.
Closes: #1007
Approved by: cgwalters
Currently in Fedora we don't sign summaries, and every use of
`rpm-ostree` would emit to the journal an error when we failed
to fetch it.
Fix this by having `OSTREE_FETCHER_REQUEST_OPTIONAL_CONTENT` tell the fetcher
not to journal 404 errors. While fixing this, we had a mix of two booleans vs
the flags; fix things so we consistently use the flags in the fetcher and pull
code.
Closes: #1004
Approved by: mbarnes
As discussed in https://github.com/ostreedev/ostree/pull/946, the
summary file is becoming an unsigned cache of ref information; any
additional metadata for the repository needs to move elsewhere in order
to remain signed. Introduce OSTREE_REPO_METADATA_REF as the well-known
name of a ref where such metadata can live, as the metadata on
contentless commits.
Don’t yet update the documentation for summary-related methods to
mention this, since it’s still hidden behind the
--enable-experimental-api configure option.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #946
Approved by: cgwalters
Use goffset rather than gsize for file sizes. More importantly, get the
unpacked_size from g_file_info_get_size() (goffset) rather than from the
splice return value, which has type gssize.
This will make a difference on 32-bit systems, where goffset is defined
as off64_t, but gsize is 32 bits.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #999
Approved by: cgwalters
When we [switched to using checkout + force_copy](e8efd1c8dc),
a side effect that went unnoticed at the time is that we started
setting directory mtimes to zero.
See the below bug where we long ago set the file times to zero, which got fixed,
so let's not regress things by setting the directory times to zero either. (Even
though AFAICS GNU tar doesn't complain about those)
This semantic is somewhat "overloaded" onto `force_copy`, but it avoids adding
yet another boolean; we don't have that many reserved boolean slots left. I
can't really think of many good use cases for `force_copy` *other* than the
`/etc` merge anyways.
https://bugzilla.redhat.com/show_bug.cgi?id=1229160
Closes: https://github.com/ostreedev/ostree/issues/995Closes: #997
Approved by: jlebon
There are a number of simple ports here. Prep for further work
in `/etc` merge.
I also stripped trailing whitespace globally.
Closes: #996
Approved by: jlebon
This verifies the collection and ref bindings in the commit metadata
against the collection ID we have stored in the remote config and ref
we want to pull from. For the HEAD commits, we also check if the
checksum of the commit we just fetched agrees with the checksum we
really wanted to pull from the ref.
For commits with explicitly specified checksums and without specified
refs, we only verify if the commit has the bindings. We are able to
only verify the collection binding, though.
Closes: #972
Approved by: cgwalters
The collection and ref bindings are stored in the commit metadata
under ostree.collection-binding and ostree.ref-binding,
respectively. They will be used to verify if the commit really comes
from the collection and ref we wanted to pull from.
Closes: #972
Approved by: cgwalters
And in general, if for some reason we can't write `user.` xattrs, provide an
error immediately rather than doing it during a later pull. This way the failure
cause is a lot more obvious.
Related: https://github.com/ostreedev/ostree/issues/991Closes: #993
Approved by: jlebon
In the storage PR I was trying to do a `pull-local` of the whole
`/ostree/repo` on the system, which ended up triggering a `g_critical()`
in the collections code, since we tried to parse a remote-prefixed ref
`fedora:fedora/26/x86_64/atomic-host` as a ref.
I'm not sure offhand what our behavior in this case *should* be. I
think git only clones local refs, but I need to check.
This corner case arises only with `pull-local`. But in any case,
while we were previously saying this is programmer error, since it's
so easy to pass various unchecked input into the pull machinery,
make invalid refs an explicit error.
Closes: #992
Approved by: jlebon
For ostree-as-host, we're the superuser, so we'll blow past
any reserved free space by default. While deltas have size
metadata, if one happens to do a loose fetch, we can fill
up the disk.
Another case is flatpak: the system helper has similar concerns
here as ostree-as-host, and for `flatpak --user`, we also
want to be nice and avoid filling up the user's quota.
Closes: https://github.com/ostreedev/ostree/issues/962Closes: #987
Approved by: jlebon
This is prep for storage space checks, where we look at free
space after parsing the metadata, before we write anything.
We did length-limited writes in the fd-based input path, but not for the
`GInputStream` path which in practice is used for HTTP pulls.
Closes: #987
Approved by: jlebon
Some of the Jenkins jobs for Fedora Atomic Host broke after updating
to 2017.7, and it turns out that we regressed handling unreadable
files in `bare-user` mode. An example of this is `/etc/shadow`, which
ends up in the ostree-as-host content as `/usr/etc/shadow`.
Now there are better fixes here; we should probably delete it and create it
during the config merge if it doesn't exist. In general, having secret files in
ostree really isn't supported, so it doesn't make sense to include them.
But let's fix this regression - when operating as an unprivileged user we don't
have `CAP_DAC_OVERRIDE` and hence will fail to open un-user-readable objects.
(We still preserve the actual `0` mode of course in the xattr and will
apply it in `bare`)
Closes: #989
Approved by: jlebon
Previously, we only supported additions in the statoverride file;
it was mainly for adding the setuid bit without having that physically
on disk.
However, for testing a change to `bare-user` handling around *unreadable*
files (which happens for `/etc/shadow` in host content), I need a way
to write that into a repo in the test suite.
I'm not actually aware of a non-test-suite use case for this; a more
sophisticated user is going to be using the API directly, which can already do
this. But we need it for tests at least.
Closes: #989
Approved by: jlebon
I had thought `glnx_link_tmpfile_at()` actually consumed the tmpfile;
it does consume the *path* but not the fd. In the non-delta path
things were fine since we used the autocleanup.
But the delta code had a tmpfile allocated in its struct that got reused, and
hence leaked the fd. Fix this by making the commit API actually consume the
tmpfile fully, just like the path path.
Closes: #986
Approved by: jlebon
This is a lot like `git clone --reference`, but we chose "localcache" as the
term "reference" is already used.
The main use case I'm targeting this for is the Fedora Atomic Host installer
case where we embed the repo content in the installer, but we may want to
kickstart and download newer content. There, while we want to get a newer ref,
we can still use the local repo as an object cache, since we have it sitting
there in memory anyways.
Another case is where one has a host ostree (say e.g. Fedora Atomic
Workstation), and one wants to create a local archive mirror of FAH. Then one
can use `pull --reference /ostree/repo` and pull the common objects (e.g.
contents of `bash.rpm` etc.)
Closes: https://github.com/ostreedev/ostree/issues/975Closes: #982
Approved by: jlebon
These are regression from #971. We were stuffing a pointer size inside a
variable of integer size. So the assignment was spilling over into other
variables' storage space. Actually use a gpointer and GPOINTER_TO_[U]INT
as was done originally.
Also bump libglnx which has static checks for this error in the future.
Update submodule: libglnx
Closes: #990
Approved by: cgwalters
An inverted condition in _ostree_repo_add_remote() was causing the
OstreeRepoFinder to delete precisely the wrong remote
configurations from memory once it was finished. It’s supposed to delete
the ones which it transiently added; but was instead deleting all the
existing remote configurations.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #985
Approved by: cgwalters
It's more natural for a few calling places. Prep for patches to go the other
way, which in turn are prep for adding a commit filter v2 that takes `struct
stat`.
`ot_gfile_type_for_mode()` was only used in this function, so inline it here.
Closes: #974
Approved by: jlebon
Previously, `ostree pull` was silent if not on a tty. I don't
see a reason not to print the final status line at least. This
is prep for more work in the test suite, so I can write assertions
on the output.
But it should also be nicer for people who e.g. do an `ostree pull` in a Jenkins
job or whatever.
Closes: #981
Approved by: jlebon
The `-z2` is annoying now since it's really a legacy; we've long
since supported typing `archive`. Convert the docs fully and
explain that.
Also do some (but not all) of the tests just to encourage newer tests to use
`archive` too.
Closes: #980
Approved by: jlebon
Obviously very minor, but I noticed this while working on `pull --reference`. If
we have a local repo, we'll have already done a hardlink and copied the detached
metadata too, so there's no reason to request it again via the fetcher path.
Closes: #978
Approved by: jlebon
I'd mostly been avoiding this file since there's always patches outstanding, but
these few functions shouldn't conflict much.
Closes: #979
Approved by: jlebon
I added `glnx_open_anonymous_tmpfile()`, but then later noticed
that the usage of this was really to be combined with `mmap()`,
and we had two versions of that in the delta code. Add a helper.
(Bigger picture...how is this different from glibc's "mmap() of /dev/zero"
approach for large chunks? One advantage is the storage can be "swapped" to
`/var/tmp`, but still deleted automatically, rather than requiring swap space)
Closes: #973
Approved by: jlebon
Use the new macros introduced recently in libglnx to make iterating over
hash tables cleaner. This is just a start, it does not migrate the whole
tree.
Update submodule: libglnx
Closes: #971
Approved by: cgwalters
There's lots of mechanically replacing `OtTmpFile` with `GLnxTmpfile`;
the biggest changes are in the commit path. Symlink commits are now
very clearly separated from regular files. Symlinks are `OtCleanupUnlinkat`,
and regular files are `GLnxTmpfile`.
The commit codepath separates those as `_ostree_repo_commit_path_final()` and
`_ostree_repo_commit_tmpf_final()`. A nice aspect of all of this is that they
both *consume* the temporary on success. This avoids an extra spurious
`unlink()` call.
One of the biggest bits of code motion is in `commit_loose_regfile_object()`,
which no longer needs to care about symlinks. For the most parth though it's
just removing conditionals.
Update submodule: libglnx
Closes: #958
Approved by: jlebon
Rather than duplicating the code. This introduces no functional changes.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #961
Approved by: cgwalters
Check that it’s in normal form and has the correct type when loading it,
since it could come from an untrusted source.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #961
Approved by: cgwalters
This makes no difference to the validity of the code, since any summary
file loaded from the cache will be verified before being read anyway;
but it will make some upcoming changes a little simpler.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #961
Approved by: cgwalters
When updating a summary file, parse additional arguments to the `ostree
summary` command as additional metadata to be put into the summary.
Add some tests for this.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #961
Approved by: cgwalters
Computing download/storage size for `archive` pulls is hard; there's
`OSTREE_REPO_COMMIT_MODIFIER_FLAGS_GENERATE_SIZES` which was from a
pre-static-deltas effort by Endless, but we aren't currently making use of this
much.
Static deltas were designed to solve this problem; we have the total
uncompressed size. Let's check free space before doing a delta pull.
Related: https://github.com/ostreedev/ostree/issues/962Closes: #963
Approved by: jlebon
The `OstreeRepoContentBareCommit` struct was basically an `OtTmpFile`, so let's
make it one. I moved the "convert to `GOutputStream`" logic into the callers,
since that bit can't fail; it makes the implementation much simpler since we can
just return the result of `ot_open_tmpfile_linkable_at()`.
Prep for `GLnxTmpfile` porting.
Closes: #957
Approved by: jlebon
The pull code also could make use of this in both the metadata and content
paths. I changed it to own the tempfile malloc (just like `GLnxTmpFile`), since
there's no reason to have different lifetimes for the filename and the file, and
that way we only have one variable rather than two.
The content path turns out to be a special case though, where
at least for mirroring archives, we directly pass the file *path*
down into `_ostree_repo_commit_loose_final()`.
This is prep for `GLnxTmpFile` porting.
Closes: #957
Approved by: jlebon
The variables here were duplicative; we don't need two booleans to distinguish
between symlinks and regular files. What we do need to handle is the "physical"
state versus the "object" state. Symlinks objects are stored as regular files in
`bare-user` and `archive`.
Prep for more cleanup.
Closes: #957
Approved by: jlebon
Instead of treating it as a programming error — given that it’s user
input, that’s not really appropriate. This modifies write_ref() and
list_collection_refs() to implement validation.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
These utilities were not needed for the initial port to support
OstreeCollectionRef, so have been delayed a bit and, in some cases, left
as FIXME comments for follow up later.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
If a repository’s summary file includes a collection ID, output that. If
it includes refs from other collections (in the ‘collection map’), output
those and include the same metadata detail as for refs in the summary
file’s main refs map.
If collection IDs are specified in the summary file, this changes the
output format from `ostree summary -v` to use (collection ID, ref name)
tuples.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
Rather than change the output format used by the existing refs command
to output collection IDs in addition to ref names, this functionality has
been hidden behind an --collections argument. If it’s not specified `ostree
refs` will output the same content as before for a given repository. If
it is specified, the collection ID for each ref will be included in the
output as (collection ID, ref name).
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
This allows newly configured remotes to have their collection ID specified,
so that refs from them can be downloaded from peers as well as the
upstream collection, using the remote’s configuration.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
This allows new repositories to be configured with a collection ID which
can be used to uniquely identify refs which originated from this
repository.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
This will pull the remotes after finding them. This potentially needs to
go in its own pull-from-remotes built-in command, but it will be fine
here for now.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
This is a wrapper around the new ostree_repo_find_remotes() method; it
tries to find available remotes which can serve updates for the
user-provided refs.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
This is a more complex implementation of OstreeRepoFinder which resolves
ref names to remote URIs by looking for refs advertised by peers on the
local network using DNS-SD records and mDNS (Avahi). The idea is to
allow OS and app updates to be propagated over local networks, without
the internet.
It requires an OSTree server and code to generate the DNS-SD adverts in
order to be fully functional — support for this will be added
separately.
Unit tests are included.
Includes fixes by Krzesimir Nowak <krzesimir@kinvolk.io>.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
This will be used in an upcoming commit. It adds a basic bloom filter
implementation, using the SipHash family of hash functions.
The implementation (including its parameter choices and hash functions)
will become a protocol detail in future, so must not be changed so that
its output is bitwise incompatible between OSTree versions.
Unit tests are included.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
This is a basic implementation of OstreeRepoFinder which resolves ref
names to remote URIs by looking for them on any currently mounted
removable storage volumes. The idea is to support OS and app updates via
USB stick.
Unit tests are included.
This bumps libostree’s maximum GLib dependency from 2.44 to 2.50 for
g_drive_is_removable(). If GLib 2.50 is not available, the call which
needs it will be omitted and the OstreeRepoFinderMount implementation
will scan all volumes (not just removable ones); this is a performance
hit, but not a functionality hit.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
This is a basic implementation of OstreeRepoFinder which resolves ref
names to remote URIs by looking their collection IDs up in the local
configuration of remotes who have their collection-id key set.
Unit tests are included.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
Add an initial OstreeRepoFinder interface (but no implementations),
which will find remote URIs by ref names and collection IDs, the
combination of which is globally unique.
The new API is used in a new ostree_repo_find_updates() function, which
resolves a list of ref names to update into a set of remote URIs to pull
them from, which can be treated as mirrors. It is an attempt to
generalise resolution of the URIs to pull from, and to generalise
determination of the order and parallelisation which they should be
downloaded from in.
Includes fixes by Krzesimir Nowak <krzesimir@kinvolk.io>.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
This adds a new collection-refs option of type a(sss), giving a list of
(collection ID, ref name, checksum) tuples to pull from the given remote.
This option is intended to supersede the refs and override-commit-ids
options, so is mutually exclusive with them.
This includes support for resolving the refs from the remote’s summary
file, or from its refs/heads and refs/mirrors directories.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
These are tuples of (collection ID, ref name) which are a globally-unique
form of local ref. They use OstreeCollectionRef as an identifier, and hence
need to be accessed using new API, as the existing API uses string
identifiers and sometimes accepts refspecs. Remote names are not
supported as part an OstreeCollectionRef.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
Add {get,set}_collection_id() methods to OstreeRepo and some documentation
about the concept of a collection ID which globally identifies an
upstream repository. See the documentation for more details.
This will be used in future commits. For now, the new API is marked as
experimental (--enable-experimental-api).
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
This is a type representing the tuple (collection ID, ref name), which is
guaranteed to be globally unique. It will be used in upcoming commits.
It introduces the concept of a ‘collection’ which is a unique, curated
set of refs which lie in the same trust domain (i.e. all signed by the
same key and validated by the same developer). Flathub might be a
collection, for example; or the set of OS refs coming from a particular
OS vendor.
It includes a function for validating collection IDs.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
This will make some future additions to regenerate_summary() easier.
This commit introduces no functional changes.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
This is followon work from previous cleanups. Basically
`stat_bare_content_object()` was the `fstatat()` logic
and `ostree_repo_read_bare_fd()` was the `openat()` implementation;
they duplicated some bits to find the object in staging, recurse
into parent etc.
Further, I wanted an internal-only version of this API which didn't allocate
`GFileInfo`/`GInputStream` but used a plain `fd` and `struct stat` to avoid
mallocs.
The end version here I think looks a lot nicer, since we deduplicate the various
`open()` calls in the different cases for example.
Closes: #952
Approved by: jlebon
Prep for future cleanup patches (in particular I want an internal-only
version at first that uses a fd+`struct stat`) to avoid allocations.
The new version avoids lots of deep nesting of conditionals as well
by hoisting the "not found" handling to an early return.
There's a bit of code duplication between the two cases but it's
quite worth the result.
Closes: #951
Approved by: jlebon
Prep for a change in `ostree_repo_load_file()`. We would crash if a
caller had `out_file_info = NULL`, because we deref `ret_file_info`
below it.
Closes: #951
Approved by: jlebon
There was a typo in it when it was first introduced. Let’s not live in
the past.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #950
Approved by: cgwalters
There was a typo in the group name. It should be OK to change the
version since this is all hidden behind the --enable-experimental-api
configure option.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #950
Approved by: cgwalters
There are a few places in the code where ad-hoc validation was being
performed. Might as well formalise it a bit more.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #948
Approved by: cgwalters
And use it in `test-symbols.sh`, to fix the `distcheck` case;
the previous change stopped distributing `libostree-devel.sym`
in release builds.
Closes: #944
Approved by: jlebon
Change the annotation of the out parameters on ostree_repo_load_file
from `(allow-none)` to `(optional) (nullable)`. `allow-none` is
ambiguous, since these parameters can be both NULL on input and set to
NULL on return.
Closes: #939
Approved by: cgwalters
Flatpak make check is failing when applying a static delta
to a bare-user-only repo due to an assert. The fix is to add
bare-user-only to the assert check.
Closes: #940
Approved by: giuseppe
We're hitting this in flathub, where we have a bunch of local builds,
but we also mirror a few refs from the gnome runtime repo into it.
Its fixable by re-doing the summary, but for a short time the
wrong version is visible.
Fixes https://github.com/ostreedev/ostree/issues/846Closes: #935
Approved by: cgwalters
The code here tried to truncate the string to the previous length,
but that doesn't work when recursing, since further calls change the
length.
What actually ended up happening was the string would get corrupted
after the first level of recursion.
Closes: #936
Approved by: jlebon
So far a lot of submitted PR have added symbols into the first
section. Split the file into `-released` and `-devel` to make
this more obvious.
To further enforce things, we hardcode a checksum of the `-released`
file in `test-symbols.sh`. Only release commits should update that
checksum.
Did you notice I like checksums?
Closes: #931
Approved by: pwithnall
This is especially interesting for the versioning symbols themselves,
as it is an indicator of when applications using introspection information
can start to use a symbol in the library to check if they have a
recent enough version of OSTree to use.
Closes: #932
Approved by: cgwalters
For the flatpak PR: https://github.com/flatpak/flatpak/pull/849
It's really more convenient if this works for HTTP pulls as well, since flatpak
does various types of pulling, and we can just set the flag everywhere.
Further, we might as well reject the content as early as possible.
Closes: #930
Approved by: alexlarsson
We saw this fail in a CI run. We've been trying to add strategic error prefixing
as a general rule, and this specific instance may help debug.
Closes: #929
Approved by: pwithnall
This imports a function that is used in rpm-ostree, and it's also intended for
use by https://github.com/advancedtelematic/aktualizr to display
what deployment we're going to boot next after the reboot.
Updated-by: Colin Walters <walters@verbum.org>
Closes: #897
Approved by: OYTIS
This is a continuation of https://github.com/ostreedev/ostree/pull/926
for directories instead of files.
See: https://github.com/flatpak/flatpak/issues/845
This option suppresses mode bits outside of `0775` for directory
checkouts. I think most people should start doing this by default,
and use explicit overrides for e.g. `/tmp` if doing a recommit based
on a checkout.
Closes: #927
Approved by: alexlarsson
This is an option which is intended mostly for flatpak;
see: https://github.com/flatpak/flatpak/issues/845
We're adding an option for pulling into *all*
repo modes that has an effect similar to the `bare-user-only`
change from https://github.com/ostreedev/ostree/pull/909
This way one can pull content into e.g. a root-owned `bare` repository and
ensure that there aren't any setuid or world-writable files.
Closes: #926
Approved by: alexlarsson
Thinking about the problem of flatpak converting from `bare-user` to `bare-user-only`
"in place" by creating a new repo and doing a `pull-local`, I realized
that we can optimize this process by doing hardlinks for both metadata
and regular files. The repo formats are *almost* compatible, the
exception being symlinks.
An earlier patch caused us to do hardlinks for metadata, this patch takes things
to the next step and special cases this specific conversion. In this case we
need to parse the source object to determine whether or not it's a symlink.
Closes: #922
Approved by: alexlarsson
Our previous logic for import-via-hardlink only tried if the repo modes match,
but we *can* hardlink metadata between e.g. `archive` and `bare-user` repos, and
that's quite useful thing to do. Our documentation encourages converting to/from
those repo modes locally for build systems.
Closes: #922
Approved by: alexlarsson
Before this, if one had repos of matching mode but different owners,
which could happen if one e.g. makes a `bare` non-root repo in
`/ostree/deploy/$stateroot/var/tmp`, every time we tried to call `linkat()`
we'd get `EPERM` and fall back to a copy.
Fix this by saving the repo owner uid, and avoid trying to call `linkat()` if we
know it's going to fail. Of course most commonly in this scenario we'll
immediately fail trying to `chown` the files to `0`, but this is prep for a
future patch to improve `bare-user` → `bare-user-only` imports where we'll be a
bit more sophisticated.
Closes: #922
Approved by: alexlarsson
This code looks like it was supposed to build a refspec, but it used a
slash as a separator rather than a colon. The following code does
recover by supporting prefix matching with slashes, but it seems like
this was perhaps not the intention.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #912
Approved by: cgwalters
Its often the case that we want to look at objects inside a commit,
before the objects the transaction is finished. For instance:
https://github.com/flatpak/flatpak/pull/837
Which tries to verify the file permissions before committing the
transaction.
And:
1e5ffa926a
Which collects the storage size of the objects so that we can
put the total download size in the commit metadata.
I tried to find all the places where we did reads from the
object directories, and in particular this fixes:
- `ostree_repo_load_file()` for `bare` repos (`archive` was already working).
- `ostree_repo_query_object_storage_size()`
- Applying deltas that reference not-yet-commited objects
Closes: #916
Approved by: cgwalters
This came up in: https://github.com/ostreedev/ostree/pull/881
Basically doing streaming for metadata is dumb. Split up the metadata/content
paths so we pass metadata around as `GVariant`. This drops the last internal
caller of `ostree_repo_write_metadata_stream_trusted()` which was the dumb
function mentioned.
Closes: #923
Approved by: jlebon
See https://github.com/ostreedev/ostree/pull/909 for more information on the
rationale. Basically there's no reason for flatpak (which uses `bare-user-only`)
to have world-writable dirs. Particularly with the presence of the system
helper.
An approach I considered instead was to parse and validate directory metadata
objects at commit time. We still may do that in addition; for file objects we *had*
to do it that way because the actual files would be laid down suid. But directories
live only as inert `.dirmeta` objects until we do a checkout (i.e. `mkdir()`), so
we can solve the problem at checkout time.
Closes: #914
Approved by: alexlarsson
Both callers of `commit_loose_object_trusted()` were passing
`OSTREE_OBJECT_TYPE_FILE`, so drop that parameter. This in turn
allows us to drop lots of checking of that inside the function.
Add a doc comment, and rename to `commit_loose_content_object()` for clarity.
Closes: #914
Approved by: alexlarsson
I noticed my previous patches incorrectly started doing `return glnx_throw*`
inside a `goto out;` function. Fix this by porting forward consistently to new
style. We just do the error prefixing in the caller.
Closes: #914
Approved by: alexlarsson
When a transaction is finished and we have moved all the staged loose
objects into the repo we fsync all the object directory, to ensure the
filenames are stable before we update the refs files to point to the
new commits.
With out this an unclean shutdown after the transaction is finished
could result in a refs file that points to an incomplete commit.
https://bugzilla.gnome.org/show_bug.cgi?id=759442Closes: #918
Approved by: cgwalters
These exist in the wild for flatpak, and aren't really a problem. The canonical
permissions are still either `0755` or `0644`, we just support the additional
writable bit for the group (i.e. extend the set to include `0775` and `0664`)
now to avoid breaking some flatpak content.
Closes: #913
Approved by: alexlarsson
Having every object in a bare-user repo (and checkouts) be executable
is ugly. I can't think of a good reason to do that; they should only
be executable if their input is. This does
for `bare-user` what we did for `bare-user-only` in
https://github.com/ostreedev/ostree/pull/909
It's also a stronger version of what we do with `checkout -U` in suppressing
suid - here we also strip world-writable files and the sticky bit (even though
that's meaningless today, it might not be in the future).
Closes: https://github.com/ostreedev/ostree/issues/907Closes: #908
Approved by: alexlarsson
This is only used internally (the header is not public), so it doesn’t
have to go in ostree-autocleanups.h. It will be used in some following
commits.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #911
Approved by: cgwalters
If there are no deltas to be listed in the summary file, don’t bother
including the key for them in the additional metadata section of the
file. This saves a few bytes in some cases.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #911
Approved by: cgwalters
This follows on from commit a946c3d4, which added formatting for
well-known summary metadata keys. This commit adds it for commits.
Currently, the only well-known commit metadata key is
ostree.commit.timestamp. Formatting this correctly is especially
important, since it’s a big-endian uint64, which is completely unusable
for mere mortals when presented as a number rather than a date.
Non-formatted output can still be retrieved using the OSTREE_DUMP_RAW
flag, and the non-formatted key name is always printed for clarity.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #911
Approved by: cgwalters
Colin Walters
Dusty Mabe
Dan Nicholson
Carlos Alberto Lopez Perez
Philip Withnall
Kalev Lember
Shaun Taheri
Alexander Larsson
Simon McVittie
Ruixin Bao
Jonathan Lebon
Matthew Leeds
Matthew Leeds
Jussi Laako
Anton Gerasimov
yceh
Guy Shapiro
Robert McQueen
Matthew Barnes
Emmanuele Bassi
Krzesimir Nowak
David Shea
Tristan Van Berkom