In the case the signature time was bad, a line prefix was missing from the
result of `ostree_gpg_verify_result_describe_variant()`.
Closes: #1092
Approved by: cgwalters
Revert the switch of _FINGERPRINT to giving the primary key ID
rather than the signing key ID, and instead add the primary
key ID as a new attribute which is available if the key is not
missing.
Closes: https://github.com/ostreedev/ostree/issues/608Closes: #1092
Approved by: cgwalters
A lof of the functions here are async and have nontrivial exits, but these ones
are all sync were straightforward ports.
Not prep for anything, just chipping away at porting.
Closes: #1146
Approved by: jlebon
We have `ot_ensure_unlinked_at()` for the "ignore ENOENT" case, and
`glnx_unlinkat()` otherwise. Port all in-tree callers to one or the other as
appropriate.
Just noticed an unprefixed error in the refs case and decided to do a tree-wide
check.
Closes: #1142
Approved by: jlebon
I'd mostly been skipping the GPG functions due to lack of autoptr for a few
things, but I noticed these bits were straightforward.
Closes: #1136
Approved by: jlebon
The vast majority of invocations of `ot_gpgme_error_to_gio_error()` were paired
with `g_prefix_error()`; let's combine them for the same reason we do
`glnx_throw_errno_prefix()`. For the few cases that don't we might as well add
some prefix.
I also changed it to `return FALSE` in prep for more style porting.
Closes: #1135
Approved by: jlebon
I was reading the pull-local command docs and realized it was somewhat unclear
that `--untrusted` *only* applied to local repo pulls; in other words that we
always treat non-local pulls as untrusted.
Tweak the docstring, and add tests that verify this explicitly.
Closes: #1130
Approved by: jlebon
However, they weren't showing up in the output HTML and I have
no idea why; I looked at what we're doing and it looks close enough
to what's going on in `GDBusConnection` that I was using as a reference.
I'm not going to spend a lot of time to debug it right now.
Closes: #1140
Approved by: jlebon
Followup from previous patch - we can now centralize the sysroot loading.
Besides the obvious cleanup value, this is also prep for dropping an
`ostree_sysroot_get_path()` user.
Closes: #1123
Approved by: jlebon
This is exactly analogous to the `ostree init` case where
we have `OSTREE_BUILTIN_FLAG_NO_REPO` to avoid trying to load
a repo when we're creating one.
Let's avoid the pointless sysroot for `init-fs`; among other
things this will then let us do `ostree_sysroot_load()` inside
the argument parsing, and drop it from every other user.
Closes: #1123
Approved by: jlebon
Rather than calling `ostree_sysroot_get_path()`, which I'd like to deprecate for
the same reason as `ostree_repo_get_path()`.
Closes: #1123
Approved by: jlebon
In almost all places. There are just a few exceptions; one tricky bit for
example is that the repo config must still have `mode=archive-z2`, since
`archive` used to mean something else. (We could very likely just get rid of
that check, but eh, later).
I also added a test that one can still do `ostree repo init --mode=archive-z2`.
Closes: #1125
Approved by: jlebon
This is for issue projectatomic/rpm-ostree#365,
an extra option of overwrite mode is added to the checkout command
so that when there is "non-directory" file already exist
during checkout, the error will be handled.
Some tests are added for regression
Closes: #1116
Approved by: cgwalters
The new --selinux-policy added in [0] exposed a subtle issue in the way
we handle labeling during commit. The CI system in rpm-ostree hit this
when trying to make use of it[1].
Basically, because of the way we use a GVariant to represent xattrs, if
a file to be committed already has an SELinux label, the xattr object
ends up with *two* label entries. This of course throws off fsck later
on, since the checksum will have gone over both entries, even though the
on-disk file will only have a single label (in which the second entry
wins).
I confirmed that the `fsck` added in the installed test fails without
the rest of this patch.
[0] https://github.com/ostreedev/ostree/pull/1114
[1] https://github.com/projectatomic/rpm-ostree/pull/953Closes: #1121
Approved by: cgwalters
For rpm-ostree, I want to move RPM files in `/boot` to `/usr/lib/ostree-boot`.
This is currently impossible without forking the libarchive code. Supporting
this is pretty straightforward; we already had pathname translation in
the libarchive code, we just need to expose it as an option.
On the command line side, I chose to wrap this as a regexp. That should be good
enough for a lot of use cases; sophisticated users should as always be making
use of the API. Note that this required some new `#ifdef LIBARCHIVE` bits to use
the new API. Following previous patterns here, we use the new API only if a
relevant option is enabled, ensuring unit test coverage of both paths.
For the test cases, I ended up changing the accounting to avoid having to
multiply the test count.
Closes: #1105
Approved by: jlebon
This was really straightforward to implement, and is useful
for dev/test scenarios mainly like we have in rpm-ostree at least.
Closes: https://github.com/ostreedev/ostree/issues/1113Closes: #1114
Approved by: jlebon
This is a follow-up to https://github.com/ostreedev/ostree/pull/1097.
We make simple_write_deployment smart enough so that it can be used for
rpm-ostree's purposes. This is mostly an upstreaming of logic that
already existed there.
Notably we correctly append NOT_DEFAULT deployments *after* the booted
deployment and we now support RETAIN_PENDING and RETAIN_ROLLBACK flags
to have more granularity on deployment pruning.
Expose these new flags on the CLI using new options (as well as expose
the previously existing NOT_DEFAULT flag as --not-as-default).
I couldn't add tests for --retain-pending because the merge deployment
is always the topmost one. Though I did check that it worked in a VM.
Closes: #1110
Approved by: cgwalters
When working with collections it can be useful to see remote refs rather
than just local and mirrored ones. This commit changes the "ostree refs
-c" output to include remote refs, and includes remote refs with
collection IDs in summary file generation as well. The former behavior
is consistent with how "ostree refs" works, and the latter behavior is
useful in facilitating P2P updates even when mirrors haven't been
configured.
To accomplish this, OstreeRepoListRefsExtFlags was extended with an
EXCLUDE_REMOTES flag. This was done rather than an INCLUDE_REMOTES flag
so that existing calls to ostree_repo_list_refs_ext continue to have the
same behavior. This flag was added to ostree_repo_list_collection_refs
(which is an experimental API break).
Also, add unit tests for the "refs -c" and summary file behavior, and
update relevant tests.
Closes: #1069
Approved by: cgwalters
Things like https://sourceware.org/libabigail/manual/abidiff.html
look interesting but in a brief look I couldn't work out
how to conveniently use them for quick ABI sanity checking without
doing a diff from a previous build (which we could do but would be
more involved).
This way will at least catch struct ABI breaks on x86_64 which
I think we'd be most likely to do accidentally when trying
to use one of the previous unused values.
I found the hole values via gdb's `pahole` command.
Closes: #1108
Approved by: jlebon
It's been this way for a long time, though not forever; I went
back to v2014.11 as a random choice and it worked then. Not
going to bother doing a full archive search for this though.
Anyone who wants more context can help themselves.
Closes: https://github.com/ostreedev/ostree/issues/1096Closes: #1106
Approved by: jlebon
This makes `ostree commit --tree=tar` honor `--owner-uid` and `--owner-gid`
for the root directory.
Prep for further commit filtering work, although mostly for the unit test cases;
this ensures we can use `ostree checkout` after autocreating a root directory.
Closes: #1104
Approved by: jlebon
The wrapping here is unnecessary, since `_ostree_repo_commit_modifier_apply()`
already does what this function did. Further, the return type was wrong.
Saw this while doing some libarchive work.
Closes: #1104
Approved by: jlebon
For both flatpak and ostree-as-host, we really want to verify up front during
pulls that we're not being downgraded. Currently both flatpak and
`OstreeSysrootUpgrader` do this before deployments, but at that point we've
already downloaded all the data, which is annoying.
Closes: https://github.com/ostreedev/ostree/issues/687Closes: #1055
Approved by: jlebon
When using the
OSTREE_SYSROOT_SIMPLE_WRITE_DEPLOYMENT_FLAGS_NOT_DEFAULT flag, the
deployment is said to be added after the booted or merge deployment.
Fix the condition to do so instead of adding it in the second place.
Closes: #1097
Approved by: cgwalters
Follow up to <https://github.com/ostreedev/ostree/pull/1079>; I was working on
the rpm-ostree updates for this, and I think it's more consistent if we have
`.img` here, since that's a closer match to the "remove $kver" that results in
`vmlinuz`. Also just best practice to have file suffix types where they make
sense.
The astute reader might notice this sneaks in a change where we'd crash if the
legacy bootdir didn't have an initramfs...yeah, should probably have test
coverage of that.
Closes: #1095
Approved by: jlebon
This is the new Fedora kernel standard layout; it has the advantage
of being in `/usr` like `/usr/lib/ostree-boot`, but it's not OSTree
specific.
Further, I think in practice forcing tree builders to compute the checksum is an
annoying stumbling block; since we already switched to e.g. computing checksums
always when doing pulls, the cost of doing another checksum for the
kernel/initramfs is tiny. The "bootcsum" becomes more of an internal
implementation detail.
Now, there is a transition; my current thought for this is that rpm-ostree will
change to default to injecting into both `/usr/lib/ostree-boot` and
`/usr/lib/modules`, and stop doing `/boot`, then maybe next year say we drop the
`/usr/lib/ostree-boot` by default.
A twist here is that the default Fedora kernel RPM layout (and what's in
rpm-ostree today) includes a kernel but *not* an initramfs in
`/usr/lib/modules`. If we looked only there, we'd just find the kernel. So we
need to look in both, and then special case this - pick the legacy layout if we
have `/usr/lib/modules` but not an initramfs.
While here, rework the code to have an `OstreeKernelLayout` struct which makes
dealing with all of the variables nicer.
Closes: #1079
Approved by: jlebon
I noticed this with a local build of an RPM:
```
/usr/include/glib-2.0/glib/glib-autocleanups.h:28:3: warning: 'help' may be used uninitialized in this function [-Wmaybe-uninitialized]
g_free (*pp);
^~~~~~~~~~~~
src/ostree/ot-main.c:82:20: note: 'help' was declared here
g_autofree char *help;
^~~~
```
Closes: #1091
Approved by: jlebon
This is simplest for now. Compare with similar logic from
`/usr/lib/tmpfiles.d/tmp.conf`:
```
R! /tmp/systemd-private-*
```
Closes: https://github.com/ostreedev/ostree/issues/393Closes: #1090
Approved by: jlebon
I actually don't quite know what the version inheritance really does, but let's
be safe and fix this. I'm being conservative here and fixing it to inherit from
2017.8, skipping .9 since that doesn't have a parent.
Related: https://github.com/ostreedev/ostree/issues/1087Closes: #1088
Approved by: jlebon
In the production case since we used `daemon()` our stderr is `/dev/null`¹
there's not much use in logging errors from `FITHAW` or `exit(1)`, and doing so
breaks the test suite which checks the return from `waitpid()`. There's nothing
we can really do if `FITHAW` fails, and in most of those cases `EINVAL`,
`EOPNOTSUPP`, we *shouldn't* do anything anyways.
¹ Though perhaps we should set up the systemd journal, but let's not
go there right now.
Closes: #1084
Approved by: jlebon
I added `waitpid()`, but that didn't actually help because we were
`daemon()`izing. Don't daemonize if we're testing so that we can `waitpid()`.
Note I still haven't reproduced this race locally, but I'm pretty sure this will
fix it.
While here, actually check the return value from `waitpid()` just in case
something goes wrong there.
Closes: #1084
Approved by: jlebon
Just a random cozy patch I made while perusing the codebase. When
determining if two OstreeDeployment objects are the same, rather than
just checking for NULL, we can just directly check for equality of
pointers to also catch the trivial case.
Closes: #1082
Approved by: cgwalters
This is mostly the `copy_file_range` changes plus the Coverity files.
```
Colin Walters (4):
localalloc: Abort on EBADF from close() by default
local-alloc: Remove almost all macros like glnx_free, glnx_unref_variant
console: Fix Coverity NULL deref warning
fdio: Merge systemd code to use copy_file_range(), use FICLONE
Jonathan Lebon (1):
console: trim useless check
Matthew Leeds (1):
dirfd: Fix typo in comment
Philip Withnall (1):
glnx-console: Add missing NULL check before writing out text
```
Update submodule: libglnx
Closes: #1081
Approved by: jlebon
/* ATTENTION:
* Please remember to update the bash-completion script (bash/ostree) and
* man page (man/ostree-$COMMANDNAME.xml) when changing the option list.
*/
Closes: #1080
Approved by: cgwalters
This essentially completes our fd-relative conversion.
While here, I cleaned up the semantics of `ostree_repo_create()` and
`ostree_repo_create_at()` to be more atomic - basically various scripts were
testing for the `objects` subdirectory, so let's formalize that.
Closes: #820
Approved by: jlebon
If one of the localcache repos has the exact same commit we resolved
from the remote, then we need to make sure to mark it as partial so that
we download the full tree.
Closes: #1074Closes: #1076
Approved by: cgwalters
Saw this in a PR result; we need to wait for the child to have written its
result to stderr before we exit, otherwise the test suite may not read it in
time.
Closes: #1070
Approved by: jlebon
This will allow us to drop the awful hack in rpm-ostree where we watch our own
stdout. In general, libraries shouldn't write to stdout.
Also we can kill the systemd journal wrapper code. There's some duplication at
each call site now...but it's easier than trying to write a `sd_journal_send()`
wrapper.
I was originally going to have this emit all of the structured data too as a
`GVariant` but decided it wasn't worth it right now.
Closes: #1052
Approved by: jlebon
Add a new error domain for GPG signing/verification errors, and use it
throughout libostree for describing verification errors. This replaces
various uses of G_IO_ERROR_FAILED, and one instance of
G_IO_ERROR_NOT_FOUND (for which some code in ot-builtin-show.c had to be
changed to ensure it was still handled correctly).
The use of a separate error domain allows failures in GPG operations to
be handled separately from network failures (where the summary file
could not be found to be downloaded, for example) or timeouts.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1064Closes: #1071
Approved by: mbarnes
I'd like to move the new canonical kernel directory to `/usr/lib/modules/$kver`,
as Fedora has done. The `get_kernel_from_tree()` function now abstracts over
parsing the data (src vs destination filenames, as well as checksum) in
preparation for adding the new case.
In preparation for this, let's change the current test suite to use the
*current* directory of `/usr/lib/ostree-boot`, and also add coverage of `/boot`.
Closes: #1053
Approved by: jlebon
When returning results from finding repos, set gpg-verify-summary=false
in their configs, since any pulls from such remotes will necessarily
involve collection IDs, and hence should be using the unsigned summary
support. In the intended deployment mode for P2P transmission of OSTree
refs, summaries *cannot* be signed, so setting gpg-verify-summary=true
would cause all the pulls to fail.
The unsigned summary support is the move of repository metadata from
the summary file (not spliceable) to the well-known ostree-metadata ref
(spliceable, as it can exist for multiple collection IDs in the same
repository).
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1066
Approved by: cgwalters
See: http://marc.info/?l=linux-fsdevel&m=149520244919284&w=2
XFS doesn't flush the journal on `syncfs()`. GRUB doesn't know how to follow the
XFS journal, so if the filesystem is in a dirty state (possible with xfs
`/boot`, extremely likely with `/`, if the journaled data includes content for
`/boot`, the system may be unbootable if a system crash occurs.
Fix this by doing a `FIFREEZE`+`FITHAW` cycle. Now, most people
probably would have replaced the `syncfs()` invocation with those two
ioctls. But this would have become (I believe) the *only* place in
libostree where we weren't safe against interruption. The failure
mode would be ugly; nothing else would be able to write to the filesystem
until manual intervention.
The real fix here I think is to land an atomic `FIFREEZETHAW` ioctl
in the kernel. I might try a patch.
In the meantime though, let's jump through some hoops and set up
a "watchdog" child process that acts as a fallback unfreezer.
Closes: https://github.com/ostreedev/ostree/issues/876Closes: #1049
Approved by: jlebon
The API for downloading a summary file can legitimately return NULL for
the summary file contents when it returns TRUE (success). This indicates
an error 404 — the summary file was not found.
Two call sites were not handling that correctly, which was causing later
assertion failures.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1061Closes: #1065
Approved by: cgwalters
(remaining > 0) is asserted by the loop condition, and remaining is not
modified between that check and the G_UNLIKELY — so the condition in the
G_UNLIKELY will always be true.
Spotted by Coverity as issue #1452617.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1059
Approved by: cgwalters
summary_timestamp is checked for non-NULL-ness above, and the function
bails if it’s NULL.
Fixes Coverity issue #1452616.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1059
Approved by: cgwalters
This parallels ostree_repo_remote_list_refs(), but returns a map of
OstreeCollectionRef → checksum, and includes refs from collection IDs
other than the remote repository’s main collection ID.
Use this in OstreeRepoFinderConfig to ensure that refs are matched
against even if they’re stored in the repository summary file’s
collection map, rather than its main ref map. This fixes false negatives
when searching for refs in some situations.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1058
Approved by: cgwalters
This catches a few failure modes in the pull code a little earlier,
before the incorrectly-NULL repo makes its way into a closure and a
worker thread, where the cause of the problem is harder to track down.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1058
Approved by: cgwalters
As the comment explains, it’s possible for a result to be freed while
ref_to_checksum is NULL, even though normally the data structure
guarantees it’s non-NULL. This was causing crashes when results were
filtered out of a find-remotes call. Guard against that.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1058
Approved by: cgwalters
The intended behaviour of ostree_repo_find_remotes() is to return
results which have the latest version of at least one of the requested
refs. Results which have some of the requested refs, but don’t have the
latest version of any of them, should be ignored. The logic to do this
was broken in the case that a result contained a positive number of the
requested refs, but none of them were the latest version. (It previously
worked when the result contained none of the requested refs.)
Fix the counting to work correctly in both cases.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #1058
Approved by: cgwalters
Coverity spotted an infloop here since we were incrementing `i++`
instead of `j++`. But adding a test revealed other bugs - we need
to keep the arrays in sync.
Coverity CID: 1452204
Closes: #1041
Approved by: pwithnall
If a delta happens to have zero objects, we could end up doing
a divide-by-zero when inferring endianness. In practice,
a zero-object delta isn't possible to generate I think, but
let's make sure the code is defensive all the same.
Spotted by Coverity.
Coverity CID: 1452208
Closes: #1041
Approved by: pwithnall
This commit sets prgname correctly so that the "ostree subcommand
--help" output prints the subcommand rather than just "ostree".
This was removed in commit f0519e541f because it tripped the thread
sanitizer, but it's being added back conditionally so most users who
don't compile with -fsanitize=adress see proper help output.
Closes: #1054
Approved by: cgwalters
Part of cleaning up our usage of libglnx; we want to use what's in GLib where we
can.
Had to change a few .c files to `#include ostree.h` early on to pick up
autoptrs for the core types.
Closes: #1040
Approved by: jlebon
There are multiple use cases where we'd like to alias refs.
First, having a "stable" alias which gets swapped across major
versions: https://pagure.io/atomic-wg/issue/228
Another case is when a ref is obsoleted;
<https://pagure.io/atomic-wg/issue/303>
This second one could be done with endoflife rebase, but I think
this case is better on the server side, as we might later change
our minds and do actual releases there.
I initially just added some test cases for symlinks in the `refs/heads` dir to
ensure this actually works (and it did), but I think it's worth having APIs.
Closes: #1033
Approved by: jlebon
I plan to at some point change rpm-ostree to read the journal messages from
libostree and render things like the time we spent in syncfs().
Closes: #1044
Approved by: jlebon
Define typedefs for read/write archives, and use the GLib
autocleanups for them. Prep for updating libglnx to drop its
custom autocleanup macros.
Closes: #1042
Approved by: jlebon
Prep for dropping `GLNX_DEFINE_CLEANUP_FUNCTION` from libglnx
in favor of using GLib's `G_DEFINE_AUTO_CLEANUP_FREE_FUNC()`.
Closes: #1042
Approved by: jlebon
It looks like `curl_multi_socket_action()` will return an error
if *one* of the requests has an error, but we already check
for that explicitly by iterating over each handle.
In libcurl, the "easy" layer doesn't really make use of this
return value. I did a bit of looking elsewhere; systemd
does check it as a runtime error, not an assertion. librepo
doesn't use the multi interface.
Closes: https://github.com/ostreedev/ostree/issues/1035Closes: #1038
Approved by: jlebon
Coverity complained that the `else if (bytes_read == 0)` was dead
code if we happened to find it was already false when testing
`else if (G_UNLIKELY (bytes_read == 0 ...`.
There was nothing wrong with the logic, but let's rework it to
only test the value once; I think it does end up nicer anyways.
Coverity CID: 1452186
Closes: #1037
Approved by: jlebon
No real problems here, but Coverity likes to see consistent checking of return
values, and I agree with it.
Coverity CID: 1452213
Coverity CID: 1452211
Closes: #1037
Approved by: jlebon
The fingerprint associated with each signature can be different to
the primary key ID (the normal one that people use to identify a
GPG key) if the signature is from a signing subkey. Try to find the
primary key and print this ID in preference to the subkey signature.
https://github.com/ostreedev/ostree/issues/608Closes: #1036
Approved by: cgwalters
Use gpgme_get_key to find the primary key for the key we are
looking for, and the primary key for each signature, and
compare these when looking up signatures.
The primary key is the first in the list of subkeys, which is
the normal key ID people use when referring to a GPG key as an
identity.
If the key has a signing subkey, signature->fpr will not match
the provided key_id, so looking up both keys and comparing the
primary key fingerprints ensures they are both canonicalised.
https://github.com/ostreedev/ostree/issues/608Closes: #1036
Approved by: cgwalters
This is a continuation of addition of journaling to libostree; see
e.g. <https://github.com/ostreedev/ostree/pull/708>.
I wanted more information at the end of fetches; in particular
some details about the delta execution (what opcodes etc.), but
this is a first step: we log things like the transferred data
as well as whether or not GPG was enabled, etc.
One awkward thing about this is how we map the fetcher options like
`tls-ca-path` back out into an enum for the code to log. But eh, hard to fix
without a bigger refactoring.
Closes: #1032
Approved by: jlebon
These were previously private, but since we expect people to use them, let's add
`#define`s like we did for some of the other commit metadata.
Closes: #1028
Approved by: jlebon
Mostly for the latest `-Wmaybe-uninitialized` fix, but while here also port some
places to newer APIs.
Update submodule: libglnx
Closes: #1027
Approved by: jlebon
Regression from previous tmpfile refactoring; unfortunately
the `OSTREE_REPO_COMMIT_MODIFIER_FLAGS_GENERATE_SIZES` option
only has coverage via gjs currently.
Might expose it via the cmdline in a later option, but in the big picture the
idea was that this data is better kept in static deltas.
Closes: https://github.com/ostreedev/ostree/issues/1014Closes: #1016
Approved by: jlebon
See: https://github.com/projectatomic/rpm-ostree/issues/885
If we get a successful Apache directory listing HTML when fetching what we
intend to be a ref, we'd dump the HTML into the error.
I did some scanning of the pull code, and this was the only case
I saw offhand where we were dumping text out into an error. Which
makes sense, since most of our formats are binary, the exeptions I
think are just `repo/config` and `repo/refs/`.
Closes: #1015
Approved by: mbarnes
Using the error prefixing in the delta processing allows us to
do new code style. Also strip trailing whitespace.
Use error prefixing in a few other random places. I didn't
hunt for all of them, just testing out the new API.
Use `glnx_fchmod()`. Also note I dropped one `fchmod (tmpf, 0600)`
which is no longer necessary.
Update submodule: libglnx
Closes: #1011
Approved by: jlebon
Prep for `ostree_repo_new_at()`. Down the line perhaps
we should extend libcurl to accept a file descriptor for cookies,
but this works OK for now.
Closes: #1010
Approved by: jlebon
Prep for `ostree_repo_new_at()`. These commands were directly accessing
`repo->repodir`, which it turns out was unnecessary since the the APIs they then
used were fd-relative. Except actually there were bugs there, so fix all of the
cookie util code to actually use the passed `dfd` and not just hardcode
`AT_FDCWD`.
Also, libsoup can't handle this (its APIs require fully qualifed paths), and
there's not a really good reason to have two implementations now; historically
it was useful to cross-check them, but I don't think we need that.
While I'm here, port to new style.
Closes: #1010
Approved by: jlebon
(Note this PR was reverted in <https://github.com/ostreedev/ostree/pull/902>;
this version should be better)
Using `${sysroot}` to mean the physical storage root: We don't want to write to
`${sysroot}/etc/ostree/remotes.d`, since nothing will read it, and really
`${sysroot}` should just have `/ostree` (ideally). Today the Anaconda rpmostree
code ends up writing there. Fix this by adding a notion of "physical" sysroot.
We determine whether the path is physical by checking for `/sysroot`, which
exists in deployment roots (and there shouldn't be a `${sysroot}/sysroot`).
In order to unit test this, I added a `--sysroot` argument to `remote add`.
However, doing this better would require reworking the command line parsing for
the `remote` argument to support specifying `--repo` or `--sysroot`, and I
didn't quite want to do that yet in this patch.
This second iteration of this patch fixes the bug we hit the first time;
embarassingly enough I broke `ostree remote list` finding system remotes.
The fix is to have `ostree_repo_open()` figure out whether it's the same
as `/ostree/repo` for now.
Down the line...we might consider having the `ostree remote` command line itself
instatiate an `OstreeSysroot` by default, but this maximizes compatibility; we
just have to pay a small cost that `ostree` usage outside of that case like
`ostree static-delta` in a releng Jenkins job or whatever will do this `stat()`
too.
Closes: https://github.com/ostreedev/ostree/issues/892Closes: #1008
Approved by: mbarnes
This came up in <https://github.com/ostreedev/ostree/pull/982>; when
we added more direct local importing, we did it synchronously.
This was actually quite a regression when doing local pulls between different
modes; in particular between a bare mode and `archive`, as we were suddenly
doing gzip {de,}compression in the main thread.
Down the line actually...a simpler fix is probably to change things so that the
local path is really only used when we know we can hardlink; everything else
would go though the fetcher codepath but with `file://`.
But this isn't a lot more code, and the speed/interactivity win is large.
Note we're only doing content async with this patch. We could do metadata as
well; we have the object already local. But the metadata code path is messier,
and metadata objects are smaller.
Another area where this comes up is that in e.g. Fedora releng, most operations
talk to a NetApp via NFS. So this has the classic network filesystem problem
that operations that are normally cheap like `stat()` can actually have
nontrivial latency. Doing as much as possible in threads is better there too.
Closes: #1006
Approved by: jlebon
There is no actual written guarantee in glib-mkenums that the template
line specified using --fhead will be added after the templates specified
inside the template file. Since the template file is only used once, we
can simply move the `#include` directive inside the template, so that it
is guaranteed to be in the right place.
Closes: #1007
Approved by: cgwalters
Currently in Fedora we don't sign summaries, and every use of
`rpm-ostree` would emit to the journal an error when we failed
to fetch it.
Fix this by having `OSTREE_FETCHER_REQUEST_OPTIONAL_CONTENT` tell the fetcher
not to journal 404 errors. While fixing this, we had a mix of two booleans vs
the flags; fix things so we consistently use the flags in the fetcher and pull
code.
Closes: #1004
Approved by: mbarnes
As discussed in https://github.com/ostreedev/ostree/pull/946, the
summary file is becoming an unsigned cache of ref information; any
additional metadata for the repository needs to move elsewhere in order
to remain signed. Introduce OSTREE_REPO_METADATA_REF as the well-known
name of a ref where such metadata can live, as the metadata on
contentless commits.
Don’t yet update the documentation for summary-related methods to
mention this, since it’s still hidden behind the
--enable-experimental-api configure option.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #946
Approved by: cgwalters
Use goffset rather than gsize for file sizes. More importantly, get the
unpacked_size from g_file_info_get_size() (goffset) rather than from the
splice return value, which has type gssize.
This will make a difference on 32-bit systems, where goffset is defined
as off64_t, but gsize is 32 bits.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #999
Approved by: cgwalters
When we [switched to using checkout + force_copy](e8efd1c8dc),
a side effect that went unnoticed at the time is that we started
setting directory mtimes to zero.
See the below bug where we long ago set the file times to zero, which got fixed,
so let's not regress things by setting the directory times to zero either. (Even
though AFAICS GNU tar doesn't complain about those)
This semantic is somewhat "overloaded" onto `force_copy`, but it avoids adding
yet another boolean; we don't have that many reserved boolean slots left. I
can't really think of many good use cases for `force_copy` *other* than the
`/etc` merge anyways.
https://bugzilla.redhat.com/show_bug.cgi?id=1229160
Closes: https://github.com/ostreedev/ostree/issues/995Closes: #997
Approved by: jlebon
There are a number of simple ports here. Prep for further work
in `/etc` merge.
I also stripped trailing whitespace globally.
Closes: #996
Approved by: jlebon
This verifies the collection and ref bindings in the commit metadata
against the collection ID we have stored in the remote config and ref
we want to pull from. For the HEAD commits, we also check if the
checksum of the commit we just fetched agrees with the checksum we
really wanted to pull from the ref.
For commits with explicitly specified checksums and without specified
refs, we only verify if the commit has the bindings. We are able to
only verify the collection binding, though.
Closes: #972
Approved by: cgwalters
The collection and ref bindings are stored in the commit metadata
under ostree.collection-binding and ostree.ref-binding,
respectively. They will be used to verify if the commit really comes
from the collection and ref we wanted to pull from.
Closes: #972
Approved by: cgwalters
And in general, if for some reason we can't write `user.` xattrs, provide an
error immediately rather than doing it during a later pull. This way the failure
cause is a lot more obvious.
Related: https://github.com/ostreedev/ostree/issues/991Closes: #993
Approved by: jlebon
In the storage PR I was trying to do a `pull-local` of the whole
`/ostree/repo` on the system, which ended up triggering a `g_critical()`
in the collections code, since we tried to parse a remote-prefixed ref
`fedora:fedora/26/x86_64/atomic-host` as a ref.
I'm not sure offhand what our behavior in this case *should* be. I
think git only clones local refs, but I need to check.
This corner case arises only with `pull-local`. But in any case,
while we were previously saying this is programmer error, since it's
so easy to pass various unchecked input into the pull machinery,
make invalid refs an explicit error.
Closes: #992
Approved by: jlebon
For ostree-as-host, we're the superuser, so we'll blow past
any reserved free space by default. While deltas have size
metadata, if one happens to do a loose fetch, we can fill
up the disk.
Another case is flatpak: the system helper has similar concerns
here as ostree-as-host, and for `flatpak --user`, we also
want to be nice and avoid filling up the user's quota.
Closes: https://github.com/ostreedev/ostree/issues/962Closes: #987
Approved by: jlebon
This is prep for storage space checks, where we look at free
space after parsing the metadata, before we write anything.
We did length-limited writes in the fd-based input path, but not for the
`GInputStream` path which in practice is used for HTTP pulls.
Closes: #987
Approved by: jlebon
Some of the Jenkins jobs for Fedora Atomic Host broke after updating
to 2017.7, and it turns out that we regressed handling unreadable
files in `bare-user` mode. An example of this is `/etc/shadow`, which
ends up in the ostree-as-host content as `/usr/etc/shadow`.
Now there are better fixes here; we should probably delete it and create it
during the config merge if it doesn't exist. In general, having secret files in
ostree really isn't supported, so it doesn't make sense to include them.
But let's fix this regression - when operating as an unprivileged user we don't
have `CAP_DAC_OVERRIDE` and hence will fail to open un-user-readable objects.
(We still preserve the actual `0` mode of course in the xattr and will
apply it in `bare`)
Closes: #989
Approved by: jlebon
Previously, we only supported additions in the statoverride file;
it was mainly for adding the setuid bit without having that physically
on disk.
However, for testing a change to `bare-user` handling around *unreadable*
files (which happens for `/etc/shadow` in host content), I need a way
to write that into a repo in the test suite.
I'm not actually aware of a non-test-suite use case for this; a more
sophisticated user is going to be using the API directly, which can already do
this. But we need it for tests at least.
Closes: #989
Approved by: jlebon
I had thought `glnx_link_tmpfile_at()` actually consumed the tmpfile;
it does consume the *path* but not the fd. In the non-delta path
things were fine since we used the autocleanup.
But the delta code had a tmpfile allocated in its struct that got reused, and
hence leaked the fd. Fix this by making the commit API actually consume the
tmpfile fully, just like the path path.
Closes: #986
Approved by: jlebon
This is a lot like `git clone --reference`, but we chose "localcache" as the
term "reference" is already used.
The main use case I'm targeting this for is the Fedora Atomic Host installer
case where we embed the repo content in the installer, but we may want to
kickstart and download newer content. There, while we want to get a newer ref,
we can still use the local repo as an object cache, since we have it sitting
there in memory anyways.
Another case is where one has a host ostree (say e.g. Fedora Atomic
Workstation), and one wants to create a local archive mirror of FAH. Then one
can use `pull --reference /ostree/repo` and pull the common objects (e.g.
contents of `bash.rpm` etc.)
Closes: https://github.com/ostreedev/ostree/issues/975Closes: #982
Approved by: jlebon
These are regression from #971. We were stuffing a pointer size inside a
variable of integer size. So the assignment was spilling over into other
variables' storage space. Actually use a gpointer and GPOINTER_TO_[U]INT
as was done originally.
Also bump libglnx which has static checks for this error in the future.
Update submodule: libglnx
Closes: #990
Approved by: cgwalters
An inverted condition in _ostree_repo_add_remote() was causing the
OstreeRepoFinder to delete precisely the wrong remote
configurations from memory once it was finished. It’s supposed to delete
the ones which it transiently added; but was instead deleting all the
existing remote configurations.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #985
Approved by: cgwalters
It's more natural for a few calling places. Prep for patches to go the other
way, which in turn are prep for adding a commit filter v2 that takes `struct
stat`.
`ot_gfile_type_for_mode()` was only used in this function, so inline it here.
Closes: #974
Approved by: jlebon
Previously, `ostree pull` was silent if not on a tty. I don't
see a reason not to print the final status line at least. This
is prep for more work in the test suite, so I can write assertions
on the output.
But it should also be nicer for people who e.g. do an `ostree pull` in a Jenkins
job or whatever.
Closes: #981
Approved by: jlebon
The `-z2` is annoying now since it's really a legacy; we've long
since supported typing `archive`. Convert the docs fully and
explain that.
Also do some (but not all) of the tests just to encourage newer tests to use
`archive` too.
Closes: #980
Approved by: jlebon
Obviously very minor, but I noticed this while working on `pull --reference`. If
we have a local repo, we'll have already done a hardlink and copied the detached
metadata too, so there's no reason to request it again via the fetcher path.
Closes: #978
Approved by: jlebon
I'd mostly been avoiding this file since there's always patches outstanding, but
these few functions shouldn't conflict much.
Closes: #979
Approved by: jlebon
I added `glnx_open_anonymous_tmpfile()`, but then later noticed
that the usage of this was really to be combined with `mmap()`,
and we had two versions of that in the delta code. Add a helper.
(Bigger picture...how is this different from glibc's "mmap() of /dev/zero"
approach for large chunks? One advantage is the storage can be "swapped" to
`/var/tmp`, but still deleted automatically, rather than requiring swap space)
Closes: #973
Approved by: jlebon
Use the new macros introduced recently in libglnx to make iterating over
hash tables cleaner. This is just a start, it does not migrate the whole
tree.
Update submodule: libglnx
Closes: #971
Approved by: cgwalters
There's lots of mechanically replacing `OtTmpFile` with `GLnxTmpfile`;
the biggest changes are in the commit path. Symlink commits are now
very clearly separated from regular files. Symlinks are `OtCleanupUnlinkat`,
and regular files are `GLnxTmpfile`.
The commit codepath separates those as `_ostree_repo_commit_path_final()` and
`_ostree_repo_commit_tmpf_final()`. A nice aspect of all of this is that they
both *consume* the temporary on success. This avoids an extra spurious
`unlink()` call.
One of the biggest bits of code motion is in `commit_loose_regfile_object()`,
which no longer needs to care about symlinks. For the most parth though it's
just removing conditionals.
Update submodule: libglnx
Closes: #958
Approved by: jlebon
Rather than duplicating the code. This introduces no functional changes.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #961
Approved by: cgwalters
Check that it’s in normal form and has the correct type when loading it,
since it could come from an untrusted source.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #961
Approved by: cgwalters
This makes no difference to the validity of the code, since any summary
file loaded from the cache will be verified before being read anyway;
but it will make some upcoming changes a little simpler.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #961
Approved by: cgwalters
When updating a summary file, parse additional arguments to the `ostree
summary` command as additional metadata to be put into the summary.
Add some tests for this.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #961
Approved by: cgwalters
Computing download/storage size for `archive` pulls is hard; there's
`OSTREE_REPO_COMMIT_MODIFIER_FLAGS_GENERATE_SIZES` which was from a
pre-static-deltas effort by Endless, but we aren't currently making use of this
much.
Static deltas were designed to solve this problem; we have the total
uncompressed size. Let's check free space before doing a delta pull.
Related: https://github.com/ostreedev/ostree/issues/962Closes: #963
Approved by: jlebon
The `OstreeRepoContentBareCommit` struct was basically an `OtTmpFile`, so let's
make it one. I moved the "convert to `GOutputStream`" logic into the callers,
since that bit can't fail; it makes the implementation much simpler since we can
just return the result of `ot_open_tmpfile_linkable_at()`.
Prep for `GLnxTmpfile` porting.
Closes: #957
Approved by: jlebon
The pull code also could make use of this in both the metadata and content
paths. I changed it to own the tempfile malloc (just like `GLnxTmpFile`), since
there's no reason to have different lifetimes for the filename and the file, and
that way we only have one variable rather than two.
The content path turns out to be a special case though, where
at least for mirroring archives, we directly pass the file *path*
down into `_ostree_repo_commit_loose_final()`.
This is prep for `GLnxTmpFile` porting.
Closes: #957
Approved by: jlebon
The variables here were duplicative; we don't need two booleans to distinguish
between symlinks and regular files. What we do need to handle is the "physical"
state versus the "object" state. Symlinks objects are stored as regular files in
`bare-user` and `archive`.
Prep for more cleanup.
Closes: #957
Approved by: jlebon
Instead of treating it as a programming error — given that it’s user
input, that’s not really appropriate. This modifies write_ref() and
list_collection_refs() to implement validation.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
These utilities were not needed for the initial port to support
OstreeCollectionRef, so have been delayed a bit and, in some cases, left
as FIXME comments for follow up later.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
If a repository’s summary file includes a collection ID, output that. If
it includes refs from other collections (in the ‘collection map’), output
those and include the same metadata detail as for refs in the summary
file’s main refs map.
If collection IDs are specified in the summary file, this changes the
output format from `ostree summary -v` to use (collection ID, ref name)
tuples.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
Rather than change the output format used by the existing refs command
to output collection IDs in addition to ref names, this functionality has
been hidden behind an --collections argument. If it’s not specified `ostree
refs` will output the same content as before for a given repository. If
it is specified, the collection ID for each ref will be included in the
output as (collection ID, ref name).
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
This allows newly configured remotes to have their collection ID specified,
so that refs from them can be downloaded from peers as well as the
upstream collection, using the remote’s configuration.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
This allows new repositories to be configured with a collection ID which
can be used to uniquely identify refs which originated from this
repository.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
This will pull the remotes after finding them. This potentially needs to
go in its own pull-from-remotes built-in command, but it will be fine
here for now.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
This is a wrapper around the new ostree_repo_find_remotes() method; it
tries to find available remotes which can serve updates for the
user-provided refs.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
This is a more complex implementation of OstreeRepoFinder which resolves
ref names to remote URIs by looking for refs advertised by peers on the
local network using DNS-SD records and mDNS (Avahi). The idea is to
allow OS and app updates to be propagated over local networks, without
the internet.
It requires an OSTree server and code to generate the DNS-SD adverts in
order to be fully functional — support for this will be added
separately.
Unit tests are included.
Includes fixes by Krzesimir Nowak <krzesimir@kinvolk.io>.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
This will be used in an upcoming commit. It adds a basic bloom filter
implementation, using the SipHash family of hash functions.
The implementation (including its parameter choices and hash functions)
will become a protocol detail in future, so must not be changed so that
its output is bitwise incompatible between OSTree versions.
Unit tests are included.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
This is a basic implementation of OstreeRepoFinder which resolves ref
names to remote URIs by looking for them on any currently mounted
removable storage volumes. The idea is to support OS and app updates via
USB stick.
Unit tests are included.
This bumps libostree’s maximum GLib dependency from 2.44 to 2.50 for
g_drive_is_removable(). If GLib 2.50 is not available, the call which
needs it will be omitted and the OstreeRepoFinderMount implementation
will scan all volumes (not just removable ones); this is a performance
hit, but not a functionality hit.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
This is a basic implementation of OstreeRepoFinder which resolves ref
names to remote URIs by looking their collection IDs up in the local
configuration of remotes who have their collection-id key set.
Unit tests are included.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
Add an initial OstreeRepoFinder interface (but no implementations),
which will find remote URIs by ref names and collection IDs, the
combination of which is globally unique.
The new API is used in a new ostree_repo_find_updates() function, which
resolves a list of ref names to update into a set of remote URIs to pull
them from, which can be treated as mirrors. It is an attempt to
generalise resolution of the URIs to pull from, and to generalise
determination of the order and parallelisation which they should be
downloaded from in.
Includes fixes by Krzesimir Nowak <krzesimir@kinvolk.io>.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
This adds a new collection-refs option of type a(sss), giving a list of
(collection ID, ref name, checksum) tuples to pull from the given remote.
This option is intended to supersede the refs and override-commit-ids
options, so is mutually exclusive with them.
This includes support for resolving the refs from the remote’s summary
file, or from its refs/heads and refs/mirrors directories.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
These are tuples of (collection ID, ref name) which are a globally-unique
form of local ref. They use OstreeCollectionRef as an identifier, and hence
need to be accessed using new API, as the existing API uses string
identifiers and sometimes accepts refspecs. Remote names are not
supported as part an OstreeCollectionRef.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
Add {get,set}_collection_id() methods to OstreeRepo and some documentation
about the concept of a collection ID which globally identifies an
upstream repository. See the documentation for more details.
This will be used in future commits. For now, the new API is marked as
experimental (--enable-experimental-api).
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
This is a type representing the tuple (collection ID, ref name), which is
guaranteed to be globally unique. It will be used in upcoming commits.
It introduces the concept of a ‘collection’ which is a unique, curated
set of refs which lie in the same trust domain (i.e. all signed by the
same key and validated by the same developer). Flathub might be a
collection, for example; or the set of OS refs coming from a particular
OS vendor.
It includes a function for validating collection IDs.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
This will make some future additions to regenerate_summary() easier.
This commit introduces no functional changes.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #924
Approved by: cgwalters
This is followon work from previous cleanups. Basically
`stat_bare_content_object()` was the `fstatat()` logic
and `ostree_repo_read_bare_fd()` was the `openat()` implementation;
they duplicated some bits to find the object in staging, recurse
into parent etc.
Further, I wanted an internal-only version of this API which didn't allocate
`GFileInfo`/`GInputStream` but used a plain `fd` and `struct stat` to avoid
mallocs.
The end version here I think looks a lot nicer, since we deduplicate the various
`open()` calls in the different cases for example.
Closes: #952
Approved by: jlebon
Prep for future cleanup patches (in particular I want an internal-only
version at first that uses a fd+`struct stat`) to avoid allocations.
The new version avoids lots of deep nesting of conditionals as well
by hoisting the "not found" handling to an early return.
There's a bit of code duplication between the two cases but it's
quite worth the result.
Closes: #951
Approved by: jlebon
Prep for a change in `ostree_repo_load_file()`. We would crash if a
caller had `out_file_info = NULL`, because we deref `ret_file_info`
below it.
Closes: #951
Approved by: jlebon
There was a typo in it when it was first introduced. Let’s not live in
the past.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #950
Approved by: cgwalters
There was a typo in the group name. It should be OK to change the
version since this is all hidden behind the --enable-experimental-api
configure option.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #950
Approved by: cgwalters
There are a few places in the code where ad-hoc validation was being
performed. Might as well formalise it a bit more.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #948
Approved by: cgwalters
And use it in `test-symbols.sh`, to fix the `distcheck` case;
the previous change stopped distributing `libostree-devel.sym`
in release builds.
Closes: #944
Approved by: jlebon
Change the annotation of the out parameters on ostree_repo_load_file
from `(allow-none)` to `(optional) (nullable)`. `allow-none` is
ambiguous, since these parameters can be both NULL on input and set to
NULL on return.
Closes: #939
Approved by: cgwalters
Flatpak make check is failing when applying a static delta
to a bare-user-only repo due to an assert. The fix is to add
bare-user-only to the assert check.
Closes: #940
Approved by: giuseppe
We're hitting this in flathub, where we have a bunch of local builds,
but we also mirror a few refs from the gnome runtime repo into it.
Its fixable by re-doing the summary, but for a short time the
wrong version is visible.
Fixes https://github.com/ostreedev/ostree/issues/846Closes: #935
Approved by: cgwalters
The code here tried to truncate the string to the previous length,
but that doesn't work when recursing, since further calls change the
length.
What actually ended up happening was the string would get corrupted
after the first level of recursion.
Closes: #936
Approved by: jlebon
So far a lot of submitted PR have added symbols into the first
section. Split the file into `-released` and `-devel` to make
this more obvious.
To further enforce things, we hardcode a checksum of the `-released`
file in `test-symbols.sh`. Only release commits should update that
checksum.
Did you notice I like checksums?
Closes: #931
Approved by: pwithnall
This is especially interesting for the versioning symbols themselves,
as it is an indicator of when applications using introspection information
can start to use a symbol in the library to check if they have a
recent enough version of OSTree to use.
Closes: #932
Approved by: cgwalters
For the flatpak PR: https://github.com/flatpak/flatpak/pull/849
It's really more convenient if this works for HTTP pulls as well, since flatpak
does various types of pulling, and we can just set the flag everywhere.
Further, we might as well reject the content as early as possible.
Closes: #930
Approved by: alexlarsson
We saw this fail in a CI run. We've been trying to add strategic error prefixing
as a general rule, and this specific instance may help debug.
Closes: #929
Approved by: pwithnall
This imports a function that is used in rpm-ostree, and it's also intended for
use by https://github.com/advancedtelematic/aktualizr to display
what deployment we're going to boot next after the reboot.
Updated-by: Colin Walters <walters@verbum.org>
Closes: #897
Approved by: OYTIS
This is a continuation of https://github.com/ostreedev/ostree/pull/926
for directories instead of files.
See: https://github.com/flatpak/flatpak/issues/845
This option suppresses mode bits outside of `0775` for directory
checkouts. I think most people should start doing this by default,
and use explicit overrides for e.g. `/tmp` if doing a recommit based
on a checkout.
Closes: #927
Approved by: alexlarsson
This is an option which is intended mostly for flatpak;
see: https://github.com/flatpak/flatpak/issues/845
We're adding an option for pulling into *all*
repo modes that has an effect similar to the `bare-user-only`
change from https://github.com/ostreedev/ostree/pull/909
This way one can pull content into e.g. a root-owned `bare` repository and
ensure that there aren't any setuid or world-writable files.
Closes: #926
Approved by: alexlarsson
Thinking about the problem of flatpak converting from `bare-user` to `bare-user-only`
"in place" by creating a new repo and doing a `pull-local`, I realized
that we can optimize this process by doing hardlinks for both metadata
and regular files. The repo formats are *almost* compatible, the
exception being symlinks.
An earlier patch caused us to do hardlinks for metadata, this patch takes things
to the next step and special cases this specific conversion. In this case we
need to parse the source object to determine whether or not it's a symlink.
Closes: #922
Approved by: alexlarsson
Our previous logic for import-via-hardlink only tried if the repo modes match,
but we *can* hardlink metadata between e.g. `archive` and `bare-user` repos, and
that's quite useful thing to do. Our documentation encourages converting to/from
those repo modes locally for build systems.
Closes: #922
Approved by: alexlarsson
Before this, if one had repos of matching mode but different owners,
which could happen if one e.g. makes a `bare` non-root repo in
`/ostree/deploy/$stateroot/var/tmp`, every time we tried to call `linkat()`
we'd get `EPERM` and fall back to a copy.
Fix this by saving the repo owner uid, and avoid trying to call `linkat()` if we
know it's going to fail. Of course most commonly in this scenario we'll
immediately fail trying to `chown` the files to `0`, but this is prep for a
future patch to improve `bare-user` → `bare-user-only` imports where we'll be a
bit more sophisticated.
Closes: #922
Approved by: alexlarsson
This code looks like it was supposed to build a refspec, but it used a
slash as a separator rather than a colon. The following code does
recover by supporting prefix matching with slashes, but it seems like
this was perhaps not the intention.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #912
Approved by: cgwalters
Its often the case that we want to look at objects inside a commit,
before the objects the transaction is finished. For instance:
https://github.com/flatpak/flatpak/pull/837
Which tries to verify the file permissions before committing the
transaction.
And:
1e5ffa926a
Which collects the storage size of the objects so that we can
put the total download size in the commit metadata.
I tried to find all the places where we did reads from the
object directories, and in particular this fixes:
- `ostree_repo_load_file()` for `bare` repos (`archive` was already working).
- `ostree_repo_query_object_storage_size()`
- Applying deltas that reference not-yet-commited objects
Closes: #916
Approved by: cgwalters
This came up in: https://github.com/ostreedev/ostree/pull/881
Basically doing streaming for metadata is dumb. Split up the metadata/content
paths so we pass metadata around as `GVariant`. This drops the last internal
caller of `ostree_repo_write_metadata_stream_trusted()` which was the dumb
function mentioned.
Closes: #923
Approved by: jlebon
See https://github.com/ostreedev/ostree/pull/909 for more information on the
rationale. Basically there's no reason for flatpak (which uses `bare-user-only`)
to have world-writable dirs. Particularly with the presence of the system
helper.
An approach I considered instead was to parse and validate directory metadata
objects at commit time. We still may do that in addition; for file objects we *had*
to do it that way because the actual files would be laid down suid. But directories
live only as inert `.dirmeta` objects until we do a checkout (i.e. `mkdir()`), so
we can solve the problem at checkout time.
Closes: #914
Approved by: alexlarsson
Both callers of `commit_loose_object_trusted()` were passing
`OSTREE_OBJECT_TYPE_FILE`, so drop that parameter. This in turn
allows us to drop lots of checking of that inside the function.
Add a doc comment, and rename to `commit_loose_content_object()` for clarity.
Closes: #914
Approved by: alexlarsson
I noticed my previous patches incorrectly started doing `return glnx_throw*`
inside a `goto out;` function. Fix this by porting forward consistently to new
style. We just do the error prefixing in the caller.
Closes: #914
Approved by: alexlarsson
When a transaction is finished and we have moved all the staged loose
objects into the repo we fsync all the object directory, to ensure the
filenames are stable before we update the refs files to point to the
new commits.
With out this an unclean shutdown after the transaction is finished
could result in a refs file that points to an incomplete commit.
https://bugzilla.gnome.org/show_bug.cgi?id=759442Closes: #918
Approved by: cgwalters
These exist in the wild for flatpak, and aren't really a problem. The canonical
permissions are still either `0755` or `0644`, we just support the additional
writable bit for the group (i.e. extend the set to include `0775` and `0664`)
now to avoid breaking some flatpak content.
Closes: #913
Approved by: alexlarsson
Having every object in a bare-user repo (and checkouts) be executable
is ugly. I can't think of a good reason to do that; they should only
be executable if their input is. This does
for `bare-user` what we did for `bare-user-only` in
https://github.com/ostreedev/ostree/pull/909
It's also a stronger version of what we do with `checkout -U` in suppressing
suid - here we also strip world-writable files and the sticky bit (even though
that's meaningless today, it might not be in the future).
Closes: https://github.com/ostreedev/ostree/issues/907Closes: #908
Approved by: alexlarsson
This is only used internally (the header is not public), so it doesn’t
have to go in ostree-autocleanups.h. It will be used in some following
commits.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #911
Approved by: cgwalters
If there are no deltas to be listed in the summary file, don’t bother
including the key for them in the additional metadata section of the
file. This saves a few bytes in some cases.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #911
Approved by: cgwalters
This follows on from commit a946c3d4, which added formatting for
well-known summary metadata keys. This commit adds it for commits.
Currently, the only well-known commit metadata key is
ostree.commit.timestamp. Formatting this correctly is especially
important, since it’s a big-endian uint64, which is completely unusable
for mere mortals when presented as a number rather than a date.
Non-formatted output can still be retrieved using the OSTREE_DUMP_RAW
flag, and the non-formatted key name is always printed for clarity.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #911
Approved by: cgwalters
It’s a bit neater to initialise the loop iterator and maximum in the
same place.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #911
Approved by: cgwalters
This makes it a bit more easily separable from the rest of the code in
the function. No functional changes.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #911
Approved by: cgwalters
For the flatpak use case where bare-user-only was introduced, we actually
don't want to support s{u,g} id files in particular.
Actually, I can't think of a reason to have anything outside of the
`0755 i.e. (u=rwx,g=rx,o=rx)` mask, so that's what we do here.
This will have the effect of treating existing `bare-user-only` repositories as
corrupted if they have files outside of that mask, but I think we should do this
now; most of the flatpak users will still be on `bare-user`, and we haven't
changed the semantics of that mode yet.
Note that in this patch we will also *reject* file content that doesn't
match this. This is somewhat asymmetric, since we aren't similarly rejecting
e.g. directory metadata. But, this will close off the biggest source
of the problem for flatpak (setuid binaries).
See: https://github.com/ostreedev/ostree/pull/908
See: https://github.com/flatpak/flatpak/pull/837Closes: #909
Approved by: alexlarsson
Copying xattrs when manipulating the GPG keyring for a repository
causes errors when the underlying filesystem doesn't support writing
xattrs - overlayfs is a common example. It also causes the selinux
attributes of the keyring files to be copied from the temporary
location instead of properly inherited from the destination directory
(ending up, for example, as unconfined_u:object_r:user_tmp_t:s0, rather
than unconfined_u:object_r:data_home_t:s0)
Closes: #910
Approved by: cgwalters
When falling back to copying, we previously would only chmod checked out
files in the non-user-checkout mode. Fix this by always doing chmod.
The file_mode was being prepared but never actually applied.
Add a basic test in the archive-z2 --> usermode checkout case in which
we're guaranteed to always fall back to copy mode.
Closes: #633Closes: #903
Approved by: cgwalters
This was making it impossible to pull or mirror a large ostree repo, and
according to Colin is no longer necessary. It works fine with a test
against a repo with 2741 commit and 451468 objects in it.
Closes: #899Closes: #904
Approved by: jlebon
This reverts commit 1eff3e8343. There
are a few issues with it. It's not a critical thing for now, so
let's ugly up the git history and revisit when we have time to
debug it and add more tests.
Besides the below issue, I noticed that the simple `ostree remote add`
now writes to `/ostree/repo/config` because we *aren't* using the
`--sysroot` argument.
Closes: https://github.com/ostreedev/ostree/issues/901Closes: #902
Approved by: mike-nguyen
There was a lot of conditionals inside `write_object()` differentating
between metadata/content, and then for content, on the different repo
types. Further, in the metadata path since the logic is simpler, can
present a non-streaming API, and further use `OtTmpfile`, etc.
Splitting them up helps drop a lot of conditionals. We introduce a small
`CleanupUnlinkat` that allows us to fully convert to the new code style in both
functions.
This itself is still prep for fully switching to `GLnxTmpfile`.
Closes: #881
Approved by: jlebon
If we have an expected checksum, call `fstatat(repo_dfd, checksum)`
early on before we do much else. This actually duplicates code,
but future work here is going to split up the metadata/content
commit paths, so they'll need to diverge anyways.
Closes: #881
Approved by: jlebon
First, the streaming metadata API is pretty dumb, since metadata
should be small. Really we should have supported a `GBytes`
version. Currently, this API *is* used when we do local pulls,
so this commit has test coverage. However, I plan to change
the object import to avoid using this. But that's fine, since
I can't think of why someone would use this API.
Next, the only difference between `ostree_repo_write_metadata()` and
`ostree_repo_write_metadata_trusted()` is whether or not we pass
an output checksum; so just dedup the implementations.
Also while I'm here break out the input length validation and do
it early in the streaming case.
Closes: #881
Approved by: jlebon
Using `${sysroot}` to mean the physical storage root: We don't want to write to
`${sysroot}/etc/ostree/remotes.d`, since nothing will read it, and really
`${sysroot}` should just have `/ostree` (ideally). Today the Anaconda rpmostree
code ends up writing there. Fix this by adding a notion of "physical" sysroot.
We determine whether the path is physical by checking for `/sysroot`, which
exists in deployment roots (and there shouldn't be a `${sysroot}/sysroot`).
In order to unit test this, I added a `--sysroot` argument to `remote add`.
However, doing this better would require reworking the command line parsing for
the `remote` argument to support specifying `--repo` or `--sysroot`, and I
didn't quite want to do that yet in this patch.
Closes: https://github.com/ostreedev/ostree/issues/892Closes: #896
Approved by: jlebon
I saw a few instances of `glnx_set_error_from_errno() + return FALSE`,
and fixed them and did a bit of style conversion.
Closes: #895
Approved by: jlebon
Having a failable accessor is annoying, since it's really common
to reference both. Instead, open the repo once when we load
the sysroot, and provide a non-failable accessor.
This is also prep for `ostree_repo_open_at()`, which collapses the separation
between `ostree_repo_new()` and `ostree_repo_open()`.
Closes: #886
Approved by: jlebon
This is prep for introducing a fd-relative `ostree_repo_new_at()`.
Previously, `ostree_repo_is_system()` compared `GFile` paths, but
there's a much simpler check we can do first - if this repository
was created via `OstreeSysroot`, it must be a system repo.
Closes: #886
Approved by: jlebon
This is a de-scoping of work I did in preparation for
rpm-ostree [live updates](https://github.com/projectatomic/rpm-ostree/pull/652).
Originally I was going to expose this as a public API.
However, I decided to do things differently, but the cleanup here for new code
style and fd-relative is nice to have anyways.
We rework things to use `OstreeDeployment*`, which the caller is expected to
already have, rather than `GFile*`s pointing to the config directories.
Closes: #741
Approved by: jlebon
The summary file can get large, but it compresses well (something
which is not true of other files in the ostree repo which are
already compressed). By sending Accept-Encoding: gzip (and
handling the compressed results) we send a lot less data.
I set up the flathub repo (http://flathub.org/repo) to enable
gzip for the summary file (only), and the result is that the
331514 byte large summary was transferred in 122889 bytes.
On my (fast) network this decreased the time i took to do
"flatpak remote-ls flathub" by about 100msec.
This fixes https://github.com/ostreedev/ostree/issues/802Closes: #882
Approved by: cgwalters
It's hard right now to do a full port to the new libglnx tmpfile
API since there are complex cases in the commit path which deal
with symlinks as well.
Let's make things more gradual by introducing the important part (struct with
autocleanup) here in libotutil, port what we can. This will make a future
complete port easier.
Closes: #871
Approved by: jlebon
A commit can now include a "ostree.endoflife-rebase" metadata key
pointing to a new ref.
When updating, the sysroot upgrader will see this and proceed to
pull and deploy the new ref instead. The origin file in the new
deployment will point to the new ref.
This functionality is planned to be used in Endless OS. We will create
a lesser tested branch for brand new, cutting edge hardware support,
and ship that on hardware platforms that require the latest drivers.
However, once our slower-moving official release is later updated to
support the new hardware, we will use this functionality to migrate
those bleeding-edge users over to the official release.
Closes: #874
Approved by: cgwalters
The whole ostree-remote.h file is only included in the public ostree.h
header if OSTREE_ENABLE_EXPERIMENTAL_API is defined, so there’s no need
to change the set of methods defined in it according to whether we’re
compiling with experimental API.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #875
Approved by: cgwalters
Make it an internal, not static, API; like _ostree_repo_add_remote(). It
will be used in many the same situations.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #875
Approved by: cgwalters
Return whether the remote already existed. This is an internal API, so
it’s not an API break. The return value will be useful in upcoming
commits for working out whether to later remove a remote again.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #875
Approved by: cgwalters
Add a name argument to the internal OstreeRemote constructor,
since this member (and several derived from it) is non-nullable,
and hence must always be set at construction time.
This changes the only call sites of the constructor to use the new API,
which is internal.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #875
Approved by: cgwalters
Just for internal documentation; g-ir-scanner doesn’t read or understand
them.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #875
Approved by: cgwalters
Now that we’ve got a public, sealed OstreeRemote structure, we can start
carefully exposing members of it as API.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #875
Approved by: cgwalters
If somehow a repo has gpg verification on but doesn't have signatures
present for the existing commit, ostree would error out if it needs to
scan the commit object (e.g. if there are no updates available).
An instance of this is currently happening in Fedora AH, in which
signatures are not shipped in the ISO due to filesystem restrictions.
Another possible scenario is if a content provider switches from not
signing commits to signing them; even if older commits are retroactively
signed, clients' local commit objects would error out if they needed
scanning.
This patch adds a check to ensure that we always attempt to fetch the
detached metadata and wait for its result (whether it exists or not)
before moving on to scan their corresponding commit objects.
See also: https://github.com/projectatomic/rpm-ostree/issues/630Closes: #873
Approved by: cgwalters
Porting a lot of this file would be hard since in many cases we do processing in
the `out:` section, so let's do what we can.
Closes: #870
Approved by: jlebon
Unbreaks mounting in CentOS. Newer systemd in Fedora pulls didn't need this, I
think due to `RequiresMountsFor=`. Anyways, this is what the fstab generator
does, and it's clearly right ✓.
Closes: https://github.com/ostreedev/ostree/issues/867Closes: #869
Approved by: jlebon
Follow up to a previous patch that addressed a double-close; I
realized we already had a helper for doing "open dfd iter, do nothing
if we get ENOENT". Raise it to libotuil, and port all consumers.
Closes: #863
Approved by: jlebon
I noticed an instance of this while working on https://github.com/ostreedev/ostree/pull/861
Which apparently I cargo-culted into the new system generator bits.
Let's break this out as a small concise change.
Closes: #866
Approved by: jlebon
If one wants to set up a mount for `/var` in `/etc/fstab`, it
won't be mounted since `ostree-prepare-root` set up a bind mount for
`/var` to `/sysroot/ostree/$stateroot/var`, and systemd will take
the already extant mount over what's in `/etc/fstab`.
There are a few options to fix this, but what I settled on is parsing
`/etc/fstab` in a generator (exactly like `systemd-fstab-generator` does),
except here we look for an explicit mount for `/var`, and if one *isn't* found,
synthesize the default ostree mount to the stateroot. Another nice property is
that if an admin creates a `var.mount` unit in `/etc` for example, that will
also override our mount.
Note that today ostree doesn't hard depend on systemd, so this behavior only
kicks in if we're built with systemd *and* libmount support (for parsing
`/etc/fstab`). I didn't really test that case though.
Initially I started writing this as a "pure libc" program, but at one point
decided to use `libostree.so` to find the booted deployment. That didn't work
out because `/boot` wasn't necessarily mounted and hence we couldn't find the
bootloader config. A leftover artifact from this is that the generator code
calls into libostree via the "cmd private" infrastructure. But it's an easy way
to share code, and doesn't hurt.
Closes: #859
Approved by: jlebon
By checking the mount status, we avoid remounting things if we don't
need to. And printing a single line per mount helps debugging when
things go wrong.
Closes: #859
Approved by: jlebon
I really have no idea what I was thinking with that list of mount points. It
seems arbitrary. Sadly `git log` doesn't help, and there's no comments.
Basically, the only mounts we should care about are those that libostree
creates. Which are just `/sysroot` and `/var`. Systemd will handle the other
things like `/tmp`, it's not our job, and we shouldn't touch them.
Closes: #859
Approved by: jlebon
In current systemd, there is:
[systemd-volatile-root](https://www.freedesktop.org/software/systemd/man/systemd-volatile-root.service.html)
which was introduced by [this commit](91214a37ef).
I'd like to make further changes to how we handle `/var`, and I don't
want to reason about the interaction of our "tmpfs var" with too many
other things.
The comment about having "all /var handling in one place" was always inaccurate
given that we rely on systemd for mounting. And in general, I don't want to
duplicate too many things systemd does - it does them well, documents them, etc.
As far as I know, it was basically just Owen who was using this for the GNOME
hardware testing effort, and I'm sure he could easily switch over to
`systemd.volatile=state`.
Closes: #856
Approved by: owtaylor
This is what caused the merge of
https://github.com/projectatomic/rpm-ostree/pull/652
to blow up, since https://github.com/ostreedev/ostree/pull/848
landed right before we tried to merge it.
When I was writing that PR I remember having an uncertain feeling
since we were doing a `mkdirat` above, but at the time I thought
we'd have test suite coverage...turns out we didn't.
For backwards compatibility, we need to continue to do a `mkdirat` here of the
parent. However...I can't think of a reason anyone would *want* that behavior.
Hence, let's add a special trick - if the destination name is `.`, we skip
`mkdirat()`. That way rpm-ostree for example can open a dfd for `/etc` and avoid
the `mkdir`.
Fold the subpath tests into `test-basic.sh` since it's not worth a separate
file. Add a test case for checking out a file.
Closes: #854
Approved by: jlebon
As $DEITY intended.
I was reading the `prepare-root.c` code and the indentation damage was
distracting. Squash tabs that have leaked into various places in the code. I
didn't yet touch the `src/libostree` bits as that has higher potential for
conflict.
Closes: #852
Approved by: jlebon
Looking at `perf record ostree checkout`, some things stand out; e.g.:
```
+ 27.63% 0.07% ostree libgio-2.0.so.0.5000.3 [.] g_file_enumerator_iterate
+ 22.74% 0.28% ostree libostree-1.so.1.0.0 [.] ostree_repo_file_tree_query_child
+ 13.74% 0.08% ostree libostree-1.so.1.0.0 [.] ot_variant_bsearch_str
```
The GIO abstractions are already fairly heavyweight, and `OstreeRepoFile` mallocs
a lot too.
Make things more efficient here by dropping the GIO bits for reading ostree data -
we just read from the variants directly and iterate over them. The end result
here is that according to perf we go from ~40% of our time in the kernel to
~70%, and things like `g_file_enumerator_iterate()` drop entirely out of the
hot set.
Closes: #848
Approved by: jlebon
Since we now have a cleaner separation of "toplevel checkout prep"
versus "recursive checkout", handle the special case of checking out
a single file at first rather than later.
Prep for future work in optimizing this function more.
Closes: #848
Approved by: jlebon
Only non-mechanical bit here was creating a local autoptr for a bit
where we'd previously done an unref for a struct member.
Closes: #847
Approved by: jlebon
Rather than `g_output_stream_splice()`, where the input is a regular
file.
See https://github.com/GNOME/libglnx/pull/44 for some more information.
I didn't try to measure the performance difference, but seeing the
read()/write() to/from userspace mixed in with the pointless `poll()` annoyed me
when reading strace.
As a bonus, we will again start using reflinks (if available) for `/etc`,
which is a regression from the https://github.com/ostreedev/ostree/pull/797
changes (which before used `glnx_file_copy_at()`).
Also, for the first time we'll use reflinks when doing commits from file-backed
content. This happens in `rpm-ostree compose tree` today for example.
Update submodule: libglnx
Closes: #817
Approved by: jlebon
Continuing to chip away at this. Using `g_file_enumerator_iterate()`
here helps notably.
I started on the much bigger `ostree_diff_dirs_with_options()` but
it's a lot messier - for later.
Closes: #844
Approved by: jlebon
Previously it was static to ostree-repo.c. Make it usable throughout
libostree so it can be used by an upcoming commit, but also expose the
typedef and reference counting functions so that opaque OstreeRemote
pointers can be used by user code, in anticipation of exposing more of
its API publicly in future.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #832
Approved by: cgwalters
This allows consumers of libostree to check at configure time whether it
supports the feature they want.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #832
Approved by: cgwalters
There are currently no unstable APIs, but some will be added in
following commits. They will be built and exposed in the libostree
global symbol list iff configured with --enable-experimental-api.
Distributions should not package OSTree with --enable-experimental-api.
This is designed for previewing new APIs on controlled platforms; any of
the APIs hidden behind this option may be changed or removed at any
point.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #832
Approved by: cgwalters
Read the http-headers (a(ss)) option in
ostree_repo_remote_fetch_summary_with_options like
ostree_repo_pull_with_options and add the headers to the fetcher. This
allows things like providing additional authorization headers to the
HTTP requests.
Closes: #839
Approved by: cgwalters
Allow GI bindings to delete refs through ostree_repo_transaction_set_ref
and ostree_repo_transaction_set_refspec by setting the checksum to NULL.
Closes: #834
Approved by: cgwalters
Rather than hard-coding the names as strings. This makes the code a
little more maintainable.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #826
Approved by: cgwalters
If a summary metadata key is well-known, like ostree.static-deltas, or
ostree.summary.last-modified, format it a little more nicely.
This is especially important for timestamps like last-modified, since
otherwise they’re formatted as a big-endian uint64, which is basically
unusable for the user.
Non-formatted output can still be retrieved using the OSTREE_DUMP_RAW
flag, and the non-formatted key name is always printed for clarity.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #826
Approved by: cgwalters
This allows a locally generated summary file to be viewed. It accepts
the same arguments as `ostree remote summary` (i.e. --raw).
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #826
Approved by: cgwalters
• Commit timestamps, so it’s easy to work out whether a given commit is
newer than the one we have locally
• Summary file timestamp, so it’s easy to work out whether the summary
file is more up to date than another summary file
• Summary file expiry time, so clients can work out when they should
expect the summary file to next be updated, and hence can query for
it at roughly the right time
The expiry time requires input from the user, so is currently never set
automatically. Programs using libostree can set it if they wish.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #826
Approved by: cgwalters
While running the testsuite under valgrind a small memory leak showed up:
==16487== 65 bytes in 1 blocks are definitely lost in loss record 773 of 1,123
==16487== at 0x4C2BBAF: malloc (vg_replace_malloc.c:299)
==16487== by 0x6048E08: g_malloc (gmem.c:94)
==16487== by 0x6062EAE: g_strdup (gstrfuncs.c:363)
==16487== by 0x54CE3E6: write_object (ostree-repo-commit.c:776)
==16487== by 0x54CF2D4: ostree_repo_write_metadata (ostree-repo-commit.c:1528)
==16487== by 0x54CF505: _ostree_repo_write_directory_meta (ostree-repo-commit.c:1712)
==16487== by 0x54D0AB4: write_dfd_iter_to_mtree_internal (ostree-repo-commit.c:2650)
==16487== by 0x54D0E2D: ostree_repo_write_dfd_to_mtree (ostree-repo-commit.c:2793)
==16487== by 0x1190C4: ostree_builtin_commit (ot-builtin-commit.c:474)
==16487== by 0x11F2EE: ostree_run (ot-main.c:200)
==16487== by 0x116F32: main (main.c:78)
The reason for this is that ot_checksum_instream_get_string returns a chunk of newly allocated memory which never got freed.
Make actual_checksum something that gets autocleanend and own the memory
assigned to it in all cases.
Signed-off-by: Sjoerd Simons <sjoerd.simons@collabora.co.uk>
Closes: #827
Approved by: pwithnall
If we're freeing the segment, it's basically always better to use
`autoptr()`. Fewer lines, more reliable, etc.
Noticed an instance of this in the pull code while reviewing a different PR,
decided to do a grep for it and fix it tree wide.
Closes: #836
Approved by: pwithnall
If one of the progress keys is set in a pull operation, a ::changed
signal is emitted on the progress object, and the callback for that
could query any of the progress keys — so they all need to be set,
otherwise we get an assertion failure in ostree_async_progress_get() due
to a named key not existing.
Spotted by Dan Nicholson in PR #819.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #835
Approved by: cgwalters
The summary URL override is looked up with "&s", which directly
exchanges the data to a pointer without allocation. This was causing a
segfault calling ostree_repo_remote_fetch_summary_with_options from
pygobject.
Closes: #829
Approved by: jlebon
g_date_time_new_from_unix_utc() will not always return a valid GDateTime
— if the input timestamp is too big, GDateTime cannot represent it, and
the constructor returns NULL.
Add some missing checks for these situations. We don’t ever expect
timestamps to be this big, but they could be as a result of corruption
or a malicious repository.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #825
Approved by: cgwalters
The keyring isn't large, so let's just fall back to copying it
rather than requiring `renameat()`.
Prep for `ostree_repo_open_at()`.
Closes: #821
Approved by: jlebon
Use the new well-known `status` key for OstreeAsyncProgress to get and
set the status atomically with other keys in an OstreeAsyncProgress
instance.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #819
Approved by: cgwalters
Rework how the status is handled in OstreeAsyncProgress so that it’s now
a well-known key in the hash table. This means that it can be retrieved
and set atomically with other keys using
ostree_async_progress_[get|set]().
The behaviour of ostree_async_progress_[get|set]_status() is preserved,
with the caveat that `status` can now also be accessed using the other
API on OstreeAsyncProgress, and has to be accessed with the right
GVariant type.
Internally, a NULL status is represented by an empty status string
(since ostree_async_progress_[get|set]_variant() deliberately don’t
allow NULL variants to be set against keys, since that would break the
ostree_async_progress_get() API).
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #819
Approved by: cgwalters
This will eliminate most of the potential races in progress reporting.
ostree_repo_pull_default_console_progress_changed() still calls three
getters, so there may still be races there, however.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #819
Approved by: cgwalters
OstreeAsyncProgress is thread-safe: it can have keys changed by one
thread while another is getting the same keys (modulo some locking
contention). However, the thread safety is done at the function call
level: if some code calls an OstreeAsyncProgress getter several times,
the key fetches are not atomic with respect to each other.
In the case of contention on the lock, this can result in consumers of
OstreeAsyncProgress data seeing an inconsistent state between the
properties they query, which could result in progress reporting
inaccuracies.
In the uncontested case, this results in the OstreeAsyncProgress lock
being locked and unlocked many times more than necessary.
Try to improve this by adding new API, which supports getting and
setting multiple keys atomically:
• ostree_async_progress_get()
• ostree_async_progress_set()
The new API uses GVariants and varargs: keys are passed as a
GVariantType string followed by arguments as for g_variant_new() or
g_variant_get(), followed by the next key, etc.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #819
Approved by: cgwalters
OstreeAsyncProgress currently does some contortions to try and avoid
allocating space for guints and guint64s (on 64-bit platforms), but this
means it uses two GHashTables. A GHashTable allocates 8 buckets even
when empty. Given that the largest usage of OstreeAsyncProgress in
libostree puts 13 uints and 5 uint64s in it, this optimisation does not
save significant (if any) memory.
Instead, change OstreeAsyncProgress to store values internally as
GVariants, and expose this with some new API:
• ostree_async_progress_get_variant()
• ostree_async_progress_set_variant()
Each GVariant is allocated on the heap. As they are immutable, they are
thread-safe once returned by a getter.
The existing API continues to work as before, except in the case where a
key is set/got as both a uint and a uint64 — there will now be a
collision (and a GVariant type checking failure) whereas previously
there was no collision. Nothing in OSTree uses OstreeAsyncProgress this
way though.
The new API can be used to share more complex data via the progress API.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #819
Approved by: cgwalters
Instead of using G_OPTION_ARG_STRING, use G_OPTION_ARG_FILENAME, which
handles filename encoding conversion differently from the locale
conversion which G_OPTION_ARG_STRING. This will fix argument handling on
systems where the filename encoding is not the same as the locale
encoding (which is fairly unlikely since most systems use UTF-8).
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Closes: #810
Approved by: cgwalters
I happened to glance at the top of my most recent patch and
noticed that I used an `throw_errno()` function in a non-errno place.
I scanned the patch for other instances of this but didn't find one.
Closes: #811
Approved by: jlebon
I was planning to change some of the object loading code in the
future, so here's some porting.
Note that I rewrote `_ostree_repo_has_loose_object()` since it
used an error return across multiple functions.
Honestly I'm not sure about this `TEMP_FAILURE_RETRY()` business...
in reality we're going to end up with a ton of code linked in
process that doesn't do it. Unix sucks =( But I'm keeping
what was there out of consistency.
Closes: #809
Approved by: jlebon
This did a `closedir` in the `goto out` section before, but it
turns out more nicely if we follow the usual pattern of doing
the `open(O_DIRECTORY)` in the callee function and handle `ENOENT`
there.
Closes: #809
Approved by: jlebon
This is a variant of the efforts in https://github.com/ostreedev/ostree/pull/741
Working on `rpm-ostree livefs`, I realized though I needed to just
check out *new* files directly into the live `/etc` (and possibly
delete obsolete files).
The way the current `/etc` merge works is fundamentally different from
that. So my plan currently is to probably do something like:
- Compute diff
- Check out each *new* file individually (as a copy)
- Optionally delete obsolete files
Also, a few other things become more important - in the current deploy code, we
copy all of the files, then relabel them. But we shouldn't expose to *live*
systems the race conditions of doing that, plus we should only relabel files we
checked out.
By converting the deploy's /etc code to use this, we fix the same TODO item
there around atomically having the label set up as we create files. And further,
if we kill the `/var` relabeling which I think is unnecessary since Anaconda
does it, we could delete large chunks of code there.
In the implementation, there are two types of things: regular files, and
symlinks. For regular files, in the `O_TMPFILE` case, we have the ability to
do *everything* atomically (including SELinux labeling) before linking it into
place. So let's just use that. For symlinks, we use `setfscreatecon()`.
Closes: #797
Approved by: jlebon
We really have an astonishing variety of similar functions which write files and
symlinks. I was working on a different PR and the duplication between the
union-mode and add-mode/none-mode checkout functions bothered me.
I realized that the "handle EEXIST" tri-state maps directly to the
`GLnxLinkTmpfileReplaceMode`, so deduping things makes even more sense.
Closes: #801
Approved by: jlebon
I was reading a strace the other day and noticed we were loading the same
`.dirmeta` object many times. Unlike the other object types, `.dirmeta` objects
don't accumulate much over time; there are only so many directory metadata types.
(Without SELinux involved it'd probably be 5-6 I'd guess offhand).
For `fedora-atomic/25/x86_64/docker-host` there are currently 34 `.dirmeta` in
the tree.
But how many times during a checkout did we load those 34 dirmeta objects?
With a quick strace:
```
$ strace -s 2048 -f -o strace.log ostree --repo=repo-build checkout -U fedora-atomic/25/x86_64/docker-host host-test-checkout
$ grep dirmeta strace.log | wc -l
7165
```
After, as you'd expect, we just loaded `34` from disk. We do
6 system calls (`openat+fstat+fstat+read+read+close`) per dirmeta,
so we dropped a total of 42780 system calls - which is about 20% of the total
system calls made.
`perf record` tells me that we're spending ~40 of our time in the kernel during
a checkout, so reducing syscall traffic helps. Though most of that appears to be
in the VFS and XFS layers for `linkat` (which isn't surprising).
So how much did perf improve? Well, on my workstation, I get a lot of
fluctuation in timing, sometimes by 30%, so this was well within the noise. But
it's well worth speeding up checkout, and I think this optimization will shine
more as we improve performance elsewhere.
Closes: #795
Approved by: jlebon
These are leftovers from the packfile code and should have been
deleted in commit: 2a0601efc7
I noticed this now since I wanted to add a new type of caching.
Closes: #795
Approved by: jlebon
I never really liked the term "osname". I feel "stateroot" is a *lot* clearer,
since the osname/stateroot mostly just holds `/var`. Further it avoids the `os`
prefix which is already overloaded.
Some of the existing docs already talked about "operating system state", which
further reinforces this.
There's *lot* more things than this which reference the term "osname", but I
don't want to change *everything* yet in this patch in case we decide to do
something different - this just gets the highlights.
Closes: #794
Approved by: jlebon
This is intended to be used for copying `/usr/etc` → `/etc` for
deployments.
A TODO here is to use `glnx_file_copy_at()` if the repo mode allows
it - then we'd use reflinks if available.
Closes: #804
Approved by: jlebon
There aren't many users of `g_file_enumerator_iterate()` left - those
remaining are usually good candidates for porting. There's some more
porting to do in this file; a mix of trivial and harder. This
one is a good candidate for an individual commit.
Closes: #803
Approved by: jlebon
I plan to make some future changes here, and wanted to do this
first.
Random side note; how about converting the do/while loops for `EINTR` to
`TEMP_FAILURE_RETRY()`? We're very inconsistent about that...
Closes: #792
Approved by: jlebon
This could help others who want to integrate with other init
systems/initramfs.
Commit-message-by: Colin Walters <walters@verbum.org>
Closes: #784
Approved by: cgwalters
`perf record ostree checkout ...` for a bare-user repo was telling
me we were spending a good 13% of our time in the depchain of `ot_lgexattrat()`.
The problem here is that traversing the `/proc` path turns out to be
somewhat expensive - there are LSM (SELinux) checks, etc.
For regular files, opening and just getting the xattr, then closing is still
quite cheap. For symlinks, we'll always need to open anyways.
This appears to shave about ~0.1 seconds off of a checkout of
`fedora-atomic/25/x86_64/docker-host` on my workstation.
Oh, and this was the last user of `ot_lgexattrat()` so we can kill it, which is
nice - the xattr code should really live in libglnx.
Closes: #796
Approved by: jlebon
This is only about 40%, and mostly simpler functions. It's
nice to switch to `g_autoptr(GMatchInfo)` instead of our inline version.
I decided to add more usage of `ot_transfer_out_value()`, though it'd
be nice to try to have a copy of that in libglnx (or possibly glib).
Closes: #791
Approved by: jlebon
Logic error introduced after refactoring; we hoisted the
`is_bare_user_symlink` variable to the top, but its computation
below. But the `is_bare` symlink depended on it.
Closes: https://github.com/ostreedev/ostree/issues/798Closes: #799
Approved by: jlebon
I think the majority of OSTree usage calls pull with refs, not
explicit commits. We even added special "override syntax" with
`@` (e.g. `ostree pull foo@ab12c34`) as a hybrid.
However, some users may want to still pull explicit commits
for whatever reason. The old static delta logic looked at
the previous commit of the ref. However, in https://github.com/ostreedev/ostree/pull/710
we enhanced the logic to look at all local commits.
It's now a lot more natural to teach the delta logic
to support revisions, e.g. `ostree pull someorigin ab12c34`.
This also fixes the problem that before, `--require-static-deltas`
was completely ignored when processing revisions.
This is a nontrivial refactoring of the logic, but the end
result feels a lot more readable to me.
Closes: https://github.com/ostreedev/ostree/issues/783Closes: #787
Approved by: cgwalters
Previously we'd assert and dump core if one used `checkout -H` without
`-U` on a bare-user repo, because we'd hit the bare-user symlink case.
Rework the code to handle this, and add tests. I hit this when I was going to
suggest to someone to use `-H` to ensure they were getting hardlinks.
Closes: #779
Approved by: jlebon
Testing a fetch of `fedora-atomic/.../docker-host` from
an nginx instance over `https://127.0.0.1` using Fedora 25
versions. Average over 3 runs:
Before: ~24.6 seconds
After: ~19 seconds
Speedup: ~30%
Closes: https://github.com/ostreedev/ostree/issues/778Closes: #780
Approved by: jlebon
In particular the 26-variable monster 👹 in `install_deployment_kernel()` is
slain🗡. I didn't touch every function here, trying to keep things gradual.
Closes: #781
Approved by: jlebon
It was reported that in the range request handling, we called `remove_pending()`
twice (once in processing it, and once potentially in the local_error cleanup),
and this could be viewed as a use-after-free. However, right now the range
cleanup and `local_error` being set are mututally exclusive.
Further, the task object already holds a strong reference, so I observed the
refcount was 2. For both of these reasons, there is no use-after-free in
practice.
Reported-By: "Siddharth Sharma" <siddharth@redhat.com>
Closes: #774
Approved by: jlebon
I was planning to change one here, decided to do a conversion
of some of the simpler functions in this file to keep up momentum.
Closes: #776
Approved by: jlebon
I was working on `rpm-ostree livefs` which does some ostree-based
filesystem diffs, and noticed that we were ending up with `/proc`
not being labeled in our base trees.
Reading the selinux-policy source, indeed we have:
```
/proc -d <<none>>
/proc/.* <<none>>
```
This dates pretty far back. We really don't want unlabeled
content in ostree. In this case it's mostly OK since the kernel
will assign a label, but again *everything* should be labeled via
OSTree so that it's all consistent, which will fix `ostree diff`.
Notably, `/proc` is the *only* file path that isn't covered when composing a
Fedora Atomic Host. So I added a hack here to hardcode it (although I'm a bit
uncertain about whether it should really be `proc_t` on disk before systemd
mounts or not).
Out of conservatism, I made this a flag, so if we hit issues down the line, we
could easily change rpm-ostree to stumble on as it did before.
Closes: #768
Approved by: jlebon
Being bitten by lack of PR testing here. There are two bugs:
- First and foremost, I forgot that GObject will call the property setters with
the defaults. This meant we were getting both path="/var/tmp/blah" and fd=-1,
and we were accepting -1 as a fd, which then got converted into AT_FDCWD
which was wrong.
- Since these properties are construct only and mutually exclusive, don't
try to handle one resetting the other. Assert that exactly one of them is set.
Closes: #769
Approved by: jlebon
This is inspired by the [Coccinelle](http://coccinelle.lip6.fr/) usage
in systemd. I also took it a bit further and added infrastructure
to have spatches which should never apply. This acts as a blacklist.
The reason to do the latter is that coccinelle is *way* more powerful than the
regular expresssions we have in `make syntax-check`.
I started with blacklisting `g_error_free()` directly. The reason that's bad is
it leaves a dangling pointer.
Closes: #754
Approved by: jlebon
I happened to read this file and realized there's a lot of cruft left over from
the time when I liked `GFile` and `malloc()`ing like 50 times just to make a
pathname string. Delete it.
Closes: #767
Approved by: jlebon
Actually trying to use this in rpm-ostree, it kept returning successfully when I
didn't expect it to... The first conditional was always succeeding even when I
was asking for a newer minor.
Closes: #766
Approved by: jlebon
I'd like to do this in rpm-ostree at least. Originally I was looking at porting
to `G_DECLARE_FINAL_TYPE` but eh, this is easier for now and won't bump our GLib
dependency which might matter for our embedded users.
For now I just did a few replacements in the `remote` command line. A full port
can come as we do other code cleanups.
This will actually break the flatpak build right now, but
that's easy to fix. And we concluded in e.g.
https://bugs.freedesktop.org/show_bug.cgi?id=95065#c5
it's a bug for downstream projects to do that.
Closes: #756
Approved by: jlebon
I didn't touch everything since at least `commit_loose_object_trusted`
does this:
```
out:
if (G_UNLIKELY (error && *error))
g_prefix_error (error, "Writing object %s.%s: ", checksum, ostree_object_type_to_string (objtype));
```
Which...it'd be interesting to make into an autocleanup. But for now just
keeping up with converting things bit by bit.
Closes: #761
Approved by: jlebon
In [this commit](6ce80f9685)
for some reason I added a `sepolicy` member to the sysroot. I
have no idea why I did that, and it's conceptually wrong
since the policy is specific to a *deployment*.
This bit me when I was working on [a pull request](https://github.com/ostreedev/ostree/pull/763)
elsewhere, since at that point it was `NULL`.
We already pass around the sepolicy in the deployment code, so just stop caching
it.
Closes: #764
Approved by: jlebon
I was playing around in a FAH vagrant box, and hit:
```
Receiving delta parts: 3/4 453.2 kB/s 1.8 MB/145.8 MB
error: opcode set-read-source: No such file object b6e54ba3471b9c116ce6b9bfbf9e55fec60d35cfdb9ae5ae1ee219af02a591b7
```
This is because this host version doesn't yet have
https://github.com/ostreedev/ostree/pull/710
which incidentally fixed this for the case where the OS vendor is using
summary files.
Some organizations may not be using summary files - at least we still try to
support that case. So let's copy the logic very recently added in that commit to
handle the legacy case too.
No new tests since this is a nice-to-have - we really do
expect people to be using summary files now.
Closes: #739
Approved by: jlebon
Robert McQueen
Colin Walters
Ruixin
Jonathan Lebon
Matthew Leeds
Guy Shapiro
Matthew Barnes
Philip Withnall
Emmanuele Bassi
Krzesimir Nowak
David Shea
Alexander Larsson
Tristan Van Berkom
Anton Gerasimov
Owen W. Taylor
Brian C. Lane
Daniel Drake
Dan Nicholson
Sjoerd Simons
Francesco Giannelli
Daniel J Walsh