Commit Graph

1574 Commits

Author SHA1 Message Date
Colin Walters d5273b34d0 lib/repo: Add API to create and list ref aliases
There are multiple use cases where we'd like to alias refs.

First, having a "stable" alias which gets swapped across major
versions: https://pagure.io/atomic-wg/issue/228

Another case is when a ref is obsoleted;
<https://pagure.io/atomic-wg/issue/303>
This second one could be done with endoflife rebase, but I think
this case is better on the server side, as we might later change
our minds and do actual releases there.

I initially just added some test cases for symlinks in the `refs/heads` dir to
ensure this actually works (and it did), but I think it's worth having APIs.

Closes: #1033
Approved by: jlebon
2017-08-02 17:33:10 +00:00
Colin Walters d1eb909cd0 lib/deploy: Add structured logging info for syncfs() times
I plan to at some point change rpm-ostree to read the journal messages from
libostree and render things like the time we spent in syncfs().

Closes: #1044
Approved by: jlebon
2017-08-02 17:19:37 +00:00
Colin Walters 4d723df2f2 lib/sysroot: Add prefixes to syncfs/fsync error messages
And clean up one other bare `glnx_throw_errno()`.

Closes: #1044
Approved by: jlebon
2017-08-02 17:19:37 +00:00
Colin Walters 40b41d2cc9 tree-wide: Use GLib autocleanups for libarchive
Define typedefs for read/write archives, and use the GLib
autocleanups for them.   Prep for updating libglnx to drop its
custom autocleanup macros.

Closes: #1042
Approved by: jlebon
2017-08-02 16:55:38 +00:00
Colin Walters ded6417aee lib/gpg: Switch to GLib autocleanups for gpgme types
Prep for dropping `GLNX_DEFINE_CLEANUP_FUNCTION` from libglnx
in favor of using GLib's `G_DEFINE_AUTO_CLEANUP_FREE_FUNC()`.

Closes: #1042
Approved by: jlebon
2017-08-02 16:55:38 +00:00
Colin Walters 6347c0fb88 lib/curl: Only check individual request errors
It looks like `curl_multi_socket_action()` will return an error
if *one* of the requests has an error, but we already check
for that explicitly by iterating over each handle.

In libcurl, the "easy" layer doesn't really make use of this
return value.  I did a bit of looking elsewhere; systemd
does check it as a runtime error, not an assertion.  librepo
doesn't use the multi interface.

Closes: https://github.com/ostreedev/ostree/issues/1035

Closes: #1038
Approved by: jlebon
2017-08-02 15:44:00 +00:00
Colin Walters 2f0707a054 lib/commit: Rework a conditional set for clarity and Coverity
Coverity complained that the `else if (bytes_read == 0)` was dead
code if we happened to find it was already false when testing
`else if (G_UNLIKELY (bytes_read == 0 ...`.

There was nothing wrong with the logic, but let's rework it to
only test the value once; I think it does end up nicer anyways.

Coverity CID: 1452186

Closes: #1037
Approved by: jlebon
2017-08-02 15:34:16 +00:00
Robert McQueen e8a9172daa gpg-verify-result: try to display the primary key ID from signatures
The fingerprint associated with each signature can be different to
the primary key ID (the normal one that people use to identify a
GPG key) if the signature is from a signing subkey. Try to find the
primary key and print this ID in preference to the subkey signature.

https://github.com/ostreedev/ostree/issues/608

Closes: #1036
Approved by: cgwalters
2017-07-31 14:37:09 +00:00
Robert McQueen 156cf23576 gpg-verify-result: canonicalise key when looking up signatures
Use gpgme_get_key to find the primary key for the key we are
looking for, and the primary key for each signature, and
compare these when looking up signatures.

The primary key is the first in the list of subkeys, which is
the normal key ID people use when referring to a GPG key as an
identity.

If the key has a signing subkey, signature->fpr will not match
the provided key_id, so looking up both keys and comparing the
primary key fingerprints ensures they are both canonicalised.

https://github.com/ostreedev/ostree/issues/608

Closes: #1036
Approved by: cgwalters
2017-07-31 14:37:09 +00:00
Colin Walters d7f953aa3a lib/pull: Journal fetch results
This is a continuation of addition of journaling to libostree; see
e.g. <https://github.com/ostreedev/ostree/pull/708>.

I wanted more information at the end of fetches; in particular
some details about the delta execution (what opcodes etc.), but
this is a first step: we log things like the transferred data
as well as whether or not GPG was enabled, etc.

One awkward thing about this is how we map the fetcher options like
`tls-ca-path` back out into an enum for the code to log. But eh, hard to fix
without a bigger refactoring.

Closes: #1032
Approved by: jlebon
2017-07-28 13:50:22 +00:00
Colin Walters d24b259cc5 grub2: Handle aarch64 (at least on Fedora/RHEL)
See: https://github.com/projectatomic/rpm-ostree-toolbox/issues/102#issuecomment-316483554

The logic here should now match what grubby does.  I have no
idea whether this applies to other distributions though.

Closes: #1021
Approved by: jlebon
2017-07-27 18:21:00 +00:00
Colin Walters fa3e07e5d7 Release 2017.9
Closes: #1029
Approved by: jlebon
2017-07-26 21:04:08 +00:00
Colin Walters e09fc83ab3 lib/core: Add #defines for ref/collection binding
These were previously private, but since we expect people to use them, let's add
`#define`s like we did for some of the other commit metadata.

Closes: #1028
Approved by: jlebon
2017-07-26 18:52:04 +00:00
Colin Walters 0985158be7 Update libglnx, port some uses to newer APIs
Mostly for the latest `-Wmaybe-uninitialized` fix, but while here also port some
places to newer APIs.

Update submodule: libglnx

Closes: #1027
Approved by: jlebon
2017-07-24 18:43:57 +00:00
Ruixin ef6f6bc688 lib: Add #define for endoflife metadata key
It is a continuation of https://github.com/ostreedev/ostree/pull/1024
It adds documentation for endoflife metadata key.

Closes: #1025
Approved by: cgwalters
2017-07-21 15:27:06 +00:00
Colin Walters 6430207e47 lib: Add #defines for current well-known metadata keys
This came up in https://github.com/projectatomic/rpm-ostree/issues/142

Let's add `#define`s for our metadata keys, with documentation so
that, well, they're documented.

Closes: #1024
Approved by: peterbaouoft
2017-07-21 14:53:33 +00:00
Colin Walters f9f7d55e79 lib/commit: Fix EBADF with GENERATE_SIZES option for commit
Regression from previous tmpfile refactoring; unfortunately
the `OSTREE_REPO_COMMIT_MODIFIER_FLAGS_GENERATE_SIZES` option
only has coverage via gjs currently.

Might expose it via the cmdline in a later option, but in the big picture the
idea was that this data is better kept in static deltas.

Closes: https://github.com/ostreedev/ostree/issues/1014

Closes: #1016
Approved by: jlebon
2017-07-20 14:01:11 +00:00
Colin Walters 779f125cbe lib/repo: Auto-recreate repo/tmp if it's deleted
We can accumulate a lot of space there; let's be nice to people who delete the
whole directory.

Closes: https://github.com/ostreedev/ostree/issues/1018

Closes: #1020
Approved by: jlebon
2017-07-19 15:01:19 +00:00
Colin Walters c740b7f6d2 core: Sanitize error text validating refs (e.g. against HTML)
See: https://github.com/projectatomic/rpm-ostree/issues/885

If we get a successful Apache directory listing HTML when fetching what we
intend to be a ref, we'd dump the HTML into the error.

I did some scanning of the pull code, and this was the only case
I saw offhand where we were dumping text out into an error.  Which
makes sense, since most of our formats are binary, the exeptions I
think are just `repo/config` and `repo/refs/`.

Closes: #1015
Approved by: mbarnes
2017-07-19 14:45:57 +00:00
Colin Walters 2a9689b76a Update libglnx, port various bits to new API
Using the error prefixing in the delta processing allows us to
do new code style.  Also strip trailing whitespace.

Use error prefixing in a few other random places.  I didn't
hunt for all of them, just testing out the new API.

Use `glnx_fchmod()`.  Also note I dropped one `fchmod (tmpf, 0600)`
which is no longer necessary.

Update submodule: libglnx

Closes: #1011
Approved by: jlebon
2017-07-18 19:18:38 +00:00
Colin Walters efd460782a lib/pull: Drop direct use of ->repodir
Prep for `ostree_repo_new_at()`.  Down the line perhaps
we should extend libcurl to accept a file descriptor for cookies,
but this works OK for now.

Closes: #1010
Approved by: jlebon
2017-07-18 19:07:56 +00:00
Colin Walters e0346c1494 Add a notion of "physical" sysroot, use for remote writing
(Note this PR was reverted in <https://github.com/ostreedev/ostree/pull/902>;
 this version should be better)

Using `${sysroot}` to mean the physical storage root: We don't want to write to
`${sysroot}/etc/ostree/remotes.d`, since nothing will read it, and really
`${sysroot}` should just have `/ostree` (ideally). Today the Anaconda rpmostree
code ends up writing there. Fix this by adding a notion of "physical" sysroot.
We determine whether the path is physical by checking for `/sysroot`, which
exists in deployment roots (and there shouldn't be a `${sysroot}/sysroot`).

In order to unit test this, I added a `--sysroot` argument to `remote add`.
However, doing this better would require reworking the command line parsing for
the `remote` argument to support specifying `--repo` or `--sysroot`, and I
didn't quite want to do that yet in this patch.

This second iteration of this patch fixes the bug we hit the first time;
embarassingly enough I broke `ostree remote list` finding system remotes.
The fix is to have `ostree_repo_open()` figure out whether it's the same
as `/ostree/repo` for now.

Down the line...we might consider having the `ostree remote` command line itself
instatiate an `OstreeSysroot` by default, but this maximizes compatibility; we
just have to pay a small cost that `ostree` usage outside of that case like
`ostree static-delta` in a releng Jenkins job or whatever will do this `stat()`
too.

Closes: https://github.com/ostreedev/ostree/issues/892

Closes: #1008
Approved by: mbarnes
2017-07-18 18:58:06 +00:00
Colin Walters 8b1f1c4428 lib/pull: Do local content imports async too
This came up in <https://github.com/ostreedev/ostree/pull/982>; when
we added more direct local importing, we did it synchronously.

This was actually quite a regression when doing local pulls between different
modes; in particular between a bare mode and `archive`, as we were suddenly
doing gzip {de,}compression in the main thread.

Down the line actually...a simpler fix is probably to change things so that the
local path is really only used when we know we can hardlink; everything else
would go though the fetcher codepath but with `file://`.

But this isn't a lot more code, and the speed/interactivity win is large.

Note we're only doing content async with this patch. We could do metadata as
well; we have the object already local. But the metadata code path is messier,
and metadata objects are smaller.

Another area where this comes up is that in e.g. Fedora releng, most operations
talk to a NetApp via NFS. So this has the classic network filesystem problem
that operations that are normally cheap like `stat()` can actually have
nontrivial latency. Doing as much as possible in threads is better there too.

Closes: #1006
Approved by: jlebon
2017-07-18 17:03:13 +00:00
Emmanuele Bassi 47a54bf876 Move the include directive to the enum template
There is no actual written guarantee in glib-mkenums that the template
line specified using --fhead will be added after the templates specified
inside the template file. Since the template file is only used once, we
can simply move the `#include` directive inside the template, so that it
is guaranteed to be in the right place.

Closes: #1007
Approved by: cgwalters
2017-07-17 15:14:44 +00:00
Colin Walters 620a90ebfa lib/pull: Avoid journaling 404s for optional content
Currently in Fedora we don't sign summaries, and every use of
`rpm-ostree` would emit to the journal an error when we failed
to fetch it.

Fix this by having `OSTREE_FETCHER_REQUEST_OPTIONAL_CONTENT` tell the fetcher
not to journal 404 errors. While fixing this, we had a mix of two booleans vs
the flags; fix things so we consistently use the flags in the fetcher and pull
code.

Closes: #1004
Approved by: mbarnes
2017-07-13 21:41:00 +00:00
Philip Withnall acb14648d7 lib/repo: Add OSTREE_REPO_METADATA_REF as a well-known metadata store
As discussed in https://github.com/ostreedev/ostree/pull/946, the
summary file is becoming an unsigned cache of ref information; any
additional metadata for the repository needs to move elsewhere in order
to remain signed. Introduce OSTREE_REPO_METADATA_REF as the well-known
name of a ref where such metadata can live, as the metadata on
contentless commits.

Don’t yet update the documentation for summary-related methods to
mention this, since it’s still hidden behind the
--enable-experimental-api configure option.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #946
Approved by: cgwalters
2017-07-11 19:50:32 +00:00
Philip Withnall 7d57459e83 lib/repo-commit: Fix types of content size cache entries
Use goffset rather than gsize for file sizes. More importantly, get the
unpacked_size from g_file_info_get_size() (goffset) rather than from the
splice return value, which has type gssize.

This will make a difference on 32-bit systems, where goffset is defined
as off64_t, but gsize is 32 bits.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #999
Approved by: cgwalters
2017-07-11 14:55:55 +00:00
Colin Walters 9d941dcebb checkout: Don't set dir mtime to 0 when doing a force copy checkout
When we [switched to using checkout + force_copy](e8efd1c8dc),
a side effect that went unnoticed at the time is that we started
setting directory mtimes to zero.

See the below bug where we long ago set the file times to zero, which got fixed,
so let's not regress things by setting the directory times to zero either. (Even
though AFAICS GNU tar doesn't complain about those)

This semantic is somewhat "overloaded" onto `force_copy`, but it avoids adding
yet another boolean; we don't have that many reserved boolean slots left. I
can't really think of many good use cases for `force_copy` *other* than the
`/etc` merge anyways.

https://bugzilla.redhat.com/show_bug.cgi?id=1229160

Closes: https://github.com/ostreedev/ostree/issues/995

Closes: #997
Approved by: jlebon
2017-07-07 15:01:51 +00:00
Colin Walters d2a05e5a09 deploy: Port some functions to new style
There are a number of simple ports here.  Prep for further work
in `/etc` merge.

I also stripped trailing whitespace globally.

Closes: #996
Approved by: jlebon
2017-07-07 14:39:47 +00:00
Krzesimir Nowak cc9a0386c4 lib/pull: Collection and ref bindings verification
This verifies the collection and ref bindings in the commit metadata
against the collection ID we have stored in the remote config and ref
we want to pull from. For the HEAD commits, we also check if the
checksum of the commit we just fetched agrees with the checksum we
really wanted to pull from the ref.

For commits with explicitly specified checksums and without specified
refs, we only verify if the commit has the bindings. We are able to
only verify the collection binding, though.

Closes: #972
Approved by: cgwalters
2017-07-06 19:08:14 +00:00
Krzesimir Nowak d91f6a0f61 lib/pull: Pass the ref together with the request
We will want to use the requested ref later for the binding
verification.

Closes: #972
Approved by: cgwalters
2017-07-06 19:08:14 +00:00
Krzesimir Nowak cf16805a2f ostree: Add collection and ref bindings to metadata on commit
The collection and ref bindings are stored in the commit metadata
under ostree.collection-binding and ostree.ref-binding,
respectively. They will be used to verify if the commit really comes
from the collection and ref we wanted to pull from.

Closes: #972
Approved by: cgwalters
2017-07-06 19:08:14 +00:00
Colin Walters 28ec43c41a build-sys: Post-release version bump
Closes: #994
Approved by: jlebon
2017-07-06 14:45:02 +00:00
Colin Walters 5a5e465492 Release 2017.8
Closes: #994
Approved by: jlebon
2017-07-06 14:45:02 +00:00
Colin Walters 23b93a3eb6 lib/repo: Immediately error creating bare-user repo on tmpfs
And in general, if for some reason we can't write `user.` xattrs, provide an
error immediately rather than doing it during a later pull. This way the failure
cause is a lot more obvious.

Related: https://github.com/ostreedev/ostree/issues/991

Closes: #993
Approved by: jlebon
2017-07-06 14:31:37 +00:00
Colin Walters 3234295324 pull: Cleanly error when doing local pulls of remote-prefixed refs
In the storage PR I was trying to do a `pull-local` of the whole
`/ostree/repo` on the system, which ended up triggering a `g_critical()`
in the collections code, since we tried to parse a remote-prefixed ref
`fedora:fedora/26/x86_64/atomic-host` as a ref.

I'm not sure offhand what our behavior in this case *should* be.  I
think git only clones local refs, but I need to check.

This corner case arises only with `pull-local`.  But in any case,
while we were previously saying this is programmer error, since it's
so easy to pass various unchecked input into the pull machinery,
make invalid refs an explicit error.

Closes: #992
Approved by: jlebon
2017-07-05 13:37:22 +00:00
Colin Walters 1f5ce1a9f7 lib/repo: Add min-free-space-percent option, default 3%
For ostree-as-host, we're the superuser, so we'll blow past
any reserved free space by default.  While deltas have size
metadata, if one happens to do a loose fetch, we can fill
up the disk.

Another case is flatpak: the system helper has similar concerns
here as ostree-as-host, and for `flatpak --user`, we also
want to be nice and avoid filling up the user's quota.

Closes: https://github.com/ostreedev/ostree/issues/962

Closes: #987
Approved by: jlebon
2017-07-04 16:15:11 +00:00
Colin Walters 8d4d638e99 lib/commit: Use provided length when doing writes
This is prep for storage space checks, where we look at free
space after parsing the metadata, before we write anything.

We did length-limited writes in the fd-based input path, but not for the
`GInputStream` path which in practice is used for HTTP pulls.

Closes: #987
Approved by: jlebon
2017-07-04 16:15:11 +00:00
Colin Walters 3348baf6eb lib/commit: Ensure bare-user objects are always user-readable
Some of the Jenkins jobs for Fedora Atomic Host broke after updating
to 2017.7, and it turns out that we regressed handling unreadable
files in `bare-user` mode.  An example of this is `/etc/shadow`, which
ends up in the ostree-as-host content as `/usr/etc/shadow`.

Now there are better fixes here; we should probably delete it and create it
during the config merge if it doesn't exist.  In general, having secret files in
ostree really isn't supported, so it doesn't make sense to include them.

But let's fix this regression - when operating as an unprivileged user we don't
have `CAP_DAC_OVERRIDE` and hence will fail to open un-user-readable objects.

(We still preserve the actual `0` mode of course in the xattr and will
 apply it in `bare`)

Closes: #989
Approved by: jlebon
2017-06-30 21:23:48 +00:00
Colin Walters 192e7b888f lib/commit: Fix a tmpfile fd leak in static delta processing
I had thought `glnx_link_tmpfile_at()` actually consumed the tmpfile;
it does consume the *path* but not the fd.  In the non-delta path
things were fine since we used the autocleanup.

But the delta code had a tmpfile allocated in its struct that got reused, and
hence leaked the fd. Fix this by making the commit API actually consume the
tmpfile fully, just like the path path.

Closes: #986
Approved by: jlebon
2017-06-30 19:48:05 +00:00
Colin Walters 4273e670ea Add "pull --localcache-repo"
This is a lot like `git clone --reference`, but we chose "localcache" as the
term "reference" is already used.

The main use case I'm targeting this for is the Fedora Atomic Host installer
case where we embed the repo content in the installer, but we may want to
kickstart and download newer content. There, while we want to get a newer ref,
we can still use the local repo as an object cache, since we have it sitting
there in memory anyways.

Another case is where one has a host ostree (say e.g. Fedora Atomic
Workstation), and one wants to create a local archive mirror of FAH. Then one
can use `pull --reference /ostree/repo` and pull the common objects (e.g.
contents of `bash.rpm` etc.)

Closes: https://github.com/ostreedev/ostree/issues/975

Closes: #982
Approved by: jlebon
2017-06-30 19:37:23 +00:00
Colin Walters 1782a1c279 lib/pull: Move check for requested content earlier
This is prep for a later patch; currently the logic is unchanged, but we'll need
this if we make local imports async.

Closes: #982
Approved by: jlebon
2017-06-30 19:37:23 +00:00
Jonathan Lebon d5dd576d20 pull: fix GLNX_HASH_TABLE_FOREACH_KV regressions
These are regression from #971. We were stuffing a pointer size inside a
variable of integer size. So the assignment was spilling over into other
variables' storage space. Actually use a gpointer and GPOINTER_TO_[U]INT
as was done originally.

Also bump libglnx which has static checks for this error in the future.

Update submodule: libglnx

Closes: #990
Approved by: cgwalters
2017-06-30 16:26:53 +00:00
Philip Withnall acace571ef lib/repo: Fix repo-finder deleting remote configs when run
An inverted condition in _ostree_repo_add_remote() was causing the
OstreeRepoFinder to delete precisely the wrong remote
configurations from memory once it was finished. It’s supposed to delete
the ones which it transiently added; but was instead deleting all the
existing remote configurations.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #985
Approved by: cgwalters
2017-06-29 23:49:25 +00:00
Colin Walters aa26db825f lib/commit: Port a few minor functions to new style
Not sure why these weren't converted before.

Closes: #984
Approved by: jlebon
2017-06-29 22:07:23 +00:00
Colin Walters d57410a7e6 lib: Add a helper to convert struct stat → GFileInfo
It's more natural for a few calling places. Prep for patches to go the other
way, which in turn are prep for adding a commit filter v2 that takes `struct
stat`.

`ot_gfile_type_for_mode()` was only used in this function, so inline it here.

Closes: #974
Approved by: jlebon
2017-06-29 18:17:28 +00:00
Colin Walters 90e0d56332 tree-wide: Replace various uses of `archive-z2` → `archive`
The `-z2` is annoying now since it's really a legacy; we've long
since supported typing `archive`.  Convert the docs fully and
explain that.

Also do some (but not all) of the tests just to encourage newer tests to use
`archive` too.

Closes: #980
Approved by: jlebon
2017-06-29 16:00:13 +00:00
Colin Walters 8d586a9da0 lib/pull: Don't fetch detached metadata twice for local pulls
Obviously very minor, but I noticed this while working on `pull --reference`. If
we have a local repo, we'll have already done a hardlink and copied the detached
metadata too, so there's no reason to request it again via the fetcher path.

Closes: #978
Approved by: jlebon
2017-06-29 15:48:59 +00:00
Colin Walters ea15025c19 lib/pull: Some small style porting
I'd mostly been avoiding this file since there's always patches outstanding, but
these few functions shouldn't conflict much.

Closes: #979
Approved by: jlebon
2017-06-29 15:21:06 +00:00
Colin Walters 250e305f73 lib/repo: Port bareuser-conversion stat to bare load
I noticed this is a simple call that's useful to port to the new internal-only
non-allocating API.

Closes: #977
Approved by: jlebon
2017-06-29 15:07:49 +00:00
Colin Walters 064d7bffef lib/deltas: More porting to new code style
Just noticed some of this while working on the previous tmpfile bits.

Closes: #973
Approved by: jlebon
2017-06-29 14:46:18 +00:00
Colin Walters ab9fef5279 lib/commit: Refactor non-failable size indexing function
It can't throw, so remove the `GError` machinery.

Closes: #973
Approved by: jlebon
2017-06-29 14:46:18 +00:00
Colin Walters 6f2ea23e8a libutil: Add a helper for O_TMPFILE + mmap()
I added `glnx_open_anonymous_tmpfile()`, but then later noticed
that the usage of this was really to be combined with `mmap()`,
and we had two versions of that in the delta code.  Add a helper.

(Bigger picture...how is this different from glibc's "mmap() of /dev/zero"
 approach for large chunks? One advantage is the storage can be "swapped" to
 `/var/tmp`, but still deleted automatically, rather than requiring swap space)

Closes: #973
Approved by: jlebon
2017-06-29 14:46:18 +00:00
Jonathan Lebon 373dc4b66c codebase: start using GLNX_HASH_TABLE_FOREACH macros
Use the new macros introduced recently in libglnx to make iterating over
hash tables cleaner. This is just a start, it does not migrate the whole
tree.

Update submodule: libglnx

Closes: #971
Approved by: cgwalters
2017-06-28 16:37:15 +00:00
Colin Walters ba918e49c5 tree-wide: Misc porting to newer libglnx APIs
- Use the new tmpfile bits
 - `glnx_try_fallocate`
 - `glnx_renameat()`

Depends: https://github.com/GNOME/libglnx/pull/57

Update submodule: libglnx

Closes: #970
Approved by: jlebon
2017-06-28 15:27:56 +00:00
Colin Walters 5776d5dcc0 Port to GLnxTmpfile
There's lots of mechanically replacing `OtTmpFile` with `GLnxTmpfile`;
the biggest changes are in the commit path.  Symlink commits are now
very clearly separated from regular files.  Symlinks are `OtCleanupUnlinkat`,
and regular files are `GLnxTmpfile`.

The commit codepath separates those as `_ostree_repo_commit_path_final()` and
`_ostree_repo_commit_tmpf_final()`. A nice aspect of all of this is that they
both *consume* the temporary on success. This avoids an extra spurious
`unlink()` call.

One of the biggest bits of code motion is in `commit_loose_regfile_object()`,
which no longer needs to care about symlinks. For the most parth though it's
just removing conditionals.

Update submodule: libglnx

Closes: #958
Approved by: jlebon
2017-06-27 22:02:14 +00:00
Philip Withnall a03f0447cb lib/pull: Use ostree_repo_verify_summary() to verify summary on pull
Rather than duplicating the code. This introduces no functional changes.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #961
Approved by: cgwalters
2017-06-27 19:19:32 +00:00
Philip Withnall 4c238c350d lib/pull: Check whether summary is in normal form when loading it
Check that it’s in normal form and has the correct type when loading it,
since it could come from an untrusted source.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #961
Approved by: cgwalters
2017-06-27 19:19:32 +00:00
Philip Withnall eb4887d619 lib/pull: Don’t cache summary file until its signature is verified
This makes no difference to the validity of the code, since any summary
file loaded from the cache will be verified before being read anyway;
but it will make some upcoming changes a little simpler.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #961
Approved by: cgwalters
2017-06-27 19:19:32 +00:00
Philip Withnall 2f297ba4d3 lib/repo: Fix a typo in a documentation comment
Looks like a copy-paste error.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #961
Approved by: cgwalters
2017-06-27 19:19:32 +00:00
Colin Walters 4796cd0454 pull: Check free space when pulling deltas
Computing download/storage size for `archive` pulls is hard; there's
`OSTREE_REPO_COMMIT_MODIFIER_FLAGS_GENERATE_SIZES` which was from a
pre-static-deltas effort by Endless, but we aren't currently making use of this
much.

Static deltas were designed to solve this problem; we have the total
uncompressed size. Let's check free space before doing a delta pull.

Related: https://github.com/ostreedev/ostree/issues/962

Closes: #963
Approved by: jlebon
2017-06-27 18:42:00 +00:00
Colin Walters c2b6afe5b9 lib/deltas: Some style porting
Just a few functions to keep up momentum.

Closes: #964
Approved by: jlebon
2017-06-27 17:19:21 +00:00
Colin Walters 7871bc3051 lib/ref: Suppress more collection ref methods from introspection
This squashes some warnings.

Closes: #965
Approved by: pwithnall
2017-06-27 11:56:44 +00:00
Colin Walters 5effceeba8 lib/commit: Fix fallocate size for bare-user symlinks
We need to account for the trailing NUL.

Closes: #957
Approved by: jlebon
2017-06-26 17:17:32 +00:00
Colin Walters af3a96755b lib: Use OtTmpFile for static delta processing
The `OstreeRepoContentBareCommit` struct was basically an `OtTmpFile`, so let's
make it one. I moved the "convert to `GOutputStream`" logic into the callers,
since that bit can't fail; it makes the implementation much simpler since we can
just return the result of `ot_open_tmpfile_linkable_at()`.

Prep for `GLnxTmpfile` porting.

Closes: #957
Approved by: jlebon
2017-06-26 17:17:32 +00:00
Colin Walters 4dee1984dc lib: Hoist unlinkat() cleanup API to fsutil, use in pull
The pull code also could make use of this in both the metadata and content
paths. I changed it to own the tempfile malloc (just like `GLnxTmpFile`), since
there's no reason to have different lifetimes for the filename and the file, and
that way we only have one variable rather than two.

The content path turns out to be a special case though, where
at least for mirroring archives, we directly pass the file *path*
down into `_ostree_repo_commit_loose_final()`.

This is prep for `GLnxTmpFile` porting.

Closes: #957
Approved by: jlebon
2017-06-26 17:17:32 +00:00
Colin Walters 1147267e4d lib/commit: Clean up commit file type handling variables
The variables here were duplicative; we don't need two booleans to distinguish
between symlinks and regular files. What we do need to handle is the "physical"
state versus the "object" state. Symlinks objects are stored as regular files in
`bare-user` and `archive`.

Prep for more cleanup.

Closes: #957
Approved by: jlebon
2017-06-26 17:17:32 +00:00
Philip Withnall d0d5f54ce2 lib/refs: Add runtime error checking for collection ID validity
Instead of treating it as a programming error — given that it’s user
input, that’s not really appropriate. This modifies write_ref() and
list_collection_refs() to implement validation.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall e3d4eeacbc lib/repo-finder: Add Avahi based OstreeRepoFinder implementation
This is a more complex implementation of OstreeRepoFinder which resolves
ref names to remote URIs by looking for refs advertised by peers on the
local network using DNS-SD records and mDNS (Avahi). The idea is to
allow OS and app updates to be propagated over local networks, without
the internet.

It requires an OSTree server and code to generate the DNS-SD adverts in
order to be fully functional — support for this will be added
separately.

Unit tests are included.

Includes fixes by Krzesimir Nowak <krzesimir@kinvolk.io>.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall 7ee4e1295a lib/bloom: Add an internal bloom filter implementation
This will be used in an upcoming commit. It adds a basic bloom filter
implementation, using the SipHash family of hash functions.

The implementation (including its parameter choices and hash functions)
will become a protocol detail in future, so must not be changed so that
its output is bitwise incompatible between OSTree versions.

Unit tests are included.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall ae335f24dc lib/repo-finder: Add mount based OstreeRepoFinder implementation
This is a basic implementation of OstreeRepoFinder which resolves ref
names to remote URIs by looking for them on any currently mounted
removable storage volumes. The idea is to support OS and app updates via
USB stick.

Unit tests are included.

This bumps libostree’s maximum GLib dependency from 2.44 to 2.50 for
g_drive_is_removable(). If GLib 2.50 is not available, the call which
needs it will be omitted and the OstreeRepoFinderMount implementation
will scan all volumes (not just removable ones); this is a performance
hit, but not a functionality hit.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall d15f83c922 lib/repo-finder: Add config-file based OstreeRepoFinder implementation
This is a basic implementation of OstreeRepoFinder which resolves ref
names to remote URIs by looking their collection IDs up in the local
configuration of remotes who have their collection-id key set.

Unit tests are included.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall 292230301d lib/repo-finder: Add basic support for finding remote URIs by ref name
Add an initial OstreeRepoFinder interface (but no implementations),
which will find remote URIs by ref names and collection IDs, the
combination of which is globally unique.

The new API is used in a new ostree_repo_find_updates() function, which
resolves a list of ref names to update into a set of remote URIs to pull
them from, which can be treated as mirrors. It is an attempt to
generalise resolution of the URIs to pull from, and to generalise
determination of the order and parallelisation which they should be
downloaded from in.

Includes fixes by Krzesimir Nowak <krzesimir@kinvolk.io>.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall 7607d94713 lib/pull: Add collection support to ostree_repo_pull_with_options()
This adds a new collection-refs option of type a(sss), giving a list of
(collection ID, ref name, checksum) tuples to pull from the given remote.
This option is intended to supersede the refs and override-commit-ids
options, so is mutually exclusive with them.

This includes support for resolving the refs from the remote’s summary
file, or from its refs/heads and refs/mirrors directories.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall fbf8df8829 lib/refs: Add methods for setting/listing collection–refs
These are tuples of (collection ID, ref name) which are a globally-unique
form of local ref. They use OstreeCollectionRef as an identifier, and hence
need to be accessed using new API, as the existing API uses string
identifiers and sometimes accepts refspecs. Remote names are not
supported as part an OstreeCollectionRef.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall 4de736fdfa lib/repo: Add collection ID support to OstreeRepo
Add {get,set}_collection_id() methods to OstreeRepo and some documentation
about the concept of a collection ID which globally identifies an
upstream repository. See the documentation for more details.

This will be used in future commits. For now, the new API is marked as
experimental (--enable-experimental-api).

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall 0a20e7d43c lib/ref: Add OstreeCollectionRef type for globally unique refs
This is a type representing the tuple (collection ID, ref name), which is
guaranteed to be globally unique. It will be used in upcoming commits.

It introduces the concept of a ‘collection’ which is a unique, curated
set of refs which lie in the same trust domain (i.e. all signed by the
same key and validated by the same developer). Flathub might be a
collection, for example; or the set of OS refs coming from a particular
OS vendor.

It includes a function for validating collection IDs.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Philip Withnall bf1f8eb0fa lib/repo: Split out ref handling from regenerate_summary()
This will make some future additions to regenerate_summary() easier.
This commit introduces no functional changes.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #924
Approved by: cgwalters
2017-06-26 15:56:07 +00:00
Colin Walters da0791f484 tests: add a syntax-check rule for glnx_prefix_error()
Same as the errno variant; the colon-space `: ` thing got me in a different
patch.

Closes: #956
Approved by: jlebon
2017-06-26 15:09:12 +00:00
Colin Walters e23071dc39 lib/deltas: Port to more to new code style
Looking at the uses of `ostree_repo_load_file()` here.

Closes: #953
Approved by: jlebon
2017-06-26 14:42:56 +00:00
Colin Walters 612c8a5fa8 lib/repo: More cleanup of load_file() internals
This is followon work from previous cleanups.  Basically
`stat_bare_content_object()` was the `fstatat()` logic
and `ostree_repo_read_bare_fd()` was the `openat()` implementation;
they duplicated some bits to find the object in staging, recurse
into parent etc.

Further, I wanted an internal-only version of this API which didn't allocate
`GFileInfo`/`GInputStream` but used a plain `fd` and `struct stat` to avoid
mallocs.

The end version here I think looks a lot nicer, since we deduplicate the various
`open()` calls in the different cases for example.

Closes: #952
Approved by: jlebon
2017-06-23 18:29:51 +00:00
Colin Walters 63ad289a9c lib/repo: Split archive/bare file parsing
Prep for future cleanup patches (in particular I want an internal-only
version at first that uses a fd+`struct stat`) to avoid allocations.

The new version avoids lots of deep nesting of conditionals as well
by hoisting the "not found" handling to an early return.

There's a bit of code duplication between the two cases but it's
quite worth the result.

Closes: #951
Approved by: jlebon
2017-06-23 14:11:36 +00:00
Colin Walters aafda9073a lib/core: Avoid NULL deref in content_file_parse() if out variable unset
Prep for a change in `ostree_repo_load_file()`.  We would crash if a
caller had `out_file_info = NULL`, because we deref `ret_file_info`
below it.

Closes: #951
Approved by: jlebon
2017-06-23 14:11:36 +00:00
Philip Withnall 9a79d13ce3 lib/remote: Fix ‘Since’ line for OstreeRemote
There was a typo in it when it was first introduced. Let’s not live in
the past.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #950
Approved by: cgwalters
2017-06-21 13:00:55 +00:00
Philip Withnall 20829a0582 lib/sym: Fix symbol versions for 2017.7 experimental symbols
There was a typo in the group name. It should be OK to change the
version since this is all hidden behind the --enable-experimental-api
configure option.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #950
Approved by: cgwalters
2017-06-21 13:00:55 +00:00
Philip Withnall 333d264c5b lib/core: Fix ‘Since’ line for ostree_validate_remote_name()
This was missed when cherry-picking it out of
https://github.com/ostreedev/ostree/pull/924#discussion_r123097919.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #950
Approved by: cgwalters
2017-06-21 13:00:55 +00:00
Colin Walters 46e7f08daa repo: Squash a gcc `-Wmaybe-uninitialized` warning
It's spurious, but unfortunately GCC doesn't currently understand that it will
always be set.

Closes: #943
Approved by: jlebon
2017-06-20 22:27:18 +00:00
Philip Withnall 20dc9454b3 lib/core: Add ostree_validate_remote_name() for remote names
There are a few places in the code where ad-hoc validation was being
performed. Might as well formalise it a bit more.

Signed-off-by: Philip Withnall <withnall@endlessm.com>

Closes: #948
Approved by: cgwalters
2017-06-20 21:52:22 +00:00
Colin Walters b69c6ba992 build-sys: Post-release version bump
Closes: #944
Approved by: jlebon
2017-06-19 15:51:27 +00:00
Colin Walters 6729b7c264 Release 2017.7
Closes: #942
Approved by: jlebon
2017-06-19 14:54:59 +00:00
David Shea 22e753176e lib/repo: Fix annotations for out parameters
Change the annotation of the out parameters on ostree_repo_load_file
from `(allow-none)` to `(optional) (nullable)`. `allow-none` is
ambiguous, since these parameters can be both NULL on input and set to
NULL on return.

Closes: #939
Approved by: cgwalters
2017-06-19 13:15:19 +00:00
Alexander Larsson 07dc33ca4a static delta apply: Work on bare-user-only repos
Flatpak make check is failing when applying a static delta
to a bare-user-only repo due to an assert. The fix is to add
bare-user-only to the assert check.

Closes: #940
Approved by: giuseppe
2017-06-19 09:09:17 +00:00
Alexander Larsson 73ba3eb686 pull: When mirroring, only replace summary if we're doing a full mirror
We're hitting this in flathub, where we have a bunch of local builds,
but we also mirror a few refs from the gnome runtime repo into it.
Its fixable by re-doing the summary, but for a short time the
wrong version is visible.

Fixes https://github.com/ostreedev/ostree/issues/846

Closes: #935
Approved by: cgwalters
2017-06-16 15:08:39 +00:00
Colin Walters 371b4a5e7e checkout: Fix SELinux policy labeling when recursing
The code here tried to truncate the string to the previous length,
but that doesn't work when recursing, since further calls change the
length.

What actually ended up happening was the string would get corrupted
after the first level of recursion.

Closes: #936
Approved by: jlebon
2017-06-16 14:54:29 +00:00
Colin Walters 2bab43fb22 lib: Split symbol versioning into -released and -devel
So far a lot of submitted PR have added symbols into the first
section.  Split the file into `-released` and `-devel` to make
this more obvious.

To further enforce things, we hardcode a checksum of the `-released`
file in `test-symbols.sh`.  Only release commits should update that
checksum.

Did you notice I like checksums?

Closes: #931
Approved by: pwithnall
2017-06-16 09:31:16 +00:00
Tristan Van Berkom 3e3a0f0766 ostreee-version.h.in: Added Since: version annotations
This is especially interesting for the versioning symbols themselves,
as it is an indicator of when applications using introspection information
can start to use a symbol in the library to check if they have a
recent enough version of OSTree to use.

Closes: #932
Approved by: cgwalters
2017-06-15 12:59:59 +00:00
Colin Walters 9529e8d435 lib/pull: Extend BAREUSERONLY_FILES flag to HTTP requests
For the flatpak PR: https://github.com/flatpak/flatpak/pull/849

It's really more convenient if this works for HTTP pulls as well, since flatpak
does various types of pulling, and we can just set the flag everywhere.

Further, we might as well reject the content as early as possible.

Closes: #930
Approved by: alexlarsson
2017-06-15 07:10:22 +00:00
Colin Walters 0e6d23835b lib/sysroot: Add some g_prefix_error() for ostree_sysroot_cleanup()
We saw this fail in a CI run. We've been trying to add strategic error prefixing
as a general rule, and this specific instance may help debug.

Closes: #929
Approved by: pwithnall
2017-06-15 00:08:29 +00:00
Anton Gerasimov 64ab8334b7 lib/sysroot: Add API to get pending/rollback for given stateroot
This imports a function that is used in rpm-ostree, and it's also intended for
use by https://github.com/advancedtelematic/aktualizr to display
what deployment we're going to boot next after the reboot.

Updated-by: Colin Walters <walters@verbum.org>

Closes: #897
Approved by: OYTIS
2017-06-14 09:56:01 +00:00
Colin Walters 0635fcbfd9 lib/checkout: Add bareuseronly_dirs option
This is a continuation of https://github.com/ostreedev/ostree/pull/926
for directories instead of files.

See: https://github.com/flatpak/flatpak/issues/845

This option suppresses mode bits outside of `0775` for directory
checkouts.  I think most people should start doing this by default,
and use explicit overrides for e.g. `/tmp` if doing a recommit based
on a checkout.

Closes: #927
Approved by: alexlarsson
2017-06-13 20:05:31 +00:00