Commit Graph
100 Commits
Author SHA1 Message Date
Colin Walters 6430207e47 lib: Add #defines for current well-known metadata keys
This came up in https://github.com/projectatomic/rpm-ostree/issues/142

Let's add `#define`s for our metadata keys, with documentation so
that, well, they're documented.

Closes: #1024
Approved by: peterbaouoft
2017-07-21 14:53:33 +00:00
Colin Walters cf6c15a6c5 ci/papr: Add a suite to run introspection-based tests without ASAN
Unfortunately we can't do gobject-introspection based tests
while compiling with `-fsanitize=address`, since it needs to hook
`malloc` early on.

Add a new suite which just runs the introspection-based tests without ASAN.

Closes: #1016
Approved by: jlebon
2017-07-20 14:01:11 +00:00
Colin Walters f9f7d55e79 lib/commit: Fix EBADF with GENERATE_SIZES option for commit
Regression from previous tmpfile refactoring; unfortunately
the `OSTREE_REPO_COMMIT_MODIFIER_FLAGS_GENERATE_SIZES` option
only has coverage via gjs currently.

Might expose it via the cmdline in a later option, but in the big picture the
idea was that this data is better kept in static deltas.

Closes: https://github.com/ostreedev/ostree/issues/1014

Closes: #1016
Approved by: jlebon
2017-07-20 14:01:11 +00:00
Colin Walters ebbd0b3ce6 ci/papr: Switch primary to libcurl, add libsoup context
Sometime in the next few releases I think we should make libcurl
the default.

Prep for more CI work.

Closes: #1016
Approved by: jlebon
2017-07-20 14:01:11 +00:00
Colin Walters 779f125cbe lib/repo: Auto-recreate repo/tmp if it's deleted
We can accumulate a lot of space there; let's be nice to people who delete the
whole directory.

Closes: https://github.com/ostreedev/ostree/issues/1018

Closes: #1020
Approved by: jlebon
2017-07-19 15:01:19 +00:00
Colin Walters c740b7f6d2 core: Sanitize error text validating refs (e.g. against HTML)
See: https://github.com/projectatomic/rpm-ostree/issues/885

If we get a successful Apache directory listing HTML when fetching what we
intend to be a ref, we'd dump the HTML into the error.

I did some scanning of the pull code, and this was the only case
I saw offhand where we were dumping text out into an error.  Which
makes sense, since most of our formats are binary, the exeptions I
think are just `repo/config` and `repo/refs/`.

Closes: #1015
Approved by: mbarnes
2017-07-19 14:45:57 +00:00
Colin Walters 2a9689b76a Update libglnx, port various bits to new API
Using the error prefixing in the delta processing allows us to
do new code style.  Also strip trailing whitespace.

Use error prefixing in a few other random places.  I didn't
hunt for all of them, just testing out the new API.

Use `glnx_fchmod()`.  Also note I dropped one `fchmod (tmpf, 0600)`
which is no longer necessary.

Update submodule: libglnx

Closes: #1011
Approved by: jlebon
2017-07-18 19:18:38 +00:00
Colin Walters efd460782a lib/pull: Drop direct use of ->repodir
Prep for `ostree_repo_new_at()`.  Down the line perhaps
we should extend libcurl to accept a file descriptor for cookies,
but this works OK for now.

Closes: #1010
Approved by: jlebon
2017-07-18 19:07:56 +00:00
Colin Walters 9430b8ad75 bin/cookies: Drop libsoup code, fix fd-relative issues, new style
Prep for `ostree_repo_new_at()`. These commands were directly accessing
`repo->repodir`, which it turns out was unnecessary since the the APIs they then
used were fd-relative. Except actually there were bugs there, so fix all of the
cookie util code to actually use the passed `dfd` and not just hardcode
`AT_FDCWD`.

Also, libsoup can't handle this (its APIs require fully qualifed paths), and
there's not a really good reason to have two implementations now; historically
it was useful to cross-check them, but I don't think we need that.

While I'm here, port to new style.

Closes: #1010
Approved by: jlebon
2017-07-18 19:07:56 +00:00
Colin Walters e0346c1494 Add a notion of "physical" sysroot, use for remote writing
(Note this PR was reverted in <https://github.com/ostreedev/ostree/pull/902>;
 this version should be better)

Using `${sysroot}` to mean the physical storage root: We don't want to write to
`${sysroot}/etc/ostree/remotes.d`, since nothing will read it, and really
`${sysroot}` should just have `/ostree` (ideally). Today the Anaconda rpmostree
code ends up writing there. Fix this by adding a notion of "physical" sysroot.
We determine whether the path is physical by checking for `/sysroot`, which
exists in deployment roots (and there shouldn't be a `${sysroot}/sysroot`).

In order to unit test this, I added a `--sysroot` argument to `remote add`.
However, doing this better would require reworking the command line parsing for
the `remote` argument to support specifying `--repo` or `--sysroot`, and I
didn't quite want to do that yet in this patch.

This second iteration of this patch fixes the bug we hit the first time;
embarassingly enough I broke `ostree remote list` finding system remotes.
The fix is to have `ostree_repo_open()` figure out whether it's the same
as `/ostree/repo` for now.

Down the line...we might consider having the `ostree remote` command line itself
instatiate an `OstreeSysroot` by default, but this maximizes compatibility; we
just have to pay a small cost that `ostree` usage outside of that case like
`ostree static-delta` in a releng Jenkins job or whatever will do this `stat()`
too.

Closes: https://github.com/ostreedev/ostree/issues/892

Closes: #1008
Approved by: mbarnes
2017-07-18 18:58:06 +00:00
Colin Walters 8b1f1c4428 lib/pull: Do local content imports async too
This came up in <https://github.com/ostreedev/ostree/pull/982>; when
we added more direct local importing, we did it synchronously.

This was actually quite a regression when doing local pulls between different
modes; in particular between a bare mode and `archive`, as we were suddenly
doing gzip {de,}compression in the main thread.

Down the line actually...a simpler fix is probably to change things so that the
local path is really only used when we know we can hardlink; everything else
would go though the fetcher codepath but with `file://`.

But this isn't a lot more code, and the speed/interactivity win is large.

Note we're only doing content async with this patch. We could do metadata as
well; we have the object already local. But the metadata code path is messier,
and metadata objects are smaller.

Another area where this comes up is that in e.g. Fedora releng, most operations
talk to a NetApp via NFS. So this has the classic network filesystem problem
that operations that are normally cheap like `stat()` can actually have
nontrivial latency. Doing as much as possible in threads is better there too.

Closes: #1006
Approved by: jlebon
2017-07-18 17:03:13 +00:00
Colin Walters 96e49a67f9 ci/papr: Update to F26
In particular F25AH will stop getting updates.

Closes: #1012
Approved by: jlebon
2017-07-18 14:01:54 +00:00
Colin Walters 620a90ebfa lib/pull: Avoid journaling 404s for optional content
Currently in Fedora we don't sign summaries, and every use of
`rpm-ostree` would emit to the journal an error when we failed
to fetch it.

Fix this by having `OSTREE_FETCHER_REQUEST_OPTIONAL_CONTENT` tell the fetcher
not to journal 404 errors. While fixing this, we had a mix of two booleans vs
the flags; fix things so we consistently use the flags in the fetcher and pull
code.

Closes: #1004
Approved by: mbarnes
2017-07-13 21:41:00 +00:00
Colin Walters 0aa20df20e tests: Run pull tests for bare/bare-user
We have variants of `test-basic` for all 4 modes, but not for pull-test, which
for some reason was named `pull-archive`, but mostly pulls *into* bare repos.
The test code was structured like the basic one where it called into a
`pull-test.sh`, so let's actually use it for 2/3 bare modes. (I tried to extend
it to `bare-user-only` but it failed, going to look at that after this).

This is related to https://github.com/ostreedev/ostree/issues/991

Closes: #998
Approved by: jlebon
2017-07-10 14:18:18 +00:00
Colin Walters 9d941dcebb checkout: Don't set dir mtime to 0 when doing a force copy checkout
When we [switched to using checkout + force_copy](https://github.com/ostreedev/ostree/commit/e8efd1c8dcaad8fbd3b05c400972d237406263e7),
a side effect that went unnoticed at the time is that we started
setting directory mtimes to zero.

See the below bug where we long ago set the file times to zero, which got fixed,
so let's not regress things by setting the directory times to zero either. (Even
though AFAICS GNU tar doesn't complain about those)

This semantic is somewhat "overloaded" onto `force_copy`, but it avoids adding
yet another boolean; we don't have that many reserved boolean slots left. I
can't really think of many good use cases for `force_copy` *other* than the
`/etc` merge anyways.

https://bugzilla.redhat.com/show_bug.cgi?id=1229160

Closes: https://github.com/ostreedev/ostree/issues/995

Closes: #997
Approved by: jlebon
2017-07-07 15:01:51 +00:00
Colin Walters d2a05e5a09 deploy: Port some functions to new style
There are a number of simple ports here.  Prep for further work
in `/etc` merge.

I also stripped trailing whitespace globally.

Closes: #996
Approved by: jlebon
2017-07-07 14:39:47 +00:00
Colin Walters 28ec43c41a build-sys: Post-release version bump
Closes: #994
Approved by: jlebon
2017-07-06 14:45:02 +00:00
Colin Walters 5a5e465492 Release 2017.8
Closes: #994
Approved by: jlebon
2017-07-06 14:45:02 +00:00
Colin Walters 23b93a3eb6 lib/repo: Immediately error creating bare-user repo on tmpfs
And in general, if for some reason we can't write `user.` xattrs, provide an
error immediately rather than doing it during a later pull. This way the failure
cause is a lot more obvious.

Related: https://github.com/ostreedev/ostree/issues/991

Closes: #993
Approved by: jlebon
2017-07-06 14:31:37 +00:00
Colin Walters 3234295324 pull: Cleanly error when doing local pulls of remote-prefixed refs
In the storage PR I was trying to do a `pull-local` of the whole
`/ostree/repo` on the system, which ended up triggering a `g_critical()`
in the collections code, since we tried to parse a remote-prefixed ref
`fedora:fedora/26/x86_64/atomic-host` as a ref.

I'm not sure offhand what our behavior in this case *should* be.  I
think git only clones local refs, but I need to check.

This corner case arises only with `pull-local`.  But in any case,
while we were previously saying this is programmer error, since it's
so easy to pass various unchecked input into the pull machinery,
make invalid refs an explicit error.

Closes: #992
Approved by: jlebon
2017-07-05 13:37:22 +00:00
Colin Walters 1f5ce1a9f7 lib/repo: Add min-free-space-percent option, default 3%
For ostree-as-host, we're the superuser, so we'll blow past
any reserved free space by default.  While deltas have size
metadata, if one happens to do a loose fetch, we can fill
up the disk.

Another case is flatpak: the system helper has similar concerns
here as ostree-as-host, and for `flatpak --user`, we also
want to be nice and avoid filling up the user's quota.

Closes: https://github.com/ostreedev/ostree/issues/962

Closes: #987
Approved by: jlebon
2017-07-04 16:15:11 +00:00
Colin Walters 8d4d638e99 lib/commit: Use provided length when doing writes
This is prep for storage space checks, where we look at free
space after parsing the metadata, before we write anything.

We did length-limited writes in the fd-based input path, but not for the
`GInputStream` path which in practice is used for HTTP pulls.

Closes: #987
Approved by: jlebon
2017-07-04 16:15:11 +00:00
Colin Walters 3348baf6eb lib/commit: Ensure bare-user objects are always user-readable
Some of the Jenkins jobs for Fedora Atomic Host broke after updating
to 2017.7, and it turns out that we regressed handling unreadable
files in `bare-user` mode.  An example of this is `/etc/shadow`, which
ends up in the ostree-as-host content as `/usr/etc/shadow`.

Now there are better fixes here; we should probably delete it and create it
during the config merge if it doesn't exist.  In general, having secret files in
ostree really isn't supported, so it doesn't make sense to include them.

But let's fix this regression - when operating as an unprivileged user we don't
have `CAP_DAC_OVERRIDE` and hence will fail to open un-user-readable objects.

(We still preserve the actual `0` mode of course in the xattr and will
 apply it in `bare`)

Closes: #989
Approved by: jlebon
2017-06-30 21:23:48 +00:00
Colin Walters cd7d35945a bin/commit: Add '=' to --statoverride
Previously, we only supported additions in the statoverride file;
it was mainly for adding the setuid bit without having that physically
on disk.

However, for testing a change to `bare-user` handling around *unreadable*
files (which happens for `/etc/shadow` in host content), I need a way
to write that into a repo in the test suite.

I'm not actually aware of a non-test-suite use case for this; a more
sophisticated user is going to be using the API directly, which can already do
this. But we need it for tests at least.

Closes: #989
Approved by: jlebon
2017-06-30 21:23:48 +00:00
Colin Walters e3a540a606 bin/commit: Port helper functions to new style
Prep for more work here.  Can't yet port the main function
without a cleanup for transactions.

Closes: #988
Approved by: jlebon
2017-06-30 20:28:48 +00:00
Colin Walters 192e7b888f lib/commit: Fix a tmpfile fd leak in static delta processing
I had thought `glnx_link_tmpfile_at()` actually consumed the tmpfile;
it does consume the *path* but not the fd.  In the non-delta path
things were fine since we used the autocleanup.

But the delta code had a tmpfile allocated in its struct that got reused, and
hence leaked the fd. Fix this by making the commit API actually consume the
tmpfile fully, just like the path path.

Closes: #986
Approved by: jlebon
2017-06-30 19:48:05 +00:00
Colin Walters 4273e670ea Add "pull --localcache-repo"
This is a lot like `git clone --reference`, but we chose "localcache" as the
term "reference" is already used.

The main use case I'm targeting this for is the Fedora Atomic Host installer
case where we embed the repo content in the installer, but we may want to
kickstart and download newer content. There, while we want to get a newer ref,
we can still use the local repo as an object cache, since we have it sitting
there in memory anyways.

Another case is where one has a host ostree (say e.g. Fedora Atomic
Workstation), and one wants to create a local archive mirror of FAH. Then one
can use `pull --reference /ostree/repo` and pull the common objects (e.g.
contents of `bash.rpm` etc.)

Closes: https://github.com/ostreedev/ostree/issues/975

Closes: #982
Approved by: jlebon
2017-06-30 19:37:23 +00:00
Colin Walters 1782a1c279 lib/pull: Move check for requested content earlier
This is prep for a later patch; currently the logic is unchanged, but we'll need
this if we make local imports async.

Closes: #982
Approved by: jlebon
2017-06-30 19:37:23 +00:00
Colin Walters aa26db825f lib/commit: Port a few minor functions to new style
Not sure why these weren't converted before.

Closes: #984
Approved by: jlebon
2017-06-29 22:07:23 +00:00
Colin Walters d57410a7e6 lib: Add a helper to convert struct stat → GFileInfo
It's more natural for a few calling places. Prep for patches to go the other
way, which in turn are prep for adding a commit filter v2 that takes `struct
stat`.

`ot_gfile_type_for_mode()` was only used in this function, so inline it here.

Closes: #974
Approved by: jlebon
2017-06-29 18:17:28 +00:00
Colin Walters 1a9a473580 cmdline/pull: Print final status even if noninteractive
Previously, `ostree pull` was silent if not on a tty.  I don't
see a reason not to print the final status line at least.  This
is prep for more work in the test suite, so I can write assertions
on the output.

But it should also be nicer for people who e.g. do an `ostree pull` in a Jenkins
job or whatever.

Closes: #981
Approved by: jlebon
2017-06-29 16:16:25 +00:00
Colin Walters 90e0d56332 tree-wide: Replace various uses of archive-z2 → archive
The `-z2` is annoying now since it's really a legacy; we've long
since supported typing `archive`.  Convert the docs fully and
explain that.

Also do some (but not all) of the tests just to encourage newer tests to use
`archive` too.

Closes: #980
Approved by: jlebon
2017-06-29 16:00:13 +00:00
Colin Walters 8d586a9da0 lib/pull: Don't fetch detached metadata twice for local pulls
Obviously very minor, but I noticed this while working on `pull --reference`. If
we have a local repo, we'll have already done a hardlink and copied the detached
metadata too, so there's no reason to request it again via the fetcher path.

Closes: #978
Approved by: jlebon
2017-06-29 15:48:59 +00:00
Colin Walters 2013db0527 tests: Fix assert_files_hardlinked
It was always succeeding because we were trying to stat the inode number, and
failing, and thus getting the empty string for both, which compared as true.

Regression from:
<https://github.com/ostreedev/ostree/commit/74e3581e>

Noticed this while working on
<https://github.com/ostreedev/ostree/pull/974>
and looking at the test results.

Closes: #976
Approved by: jlebon
2017-06-29 15:34:57 +00:00
Colin Walters ea15025c19 lib/pull: Some small style porting
I'd mostly been avoiding this file since there's always patches outstanding, but
these few functions shouldn't conflict much.

Closes: #979
Approved by: jlebon
2017-06-29 15:21:06 +00:00
Colin Walters 250e305f73 lib/repo: Port bareuser-conversion stat to bare load
I noticed this is a simple call that's useful to port to the new internal-only
non-allocating API.

Closes: #977
Approved by: jlebon
2017-06-29 15:07:49 +00:00
Colin Walters 064d7bffef lib/deltas: More porting to new code style
Just noticed some of this while working on the previous tmpfile bits.

Closes: #973
Approved by: jlebon
2017-06-29 14:46:18 +00:00
Colin Walters ab9fef5279 lib/commit: Refactor non-failable size indexing function
It can't throw, so remove the `GError` machinery.

Closes: #973
Approved by: jlebon
2017-06-29 14:46:18 +00:00
Colin Walters 6f2ea23e8a libutil: Add a helper for O_TMPFILE + mmap()
I added `glnx_open_anonymous_tmpfile()`, but then later noticed
that the usage of this was really to be combined with `mmap()`,
and we had two versions of that in the delta code.  Add a helper.

(Bigger picture...how is this different from glibc's "mmap() of /dev/zero"
 approach for large chunks? One advantage is the storage can be "swapped" to
 `/var/tmp`, but still deleted automatically, rather than requiring swap space)

Closes: #973
Approved by: jlebon
2017-06-29 14:46:18 +00:00
Colin Walters ba918e49c5 tree-wide: Misc porting to newer libglnx APIs
- Use the new tmpfile bits
 - `glnx_try_fallocate`
 - `glnx_renameat()`

Depends: https://github.com/GNOME/libglnx/pull/57

Update submodule: libglnx

Closes: #970
Approved by: jlebon
2017-06-28 15:27:56 +00:00
Colin Walters 5776d5dcc0 Port to GLnxTmpfile
There's lots of mechanically replacing `OtTmpFile` with `GLnxTmpfile`;
the biggest changes are in the commit path.  Symlink commits are now
very clearly separated from regular files.  Symlinks are `OtCleanupUnlinkat`,
and regular files are `GLnxTmpfile`.

The commit codepath separates those as `_ostree_repo_commit_path_final()` and
`_ostree_repo_commit_tmpf_final()`. A nice aspect of all of this is that they
both *consume* the temporary on success. This avoids an extra spurious
`unlink()` call.

One of the biggest bits of code motion is in `commit_loose_regfile_object()`,
which no longer needs to care about symlinks. For the most parth though it's
just removing conditionals.

Update submodule: libglnx

Closes: #958
Approved by: jlebon
2017-06-27 22:02:14 +00:00
Colin Walters 7d0e6fb63f ci: Actually run installed tests again
We did it *again* 😭

Closes: #969
Approved by: jlebon
2017-06-27 21:21:18 +00:00
Colin Walters 4796cd0454 pull: Check free space when pulling deltas
Computing download/storage size for `archive` pulls is hard; there's
`OSTREE_REPO_COMMIT_MODIFIER_FLAGS_GENERATE_SIZES` which was from a
pre-static-deltas effort by Endless, but we aren't currently making use of this
much.

Static deltas were designed to solve this problem; we have the total
uncompressed size. Let's check free space before doing a delta pull.

Related: https://github.com/ostreedev/ostree/issues/962

Closes: #963
Approved by: jlebon
2017-06-27 18:42:00 +00:00
Colin Walters 54db9ecab3 build: Don't scan ostree-remote.h for introspection if !experimental-api
It's a bit unfortunate that the scanner doesn't error on this, but
anyways

Closes: https://github.com/ostreedev/ostree/issues/966

Closes: #967
Approved by: jlebon
2017-06-27 17:33:08 +00:00
Colin Walters c2b6afe5b9 lib/deltas: Some style porting
Just a few functions to keep up momentum.

Closes: #964
Approved by: jlebon
2017-06-27 17:19:21 +00:00
Colin Walters 21cb4d1715 ci: Make introspection warnings fatal
Closes: #965
Approved by: pwithnall
2017-06-27 11:56:44 +00:00
Colin Walters 7871bc3051 lib/ref: Suppress more collection ref methods from introspection
This squashes some warnings.

Closes: #965
Approved by: pwithnall
2017-06-27 11:56:44 +00:00
Colin Walters 5effceeba8 lib/commit: Fix fallocate size for bare-user symlinks
We need to account for the trailing NUL.

Closes: #957
Approved by: jlebon
2017-06-26 17:17:32 +00:00
Colin Walters af3a96755b lib: Use OtTmpFile for static delta processing
The `OstreeRepoContentBareCommit` struct was basically an `OtTmpFile`, so let's
make it one. I moved the "convert to `GOutputStream`" logic into the callers,
since that bit can't fail; it makes the implementation much simpler since we can
just return the result of `ot_open_tmpfile_linkable_at()`.

Prep for `GLnxTmpfile` porting.

Closes: #957
Approved by: jlebon
2017-06-26 17:17:32 +00:00
Colin Walters 4dee1984dc lib: Hoist unlinkat() cleanup API to fsutil, use in pull
The pull code also could make use of this in both the metadata and content
paths. I changed it to own the tempfile malloc (just like `GLnxTmpFile`), since
there's no reason to have different lifetimes for the filename and the file, and
that way we only have one variable rather than two.

The content path turns out to be a special case though, where
at least for mirroring archives, we directly pass the file *path*
down into `_ostree_repo_commit_loose_final()`.

This is prep for `GLnxTmpFile` porting.

Closes: #957
Approved by: jlebon
2017-06-26 17:17:32 +00:00
Colin Walters 1147267e4d lib/commit: Clean up commit file type handling variables
The variables here were duplicative; we don't need two booleans to distinguish
between symlinks and regular files. What we do need to handle is the "physical"
state versus the "object" state. Symlinks objects are stored as regular files in
`bare-user` and `archive`.

Prep for more cleanup.

Closes: #957
Approved by: jlebon
2017-06-26 17:17:32 +00:00
Colin Walters da0791f484 tests: add a syntax-check rule for glnx_prefix_error()
Same as the errno variant; the colon-space `: ` thing got me in a different
patch.

Closes: #956
Approved by: jlebon
2017-06-26 15:09:12 +00:00
Colin Walters 553b99642c cmd/fsck: Port to new style
Happened to look at this code too, it's a straightforward port.

Closes: #955
Approved by: jlebon
2017-06-26 14:54:32 +00:00
Colin Walters e23071dc39 lib/deltas: Port to more to new code style
Looking at the uses of `ostree_repo_load_file()` here.

Closes: #953
Approved by: jlebon
2017-06-26 14:42:56 +00:00
Colin Walters 612c8a5fa8 lib/repo: More cleanup of load_file() internals
This is followon work from previous cleanups.  Basically
`stat_bare_content_object()` was the `fstatat()` logic
and `ostree_repo_read_bare_fd()` was the `openat()` implementation;
they duplicated some bits to find the object in staging, recurse
into parent etc.

Further, I wanted an internal-only version of this API which didn't allocate
`GFileInfo`/`GInputStream` but used a plain `fd` and `struct stat` to avoid
mallocs.

The end version here I think looks a lot nicer, since we deduplicate the various
`open()` calls in the different cases for example.

Closes: #952
Approved by: jlebon
2017-06-23 18:29:51 +00:00
Colin Walters 63ad289a9c lib/repo: Split archive/bare file parsing
Prep for future cleanup patches (in particular I want an internal-only
version at first that uses a fd+`struct stat`) to avoid allocations.

The new version avoids lots of deep nesting of conditionals as well
by hoisting the "not found" handling to an early return.

There's a bit of code duplication between the two cases but it's
quite worth the result.

Closes: #951
Approved by: jlebon
2017-06-23 14:11:36 +00:00
Colin Walters aafda9073a lib/core: Avoid NULL deref in content_file_parse() if out variable unset
Prep for a change in `ostree_repo_load_file()`.  We would crash if a
caller had `out_file_info = NULL`, because we deref `ret_file_info`
below it.

Closes: #951
Approved by: jlebon
2017-06-23 14:11:36 +00:00
Colin Walters 027c77c621 ci: Enable -Werror=maybe-uninitialized
We don't want to inject this warning by default for every build
like the other ones in `configure.ac`, since it can be spruriously
wrong.  But there's no reason not to have a more extended set of
warnings for well-known toolchains (e.g. f25).

Closes: #943
Approved by: jlebon
2017-06-20 22:27:18 +00:00
Colin Walters 46e7f08daa repo: Squash a gcc -Wmaybe-uninitialized warning
It's spurious, but unfortunately GCC doesn't currently understand that it will
always be set.

Closes: #943
Approved by: jlebon
2017-06-20 22:27:18 +00:00
Colin Walters b69c6ba992 build-sys: Post-release version bump
Closes: #944
Approved by: jlebon
2017-06-19 15:51:27 +00:00
Colin Walters 2a082b20b8 main: DevelBuild=yes to ostree --version for devel builds
And use it in `test-symbols.sh`, to fix the `distcheck` case;
the previous change stopped distributing `libostree-devel.sym`
in release builds.

Closes: #944
Approved by: jlebon
2017-06-19 15:51:27 +00:00
Colin Walters 6729b7c264 Release 2017.7
Closes: #942
Approved by: jlebon
2017-06-19 14:54:59 +00:00
Colin Walters a45dc0fd0b build-sys: Add "release build" flag, use for symbol versioning
I was trying to do a release and move the symbols from `-devel.sym` into
`-release.sym`, but it turns out that at least GNU binutils `ld` treats an empty
version script as a syntax error.  Fix this by adding a "release build"
flag, and only include `-devel` in non-release builds.

This would also make it easier to inject that flag into our `.pc` and
`ostree-version.h` and `ostree --version` metadata, but I didn't do that yet.

EDIT: Turns out a simpler fix is just to add an empty section. However I kept
this commit since it's a useful sanity check for whether we should include
`-devel.sym` in builds, and we may want to inject the metadata later.

Closes: #942
Approved by: jlebon
2017-06-19 14:54:59 +00:00
Colin Walters fb2c3c1db3 tests: Fix previous commit for selinux testing
I only checked the test passed, I didn't read the output closely, and
made it succeed without testing anything.

Fix the absolute/relative `/etc` references.

Closes: #937
Approved by: jlebon
2017-06-16 15:46:07 +00:00
Colin Walters 371b4a5e7e checkout: Fix SELinux policy labeling when recursing
The code here tried to truncate the string to the previous length,
but that doesn't work when recursing, since further calls change the
length.

What actually ended up happening was the string would get corrupted
after the first level of recursion.

Closes: #936
Approved by: jlebon
2017-06-16 14:54:29 +00:00
Colin Walters 2bab43fb22 lib: Split symbol versioning into -released and -devel
So far a lot of submitted PR have added symbols into the first
section.  Split the file into `-released` and `-devel` to make
this more obvious.

To further enforce things, we hardcode a checksum of the `-released`
file in `test-symbols.sh`.  Only release commits should update that
checksum.

Did you notice I like checksums?

Closes: #931
Approved by: pwithnall
2017-06-16 09:31:16 +00:00
Colin Walters 9529e8d435 lib/pull: Extend BAREUSERONLY_FILES flag to HTTP requests
For the flatpak PR: https://github.com/flatpak/flatpak/pull/849

It's really more convenient if this works for HTTP pulls as well, since flatpak
does various types of pulling, and we can just set the flag everywhere.

Further, we might as well reject the content as early as possible.

Closes: #930
Approved by: alexlarsson
2017-06-15 07:10:22 +00:00
Colin Walters 0e6d23835b lib/sysroot: Add some g_prefix_error() for ostree_sysroot_cleanup()
We saw this fail in a CI run. We've been trying to add strategic error prefixing
as a general rule, and this specific instance may help debug.

Closes: #929
Approved by: pwithnall
2017-06-15 00:08:29 +00:00
Colin Walters 584735b1c9 build-sys: post-release version bump
Per request by flatpak.

Closes: #928
Approved by: alexlarsson
2017-06-14 14:51:25 +00:00
Colin Walters 0635fcbfd9 lib/checkout: Add bareuseronly_dirs option
This is a continuation of https://github.com/ostreedev/ostree/pull/926
for directories instead of files.

See: https://github.com/flatpak/flatpak/issues/845

This option suppresses mode bits outside of `0775` for directory
checkouts.  I think most people should start doing this by default,
and use explicit overrides for e.g. `/tmp` if doing a recommit based
on a checkout.

Closes: #927
Approved by: alexlarsson
2017-06-13 20:05:31 +00:00
Colin Walters 6ed824bf00 lib/pull: Add OSTREE_REPO_PULL_FLAGS_BAREUSERONLY_FILES
This is an option which is intended mostly for flatpak;
see: https://github.com/flatpak/flatpak/issues/845

We're adding an option for pulling into *all*
repo modes that has an effect similar to the `bare-user-only`
change from https://github.com/ostreedev/ostree/pull/909

This way one can pull content into e.g. a root-owned `bare` repository and
ensure that there aren't any setuid or world-writable files.

Closes: #926
Approved by: alexlarsson
2017-06-13 18:44:28 +00:00
Colin Walters 74e3581ed6 lib/repo: Support hardlink conversions from bare-user to bu-only
Thinking about the problem of flatpak converting from `bare-user` to `bare-user-only`
"in place" by creating a new repo and doing a `pull-local`, I realized
that we can optimize this process by doing hardlinks for both metadata
and regular files.  The repo formats are *almost* compatible, the
exception being symlinks.

An earlier patch caused us to do hardlinks for metadata, this patch takes things
to the next step and special cases this specific conversion. In this case we
need to parse the source object to determine whether or not it's a symlink.

Closes: #922
Approved by: alexlarsson
2017-06-13 12:02:12 +00:00
Colin Walters b614c65eab lib/repo: Import metadata via hardlink even for distinct repo modes
Our previous logic for import-via-hardlink only tried if the repo modes match,
but we *can* hardlink metadata between e.g. `archive` and `bare-user` repos, and
that's quite useful thing to do. Our documentation encourages converting to/from
those repo modes locally for build systems.

Closes: #922
Approved by: alexlarsson
2017-06-13 12:02:12 +00:00
Colin Walters 695771667c lib/repo: Skip import via hardlink if repo owners don't match
Before this, if one had repos of matching mode but different owners,
which could happen if one e.g. makes a `bare` non-root repo in
`/ostree/deploy/$stateroot/var/tmp`, every time we tried to call `linkat()`
we'd get `EPERM` and fall back to a copy.

Fix this by saving the repo owner uid, and avoid trying to call `linkat()` if we
know it's going to fail. Of course most commonly in this scenario we'll
immediately fail trying to `chown` the files to `0`, but this is prep for a
future patch to improve `bare-user` → `bare-user-only` imports where we'll be a
bit more sophisticated.

Closes: #922
Approved by: alexlarsson
2017-06-13 12:02:12 +00:00
Colin Walters 848b7c0201 lib/repo: Refactor object copy import function
This came up in: https://github.com/ostreedev/ostree/pull/881

Basically doing streaming for metadata is dumb. Split up the metadata/content
paths so we pass metadata around as `GVariant`. This drops the last internal
caller of `ostree_repo_write_metadata_stream_trusted()` which was the dumb
function mentioned.

Closes: #923
Approved by: jlebon
2017-06-12 21:13:23 +00:00
Colin Walters 8edb5161db lib/checkout: Ignore world-writable dirs for bare-user-only checkout
See https://github.com/ostreedev/ostree/pull/909 for more information on the
rationale. Basically there's no reason for flatpak (which uses `bare-user-only`)
to have world-writable dirs. Particularly with the presence of the system
helper.

An approach I considered instead was to parse and validate directory metadata
objects at commit time. We still may do that in addition; for file objects we *had*
to do it that way because the actual files would be laid down suid.  But directories
live only as inert `.dirmeta` objects until we do a checkout (i.e. `mkdir()`), so
we can solve the problem at checkout time.

Closes: #914
Approved by: alexlarsson
2017-06-12 14:24:22 +00:00
Colin Walters 18ae8e5267 lib/commit: Drop some conditionals/clarify code in content path
Both callers of `commit_loose_object_trusted()` were passing
`OSTREE_OBJECT_TYPE_FILE`, so drop that parameter.  This in turn
allows us to drop lots of checking of that inside the function.

Add a doc comment, and rename to `commit_loose_content_object()` for clarity.

Closes: #914
Approved by: alexlarsson
2017-06-12 14:24:22 +00:00
Colin Walters aed8a6b09a lib/commit: Port final object writing function to new code style
I noticed my previous patches incorrectly started doing `return glnx_throw*`
inside a `goto out;` function. Fix this by porting forward consistently to new
style. We just do the error prefixing in the caller.

Closes: #914
Approved by: alexlarsson
2017-06-12 14:24:22 +00:00
Colin Walters 5de201df26 repo: Fix leak of superblock fds when generating summary
Related: https://github.com/ostreedev/ostree/issues/920

Closes: #921
Approved by: alexlarsson
2017-06-12 14:15:14 +00:00
Colin Walters e0e07e2a48 ci: Add CentOS 7 build
For similar reasons as https://github.com/projectatomic/rpm-ostree/pull/824

Closes: #919
Approved by: jlebon
2017-06-09 18:30:15 +00:00
Colin Walters a98faa9113 ci: Update to match current rpm-ostree
This copies the `ci/` directory from rpm-ostree, with much the same rationale;
among other things we don't want to depend on the Docker hub.

The specific reason I'm doing this is that I want to add a CentOS7 build, but
that means we can't use `projectatomic/ostree-tester`, and at that point we
might as well unwind it all.

Closes: #917
Approved by: jlebon
2017-06-08 21:08:31 +00:00
Colin Walters c81252c1e0 repo/commit: Support group-writable files for bare-user-only
These exist in the wild for flatpak, and aren't really a problem. The canonical
permissions are still either `0755` or `0644`, we just support the additional
writable bit for the group (i.e. extend the set to include `0775` and `0664`)
now to avoid breaking some flatpak content.

Closes: #913
Approved by: alexlarsson
2017-06-08 06:58:54 +00:00
Colin Walters 81e1f7761f tests: Add a test for bare-user-only failing to commit suid content
We didn't have coverage of this before, and adding the test infrastructure will
help ensure we have coverage for more changes here.

Closes: #913
Approved by: alexlarsson
2017-06-08 06:58:54 +00:00
Colin Walters 5913b22944 lib/repo: For bare-user, mask content object modes with 0775
Having every object in a bare-user repo (and checkouts) be executable
is ugly.  I can't think of a good reason to do that; they should only
be executable if their input is.  This does
for `bare-user` what we did for `bare-user-only` in
https://github.com/ostreedev/ostree/pull/909
It's also a stronger version of what we do with `checkout -U` in suppressing
suid - here we also strip world-writable files and the sticky bit (even though
that's meaningless today, it might not be in the future).

Closes: https://github.com/ostreedev/ostree/issues/907

Closes: #908
Approved by: alexlarsson
2017-06-08 06:50:16 +00:00
Colin Walters d3900f90f4 lib/repofile: Follow symlinks for g_file_read()
This avoids `ostree cat /path/to/symlink` crashing, a longstanding embarassing
issue.

Closes: #915
Approved by: jlebon
2017-06-07 20:54:32 +00:00
Colin Walters 807a804b16 lib/repofile: Port mostly to new code style
Prep for a bugfix.

Closes: #915
Approved by: jlebon
2017-06-07 20:54:32 +00:00
Colin Walters 6b402e53f4 builtins/cat: Port to new code style
Definitely better.  Prep for another fix.

Closes: #915
Approved by: jlebon
2017-06-07 20:54:32 +00:00
Colin Walters 0c4b3a2b6d Canonicalize bare-user-only perms with 0755 mask
For the flatpak use case where bare-user-only was introduced, we actually
don't want to support s{u,g} id files in particular.

Actually, I can't think of a reason to have anything outside of the
`0755 i.e. (u=rwx,g=rx,o=rx)` mask, so that's what we do here.

This will have the effect of treating existing `bare-user-only` repositories as
corrupted if they have files outside of that mask, but I think we should do this
now; most of the flatpak users will still be on `bare-user`, and we haven't
changed the semantics of that mode yet.

Note that in this patch we will also *reject* file content that doesn't
match this.  This is somewhat asymmetric, since we aren't similarly rejecting
e.g. directory metadata.  But, this will close off the biggest source
of the problem for flatpak (setuid binaries).

See: https://github.com/ostreedev/ostree/pull/908
See: https://github.com/flatpak/flatpak/pull/837

Closes: #909
Approved by: alexlarsson
2017-06-07 15:13:55 +00:00
Colin Walters e18cacb06e Don't install trivial-httpd man page if not enabled
I just noticed this scroll by in a file listing.

Closes: #905
Approved by: jlebon
2017-06-02 19:36:56 +00:00
Colin Walters cad42d9601 Revert "Add a notion of "physical" sysroot, use for remote writing"
This reverts commit 1eff3e8343. There
are a few issues with it.  It's not a critical thing for now, so
let's ugly up the git history and revisit when we have time to
debug it and add more tests.

Besides the below issue, I noticed that the simple `ostree remote add`
now writes to `/ostree/repo/config` because we *aren't* using the
`--sysroot` argument.

Closes: https://github.com/ostreedev/ostree/issues/901

Closes: #902
Approved by: mike-nguyen
2017-06-02 14:11:58 +00:00
Colin Walters a094879f3a lib/repo: Delete unused private prototypes
The implementations were removed in: 6ffcb24d22
I noticed this while looking at the commit code.

Closes: #898
Approved by: jlebon
2017-06-01 21:30:51 +00:00
Colin Walters f4f1330789 repo/commit: Split up metadata/content commit paths
There was a lot of conditionals inside `write_object()` differentating
between metadata/content, and then for content, on the different repo
types.  Further, in the metadata path since the logic is simpler, can
present a non-streaming API, and further use `OtTmpfile`, etc.

Splitting them up helps drop a lot of conditionals. We introduce a small
`CleanupUnlinkat` that allows us to fully convert to the new code style in both
functions.

This itself is still prep for fully switching to `GLnxTmpfile`.

Closes: #881
Approved by: jlebon
2017-06-01 18:43:38 +00:00
Colin Walters ec1964dd44 repo/commit: Don't renormalize trusted metadata
As the comment in the code says; in the expected checksum case, the caller
really has to have a normal form already.

Closes: #881
Approved by: jlebon
2017-06-01 18:43:38 +00:00
Colin Walters 6ba4dac6f2 repo/commit: In the expected checksum case, check existence early
If we have an expected checksum, call `fstatat(repo_dfd, checksum)`
early on before we do much else.  This actually duplicates code,
but future work here is going to split up the metadata/content
commit paths, so they'll need to diverge anyways.

Closes: #881
Approved by: jlebon
2017-06-01 18:43:38 +00:00
Colin Walters d2a92df155 repo/commit: Dedup content writing API implementation
Similar to metadata, for `write_content_trusted()` we can just
call `_write_content()` with a `NULL` output checksum.

Closes: #881
Approved by: jlebon
2017-06-01 18:43:38 +00:00
Colin Walters 22b1234f52 repo/commit: Dedup metadata writing API implementations
First, the streaming metadata API is pretty dumb, since metadata
should be small.  Really we should have supported a `GBytes`
version.  Currently, this API *is* used when we do local pulls,
so this commit has test coverage.  However, I plan to change
the object import to avoid using this.  But that's fine, since
I can't think of why someone would use this API.

Next, the only difference between `ostree_repo_write_metadata()` and
`ostree_repo_write_metadata_trusted()` is whether or not we pass
an output checksum; so just dedup the implementations.

Also while I'm here break out the input length validation and do
it early in the streaming case.

Closes: #881
Approved by: jlebon
2017-06-01 18:43:38 +00:00
Colin Walters 1eff3e8343 Add a notion of "physical" sysroot, use for remote writing
Using `${sysroot}` to mean the physical storage root: We don't want to write to
`${sysroot}/etc/ostree/remotes.d`, since nothing will read it, and really
`${sysroot}` should just have `/ostree` (ideally). Today the Anaconda rpmostree
code ends up writing there. Fix this by adding a notion of "physical" sysroot.
We determine whether the path is physical by checking for `/sysroot`, which
exists in deployment roots (and there shouldn't be a `${sysroot}/sysroot`).

In order to unit test this, I added a `--sysroot` argument to `remote add`.
However, doing this better would require reworking the command line parsing for
the `remote` argument to support specifying `--repo` or `--sysroot`, and I
didn't quite want to do that yet in this patch.

Closes: https://github.com/ostreedev/ostree/issues/892

Closes: #896
Approved by: jlebon
2017-06-01 18:15:56 +00:00
Colin Walters 2414704609 lib/util: Some style conversion
I saw a few instances of `glnx_set_error_from_errno() + return FALSE`,
and fixed them and did a bit of style conversion.

Closes: #895
Approved by: jlebon
2017-05-30 20:33:28 +00:00
Colin Walters 9a3555a74b cmd: Use autoptr for GKeyFile
Prep for code style conversion.

Closes: #891
Approved by: jlebon
2017-05-29 14:09:08 +00:00
Colin Walters 88a1fc92a9 tree-wide: Add+run spatch to use glnx_throw()
I had to run a sed job to add whitespace after, but otherwise this was easy.

Closes: #890
Approved by: jlebon
2017-05-26 19:27:11 +00:00