Commit Graph
100 Commits
Author SHA1 Message Date
Colin Walters 882fa906e2 bin/main: Remove duplicated usage output on unknown commands
It's been this way for a long time, though not forever; I went
back to v2014.11 as a random choice and it worked then.  Not
going to bother doing a full archive search for this though.
Anyone who wants more context can help themselves.

Closes: https://github.com/ostreedev/ostree/issues/1096

Closes: #1106
Approved by: jlebon
2017-08-23 17:51:17 +00:00
Colin Walters 89801dd481 build/maint.mk: Comment out setting of LC_ALL
This triggers obscure bugs; really we shouldn't be overriding
the global locale here.  In practice, production build systems
will be using fixed locales anyways.

Also, we only use a small subset of this file (`make syntax-check`),
which appears to work OK without this.

I will probably try to work out where to submit this as at
least an issue report for upstream gnulib.

Closes: https://github.com/ostreedev/ostree/issues/1101

Closes: #1107
Approved by: heftig
2017-08-23 17:41:06 +00:00
Colin Walters eb6f7c6db8 lib/commit: Honor commit filter for libarchive --tar-autocreate-parents
This makes `ostree commit --tree=tar` honor `--owner-uid` and `--owner-gid`
for the root directory.

Prep for further commit filtering work, although mostly for the unit test cases;
this ensures we can use `ostree checkout` after autocreating a root directory.

Closes: #1104
Approved by: jlebon
2017-08-23 14:48:12 +00:00
Colin Walters 556e2deb93 lib/commit: Remove duplicated function for filter processing
The wrapping here is unnecessary, since `_ostree_repo_commit_modifier_apply()`
already does what this function did. Further, the return type was wrong.

Saw this while doing some libarchive work.

Closes: #1104
Approved by: jlebon
2017-08-23 14:48:12 +00:00
Colin Walters 618617d68b lib/pull: Add support for timestamp-check option, use in upgrader
For both flatpak and ostree-as-host, we really want to verify up front during
pulls that we're not being downgraded. Currently both flatpak and
`OstreeSysrootUpgrader` do this before deployments, but at that point we've
already downloaded all the data, which is annoying.

Closes: https://github.com/ostreedev/ostree/issues/687

Closes: #1055
Approved by: jlebon
2017-08-21 18:24:07 +00:00
Colin Walters 1e5b06be5b lib/deploy: Add .img to end of initramfs in /usr/lib/modules
Follow up to <https://github.com/ostreedev/ostree/pull/1079>; I was working on
the rpm-ostree updates for this, and I think it's more consistent if we have
`.img` here, since that's a closer match to the "remove $kver" that results in
`vmlinuz`. Also just best practice to have file suffix types where they make
sense.

The astute reader might notice this sneaks in a change where we'd crash if the
legacy bootdir didn't have an initramfs...yeah, should probably have test
coverage of that.

Closes: #1095
Approved by: jlebon
2017-08-21 16:41:50 +00:00
Colin Walters 7b3e55a3f4 docs/build: s/libOSTree/libostree/
I find "libOSTree" awkward to type and really to look at. Let's be nicer on
people's pinky fingers and eyes and drop it all down to lowercase.

Closes: #1093
Approved by: jlebon
2017-08-18 17:51:09 +00:00
Colin Walters 3ab0d5e664 lib/sysroot: Support /usr/lib/modules/$kver for kernel/initramfs
This is the new Fedora kernel standard layout; it has the advantage
of being in `/usr` like `/usr/lib/ostree-boot`, but it's not OSTree
specific.

Further, I think in practice forcing tree builders to compute the checksum is an
annoying stumbling block; since we already switched to e.g. computing checksums
always when doing pulls, the cost of doing another checksum for the
kernel/initramfs is tiny. The "bootcsum" becomes more of an internal
implementation detail.

Now, there is a transition; my current thought for this is that rpm-ostree will
change to default to injecting into both `/usr/lib/ostree-boot` and
`/usr/lib/modules`, and stop doing `/boot`, then maybe next year say we drop the
`/usr/lib/ostree-boot` by default.

A twist here is that the default Fedora kernel RPM layout (and what's in
rpm-ostree today) includes a kernel but *not* an initramfs in
`/usr/lib/modules`. If we looked only there, we'd just find the kernel. So we
need to look in both, and then special case this - pick the legacy layout if we
have `/usr/lib/modules` but not an initramfs.

While here, rework the code to have an `OstreeKernelLayout` struct which makes
dealing with all of the variables nicer.

Closes: #1079
Approved by: jlebon
2017-08-18 17:34:36 +00:00
Colin Walters 984d22303d bin: Squash some -Wuninit warnings with porting to new style
I noticed this with a local build of an RPM:

```
/usr/include/glib-2.0/glib/glib-autocleanups.h:28:3: warning: 'help' may be used uninitialized in this function [-Wmaybe-uninitialized]
   g_free (*pp);
   ^~~~~~~~~~~~
src/ostree/ot-main.c:82:20: note: 'help' was declared here
   g_autofree char *help;
                    ^~~~
```

Closes: #1091
Approved by: jlebon
2017-08-17 21:04:04 +00:00
Colin Walters e6a4203c52 boot: Add a tmpfiles.d snippet to clean up /var/tmp/ostree-ovl.XXX
This is simplest for now.  Compare with similar logic from
`/usr/lib/tmpfiles.d/tmp.conf`:
```
R! /tmp/systemd-private-*
```

Closes: https://github.com/ostreedev/ostree/issues/393

Closes: #1090
Approved by: jlebon
2017-08-17 18:38:45 +00:00
Colin Walters bd9e931b33 build-sys: Post-release version bump
Closes: #1089
Approved by: jlebon
2017-08-17 18:18:21 +00:00
Colin Walters 395676917b Release 2017.10
Closes: #1089
Approved by: jlebon
2017-08-17 18:18:21 +00:00
Colin Walters c4148d4ddd lib: Fix v2017.10 symbols to inherit from v2017.8
I actually don't quite know what the version inheritance really does, but let's
be safe and fix this. I'm being conservative here and fixing it to inherit from
2017.8, skipping .9 since that doesn't have a parent.

Related: https://github.com/ostreedev/ostree/issues/1087

Closes: #1088
Approved by: jlebon
2017-08-17 16:57:33 +00:00
Colin Walters ceeefa69f5 docs/related-projects: Tweak client side snapshot text, add casync
The latter came up on the list.

Closes: #1086
Approved by: jlebon
2017-08-17 16:48:25 +00:00
Colin Walters 6709aa83a8 lib/sysroot: Port a few functions to new style
Not sure why we didn't do this earlier. Just noticed them when looking at the
code for a different reason.

Closes: #1085
Approved by: jlebon
2017-08-17 14:37:49 +00:00
Colin Walters 40a64bcfe9 lib/deploy: Ignore errors from FITHAW
In the production case since we used `daemon()` our stderr is `/dev/null`¹
there's not much use in logging errors from `FITHAW` or `exit(1)`, and doing so
breaks the test suite which checks the return from `waitpid()`. There's nothing
we can really do if `FITHAW` fails, and in most of those cases `EINVAL`,
`EOPNOTSUPP`, we *shouldn't* do anything anyways.

¹ Though perhaps we should set up the systemd journal, but let's not
  go there right now.

Closes: #1084
Approved by: jlebon
2017-08-16 18:01:04 +00:00
Colin Walters ba28684ac2 lib/deploy: Really close testing race condition
I added `waitpid()`, but that didn't actually help because we were
`daemon()`izing. Don't daemonize if we're testing so that we can `waitpid()`.

Note I still haven't reproduced this race locally, but I'm pretty sure this will
fix it.

While here, actually check the return value from `waitpid()` just in case
something goes wrong there.

Closes: #1084
Approved by: jlebon
2017-08-16 18:01:04 +00:00
Colin Walters eb35821a51 build-sys: Move bash completions to /usr/share/ by default
This is in line with the "/etc is for sysadmins", "/usr is OS" model;
e.g. systemd's bash completions go there.

Making this change since I was looking at the required spec file changes.

Closes: #1083
Approved by: mbarnes
2017-08-16 15:14:31 +00:00
Colin Walters 6063bdb013 Update libglnx
This is mostly the `copy_file_range` changes plus the Coverity files.

```
Colin Walters (4):
      localalloc: Abort on EBADF from close() by default
      local-alloc: Remove almost all macros like glnx_free, glnx_unref_variant
      console: Fix Coverity NULL deref warning
      fdio: Merge systemd code to use copy_file_range(), use FICLONE

Jonathan Lebon (1):
      console: trim useless check

Matthew Leeds (1):
      dirfd: Fix typo in comment

Philip Withnall (1):
      glnx-console: Add missing NULL check before writing out text
```

Update submodule: libglnx

Closes: #1081
Approved by: jlebon
2017-08-16 12:56:48 +00:00
Colin Walters fd98bda3c7 repo: Introduce ostree_repo_open_at() and ostree_repo_create_at()
This essentially completes our fd-relative conversion.

While here, I cleaned up the semantics of `ostree_repo_create()` and
`ostree_repo_create_at()` to be more atomic - basically various scripts were
testing for the `objects` subdirectory, so let's formalize that.

Closes: #820
Approved by: jlebon
2017-08-15 12:35:10 +00:00
Colin Walters 95c832b4e2 lib/deploy: Close test suite race condition
Saw this in a PR result; we need to wait for the child to have written its
result to stderr before we exit, otherwise the test suite may not read it in
time.

Closes: #1070
Approved by: jlebon
2017-08-10 14:42:58 +00:00
Colin Walters de153dea30 lib/sysroot: Add journal-msg signal
This will allow us to drop the awful hack in rpm-ostree where we watch our own
stdout. In general, libraries shouldn't write to stdout.

Also we can kill the systemd journal wrapper code. There's some duplication at
each call site now...but it's easier than trying to write a `sd_journal_send()`
wrapper.

I was originally going to have this emit all of the structured data too as a
`GVariant` but decided it wasn't worth it right now.

Closes: #1052
Approved by: jlebon
2017-08-10 14:20:00 +00:00
Colin Walters f1102763df lib/sysroot-deploy: Refactor kernel layout parsing
I'd like to move the new canonical kernel directory to `/usr/lib/modules/$kver`,
as Fedora has done. The `get_kernel_from_tree()` function now abstracts over
parsing the data (src vs destination filenames, as well as checksum) in
preparation for adding the new case.

In preparation for this, let's change the current test suite to use the
*current* directory of `/usr/lib/ostree-boot`, and also add coverage of `/boot`.

Closes: #1053
Approved by: jlebon
2017-08-09 13:46:17 +00:00
Colin Walters 8b60f63f58 lib/sysroot-deploy: Port a kernel finding logic to new style
Prep for more work here.

Closes: #1053
Approved by: jlebon
2017-08-09 13:46:17 +00:00
Colin Walters 8642ef5ab3 lib/deploy: Use a FIFREEZE/FITHAW cycle for /boot
See: http://marc.info/?l=linux-fsdevel&m=149520244919284&w=2

XFS doesn't flush the journal on `syncfs()`. GRUB doesn't know how to follow the
XFS journal, so if the filesystem is in a dirty state (possible with xfs
`/boot`, extremely likely with `/`, if the journaled data includes content for
`/boot`, the system may be unbootable if a system crash occurs.

Fix this by doing a `FIFREEZE`+`FITHAW` cycle.  Now, most people
probably would have replaced the `syncfs()` invocation with those two
ioctls.  But this would have become (I believe) the *only* place in
libostree where we weren't safe against interruption.  The failure
mode would be ugly; nothing else would be able to write to the filesystem
until manual intervention.

The real fix here I think is to land an atomic `FIFREEZETHAW` ioctl
in the kernel.  I might try a patch.

In the meantime though, let's jump through some hoops and set up
a "watchdog" child process that acts as a fallback unfreezer.

Closes: https://github.com/ostreedev/ostree/issues/876

Closes: #1049
Approved by: jlebon
2017-08-08 16:09:04 +00:00
Colin Walters 9f8f351cd4 lib: Port gpg verification for remotes to fd-relative
This was the last use of `repo->repodir` internally, and will help finally add
`ostree_repo_open_at()`.

Closes: #1034
Approved by: jlebon
2017-08-08 15:50:25 +00:00
Colin Walters 6d861dd92d tree-wide: Remove trailing semicolon from autoptr declarations
It confuses `g-ir-scanner`, and isn't necessary.

Closes: #1056
Approved by: pwithnall
2017-08-07 17:42:32 +00:00
Colin Walters 48f8f32921 bin/pull: Fix @ override syntax when pulling multiple refs
Coverity spotted an infloop here since we were incrementing `i++`
instead of `j++`.  But adding a test revealed other bugs - we need
to keep the arrays in sync.

Coverity CID: 1452204

Closes: #1041
Approved by: pwithnall
2017-08-07 16:10:08 +00:00
Colin Walters 8eec337fee lib/deltas: Squash Coverity warning for div-by-zero in delta show
If a delta happens to have zero objects, we could end up doing
a divide-by-zero when inferring endianness.  In practice,
a zero-object delta isn't possible to generate I think, but
let's make sure the code is defensive all the same.

Spotted by Coverity.

Coverity CID: 1452208

Closes: #1041
Approved by: pwithnall
2017-08-07 16:10:08 +00:00
Colin Walters 4e068f3924 tree-wide: Fix the build with old glib (Ubuntu Trusty etc.)
This regressed with <https://github.com/ostreedev/ostree/pull/1040>
but currently the Travis builds aren't gating.

Closes: #1051
Approved by: jlebon
2017-08-03 16:23:41 +00:00
Colin Walters d3897e52e7 lib: Define and use an autoptr cleanup for gpgme_key_t
Followup for previous patch, allows porting a bit to new code style.

Closes: #1039
Approved by: jlebon
2017-08-03 15:01:57 +00:00
Colin Walters b929b620ae tree-wide: Use g_autoptr(Ostree*)
Part of cleaning up our usage of libglnx; we want to use what's in GLib where we
can.

Had to change a few .c files to `#include ostree.h` early on to pick up
autoptrs for the core types.

Closes: #1040
Approved by: jlebon
2017-08-03 13:48:12 +00:00
Colin Walters 7f33d94dde tests/libtest-core.sh: Add a comment that this copy is canonical
Ref: https://github.com/projectatomic/bubblewrap/pull/203

Closes: #1047
Approved by: jlebon
2017-08-02 17:55:50 +00:00
Colin Walters e0f17b41ba lib/pull: Log state of summary/commit GPG verification
Since we have both, we should clearly log the state of both of
them.  Split this out of a larger patch.

Closes: #1046
Approved by: jlebon
2017-08-02 17:45:49 +00:00
Colin Walters d5273b34d0 lib/repo: Add API to create and list ref aliases
There are multiple use cases where we'd like to alias refs.

First, having a "stable" alias which gets swapped across major
versions: https://pagure.io/atomic-wg/issue/228

Another case is when a ref is obsoleted;
<https://pagure.io/atomic-wg/issue/303>
This second one could be done with endoflife rebase, but I think
this case is better on the server side, as we might later change
our minds and do actual releases there.

I initially just added some test cases for symlinks in the `refs/heads` dir to
ensure this actually works (and it did), but I think it's worth having APIs.

Closes: #1033
Approved by: jlebon
2017-08-02 17:33:10 +00:00
Colin Walters d1eb909cd0 lib/deploy: Add structured logging info for syncfs() times
I plan to at some point change rpm-ostree to read the journal messages from
libostree and render things like the time we spent in syncfs().

Closes: #1044
Approved by: jlebon
2017-08-02 17:19:37 +00:00
Colin Walters 4d723df2f2 lib/sysroot: Add prefixes to syncfs/fsync error messages
And clean up one other bare `glnx_throw_errno()`.

Closes: #1044
Approved by: jlebon
2017-08-02 17:19:37 +00:00
Colin Walters 40b41d2cc9 tree-wide: Use GLib autocleanups for libarchive
Define typedefs for read/write archives, and use the GLib
autocleanups for them.   Prep for updating libglnx to drop its
custom autocleanup macros.

Closes: #1042
Approved by: jlebon
2017-08-02 16:55:38 +00:00
Colin Walters ded6417aee lib/gpg: Switch to GLib autocleanups for gpgme types
Prep for dropping `GLNX_DEFINE_CLEANUP_FUNCTION` from libglnx
in favor of using GLib's `G_DEFINE_AUTO_CLEANUP_FREE_FUNC()`.

Closes: #1042
Approved by: jlebon
2017-08-02 16:55:38 +00:00
Colin Walters 6347c0fb88 lib/curl: Only check individual request errors
It looks like `curl_multi_socket_action()` will return an error
if *one* of the requests has an error, but we already check
for that explicitly by iterating over each handle.

In libcurl, the "easy" layer doesn't really make use of this
return value.  I did a bit of looking elsewhere; systemd
does check it as a runtime error, not an assertion.  librepo
doesn't use the multi interface.

Closes: https://github.com/ostreedev/ostree/issues/1035

Closes: #1038
Approved by: jlebon
2017-08-02 15:44:00 +00:00
Colin Walters 2f0707a054 lib/commit: Rework a conditional set for clarity and Coverity
Coverity complained that the `else if (bytes_read == 0)` was dead
code if we happened to find it was already false when testing
`else if (G_UNLIKELY (bytes_read == 0 ...`.

There was nothing wrong with the logic, but let's rework it to
only test the value once; I think it does end up nicer anyways.

Coverity CID: 1452186

Closes: #1037
Approved by: jlebon
2017-08-02 15:34:16 +00:00
Colin Walters fe8938227e tests/lzma: Fix off-by-one in buffer size
Coverity spotted that we had an off-by-one here since we were using
`i+1`.  Fix this by adding a `-1` to the bounds check.  Also use
`sizeof()` to ensure the data and size are coupled.

Coverity CID: 1452207

Closes: #1037
Approved by: jlebon
2017-08-02 15:34:16 +00:00
Colin Walters e70db67c01 rofiles-fuse: Add missing error handling for fcntl()
Spotted by Coverity.

Coverity CID: 1452201

Closes: #1037
Approved by: jlebon
2017-08-02 15:34:16 +00:00
Colin Walters 5daae8629e tests/lzma: Use GRand over random()
It's designed for test suites and non-critical random uses like this. This
silences a Coverity warning about weak randomness.

Closes: #1037
Approved by: jlebon
2017-08-02 15:34:16 +00:00
Colin Walters 78348fa296 bin/refs: Fix error handling logic
Spotted by Coverity.

Coverity CID: 1452202

Closes: #1037
Approved by: jlebon
2017-08-02 15:34:16 +00:00
Colin Walters 36ab8cd0e1 tests: Add some return value checking to pacify Coverity
No real problems here, but Coverity likes to see consistent checking of return
values, and I agree with it.

Coverity CID: 1452213
Coverity CID: 1452211

Closes: #1037
Approved by: jlebon
2017-08-02 15:34:16 +00:00
Colin Walters fbefe875c7 docs/pull: Finish incomplete docs for pull@hash syntax
Closes: https://github.com/ostreedev/ostree/issues/1043

Closes: #1045
Approved by: guyshapiro
2017-08-01 21:51:17 +00:00
Colin Walters d7f953aa3a lib/pull: Journal fetch results
This is a continuation of addition of journaling to libostree; see
e.g. <https://github.com/ostreedev/ostree/pull/708>.

I wanted more information at the end of fetches; in particular
some details about the delta execution (what opcodes etc.), but
this is a first step: we log things like the transferred data
as well as whether or not GPG was enabled, etc.

One awkward thing about this is how we map the fetcher options like
`tls-ca-path` back out into an enum for the code to log. But eh, hard to fix
without a bigger refactoring.

Closes: #1032
Approved by: jlebon
2017-07-28 13:50:22 +00:00
Colin Walters d24b259cc5 grub2: Handle aarch64 (at least on Fedora/RHEL)
See: https://github.com/projectatomic/rpm-ostree-toolbox/issues/102#issuecomment-316483554

The logic here should now match what grubby does.  I have no
idea whether this applies to other distributions though.

Closes: #1021
Approved by: jlebon
2017-07-27 18:21:00 +00:00
Colin Walters 79a88ac249 build-sys: Post-release version bump
Closes: #1029
Approved by: jlebon
2017-07-26 21:04:08 +00:00
Colin Walters fa3e07e5d7 Release 2017.9
Closes: #1029
Approved by: jlebon
2017-07-26 21:04:08 +00:00
Colin Walters f1f199578e ci: Enable libcurl by default on Fedora
The insttest fell over since its build used libsoup, but that just dropped out
of FAH.

Closes: #1030
Approved by: jlebon
2017-07-26 20:54:35 +00:00
Colin Walters e09fc83ab3 lib/core: Add #defines for ref/collection binding
These were previously private, but since we expect people to use them, let's add
`#define`s like we did for some of the other commit metadata.

Closes: #1028
Approved by: jlebon
2017-07-26 18:52:04 +00:00
Colin Walters 0985158be7 Update libglnx, port some uses to newer APIs
Mostly for the latest `-Wmaybe-uninitialized` fix, but while here also port some
places to newer APIs.

Update submodule: libglnx

Closes: #1027
Approved by: jlebon
2017-07-24 18:43:57 +00:00
Colin Walters 8456fd5057 build: Turn off default warnings if we find -Werror specified
Our CI runs use `-Werror`; there's no point to our default warning set kicking
in, it just bloats the command line output.

Closes: #1023
Approved by: jlebon
2017-07-21 20:45:47 +00:00
Colin Walters 0ded552cf1 ci: Enable -Werror for clang
I hit an unused-variable warning with `GLNX_AUTO_PREFIX_ERROR` for
rpm-ostree and led me to wonder why ostree didn't fail, then I noticed
we had lost the special `-Werror=unused-variable` bit.  Let's go
ahead and use `-Werror` for clang too.

Closes: #1023
Approved by: jlebon
2017-07-21 20:45:47 +00:00
Colin Walters ed99b4169e tests: More fixes for gjs tests
Previous to this commit, the gjs tests were installed-only; and our
logic for handling the "--enable-installed-tests=exclusive" logic
actually also meant they weren't installed.

It did work for me locally with `--enable-installed-tests`.

However, to make things fully symmetric, let's enable the js tests to also be
run under `make check`.

Also remove `corrupt-repo-ref.js` from the PAPR invocation since it's not
actually a unit test, it's a utility helper.

Closes: #1022
Approved by: jlebon
2017-07-21 15:45:27 +00:00
Colin Walters 6430207e47 lib: Add #defines for current well-known metadata keys
This came up in https://github.com/projectatomic/rpm-ostree/issues/142

Let's add `#define`s for our metadata keys, with documentation so
that, well, they're documented.

Closes: #1024
Approved by: peterbaouoft
2017-07-21 14:53:33 +00:00
Colin Walters cf6c15a6c5 ci/papr: Add a suite to run introspection-based tests without ASAN
Unfortunately we can't do gobject-introspection based tests
while compiling with `-fsanitize=address`, since it needs to hook
`malloc` early on.

Add a new suite which just runs the introspection-based tests without ASAN.

Closes: #1016
Approved by: jlebon
2017-07-20 14:01:11 +00:00
Colin Walters f9f7d55e79 lib/commit: Fix EBADF with GENERATE_SIZES option for commit
Regression from previous tmpfile refactoring; unfortunately
the `OSTREE_REPO_COMMIT_MODIFIER_FLAGS_GENERATE_SIZES` option
only has coverage via gjs currently.

Might expose it via the cmdline in a later option, but in the big picture the
idea was that this data is better kept in static deltas.

Closes: https://github.com/ostreedev/ostree/issues/1014

Closes: #1016
Approved by: jlebon
2017-07-20 14:01:11 +00:00
Colin Walters ebbd0b3ce6 ci/papr: Switch primary to libcurl, add libsoup context
Sometime in the next few releases I think we should make libcurl
the default.

Prep for more CI work.

Closes: #1016
Approved by: jlebon
2017-07-20 14:01:11 +00:00
Colin Walters 779f125cbe lib/repo: Auto-recreate repo/tmp if it's deleted
We can accumulate a lot of space there; let's be nice to people who delete the
whole directory.

Closes: https://github.com/ostreedev/ostree/issues/1018

Closes: #1020
Approved by: jlebon
2017-07-19 15:01:19 +00:00
Colin Walters c740b7f6d2 core: Sanitize error text validating refs (e.g. against HTML)
See: https://github.com/projectatomic/rpm-ostree/issues/885

If we get a successful Apache directory listing HTML when fetching what we
intend to be a ref, we'd dump the HTML into the error.

I did some scanning of the pull code, and this was the only case
I saw offhand where we were dumping text out into an error.  Which
makes sense, since most of our formats are binary, the exeptions I
think are just `repo/config` and `repo/refs/`.

Closes: #1015
Approved by: mbarnes
2017-07-19 14:45:57 +00:00
Colin Walters 2a9689b76a Update libglnx, port various bits to new API
Using the error prefixing in the delta processing allows us to
do new code style.  Also strip trailing whitespace.

Use error prefixing in a few other random places.  I didn't
hunt for all of them, just testing out the new API.

Use `glnx_fchmod()`.  Also note I dropped one `fchmod (tmpf, 0600)`
which is no longer necessary.

Update submodule: libglnx

Closes: #1011
Approved by: jlebon
2017-07-18 19:18:38 +00:00
Colin Walters efd460782a lib/pull: Drop direct use of ->repodir
Prep for `ostree_repo_new_at()`.  Down the line perhaps
we should extend libcurl to accept a file descriptor for cookies,
but this works OK for now.

Closes: #1010
Approved by: jlebon
2017-07-18 19:07:56 +00:00
Colin Walters 9430b8ad75 bin/cookies: Drop libsoup code, fix fd-relative issues, new style
Prep for `ostree_repo_new_at()`. These commands were directly accessing
`repo->repodir`, which it turns out was unnecessary since the the APIs they then
used were fd-relative. Except actually there were bugs there, so fix all of the
cookie util code to actually use the passed `dfd` and not just hardcode
`AT_FDCWD`.

Also, libsoup can't handle this (its APIs require fully qualifed paths), and
there's not a really good reason to have two implementations now; historically
it was useful to cross-check them, but I don't think we need that.

While I'm here, port to new style.

Closes: #1010
Approved by: jlebon
2017-07-18 19:07:56 +00:00
Colin Walters e0346c1494 Add a notion of "physical" sysroot, use for remote writing
(Note this PR was reverted in <https://github.com/ostreedev/ostree/pull/902>;
 this version should be better)

Using `${sysroot}` to mean the physical storage root: We don't want to write to
`${sysroot}/etc/ostree/remotes.d`, since nothing will read it, and really
`${sysroot}` should just have `/ostree` (ideally). Today the Anaconda rpmostree
code ends up writing there. Fix this by adding a notion of "physical" sysroot.
We determine whether the path is physical by checking for `/sysroot`, which
exists in deployment roots (and there shouldn't be a `${sysroot}/sysroot`).

In order to unit test this, I added a `--sysroot` argument to `remote add`.
However, doing this better would require reworking the command line parsing for
the `remote` argument to support specifying `--repo` or `--sysroot`, and I
didn't quite want to do that yet in this patch.

This second iteration of this patch fixes the bug we hit the first time;
embarassingly enough I broke `ostree remote list` finding system remotes.
The fix is to have `ostree_repo_open()` figure out whether it's the same
as `/ostree/repo` for now.

Down the line...we might consider having the `ostree remote` command line itself
instatiate an `OstreeSysroot` by default, but this maximizes compatibility; we
just have to pay a small cost that `ostree` usage outside of that case like
`ostree static-delta` in a releng Jenkins job or whatever will do this `stat()`
too.

Closes: https://github.com/ostreedev/ostree/issues/892

Closes: #1008
Approved by: mbarnes
2017-07-18 18:58:06 +00:00
Colin Walters 8b1f1c4428 lib/pull: Do local content imports async too
This came up in <https://github.com/ostreedev/ostree/pull/982>; when
we added more direct local importing, we did it synchronously.

This was actually quite a regression when doing local pulls between different
modes; in particular between a bare mode and `archive`, as we were suddenly
doing gzip {de,}compression in the main thread.

Down the line actually...a simpler fix is probably to change things so that the
local path is really only used when we know we can hardlink; everything else
would go though the fetcher codepath but with `file://`.

But this isn't a lot more code, and the speed/interactivity win is large.

Note we're only doing content async with this patch. We could do metadata as
well; we have the object already local. But the metadata code path is messier,
and metadata objects are smaller.

Another area where this comes up is that in e.g. Fedora releng, most operations
talk to a NetApp via NFS. So this has the classic network filesystem problem
that operations that are normally cheap like `stat()` can actually have
nontrivial latency. Doing as much as possible in threads is better there too.

Closes: #1006
Approved by: jlebon
2017-07-18 17:03:13 +00:00
Colin Walters 96e49a67f9 ci/papr: Update to F26
In particular F25AH will stop getting updates.

Closes: #1012
Approved by: jlebon
2017-07-18 14:01:54 +00:00
Colin Walters 620a90ebfa lib/pull: Avoid journaling 404s for optional content
Currently in Fedora we don't sign summaries, and every use of
`rpm-ostree` would emit to the journal an error when we failed
to fetch it.

Fix this by having `OSTREE_FETCHER_REQUEST_OPTIONAL_CONTENT` tell the fetcher
not to journal 404 errors. While fixing this, we had a mix of two booleans vs
the flags; fix things so we consistently use the flags in the fetcher and pull
code.

Closes: #1004
Approved by: mbarnes
2017-07-13 21:41:00 +00:00
Colin Walters 0aa20df20e tests: Run pull tests for bare/bare-user
We have variants of `test-basic` for all 4 modes, but not for pull-test, which
for some reason was named `pull-archive`, but mostly pulls *into* bare repos.
The test code was structured like the basic one where it called into a
`pull-test.sh`, so let's actually use it for 2/3 bare modes. (I tried to extend
it to `bare-user-only` but it failed, going to look at that after this).

This is related to https://github.com/ostreedev/ostree/issues/991

Closes: #998
Approved by: jlebon
2017-07-10 14:18:18 +00:00
Colin Walters 9d941dcebb checkout: Don't set dir mtime to 0 when doing a force copy checkout
When we [switched to using checkout + force_copy](https://github.com/ostreedev/ostree/commit/e8efd1c8dcaad8fbd3b05c400972d237406263e7),
a side effect that went unnoticed at the time is that we started
setting directory mtimes to zero.

See the below bug where we long ago set the file times to zero, which got fixed,
so let's not regress things by setting the directory times to zero either. (Even
though AFAICS GNU tar doesn't complain about those)

This semantic is somewhat "overloaded" onto `force_copy`, but it avoids adding
yet another boolean; we don't have that many reserved boolean slots left. I
can't really think of many good use cases for `force_copy` *other* than the
`/etc` merge anyways.

https://bugzilla.redhat.com/show_bug.cgi?id=1229160

Closes: https://github.com/ostreedev/ostree/issues/995

Closes: #997
Approved by: jlebon
2017-07-07 15:01:51 +00:00
Colin Walters d2a05e5a09 deploy: Port some functions to new style
There are a number of simple ports here.  Prep for further work
in `/etc` merge.

I also stripped trailing whitespace globally.

Closes: #996
Approved by: jlebon
2017-07-07 14:39:47 +00:00
Colin Walters 28ec43c41a build-sys: Post-release version bump
Closes: #994
Approved by: jlebon
2017-07-06 14:45:02 +00:00
Colin Walters 5a5e465492 Release 2017.8
Closes: #994
Approved by: jlebon
2017-07-06 14:45:02 +00:00
Colin Walters 23b93a3eb6 lib/repo: Immediately error creating bare-user repo on tmpfs
And in general, if for some reason we can't write `user.` xattrs, provide an
error immediately rather than doing it during a later pull. This way the failure
cause is a lot more obvious.

Related: https://github.com/ostreedev/ostree/issues/991

Closes: #993
Approved by: jlebon
2017-07-06 14:31:37 +00:00
Colin Walters 3234295324 pull: Cleanly error when doing local pulls of remote-prefixed refs
In the storage PR I was trying to do a `pull-local` of the whole
`/ostree/repo` on the system, which ended up triggering a `g_critical()`
in the collections code, since we tried to parse a remote-prefixed ref
`fedora:fedora/26/x86_64/atomic-host` as a ref.

I'm not sure offhand what our behavior in this case *should* be.  I
think git only clones local refs, but I need to check.

This corner case arises only with `pull-local`.  But in any case,
while we were previously saying this is programmer error, since it's
so easy to pass various unchecked input into the pull machinery,
make invalid refs an explicit error.

Closes: #992
Approved by: jlebon
2017-07-05 13:37:22 +00:00
Colin Walters 1f5ce1a9f7 lib/repo: Add min-free-space-percent option, default 3%
For ostree-as-host, we're the superuser, so we'll blow past
any reserved free space by default.  While deltas have size
metadata, if one happens to do a loose fetch, we can fill
up the disk.

Another case is flatpak: the system helper has similar concerns
here as ostree-as-host, and for `flatpak --user`, we also
want to be nice and avoid filling up the user's quota.

Closes: https://github.com/ostreedev/ostree/issues/962

Closes: #987
Approved by: jlebon
2017-07-04 16:15:11 +00:00
Colin Walters 8d4d638e99 lib/commit: Use provided length when doing writes
This is prep for storage space checks, where we look at free
space after parsing the metadata, before we write anything.

We did length-limited writes in the fd-based input path, but not for the
`GInputStream` path which in practice is used for HTTP pulls.

Closes: #987
Approved by: jlebon
2017-07-04 16:15:11 +00:00
Colin Walters 3348baf6eb lib/commit: Ensure bare-user objects are always user-readable
Some of the Jenkins jobs for Fedora Atomic Host broke after updating
to 2017.7, and it turns out that we regressed handling unreadable
files in `bare-user` mode.  An example of this is `/etc/shadow`, which
ends up in the ostree-as-host content as `/usr/etc/shadow`.

Now there are better fixes here; we should probably delete it and create it
during the config merge if it doesn't exist.  In general, having secret files in
ostree really isn't supported, so it doesn't make sense to include them.

But let's fix this regression - when operating as an unprivileged user we don't
have `CAP_DAC_OVERRIDE` and hence will fail to open un-user-readable objects.

(We still preserve the actual `0` mode of course in the xattr and will
 apply it in `bare`)

Closes: #989
Approved by: jlebon
2017-06-30 21:23:48 +00:00
Colin Walters cd7d35945a bin/commit: Add '=' to --statoverride
Previously, we only supported additions in the statoverride file;
it was mainly for adding the setuid bit without having that physically
on disk.

However, for testing a change to `bare-user` handling around *unreadable*
files (which happens for `/etc/shadow` in host content), I need a way
to write that into a repo in the test suite.

I'm not actually aware of a non-test-suite use case for this; a more
sophisticated user is going to be using the API directly, which can already do
this. But we need it for tests at least.

Closes: #989
Approved by: jlebon
2017-06-30 21:23:48 +00:00
Colin Walters e3a540a606 bin/commit: Port helper functions to new style
Prep for more work here.  Can't yet port the main function
without a cleanup for transactions.

Closes: #988
Approved by: jlebon
2017-06-30 20:28:48 +00:00
Colin Walters 192e7b888f lib/commit: Fix a tmpfile fd leak in static delta processing
I had thought `glnx_link_tmpfile_at()` actually consumed the tmpfile;
it does consume the *path* but not the fd.  In the non-delta path
things were fine since we used the autocleanup.

But the delta code had a tmpfile allocated in its struct that got reused, and
hence leaked the fd. Fix this by making the commit API actually consume the
tmpfile fully, just like the path path.

Closes: #986
Approved by: jlebon
2017-06-30 19:48:05 +00:00
Colin Walters 4273e670ea Add "pull --localcache-repo"
This is a lot like `git clone --reference`, but we chose "localcache" as the
term "reference" is already used.

The main use case I'm targeting this for is the Fedora Atomic Host installer
case where we embed the repo content in the installer, but we may want to
kickstart and download newer content. There, while we want to get a newer ref,
we can still use the local repo as an object cache, since we have it sitting
there in memory anyways.

Another case is where one has a host ostree (say e.g. Fedora Atomic
Workstation), and one wants to create a local archive mirror of FAH. Then one
can use `pull --reference /ostree/repo` and pull the common objects (e.g.
contents of `bash.rpm` etc.)

Closes: https://github.com/ostreedev/ostree/issues/975

Closes: #982
Approved by: jlebon
2017-06-30 19:37:23 +00:00
Colin Walters 1782a1c279 lib/pull: Move check for requested content earlier
This is prep for a later patch; currently the logic is unchanged, but we'll need
this if we make local imports async.

Closes: #982
Approved by: jlebon
2017-06-30 19:37:23 +00:00
Colin Walters aa26db825f lib/commit: Port a few minor functions to new style
Not sure why these weren't converted before.

Closes: #984
Approved by: jlebon
2017-06-29 22:07:23 +00:00
Colin Walters d57410a7e6 lib: Add a helper to convert struct stat → GFileInfo
It's more natural for a few calling places. Prep for patches to go the other
way, which in turn are prep for adding a commit filter v2 that takes `struct
stat`.

`ot_gfile_type_for_mode()` was only used in this function, so inline it here.

Closes: #974
Approved by: jlebon
2017-06-29 18:17:28 +00:00
Colin Walters 1a9a473580 cmdline/pull: Print final status even if noninteractive
Previously, `ostree pull` was silent if not on a tty.  I don't
see a reason not to print the final status line at least.  This
is prep for more work in the test suite, so I can write assertions
on the output.

But it should also be nicer for people who e.g. do an `ostree pull` in a Jenkins
job or whatever.

Closes: #981
Approved by: jlebon
2017-06-29 16:16:25 +00:00
Colin Walters 90e0d56332 tree-wide: Replace various uses of archive-z2 → archive
The `-z2` is annoying now since it's really a legacy; we've long
since supported typing `archive`.  Convert the docs fully and
explain that.

Also do some (but not all) of the tests just to encourage newer tests to use
`archive` too.

Closes: #980
Approved by: jlebon
2017-06-29 16:00:13 +00:00
Colin Walters 8d586a9da0 lib/pull: Don't fetch detached metadata twice for local pulls
Obviously very minor, but I noticed this while working on `pull --reference`. If
we have a local repo, we'll have already done a hardlink and copied the detached
metadata too, so there's no reason to request it again via the fetcher path.

Closes: #978
Approved by: jlebon
2017-06-29 15:48:59 +00:00
Colin Walters 2013db0527 tests: Fix assert_files_hardlinked
It was always succeeding because we were trying to stat the inode number, and
failing, and thus getting the empty string for both, which compared as true.

Regression from:
<https://github.com/ostreedev/ostree/commit/74e3581e>

Noticed this while working on
<https://github.com/ostreedev/ostree/pull/974>
and looking at the test results.

Closes: #976
Approved by: jlebon
2017-06-29 15:34:57 +00:00
Colin Walters ea15025c19 lib/pull: Some small style porting
I'd mostly been avoiding this file since there's always patches outstanding, but
these few functions shouldn't conflict much.

Closes: #979
Approved by: jlebon
2017-06-29 15:21:06 +00:00
Colin Walters 250e305f73 lib/repo: Port bareuser-conversion stat to bare load
I noticed this is a simple call that's useful to port to the new internal-only
non-allocating API.

Closes: #977
Approved by: jlebon
2017-06-29 15:07:49 +00:00
Colin Walters 064d7bffef lib/deltas: More porting to new code style
Just noticed some of this while working on the previous tmpfile bits.

Closes: #973
Approved by: jlebon
2017-06-29 14:46:18 +00:00
Colin Walters ab9fef5279 lib/commit: Refactor non-failable size indexing function
It can't throw, so remove the `GError` machinery.

Closes: #973
Approved by: jlebon
2017-06-29 14:46:18 +00:00
Colin Walters 6f2ea23e8a libutil: Add a helper for O_TMPFILE + mmap()
I added `glnx_open_anonymous_tmpfile()`, but then later noticed
that the usage of this was really to be combined with `mmap()`,
and we had two versions of that in the delta code.  Add a helper.

(Bigger picture...how is this different from glibc's "mmap() of /dev/zero"
 approach for large chunks? One advantage is the storage can be "swapped" to
 `/var/tmp`, but still deleted automatically, rather than requiring swap space)

Closes: #973
Approved by: jlebon
2017-06-29 14:46:18 +00:00
Colin Walters ba918e49c5 tree-wide: Misc porting to newer libglnx APIs
- Use the new tmpfile bits
 - `glnx_try_fallocate`
 - `glnx_renameat()`

Depends: https://github.com/GNOME/libglnx/pull/57

Update submodule: libglnx

Closes: #970
Approved by: jlebon
2017-06-28 15:27:56 +00:00
Colin Walters 5776d5dcc0 Port to GLnxTmpfile
There's lots of mechanically replacing `OtTmpFile` with `GLnxTmpfile`;
the biggest changes are in the commit path.  Symlink commits are now
very clearly separated from regular files.  Symlinks are `OtCleanupUnlinkat`,
and regular files are `GLnxTmpfile`.

The commit codepath separates those as `_ostree_repo_commit_path_final()` and
`_ostree_repo_commit_tmpf_final()`. A nice aspect of all of this is that they
both *consume* the temporary on success. This avoids an extra spurious
`unlink()` call.

One of the biggest bits of code motion is in `commit_loose_regfile_object()`,
which no longer needs to care about symlinks. For the most parth though it's
just removing conditionals.

Update submodule: libglnx

Closes: #958
Approved by: jlebon
2017-06-27 22:02:14 +00:00
Colin Walters 7d0e6fb63f ci: Actually run installed tests again
We did it *again* 😭

Closes: #969
Approved by: jlebon
2017-06-27 21:21:18 +00:00
Colin Walters 4796cd0454 pull: Check free space when pulling deltas
Computing download/storage size for `archive` pulls is hard; there's
`OSTREE_REPO_COMMIT_MODIFIER_FLAGS_GENERATE_SIZES` which was from a
pre-static-deltas effort by Endless, but we aren't currently making use of this
much.

Static deltas were designed to solve this problem; we have the total
uncompressed size. Let's check free space before doing a delta pull.

Related: https://github.com/ostreedev/ostree/issues/962

Closes: #963
Approved by: jlebon
2017-06-27 18:42:00 +00:00