I often want to have "idempotent" systems that iterate to a known
state. If after generating a commit, the system is interrupted, I'd
like the next run to still generate a delta. But we don't want to
regenerate if one exists, hence this option.
Closes: #375
Approved by: jlebon
I've seen a few people hit this and wonder why checkouts are slow/take
space. Really, ensuring this happens is the *point* of OSTree.
Physical copies should be a last resort fallback for very unusual
situations (one of those is rpm-ostree checking out the db since
librpm doesn't know how to read from libostree).
Even I hit the fact that `/var` is a mountpoint disallowing hardlinks
with `/ostree` once and was confused. =)
Add this to the rofiles-fuse test case because it creates a mount
point.
Closes: #368
Approved by: jlebon
The program is called ro*files* and ostree creates physical
copies of directories, so changing them is fine.
I hit this when trying to do a copy checkout onto an rofiles-fuse
mount.
Closes: #368
Approved by: jlebon
Import `gs_file_enumerator_iterate()` for the next six months or
so...after RHEL 7.3 is released I'm strongly considering hard
requiring 2.46 or so.
Likely at some point we should figure out how to share more "glib
backport" code with NetworkManager at least.
Closes: #341
Approved by: jlebon
This was the last caller of libgsystem that isn't
`gs_file_get_path_cached()`. I think the use case ostree has where
the same code can be called via command line and via a shared library
*and* via a daemon is rather unusual, so let's just copy the code for
logging from libgsystem into here.
For example rpm-ostree hard depends on a daemon mode, so it'll just
use `sd_journal` directly.
Closes: #341
Approved by: jlebon
I was easily reproducing a hang on pulls with thousands of requests on
current git master. The initial symptom seemed to be that there are
multiple code paths where we don't invoke
`session_thread_process_pending_queue()`. We really need to do
that any time we remove something from the outstanding queue,
to ensure it gets filled again.
A further issue is that we were tying the lifecycle of the pending
object to the `GTask`, but the task could be unref'd from the main
thread (via a `GSource` on the main thread), and that introduced
threadsafety issues, because the hash table and other data suddenly
could be concurrently modified.
Both of these need to be fixed together. First, we introduce
`Arc<Pending>`, and ensure that both the main and worker threads hold
references.
Second, we ensure that we re-process the queue *immediately* whenever
a task is done, inside the worker thread, rather than doing it
incidentally via an unref. This architecture is quite similar to what
the outside pull code is doing.
Closes: #350
Approved by: jlebon
For some reason I'm really not inclined to debug right now, `libtool`
is eating the `-fsanitize=address` argument when linking
`libreaddir-rand.so`, which causes failures since we're trying to
`LD_PRELOAD`.
Closes: #351
Approved by: jlebon
It's unfortunate that in automake one has to explicitly include the globa
`$(AM_CFLAGS)` if one sets `CFLAGS`, and similarly for other variables.
I'm trying to use `-fsanitize=address`, and not including it was
causing linker failures.
We also weren't inheriting the global warnings etc., so I had to fix a
decl-after-statement.
Closes: #351
Approved by: jlebon
The previous code was subject to a divide by zero if less than a
second had passed. Rework it so we only do the divide if more than a
second has passed.
Closes: #349
Approved by: Mathnerd314
I find the "-z2" is really a long ago relic of the past when I changed
the format. We no longer have anything to do with the original
`archive`, so let's start allowing people to type `--mode=archive`
which just looks saner.
At some point later I'll update the docs too, but it'll be an annoying
transition period as we'll have to say "On older OSTree, use -z2" etc.
Closes: #346
Approved by: giuseppe
Just noticed this while reading some code, we didn't have many manual
`out: close()` bits left, this pushes us over the edge to autocleanup
almost everywhere.
Closes: #332
Approved by: jlebon
The recent memleak fixes motivated me to look at the bitrotted code to
run invocations of `ostree` in the test suite underneath valgrind.
There are a few things here. First, update suppressions file from
libhif, since I recently worked on it.
When running *uninstalled* as we now support, we need
`libtool --mode=execute` in the mix so it expands out to
the uninstalled binary and we don't valgrind the intermediate shell.
However, it's harder than that because we chdir into a tmpdir,
which defeats the libtool logic. AFAICS, the only fix for this
is to determine the realbin path before we chdir, and then unfortunately
we need to change every use of `ostree` to `${OSTREE}` =(
Then this immediately breaks for me on RHEL7 because my ancient
copy of `valgrind-3.10.0-16.el7.x86_64` is unaware of syscall 306, i.e.
`syncfs`.
But let's do this first before I dive into that.
Closes: #292
Approved by: krnowak
As the docs say, `g_regex_match()` still allocates a match even if it
returns `FALSE`. Using `g_autoptr` is just plain better.
Closes: #292
Approved by: krnowak
This kills another GSystem consumer...I think down the line I'd like
to do something like "detect whether file is > 1k if so, mmap,
otherwise just readall()" so we can use this helper in more places.
Closes: #319
Approved by: jlebon
We have a lot of "allow_noent" type wrapper functions since
a common pattern is to allow files to not exist, but still
throw cleanly on other issues.
This is another instance of that, and cleans up duplicated error
handling code.
Part of this is prep for moving away from `GFile` consumers.
Closes: #319
Approved by: jlebon
We have had in the past issues with running `ostree_repo_pull()`
multiple times in the same process, embarassingly enough. Nothing in
the current test suite covers this, so let's start.
Closes: #322
Approved by: jlebon
I was extending the C-based test suite to cover more, and noticed that
we weren't aborting if a command failed. That made the tests somewhat
less useful, so let's fix it.
Closes: #321
Approved by: jlebon
The "no atime" thing was mostly useful only before "relative atime"
updates landed. Users who care about performance will turn it off
entirely anyways.
Closes: #316
Approved by: jlebon
In addition to generic fd relative porting,
this is a necessary preparatory step for libglnx porting, because
when I tried to use `g_mapped_file_new` I hit an issue with
it using a different error domain from GIO.
Thankfully libglnx consistently uses the GIO error domain, and here
we're now using it for the `open()` call.
Closes: #317
Approved by: jlebon
This is similar to changes Krzesimir has been doing recently - we
really don't need the ergonomics of floating refs since we have
autocleanups.
We should continue to change most of our code to sink refs.
Specifically here it was pretty broken that the `_map()` API was
sinking but the other two weren't, and this broke some refactoring I
was trying to do later.
Closes: #317
Approved by: jlebon
I happened to have the github page open with my commit after it was
merged, and then noticed a bug.
We still need to throw if we hit a non-`ENOENT` error.
Closes: #315
Approved by: giuseppe
When we added enum type generation, the generated symbols used
the `ostree_` prefix, and at the time that implied they were public.
So we started (if built with libsoup) exporting
`ostree_fetcher_config_flags_get_type`.
I think it's not worth confusing ABI checkers, so let's export the
dummy symbol forever, and switch enums to be private by default.
We should revisit this and also export some auto-generated enum
types for public enums, but that's a separate patch.
Closes: #296
Approved by: gatispaeglis
I was briefly looking at building with clang mostly since it
detects unused variables with cleanup attributes, but then
I hit this fatal error.
It's a hard compile error with `-Wformat-nonliteral` since clang
doesn't know it's a format string.
It's actually very nice to be able to declare loop variables inside
the initializer.
Ideally we could turn off nested functions though.
Closes: #284
Approved by: jlebon
I ended up deciding to move this one into libglnx, seems like
something other libglnx-using software might want to do, even though
xdg-app doesn't right now.
Closes: #282
Approved by: jlebon
The filesystem commit code will never give us potentially hostile
filenames, and when importing from archives, we do some validation.
However, we should be extra paranoid and also add error messages in
the mtree in case someone tries to import a hostile
libarchive-supported format.
Closes: #283
Approved by: jlebon
We were arbitrarily only deleting content after exactly one day. Some
use cases may want something else; make it configurable.
Closes: #170
Approved by: jlebon
We had a policy of cleaning up all files in `$repo/tmp` older
than one day, but we should really clean up previous bootid staging
directories too, as they can potentially take up a lot of disk space.
https://bugzilla.gnome.org/show_bug.cgi?id=760531Closes: #170
Approved by: jlebon
Setting this causes commit to error out. There are other ways we
could do this in a more sophisticated fashion, such as via SystemTap
etc. But this has low-tech applicablity, works as non-root.
The reason I'm adding this is so that we can add test cases for
cleanup of the `tmp/staging-` directory.
Closes: #170
Approved by: jlebon
I have a cache drive I often mount read-only, and the previous commit
for opening `tmp/cache` broke since `errno == EROFS`, not `EPERM`.
It turns out we already had the concept of a "writable" repo, so just
piggy back off that.
Closes: #281
Approved by: giuseppe
To GLnxConsoleRef. There were some subtleties here, for example we
used to reference `GSConsole` inside the progress changed function,
which at first seems like an ABI hazard, because e.g. rpm-ostree or
xdg-app could still be passing a `GSConsole` instance there. Luckily,
it turns out to be compatible to just start calling libglnx here.
Another issue was that due to libglnx's use of the cleanup function,
we needed to ensure we always called `ostree_async_progress_finish()`
*before* the cleanup function was invoked.
Closes: #280
Approved by: giuseppe
It's actually just easier for build systems (e.g. rpm-ostree)
using dracut to use `--add ostree` rather than indirecting
through the conf file.
This makes it easier for yum-managed systems to install ostree without
side effects.
Closes: #279
Approved by: gatispaeglis
There was some leftover intermediate cruft here I noticed
while reviewing another patch:
- We had an output `GFile*` for that was never used
- We required the caller to allocate the loose pathbuf, but
none of them ever reused it
- We had an extra intermediate function
Also while looking at this, I'm now uncertain whether some of the
callers of `_ostree_repo_has_loose_object` should really be invoking
`ostree_repo_has_object()`, but let's leave that aside for now.
Closes: #272
Approved by: alexlarsson
I'm not sure if something changed or there was a race condition , or
if extraterrestrials intervened to make the tests pass before, but
in the Docker container `/dev/fuse` does exist and since we appear
to be running as uid 0 it'll be writable.
We previously discussed that another requirement for FUSE (and most
setuid binaries) is having `CAP_SYS_ADMIN`, so let's check for that in
the bounding set too.
Closes: #266
Approved by: jlebon
We keep forgetting to update `apidoc/ostree-sections.txt`, so let's
start enforcing it. Of course it turns out we had some bugs here
like symbols marked as public but never implemented, etc. Those
are fixed in the prior commits.
Closes: #263
Approved by: giuseppe
This was removed in 14d682305b - long
before we had a stable library.
I again claim removing unimplemented symbols from the header is not an
API/ABI break.
Closes: #263
Approved by: giuseppe
If a symbol falls in a git merkle tree forest, but no one hears it,
did it ever exist?
I'm claiming this isn't an API/ABI break because nothing could
have actually used this. `git log -S checksum_update_meta` leads
me to 38ef75e6e0 which was before
we really had a stable shared library.
Closes: #263
Approved by: giuseppe
It's not quite namespaced enough to have
`/usr/libexec/grub2-15_ostree`, and the Fedora spec file expects
things in `/usr/libexec/ostree`. Changing the spec file would be
annoying as we'd need conditionals.
Closes: #249
Approved by: gatispaeglis
I think it's cleaner if we use `remote_repo_local` to know
that we have a local repo. In reality, it might be nicest
if we didn't even create an `OstreeFetcher` for this case,
but untangling the code is tricky.
Closes: #239
Approved by: alexlarsson
If one happens to not have grub2 installed, the previous commit
would cause us to fall back to `grub-mkconfig` with bad results.
We should likely just do dynamic detection and avoid pain, but
there's no harm in also allowing this to be statically determined.
The automagic here is that if `/etc/os-release` says `ID_LIKE=fedora`,
we know to use `grub2-mkconfig`. But distro scripts should set this
in the build configurations using `--with-grub2-mkconfig=`.
Closes: #240
Approved by: jlebon
It's not working for me in `make check` on a RHEL 7 Workstation,
apparently because no GPG agent is spawned. I'm guessing this has
something to do with the GPG version?
The downside of this is we will be less likely to notice if GPG
changes again and we start leaking agents like we're in The Matrix
Reloaded. But the real solution to that is containers anyways.
Closes: #233
Approved by: smcv
The API supports this, and it's not hard for us to do in the command
line as well. One possible use case is separating "content
generation" in a separate server.
Related: https://github.com/ostreedev/ostree/pull/223Closes: #227
Approved by: jlebon
When I'm doing local development builds, it's quite common for me not
to want to accumulate history. There are also use cases for this on
build servers as well.
In particular, using this, one could write a build system that didn't
necessarily need to have access to (a copy of) the OSTree repository.
Instead, the build system would determine the last commit ID on the
branch, and pass that to a worker node, then sync the generated
content back.
The API supported generating custom commits that don't necessarily
reference the previous commit on the same branch, let's just expose
this in the command line for convenience.
I plan to also support this rpm-ostree.
Closes: #223
Approved by: jlebon
Just keeping my promise to write more documentation. There could be a
lot more to write here, but I'm trying to get a start done.
Closes: #222
Approved by: jlebon
I'm trying to improve the developer experience on OSTree-managed
systems, and I had an epiphany the other day - there's no reason we
have to be absolutely against mutating the current rootfs live. The
key should be making it easy to rollback/reset to a known good state.
I see this command as useful for two related but distinct workflows:
- `ostree admin unlock` will assume you're doing "development". The
semantics hare are that we mount an overlayfs on `/usr`, but the
overlay data is in `/var/tmp`, and is thus discarded on reboot.
- `ostree admin unlock --hotfix` first clones your current deployment,
then creates an overlayfs over `/usr` persistent
to this deployment. Persistent in that now the initramfs switchroot
tool knows how to mount it as well. In this model, if you want
to discard the hotfix, at the moment you roll back/reboot into
the clone.
Note originally, I tried using `rofiles-fuse` over `/usr` for this,
but then everything immediately explodes because the default (at least
CentOS 7) SELinux policy denies tons of things (including `sshd_t`
access to `fusefs_t`). Sigh.
So the switch to `overlayfs` came after experimentation. It still
seems to have some issues...specifically `unix_chkpwd` is broken,
possibly because it's setuid? Basically I can't ssh in anymore.
But I *can* `rpm -Uvh strace.rpm` which is handy.
NOTE: I haven't tested the hotfix path fully yet, specifically
the initramfs bits.
In some cases (such as `ostree-sysroot-cleanup.c`), the surrounding
code would be substantially cleaner if it was also ported to
fd-relative, but I'm going to do that in a separate patch.
That way these patches are easier to review for mechanical
correctness. I used an Emacs keyboard macro as the poor man's
[Coccinelle](http://coccinelle.lip6.fr/).
Since we hard-depend on GLib 2.40, we can start using GSubprocess.
This is part of dropping our dependency on libgsystem, which is
deprecated in favor of libglnx (as well as migrating things to GLib).
I'd like to encourage people to make OSTree-managed systems more
strictly read-only in multiple places. Ideally everywhere is
read-only normally besides `/var/`, `/tmp/`, and `/run`.
`/boot` is a good example of something to make readonly. Particularly
now that there's work on the `admin unlock` verb, we need to protect
the system better against things like `rpm -Uvh kernel.rpm` because
the RPM-packaged kernel won't understand how to do OSTree right.
In order to make this work of course, we *do* need to remount `/boot`
as writable when we're doing an upgrade that changes the kernel
configuration. So the strategy is to detect whether it's read-only,
and if so, temporarily mount read-write, then remount read-only when
the upgrade is done.
We can generalize this in the future to also do `/etc` (and possibly
`/sysroot/ostree/` although that gets tricky).
One detail: In order to detect "is this path a mountpoint" is
nontrivial - I looked at copying the systemd code, but the right place
is to use `libmount` anyways.
This content currently lives here:
<https://wiki.gnome.org/Projects/OSTree/RelatedProjects>. Moving it
into the manual in Markdown:
- Makes it look better
- It's more useful alongside the rest of the docs
- Is much less crummy in general than the GNOME wiki
GNOME Continuous uses ginstest-runner --report-directory, which causes
the tests to save their tmpdirs persistently. This also means the
result directories didn't match the `/(var/)?tmp` regexp, which broke
the ostree tests in GContinuous.
Fix this by simply asserting that the tmpdir either has `.tmpdir` or
nothing.
We were considering using this for Docker integration, but we may end
up going a different architectural path. Anyways, it doesn't hurt to
have the bindings in here - they can do a few things.
I decided to fork some of the core code from
https://github.com/dradtke/gotk3 because...well, what we really need a
GIR-based core generator but I didn't want to start on the fully
correct thing until we knew we wanted it, and this was a quick hack.
Also, let's make a `contrib/` directory for things like this.
Otherwise $(installed_testdir) is empty so we try to put content in
`/`, which I noticed when trying to build an RPM (it works works fine
`sudo make install`).
%Z only uses seconds, so it's possible that we did the commit
in the same second, which made this test racy.
- Switch to full nanosecond precision using '%.Y' so it always differs
- Fix the inverted `cmp` usage
- Add a missing `ok`
This will allow daemons like rpm-ostree to detect if there are any new
deployments efficiently, in combination with using inotify. If there
are any changes, rpm-ostree wants publish them on DBus.
While we're here, add some changes to start doing unit C testing of
the sysroot API.
I want to be able to easily test the C API on actual data in an OSTree
repo. The shell `libtest.sh` has code to generate it. Bridge the two
worlds by introducing a little `libostreetest` library which has a C
API which spawns a shell that runs things in `libtest.sh`.
Yes, this is about as beautiful as it sounds, which is to say, it's
not. But it works!
Note while we were here, I realized we were actually now creating
*two* tmpdirs per test in `make check` because the tap driver was
already doing that. Unify it so we know the C code can rely on it.
And change the command line to use it. rpm-ostree had a copy
of this code, and thus there's a clear reason to have an API.
While we're moving this into API, ensure the mtime on deploy is bumped
after an osname is created, so that daemons like rpm-ostree can notice
changes. (In reality, creating the directory should do this, but
let's be double sure)
Colin Walters