It's hard right now to do a full port to the new libglnx tmpfile
API since there are complex cases in the commit path which deal
with symlinks as well.
Let's make things more gradual by introducing the important part (struct with
autocleanup) here in libotutil, port what we can. This will make a future
complete port easier.
Closes: #871
Approved by: jlebon
Follow up to a previous patch that addressed a double-close; I
realized we already had a helper for doing "open dfd iter, do nothing
if we get ENOENT". Raise it to libotuil, and port all consumers.
Closes: #863
Approved by: jlebon
The keyring isn't large, so let's just fall back to copying it
rather than requiring `renameat()`.
Prep for `ostree_repo_open_at()`.
Closes: #821
Approved by: jlebon
`perf record ostree checkout ...` for a bare-user repo was telling
me we were spending a good 13% of our time in the depchain of `ot_lgexattrat()`.
The problem here is that traversing the `/proc` path turns out to be
somewhat expensive - there are LSM (SELinux) checks, etc.
For regular files, opening and just getting the xattr, then closing is still
quite cheap. For symlinks, we'll always need to open anyways.
This appears to shave about ~0.1 seconds off of a checkout of
`fedora-atomic/25/x86_64/docker-host` on my workstation.
Oh, and this was the last user of `ot_lgexattrat()` so we can kill it, which is
nice - the xattr code should really live in libglnx.
Closes: #796
Approved by: jlebon
I happened to read this file and realized there's a lot of cruft left over from
the time when I liked `GFile` and `malloc()`ing like 50 times just to make a
pathname string. Delete it.
Closes: #767
Approved by: jlebon
Add an OpenSSL backend to the checksum input stream, which is where we do a lot
of checksumming (object commit, static deltas).
The raw OpenSSL performance is
[approximately double](https://gist.github.com/cgwalters/169349fd1c06fd4fb4d3a7ce33303222) on
my laptop; not only does OpenSSL have e.g. hand-tuned x86_64 assembly, the
current implementation uses the
[Intel SHA extensions](https://en.wikipedia.org/wiki/Intel_SHA_extensions).
Another reason to do this is I was idly thinking about adding
[Curve25519](https://en.wikipedia.org/wiki/Curve25519) signatures (like e.g.
Alpine does) instead of/in addition to GPG. The rationale for that is
that GPG is pretty heavyweight, both in code footprint and the simple
fact that EC keys are way smaller.
I didn't benchmark ostree with this; we have bigger performance problems
really like the fact we just malloc way too much. But, it's a step
in the right direction I think in combination with the libcurl work
where we're linking to openssl anyways.
Closes: #738
Approved by: jlebon
The current `OstreeChecksumInputStream` is public due to a historical
mistake. I'd like to add an OpenSSL checksum backend, but that's
harder without breaking this API.
Let's ignore it and create a new private version, so it's easier to do the
GLib/OpenSSL abstraction in one place.
Closes: #738
Approved by: jlebon
While looking at a slow update issue (which I'm guessing is
unpredictable I/O latency in an OpenStack instance), I noticed
in one of the traces we were inside a fsync here.
Dropping the fsync here is just another of a long series of unwinding
them - we `syncfs()` the sysroot fd and `/boot` and we have a big
`sync()` anyways.
Closes: #508
Approved by: jlebon
Lots and lots of preparation led to this moment - when nothing
apparent changes for users! Woo!
But seriously, having the extra dependency is a minor annoyance, and
in the big picture I think the libgsystem idea was wrong - we need to
land things in GLib, and use git submodules for API-unstable or
Linux-specific sharing. For a lot of OSTree, the libgsystem `GFile*`
orientation was also wrong, we really want fd-relative.
Closes: #444
Approved by: jlebon
We have a better API now, drop use of the internal helper, which also
depended on libgsystem.
This required bumping libglnx to pull in a fix.
Closes: #429
Approved by: giuseppe
Import `gs_file_enumerator_iterate()` for the next six months or
so...after RHEL 7.3 is released I'm strongly considering hard
requiring 2.46 or so.
Likely at some point we should figure out how to share more "glib
backport" code with NetworkManager at least.
Closes: #341
Approved by: jlebon
This was the last caller of libgsystem that isn't
`gs_file_get_path_cached()`. I think the use case ostree has where
the same code can be called via command line and via a shared library
*and* via a daemon is rather unusual, so let's just copy the code for
logging from libgsystem into here.
For example rpm-ostree hard depends on a daemon mode, so it'll just
use `sd_journal` directly.
Closes: #341
Approved by: jlebon
Just noticed this while reading some code, we didn't have many manual
`out: close()` bits left, this pushes us over the edge to autocleanup
almost everywhere.
Closes: #332
Approved by: jlebon
This kills another GSystem consumer...I think down the line I'd like
to do something like "detect whether file is > 1k if so, mmap,
otherwise just readall()" so we can use this helper in more places.
Closes: #319
Approved by: jlebon
We have a lot of "allow_noent" type wrapper functions since
a common pattern is to allow files to not exist, but still
throw cleanly on other issues.
This is another instance of that, and cleans up duplicated error
handling code.
Part of this is prep for moving away from `GFile` consumers.
Closes: #319
Approved by: jlebon
In addition to generic fd relative porting,
this is a necessary preparatory step for libglnx porting, because
when I tried to use `g_mapped_file_new` I hit an issue with
it using a different error domain from GIO.
Thankfully libglnx consistently uses the GIO error domain, and here
we're now using it for the `open()` call.
Closes: #317
Approved by: jlebon
This is similar to changes Krzesimir has been doing recently - we
really don't need the ergonomics of floating refs since we have
autocleanups.
We should continue to change most of our code to sink refs.
Specifically here it was pretty broken that the `_map()` API was
sinking but the other two weren't, and this broke some refactoring I
was trying to do later.
Closes: #317
Approved by: jlebon
There's no need to allocate the variant builder on a heap, so allocate
it on the stack and avoid a memory leak at the same time.
Closes: #307
Approved by: cgwalters
This can be used as a fingerprint to determine whether two
OstreeSePolicy objects are equivalent.
Also add documentation for ostree_sepolicy_get_name().
Closes: #219
Approved by: cgwalters
⚠️ There is a notable spiked pit trap here around
`posix_fallocate()` and `errno`. This has bit other projects,
see e.g.
7bb87460e6
Otherwise the port was straightforward.
Right now though, almost all of the details of deltas are private, so
we can't do the "honest thing" and have the command line just use the
shared library.
Eventually some of this should appear in the API, but for now add
command line which is useful for debugging.
There might be a race here in that we create new symlink files *after*
calling `syncfs`, and they are not guaranteed to end up on disk.
Rework the code so that we create symlinks before, and then only
rename them after (and `fsync()` the directory for good measure).
Additional-fixes-by: Giuseppe Scrivano <gscrivan@redhat.com>
Tested-by: Giuseppe Scrivano <gscrivan@redhat.com>
This still needs verification that we're fixing a real bug; but I'm
fairly confident this won't make the fsync situation worse.
https://bugzilla.gnome.org/show_bug.cgi?id=755595
An OSTree user noticed that `ostree fsck` would produce `missing
object` errors in the case of interrupted pulls.
It's possible to do e.g. `ostree pull --subpath=/usr/share/rpm ...`,
which gets you just that portion of the commit. The use case for this
was being able to see what changes would appear in an update before
actually downloading all of it.
(I think this would be better covered by static deltas, but those
aren't final yet, and `--subpath` predates it)
Further, `.commitpartial` is used as a successor to the `transaction`
symlink for more precise knowledge in the case where a pull was
interrupted that we needed to resume scanning.
So it makes sense for `ostree fsck` to be aware of it.
The function returns a gboolean, replace g_return_if_fail with
g_return_val_if_fail.
Add similar checks to the other functions.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Starting down the path of not using libgsystem. The main win here
will be code sharing between ostree/rpm-ostree as well as going down
the path of not using GFile * for local files.
This is a noticeable cleanup, and fixes another big user of GFile* in
performance/security sensitive codepaths.
I'm specifically making this change because the static deltas code was
leaking temporary files, and cleaning that up nicely would be best if
we were fd relative.
This is just an efficiency optimization. We're getting fairly close
to all of the hot code paths using `*at()`.
Note that we end up maintaining a half-duplicate code path set here,
because we still need to support commits from an arbitrary GFile *,
which in a possible common case is an OSTree commit.
I think it's worth it though.
For Fedora and potentially other distributions which use globally
distributed mirrors, metalink is a popular solution to redirect
clients to a dynamic set of mirrors.
In order to make metalink work though, it needs *one* file which can
be checksummed. (Well, potentially we could explode all refs into the
metalink.xml, but that would be a lot more invasive, and a bit weird
as we'd end up checksumming the checksum file).
This commit adds a new command:
$ ostree summary -u
To regenerate the summary file. Can only be run by one process at a
time.
After that's done, the metalink can be generated based on it, and the
client fetch code will parse and load it.
https://bugzilla.gnome.org/show_bug.cgi?id=729585
To be really sure that any directory entries have hit disk we need to
call fsync() on the directory fd. This API allows us to conveniently
create a directory hierarchy, fsyncing all of it along the way.
Let's be a bit more conservative here and actually fdatasync() the
configurations we're generating.
I'm seeing an issue at the moment where syslinux isn't finding the
config sometimes, and while I don't think this is the issue, let's try
it.
This has a very basic level of functionality (deltas can be generated,
and applied offline). There is only some stubbed out pull code to
fetch them via HTTP.
But, better to commit this now and improve it from a known starting
point, rather than have it languish in a branch.
Add an optional dependency on gpgme to add GPG signatures into the
detached metadata, with the key "ostree.gpgsigs", as an "aay", an
array of signatures (treated as binary data).
The commit command gains a --gpg-sign=<key-id> argument. Also add an
argument --gpg-homedir to set the GPG homedir where we look for
keyrings.
We used to have a version of this, but since I'm trying to use
GBytes more, this became a more common operation, and it's annoying
to type out the whole G_VARIANT_TYPE ("ay") each time, and pass
TRUE for trusted.
https://bugzilla.gnome.org/show_bug.cgi?id=706031
We may revive this later, but commits in their current form aren't
very useful for humans to read, so it doesn't make sense to have a
tool to show a history of useless stuff.
More interesting things are diffs between commits, object statistics,
etc.
Colin Walters
Alexander Larsson
Krzesimir Nowak
Jonathan Lebon
Giuseppe Scrivano
Matthew Barnes
Daniel Svensson
Jeremy Whiting
Stef Walter