Commit Graph

455 Commits

Author SHA1 Message Date
Colin Walters e127070550 repo: Only use mmap() for metadata > 16k
See http://stackoverflow.com/questions/258091/when-should-i-use-mmap-for-file-access
and
https://lwn.net/Articles/591978/

I didn't really notice much performance difference in some small
tests, but I happened to be stracing and realized we were `mmap()`ing
even for 50 bytes which is not very useful, so let's not do it.

Closes: #489
Approved by: alexlarsson
2016-09-08 14:56:30 +00:00
Colin Walters 3ef4cc2e5b lib: Add an API to list only "our" objects, fix prune to use it
When doing a prune, we should not try to delete objects in parent
repos, since it'll fail.  There is a bigger discussion about the
semantics of `parent=` to be had, but this will fix trying to use
`ostree prune --repo=/ostree/repo/extensions/rpmostree/pkgcache`.

Closes: https://github.com/ostreedev/ostree/issues/467

Closes: #471
Approved by: jlebon
2016-08-25 20:01:37 +00:00
Colin Walters 683e060099 repo: Add prefixes to errors for querying size/deleting
I hit these while causing errors in prune.  Let's add the specific
object we were looking for.

Closes: #471
Approved by: jlebon
2016-08-25 20:01:37 +00:00
Dan Nicholson b1d13bb356 repo: Really ignore progress changed user data
The documentation says this is ignored, implying that you should pass
NULL to it. However, the function immediately returns in this case even
though the argument isn't used anywhere.

Closes: #458
Approved by: cgwalters
2016-08-12 14:56:07 +00:00
Colin Walters 6b089304bf repo: Drop more internally unused GFile members
I forgot to actually remove `config_file` in the previous
commit, the txn lock hasn't been used in a long time, and
for the uncompressed cache, everything uses the fd already.

Closes: #433
Approved by: giuseppe
2016-08-08 08:31:59 +00:00
Colin Walters d55655b096 repo: Drop internal GFile config_file
The remote parsing code still uses GFiles but this is a start.

Closes: #432
Approved by: giuseppe
2016-08-05 08:26:07 +00:00
Colin Walters 18a576c390 lib: Drop GFiles for self->{objects_dir,state_dir}
Everything that used to reference these is now fd-relative.

Closes: #432
Approved by: giuseppe
2016-08-05 08:26:07 +00:00
Colin Walters 24671f7d25 repo: Port to g_autoptr() rather than old style cleanup macros
This notably kills a few libgsystem uses here.

Closes: #423
Approved by: giuseppe
2016-08-01 21:00:12 +00:00
Colin Walters 421c902125 repo: Port sign_data() to libglnx tmpfile APIs
Part of dropping libgsystem.

Closes: #423
Approved by: giuseppe
2016-08-01 21:00:12 +00:00
Colin Walters 5334758ba7 repo: Make ostree_repo_create() nonfatal on existing repos
In general we want to support "idempotentcy" or "state
synchronization" across interruption.  If a repo is only partially
created due to a crash or whatever, it's hard for a user to know that.
Let's just make `ostree_repo_create()` idempotent. Since all we're
doing is a set of `mkdirat()` invocations, it's quite simple.

This also involved porting to fd-relative, which IMO makes the
code a lot clearer.

Closes: #422
Approved by: 14rcole
2016-08-01 15:12:14 +00:00
Dan Nicholson 4ebcd57fec repo: Fix annotation for ostree_repo_add_gpg_signature_summary
GI needs to know this is a NULL terminated array of gchar or you'll
segfault using it from a binding.

Closes: #387
Approved by: cgwalters
2016-07-08 22:22:28 +00:00
Alexander Larsson fc4a7ec35e pull: Correctly handle repo->parent_repo when applying static deltas
In flatpak i was using a parent repo, and it failed to update
with ENOENT when dispatching an set-read-source opcode, because the
object it referenced was in the parent repo.

This fixes that by making _ostree_repo_read_bare_fd look
at parent_repo.

Closes: #362
Approved by: cgwalters
2016-06-23 11:36:33 +00:00
Mathnerd314 23049bbd01 core: Add OSTREE_OBJECT_TYPE_COMMIT_META
This is cleaner than the loose_path_with_suffix approach

Closes: #359
Approved by: cgwalters
2016-06-22 16:10:01 +00:00
Mathnerd314 4cb77c51db core: Use OSTREE_SHA256_STRING_LEN instead of 64
Closes: #359
Approved by: cgwalters
2016-06-22 16:10:01 +00:00
Colin Walters f6ce04e480 libglnx porting: Drop uses of gs_file_openat_noatime
We're not really doing the "noatime" thing anymore.

Closes: #341
Approved by: jlebon
2016-06-21 18:24:17 +00:00
Colin Walters 90b9a06277 lib: Use g_file_enumerator_iterate() if available, with fallback
Import `gs_file_enumerator_iterate()` for the next six months or
so...after RHEL 7.3 is released I'm strongly considering hard
requiring 2.46 or so.

Likely at some point we should figure out how to share more "glib
backport" code with NetworkManager at least.

Closes: #341
Approved by: jlebon
2016-06-21 18:24:17 +00:00
Colin Walters 4819b44189 libglnx porting: Use of GSDirFdIterator
This one was pretty simple.  One of the uses in `repo.c` was just a
leftover variable.

Closes: #341
Approved by: jlebon
2016-06-21 18:24:17 +00:00
Colin Walters 651f4bc3b9 repo: Avoid a possible divide by zero in progress
The previous code was subject to a divide by zero if less than a
second had passed.  Rework it so we only do the divide if more than a
second has passed.

Closes: #349
Approved by: Mathnerd314
2016-06-17 02:30:13 +00:00
Colin Walters ab47a8a030 Add "archive" as an alias for "archive-z2"
I find the "-z2" is really a long ago relic of the past when I changed
the format.  We no longer have anything to do with the original
`archive`, so let's start allowing people to type `--mode=archive`
which just looks saner.

At some point later I'll update the docs too, but it'll be an annoying
transition period as we'll have to say "On older OSTree, use -z2" etc.

Closes: #346
Approved by: giuseppe
2016-06-16 14:34:23 +00:00
Colin Walters 7847bc7394 lib: Port some manual close() cleanups to be glnx_fd_close
Just noticed this while reading some code, we didn't have many manual
`out: close()` bits left, this pushes us over the edge to autocleanup
almost everywhere.

Closes: #332
Approved by: jlebon
2016-06-13 14:58:55 +00:00
Colin Walters 0f7bf7be95 lib: Fix a leak in remote parsing
As the docs say, `g_regex_match()` still allocates a match even if it
returns `FALSE`.  Using `g_autoptr` is just plain better.

Closes: #292
Approved by: krnowak
2016-06-09 21:10:35 +00:00
Mathnerd314 1b88dc7f90 docs: Get API docs working again
This changes around a few things that didn't work for me:
* Section names seem to be ostree-* instead of libostree-*
* Also XML files are ostree-* (they didn't show up at all)
- gtk-doc doesn't seem to parse const _OSTREE_PUBLIC correctly
* pull documentation is now on the actual functions rather than stubs
* Update gitignore with some more files

And there some changes to make gtk-doc give fewer warnings (not finished)

Closes: #327
Approved by: cgwalters
2016-06-09 18:15:49 +00:00
Mathnerd314 693b4ec4be pull: Move pull and pull_onedir back into ostree-repo.c
They call pull_with_options, which will fail anyway.

Closes: #327
Approved by: cgwalters
2016-06-09 18:15:49 +00:00
Mathnerd314 da5cb099a6 pull: Move libsoup-related code from ostree-repo.c to ostree-repo-pull.c
This centralizes the ifdef's in one file, which will make it
easier to write new pull backends.

ostree-repo-pull.c is now built unconditionally

Closes: #327
Approved by: cgwalters
2016-06-09 18:15:49 +00:00
Mathnerd314 0e9a875393 repo: use OSTREE_TIMESTAMP (=1) for checked-out files
1 is a better choice than 0 because some programs use 0
as a special value; for example, GNU Tar warns of an
"implausibly old timestamp" with 0.

Closes: #330
Approved by: cgwalters
2016-06-09 18:04:55 +00:00
Colin Walters f94b191ae5 repo: (minor) Drop a GFile usage in signature creation
Just one user of hundreds but, the rest of the function was ported
now.

Closes: #319
Approved by: jlebon
2016-06-09 14:39:09 +00:00
Colin Walters 70e5489258 lib: Add a helper for mmap->bytes with openat(), use it in repo
This kills another GSystem consumer...I think down the line I'd like
to do something like "detect whether file is > 1k if so, mmap,
otherwise just readall()" so we can use this helper in more places.

Closes: #319
Approved by: jlebon
2016-06-09 14:39:09 +00:00
Colin Walters 3a03a35071 lib: Add `_ALLOW_NOENT` flag to internal variant mapping API
We have a lot of "allow_noent" type wrapper functions since
a common pattern is to allow files to not exist, but still
throw cleanly on other issues.

This is another instance of that, and cleans up duplicated error
handling code.

Part of this is prep for moving away from `GFile` consumers.

Closes: #319
Approved by: jlebon
2016-06-09 14:39:09 +00:00
Colin Walters 5a996c04de glnx porting: Drop use of gs_file_openat_noatime
As noted previously the "noatime" thing is mostly obsoleted by
relatime.

Closes: #319
Approved by: jlebon
2016-06-09 14:39:09 +00:00
Krzesimir Nowak b5da2f524c repo: Fix build on 32-bit systems
On 32-bit systems the modifier for printing 64bit values should be
%llu instead of %lu. Just use appriopriate macros that do the right
thing.

Closes: #329
Approved by: giuseppe
2016-06-09 11:25:29 +00:00
Colin Walters 900c085f21 libglnx porting: Port away from gs_file_map_noatime()
The "no atime" thing was mostly useful only before "relative atime"
updates landed.  Users who care about performance will turn it off
entirely anyways.

Closes: #316
Approved by: jlebon
2016-06-02 14:52:40 +00:00
Colin Walters c148631a98 lib: Drop GFile variant mapping API for fd-relative
In addition to generic fd relative porting,
this is a necessary preparatory step for libglnx porting, because
when I tried to use `g_mapped_file_new` I hit an issue with
it using a different error domain from GIO.

Thankfully libglnx consistently uses the GIO error domain, and here
we're now using it for the `open()` call.

Closes: #317
Approved by: jlebon
2016-06-01 15:02:41 +00:00
Sam Spilsbury a189b19fd0 repo: Display estimated time remaining when pulling
Bug 765429 said that not having a time estimate can be annoying
when working with large pulls.

There isn't any complex time estimation logic here - we just take
the number of bytes remaining and do a linear projection of
the bytes per second rate at the current point in time.

Closes: #318
Approved by: cgwalters
2016-06-01 14:09:41 +00:00
Colin Walters 6e57987c7c libglnx porting: Port away from gs_file_ensure_directory()
The init-fs code in particular gets (IMO) a lot cleaner being fd
relative.

Closes: #312
Approved by: giuseppe
2016-05-30 11:33:28 +00:00
Colin Walters e5eb6f3a22 libglnx porting: Port callers of gs_file_unlink to just unlink()
In the second case it's actually less code too.

Closes: #312
Approved by: giuseppe
2016-05-30 11:33:28 +00:00
Krzesimir Nowak 4929ab4033 repo: Add functions for verifying any data in repository
This can be useful for validating the 3rd party data that is put in
the extensions directory and is signed with the same keys as commits
or the summary file.

Closes: #310
Approved by: cgwalters
2016-05-27 11:20:00 +00:00
Krzesimir Nowak 27f0c6980a repo: Factor out setting up a GPG verifier
Moved out setting up a GPG verifier to a separate function, as I would
like to use it for the any data verification function in the following
commit.

Closes: #310
Approved by: cgwalters
2016-05-27 11:20:00 +00:00
Krzesimir Nowak 582169f83e repo: Add a missing Returns paragraph
Closes: #310
Approved by: cgwalters
2016-05-27 11:20:00 +00:00
Krzesimir Nowak 89bfb1d503 repo: Factor out the check of gpg result to a separate function
I plan to add a function for verifying any data which may return the
error about lack of trusted signatures, so let's avoid the redundancy
and put the check in the separate function.

Closes: #310
Approved by: cgwalters
2016-05-27 11:20:00 +00:00
Krzesimir Nowak cb45456708 libostree: Fix bracket-missing warnings
Apparently I got the bracketing wrong in
862e6ecdcc58f025696b1394adfc0fcf7322df23:

src/libostree/ostree-repo.c: In function 'ostree_repo_delete_object':
src/libostree/ostree-repo.c:3538:11: warning: missing braces around
initializer [-Wmissing-braces]
           g_auto(GVariantBuilder) builder = {0,};

Closes: #298
Approved by: cgwalters
2016-05-13 12:22:59 +00:00
Krzesimir Nowak 862e6ecdcc libostree: Variant-related leak plugs and fixes
This tries to avoid leaking GVariantBuilders and GVariants in some
situations. The leaks were usually happening when some error occurred
or because of unclear variant ownership situation.

The former is mostly about making sure that g_variant_builder_clear is
called on builders that didn't finish their variant building process.

The latter is surely more work - sometimes the result of
g_variant_builder_end() should not be passed directly to a function,
but rather stored in a g_autoptr(GVariant), sunk and then passed to a
function. IMO, with an advent of g_autoptr, GVariants should be always
sunk instead of relying on some receiver function sinking it. This
would make an easy-to-follow policy of always sinking your
variants. Functions could then assume that the passed variant is
already sunk. These leaks are still happenning in commands, but they
are less harmful, since that code will not be used by some daemon as a
library routine.

Closes: #291
Approved by: cgwalters
2016-05-12 11:17:09 +00:00
Krzesimir Nowak aa946cc136 repo: Allow using options for fetching summary
This adds a _with_options variant of the
ostree_repo_remote_fetch_summary function, so we can tell the fetcher
to use a specific URL instead taking it from the remote config.

Closes: #290
Approved by: cgwalters
2016-05-10 13:47:36 +00:00
Krzesimir Nowak bdd73913fb repo: Plug a leak
The base URI created in this line was always (sans erroneous
situations) overwritten in the code block below without freeing it
previously, so it leaked.

Closes: #290
Approved by: cgwalters
2016-05-10 13:47:36 +00:00
Jonathan Lebon 65e7969b0b OstreeRepo: add extensions directory
It's very useful for third-party applications to have someplace to store
their data guaranteed to be on the same device as the repo (thus
ensuring hardlinks) while still being shielded away from any of OSTree's
timely garbage collections.

We create a new "extensions/" subdirectory where apps can include
whatever they wish in "extensions/myapp/". This subdirectory is
completely unmanaged by ostree.

NB: I didn't bother making it a member of the OstreeRepo proper since we
don't really use it for anything else yet.

Closes: #286
Approved by: cgwalters
2016-05-09 14:43:16 +00:00
Colin Walters 6724519080 libglnx porting: Migrate to glnx_stream_fstat()
I ended up deciding to move this one into libglnx, seems like
something other libglnx-using software might want to do, even though
xdg-app doesn't right now.

Closes: #282
Approved by: jlebon
2016-05-06 14:29:59 +00:00
Colin Walters 7021c4f876 repo: Make repo/tmp expiry configurable via tmp-expiry-seconds
We were arbitrarily only deleting content after exactly one day.  Some
use cases may want something else; make it configurable.

Closes: #170
Approved by: jlebon
2016-05-02 18:44:44 +00:00
Colin Walters a56ba6081a repo: Clean up staging directory for previous boot IDs
We had a policy of cleaning up all files in `$repo/tmp` older
than one day, but we should really clean up previous bootid staging
directories too, as they can potentially take up a lot of disk space.

https://bugzilla.gnome.org/show_bug.cgi?id=760531

Closes: #170
Approved by: jlebon
2016-05-02 18:44:44 +00:00
Colin Walters e3ec83a934 repo: Add OSTREE_BOOTID override envvar for debugging
It's useful for test cases to be able to influence this.

Conflicts:
	src/libostree/ostree-repo.c

Closes: #170
Approved by: jlebon
2016-05-02 18:44:44 +00:00
Colin Walters 15b3cab65e repo: Add OSTREE_REPO_TEST_ERROR=pre-commit env var
Setting this causes commit to error out.  There are other ways we
could do this in a more sophisticated fashion, such as via SystemTap
etc.  But this has low-tech applicablity, works as non-root.

The reason I'm adding this is so that we can add test cases for
cleanup of the `tmp/staging-` directory.

Closes: #170
Approved by: jlebon
2016-05-02 18:44:44 +00:00
Colin Walters 3111248373 repo: Fix reading repos on read-only media
I have a cache drive I often mount read-only, and the previous commit
for opening `tmp/cache` broke since `errno == EROFS`, not `EPERM`.

It turns out we already had the concept of a "writable" repo, so just
piggy back off that.

Closes: #281
Approved by: giuseppe
2016-05-02 14:27:33 +00:00