Commit Graph
72 Commits
Author SHA1 Message Date
Alexander Larsson d258b124a2 tests: Exit valgrind tests if a leak is detected
This fails the test if OT_TESTS_VALGRIND is set and valgrind
detects a leak.

Closes: #334
Approved by: cgwalters
2016-06-10 12:14:40 +00:00
Alexander Larsson f6b70a32e6 pull: Fix leak of refs_to_fetch
Closes: #333
Approved by: giuseppe
2016-06-10 11:12:59 +00:00
Alexander Larsson c9c4ab851e pull: Fix leaked uri in request_static_delta_superblock_sync
Closes: #333
Approved by: giuseppe
2016-06-10 11:12:59 +00:00
Alexander Larsson 785f894de1 pull: Fix leak of FetchObjectData in some cases
Closes: #333
Approved by: giuseppe
2016-06-10 11:12:59 +00:00
Alexander Larsson bd4e250e93 pull: Fix leak of debug temporary strings
Closes: #333
Approved by: giuseppe
2016-06-10 11:12:59 +00:00
Alexander Larsson 8e3d558f23 libotutil: Fix leak of VariantMapData
Closes: #333
Approved by: giuseppe
2016-06-10 11:12:59 +00:00
Alexander Larsson d368624798 Build on older versions of glib
Various places need to include libglnx.h for the autoptr backport
fallbacks to be there before ostree-autocleanups.h is included.

This fixes the build on centos7·

Closes: #309
Approved by: giuseppe
2016-05-25 14:01:39 +00:00
Alexander Larsson 86ac8062f5 tests: Fix build with old glib
test-checksum.c was using g_autofree without including libglnx.h which
has the backport for that.

Closes: #304
Approved by: cgwalters
2016-05-20 11:48:02 +00:00
Alexander Larsson f2fd1f50e2 repo: Handle parent repos with different remote configuration
In the case we have a repo with a parent, and the child repo has a
remote called "foo", but some option is unset. Then when we look up
the parent repo for a value before using the default we will fail due
to the parent not having the "foo" remote. As soon as we find the
requested remote at some point in the hierarchy we need to ignore further
errors and use the default value.

Closes: #274
Approved by: giuseppe
2016-04-22 14:38:00 +00:00
Alexander Larsson ad019def9a repo: Fix temporary directory allocation
We looked for and locked old temporary directories so we can
reuse them if not in use. However, once we found one that
we can reuse we didn't stop iterating, and eventually we
reached the end. This means we can lock multiple dirs.

Closes: #273
Approved by: giuseppe
2016-04-22 08:46:53 +00:00
Alexander Larsson 8f8ab56211 repo: Allow loading files staged in the transaction
Currently we can load metadata from the stage dir, but not file
data, which makes no sense.

Closes: #269
Approved by: cgwalters
2016-04-21 19:05:06 +00:00
Alexander Larsson 5595664e47 ostree export: Add --prefix option
This lets you set a prefix for the resulting archive patsh.
Especially useful in combination with --subpath, for instance
--subpath=subdir --prefix=subdir to extract just subdir.

Closes: #265
Approved by: cgwalters
2016-04-19 12:28:06 +00:00
Alexander Larsson 5079f70ec0 ostree export: Add --subpath support
This lets you export a subset of a commit.

Closes: #265
Approved by: cgwalters
2016-04-19 12:28:06 +00:00
Alexander Larsson b4b26907c8 static-delta: Initialize read_source_fd to -1
If not, we'll get ESPIPE when seeking on fd 0.

Closes: #260
Approved by: cgwalters
2016-04-15 11:56:06 +00:00
Alexander Larsson a08b7765b7 static-delta: Put temp files in /var/tmp
We may not have write permissions in the current directory.

Closes: #259
Approved by: cgwalters
2016-04-14 20:49:32 +00:00
Alexander Larsson 7ac8b0442c Look for $remotename.trustedkeys.gpg in remotes.d dir
This is a nice way to add gpg keys for system configured remotes
without making them globally trusted.

Closes: #247
Approved by: cgwalters
2016-04-14 16:21:01 +00:00
Alexander Larsson c86e4f0c90 Add remotes-config-dir to OstreeRepo
This allows you to replace the default
$sysroot/$sysconfdir/ostree/repos.d string value, and to use a similar
feature for repos that are not the system repo.

In particular, this allows us to support /etc/xdg-app/remotes.d for
xdg-app.

Closes: #247
Approved by: cgwalters
2016-04-14 16:21:01 +00:00
Alexander Larsson 9e7e594907 Add OstreeRepo option for an out-of-band cache dir
This allows you to have a writable cache dir even for a system-owned
repository.

Closes: #250
Approved by: cgwalters
2016-04-14 15:55:08 +00:00
Alexander Larsson b787fce612 Add cache_dir_fd to OstreeRepo
This will allow us later to easily swap out the cache dir.

Closes: #250
Approved by: cgwalters
2016-04-14 15:55:08 +00:00
Alexander Larsson 77ea287cd2 commit: Fix crash if dfd_iter is NULL
in write_directory_content_to_mtree_internal dfd_iter can be NULL,
for instance if commiting from --tree=ref=FOO. Don't blindly de-ref
it to avoid crashing.

Closes: #256
Approved by: cgwalters
2016-04-13 19:37:06 +00:00
Alexander Larsson 6a57d0a2f0 fetcher: Initialize output_stream_set_lock mutex
ostree pull-local crashed for me in thread_closure_unref () doing:
    g_mutex_clear (&thread_closure->output_stream_set_lock);

Seems like we never initialize this mutex.

Closes: #254
Approved by: cgwalters
2016-04-13 14:11:46 +00:00
Alexander Larsson e62beae695 Fix the symbol versions for ostree_repo_get_remote_*option
These were accidentally added to 2016.4 instead of 2016.5

Closes: #251
Approved by: cgwalters
2016-04-11 13:07:07 +00:00
Alexander Larsson ddda8e5b8b Add support for ostree static-delta delete
Closes: #245
Approved by: giuseppe
2016-04-08 13:42:43 +00:00
Alexander Larsson 6aa447a917 Add test case for inheriting remote options
Closes: #236
Approved by: cgwalters
2016-04-07 20:02:46 +00:00
Alexander Larsson d7181f22a6 Inherit remotes and remote options from parent repo
Closes: #236
Approved by: cgwalters
2016-04-07 20:02:46 +00:00
Alexander Larsson 37382590dc Export ostree_repo_get_remote_option* functions
These are useful for ostree users (like xdg-app) that have custom
options for remotes. In particular they are useful when we later make them
all respect self->parent_repo.

Closes: #236
Approved by: cgwalters
2016-04-07 20:02:46 +00:00
Alexander Larsson 91734a8a18 Support pathnames for --subpath=...
This allows you to pull a single file, rather than just a directory.

Closes: #244
Approved by: cgwalters
2016-04-07 15:15:39 +00:00
Alexander Larsson ace0d4650b pull-local: Support --gpg-verify and --gpg-verify-summary
Force the otherwise disabled gpg verifications on.
Note: You need to pass --remote=foo so we know what gpg keys to verify
against.

Closes: #237
Approved by: cgwalters
2016-04-05 19:12:09 +00:00
Alexander Larsson 87298c84b8 Fix local-pull test
This was not being run for whatever reason. Also, the test
count was wrong.

Closes: #237
Approved by: cgwalters
2016-04-05 19:12:09 +00:00
Alexander Larsson eabb4e3545 pull: Don't try to cache summaries for pull-local
Not only does this not make sense from a performance perspective, but
it also doesn't work because we can't use a url as a path element.

Closes: #237
Approved by: cgwalters
2016-04-05 19:12:09 +00:00
Alexander Larsson 456f515522 Add --untrusted option to pull and pull-local
https://bugzilla.gnome.org/show_bug.cgi?id=764125

Closes: #221
Approved by: cgwalters
2016-03-25 12:56:55 +00:00
Alexander Larsson ed1e0c6d04 pull: Add OSTREE_REPO_PULL_FLAGS_UNTRUSTED flag
If this is set we verify all objects we pull, even for local remotes,
and we avoid hard-linking into local source repos.

https://bugzilla.gnome.org/show_bug.cgi?id=764125

Closes: #221
Approved by: cgwalters
2016-03-25 12:56:55 +00:00
Alexander Larsson ddf0a02f88 pull local: Don't import objects we already have
In particular, I noticed this because the pull failed when trying to
pull from a shallow repo which only had the new objects that was
needed.

https://bugzilla.gnome.org/show_bug.cgi?id=764059
2016-03-23 14:58:08 -04:00
Alexander Larsson cb60de0f95 Don't fail "ostree remote refs" if writing the summary cache is not permitted
It used to be allowed to run something like "ostree remote refs" on
a read-only (e.g. system) repo. However, the summary cache caused that to
break. This commit just makes it not save the cache if we get some kind
of permission error when writing it. It'll still work, even without the
cache.

https://bugzilla.gnome.org/show_bug.cgi?id=763855
2016-03-18 17:33:39 +01:00
Alexander Larsson 89624ee57d rofiles-fuse: Fix permission comparison
We want to allow write if the devinode is in the set,
not the other way around.

https://bugzilla.gnome.org/show_bug.cgi?id=763676
2016-03-16 16:30:53 +01:00
Alexander Larsson 9b919c8dd2 repo: Fix the skip-summary-if-summary.sig-is-same cache
This code was always downloading the summary even for a cache hit due
to a missing else.
2016-03-15 12:04:43 +01:00
Alexander Larsson 6c285d2e4a rofiles-fuse: Handle operations on the root
ENSURE_RELPATH breaks when path is "/". In that case we need
to return "." instead of "".

https://bugzilla.gnome.org/show_bug.cgi?id=762596
2016-02-24 20:43:23 +01:00
Alexander Larsson 1833142b14 deltas: Fix regression in ostree_repo_static_delta_execute_offline
xdg-app passed this a filename directly, and in this case it should be
used as is. This regressed to always look for "superblock" in the same
directory as the passed in filename.

https://bugzilla.gnome.org/show_bug.cgi?id=762617
2016-02-24 15:54:09 +01:00
Alexander Larsson 9f9d713a56 build: Use threadsafe GPGME
https://bugzilla.gnome.org/show_bug.cgi?id=761863
2016-02-22 18:07:30 -05:00
Alexander Larsson 96eed95720 repo: Allocate a tmpdir for each OstreeFetcher to isolate concurrent downloads
This way two pulls will not use the same tmpdir and accidentally
overwrite each other. However, consecutive OstreeFetchers will reuse
the tmpdirs, so that we can properly resume downloading large objects.

https://bugzilla.gnome.org/show_bug.cgi?id=757611
2015-12-14 08:39:11 +01:00
Alexander Larsson f771461b4a repo: Use per-transaction staging dir
Concurrent pulls break since we're sharing the staging directory for
all transactions in the repo. This makes us use a per-transaction directory.

However, in order for resumes to work we first look for existing
staging directories and try to aquire an exclusive lock for them. If
we can't find any staging directory or they are all already locked,
then we create a new one.

https://bugzilla.gnome.org/show_bug.cgi?id=757611
2015-12-14 08:38:51 +01:00
Alexander Larsson be19c88861 repo: Add _ostree_repo_allocate_tmpdir helper
This creates a subdirectory of the tmp dir with a selected prefix,
and takes a lockfile to ensure that nobody else is using the same directory.
However, if a directory with the same prefix already exists and is
not locked that is used instead.

The later is useful if you want to support some kind of resumed operation
on the tmpdir.

touch reused dirs

https://bugzilla.gnome.org/show_bug.cgi?id=757611
2015-12-14 08:37:55 +01:00
Alexander Larsson 211ca6795c Update to latest libglnx 2015-12-14 08:37:17 +01:00
Alexander Larsson 1c056eb282 deltas: Support including detached metadata in static deltas
This is very useful for the inline-parts case, as you can then include
detached signatures in a single file representing the commit.

It is not as important for the generic pull case, as the detached
metadata is only a single small file. Additionally the detached
metadata is not content referenced and may change after the static
delta file was created, so we need to pull the latest version anyway.
2015-11-10 08:56:16 +01:00
Alexander Larsson 598afd5030 pull: Verify checksums from static deltas unless gpg signed summary
Otherwise untrusted repos can lie about the commit ids.
2015-11-10 08:56:15 +01:00
Alexander Larsson ec56fea821 deltas: Support passing filename to ostree_repo_static_delta_execute_offline
If you pass a diriectory it will look for the "superblock" child, otherwise
it will use the file as the superblock. I need this in xdg-app to be able
to install any filename as a bundle.
2015-11-10 08:56:15 +01:00
Alexander Larsson dac57b6410 deltas: Support passing filename to delta generator 2015-11-10 08:56:15 +01:00
Alexander Larsson b22b859dc5 deltas: Make min-fallback-size 0 actually disable fallbacks
This is what the docs specify.
2015-11-10 08:56:14 +01:00
Alexander Larsson 78f14555c8 deltas: Verify checksums in apply-offline unless skip_validate is TRUE 2015-11-10 08:56:14 +01:00
Alexander Larsson 7a4fb1b2f1 Add _ostree_repo_open|commit_untrusted_content_bare
Also renames OstreeRepoTrustedContentBareCommit to
OstreeRepoContentBareCommit so that it can be used by both.

This will be needed when we introduce checksum verification of objects
in static deltas.
2015-11-10 08:56:14 +01:00
Alexander Larsson 3fa3e12e02 static-delta apply-offline: Don't skip validation
This makes no sense, at least by default.
2015-11-10 08:56:13 +01:00
Alexander Larsson e418acb31b delta: Ensure the from commit exists when applying static delta 2015-11-10 08:56:13 +01:00
Alexander Larsson eaa678eaa4 deltas: Make apply-offline only read the parts once
No need to read() the file for the checksum if we then directly mmap it.
Instead we just mmap it initially and checksum from that.
2015-11-10 08:56:13 +01:00
Alexander Larsson ee70ffbbcb Add tests for inline static deltas 2015-11-10 08:56:13 +01:00
Alexander Larsson ccb77d3bd8 static-deltas generate: Add --inline option to CLI tool 2015-11-10 08:56:12 +01:00
Alexander Larsson 11a79220e2 static deltas: Add support for inline-parts
In this mode the parts are stored in the metadata of the main delta
superblock file.  This can be useful if you want a single-file delta
for easy transport, or for http in the case the delta is very small.
2015-11-10 08:56:12 +01:00
Alexander Larsson 30768a979f generate-static-delta: Support min-fallback-size 0 to disable fallbacks 2015-11-10 08:56:12 +01:00
Alexander Larsson fd6c572c42 _ostree_static_delta_part_validate: Take a stream instead of a file as arg
This is only called in one place, and we have already opened the file there,
no need to open it twice.
2015-09-13 22:25:21 -04:00
Alexander Larsson 1ee4007a75 offline delta apply: Handle the version field
It seems the format changed and this code was not updated.
2015-09-13 22:23:56 -04:00
Alexander Larsson 74f4fc2bc8 Fix build with !HAVE_LIBSOUP
Fix up ostree_repo_pull to match the new declaration
2015-03-23 14:26:01 +01:00
Alexander Larsson d26359f823 Fix build failure on g_autoptr(gchar) with glib master
This was removed in favour of g_autofree
2015-03-20 15:36:56 +01:00
Alexander Larsson 638431045c static delta generation: Separate max chunk size from min fallback size
There is no particular reason these have to be the same.

https://bugzilla.gnome.org/show_bug.cgi?id=721799
2015-01-20 14:45:45 +01:00
Alexander Larsson 5b721a5b08 Allow creating and pulling static deltas starting from "empty"
You create these with something like:
  ostree static-delta generate --empty --to=master

These will be automatically used during pull if no previous revision
exists in the target repo.

These work very much like the normal static deltas except they
are named just by the "to" revision. I.e:

deltas/94/f7d2dc23759dd21f9bd01e6705a8fdf98f90cad3e0109ba3f6c091c1a3774d

for a from-scratch to 94f7d2dc23759dd21f9bd01e6705a8fdf98f90cad3e0109ba3f6c091c1a3774d delta.

https://bugzilla.gnome.org/show_bug.cgi?id=721799
2015-01-14 14:43:32 +01:00
Alexander Larsson 82ed6c43ed Fix ostree_repo_list_static_delta_names
The current layout uses a prefix of two bytes as the initial dir
and a second directory inside that with the superblock. This
updates the list code to handle that.

https://bugzilla.gnome.org/show_bug.cgi?id=721799
2015-01-14 14:36:35 +01:00
Alexander Larsson e908ebd9b3 test-local-pull: Sort find output to make test robust
There is no guarantee that find will produce output in the same
order, so we need to sort the output to ensure we always
get the same output.

https://bugzilla.gnome.org/show_bug.cgi?id=741125
2014-12-08 20:28:09 +01:00
Alexander Larsson dbf717ac4b Add local-pull archive-z2 <=> bare-user roundtrip test
This creates a archive-z2 repo, pull-locals it to bare-user and then
again back to archive-z2 making sure things fsck along the way.
Then it checks out all repos and makes sure each one reproduces
the same result.

Unfortunately we can't install this as a real test because
it doesn't work in the test-runner because tmpfs doesn't support
user xattrs.

https://bugzilla.gnome.org/show_bug.cgi?id=741125
2014-12-08 12:03:27 +01:00
Alexander Larsson fcd3caf6dd Add test-basic-user.sh testing for bare-user repos
This just does whatever test-basic.sh does, but on a bare-user
repo.

This works standalone, but unfortunately it breaks in
gnome-desktop-testing-runner as /tmp doesn't support
xattrs, so it is not installed atm.

https://bugzilla.gnome.org/show_bug.cgi?id=741125
2014-12-08 10:39:44 +01:00
Alexander Larsson a3422791d4 Split out basic tests from test-basic.sh
This will let us reuse them with other repo types

https://bugzilla.gnome.org/show_bug.cgi?id=741125
2014-12-08 10:39:44 +01:00
Alexander Larsson 47c612e5a0 Support for "bare-user" repo format
This format is pretty much the same as the "bare" format, except the
file ownership and xattrs is not stored in the actual filesystem object, but
rather on the side in a user xattr. This means two things:

1) An unprivileged user can store such a repo independent of the types
   of files in it or their xattrs. And you can later (as root)
   reconstruct the real filesystem tree with ownership. Although you
   can't do that using hardlink-sharing. This also means ostree
   fsck does a full verification.

2) Such a repository can be checked out with user-mode (checkout -U)
   as an unprivileged user using hardlinks for space sharing.

Additionally, symlinks are stored as regular files (with the content
being the symlink target) because user xattrs are not supported on
symlinks. We know at checkout time if the file is a symlink because
the original st_mode is stored in the xattr metadata.

https://bugzilla.gnome.org/show_bug.cgi?id=741125
2014-12-08 10:39:39 +01:00
Alexander Larsson 26a47b9cca Add ot_lgetxattrat and ot_lsetxattrat utils
These are implementation of the missing corresponding syscalls that
are done with the /proc/self/fd mechanism described at:

https://mail.gnome.org/archives/ostree-list/2014-February/msg00017.html

https://bugzilla.gnome.org/show_bug.cgi?id=741125
2014-12-08 10:39:39 +01:00
Alexander Larsson 22ed7d0fbf union checkout: Fix symlink handling for xattrs
Applying xattrs on a symlink during checkout failed since
it was setting the xattrs on the final filename, not the
temporary name.

This made the "checkout union 1" test in test-basic.sh
fail.

https://bugzilla.gnome.org/show_bug.cgi?id=741125
2014-12-05 15:53:42 +01:00
Alexander Larsson bb82c17701 checkout: fchown symlink to proper uid/gid
When commiting a symlink we do store the uid/gid of the actual
symlink (i.e. not target). However, this was not restored
on non-user-mode checkout as it should.

This commit fixes that, and additionally it ensures xattrs
on symlinks are not set in user-mode checkout.

https://bugzilla.gnome.org/show_bug.cgi?id=741125
2014-12-04 20:30:10 +01:00