Commit Graph

1800 Commits

Author SHA1 Message Date
Colin Walters d750e2e2eb commit: Add missing (allow-none) in write_ref_immediate()
Spotted by Adam Coldrick.
2015-03-03 21:16:33 -05:00
Colin Walters 67cec3a4de deltas: Use mmap() instead of copying input file
It's more efficient.
2015-03-03 18:39:45 -05:00
Colin Walters 3e3eb01335 deltas: Gather statistics on total number rollsum'd and bsdiff'd
Useful for debugging at least.  Though in the future it'd be nice to
store this inside the delta metadata maybe?
2015-03-03 18:32:24 -05:00
Giuseppe Scrivano 948327fb67 ostree_repo_static_delta_generate: add new param "verbose"
it will be used by the next patch that adds "--generate-static-delta"
to the commit command.

As part of the patch, update the list of supported "params" in the
documentation.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-03-03 12:16:17 -05:00
Colin Walters 392929d9bc static-delta: Add --disable-bsdiff option
A builder may not want to pay the CPU penalty.  Also it's useful for
evaluating performance.
2015-03-03 12:16:17 -05:00
Giuseppe Scrivano 494d3535e4 Fix "make syntax-check" failures.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-03-03 12:16:17 -05:00
Giuseppe Scrivano 439e1c1b97 static-delta: increase threshold for rollsum to 50%
It favours bsdiff usage for files that are not very rollsum friendly.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-03-03 12:16:17 -05:00
Giuseppe Scrivano 3f3bb8e37d Add bsdiff support to deltas
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-03-03 12:16:17 -05:00
Colin Walters 488efac728 repo: Port hardlink-scanning code to fd-relative calls
Continuing the migration.
2015-03-03 11:48:41 -05:00
Colin Walters 6e0d92db00 repo: Drop internal GFile* API helper
There are no users anymore.
2015-03-03 11:48:41 -05:00
Colin Walters 5b3a495018 deploy: Use glnx file copy code
This is centralizing more generic utility code in libglnx.
2015-03-03 11:48:41 -05:00
Matthew Barnes c107561171 core: Fix duplication bug in _ostree_detached_metadata_append_gpg_sig()
g_variant_builder_add() does not replace identical keys in a VARDICT
variant, so signing a commit multiple times results in multiple copies
of "ostree.gpgsigs" in the metadata.  And since g_variant_lookup_value()
stops on the first match, subsequent signatures have no net effect.

Instead of GVariantBuilder use GVariantDict, which behaves more like a
hash table.
2015-03-02 15:13:49 -05:00
Matthew Barnes c04f9f47e3 core: Add definitions for GPG signature metadata
_OSTREE_METADATA_GPGSIGS_NAME
_OSTREE_METADATA_GPGSIGS_TYPE
2015-03-02 15:12:06 -05:00
Matthew Barnes fdda36eb77 libotutil: Allow no variant in ot_util_variant_builder_from_variant()
For programming convenience in cases where we may or may not have a
GVariant to initialize the builder with.
2015-03-02 15:12:06 -05:00
Matthew Barnes b05d670031 libotutil: Remove ot_variant_new_from_bytes()
Not needed anymore - use g_variant_new_from_bytes().
2015-03-02 15:12:06 -05:00
Matthew Barnes 85933c7cca configure.ac: Bump GLib requirement to 2.40
Mainly for GVariantDict, useful for metadata.
2015-03-02 15:06:46 -05:00
Matthew Barnes 0bae648977 gpg: Remove _ostree_gpg_verifier_set_homedir()
Never called, and the setting is never applied anyway.
2015-03-02 12:03:04 -05:00
Matthew Barnes a5b002dae6 ostree: Add gpg-sign command
Signs a commit with one or more GPG keys.
2015-02-26 12:53:01 -05:00
Colin Walters 152dcf89b5 repo: detached sigs: Use error prefixing instead of overwriting
Noted when "rpm-ostree compose sign" failed to write to a repo due to
permissions.
2015-02-25 10:39:15 -05:00
Colin Walters ace982b51c prepare-root: Move /sysroot instead of unmounting it
I originally thought this would fix a regression, but it turns out
this wasn't the bug.  But pushing anyways as it's just cleaner.

https://bugzilla.gnome.org/show_bug.cgi?id=743891
2015-02-24 15:06:35 -05:00
Giuseppe Scrivano 22af1d9633 ostree-repo-static-delta-processing: initialize "modev"
Can be gs_unref_variant'ed on an early error.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-02-23 16:41:42 -05:00
Giuseppe Scrivano 52822efd32 Fix GObject introspection annotation
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-02-23 16:41:42 -05:00
Giuseppe Scrivano 50cd3e971a ostree-repo-traverse.c: Fix documentation parameter name
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-02-23 16:41:42 -05:00
Colin Walters 3b3708c312 repo: Port APIs used by prune to fd-relative *at calls
More of the general trend away from GFile * to the faster and more
secure world of *at().
2015-02-22 23:20:12 -05:00
Colin Walters 46bd4657b2 checkout: Drop internal use of GFile *
Checkout was one of the first complex code paths I tried to convert to
*at().  I ended up keeping both, because I hit the "xattrs for a
symlink" problem.  Later, Florian gave me a workaround, and we started
using it here, but the GFile * parameters weren't deleted.  They're
not used, so do so now.
2015-02-22 21:30:23 -05:00
Colin Walters 9539408bb3 deploy: Also look for /usr/lib/os-release
With newer versions of systemd, the file has moved there.
2015-02-22 21:02:27 -05:00
Colin Walters 965a304a17 Use libglnx
Starting down the path of not using libgsystem.  The main win here
will be code sharing between ostree/rpm-ostree as well as going down
the path of not using GFile * for local files.
2015-02-22 21:02:27 -05:00
Matthew Barnes 5aa0d51d7a repo: Check for OSTREE_REPO in ostree_repo_new_default()
Convenience feature to avoid having to pass --repo options repeatedly.

Before falling back to the default system repository path, check for a
repository path defined by the OSTREE_REPO environment variable.
2015-02-19 20:44:34 -05:00
Daniel Drake 7c267d6089 libostree: set directory mtimes to 0 on checkout
We already set all file mtimes to 0 so that they are constant
over all checkouts, and can be made constant with a known value from
the system where the ostree was created.

However, this was not happening for directories. Zero their mtimes too.

This is important for shipping a fontconfig cache in the ostree;
the fontconfig cache files embed a directory mtime.
2015-02-18 18:27:18 -05:00
Colin Walters c54df89771 deltas: Search for similar objects (possibly renamed across directories)
The previous diff algorithm was file tree based, and only looked
at modified files that lived at the same path.

However, components like the Linux kernel have versioned
subdirectories, e.g. /usr/lib/modules/$kver/.../ext4.ko.  We want to
be able to detect these "modified renames" so that we can compute
diffs (rollsum, bsdiff).
2015-02-16 10:10:35 -05:00
Colin Walters 0f74ed62b7 repo: Add a new iterator traversal API for commits
This is a more optimized version of the GFile * APIs, and is now used
internally by the previous ostree_repo_traverse().
2015-02-16 10:10:35 -05:00
Colin Walters 08476ce254 deltas: Prune deltas when the corresponding "to" commit vanishes
We want prune to actually give you back disk space when using deltas.
2015-02-16 10:10:35 -05:00
Colin Walters 9aa7e30b38 deltas: Implement rollsums
This does an rsync-style prepared delta basically.  On my test data,
it shaves ~6MB of uncompressed data.  Not a huge amount, but I expect
this to be more useful for things like binaries which embed data, etc.
2015-02-16 10:10:35 -05:00
Colin Walters 345754a564 deltas: Initial code to copy content from existing objects
This is preparatory work for implementing rollsum support.
2015-02-16 10:10:35 -05:00
Colin Walters 7900c82a36 deltas: Flesh out the open/write/close opcodes
Refactor open-splice-and-close to call open/close.  We can't just call
write as that would require duplicating the object size parameter.
2015-02-16 10:10:35 -05:00
Colin Walters 96181da26a deltas: Use the new internal streaming APIs
This is significantly more efficient.
2015-02-16 10:10:35 -05:00
Colin Walters 7aea18cf0d deltas: Stub out a few more opcodes 2015-02-16 10:10:35 -05:00
Colin Walters 8fb29c9e5d deltas: Print total size of rollsums we would use 2015-02-16 10:10:35 -05:00
Colin Walters 3df8be0d92 deltas: Compute rollsum targets 2015-02-16 10:10:35 -05:00
Colin Walters d749932f6b deltas: Rework format to allow streaming
There's still some silliness here, but there is now only one opcode
open-splice-and-close, that writes a single chunk from the payload.
This is really all we need for metadata, and small content objects are
also fine with this.

We get some deduplication between content objects by creating a
dictionary for (uid,gid,mode) tuples and xattrs.

This still keeps the operation/payload code in, so we could do
rollsums in a future update easily.
2015-02-16 10:10:35 -05:00
Colin Walters 513d47a90c deltas: Add _V0 to part #define
To make more explicitly clear that this is the version that matches
the version in the metadata.
2015-02-16 10:10:35 -05:00
Colin Walters 3c2a36eab0 deltas: Remove support for gzipped delta parts
XZ is really, really good.
2015-02-16 10:10:35 -05:00
Colin Walters 92cc3b5968 deltas: Use base64 for csums, add version to parts 2015-02-16 10:10:35 -05:00
Colin Walters ee4e393fa1 repo: Store pending objects in prefixed subdirectory
I was hitting a bug in libguestfs/guestmount/FUSE where it blew up
with EINVAL on directories containing lots of files (more than
32000?).  We really want to use prefixed subdirs just like the real
objects/ directory does.

This allows us to share more code between the paths, is more
efficient, etc.
2015-02-15 15:30:19 -05:00
Colin Walters 49bdbf1db0 repo: Fix major performance regression with --scan-hardlinks
gnome-continuous uses the ostree_repo_scan_hardlinks() mode to
avoid re-checksumming everything.  However, when I ported the commit
code to use openat() and friends, this optimization was lost.

Re add it.  The difference is about 15s versus 5 minutes.
2015-02-13 16:36:07 -05:00
Colin Walters 64363c26ac pull-local: Fix regression with absolute paths
Don't add cwd unless the path is relative.
2015-02-13 11:39:36 -05:00
Colin Walters 9cc9804195 Change pull-local to just be a wrapper for pull with file:///
This follows up from the previous commit; now that pull knows how to
do the efficient link() or copy for local files, we can just have
pull-local call into ostree_repo_pull().

As part of this:
 - pull() can also accept a file:/// URI instead
   of a remote name (since pull local supports anonymous pulls)
 - pull() knows an "override-remote-name" option, since pull-local
   supported writing a ref out even if there wasn't a remote with
   that name
2015-02-08 04:43:06 -05:00
Colin Walters ab3bf493dd pull: Optimize file:/// URIs to skip libsoup and hardlink if possible
It's always been suboptimal to have both pull and pull-local; as we go
beyond the raw object data into things like deltas and summary files,
the logic to perform e.g. mirroring should only be in one place.

This will be used by Pulp's OSTree content plugin at least to perform
promotions.
2015-02-06 10:02:57 -05:00
Colin Walters be4ad54800 Add an internal API to get a read fd for a content object
This will be used by the static deltas work.
2015-02-05 21:50:26 -05:00
Colin Walters 1fdecbd263 pull: Copy the upstream summary file when doing a pull --mirror
While it could be regenerated downstream, there might be other
metadata upstream, and the goal here is a mirror.

https://bugzilla.gnome.org/show_bug.cgi?id=739377
2015-02-05 21:24:21 -05:00
Colin Walters fab1e113db When mirroring, write content directly, do not verify
When doing a pull --mirror from an archive-z2 repository into another
archive-z2 repository, currently we gunzip/checksum/gzip each content
object.  The re-gzip process in particular is fairly expensive.

This does assume that the upstream content is trusted and correct.
It'd be nice in the future to do at least a CRC check, if not the full
checksum.  (Could we append CRC data to the end of filez objects?)

We could also choose to only do this optimization if fetching over
TLS.

before: 1626 metadata, 20320 content objects fetched; 299634 KiB transferred in 62 seconds
after : 1626 metadata, 20320 content objects fetched; 299634 KiB transferred in 11 seconds
2015-02-05 21:24:21 -05:00
Colin Walters 247866a9bc Add an internal API to stream content objects
For future delta work where we do more interesting things than just
"tar of new objects", this lays the groundwork for doing streaming
writes into content objects.

It's also more efficient, as we avoid many intermediate allocations
and virtual calls.  Just a single `g_output_stream_write_all` for the
splice case.

Conflicts:
	src/libostree/ostree-repo-private.h
	src/libostree/ostree-repo-static-delta-processing.c
2015-02-05 19:15:27 -05:00
Colin Walters 65afe1110d util: Add an API to atomic-replace a file, dirfd relative, optional fsync
This can be used in other places too, but I plan to use it to write
the summary file.
2015-02-05 16:59:52 -05:00
Colin Walters 3b8ed12ab2 repo: Hold an fd "repo_dir_fd" open for the toplevel too
We could just make everything relative to this, but the objects/ and
tmp/ are accessed very often, so I think it's worth holding individual
fds.

This fd can cover everything else: refs, deltas, etc.
2015-02-05 14:15:34 -05:00
Colin Walters e739677ea1 prepare-root: Update comments 2015-02-04 05:32:45 -05:00
Daniel Drake 4f75d4ea0b prepare-root: avoid double-stacked /sysroot mount
prepare-root works with the mount that has been set up at /sysroot.
It creates a bind-mount within /sysroot (the deployment) and then moves
that mount to /sysroot.

Now we have 2 mounts both at /sysroot, and once we do switch_root, we will
never be able to unmount both of them. I'm not sure if this is ultimately
a kernel bug, but either way, ostree could do a bit more tidying up
after itself.
http://thread.gmane.org/gmane.linux.file-systems/92411

Easy way to reproduce:
1. Boot with rd.break param
2. At initramfs shell, run: ostree-prepare-root /sysroot
3. Observe two /sysroot mounts in /proc/mounts

Fix this by setting up the mounts at /sysroot.tmp, and unmounting the
original /sysroot before our new mount is MS_MOVEd on top of it.
2015-02-03 20:28:37 -05:00
Giuseppe Scrivano 6a3959c895 syntax-check: Remove empty lines at the end of file
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-02-02 15:07:56 +01:00
Giuseppe Scrivano 27a45e2edb pull: use a single per-transaction syncfs instead of fsync
Do not write directly to objects/ but maintain pulled files under tmp/
with a "tmpobject-$CHECKSUM.$OBJTYPE" name until they are syncfs'ed to
disk.

Move them under objects/ at ostree_repo_commit_transaction cleanup
time.

Before (test done on a local network):

$ LANG=C sudo time ./ostree --repo=repo pull origin master

0 metadata, 3 content objects fetched; 83820 KiB; 4 delta parts
fetched, transferred in 417 seconds
16.42user 6.73system 6:57.19elapsed 5%CPU (0avgtext+0avgdata
248428maxresident)k
24inputs+794472outputs (0major+233968minor)pagefaults 0swaps

After:

$ LANG=C sudo time ./ostree --repo=repo pull origin master

0 metadata, 3 content objects fetched; 83820 KiB; 4 delta parts
fetched, transferred in 9 seconds
14.70user 2.87system 0:09.99elapsed 175%CPU (0avgtext+0avgdata
256168maxresident)k
0inputs+794472outputs (0major+164333minor)pagefaults 0swaps

https://bugzilla.gnome.org/show_bug.cgi?id=728065

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-01-30 14:03:42 -05:00
Giuseppe Scrivano e8cbd4b8c5 Remove magic argument numbers to exit(2)
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-01-30 15:27:36 +01:00
Giuseppe Scrivano ea4683ba06 Remove unused <dirent.h>
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-01-30 15:27:36 +01:00
Giuseppe Scrivano 7154193ae0 Remove unused include <assert.h>
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-01-30 15:27:36 +01:00
Giuseppe Scrivano b726796f41 Remove trailing dot from error message
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-01-30 15:27:36 +01:00
Giuseppe Scrivano bb5d7bcf3a Fix repeated words.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-01-30 15:27:36 +01:00
Colin Walters 5b4500faf6 pull: (trivial) Fix English in function name
s/writed/written/
2015-01-25 09:14:21 -05:00
Colin Walters 6ce80f9685 Explicitly label .origin files as configuration
subscription-manager has a daemon that runs in a confined domain,
and it doesn't have permission to write usr_t, which is the default
label of /ostree/deploy/$osname/deploy.

A better long term fix is probably to move the origin file into the
deployment root as /etc/ostree/origin.conf or so.

In the meantime, let's ensure the .origin files are labeled as
configuration.
2015-01-23 12:44:06 -05:00
Colin Walters 8f4999c854 build: Add --disable-static-deltas
Since they're unstable, we want to allow organizations shipping ostree
now to be future proof against any changes.
2015-01-22 21:21:34 -05:00
Colin Walters f2e4830409 pull: Further extend static delta progress
With deltas, we have an accurate total size, among other things.
2015-01-22 09:07:13 -05:00
Colin Walters e40b86221a pull: Distingiush delta pulls from loose better 2015-01-22 09:07:13 -05:00
Colin Walters 549f1ec7df commit: Fix segfault on async writes if object exists and checksum requested
If an object already existed and we somehow tried to pull it, the
caller would still expect a returned checksum.

This appears to happen with static deltas for some reason; we might be
including duplicate metadata objects.  Regardless, this is a bug that
should be fixed.
2015-01-21 21:17:47 -05:00
Colin Walters 28e5e6f27f traverse: Fix transfer on hash table
Otherwise, bindings try to erroneously unref the key/values as well.
2015-01-21 11:59:32 -05:00
Colin Walters 8195fd18e2 deltas: Use trusted writes
We have a chain of checksums from the root up until here.  While doing
checksums of the objects individually would be a good redundancy check
for test cases and the like, when doing a pull there's no good reason
to burn cycles on SHA256.
2015-01-20 23:28:26 -05:00
Colin Walters 6d1de23f87 deltas: Drop async content writes
This caused deadlocks and/or EMFILE due to the interaction between
threads and fds.  What we really want here is a better pull-based
model for parsing content objects.

Another idea would be to change static deltas so that content objects
have a special opcode that includes their metadata first, and then do
rollsums etc. only over actual content.
2015-01-20 23:21:26 -05:00
Colin Walters d49fc876bb deltas: Unlink temporary metadata files before processing
Leaking them is bad.
2015-01-20 23:16:30 -05:00
Giuseppe Scrivano 4991e9ab6a static-delta: limit the number of writes in process to 1
This will avoid too many open files at the same time that could cause
an EMFILE error.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
(cherry picked from commit bc092b06f0e34e93f7d6102957bf55fd7ffd1b9e)
2015-01-20 09:51:02 -05:00
Alexander Larsson 638431045c static delta generation: Separate max chunk size from min fallback size
There is no particular reason these have to be the same.

https://bugzilla.gnome.org/show_bug.cgi?id=721799
2015-01-20 14:45:45 +01:00
Colin Walters 6bbfa5f85a admin: Add set-origin command
See projectatomic/rpm-ostree#42 for rationale. There are two high
level use cases:

 - If the OS comes unconfigured, this is a way to point it at a repo of your choice.
 - To switch between repositories while keeping the same branch easily.
2015-01-19 13:55:20 -05:00
Colin Walters 886913abdc sysroot: Add ostree_sysroot_write_origin_file() API
We want to allow admins to change the origin file without doing a new
deployment, so this will be part of a future "admin set-origin"
command.
2015-01-19 13:45:11 -05:00
Colin Walters 1d216a8c60 cmdline: (cleanup) Add internal helper to parse key=value options
This will be used by a later "ostree admin set-origin" as well.
2015-01-19 13:45:11 -05:00
Colin Walters 0eac91a253 admin: (cleanup) Add internal API to find a deployment given an index
At some point, we might want to expose a uniform way to refer
to deployments by an index.  At the moment undeploy is the only
command that does.

I plan to introduce another command which optionally takes an index,
so prepare a helper function for this.
2015-01-19 13:45:11 -05:00
Giuseppe Scrivano ce957f8649 editor: honor arguments environment variable
My EDITOR looks like "EDITOR=emacs -nw", ensure that it is honored.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-01-16 09:21:30 -05:00
Giuseppe Scrivano f2ae405f9f static-delta: do not accept both --from=REV and --empty for generate
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2015-01-15 22:54:56 -05:00
Colin Walters 2f9567ad56 deltas: Use *at() for writes
Yet another GFile* user goes away in a performance sensitive path.
2015-01-14 22:29:45 -05:00
Colin Walters ebd03216a1 pull: Delete processed delta parts
Otherwise they stay around until a much later GC.
2015-01-14 22:18:00 -05:00
Colin Walters 9020fe2547 Change OstreeFetcher to be dirfd-relative
This is a noticeable cleanup, and fixes another big user of GFile* in
performance/security sensitive codepaths.

I'm specifically making this change because the static deltas code was
leaking temporary files, and cleaning that up nicely would be best if
we were fd relative.
2015-01-14 22:12:08 -05:00
Colin Walters a7300a828d core: Add an API to parse a content file using dirfd relative lookup
This will be used for a later change to use openat() for the fetching
code.  Note that we drop the code to use mmap() - it was an attempt to
avoid keeping a fd open, but we do correctly close anyways.
2015-01-14 22:03:02 -05:00
Colin Walters 9b6b352181 checksumutils: Support splicing stream to arbitrary checksum type
This will be used later by the metalink code; you can splice with a
NULL output stream to an arbitrary GChecksum instead of just a SHA256
one.
2015-01-14 22:01:54 -05:00
Colin Walters 92c338de74 deltas: (trivial) delete some debugging prints
They create too much noise.
2015-01-14 11:38:10 -05:00
Alexander Larsson 5b721a5b08 Allow creating and pulling static deltas starting from "empty"
You create these with something like:
  ostree static-delta generate --empty --to=master

These will be automatically used during pull if no previous revision
exists in the target repo.

These work very much like the normal static deltas except they
are named just by the "to" revision. I.e:

deltas/94/f7d2dc23759dd21f9bd01e6705a8fdf98f90cad3e0109ba3f6c091c1a3774d

for a from-scratch to 94f7d2dc23759dd21f9bd01e6705a8fdf98f90cad3e0109ba3f6c091c1a3774d delta.

https://bugzilla.gnome.org/show_bug.cgi?id=721799
2015-01-14 14:43:32 +01:00
Alexander Larsson 82ed6c43ed Fix ostree_repo_list_static_delta_names
The current layout uses a prefix of two bytes as the initial dir
and a second directory inside that with the superblock. This
updates the list code to handle that.

https://bugzilla.gnome.org/show_bug.cgi?id=721799
2015-01-14 14:36:35 +01:00
Colin Walters 97fbd872ae deltas: Do not traverse all objects, only both commits
That's the way they were designed.  We really don't want to include
all intervening objects.

https://bugzilla.gnome.org/show_bug.cgi?id=721799
2015-01-13 21:32:39 -05:00
Colin Walters 019635d9c2 repo: Fix bare-user file loads
Regression from 86764dbf00

This function is kind of fiendish now that we have 3 cases, each of
which want to be optimized somewhat to only load what's necessary
(e.g. don't open the file if we don't have an output for stream
requested).

Clean things up so that BARE_USER and BARE are separate conditionals
that share as much as possible, and fix the bug that asserted we
were in BARE mode.

I tested this by running test-basic-user.sh by hand.
2015-01-12 12:43:33 -05:00
Colin Walters 25a5909500 repo: Deduplicate some code in load_file 2015-01-12 12:43:33 -05:00
Colin Walters e3f9f331a2 Two 32 bit compilation fixes 2015-01-12 12:43:03 -05:00
Matthew Barnes 6ff841d3b0 repo: Report metadata fetch progress separately
Partially resolves https://bugzilla.gnome.org/740276
2015-01-12 11:21:18 -05:00
Matthew Barnes 4b5b450d5c pull: Prioritize fetching metadata objects over content objects 2015-01-11 18:48:21 -05:00
Matthew Barnes 5c26e392ec fetcher: Add a priority value to async requests 2015-01-11 18:48:21 -05:00
Colin Walters f23f556f03 checkout: Add --fsync=false
Some use cases for checkouts don't need to fsync during checkout.
Installer programs for example will just do a global fsync at the end.

In the future, the default "ostree admin" core could also be
rearchitected to only do a transaction commit right before reboot, and
do the fsync then.

https://bugzilla.gnome.org/show_bug.cgi?id=742482
2015-01-07 11:41:46 -05:00
Colin Walters 687a6f8314 Add internal ot_openat_read_stream() helper
We had two cases which were creating an input stream using openat().
2015-01-07 08:41:45 -05:00
Colin Walters 026c5c60d3 Use *at() functions for native filesystem commits
This is just an efficiency optimization.  We're getting fairly close
to all of the hot code paths using `*at()`.

Note that we end up maintaining a half-duplicate code path set here,
because we still need to support commits from an arbitrary GFile *,
which in a possible common case is an OSTree commit.

I think it's worth it though.
2015-01-06 22:43:14 -05:00
Colin Walters cf8d6848b3 refs: Use G_IO_ERROR_NOT_FOUND when a ref lookup fails
rpm-ostree had code to check for this, which didn't actually work.

I don't see a no backwards compatibility concern in changing this, as
it's unlikely a caller would try to sensibly disambiguate FAILED.
2015-01-06 18:47:04 -05:00
Colin Walters 86764dbf00 repo: Improve ostree_repo_load_file() to use `*at()` for xattrs
We were already using openat() for the contents, but not the xattrs.
Now that libgsystem 2014.3 has gs_fd_get_all_xattrs(), make use of it.

Clean things up a bit so we only open the fd once.
2015-01-06 16:47:03 -05:00
Colin Walters 6dab41ba77 deploy: Update to use latest libgsystem API 2015-01-06 12:46:21 -05:00
Colin Walters 01f6d68c60 deploy: Actually copy xattrs of modified config files
Regression from 7b01bd2e43
where we stopped using g_file_copy() - we lost copying xattrs.

This specifically breaks /etc/shadow SELinux labeling, with the
obvious bad consequences.

https://bugzilla.gnome.org/show_bug.cgi?id=742289
2015-01-06 10:55:44 -05:00
Matthew Barnes 7727fe84d9 Require a PREFIX when deleting refs
Also fix the "ostree refs" help output to not give the impression that
the --delete option takes its own PREFIX argument.

https://bugzilla.gnome.org/show_bug.cgi?id=742454
2015-01-06 10:33:12 -05:00
Colin Walters 1e8e070102 Port to libgsystem errno API, hard depend on 2014.3
This is long overdue to make it to libgsystem.  Update our dependency.
2015-01-04 21:17:11 -05:00
Colin Walters 1bcc7a8e3a Merge branch 'giuseppe/staticdeltas' of https://github.com/giuseppe/ostree 2014-12-19 16:31:31 -05:00
Colin Walters 125889fd7e Enforce 'rdev' (device file major/minor) is 0
Historically OSTree supported device files, but it wasn't useful, and
added attack surface.  Support was removed in

https://git.gnome.org/browse/ostree/commit/?id=62a896350bd54bff5a9413d2ee0fad7ff4364f9a

Perform a further cleanup by enforcing internally that the device
major/minor must be 0.

Conflicts:
	src/libostree/ostree-core.c
2014-12-19 10:20:05 -05:00
Colin Walters c4efbf6718 core: Deduplicate code converting struct stat -> GFileInfo
We were doing the same thing in a number of places, make a helper
function.
2014-12-19 10:18:59 -05:00
Matthew Barnes 880328ba03 Add ostree_repo_pull_default_console_progress_changed()
Replaces ot_common_pull_progress() in ostree binary, so it can be shared
with rpm-ostree.
2014-12-18 21:31:53 -05:00
Colin Walters abb88336b3 repo: Minor cleanup: Move size generation code initialization
It has a global effect, so do it in the entrypoint.
2014-12-18 18:06:56 -05:00
Giuseppe Scrivano 6e60c05d2f Remove unused variable warning 2014-12-18 12:48:47 +01:00
Giuseppe Scrivano ed2b56a430 fix --help for static-delta
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2014-12-18 12:48:47 +01:00
Colin Walters 5c47389243 test-rollsum: Process all input, print more statistics
Copying the bup code, we need to loop over all chunks even after
hitting the rollsum returning 0.

Also print more statistics.
2014-12-18 12:48:47 +01:00
Colin Walters ca678224be Static deltas support
https://bugzilla.gnome.org/show_bug.cgi?id=721799
2014-12-18 12:48:47 +01:00
Giuseppe Scrivano a23b3b332e Revert "ostree-repo-pull: Remove currently #if 0 static delta code"
This reverts commit 1c2e20af25.
2014-12-18 12:48:47 +01:00
Matthew Barnes 880940f93b doc: Update gtk-docs 2014-12-17 19:34:17 -05:00
Matthew Barnes e54d48be39 Add "ostree remote list" command
Lists available remote names.
2014-12-17 16:05:53 -05:00
Matthew Barnes 39be27fc93 Add ostree_repo_remote_list()
Lists available remote names.
2014-12-17 16:05:37 -05:00
Colin Walters f6a6e68412 Add more flexible _remote_change() API , expose via 'ostree remote'
For Anaconda, I needed OSTREE_REPO_REMOTE_CHANGE_ADD_IF_NOT_EXISTS,
with the GFile *sysroot argument to avoid ugly hacks.  We want to
write the content provided via "ostreesetup" as a remote to the target
chroot only in the case where it isn't provided as part of the tree
content itself.

This is also potentially useful in idempotent systems management tools
like Ansible.

https://bugzilla.gnome.org/show_bug.cgi?id=741577
2014-12-15 21:28:09 -05:00
Colin Walters 8067e977a7 repo: (cleanup) Use _is_system() rather than inlining it
No reason to duplicate the code.
2014-12-15 13:59:46 -05:00
Matthew Barnes fa6e7b4b01 Improve "ostree remote" help output
Must have glossed over these because the commands are so simple.

 - List subcommands for "ostree remote --help".

 - Only show options relevant to COMMAND for
   "ostree remote COMMAND --help".
2014-12-09 09:56:30 -05:00
Colin Walters f7c926c5e9 trivial-httpd: use PR_SET_PDEATHSIG by default
If we're not daemonizing, this is a useful way to ensure we go away if
our parent does.

https://bugzilla.gnome.org/show_bug.cgi?id=741264
2014-12-08 14:47:37 -05:00
Matthew Barnes ac882f2128 repo: Add private functions to access remote options
ostree_repo_pull_with_options() needs this, and I'd rather keep the
OstreeRemote struct definition tucked away in ostree-repo.c with its
own internal API.
2014-12-08 12:49:25 -05:00
Matthew Barnes f79896693e repo: Add ostree_repo_remote_get_url()
Peeking at remote details by way of ostree_repo_copy_config() doesn't
work anymore.
2014-12-08 12:49:23 -05:00
Matthew Barnes 17b9e399b8 repo: Add an internal struct to manage remotes
OstreeRemote is a reference-counted struct that encompasses data about a
remote, whether read from a configuration file or created explicitly via
ostree_repo_remote_add().

OstreeRemotes are held in an internal table indexed by remote name.
This solves some problems caused by merging system-wide remote data into
the OstreeRepo's internal config key file.

Also fixes https://bugzilla.gnome.org/show_bug.cgi?id=740911
2014-12-08 12:49:10 -05:00
Matthew Barnes f3dcb7a052 libotutil: Add ot_keyfile_copy_group()
Copies all the keys of a group from one GKeyFile to another.
2014-12-08 12:47:19 -05:00
Alexander Larsson 47c612e5a0 Support for "bare-user" repo format
This format is pretty much the same as the "bare" format, except the
file ownership and xattrs is not stored in the actual filesystem object, but
rather on the side in a user xattr. This means two things:

1) An unprivileged user can store such a repo independent of the types
   of files in it or their xattrs. And you can later (as root)
   reconstruct the real filesystem tree with ownership. Although you
   can't do that using hardlink-sharing. This also means ostree
   fsck does a full verification.

2) Such a repository can be checked out with user-mode (checkout -U)
   as an unprivileged user using hardlinks for space sharing.

Additionally, symlinks are stored as regular files (with the content
being the symlink target) because user xattrs are not supported on
symlinks. We know at checkout time if the file is a symlink because
the original st_mode is stored in the xattr metadata.

https://bugzilla.gnome.org/show_bug.cgi?id=741125
2014-12-08 10:39:39 +01:00
Alexander Larsson 26a47b9cca Add ot_lgetxattrat and ot_lsetxattrat utils
These are implementation of the missing corresponding syscalls that
are done with the /proc/self/fd mechanism described at:

https://mail.gnome.org/archives/ostree-list/2014-February/msg00017.html

https://bugzilla.gnome.org/show_bug.cgi?id=741125
2014-12-08 10:39:39 +01:00
Alexander Larsson 22ed7d0fbf union checkout: Fix symlink handling for xattrs
Applying xattrs on a symlink during checkout failed since
it was setting the xattrs on the final filename, not the
temporary name.

This made the "checkout union 1" test in test-basic.sh
fail.

https://bugzilla.gnome.org/show_bug.cgi?id=741125
2014-12-05 15:53:42 +01:00
Alexander Larsson bb82c17701 checkout: fchown symlink to proper uid/gid
When commiting a symlink we do store the uid/gid of the actual
symlink (i.e. not target). However, this was not restored
on non-user-mode checkout as it should.

This commit fixes that, and additionally it ensures xattrs
on symlinks are not set in user-mode checkout.

https://bugzilla.gnome.org/show_bug.cgi?id=741125
2014-12-04 20:30:10 +01:00
Colin Walters 37a059925f deploy: Ensure that we can deploy using only /usr/lib/ostree-boot
rpm-ostree at least has the option to generate a tree with just that
instead of /boot, but while we were enumerating the latter, we'd still
return paths from /boot.

https://bugzilla.gnome.org/show_bug.cgi?id=740947
2014-11-30 23:14:05 -05:00
Colin Walters 387ee4cd65 trivial: Fix typo in previous commit 2014-11-25 13:47:54 -05:00
Colin Walters 12187994e2 grub2: If using --sysroot, run in chroot
In Anaconda, we're using "ostree admin --sysroot=/mnt/sysimage
instutil set-kargs", and it was working before, but newer versions of
lorax strip out /etc/system-release which grub2 wants.

That was wrong anyways as we want the /etc/system-release from the
target root.

(Man, grub2 sucks...give me a declarative config file format I can just
 write)

https://bugzilla.gnome.org/show_bug.cgi?id=740697
2014-11-25 12:24:06 -05:00
Matthew Barnes 97558276e4 Refactor command-line parsing
Refactor command-line parsing to better utilize GOptionContext.  This
eliminates most of the manual parsing and global options are now shown
in the help output.

Here's a sample:

    $ ostree admin --help
    Usage:
      ostree admin [OPTION...] --print-current-dir|COMMAND

    Builtin "admin" Commands:
      cleanup
      config-diff
      deploy
      init-fs
      instutil
      os-init
      status
      switch
      undeploy
      upgrade

    Help Options:
      -h, --help         Show help options

    Application Options:
      --sysroot=PATH     Create a new OSTree sysroot at PATH
      -v, --verbose      Print debug information during command processing
      --version          Print version information and exit

https://bugzilla.gnome.org/show_bug.cgi?id=740295
2014-11-24 19:36:07 -05:00
Matthew Barnes 3a9127f103 Various manpage / usage blurb improvements
Corrections, clarifications, consistency.

Not a comprehensive overhaul of the manpages; they do still need work.
These are just flaws I've noted while studying the ostree command syntax.

https://bugzilla.gnome.org/show_bug.cgi?id=740097
2014-11-16 15:52:51 -05:00
Giuseppe Scrivano f699153f67 ostree-fetcher: move more logic into ostree_fetcher_request_uri_internal
Make _ostree_fetcher_request_uri_with_partial_async and
ostree_fetcher_stream_uri_async simple wrapper around the same
function, all the requests are created in the same place now.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2014-11-12 21:20:28 -05:00
Giuseppe Scrivano a5491f98cb ostree-fetcher: make _ostree_fetcher_stream_uri_sync private
Rename _ostree_fetcher_contents_membuf_sync to
ostree_fetcher_request_uri_to_membuf and drop unused argument
user_data.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2014-11-12 21:20:28 -05:00
Giuseppe Scrivano c2bc99bc16 ostree-fetcher: Remove _ostree_fetcher_request_uri_to_stream function
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2014-11-12 21:20:28 -05:00
Giuseppe Scrivano 5546c4d2fc ostree-metalink: use _ostree_fetcher_contents_membuf_sync
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2014-11-12 21:20:28 -05:00
Giuseppe Scrivano d48aca5645 ostree-fetcher: add max_size argument to change _ostree_metalink_request_sync
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2014-11-12 21:20:28 -05:00
Giuseppe Scrivano a4a4921d3f ostree-fetcher: remove two unused functions
_ostree_fetcher_query_state_text() and_ostree_fetcher_get_n_requests()
have no callers, so remove them.

If they will be needed, they can be easily copied back from the git
history.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2014-11-12 21:20:28 -05:00
Giuseppe Scrivano c832e9b751 metalink: have a single entry to the metalink module
Replace _ostree_metalink_request_async with a synchronous version.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2014-11-12 21:20:28 -05:00
Giuseppe Scrivano d5d73debd8 ostree_fetcher: new function _ostree_fetcher_contents_membuf_sync
Move code from ostree-repo-pull.c

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2014-11-12 21:20:28 -05:00
Matthew Barnes 5c9e83c02c syslinux: Adapt to bootloader UI changes
This was a side-effect of the bootloader UI changes in bug 739416,
but should now be sufficiently future-proofed with code comments.
2014-11-10 21:31:32 -05:00
Giuseppe Scrivano 1c2e20af25 ostree-repo-pull: Remove currently #if 0 static delta code
We can readd this in the static deltas work.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2014-11-06 18:48:29 -05:00
Matthew Barnes 59fd2b1e43 repo: Remove unused remote-cache dir
Unused leftover from when ostree had pack files; minor cleanup.

https://bugzilla.gnome.org/show_bug.cgi?id=739746
2014-11-06 13:31:58 -05:00
Matthew Barnes e4caf3c3cd Fix GFile leak in ostree_repo_constructed() 2014-11-05 15:41:33 -05:00
Colin Walters a35c4a564d commit: Split out file processing into helper function
There should be no logic change here, just reducing indentation.
2014-11-03 21:13:22 -05:00
Matthew Barnes e5b147c643 libostree: Reformat bootloader titles
Use the pattern:

  $PRETTY_NAME [$COMMIT_VERSION] (ostree[:$OSNAME][:$DEPLOYMENT_INDEX])

$OSNAME is only shown if there are multiple values.

$COMMIT_VERSION refers to the version tag in the commit's metadata.

$DEPLOYMENT_INDEX is only shown if no $COMMIT_VERSION is available.

https://bugzilla.gnome.org/show_bug.cgi?id=739416
2014-11-03 20:43:36 -05:00
Giuseppe Scrivano 0a7c1851d9 diff: Use gs_unref_object not gs_free on a GFile
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2014-10-30 12:17:32 -04:00
Giuseppe Scrivano cb053ae0f6 pull: Fix a used before initialization warning
src/libostree/ostree-repo-pull.c:1676:22: warning: 'flags' may be used uninitialized in this function [-Wmaybe-uninitialized]

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2014-10-30 12:16:59 -04:00
Giuseppe Scrivano 65c0594011 ostree-repo-pull: Fix inverted assert condition for maxdepth
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2014-10-30 11:46:02 -04:00
Colin Walters bcf40b4902 pull: Add depth support
For mirroring in particular, we really want to be able to traverse
all history.

$ ostree --repo=repo pull --mirror --depth=-1

https://bugzilla.gnome.org/show_bug.cgi?id=739240
2014-10-28 11:16:55 -04:00
Colin Walters 64dec0add8 libostree: Add ostree_repo_pull_with_options()
We potentially need a lot of argument types for pull.  Rather than
have a C function with tons of arguments, let's use a GVariant a{sv}
as a handy extensible (and immutable) bag of properties.

This is prepratory work for adding an option to pull to traverse
history.

https://bugzilla.gnome.org/show_bug.cgi?id=737844
2014-10-28 11:16:51 -04:00
Colin Walters fb3ad0037f grub2: Fix bad substitution in review followup
This caused GRUB2+EFI installations to fail.
2014-10-27 12:19:32 -04:00
Giuseppe Scrivano 7973a58607 checkout: permit checkout of a single file
fixes a coredump when using a command like:

$ ostree --repo=repo checkout -U --subpath=/usr/lib/passwd \
  fedora-atomic/rawhide/x86_64/docker-host usrlib-new

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2014-10-24 11:18:45 -04:00
Colin Walters 0c89abee6d Add (non-atomic) support for GRUB2 + UEFI
We need basic support for UEFI - many newer servers don't support
BIOS compatibility mode anymore.

However, this patch only implements non-atomic because UEFI is FAT, and
we can't do the previous design for OSTree of atomic swap of
/boot/loader.

The Fedora/RHEL UEFI layout has the kernels on a "real" /boot
partition, and /boot/efi/EFI/$vendor just holds the grub2 UEFI binary
and grub.cfg.

Following this, /boot/loader is still on the OS boot partition, and we
still atomically swap it.  This potentially paves the way to atomic
upgrades in the future.

https://bugzilla.gnome.org/show_bug.cgi?id=724246
2014-10-23 21:42:30 -04:00
James Antill 81a484459a admin status: Print the version 2014-10-23 11:24:29 -04:00
James Antill aab6e7bdf2 dump: Print the version when dumping a commit, log/show/etc. 2014-10-23 10:21:17 -04:00
James Antill 52c0d89a8c admin: Add ot_admin_checksum_version to get a dup of the version for a commit 2014-10-23 10:21:17 -04:00
Giuseppe Scrivano 0785c398b0 fsck: report the correct error when a present file cannot be loaded
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2014-10-22 18:06:29 +02:00
Giuseppe Scrivano 26bdfae5bc make ostree_cmd__private__ ret value const 2014-10-21 16:18:30 -04:00
Colin Walters 5fb8bb955e repo: When writing to a non-root owned repo, chown() objects to match
Some package systems need to be run as root, so the process linking to
libostree may also be root.  However, it's reasonable to have the
target repository be owned by a uid other than root.

This patch makes it Just Work by chowning the file content to match.

Note this only operates on archive-z2 repositories, because you can't
usefully serve bare repositories via HTTP.

https://bugzilla.gnome.org/show_bug.cgi?id=738954
2014-10-21 15:43:43 -04:00
Colin Walters cdfcf09316 Add "ostree remote delete" and corresponding API
For Anaconda, we have an ugly bootstrapping problem where we need to
add the remote to the repository's config, then do a pull+deploy, then
remove and re-add the config, because /etc/ostree/remotes.d doesn't
exist yet in the target system.

https://bugzilla.gnome.org/show_bug.cgi?id=738698
2014-10-17 19:09:42 -04:00
Colin Walters 884a9b27e9 repo: Disable uncompressed object cache on non-writable repos
While we did support disabling the uncompressed-objects-cache
per-repository:

1) We didn't actually respect that operation when doing
   CHECKOUT_MODE_USER on archive-z2 repositories
2) It'd be better to automatically detect we can't write to the
   repo and disable the uncompressed cache then.
2014-10-16 14:54:34 -04:00
Colin Walters d546abfa2a libostree: Add initial GRUB2 support
In this approach, we drop a /etc/grub.d/15_ostree file which is a
hybrid of shell/C that picks up bits from the GRUB2 library (e.g. the
block device script generation), and then calls into libostree's
GRUB2 code which knows about the BLS entries.

This is admittedly ugly.  There exists another approach for GRUB2 to
learn the BLS specification.  However, the spec has a few issues:

https://www.redhat.com/archives/anaconda-devel-list/2014-July/msg00002.html

This approach also gives a bit more control to the admin via the
naming of the 15_ostree symlink; they can easily disable it:

Or reorder the ostree entries ahead of 10_linux:

Also, this approach doesn't require patches for grub2, which is an
issue with the pressure to backport (rpm-)OSTree to EL7.
2014-10-16 14:15:00 -04:00
Giuseppe Scrivano 92c5a9f992 http: set the HTTP status on directory listing
Fix a HTTP response header like the following:

HTTP/1.1 0 (null)
Server: ostree-httpd libsoup/2.48.0
Date: Tue, 07 Oct 2014 11:19:22 GMT
Content-Type: text/html
Content-Length: 12533

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2014-10-07 13:19:36 +02:00
Colin Walters 5dba2442a1 trivial-httpd: Another followup fix - need to listen on all interfaces
This was the intended design, but it regressed when updating.
2014-10-06 17:43:02 -04:00
Giuseppe Scrivano 68b37dc001 libostree: fix a gobject-introspection warning
src/libostree/ostree-repo.c:1759: Warning: OSTree:
  ostree_repo_import_object_from: unknown parameter 'checksum' in
  documentation comment, should be 'sha256'

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
2014-10-06 19:03:34 +02:00
Colin Walters b3ad113f78 Add "unconfigured-state" concept to origin files
Some operating systems may come with external tools for subscription
management that drive access to the content.  In that case, the origin
file may not be useful (for example, it could refer to an installer
ISO).

This patch will allow OS installers to inject that state, with a
useful error message, directing the system administrator to an
external tool.

See: https://github.com/projectatomic/rpm-ostree/issues/31

https://bugzilla.gnome.org/show_bug.cgi?id=737686
2014-10-03 14:03:55 -04:00
Colin Walters 63abc1b513 pull: Support full recursive mirrors of repositories with summary file
Now that we have a summary file, we can use it to allow a simple:

ostree pull --mirror

To download the latest commit on every branch.  Also, for a case I'm
dealing with there's only one branch, but I don't want mirror users to
have to hardcode it.

https://bugzilla.gnome.org/show_bug.cgi?id=737807
2014-10-03 13:56:40 -04:00
Colin Walters 7ab1fb2369 Add API to directly link() objects between repositories
And use it in pull-local.  As one might expect, this is blazingly fast
if they're on the same filesystem.

I'll be using this to "promote" builds between different repositories.
2014-10-01 23:20:35 -04:00
Colin Walters 1b8759a67e upgrader: Hold a ref to the origin
We unref it in _finalize, so we need to hold a ref.  I *thought*
this was the source of
https://github.com/projectatomic/rpm-ostree/issues/30

But apparently not =/
2014-09-30 11:02:14 -04:00
Owen W. Taylor d64d003af0 ostree admin: Add a --print-current-dir option
Add an option --print-current-dir that prints the current deployment
directory to stdout and exits.

https://bugzilla.gnome.org/show_bug.cgi?id=731051
2014-09-26 10:40:38 -04:00
Owen W. Taylor 262cba09c0 ostree admin instutil set-kargs: make more flexible
Add command line arguments:
 --import-proc-cmdline: import values from /proc/cmdline
 --merge: import current values
 --replace=ARG=VALUE: replace value
 --append=ARG=VALUE: append a new argument

Extra command line arguments are treated like --append=, which
gives backwards compatibility.

https://bugzilla.gnome.org/show_bug.cgi?id=731051
2014-09-26 10:40:37 -04:00
Colin Walters edf9cf5a7f trivial-httpd: Further fixes for previous commit 2014-09-25 16:03:06 -04:00
Colin Walters 1b19a989fb Followup to SoupServer port deprecation
My patch didn't actually work with newer libsoup.
2014-09-25 15:57:58 -04:00
Owen W. Taylor 3400f2d2ae --help should always go to stdout
The standard convention is that the  output of --help should go to standard
output (so that it can be piped to a pager and searched.) See, e.g., the
GNU coding standards.

https://bugzilla.gnome.org/show_bug.cgi?id=737194
2014-09-25 14:28:58 -04:00
Owen W. Taylor c9018c785c ostree admin instutil: make --help work for subcommands
Even though the subcommands don't take arguments, use a GOptionContext
so that --help works as expected.

https://bugzilla.gnome.org/show_bug.cgi?id=737194
2014-09-25 03:08:06 -04:00
Owen W. Taylor 9d72ff2d8e Fix help output for nested subcommands
Only 'ostree admin' was appearing in the help message for commands
nested within ostree admin.

https://bugzilla.gnome.org/show_bug.cgi?id=737194
2014-09-25 03:08:06 -04:00
Owen W. Taylor 1dca556691 Pass --help to the most nested subcommand
'ostree admin <x> --help' and 'ostree admin instutil <x> --help' should
give help on the deepest subcommand, not on 'ostree admin'.

https://bugzilla.gnome.org/show_bug.cgi?id=737194
2014-09-25 03:08:06 -04:00
Owen W. Taylor 40f490ed11 ostree admin: Fix return value from 'ostree admin [instutil]'
'ostree admin' and 'ostree admin instuil' with no arguments were meant to fail,
but the logic was wrong; add an assertion on the return value from all ostree
commands to catch similar problems in the future.

https://bugzilla.gnome.org/show_bug.cgi?id=737194
2014-09-25 03:08:06 -04:00
Colin Walters 8f4ffa6950 deploy: Copy parent directories for modified config files
Previously, in the case where a parent directory of a modified config
file was removed, we would throw an exception.  This happens when
switching from a tree that has some software (e.g. firewalld), to one
that does not.

While it's nice to have this warning that your config file probably no
longer applies, there's no need to make it so...fatal.

It's particularly problematic that the only easy workaround is to
remove the config files from your current tree - which breaks
rollback.

The solution then is for for us to take ownership of the parent
directories too into the new /etc.  Admins can clean up these files
afterwards at any time.

https://bugzilla.gnome.org/show_bug.cgi?id=734293
2014-09-16 12:06:10 -04:00
Colin Walters b756a13a65 Extract opendirat() helper function into libotutil
We were duplicating the code to do an opendirat() in a few places.
2014-09-16 11:34:39 -04:00
Colin Walters dfeb27eca5 deploy: Remove legacy "current" symbolic links
Per comment, this was a temporary non-atomic hack, and it's time to
remove it.

https://bugzilla.gnome.org/show_bug.cgi?id=731051
2014-09-13 10:44:29 -04:00
Colin Walters 12e3ed83d3 deploy: Clean up leftover state before creating new directories
This fixes a regression introduced with https://git.gnome.org/browse/ostree/commit/?id=7baa600e237b326899de2899a9bc54a6b863943c

The original code in "ostree admin upgrade" had a comment:

 /* Here we perform cleanup of any leftover data from previous
  * partial failures. This avoids having to call gs_shutil_rm_rf()
  * at random points throughout the process. */

But since I deleted that initial cleanup call, we *do* need to do the
cleanup during the process run.  It turns out there are only a few
places this is necessary.

https://bugzilla.gnome.org/show_bug.cgi?id=733030
2014-09-13 10:43:14 -04:00
Colin Walters 7b01bd2e43 deploy: Consistently use fd-relative API
While looking to fix a different bug here, I found the current
state of things where we had a mix of fd-relative API versus not
frustrating.

Change the code around to consistently use *at, and also add some more
tests.
2014-09-13 10:41:59 -04:00
Colin Walters 34c336c1f3 Work around deprecation of SoupServer's port property
See https://bugzilla.gnome.org/show_bug.cgi?id=732184
2014-09-09 09:30:02 -04:00
Colin Walters a593078722 pull: Squash some uninitialized variable warnings 2014-09-08 13:43:12 -04:00
Colin Walters b30f32e0da Drop g_type_init() call now that we depend on GLib 2.36 2014-09-07 12:58:10 -04:00
Colin Walters f8f5da219e Add repository "summary" file and metalink support
For Fedora and potentially other distributions which use globally
distributed mirrors, metalink is a popular solution to redirect
clients to a dynamic set of mirrors.

In order to make metalink work though, it needs *one* file which can
be checksummed.  (Well, potentially we could explode all refs into the
metalink.xml, but that would be a lot more invasive, and a bit weird
as we'd end up checksumming the checksum file).

This commit adds a new command:

$ ostree summary -u

To regenerate the summary file.  Can only be run by one process at a
time.

After that's done, the metalink can be generated based on it, and the
client fetch code will parse and load it.

https://bugzilla.gnome.org/show_bug.cgi?id=729585
2014-09-03 13:21:52 -04:00
Colin Walters c08a4503ce fetcher: Close request body on error paths
Otherwise, we're potentially holding up subsequent requests.

I was hitting this when testing the metalink code, where we want to
continue doing more fetches after hitting a 404.

https://bugzilla.gnome.org/show_bug.cgi?id=729585
2014-08-27 14:00:31 -04:00
Anne LoVerso 6dfe99a283 pull: Fix use-after-free
The strchr() was pointing into a string we were freeing.
2014-08-21 13:57:31 -04:00
Anne LoVerso 3742c32945 repo-pull: Allow pulling only one directory
Changes the pull API to allow pulling only a single directory instead
of the whole deployment.  This option is utilized by the check-diff
option in rpm-ostree.

Add a new state directory to hold <checksum>.commitpartial files, so
we know that we've only downloaded partial state.
2014-08-20 15:09:32 -04:00
Colin Walters 6a55d2c32f sysroot: Add precondition that load() has been called for some API calls
We require the caller to invoke load() at the moment to do most
things.
2014-08-15 15:21:40 -04:00
Colin Walters 82fb07e4ae deploy: Also honor multiply-specified kernel args with existing config
If we deployed a new tree inside an existing OS, inheriting kernel
args, we need to use append() instead of replace() to avoid collapsing
multiply specified kernel arguments like console=/dev/foo
console=/dev/bar.

Reported-by: Dusty Mabe <dusty@dustymabe.com>
2014-08-03 14:34:28 -04:00
Colin Walters 05114f8565 deploy: Propertly append multiple kernel arguments with --karg-proc-cmdline
We want to allow multiply specified arguments, as it's useful for
things like console= and rd.lvm.lv.

See: https://bugzilla.gnome.org/show_bug.cgi?id=721136

Reported-by: Dusty Mabe <dustymabe@gmail.com>
2014-08-03 14:11:40 -04:00
Colin Walters b97a5f59df core: Support a per-remote "proxy" configuration option
We don't want to have to force people to set it in the environment.

https://bugzilla.gnome.org/show_bug.cgi?id=733734
2014-07-28 13:49:38 -04:00
James Antill 43bba232a0 Display bytes/second during pull
Add start time to the AsyncProgress object, use it in the common
progress.
2014-07-28 08:37:15 -04:00
James Antill 249da7aef2 pull: override max-conns-per-host to 8, for faster downloads
Modern web browsers are all converging on 6 or 8.  Let's match that.
2014-07-25 10:22:55 -04:00
Colin Walters d5e813c52c refs: Don't try searching for input strings that can't be objects
I noticed OSTree was a bit slower, did some investigation
and saw we were enumerating all objects for things like

$ ostree rev-parse blah

Since "blah" can never be an object (because of the 'l' and 'h'), just
return no matches.
2014-07-23 15:09:24 -04:00
Colin Walters 9bd229dff5 pull-local: Write detached metadata into the correct repository
We want to write to the dest repo, not src.

Noticed while reviewing this code for some other bug.

https://bugzilla.gnome.org/show_bug.cgi?id=733579
2014-07-22 15:43:33 -04:00
Colin Walters cdaa9517af repofile: Avoid segfault if querying child in non-directory
The user might "ostree ls /usr/bin/bash/blah", which previously would
segfault.

A somewhat related future enhancement here would be for "ostree ls" to
follow symbolic links.

Reported-by: Dusty Mabe <dustymabe@gmail.com>

https://bugzilla.gnome.org/show_bug.cgi?id=733476
2014-07-21 07:58:07 -04:00
Colin Walters e392820541 core: Do enumerate ff/ objects as well
An embarassing off-by-one here.  I noticed we weren't pruning them.

https://bugzilla.gnome.org/show_bug.cgi?id=733458
2014-07-21 07:25:51 -04:00
Colin Walters 1834a71b1f core: Do enumerate content objects in archive-z2 repositories
Prune has worked fine on bare repositories for some time, but now that
I finally try to delete data on the server side, I notice we weren't
actually enumerating content objects =/

That caused them to not be pruned.

https://bugzilla.gnome.org/show_bug.cgi?id=733458
2014-07-21 07:25:51 -04:00
Colin Walters 72da2e0c36 core: Unify object deletion code with prune
The prune API duplicated logic to delete objects, and furthermore the
core API to delete an object didn't clean up detached metadata.

Fix the duplication by doing the obvious thing: prune should call
_delete.

https://bugzilla.gnome.org/show_bug.cgi?id=733452
2014-07-20 08:57:37 -04:00
Colin Walters 1dd4435936 Fix typo in previous commit 2014-07-18 15:49:40 -04:00
Anne LoVerso fd56952d6b ostree-repo-resolve-rev: Add function to accept a partial checksum
This patch adds a function that will parse a partial checksum when
resolving a refspec.  If the inputted refspec matches a truncated
existing checksum, it will return that checksum to be parsed.  If
multiple truncated checksums match the partial refspec, it is not
unique and will return false.  This addition is inspired by the same
functionality in Docker, which allows a user to reference a specific
commit without typing the entire checksum.

partial checksums: Add function to abstract comparison

This modifies the list_objects and list_objects_at functions
to take an additional argument for the string that a commit starts
with.  If this string arg is not null, it will only list commit
objects beginning with that string.  This allows for a new function
ostree_repo_list_commit_objects_starting_with to pass a partial string
and return a list of all matching commits.  This improves on the
previous strategy of listing refs because it will list all commit objects,
even ones in past history.  This update also includes bugfixes on
error handling and string comparison, and changes the output structure
of resolve_partial_checksum. The new strcuture will no longer return FALSE
without error.  Also, the hashtable foreach now uses iter.  Also
includes modified test file
2014-07-18 15:15:44 -04:00
James Antill 3645afc6d7 core: Use ostree_commit_get_parent in ostree_repo_resolve_rev 2014-07-18 10:10:06 -04:00
Owen W. Taylor ff6883ca06 ostree-remount: mount a tmpfs on /var if necessary so /var is read-write
/var needs to be read-write for a functioning system. Various
systemd services will fail to start if /var is read-only. After we
remount /var (or if we skip remounting /var because / is read-only),
if /var is still readonly, mount a tmpfs file system on /var.

While this isn't strictly part of ostree, keeping it here makes sense
because it keeps twiddling around with the /var mount in one place
for easier coordination and maintenance. This will likely need updating
if systemd gains better support for a readonly root filesystem.

https://bugzilla.gnome.org/show_bug.cgi?id=732979
2014-07-09 21:24:07 -04:00
Colin Walters 772409e945 pull: Only set ca path if we actually have a setting for it
Otherwise, GTlsFileDatabase ends up accepting NULL for anchors, and
then we don't trust any CAs at all.

https://bugzilla.gnome.org/show_bug.cgi?id=726256
2014-07-01 08:53:39 -04:00
Colin Walters f60bac45fd core: add "tls-ca-path" option
Some organizations will want to use private Certificate Authorities to
serve content to their clients.  While it's possible to add the CA
to the system-wide CA store, that has two drawbacks:

1) Compromise of that cert means it can be used for other web traffic
2) All of ca-certificates is trusted

This patch allows a much stronger scenario where *only* the CAs in
tls-ca-path are used for verification from the given repository.

https://bugzilla.gnome.org/show_bug.cgi?id=726256
2014-06-27 13:16:47 -04:00
Anne LoVerso 9cb9a31b4f manpage: Add separate manpages for each command
Using docker as a model, this update creates separate manpages
for each OSTree command, such that the main manpage is not
cluttered and the separate pages can provide more in-depth detail
and exanples that might be useful to a user.  Each page includes
synopsis, description, example, and a list of options if needd.
This update also alphabetizes the usage error output for ostree
and ostree admin so that it matches the list on the manpage.
2014-06-23 16:02:37 -04:00
Colin Walters 40f99f764e [staticanalysis]: Delete an unused variable
https://bugzilla.gnome.org/show_bug.cgi?id=732020
2014-06-21 18:08:49 -04:00
Colin Walters 4d04b148d8 [staticanalysis]: Fix in_status_line
We need to end the status line *after* we've done a pull, as ostree
admin upgrade does.  Also add the correct in_status_line assignment.

https://bugzilla.gnome.org/show_bug.cgi?id=732020
2014-06-21 18:08:49 -04:00
Colin Walters 5936740b95 [staticanalysis]: Fix some dead code
One was an unused variable, the other is actually dead because we
can't have mfile != NULL.

https://bugzilla.gnome.org/show_bug.cgi?id=732020
2014-06-21 18:08:49 -04:00
Colin Walters 54079982d8 [staticanalysis]: Actually check errors on splice() of objects
We were using unsigned size when we should have been using signed,
this means we basically weren't checking for errors on write...ouch.

Luckily if we e.g. hit ENOSPC during a pull, the checksums wouldn't
match and we'd return an error anyways.  However when writing an
object, we'd end up silently ignoring it =/

https://bugzilla.gnome.org/show_bug.cgi?id=732020
2014-06-21 18:08:49 -04:00
Colin Walters 2dc0cea503 [staticanalysis]: Add assertion to pacify analyzer
This condition can't actually be hit, let's hint that's the case.

https://bugzilla.gnome.org/show_bug.cgi?id=732020
2014-06-21 18:08:49 -04:00
Colin Walters dfda6e3a46 [staticanalysis]: Add missing va_end()
https://bugzilla.gnome.org/show_bug.cgi?id=732020
2014-06-21 18:08:49 -04:00
Colin Walters d7067975e7 [staticanalysis]: Fix two uses of uninitialized variables
https://bugzilla.gnome.org/show_bug.cgi?id=732020
2014-06-21 18:08:49 -04:00
Colin Walters 6eac108560 remote: Minor argument parsing cleanup
Only access relevant portion of argv after we've verified argc.

https://bugzilla.gnome.org/show_bug.cgi?id=731984
2014-06-21 15:57:43 -04:00
Colin Walters 1962623bbc remote: Fix regression in previous commit wrt adding branches
branches is always NULL at that point, we want to look at argc.

https://bugzilla.gnome.org/show_bug.cgi?id=731984
2014-06-21 15:57:43 -04:00
Colin Walters 2265ccdeb3 libostree: Add ostree_repo_remote_add() API, port "ostree remote add"
At least one external tool is using the API, and wants to add a
remote, but all of the logic right now is in the tool.  Move it to the
library.

https://bugzilla.gnome.org/show_bug.cgi?id=731984
2014-06-20 15:36:46 -04:00
Colin Walters bb043b319f pull: Emit a better error if the remote isn't found
The generic GKeyFile error isn't quite informative enough here.

I hit this with the new compose process where we don't automatically
inject a configured remote into the generated disk images; we expect
people to add them.

https://bugzilla.gnome.org/show_bug.cgi?id=731346
2014-06-19 04:39:28 -04:00
James Antill d4d4cbda7a core: Fix g_file_get_uri/get_parse_name for non-root ostree files 2014-06-18 17:34:04 -04:00
James Antill 54c271001f commit: Hide --disable-fsync option, add --fsync which takes a boolean
Per discussion on mailing list, the --disable-fsync has a
double-negative thing going on; --fsync=false is clearer.
2014-06-18 17:33:03 -04:00
Anne LoVerso f2f3b67804 cat: Fix a typo in usage error printout 2014-06-17 16:06:15 -04:00
Colin Walters 6a5e66b152 pull: Support --mirror option
There's several use cases for calling into ostree itself to do
mirroring, instead of using bare rsync.  For example, it's a bit more
efficient as it doesn't require syncing the objects/ directory.

https://bugzilla.gnome.org/show_bug.cgi?id=728351
2014-06-16 17:14:13 -04:00
Colin Walters 4ac27caefd log: Print a friendly error if we haven't downloaded the complete history
For the local repository on the system, it's not the usual case to
have the complete compose history.  Rather than erroring out, provide
a bit more friendly message.

https://bugzilla.gnome.org/show_bug.cgi?id=731538
2014-06-11 16:41:35 -04:00
Colin Walters e31daf448a libostree: Silently ignore EPERM when setting EXT2_IMMUTABLE_FL
In the case of running ostree as non-root on a regular filesystem (not
tmpfs which doesn't support immutable), we should just silently do
nothing if we encounter EPERM.  Cache the result to avoid spam in
strace.

https://bugzilla.gnome.org/show_bug.cgi?id=728006
2014-06-09 14:44:17 -04:00
Colin Walters 24c64d6b4f libostree: Make OstreeFetcher explicitly private
Per previous commit.  We're not attempting to export an HTTP to
clients, the high level API is ostree_repo_pull().

https://bugzilla.gnome.org/show_bug.cgi?id=731369
2014-06-09 14:04:53 -04:00
Colin Walters 4ce93db595 libostree: Make OstreeTlsCertInteraction explicitly private
Similar to previous commit, this should be private.

https://bugzilla.gnome.org/show_bug.cgi?id=731369
2014-06-09 14:04:53 -04:00
Colin Walters 9f734494ff libostree: Make OstreeLibarchiveInputStream explicitly private
We weren't installing the headers, but at the moment all symbols
starting with ostree_ were being exported.  Fix that by prefixing
non-static symbols with '_'.

https://bugzilla.gnome.org/show_bug.cgi?id=731369
2014-06-09 14:04:53 -04:00
Colin Walters 51bf1fab0f repo: Support fsync=false configuration
See rationale in the updated docs.  Basically developer use cases as
well as UPS-backed servers.
2014-06-05 17:35:50 -04:00
Vadim Rutkovsky ef00572999 Honor disable_fsync during checkout also
Finally, fsync to ensure all entries are on disk, unless disabled.
 We support disabling this for cases like server-side buildroot
 construction where we don't need to be robust against power loss
2014-06-04 16:14:46 +02:00
Colin Walters cb43d2942f ostree-remount: Check for / being *mounted* read-only, not necessarily writable
The previous S_IMMUTABLE commit broke ostree-remount; / is now not
actually writable.  All we really wanted to know though was whether it
was *mounted* writable, so check that via statvfs() which is cleaner
anyways (i.e. not via access() which kernel people hate).

https://bugzilla.gnome.org/show_bug.cgi?id=728006
2014-06-03 17:38:00 -04:00
James Antill f22fa92aef Add --disable-fsync option to both commit and pull (non-local)
On some storage configurations, fsync() can be extremely expensive.
Developers and users with slow hard drives may want the ability to opt
for speed over safety.

Furthermore, many production servers have UPS and stable kernels, and
the risk of not fsync'ing in that scenario is fairly low.  These users
should also be able to opt out.
2014-06-03 15:41:02 -04:00
Colin Walters b4d21e9406 deploy: Set the immutable bit on the deployment root
This prevents people from creating new directories there and expecting
them to be persisted.  The OSTree model has all local state to be in
/etc and /var.

This introduces a compile-time dependency on libe2fsprogs.

We're only doing this for the root directory at the moment.

https://bugzilla.gnome.org/show_bug.cgi?id=728006
2014-06-03 13:41:48 -04:00
Colin Walters 47610b45c2 Limit metadata to 10 MiB
If fetching GPG-signed commits over plain HTTP, a MitM attacker can
fill up the drive of targets by simply returning an enormous stream
for the commit object.

Related to this, an attacker can also cause OSTree to perform large
memory allocations by returning enormous GVariants in the metadata.

This helps close that attack by limiting all metadata objects to 10
MiB, so the initial fetch will be truncated.

But now the attack is only slightly more difficult as the attacker
will have to return a correctly formed commit object, then return a
large stream of < 10 MiB dirmeta/dirtree objects.

https://bugzilla.gnome.org/show_bug.cgi?id=725921
2014-05-27 14:15:27 -04:00
Colin Walters 6002356747 fetcher: Unref pending result when completing
Otherwise we were just leaking it.

https://bugzilla.gnome.org/show_bug.cgi?id=725921
2014-05-27 12:01:14 -04:00
Colin Walters f1ccaa581e repo: Don't require a txn for writing
The current "transaction" symlink was introduced to fix issues with
interrupted pulls; normally we assume that if we have a metadata
object, we also have all objects to which it refers.

There used to be a "summary" which had all the available refs, but I
deleted it because it wasn't really used, and was still racy despite
the transaction bits.

We still want the pull process to use the transaction link, so don't
delete the APIs, just relax the restriction on object writing, and
introduce a new ostree_repo_set_ref_immediate().
2014-05-26 18:49:17 -04:00
Colin Walters a4d01976bb repo: Only load /etc/ostree/remotes.d for system repo
They shouldn't be loaded for random test/personal repositories.  Doing
so triggers another bug in that we return them from
ostree_repo_get_config() when then causes clients to write them out
permanently to disk with ostree_repo_write_config().  This caused test
suite failures.
2014-05-26 18:38:35 -04:00
Colin Walters 62c2a204c9 trivial: commit: Fix docstring typos 2014-05-26 17:57:36 -04:00
Colin Walters 78726244a0 deploy: Convert remaining g_print() to systemd journal messages
These bits should be logged more sanely.
2014-05-24 13:19:29 -04:00
Colin Walters bf37a9c741 deploy: Remove deployment root print
It doesn't look very professional; in the future though we should have
a progress bar here or something.
2014-05-24 12:30:56 -04:00
Colin Walters 19abd8277f selinux-ensure-labeled: Support no arguments to just traverse deployment root
This makes it easy to use for the case where rpm-ostree-toolbox is
injecting systemd services into the deployment root, and we don't
actually need to traverse the whole FS.
2014-05-22 22:51:21 -04:00
Colin Walters a864190a04 admin switch: Allow switching just remote names
This is a followup to the previous commit; for the installation media
case we want to keep the current origin ref, and only switch remotes.
2014-05-15 08:58:44 -04:00
Colin Walters 24e1bf2552 admin switch: Support switching remotes as well
Say I have an installation from CDROM; the remote name may be
"installmedia" or something like that.  We want to allow also
switching remotes.
2014-05-12 17:31:34 -04:00
Colin Walters f47a20fb81 Support /etc/ostree/remotes.d
For many OS install scenarios, one runs through an installer which may
come with embedded data, and then the OS is configured post-install to
receive updates.

In this model, it'd be nice to avoid the post-install having to rewrite
the /ostree/repo/config file.

Additionally, it feels weird for admins to interact with "/ostree" -
let's make the system feel more like Unix and have our important
configuration in /etc.

https://bugzilla.gnome.org/show_bug.cgi?id=729343
2014-05-08 18:59:24 -04:00
Owen W. Taylor 466671407d trivial-httpd: flush after writing the port to stdout
The option --port-file=- is most useful when the stdout of the daemon
is programatically redirected and not going to a terminal. The
flush-after-a-line behavior of stdout is specific to terminals, so
we need an explicit flush.

https://bugzilla.gnome.org/show_bug.cgi?id=729609
2014-05-05 19:02:59 -04:00
Colin Walters 25ad4a9f65 pull: Add tls-client-cert-{path,key} (if we have new enough libsoup)
This is an actually working version of client-side certificates.
Depends on:
See: https://bugzilla.gnome.org/show_bug.cgi?id=334021

We detect whether libsoup is new enough for this.

https://bugzilla.gnome.org/show_bug.cgi?id=729356
2014-05-01 17:13:13 -04:00
Colin Walters 64b4ec6ef0 Revert "pull: Add tls-client-cert-{path,key}"
This reverts commit 94f9ee7bce.
Doesn't actually work, see
https://bugzilla.gnome.org/show_bug.cgi?id=334021
2014-05-01 10:23:11 -04:00
Colin Walters 94f9ee7bce pull: Add tls-client-cert-{path,key}
These can be used to present a client certificate when making requests
to a repository.
2014-05-01 07:57:06 -04:00
Colin Walters 7cc4a20f3b pull: Correctly handle error state when fetching optional data
For the static deltas work, we're using the already-extant internal
API to perform a HTTP fetch for optional data - static deltas are
optional.

Except that we didn't correctly unset the error if we were doing an
optional fetch and the data wasn't found.
2014-04-29 10:59:57 -04:00
Colin Walters 626fc8519b pull: Stay in mainloop if we're synchronously fetching URI
The static deltas work will be doing some synchronous fetching even
after refs are downloaded.
2014-04-29 10:59:57 -04:00
Colin Walters 006e8f0073 deltas: Link to liblzma, add internal API to use it
For future static deltas work, we'll be linking to liblzma.  Since
it's fairly widespread, let's just make it a hard dependency.
2014-04-29 10:59:57 -04:00
Colin Walters f040c02048 libostree: Add _finish() API to async progress
Since OstreeAsyncProgress queues to the mainloop, we might "lose" the
last message.  Give callers a way to force a flush.
2014-04-29 10:59:57 -04:00
Colin Walters 349083194d otutil: Correctly add ref to bytes when creating GVariant
This one went undiscovered for a while because it turned out we
weren't using it...
2014-04-29 08:53:28 -04:00
Colin Walters 959db9f0de core: Add _STRING variants of GVariant object formats
For the static deltas work, we're going to embed a commit in the delta
superblock, so we need a format string without the G_VARIANT_TYPE().
2014-04-29 08:51:00 -04:00
Colin Walters 021c4ee9d3 show: Fix segfault if we can't find an object
We need to use the original rev here.
2014-04-29 07:42:30 -04:00
Colin Walters 5346e0615b libotutil: Fix two bugs in usage of posix_fallocate()
* It's invalid to call it with a size of 0, so don't do that.

* posix_* apparently don't set errno.  So capture the return value and
  use that.
2014-04-29 07:40:25 -04:00
Colin Walters 22ab7b3e79 core: Use posix_fallocate() when writing objects
This helps ensure the filesystem allocates space efficiently.
2014-04-28 19:42:12 -04:00
Cosimo Cecchi 4903fba08c os-init: don't create /var/log/journal on deploy
Leave the policy of whether to persistently store journal logs to the
system integrator when the default journald configuration is in use.

https://bugzilla.gnome.org/show_bug.cgi?id=728762
2014-04-25 16:47:02 -04:00
Colin Walters 2c5d726076 deploy: fsync() copy of /etc
This unfortunately requires reimplementing gs_shutil_cp_a(), except
while we're here, we also use the *at calls.
2014-04-11 18:43:58 -04:00
Colin Walters b423d9e9cf deploy: fsync() kernel/initramfs and bootloader config parent directories
Ensure they've hit disk.
2014-04-11 18:43:27 -04:00
Colin Walters 646c8be8dc deploy: fdatasync() bootloader configuration files
Yet more data we're writing out that needs to be sync'd.
2014-04-11 18:43:27 -04:00
Colin Walters 18aaa49724 libotutil: Make use of dirfd-relative API in ot_gfile_replace_contents_fsync()
It's just more efficient.
2014-04-11 18:43:27 -04:00
Colin Walters 053dd72649 deploy: Ensure .origin file is fsync()ed on disk
Along with its parent directory.
2014-04-11 18:43:27 -04:00
Colin Walters 0b8495f09a deploy: Also fsync parent directory of modified config files 2014-04-11 18:43:13 -04:00
Colin Walters 38feca0e4d deploy: Ensure that any modified config files are fsync()d
It really wouldn't do for one to be missing one's ssh keys for
example...
2014-04-11 18:43:13 -04:00
Colin Walters 1b16553600 checkout: fsync() directory on checkouts
We want to be really sure that our deployment roots have hit the disk.
2014-04-11 18:43:13 -04:00
Colin Walters 7acad0a2e5 deploy: Ensure that all directories we create are fsync()d
There's two halves to this; first, when we create an hierarchy, we
need to call fsync().  Second, we need to fsync again anytime after
we've modified a directory.
2014-04-11 18:43:13 -04:00
Colin Walters b19aea441a libotutil: Add API to create directory hierarchy recursively *and* fsync
To be really sure that any directory entries have hit disk we need to
call fsync() on the directory fd.  This API allows us to conveniently
create a directory hierarchy, fsyncing all of it along the way.
2014-04-11 18:43:13 -04:00
Colin Walters d27c78eab5 pull: Display download progress of individual objects as we get it
It was kind of annoying at least for rpm-ostree upgrades since /boot
happens to be first and we eat a 21MB initramfs with no download
progress.

https://bugzilla.gnome.org/show_bug.cgi?id=726348
2014-04-11 01:31:14 -04:00
Colin Walters d4d4ef552d libotutil: Fix mistaken return of TRUE in error path
Gah, must have been a refactoring bug.
2014-04-09 01:51:43 -04:00
Colin Walters 1bdabda5f3 Use external libgsystem 2014.2
It's been split off for a while, let's kill the code duplication.

Among other things, this fixes the systemd detection for the journal
logging.
2014-04-04 16:52:37 -04:00
Colin Walters 4bd8b5c850 switch: Always allow chronological downgrades
Ideally we'd have something a bit more strict, but...without
downloading the parentage, this is all we can do at the moment.
2014-04-04 16:51:37 -04:00
Colin Walters 76a976817f deploy: Call fsync() on parent directory before/after symlink swap
Let's be conservative here and try hard to ensure the symlink has the
correct content on disk.
2014-04-03 22:39:41 -04:00
Colin Walters 829e0382ab bootloader: fdatasync() bootloader configuration
Let's be a bit more conservative here and actually fdatasync() the
configurations we're generating.

I'm seeing an issue at the moment where syslinux isn't finding the
config sometimes, and while I don't think this is the issue, let's try
it.
2014-04-03 22:32:30 -04:00
Colin Walters cc8bacf8b4 bootloaders: Always write out bootloader config file
There was an attempted optimization to only write if changed, but this
is broken - we always write the bootloader config into a new
directory.

In theory we should only be writing if it changed, but let's not do a
broken optimization.
2014-04-03 21:08:03 -04:00
Colin Walters ac40de5773 deploy: Add a debug message for which bootloader is used
Debugging something...
2014-04-03 19:39:08 -04:00
Colin Walters 890d685c79 deploy: Add an assertion that deployment directory exists
I may be seeing something going wrong in rpm-ostree, just adding this
assertion for my own sanity.
2014-04-03 18:29:07 -04:00
Colin Walters 49d736c265 tool: End status line after pull
Otherwise we potentially get overlapped output.
2014-04-03 08:04:32 -04:00
Colin Walters edc8665206 fetcher: Honor http_proxy environment variable
https://bugzilla.gnome.org/show_bug.cgi?id=706809
2014-04-01 19:12:40 -04:00
Colin Walters 7bd00e3005 checkout: Always do chmod even in _MODE_USER
The previous commit here changed things so that we do mkdir(x, 0700),
then fchmod later only if we created the directory.

However the logic was incorrect; we still need to chmod even in
MODE_USER if we created the directory.
2014-04-01 13:07:17 -04:00
Colin Walters 956b18ace7 tools: Fix unintialized variables 2014-04-01 10:00:18 -04:00
Colin Walters 5bca569506 upgrade: Support --allow-downgrade again
This was unintentially dropped with the OstreeSysrootUpgrader rebase.
2014-03-31 22:27:56 -04:00
Colin Walters 913337c557 sysroot-upgrader: Check merge deployment, not ref for differences
Otherwise this broke atomicity; we could fetch/store the ref, then
crash, and then not upgrade the next time we tried upgrading.

The correct model is: the tree has changed if the new ref is different
from the merge deployment.
2014-03-31 14:41:13 -04:00
Colin Walters 8d13e5bb74 sysroot: Be more conservative with bootlink optimization
Trying to implement "rpm-ostree rollback", in the case where we have 2
deployments with the same bootconfig that we're reordering, we need to
write bootconfig, not just swap the bootlinks.
2014-03-28 21:06:23 -04:00
Colin Walters ad46467f2b tool: End status line in switch/upgrade too 2014-03-27 07:45:13 -04:00
Colin Walters 11ecf9267e libostree/upgrader: Don't pull if there is no remote
In this case we're just reading the local repo.
2014-03-27 07:36:50 -04:00
Colin Walters 9389b7961c libostree/upgrader: Add an API to retrieve an origin description
This will be used by "rpm-ostree upgrade".
2014-03-25 16:46:39 -04:00
Colin Walters 90883674a9 libostree/upgrader: Throw an error if osname is empty
This shouldn't happen; I'm just adding the check to force a rebuild.
2014-03-25 15:47:20 -04:00
Colin Walters 7baa600e23 Add an OstreeSysrootUpgrader API
This moves some utility code from the ostree tool into the shared
library, which will make it easier to consume by external tools.
2014-03-24 18:08:22 -04:00
Colin Walters ffb9d34671 Move basic commit API into ostree_sysroot_simple_write_deployment()
The admin commands had this shared in tool common, but we want to
encourage external programs to do this as well.
2014-03-23 11:36:09 -04:00
Colin Walters a80872afa0 pull: Don't print, just use progress callback
More work on removing g_print() from the library.
2014-03-21 14:52:56 -04:00
Colin Walters 44c386c8f9 pull: End status line on error as well
This way we don't append the error to the current line.
2014-03-19 09:49:55 -04:00
Colin Walters b68f8f95f0 core: Add "admin instutil set-kargs"
This will be used by Anaconda as a convenience command to set the
bootloader arguments.
2014-03-19 09:49:55 -04:00
Colin Walters 4309b49866 libostree: Add ostree_sysroot_deployment_set_kargs()
It turns out people sometimes want to be able to change the kernel
arguments.  Add a convenient API to do so for the current deployment.

This will be used by Anaconda.
2014-03-19 09:49:55 -04:00
Colin Walters 9fed088d6e libostree: Fix ostree_deployment_clone() to also clone bootconfig
This way one can easily mutate it for a new deployment list.
2014-03-19 09:49:55 -04:00
Colin Walters 606918e763 libostree: Add ostree_bootconfig_parser_clone()
This will be necessary to fix ostree_deployment_clone(), but is
potentially useful on its own for other consumers.
2014-03-19 09:49:55 -04:00
Colin Walters 24b1e9c0ac Add "ostree admin instutil", move selinux-ensure-labeled there
There are going to be a few utilities that are only useful for
installers and disk image creation tools.  Let's not expose them all
at the toplevel; instead, hide them under "instutil".
2014-03-19 09:49:55 -04:00
Colin Walters fbc232d72b TODO: update 2014-03-15 19:05:52 -04:00
Colin Walters b6b7d37c9d libostree: Fix crash if output is not a tty
This was a recent regression.
2014-03-14 21:59:36 -04:00
Colin Walters cbc11f42e5 libostree: Add API to append a GPG signature
This will be used by rpm-ostree which needs to use an external program
to sign commits.
2014-03-14 21:57:48 -04:00
Colin Walters 4995352fe0 libostree: Add a better error if we fail to read keyring directory
I had accidentally put it in the -devel package and not noticed.
2014-03-14 14:42:46 -04:00
Colin Walters e11de9357c admin: selinux-ensure-labeled: new builtin
Code like rpm-ostree generates disk images directly.  In order to
ensure SELinux labeling is correct, it currently has a helper program
that runs over the deployment root, then over the whole disk and to
only set a label if none exist.

In order to make it easier to write installers such as Anaconda
without having them depend on rpm-ostree (or whatever other
build-server side program), pull in the helper code here.
2014-03-13 08:21:45 -04:00
Colin Walters 1f7f7382b1 deploy: Less usage of g_print
No need to spam the console here.
2014-03-11 13:16:38 -04:00
Colin Walters bf918f1586 libostree: Remove g_print() from bootloader code
No need to be so chatty.
2014-03-11 13:15:57 -04:00
Colin Walters 5c839f0267 pull: Drop some g_print(), replace others with async progress
We shouldn't g_print() from a library, particularly when the
expectation is that the client has an async progress set up.

This should fix the pull output extending the status line.
2014-03-11 09:36:54 -04:00
Colin Walters c2123bfc71 pull: Ensure temporary data that appears corrupted is deleted
If a MITM attacker (or just network corruption) causes a temporary
downloaded object in tmp/ to be corrupted, we'll end up
continually trying to commit it, and fail.

Fix this unlinking the temp file immediately after opening it.  This
will ensure that if we exit due to an error (or crash), the kernel
will clean up the space for us.

https://bugzilla.gnome.org/show_bug.cgi?id=725924
2014-03-07 19:36:55 -05:00
Colin Walters 79fa7ca692 Add /run/ostree-booted
The idea with this is that things like yum should be able to look for
it and determine whether or not they should assume that they can
change things on the system.

https://bugzilla.gnome.org/show_bug.cgi?id=725380
2014-02-28 03:49:25 -05:00
Colin Walters 4f04a55a5d upgrade/switch: Fix status line being overwritten with pull progress 2014-02-27 16:27:26 -05:00
Colin Walters d8852d099a checkout: Use fd-relative open of newly created directory
We were walking the full path again on our directories, no need to do
that.
2014-02-27 11:49:49 -05:00
Colin Walters b6704f8346 checkout: Only fchown/fchmod directories after we're done populating them
See https://mail.gnome.org/archives/ostree-list/2014-February/msg00020.html
2014-02-27 11:19:33 -05:00
Colin Walters 683a719b7e Update libgsystem, use it to set dirfd-relative xattrs on symlinks
This is a bit more efficient in that we're not walking full paths, and
it helps avoid security/reliability issues if an attacker (or just a
misbehaving process) has the ability to mutate paths in the middle.
2014-02-26 16:34:37 -05:00
Colin Walters 0eb8c37ecb upgrade: Properly set origin_refspec variable for resolve/printing
1) We were ignoring the remote, which is broken
2) We were printing NULL later on
2014-02-26 15:13:12 -05:00
Colin Walters c85682cb85 switch: Don't check whether revision matches
It's quite possible that say "buildmaster" and "smoketested" are the
same revision - but we should allow switching between them.
2014-02-26 15:12:08 -05:00
Colin Walters b762c2f8f1 pull: Remove explicit threading
Mixing async and threads has proved to be too much for my little mind.
It has race conditions that I've tried repeatedly to fix, but failed.

The threading here was scanning metadata objects - and there are
two parts to that:

1) Physically loading them from disk
2) Parsing them

Now #1 has been partially addressed by avoiding a storm of lstat() if
we're starting from a known working state.  If pull gets interrupted,
then we do need to rescan all objects.  Also, we can address this with
local metadata packfiles.

The other potentially slow bit is that we recurse across the metadata,
blocking the main thread.  We could ameliorate that in the future by
scheduling metadata parsing as idle "chunks".

Anyways, let's move the needle back to reliability, and readd speed
more carefully.

https://bugzilla.gnome.org/show_bug.cgi?id=706456
2014-02-21 15:04:23 -05:00
Colin Walters f2e0162846 upgrade: Refuse chronologically older commits unless --allow-downgrade
We don't want to allow MITM attackers to intercept upgrade requests
and provide clients with older OS versions vulnerable to security
flaws.

Only "ostree admin upgrade" gets this behavior for now - whether we
want to do it for "ostree admin switch" is another question.
2014-02-20 18:25:56 -05:00
Colin Walters 9fc9008af8 repo: Fix crash without SELinux policy enabled during commit 2014-02-20 14:19:49 -05:00
Colin Walters 071916196c build: Fix build without SELinux 2014-02-20 10:12:53 -05:00
Colin Walters 3337334be5 libostree: Split off SELinux OstreeSePolicy class
It's better if this is independent from the OstreeSysroot; for
example, a policy is active in a given deployment root at once, not
for a sysroot globally.

We can also collect SELinux-related API in one place.

Unfortunately at the moment there can be only one instance of this
class per process.
2014-02-19 08:43:45 -05:00
Daniel Drake 8ccd603cba fetcher: set timeouts on HTTP connections
We're seeing some hangs while ostree is fetching updates.
I imagine the fact that SoupSessionAsync has no timeout by default
could be the cause of this.

Set timeout values to 60 seconds, which is the default for the new
SoupSession API which we may switch to later.

https://bugzilla.gnome.org/show_bug.cgi?id=724310
2014-02-13 11:21:26 -06:00
Cosimo Cecchi d4f86fdef2 os-init: also create a symlink for /var/lock
After creating one for /var/run. This is needed at least on Debian
systems.
2014-02-12 14:05:13 -08:00
Colin Walters ad08799ef1 deploy: Remove now-unimplemented --no-bootloader argument 2014-02-12 16:47:39 -05:00
Daniel Drake c8b16d9313 boot/ostree-remount.service: run before tmpfiles.d
tmpfiles.d configurations generally require write access to some places
that are read-only until ostree-remount runs.

Make sure ostree-remount has run first.

Thanks to Cosimo Cecchi for finding and diagnosing this problem.

https://bugzilla.gnome.org/show_bug.cgi?id=724183
2014-02-11 17:27:38 -05:00
Colin Walters 3ffdef07a4 repo: Split generic GPG commit verification out into helper
This will be used for a future commit which GPG verifies static
deltas.
2014-02-10 09:40:35 -05:00
Colin Walters 60c4d467aa deltas: Add a timestamp to delta metadata 2014-02-10 09:40:35 -05:00
Colin Walters f526fd4e3c pull: Remove a duplicate hash table
Not sure why we had two...perhaps the code originally had them
separate.
2014-02-10 09:40:35 -05:00
Colin Walters 2ee2751fd1 repo: Don't set GPG engine executable path
The instructions one finds on the internets are apparently wrong, we
really need to keep the default here, since gpgme uses it to actually
find the helper binary it runs.

This fixes the GPG tests for me on EL7 at least.
2014-02-10 09:12:00 -05:00
Colin Walters 8dd7b5575e Drop refs/summary
I'm not aware of anyone using this, and it's not efficient to write a
whole file every time a ref changes, plus it's not atomic.
2014-02-09 14:01:27 -05:00
Colin Walters a30fcba273 sysroot: Add a log with MESSAGE_ID when deployment is complete 2014-02-09 13:17:24 -05:00
Colin Walters d744436a50 pull: Don't crash if the URL is not found 2014-02-06 03:46:14 -05:00
Colin Walters 2d6374822b Initial basic static delta code drop
This has a very basic level of functionality (deltas can be generated,
and applied offline).  There is only some stubbed out pull code to
fetch them via HTTP.

But, better to commit this now and improve it from a known starting
point, rather than have it languish in a branch.
2014-02-04 10:31:44 -05:00
Colin Walters 844c5ea652 core: Import bup's "rollsum" code, add a test case
For static deltas, one strategy that will be employed is to split each
object into chunks, and only include changed chunks in the deltas.
2014-02-04 10:12:56 -05:00
Colin Walters 3007aeb160 build: Fix --without-selinux case 2014-02-02 11:49:43 -05:00
Colin Walters e580a88f4e SELinux: Ensure we label /var, and fix /etc merge wrt xattrs
First, /var needs to be labeled at least once.  We should probably
rearrange things so that /var is only created (and labeled) on the
first deployment, but this patch adds a /var/.ostree-selabeled file
instead.

Second, when doing the /etc merge, we compare the xattrs of the old
/usr/etc versus the current /etc.  The problem with that is that the
policy has different labels for /usr/etc on disk than the real /etc.

The correct fix for this is a bit invasive - we have to take the
physical content of the old /usr/etc, but compare the labels as if
they were really in /etc.

Instead for now, just ignore changes to xattrs.  If the file
content/mode changes, then we take the new file (including any changed
xattrs).

Bottom line: just doing chcon -t blah_t /etc/foo.conf may be lost on
upgrade (for now).
2014-02-02 11:32:52 -05:00
Colin Walters f86a132eb0 libostree: Also use xattr callback for directories
They need labels too, obviously.
2014-02-01 11:30:10 -05:00
Colin Walters 234ae70b4c Add --disable-fsync option to pull-local, and API to repo
This will be used by guestmount - it's WAY faster.  We only take disks
as a unit, so it's safe.  If the process fails halfway through, we
just start over from scratch the next time anyways.
2014-01-31 22:18:13 -05:00
Colin Walters 2313bdcb62 Add SELinux support
The trees as shipped come with /usr/etc, which should just be labeled
as usr_t.  When we do a deployment, we need to relabel the copies of
the files we're making in /etc.

SELinux support is compile and runtime optional.
2014-01-31 08:10:55 -05:00
Colin Walters be1acfafa0 repo: Add API to provide xattrs
This will be used by rpm-ostree to provide SELinux security contexts,
without requiring us to actually label the disk.
2014-01-30 20:37:27 -05:00
Colin Walters c7bcfc1c61 build: Look for /usr/bin/gpgv2 vs /usr/bin/gpgv
For some reason, RHEL has gpgv, but Fedora doesn't.  We need to detect
which to use, since presumably Debian only has gpgv.
2014-01-30 19:32:59 -05:00
Colin Walters f36cc6a44c repo: Improve GPG error messages
The signing test is failing here on EL7 beta for me - it seems like
gnupg isn't honoring the homedir.
2014-01-30 17:17:36 -05:00
Colin Walters ffb19aef9a libostree: Actually trusted.gpg.d/*.gpg for GPG verification
The intent of this code I'm fairly certain was to use *.gpg from the
trusted.gpg.d, directory.  But right now, we're only using
"pubring.gpg" from that directory, which is odd.

Let's fix this to use all keys ending in .gpg, which will also
include pubring.gpg.
2014-01-30 15:40:46 -05:00
Colin Walters bffd30ff27 ostree-prepare-root.service: Also order before plymouth-switch-root.service
In the OSTree model, /sysroot gets set up twice.  We need to ensure
that the /sysroot plymouth sees is only after OSTree has set it up.
2014-01-27 13:57:18 -05:00
Colin Walters daaddcf536 boot/ostree-remount.service: Run before plymouth-read-write.service
The plymouth service needs a writable /var, so ensure that we run
before it does.
2014-01-27 11:57:35 -05:00
Colin Walters d64d2b6636 pull: Be less chatty with G_MESSAGES_DEBUG=all
Only note state *transitions*, don't spam on simple checks.
2014-01-22 15:02:01 +00:00
Colin Walters 390b781d3a pull: Add remotename:ref syntax
This is really the common case.
2014-01-21 08:57:34 +00:00
Colin Walters 43fdfa0649 repo: Document ostree_repo_sign_commit(), and add introspection data
The important bit for introspection is (allow-none) on the homedir.
2014-01-20 12:13:37 -05:00
Colin Walters 9640171711 pull: Close another race
Only send _IDLE messages if and only if we state transition the main
thread (from idle -> !idle or !idle -> idle).  This ensures that we
don't send IDLE, then get it back, and process that when we're !idle.
2014-01-20 06:26:49 -05:00
Colin Walters 3cd866556c pull: Hopefully squash race where we would exit early
This is a redesign (again) of the pull code.  It is simpler and
survives 20 minutes of testing in a loop, whereas the old code would
only go from 30 seconds to 2 minutes.

The problem with the old code was that there was a race where we might
determine idle state even when there are content requests in flight
between the metadata thread and the main one.

This code majorly reworks things - there's now only one IDLE message,
sent in a circle from the main thread, through the metadata scanner,
and back to the main one.

Crucially it's only sent when the *main* thread is idle.  Previously
we were looking at whether the metadata scanner is idle, but that
doesn't make a lot of sense.  First let's make sure the main thread is
idle, then verify that the metadata one is.

This closes the loop because we'll have ensured we get any pending
requests.

https://bugzilla.gnome.org/show_bug.cgi?id=706456
2014-01-19 18:12:44 -05:00
Colin Walters f841313206 libostree: Ensure we set standard::type when querying files
This makes the obvious code to recursively enumerate directories
operate more sanely.

Noticed this while trying to write corrupt-repo-ref.js.
2014-01-19 13:19:10 -05:00
Colin Walters 3ce687ef1b pull: Add a few more g_debug() and assertions
To help track down the race condition better.
2014-01-19 12:39:18 -05:00
Colin Walters d98eb901c4 deploy/main: Unify some bits between admin-switch and admin-upgrade 2014-01-19 11:28:30 -05:00
Colin Walters 878a43411e admin/switch: New builtin to switch between trees
This is something I want to make easier, as it better showcases the
flexibility of OSTree.
2014-01-18 17:47:16 -05:00
Colin Walters 4fff43da1b Squash some harmless compiler warnings
None of these AFAICS actually can occur, but let's silence gcc.
2014-01-18 04:53:16 -05:00
Colin Walters d0d98d3abe libotutil: Fix a possible uninitialized free() in error path
Spotted by gcc.
2014-01-18 04:52:45 -05:00
Colin Walters 5034bf3a9d commit: Reject non-regular/non-symlinks earlier with better error message
Also avoid _NOT_SUPPORTED as that triggers the --help behavior from
the commandline; just use _FAILED.

https://bugzilla.gnome.org/show_bug.cgi?id=722410
2014-01-17 10:39:36 -05:00
Colin Walters b2d0ba7ac1 deploy: Rework kernel arguments, add --karg-append to "admin deploy"
The "ordered hash" code was really just for kernel arguments.  And it
turns out it needs to be a multihash (for e.g. multiple console=
arguments).

So turn the OstreeOrderedHash into OstreeKernelArgs, and move the bits
to split key=value and such into there.

Now we're not making this public API yet - the public OstreeSysroot
just takes char **kargs.  To facilitate code reuse between ostree/ and
libostree/, make it a noinst libtool library.  It'll be duplicated in
the binary and library, but that's OK for now.  We can investigate
making OstreeKernelArgs public later.

https://bugzilla.gnome.org/show_bug.cgi?id=721136
2014-01-16 15:07:55 -05:00
Colin Walters fd8bd356ae Update libgsystem
This has a GsConsole fix.
2014-01-15 09:19:32 -05:00
Colin Walters 108546c08c admin: Add --karg-proc-cmdline argument
When installing a new tree inside an existing OS, this is a convenient
way to include the command line arguments one needs (such as root=).
2014-01-15 09:19:32 -05:00
Colin Walters 54e58a51ca deploy: Write bootloader config even if just kernel arguments changed
The official way to add bootloader arguments to the current deployment
is to redeploy with --karg.  However, doing so tripped up an
optimization made inside the deployment code to just swap the
bootlinks if we're keeping the same "bootcsum".

Change this optimization to look at the pair of (bootcsum, options).
2014-01-15 09:19:32 -05:00
Colin Walters ff2c592ac9 pull: Ensure we begin a status line
Otherwise if the operation completes before anything happens, we hit
an assertion trying to end a status line when we didn't begin one.
2014-01-10 08:15:54 -05:00
Colin Walters 091523a3bd sysroot: Error out on deploy --os=<unknown> which has not been initialized
We should hard require "ostree admin os-init foo" before letting
deployments go there; it's too easy to typo the argument.
2014-01-08 18:30:38 -05:00
Colin Walters aaeeb45fba Remove 'write-refs' builtin
See https://bugzilla.gnome.org/show_bug.cgi?id=705979

This was just a performance hack for gnome-continuous back before it
used libostree via g-i.
2013-12-25 14:24:49 -05:00
Colin Walters 39e3c1bb8a ostree/remote-add: Add --no-gpg-verify option
To more conveniently disable GPG verification.
2013-12-25 14:17:36 -05:00
Colin Walters 9adbc165e7 repo: Introspect _sign_commit() and _verify_commit()
We can't use #ifdef in the headers, since then g-ir-scanner won't pick
up the functions (unless we included config.h).  Let's instead always
have the symbols, but just set an error if we were built without
support for it, just like how pull works.
2013-12-23 11:40:57 -05:00
Colin Walters a58d3d079c prune: Remove redundant 'bytes' in format string 2013-12-20 17:44:13 -05:00
Colin Walters 76d9d92916 repo: Provide a friendlier error message if no signatures are found
I always forget the key to disable it...
2013-12-17 23:00:01 -05:00
Colin Walters 68c9c45e15 repo-pull: Remove leftover unused %s 2013-12-16 18:24:03 -05:00
Roddy Shuler fe5ed36461 core: Set mtime of content objects to 0
This is necessary to satisfy tools such as guile and python, which
compare mtimes to determine whether or not source files need to be
compiled.

https://bugzilla.gnome.org/show_bug.cgi?id=720363
2013-12-15 11:32:34 -05:00
Colin Walters 41df7efc88 Update libgsystem 2013-11-28 23:45:48 -05:00
Colin Walters a24afd68f0 Move xattr handling into libgsystem, fix sysroot to handle directory ownership/perms
This large patch moves the core xattr logic down into libgsystem,
which allows the gs_shutil_cp_a() API to copy them.  In turn, this
allows us to just use that API instead of rolling our own recursive
copy here.

As noted in the new comment though, one case that we are explicitly
regressing is where the new /etc removes a parent directory that's
needed by a modified file.  This seems unlikely for most vendors now,
but let's do that as a separate bug.

https://bugzilla.gnome.org/show_bug.cgi?id=711058
2013-11-28 23:28:13 -05:00
Colin Walters a897b42a7a ostree-remount.service: Run before systemd-random-seed.service
It's essential that we've mounted /var read-write in order for
random-seed.service to work.
2013-11-22 16:42:20 -05:00
Colin Walters 2b8c717c33 fetcher: Display incremental download progress
Previously the progress meter would bump in large chunks after we
completed a download.  Instead, poll in progress files via fstat() for
their size, and add those to the running total.
2013-11-21 14:34:47 -05:00
Colin Walters e9b35deba8 Update libgsystem
For https://bugzilla.gnome.org/show_bug.cgi?id=711057
2013-11-03 21:35:45 -05:00
Colin Walters affccb343a main: Treat default osname more consistently
The libostree already treats passing NULL for osname as "booted
osname, if any".  We should do the same inside the tools.  The upgrade
builtin had this logic duplicated there; we should be able to safely
remove it.

https://bugzilla.gnome.org/show_bug.cgi?id=710970
2013-10-28 09:15:32 -04:00
Daniel Narvaez 7ecfbff26e Fix crash when deploying with implicit os name
When booted into an ostree you can deploy without passing
an --os option. That was crashing though, because
ot_admin_complete_deploy_one is called with NULL
osname but it was not handling it properly.

https://bugzilla.gnome.org/show_bug.cgi?id=710970
2013-10-28 09:15:32 -04:00
Daniel Narvaez 03aa10f17d main/remote: Add a show-url operation to the remote command
Useful to get the remote url in scripts.

https://bugzilla.gnome.org/show_bug.cgi?id=710967
2013-10-28 09:14:23 -04:00
Colin Walters 73ff642b15 libostree: Remove extra : in comment
g-ir-scanner handles this, but it stuck out.
2013-10-24 20:32:20 -04:00
Colin Walters 379db715d7 libostree: Squash a compiler warning 2013-10-24 14:37:50 -04:00
Colin Walters 87922259ee libostree: Squash two g-ir-scanner warnings 2013-10-24 14:37:35 -04:00
Daniel Narvaez ccb10d592d Add support for mkinitcpio
https://bugzilla.gnome.org/show_bug.cgi?id=710682
2013-10-24 14:27:49 -04:00
Colin Walters c65923e642 Add OstreeAsyncProgress, use it for ostree_repo_pull
Several APIs in libostree were moved there from the commandline code,
and have hardcoded g_print() for progress and notifications.  This
isn't useful for people who want to write PackageKit backends, custom
GUIs and the like.

From what I can tell, there isn't really a winning precedent in GLib
for progress notifications.

PackageKit has the model where the source has GObject properties that
change as async ops execute, which isn't bad...but I'd like something
a bit more general where say you can have multiple outstanding async
ops and sensibly track their state.

So, OstreeAsyncProgress is basically a threadsafe property bag with a
change notification signal.

Use this new API to move the GSConsole usage (i.e. g_print()) out from
libostree/ and into ostree/.
2013-10-24 14:27:13 -04:00
Colin Walters 9aaa29598e Update libgsystem
Just to test the new code.
2013-10-23 15:43:29 -04:00
Jeremy Whiting f583c4ab0b core: Add size information to commit metadata
Add a --generate-sizes option to commit to add size information to the
commit metadata.  This will be used by higher level code which wants
to determine the total size necessary for downloading.
2013-10-19 11:56:51 -04:00
Colin Walters 7c903eb889 repo: Remove set-but-unused variable
The gpgme examples use this, but from what I can tell we don't really
need to because we don't need detailed results; we only care whether
we signed it at all.
2013-10-15 16:44:19 -04:00
Colin Walters 4e5b31b8fc trivial: Don't include config.h in headers
This should be included by each .c file.  This fixes using libostree
from a "plain" project without config.h.
2013-10-15 15:26:37 -04:00
Colin Walters b2a98f4ca3 sysroot: Handle deleting directories in /etc
We need to use the full shutil_rm_rf() in order to actually delete
complete directories.

Test suite code based on a patch from Sjoerd Simons <sjored@luon.net>

https://bugzilla.gnome.org/show_bug.cgi?id=710097
2013-10-15 14:56:43 -04:00
Colin Walters 3b700ccb50 core: Add code to read/write "varints"
Adapted from Google protobufs.  For several cases, we want to support
e.g. file sizes up to guint64, but paying the cost of 8 bytes for each
number is too high.

This will be used for static deltas and sizes metadata.
2013-10-11 12:21:37 -04:00
Jasper St. Pierre 3b31f2b4d1 Update libgsystem 2013-10-09 16:17:38 -04:00
Colin Walters c62141004a libostree: Fix an introspection scanner warning 2013-10-09 12:53:12 -04:00
Colin Walters 2708124190 repo: Tweak traversal API
It's convenient for bindings if we have a version that doesn't mutate
the hash table, because they pass temporary hash tables as input.
2013-10-09 12:05:56 -04:00
Colin Walters cb251ae5ca sysroot: Move ostree_sysroot_origin_new_from_refspec here
Rather than having it live in admin.  This is useful for other
consumers like the test suite.
2013-10-03 18:34:24 -04:00
Colin Walters ae2234b183 sysroot: Ensure we create /boot/loader.%d/entries even if there are no deployments
Not doing so breaks things, and we should support this.
2013-10-03 18:34:24 -04:00
Colin Walters 1641ade728 sysroot: Avoid bad double-free
If a deployment is somehow in the list twice, the hash table will free
the *new* value with g_hash_table_insert which gets all broken.  Just
use g_hash_table_replace().
2013-10-03 18:34:24 -04:00
Colin Walters 2b2c6ead22 sysroot: Update some annotations
We were double-freeing before.
2013-10-03 18:34:04 -04:00
Colin Walters 07904c2457 sysroot: Support more arbitrary deployment changes
This commit changes the sysroot API so that one can create arbitrary
new deployment checkouts, then commit them as one step.  This is to
enable things like an automatic bisection tool which say create 50
deployments at once, then when done clean them up.

This also moves some printfs from the library into src/ostree.
2013-10-02 20:18:06 -04:00
Colin Walters f9379b0ce3 repo: Only delete temp files older than a day
This is somewhat lame, but to do better we need a reliable
multiprocess synchronization mechanism.

https://bugzilla.gnome.org/show_bug.cgi?id=709115
2013-10-02 18:17:27 -04:00
Colin Walters 7b119370a2 repo: Enumerate objects using openat() too, make more efficient
This drops several calls to malloc()/g_object_new() per object
enumeration.  Just a followup to using openat() and friends in other
places.
2013-09-30 08:51:25 -04:00
Colin Walters 8592922c2c core: Make ostree_get_relative_object_path() private
I plan to rename all of these APIs to use the term 'loose', so that it
makes more sense after pack files are introduced.  External users
should not use them; instead use _load_variant() or _read_commit().
2013-09-29 20:17:18 -04:00
Jeremy Whiting 7f9eefb62d pull: Verify commits with gpg signatures from detached metadata
This uses gpgv for verification against DATADIR/ostree/pubring.gpg by
default.  The keyring can be overridden by specifying OSTREE_GPG_HOME.

Add a unit test for commit signing with gpg key and verifying on pull;
to implement this we ship a test GPG key generated with no password
for Ostree Tester <test@test.com>.

Change all of the existing tests to disable GPG verification.
2013-09-29 14:49:47 -04:00
Colin Walters b064581577 commit: Store detached metadata in normal form
Otherwise corrupted variants may crash us.
2013-09-29 14:00:53 -04:00
Jeremy Whiting 7d5aa74dae core: Use libgpgme to add GPG signatures to detached metadata for commit object
Add an optional dependency on gpgme to add GPG signatures into the
detached metadata, with the key "ostree.gpgsigs", as an "aay", an
array of signatures (treated as binary data).

The commit command gains a --gpg-sign=<key-id> argument.  Also add an
argument --gpg-homedir to set the GPG homedir where we look for
keyrings.
2013-09-28 16:12:35 -04:00
Colin Walters 0f486105db remote-add: Add --set=KEY=VALUE option
This can be used to add a remote and set e.g. tls-permissive=true, or
gpgverify=false.
2013-09-28 12:00:16 -04:00
Colin Walters 38a5f6e5ed show: Handle keys of any type, and set an error if key doesn't exist
Previously we were just handling strings, and silently doing nothing
if the key didn't exist, which is pretty broken.
2013-09-27 12:57:01 -04:00
Colin Walters 2ef652439d repo-pull: Allocate with g_new0 rather than g_new.
Conflicts:

	src/libostree/ostree-repo-pull.c
2013-09-26 14:38:37 -04:00
Colin Walters 7d72a4b941 pull: Also fetch detached commitmeta files
These will contain GPG signatures and the like in the future, so we
should fetch them now.
2013-09-26 14:36:06 -04:00
Colin Walters 10cf4ccabc repo: When adding detached metadata, ensure the objdir exists
We may be creating the commitmeta file before the commit, so we need
to call mkdir(<first two checksum characters>) here too.
2013-09-26 13:59:08 -04:00
Colin Walters edcfcd21a3 show: Print an error message if commit doesn't have detached metadata
...instead of segfaulting.
2013-09-26 13:57:59 -04:00
Colin Walters 7959ad9eaf fetcher: Fix previous commit
I was getting hangs in the test suite, and looking at the previous
commit, we were calling the async completion functions out of the
finalizer for the URI, which is weird.  I didn't analyze what's going
wrong, but what we really should be doing is processing our internal
queue after we've downloaded a file, and the request is about to be
finalized.

I suspect doing queue management from the finalizer created a circular
reference type situation.

This patch deduplicates the queue processing bits too.

https://bugzilla.gnome.org/show_bug.cgi?id=708126
2013-09-25 11:59:43 -04:00
Sjoerd Simons 5f310868f7 fetcher: Add a queue to limit requests sent to soup
On a large ostree repository pulling over http slows to a crawl. Pulling
from localhost results in:
 5944 metadata, 63734 content objects fetched; 850509 KiB transferred in
 1106 seconds
In other words about 800KiB/s. Some profiling shows that essentially
all of the CPU goes into libsoup doing its request bookkeeping instead
of into the actual downloading.

Adding a simple queue to limit to number of active request sent into
libsoup makes for a dramatic improvement:
 5944 metadata, 63734 content objects fetched; 850509 KiB transferred
 in 89 seconds
So around 9450 KiB/s.

https://bugzilla.gnome.org/show_bug.cgi?id=708126
2013-09-22 23:40:51 +02:00
Colin Walters 5c406132ce libostree: Document and annotate some methods to quash g-ir-scanner warnings
Since warnings are bad.  Introspection is good.
2013-09-20 14:00:31 -04:00
Colin Walters b4bf5af5fa libostree: Remove private header file from ostree.h
Was breaking pkgsys-ostree.
2013-09-20 13:54:43 -04:00
Colin Walters 0f65d73581 upgrade: Don't segfault if there's no previous deployment
Obviously.
2013-09-20 11:21:08 -04:00
Colin Walters 298625d7f8 deploy: Correctly swap bootloader version with new boot checksums
If we had two deployments with different boot checksums, and were
trying to remove the one that was the same and add a new one (the
normal case), we'd end up assuming due to comparison with 0 that
we only needed to do the fast subbootversion swap.

Fix this by actually putting 1 where we really mean 1.

And update the tests to verify the fix; I have double-verified by
undoing the fix, and noting that the test fails.

https://bugzilla.gnome.org/show_bug.cgi?id=708351
2013-09-20 11:21:08 -04:00
Colin Walters 08b8734576 deploy: Remove checksum from generated loader entries
The actual deployment checksum shouldn't be in there, because we may
just swap bootlinks, rendering the name of the old bootloader entry
file invalid.  Thankfully nothing actually parsed the names of these
files, so let's just use the index.
2013-09-19 22:41:33 -04:00
Colin Walters d03b8bbad9 deploy: Do a full system sync after writing bootloader config
This is just something I noticed on inspection; we should catch any
changes to /boot in the sync(), even though theoretically gio should
have done fdatasync().
2013-09-19 22:26:43 -04:00
Colin Walters ae68b8380f deploy: Only query bootloader if we have to make a loader change
This will also help distinguish in the output when we're just swapping
bootlinks versus when we need to change the loader config.
2013-09-19 22:25:12 -04:00
Colin Walters d1bc9e2acf deploy: Some minor assertion and error message improvements
To help debug https://bugzilla.gnome.org/show_bug.cgi?id=708351

Plus run a status consistently in the tests.
2013-09-19 22:23:19 -04:00
Colin Walters 9b31b526cc libostree: Add API to write metadata as stream
And use it in pull-local.  The goal here is to kill users of
ostree_repo_load_variant(), and move a bit more towards hiding the
variants.
2013-09-18 18:44:57 -04:00
Colin Walters 4c1a69f74e main: Print error: in red bold just for fun
It's easier to see.  Also drop old dead ot_main() function.
2013-09-18 16:34:14 -04:00
Colin Walters 0818a462c2 sysroot: Look for booted deployment correctly
We need to look at the list we just gathered, not potentially NULL.
2013-09-17 10:53:41 -04:00
Colin Walters 8f1ea1b50a sysroot: Clean up API
Now that we have a real GObject for the sysroot, we have a convenient
place to keep track of 4 pieces of state:

* The current deployment list
* The current bootversion
* The current subbootversion
* The current booted deployment (if any)

Avoid requiring callers to pass all of this around and load it
piecemeal; instead the new thing is ostree_sysroot_load().
2013-09-16 18:32:13 -04:00
Colin Walters c07c84cb6f libostree: Use ostree_repo_create() rather than spawning subprocess
We have API now, yay!
2013-09-16 08:52:16 -04:00
Colin Walters b88c4b5c1c doc: Add some more OstreeSysroot docs 2013-09-15 22:44:25 -04:00
Colin Walters ac0154713d libostree: Make OstreeBootloader private
It was only temporarily public while functionality was being merged
down; that's done now.
2013-09-15 20:26:13 -04:00
Colin Walters 4abaf27e57 admin: Drop two uses of ostree_sysroot_get_path()
Moving towards it being a full library.
2013-09-15 20:21:49 -04:00
Colin Walters c6292942ff libostree: Nearly complete move of API into OstreeSysroot
Move the deployment code too.
2013-09-15 20:16:20 -04:00
Colin Walters 95f07d486a libostree: Move a lot more sysroot API here
OstreeBootloader is temporarily public API.
2013-09-15 18:08:06 -04:00
Colin Walters 6f929ca5af libostree: Move sysroot initialization API here 2013-09-15 15:16:56 -04:00
Colin Walters 35bab87691 Move Deployment and BootconfigParser into libostree
As part of moving admin functionality there.  While we are doing this,
rename OtConfigParser to OstreeBootConfig parser since it's a better
name.
2013-09-15 15:06:31 -04:00
Colin Walters af0f888057 libostree: Add new OstreeSysroot class
At the moment, just a container for a path, but we will start moving
admin functionality here.
2013-09-15 14:33:57 -04:00
Colin Walters d302c3d7ff checkout: Fix leak when doing uncompressed cache + copying checkouts
We were reading into "input" twice, leading a fd.  This was
breaking the gnome-continuous integrationtest.
2013-09-13 17:15:30 -04:00
Colin Walters 94e42c521c pull: Drop usage of 2.36 symbol
Really need deprecation warnings while using RHEL6.4...maybe Software
Collections will save me.
2013-09-13 16:45:43 -04:00
Colin Walters a9e82e71fa Update libgsystem
With tempfile fix.
2013-09-13 16:43:27 -04:00
Colin Walters 2e6b64d21b doc: Add OstreeMutableTree, other minor doc updates 2013-09-12 23:53:13 -04:00
Colin Walters ce0fbbfdee admin: Silence a gcc warning
It can't happen, but anyways.
2013-09-12 14:34:38 -04:00
Colin Walters 081da0033a COPYING: Now fully LGPLv2+
I ran into Jeremy Katz today, and he gave me permission to relicense
the small bits of switch-root.c to LGPLv2+.  This combined with
permission from Peter Jones allows OSTree to become fully LGPLv2+.

Not a big deal, it's just a lot clearer to only have one license, and
it makes it easier to turn application code into library code.
2013-09-11 19:57:05 -04:00
Colin Walters 6a01ec1032 checkout: Fix previous commit
Don't try to close in cases where we don't have input.
2013-09-11 11:54:53 -04:00
Colin Walters 0238a2c26c checkout: Fix file descriptor leak for copying checkouts
Hardlink checkouts didn't hit this, but we need to close the input
stream.
2013-09-11 10:58:22 -04:00
Colin Walters c8744beb17 Update libgsystem
And while we're here, also update the TODO slightly.
2013-09-11 08:28:09 -04:00
Colin Walters 27b46cb45c libostree: Fix compiler warning 2013-09-10 12:25:13 -04:00
Colin Walters 4f236ebc1b checkout: Add g_prefix_error() around more failures
So we can debug what's going wrong more easily.
2013-09-10 12:21:04 -04:00
Jasper St. Pierre fb1e140dba repo: Make the optimization for reusing checksums clearer
The code here is a bit hard to understand, so make it clearer by cleaning
up the flow control and adding some comments.

https://bugzilla.gnome.org/show_bug.cgi?id=707727
2013-09-09 23:00:41 -04:00
Jasper St. Pierre 223bccf2f6 repo: Make read_commit spit out a resolved commit ref as well
read_commit resolves the ref to a commit, and a lot of consumers want
the resolved commit for their own purposes; this prevents them from
calling resolve_rev themselves.

https://bugzilla.gnome.org/show_bug.cgi?id=707727
2013-09-09 22:58:39 -04:00
Jasper St. Pierre 6b4ef97c18 repo-libarchive: Apply commit modifiers to libarchive archives as well
And document the libarchive methods as well, so we can pass a NULL
commit modifier.

https://bugzilla.gnome.org/show_bug.cgi?id=707727
2013-09-09 22:51:43 -04:00
Jasper St. Pierre c7f9fc9c63 repo: Change the pairs of checksums to instead be based on OstreeRepoFiles
We want an OstreeRepoFile to be the way to reference a "filesystem
tree" that's stored in the repo, which is a combination of a DIR_TREE
and a DIR_META. The idea is that once you write an mtree to the repo
using ostree_repo_write_mtree, it becomes serialized and you get an
OstreeRepoFile in return.

Change any APIs that care about DIR_TREE / DIR_META checksums to care
about OstreeRepoFiles instead, which right now is mostly is
ostree_repo_write_commit.

https://bugzilla.gnome.org/show_bug.cgi?id=707727
2013-09-09 22:51:33 -04:00
Jasper St. Pierre 33db74d0ce repo-file: Base OstreeRepoFile on trees instead of commits
We want an OstreeRepoFile to be the way to represent a filesystem tree
inside an ostree repository. In order to do this, we need to drop the
commit from an OstreeRepoFile, and make that go to callers.

Switch all current users of ostree_repo_file_new_root to
ostree_repo_read_commit, and make the actual constructor private.

https://bugzilla.gnome.org/show_bug.cgi?id=707727
2013-09-09 22:31:59 -04:00
Colin Walters ac2d61dd51 core: Add detached metadata, readd metadata to commits
Previously I thought we'd have to ditch the current commit
format to avoid a{sv} due to

See https://bugzilla.gnome.org/show_bug.cgi?id=673012

But I realized that we don't really have to care about
unpacking/repacking commit objects, so let's just re-expose the
existing metadata a{sv} in commits in the API.

Also, add support for "detached" metadata that can be updated at any
time post-commit.  This is specifically designed for GPG signatures.

https://bugzilla.gnome.org/show_bug.cgi?id=707379
2013-09-09 17:01:32 -04:00
Colin Walters b4b700c163 Update libgsystem
Now with new shiny implementation of gs_shutil_rm_rf().
2013-09-09 16:06:51 -04:00
Vivek Dasmohapatra f802822034 pull: ref the thread default main context during init, unref in cleanup
...get_thread_default returns NULL when the thread default is also the global
default, so this only shows up when running in a thread (eg g_task_run_in_thread)
2013-09-09 12:06:01 -04:00
Colin Walters 3ff4b1874c core: Use openat() for reading bare file objects too
...unless we want xattrs, in which case we have to fallback to path
lookup due to lack of llistxattrat().

This looks nicer in strace.
2013-09-08 19:31:44 -04:00
Colin Walters a071a3ec3a core: Delete unused OstreeRepoFile API
Was just unused, and it was using further API i want to remove.
2013-09-08 16:49:21 -04:00
Colin Walters d863ef5838 repo: Delete leftover repo/pending dir
This is a relic from long ago when we were trying to stage objects
before finally committing them all in one go in the pull code.

We're no longer doing that, so stop trying to make the directory.
This also fixes trying to use ostree as non-root to read the
root-owned repo, since we'd fail to create the pending dir.
2013-09-08 15:04:35 -04:00
Colin Walters 33e589c018 core: Use at-relative functions for checking out tree copies too
For the cases where we can't hardlink, use at-relative walking of the
path where possible.  We still don't have lsetxattrat, so we also need
to deal with pathnames, but that is now only for symlinks.

Again, the advantages of this are a lot less malloc() of pathnames in
ostree, and much less time spent traversing paths inside the kernel.

https://bugzilla.gnome.org/show_bug.cgi?id=707733
2013-09-08 14:42:50 -04:00
Colin Walters 0c2ea54e68 core: Make ostree_set_xattrs() private
Nothing external uses it.  We keep ostree_get_xattrs_for_file() public
because it's convenient for external consumers to get xattrs in
exactly the format we desire.

https://bugzilla.gnome.org/show_bug.cgi?id=707733
2013-09-08 14:41:51 -04:00
Colin Walters 27c3e7884e core: Make write_object() a bit more efficient
Do as many operations as we can using the original file descriptor
while we have it open, rather than writing, closing, then reopening.

This necessitated very explicitly special casing symbolic links,
mainly due to the lack of lsetxattrat().

https://bugzilla.gnome.org/show_bug.cgi?id=707733
2013-09-08 14:40:52 -04:00
Colin Walters 9846fb27fd core: Use linkat() for hardlink checkouts too
Clean up how we deal with the uncompressed object cache; we now use
openat()/linkat() and such just like we do for the main objects/.

Use linkat() between the objects and the destination, if possible.

https://bugzilla.gnome.org/show_bug.cgi?id=707733
2013-09-08 14:40:09 -04:00
Jasper St. Pierre db148cc9ba repo: Make the body parameter to ostree_repo_commit optional
https://bugzilla.gnome.org/show_bug.cgi?id=707727
2013-09-08 11:50:51 -04:00
Jasper St. Pierre 1f8db2a873 repo: Drop the branch parameter from ostree_repo_commit
It's unused. Make users explicitly write a ref if they want this;
high-level convenience API will be introduced later.

https://bugzilla.gnome.org/show_bug.cgi?id=707727
2013-09-08 11:50:51 -04:00
Jasper St. Pierre f49ed9e74d builtin-commit: Don't parse the parent's GVariant by hand
Instead, use OstreeRepoFile as a handle for the parent commit.
We need to add an accessor for the metadata checksum, as that
hasn't been exposed before.

https://bugzilla.gnome.org/show_bug.cgi?id=707727
2013-09-08 11:50:51 -04:00
Jasper St. Pierre 8ac0f99ed6 repo-file: s/content_checksum/contents_checksum/
This is what we call it everywhere else, so just be consistent.
It also lines up with metadata_checksum better.

https://bugzilla.gnome.org/show_bug.cgi?id=707727
2013-09-08 11:50:51 -04:00
Colin Walters 75225166da repo: Fix file descriptor leak from previous commit
I keep forgetting mmap doesn't take ownership of the fd.
2013-09-08 10:41:18 -04:00
Colin Walters 5a685ff989 core: Use at-relative lookups for archive-z2 content
We can't quite do it for bare repositories yet because we need to have
a way to go from struct stat -> GFileInfo, and that's buried in gio's
private GLocalFile class.
2013-09-07 10:10:21 -04:00
Colin Walters d2c6e19278 core: Use at-relative lookups for metadata reading
Just use openat() for locating variants, rather than doing the lstat()
+ open().  This also drops several malloc+object allocations from the
lookup path.
2013-09-07 05:03:48 -04:00
Colin Walters b97249d4f9 core: Use at-relative lookup for locating objects too
Add new internal API to both fstatat() and write a pathname for the
given object.  Use it in commit, and also wrapped in the old
GFile-based API.

This is more efficient.
2013-09-07 04:47:40 -04:00
Colin Walters a7c0992d94 core: Add malloc-free API for objects, use *at functions for storing
This is more efficient; we avoid malloc of a number of pathname +
GFile objects, plus the kernel doesn't have to traverse the repo path
again.
2013-09-07 04:18:41 -04:00
Jasper St. Pierre 2506e8fb26 repo: Make commit_transaction introspectable 2013-09-07 01:46:50 -04:00
Jasper St. Pierre 3dcff0b2c4 Fix build
Before I pushed the "move code to a different file" commit, I rebased,
but forgot to test the build. Sorry.
2013-09-06 20:47:13 -04:00
Jasper St. Pierre 91d3b90c9f Move ref writing to be transaction-based
Rather than having separate write_ref calls, make clients start a
transaction, add some refs, and then commit it. While this doesn't
make it 100% atomic, it makes it easier for us to use an atomic
model, and it means we don't do as much I/O updating the summary
file and such.

https://bugzilla.gnome.org/show_bug.cgi?id=707644
2013-09-06 20:31:12 -04:00
Jasper St. Pierre bd2948e964 repo: Move the scanning hardlinks optimization out of prepare_transaction
This is just a terrible API to have. Make the scanning a separate method,
and document it as an optimization.
2013-09-06 20:31:12 -04:00
Jasper St. Pierre f84504a8c4 repo: Make abort_transaction silently succeed if we're not in a transaction
This helps callers out a lot, and means we can always call abort_transaction
at the end of a function.

https://bugzilla.gnome.org/show_bug.cgi?id=707644
2013-09-06 20:31:12 -04:00
Jasper St. Pierre ef61724269 repo: Move the transaction stats to a separate struct
This is much easier for callers to handle, and simplifies
the API a lot.

https://bugzilla.gnome.org/show_bug.cgi?id=707644
2013-09-06 20:31:12 -04:00
Jasper St. Pierre 5082e1d8e9 repo: Rename "stage" to "write" in the API
An earlier version of this API acted like git in that some objects
would be staged in a temporary directory which would be then committed
in one go by moving files around. The API doesn't match most users
expectations though, as while the stage is nice as a high-level API
it isn't really suited for low-level APIs.

While the stage was removed, the APIs were never renamed. Rename
them now so that they match expectations.

https://bugzilla.gnome.org/show_bug.cgi?id=707644
2013-09-06 20:31:12 -04:00
Jasper St. Pierre c817217ad8 repo: Move commit code to another file
ostree-repo.c is a bit too big, and most of the commit code is
fairly standalone.

https://bugzilla.gnome.org/show_bug.cgi?id=707644
2013-09-06 20:31:11 -04:00
Jasper St. Pierre 89f4817bdf repo: Make prepare_transaction introspectable
We need to document the out pointer as an out pointer.

https://bugzilla.gnome.org/show_bug.cgi?id=707644
2013-09-06 20:28:42 -04:00
Jasper St. Pierre b36e61f813 repo: Make OSTreeCommitModifier introspectable
Callbacks need GDestroyNotifies to be usable from bindings.

https://bugzilla.gnome.org/show_bug.cgi?id=707644
2013-09-06 20:28:42 -04:00
Jasper St. Pierre f5f7fe5e9a repo: Make the ordering consistent between abort/complete_transaction
This just makes the functions nicer to stare at.

https://bugzilla.gnome.org/show_bug.cgi?id=707644
2013-09-06 20:28:42 -04:00
Colin Walters 4dcf1a4282 core: When using hardlinks, always use linkat() for destination
This avoids repeatedly traversing the target pathname, and is just
more efficient.

Part of a prelude to using fd-relative API for the source object path
too.
2013-09-06 18:45:41 -04:00
Colin Walters 2b61caa2fe core: Use fd-relative creation for tmp/
Update libgsystem submodule for a bugfix.

This is both more efficient from a kernel perspective, and avoids us
calling gs_file_get_path_cached() on tmp_dir constantly, which
triggered another bug due to lack of locking.
2013-09-06 18:45:16 -04:00
Jasper St. Pierre 02adfc8c03 repo: Rename ostree_repo_check to ostree_repo_open
As it more clearly describes what the function does: load the
repo from disk and initialize it.

At the same time, add a cancellable parameter.

https://bugzilla.gnome.org/show_bug.cgi?id=707582
2013-09-05 14:07:03 -04:00
Jasper St. Pierre 57d285f619 repo: Move the 'init' builtin command to a public API, ostree_repo_create
This continues the goal of making more of ostree accessible by API,
rather than forking out to subprocesses.

https://bugzilla.gnome.org/show_bug.cgi?id=707582
2013-09-05 14:07:03 -04:00
Jasper St. Pierre eb6f017ef8 repo: Use constructed, not constructor
It's simpler to do this after the object is constructed, rather
than in the middle of construction.

https://bugzilla.gnome.org/show_bug.cgi?id=707582
2013-09-05 13:54:36 -04:00
Colin Walters 8a59f6561c main: Call setlocale()
While we're not translating anything right now, we should still
setlocale() so that our output is formatted as command line users will
expect.
2013-09-04 16:45:51 -04:00
Colin Walters 1ae83002b5 core: Delete unused prototype 2013-09-04 08:22:13 -04:00
Colin Walters eeb1f9e07b core: Delete unused ostree_create_temp_dir() API
Nothing was using it.  Poof.  Gone!
2013-09-04 08:18:45 -04:00
Colin Walters 36815f52b5 libostree: Move file creation APIs out of core, into checkout.c
Since this was the only user, let's not have generic code to go from
OSTree representation -> filesystem here.  It should live in checkout.
2013-09-04 08:17:42 -04:00
Colin Walters 79c922a00b core: Remove unused prototypes 2013-09-04 07:54:32 -04:00
Colin Walters ee2b66ef00 core: Make a helper function private
Nothing was using this, so make it private.
2013-09-04 07:53:02 -04:00
Colin Walters 2534714501 core: Make ostree_write_variant_with_size() private
More work making the file formats and utilities private.
2013-09-04 07:49:41 -04:00
Colin Walters 0d72168961 libostree: Make the format of file objects private
It's now isolated almost entirely to ostree-core.c, except
ostree-repo.c needs to know how to create archive-z2 file headers.  So
give it a private API for that.
2013-09-03 18:43:42 -04:00
Colin Walters 356de33b49 repo: Drop ostree_repo_load_variant_c API
Originally we had this to avoid forcing callers to malloc() if they
had a csum, but nowadays we have in-place conversion APIs that are
fast enough.
2013-09-03 18:23:11 -04:00
Colin Walters dd7d2f7b43 repo: Only apply setuid/xattrs after checksum validation
See the new comment in the source; basically if we're fetching content
over http, then someone with the capability to MITM the network could
create a transient setuid binary on disk with arbitrary content.  If
they also had a process running on the system (such as an application)
it could be escalated to root.

https://bugzilla.gnome.org/show_bug.cgi?id=707139
2013-09-02 15:31:55 -04:00
Colin Walters 597da6ca6b libostree: Change synchronous fetching API to return a stream
There's not a good reason to write small things such as repo/config to
the filesystem, only to read them back in again.  Change the
non-partial API to just return a stream, then read it into a memory
buffer.

https://bugzilla.gnome.org/show_bug.cgi?id=707157
2013-09-02 14:48:21 -04:00
Colin Walters 95a78542e9 fetcher: Only open files when we are ready to write to them
Otherwise we quickly run out of file descriptors when doing large
requests.

https://bugzilla.gnome.org/show_bug.cgi?id=707157
2013-09-02 14:48:21 -04:00
Colin Walters 8b5f684b68 fetcher: Clean up code to avoid intermediate files
I think originally we had the .part/.done separation because we were
trying to support partial downloads of files like repo/config and
repo/refs.

But now that the http server configuration won't give us partial
results, we don't need to support caching those files between runs.

And thus, there's no reason to have the .part/.done and do the dance
with renaming them.

When fetching objects/ and other things that use _with_async, we
continue to use _append_to(), and if the returned range tells us we
have all the bytes, then we hand the full file over to the caller.

Don't attempt to shortcut in the case where the last run told us we
already have the object; the object fetcher code will not make a
request.

While we're here, also clean up use of GError and consistently use the
cancellable from the pending.

https://bugzilla.gnome.org/show_bug.cgi?id=707157
2013-09-02 14:48:20 -04:00
Colin Walters 9e497a4ce7 prepare-root: Fix ostree= kernel argument at end
Extracting the code for parse_ostree_cmdline() and running it on some
test input (on RHEL6.4 glibc), I can reproduce the odd behavior from
getline() where it apparently returns the size of the default malloc
buffer in the size output, and some non-zero value.

This behavior would be OK except that it breaks the logic for
stripping off the trailing newline, which in turn breaks booting
because we return "ostree=foo\n".

This has worked so far in gnome-ostree because syslinux apparently
injects initrd=/path/to/initrd as a final kernel argment.

Anyways, we don't handle NUL characters here in /proc/cmdline, so
let's just call strlen () to be safe.

https://bugzilla.gnome.org/show_bug.cgi?id=707192
2013-09-02 13:27:41 -04:00
Colin Walters 72a8f41ad4 admin: Error out if we see malformed or mismatched checksums for /boot
The kernel/initramfs both need to end in -SHA256 and match.
2013-08-30 18:10:35 -04:00