I run builds on my laptop, but it also crashes about 1/4 of the time
while suspending. It's definitely undesrirable to get e.g. empty
.dirtree objects because they corrupt builds. Concretely, I was
getting empty contents committed for xorg-util-macros.
Now, we used to write out temporary files using g_file_replace() which
does a fsync() during close, but then switched to a more "manual"
g_file_append_to().
We could switch back to g_file_replace(), but the problem is, we don't
want to call fsync() on temporary files in the case where we already
have the object. Attempting to add an object we already have is a
*very* common case.
This is both the old and new code sequence for the case where an
object is already stored:
open(temp, O_WRONLY)
write() write() write()
close()
lstat(objects/3a/9fe332...) = 0
unlink(temp)
In the *new* code, here's the case where an object *isn't* stored:
open(temp, O_WRONLY)
write() write() write()
close()
lstat(objects/3a/9fe332...) = -1
open(temp, O_RDONLY)
fdatasync()
close()
rename(temp, objects/3a/9fe332)
Compare with the *old* code path for when an object isn't stored:
open(temp, O_WRONLY)
write() write() write()
close()
lstat(objects/3a/9fe332...) = -1
link(temp, objects/3a/9fe332)
unlink(temp)
The problem with this is we really need to fdatasync(). Also doing
just rename() instead of the weird link()/unlink() helps us express to
the filesystem that we want atomic semantics. For example, BTRFS has
special handling for rename().
We really don't have a sane story for private files. This is a
defensive step ensuring that with old versions of gnome-ostree,
components that mistakenly have un-world-readable files don't break
pulls.
Use the new update-cache mode of pango-querymodules, which
automatically finds the correct cache file location.
Updated to look in both places by Colin Walters <walters@verbum.org>
https://bugzilla.gnome.org/show_bug.cgi?id=682411
This command regenerates kernel-dependent files. It is meant to
be run after kernel upgrades in the host distribution, to keep
the ostree installation running, and attempts to figure out the
right version automatically (and without changing the ostree checkout)
As the manual page doesn't say, but the in-code kernel documentation
shows, hardlinking for normal users can fail for a variety of
reasons (including very common situations such as non regular file
or non writable file), if the owner of the file does not match
the user linking (e.g. when checking out a shadow repo with a root-
owned master).
If that happens, fail back silently to copying instead of aborting
the whole operation.
https://bugzilla.gnome.org/show_bug.cgi?id=682298
Common code was duplicating the command name and not shifting
arguments properly, which thus required the builtins to be aware
of it, instead of being treated like subcommands.
The qemu helper really wants to copy kernel modules, but not
update the system bootloader. Allow it to reuse ostadmin for
this.
Note that our previous path of shelling out to "cp -al" broke because
it refused to make cross-device links.
# Please enter the commit message for your changes. Lines starting
# with '#' will be ignored, and an empty message aborts the commit.
# On branch master
# Your branch is ahead of 'origin/master' by 1 commit.
#
# Changes to be committed:
# (use "git reset HEAD <file>..." to unstage)
#
# modified: src/libotutil/ot-gio-utils.c
# modified: src/libotutil/ot-gio-utils.h
# modified: src/ostadmin/ot-admin-builtin-deploy.c
#
# Untracked files:
# (use "git add <file>..." to include in what will be committed)
#
# embedded-dependencies/glib/
# embedded-dependencies/libsoup/
The old command line syntax assumed you wanted the latest from
a given revision, but to bisect, you really want the ability
to deploy any given one.
Old:
$ ostadmin deploy gnomeos-3.6-i686-{runtime,devel}
New:
$ ostadmin deploy trees/gnomeos-3.6-i686-runtime
$ ostadmin deploy trees/gnomeos-3.6-i686-devel
And note that the deploy directory now changes to:
/ostree/trees/gnomeos-3.6-i686-devel
And now it's possible to:
$ ostadmin deploy trees/gnomeos-3.6-i686-devel trees/gnomeos-3.6-i686-devel^^^
This helps make things less confusing when we try to create a file not
owned by our uid, such as running ostree-pull as non-root on a
repository owned by root.
The "status text" code was assuming that libsoup was only doing one
thing at a time, but that's never been the case. Change the code to
display the status of all active requests.
This can be a large performance win in certain circumstances:
* Cold buffer cache (we don't block the whole process)
* Requiring a copy instead of hardlink
The previous fix to just ignore symbolic links for hard linking isn't
really good enough, since it can happen for empty files too.
Since this is an optimization, when we get EMLINK, let's instead just
fall back to copying. This also applies to EXDEV.
Rather than always doing:
1) make temporary link
2) unlink() target
3) rename()
Just try making the link, and only do the second two if the file
already exists. This reduces system call traffic a lot.
We should explicitly distinguish between the case where we have a git
branch we're following, versus an immutable tag. In the latter case,
we can entirely avoid running 'git fetch' for it once we have it.
This is a noticeable speedup in our current scenario of pinning WebKit
to a tag.
The git mirroring code now has a --fetch-skip-secs=X option which
allows us to basically run it in a loop, without hitting remote git
repositories too often.
This is the new workflow:
$ ostbuild import-tree wip-gnome-menus-3.6
$ $EDITOR ~/build/ostbuild/snapshots/wip-gnome-menus-3.6-2012.1-abcd.json
$ ostbuild build
First, "resolve" now just picks git commits. We don't expand
config-opts and patches, nor do we generate tree contents. This makes
the generated files *much* more human editable.
Next, fold "build-components" and "compose" into just "build". One
never really wants to just build components. This lets us eliminate
binary snapshots as a concept; instead we always have a combination of
source snapshot and component/ refs.
First, for binary snapshots we need to include the exact revision of
the architecture buildroot. To do that, introduce
"architecture-buildroots2".
Second, for bin-to-src, we'd like to allow "partial" builds. So while
we do expand everything to source, *also* include the binary
ostree-revision. This will allow building just one component, while
reusing the previously built binaries.
By default, don't delete loose objects which have hard links. This
has the natural semantics that if you delete all the checkouts, you
probably want it packed.
Conversely, if it has a hard link, we do want further checkouts to
share storage, even if we pack in between them.
In the case of not being able to check the repository or not being able
to parse command line arguments, attributes from "pull_data" would be used
with garbage in them. This patch makes initialization of the sctructure to
happen earlier so it is safe to use in those cases.
Call self.parse_config() so that all necessary directories are parsed.
Also don't call nonexistant self.parse_active_branch() and just use
self.active_branch since it's already have been initialized.
For components specified on the command line, we now force a rebuild.
There's also a new option to skip if the git revision matches, but
metadata changed (useful for now to avoid cascading builds if you know
you don't need it).
This builtin did not work for me. It did not initialize self.repo,
it used the nonexisting parse_active_branch function unconditionally,
and it did not actually find the list of components in contents.json.
This commit fixes all three of these issues.
This fixes the case where /tmp resides in one volume, and the build
directory in a different one: by storing the temporary file in the
same directory as the target one, we avoid the case in which os.rename()
would cross file system boundaries.
Reviewed-by: Colin Walters <walters@verbum.org>
By default, when doing a commit, scan all of our loose objects and
build up a (device,inode) -> checksum hash. Then when we're doing a
commit, if we see a file with that (device,inode) pair, we can avoid
checksumming it.
This will allow us to use hard links again for user-mode checkouts,
rather than the hackish link cache. It was pretty silly anyways to
have file objects be stored with just a small metadata header
prepended, but uncompressed.
Either they should be hardlinkable, or compressed (in pack files).
Rather than passing xattr/file_info for all objects, change the API to
assume we're passing the defined object stream for each type. Namely,
for OSTREE_OBJECT_TYPE_FILE, we're now giving the "archive file" data.
This significantly cleans up the code for committing to archive mode
repositories, at the cost of having to (at present) create an
intermediate temporary file when committing to raw repositories.
This will be useful for ostbuild; a user can create their own archive
mode repository which transparently inherits objects from the
root-owned one in /ostree.
The ostree-switch-root tool expects three arguments (argc=4): new root, OS
tree target, and init(8) binary to launch inside it. Also, the error message
when not enough arguments are passed now tells about the second argument
being the target OS tree.
Reviewed-by: Colin Walters <walters@verbum.org>
This is a convenient way to have a lookaside directory of hard links,
which can greatly speed up checkouts. In the future we probably want
to push this down into the repository.
Having the archived vs not distinction in the object system wasn't
useful in light of pack files. In fact, we should probably move
towards generating a pack file per commit by default.
Don't expose GChecksum in APIs. Add a new stream class which allows
us to pass an input stream somewhere, but gather a checksum as it's
read.
Move some bits of the internals towards binary csums.
Colin Walters
Matthias Clasen
Giovanni Campagna
Stef Walter
Adrian Perez
Carl-Anton Ingmarsson
Diego Escalante Urrelo