Hacktree This is a combined build system and runtime daemon that builds and manages root filesystems. They both share an executable name "hacktree", but are conceptually split up into two parts: hacktree-build hacktree-root-manager == The recipe set == A recipe is similar to Bitbake's format, except just have metadata - we don't allow arbitrary Python scripts. Also, we assume autotools. Example: SUMMARY = "The basic file, shell and text manipulation utilities." HOMEPAGE = "http://www.gnu.org/software/coreutils/" BUGTRACKER = "http://debbugs.gnu.org/coreutils" LICENSE = "GPLv3+" LIC_FILES_CHKSUM = "file://COPYING;md5=d32239bcb673463ab874e80d47fae504\ file://src/ls.c;startline=5;endline=16;md5=e1a509558876db58fb6667ba140137ad" SRC_URI = "${GNU_MIRROR}/coreutils/${BP}.tar.gz \ file://remove-usr-local-lib-from-m4.patch \ " DEPENDS = "gmp foo" Each recipe will output one or more artifacts. == The Root Repository == A root repository is simply a git repository (at first). Keep reading. == A Root == A root is a filesystem tree, stored in the repository using git. The filesystem tree is built using a set of artifacts. Roots are actually read-only bind-mounts on top of a real writable location, which isn't "public". Each root is checkout of a branch in the repo. TODO can we modify git to add the concept of a "raw" object type only used for blobs? The metadata for it could be stored in extended attributes. Then we could simply hard link the object for regular files to their checkout, and almost totally deduplicate. http://thread.gmane.org/gmane.comp.version-control.git/102001/focus=102094 In GNOME, we will have a root per: - major version (3.0, 3.2) - "runtime", "sdk", and "devel" - Build type (opt, debug) - Architecture (ia32, x86_64) /gnome/root-3.2-runtime-opt-x86_64/{etc,bin,share,usr,lib} /gnome/root-3.2-devel-debug-x86_64/{etc,bin,share,usr,lib} /gnome/.real/root-3.2-runtime-opt-x86_64 /gnome/.real/root-3.2-devel-debug-x86_64 A "runtime" root is what's necessary to run applications. A SDK root is that, plus all the command line developer tools (gcc, gdb, make, strace). And finally the "devel" root has all the API-unstable headers not necessary for applications (NetworkManager.h etc.) Hmm, maybe we should punt developer tools into a Unix app framework. == Artifact == An artifact is a binary result of compiling a recipe (there may be multiple). Think of an artifact as like a Linux distribution "package", except there are no runtime dependencies, conflicts, or pre/post scripts. It's basically just a gzipped tarball, and we encode metadata in the filename. Example: gdk-pixbuf-runtime,o=master,r=3.2-opt-x86_64,b=opt,v=2.24.0-10-g1d39095,.tar.gz This is an artifact from the gdk-pixbuf component. Here's a decoding of the key/value pairs: o: The origin of the build - there are just "master" and "local" r: The name of the root this artifact was compiled against b: The name of the build flavor (known values are "opt" and "debug") v: The output of "git describe". To build a root, we simply unpack the artifacts that compose it, and run "git commit". hacktree will default to splitting up shared libraries' unversioned .so link and header files into -devel, and the rest into -runtime. All binaries default to runtime. == Configuration Management == Have you ever been a system administrator on a zypper/yum system, done an RPM update, which then drops .rpmnew files in your /etc/ that you have to go and hunt for with "find" or something, and said to yourself, "Wow, this system is awesome!!!" ? Right, that's what I thought. Configuration (and systems) management is a tricky problem, and I certainly don't have a magic bullet. However, one large conceptual improvement I think is defaulting to "rebase" versus "merge". This means that we won't permit direct modification of /etc - instead, you HAVE to write a script which accomplishes your goals. It's recommended to make this script revision controlled. We just execute /etc/gnomeos/config.d/* in alphabetical order, while the rootfs is still mounted read-write. Hmm, probably we need to define it to operate on a shadow tree, which we then diff? If the script fails, we can roll back the update, or drop to a shell if interactive. == Local modifications == A key point of this whole endeavour is that we want developers to be able to do local builds. This is surprisingly something not well supported by the Debian/Fedora's tools at least. === Updating a root with a new local artifact === Whenever you install a local artifact, if no "local" branch exists for that root, it's created. Let's say we're debugging gdk-pixbuf, tracking down a memory corruption bug. We've added a few printfs, and want to rerun things. GCC optimization is screwing us, so we build it in debug mode (-O0). The active root is root-3.2-opt. $ pwd ~/src/gdk-pixbufroot $ echo $HACKTREE_ROOT /gnome/root-3.2-opt $ hacktree make debug