From 4c58d2f0cb54f65d1855ba1f38e4f80ad37d0b0e Mon Sep 17 00:00:00 2001 From: Colin Walters Date: Sat, 15 Oct 2011 16:46:33 -0400 Subject: [PATCH] Have an introductory README.md that makes sense Skip the "how do we build" for now. --- DESIGN | 291 ----------------------------------------------- README-build.md | 128 +++++++++++++++++++++ README.md | 293 ++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 421 insertions(+), 291 deletions(-) delete mode 100644 DESIGN create mode 100644 README-build.md create mode 100644 README.md diff --git a/DESIGN b/DESIGN deleted file mode 100644 index dbbda153..00000000 --- a/DESIGN +++ /dev/null @@ -1,291 +0,0 @@ -Hacktree - -This is a combined build system and runtime daemon that builds and -manages root filesystems. They both share an executable name -"hacktree", but are conceptually split up into two parts: - -hacktree-build -hacktree-root-manager - -== Problem statement == - -Hacking on the core operating system is painful - this includes most -of GNOME from upower and NetworkManager up to gnome-shell. I want a -system that matches these requirements: - -0) Does not disturb your existing OS -1) Is not terribly slow to use -2) Shares your $HOME - you have your data -3) Allows easy rollback -4) Ideally allows access to existing apps - -== Comparison with existing tools == - -Virtualization: - Fails on points 1) and 2). - -Rebuilding OS packages: - Fails on points 0) and 3). Is also just very annoying. - -jhbuild + OS packages: - The state of the art in GNOME - but can only build non-root things - - this means you can't build NetworkManager, and thus are permanently - stuck on whatever the distro provides. - -== Who is hacktree for? == - -First - operating system developers and testers. I specifically keep -a few people in mind - Dan Williams and Eric Anholt, as well as myself -obviously. For Eric Anholt, a key use case for him is being able to -try out the latest gnome-shell, and combine it with his work on Mesa, -and see how it works/performs - while retaining the ability to roll -back if one or both breaks. - -The rollback concept is absolutely key for shipping anything to -enthusiasts or knowledable testers. With a system like this, a tester -can easily perform a local rollback - something just not well -supported by dpkg/rpm. (Why not Conary? AIUI Conary is targeted at -individual roots, so while you could roll back a given root, it would -use significantly more disk space than hacktree) - -Also, distributing operating system trees (instead of packages) gives -us a sane place to perform automated QA **before** we ship it to -testers. We should never be wasting these people's time. - -Even better, this system would allow testers to bisect across -operating system builds efficiently. - -== The core idea == - -chroots are the original lightweight "virtualization". Let's use -them. So basically, you install a mainstream distribution (say -Debian). It has a root filesystem with a regular layout, /etc, /usr, -/lib etc. - -Now, what we can do is have a system that installs chroots, like: - -/gnomeos/root-3.0-opt -/gnomeos/root-3.2-opt - -These live in the same root filesystem as your regular distribution -(Note though, the root partition should be reasonably sized, or - hopefully you've used just one big partition). - -You should be able to boot into one of these roots. Since hacktree -lives inside a distro created partition, a tricky part here is that we -need to know how to interact with the installed distribution's grub. -This is an annoying but tractable problem. - -Hacktree will allow efficiently parallel installing and downloading OS -builds. - -== The recipe set == - -A recipe is similar to Bitbake's format, except just have metadata - -we don't allow arbitrary Python scripts. Also, we assume -autotools. Example: - -SUMMARY = "The basic file, shell and text manipulation utilities." -HOMEPAGE = "http://www.gnu.org/software/coreutils/" -BUGTRACKER = "http://debbugs.gnu.org/coreutils" -LICENSE = "GPLv3+" -LIC_FILES_CHKSUM = "file://COPYING;md5=d32239bcb673463ab874e80d47fae504\ - file://src/ls.c;startline=5;endline=16;md5=e1a509558876db58fb6667ba140137ad" -SRC_URI = "${GNU_MIRROR}/coreutils/${BP}.tar.gz \ - file://remove-usr-local-lib-from-m4.patch \ - " -DEPENDS = "gmp foo" - -Each recipe will output one or more artifacts. - -== The Root Repository == - -A root repository is simply a git repository (at first). - -Keep reading. - -== A Root == - -A root is a filesystem tree, stored in the repository using git. The -filesystem tree is built using a set of artifacts. Roots are actually -read-only bind-mounts on top of a real writable location, which isn't -"public". - -Each root is checkout of a branch in the repo. - -TODO can we modify git to add the concept of a "raw" object type only -used for blobs? The metadata for it could be stored in extended -attributes. Then we could simply hard link the object for regular -files to their checkout, and almost totally deduplicate. -http://thread.gmane.org/gmane.comp.version-control.git/102001/focus=102094 - -In GNOME, we will have a root per: - - major version (3.0, 3.2) - - "runtime", "sdk", and "devel" - - Build type (opt, debug) - - Architecture (ia32, x86_64) - -/gnome/root-3.2-runtime-opt-x86_64/{etc,bin,share,usr,lib} -/gnome/root-3.2-devel-debug-x86_64/{etc,bin,share,usr,lib} -/gnome/.real/root-3.2-runtime-opt-x86_64 -/gnome/.real/root-3.2-devel-debug-x86_64 - -A "runtime" root is what's necessary to run applications. A SDK root -is that, plus all the command line developer tools (gcc, gdb, make, -strace). And finally the "devel" root has all the API-unstable -headers not necessary for applications (NetworkManager.h etc.) - -Hmm, maybe we should punt developer tools into a Unix app framework. - -== Artifact == - -An artifact is a binary result of compiling a recipe (there may be -multiple). Think of an artifact as like a Linux distribution -"package", except there are no runtime dependencies, conflicts, or -pre/post scripts. It's basically just a gzipped tarball, and we -encode metadata in the filename. - -Example: - -gdk-pixbuf-runtime,o=master,r=3.2-opt-x86_64,b=opt,v=2.24.0-10-g1d39095,.tar.gz - -This is an artifact from the gdk-pixbuf component. Here's a decoding of the key/value pairs: - -o: The origin of the build - there are just "master" and "local" -r: The name of the root this artifact was compiled against -b: The name of the build flavor (known values are "opt" and "debug") -v: The output of "git describe". - -To build a root, we simply unpack the artifacts that compose it, and -run "git commit". - -hacktree will default to splitting up shared libraries' unversioned .so -link and header files into -devel, and the rest into -runtime. - -All binaries default to runtime. - -== Configuration Management == - -Have you ever been a system administrator on a zypper/yum system, done -an RPM update, which then drops .rpmnew files in your /etc/ that you -have to go and hunt for with "find" or something, and said to -yourself, "Wow, this system is awesome!!!" ? Right, that's what I -thought. - -Configuration (and systems) management is a tricky problem, and I -certainly don't have a magic bullet. However, one large conceptual -improvement I think is defaulting to "rebase" versus "merge". - -This means that we won't permit direct modification of /etc - instead, -you HAVE to write a script which accomplishes your goals. It's -recommended to make this script revision controlled. We just execute -/etc/gnomeos/config.d/* in alphabetical order, while the rootfs is -still mounted read-write. - -Hmm, probably we need to define it to operate on a shadow tree, which -we then diff? - -If the script fails, we can roll back the update, or drop to a shell -if interactive. - -== Local modifications == - -A key point of this whole endeavour is that we want developers to be -able to do local builds. This is surprisingly something not well -supported by the Debian/Fedora's tools at least. - -=== Updating a root with a new local artifact === - -Whenever you install a local artifact, if no "local" branch exists for -that root, it's created. - -Let's say we're debugging gdk-pixbuf, tracking down a memory -corruption bug. We've added a few printfs, and want to rerun things. -GCC optimization is screwing us, so we build it in debug mode (-O0). -The active root is root-3.2-opt. - -$ pwd -~/src/gdk-pixbufroot -$ echo $HACKTREE_ROOT -/gnome/root-3.2-opt - -$ hacktree make debug -