From 6647050978c40f023db7400302f66f4cdbf1225a Mon Sep 17 00:00:00 2001
From: Colin Walters <walters@verbum.org>
Date: Fri, 3 Feb 2012 08:42:07 -0500
Subject: [PATCH] Update READMEs a bit

---
 README-build.md | 128 ------------------------------------------------
 README.md       |  87 ++++++++++++++++++--------------
 gnomeos/README  |  42 ++++++----------
 3 files changed, 66 insertions(+), 191 deletions(-)
 delete mode 100644 README-build.md

diff --git a/README-build.md b/README-build.md
deleted file mode 100644
index a9311c8c..00000000
--- a/README-build.md
+++ /dev/null
@@ -1,128 +0,0 @@
-NOTE THIS STUFF IS OUT OF DATE!  I'm working on merging some of these
-ideas into jhbuild for now.
-
-== The recipe set ==
-
-A recipe is similar to Bitbake's format, except just have metadata -
-we don't allow arbitrary Python scripts.  Also, we assume
-autotools. Example:
-
-SUMMARY = "The basic file, shell and text manipulation utilities."
-HOMEPAGE = "http://www.gnu.org/software/coreutils/"
-BUGTRACKER = "http://debbugs.gnu.org/coreutils"
-LICENSE = "GPLv3+"
-LIC_FILES_CHKSUM = "file://COPYING;md5=d32239bcb673463ab874e80d47fae504\
-                    file://src/ls.c;startline=5;endline=16;md5=e1a509558876db58fb6667ba140137ad"
-SRC_URI = "${GNU_MIRROR}/coreutils/${BP}.tar.gz \
-           file://remove-usr-local-lib-from-m4.patch \
-          "
-DEPENDS = "gmp foo"
-
-Each recipe will output one or more artifacts.
-
-
-In GNOME, we will have a root per:
- - major version (3.0, 3.2)
- - "runtime", "sdk", and "devel"
- - Build type (opt, debug)
- - Architecture (ia32, x86_64)
-
-/gnome/root-3.2-runtime-opt-x86_64/{etc,bin,share,usr,lib}
-/gnome/root-3.2-devel-debug-x86_64/{etc,bin,share,usr,lib}
-/gnome/.real/root-3.2-runtime-opt-x86_64
-/gnome/.real/root-3.2-devel-debug-x86_64
-
-A "runtime" root is what's necessary to run applications.  A SDK root
-is that, plus all the command line developer tools (gcc, gdb, make,
-strace).  And finally the "devel" root has all the API-unstable
-headers not necessary for applications (NetworkManager.h etc.)
-
-Hmm, maybe we should punt developer tools into a Unix app framework.
-
-== Artifact ==
-
-An artifact is a binary result of compiling a recipe (there may be
-multiple).  Think of an artifact as like a Linux distribution
-"package", except there are no runtime dependencies, conflicts, or
-pre/post scripts.  It's basically just a gzipped tarball, and we
-encode metadata in the filename.
-
-Example:
-
-gdk-pixbuf-runtime,o=master,r=3.2-opt-x86_64,b=opt,v=2.24.0-10-g1d39095,.tar.gz
-
-This is an artifact from the gdk-pixbuf component.  Here's a decoding of the key/value pairs:
-
-o: The origin of the build - there are just "master" and "local"
-r: The name of the root this artifact was compiled against
-b: The name of the build flavor (known values are "opt" and "debug")
-v: The output of "git describe".
-
-To build a root, we simply unpack the artifacts that compose it, and
-run "git commit".
-
-hacktree will default to splitting up shared libraries' unversioned .so
-link and header files into -devel, and the rest into -runtime.
-
-All binaries default to runtime.
-
-Local modifications ==
-
-A key point of this whole endeavour is that we want developers to be
-able to do local builds.  This is surprisingly something not well
-supported by the Debian/Fedora's tools at least.
-
-=== Updating a root with a new local artifact ===
-
-Whenever you install a local artifact, if no "local" branch exists for
-that root, it's created.
-
-Let's say we're debugging gdk-pixbuf, tracking down a memory
-corruption bug.  We've added a few printfs, and want to rerun things.
-GCC optimization is screwing us, so we build it in debug mode (-O0).
-The active root is root-3.2-opt.
-
-$ pwd
-~/src/gdk-pixbufroot
-$ echo $HACKTREE_ROOT
-/gnome/root-3.2-opt
-<hack hack hack>
-$ hacktree make debug
-<time passes, hopefully not too much>
-$ ls gdk-pixbuf*.tar.gz
-gdk-pixbuf-runtime,o=master,r=3.2-opt,b=debug,v=2.24.0-10-g1d39095,.tar.gz
-gdk-pixbuf-devel,o=master,r=3.2-opt,b=debug,v=2.24.0-10-g1d39095,.tar.gz
-gdk-pixbuf-manifests,o=master,r=3.2-opt,b=debug,v=2.24.0-10-g1d39095,.tar.gz
-$ hacktree install gdk-pixbuf*,o=master,r=3.2-opt,b=debug,v=2.24.0-10-g1d39095,.tar.gz
-<policykit auth dialog>
-
-Now here's where the cool stuff happens.  hacktree takes
-/gnome/root-3.2-opt (the which is given in the r= above), and looks
-for the corresponding git branch (root-3.2-opt).  Now hacktree notices
-there's no corresponding "local" branch, i.e. local-3.2-opt.  One is
-created and checked out:
-
-# pwd
-/gnome/repo.git
-# git branch local-3.2-opt root-3.2-opt
-# git clone --branch local-3.2-opt /gnome/repo.git /gnome/.real-local-3.2-opt
-
-Now, the artifacts specified are overlaid:
-
-# cd /gnome/.real-local-3.2-opt
-# tar xvf
-
-Ok, now we need to remove old no longer shipped files from the root.
-Thus, we need a list of files corresponding to each original artifact,
-and to know which artifacts are in a root.  Note above that one of the
-artifacts produced was "manifests".  This contains files like:
-
-/meta/manifests/gdk-pixbuf-runtime.list
-/meta/manifests/gdk-pixbuf-devel.list
-
-Thus we diff the manifests, and clean up any leftover files.
-
-# git commit -a -m "Install artifact gdk-pixbuf-runtime,o=master,r=3.2-opt,b=debug,v=2.24.0-10-g1d39095,.tar.gz"
-# git checkout 
-
-
diff --git a/README.md b/README.md
index 36d003d0..7e2ecc5d 100644
--- a/README.md
+++ b/README.md
@@ -1,5 +1,5 @@
 OSTree
-========
+======
 
 Problem statement
 -----------------
@@ -67,7 +67,7 @@ Comparison with existing tools
     stuck on whatever the distro provides.
 
 Who is ostree for?
-------------------------------
+------------------
 
 First - operating system developers and testers.  I specifically keep
 a few people in mind - Dan Williams and Eric Anholt, as well as myself
@@ -115,16 +115,22 @@ lives inside a distro created partition, a tricky part here is that we
 need to know how to interact with the installed distribution's grub.
 This is an annoying but tractable problem.
 
-OSTree will allow efficiently parallel installing and downloading OS
-builds.
+First, we install a kernel+initramfs alongside the distribution's.
+Then, we have a "trampoline" ostree-init binary which is statically
+linked, and boot the kernel with init=/ostree/ostree-init.  This then
+takes care of chrooting and running the init binary.
 
-An important note here is that we explicitly link /home in each root
-to the real /home.  This means you have your data.  This also implies
-we share uid/gid, so /etc/passwd will have to be in sync.  Probably
-what we'll do is have a script to pull the data from the "host" OS.
+An important note here is that we bind mount the real /home.  This
+means you have your data.  This also implies we share uid/gid, so
+/etc/passwd will have to be in sync.  Probably what we'll do is have a
+script to pull the data from the "host" OS.
 
-Other shared directories are /var and /root.  Note that /etc is
-explicitly NOT shared!
+I've decided for now to move /var into /ostree to avoid sharing it
+with the "host" distribution, because in practice we're likely
+to hit incompatibilities.
+
+Do note however /etc lives *inside* the OSTree; it's presently
+versioned and readonly like everything else.
 
 On a pure OSTree system, the filesystem layout will look like this:
 
@@ -132,6 +138,7 @@ On a pure OSTree system, the filesystem layout will look like this:
 		|-- boot
 		|-- home
 		|-- ostree
+		|   |-- var
 		|   |-- current -> gnomeos-3.2-opt-7e9788a2
 		|   |-- gnomeos-3.0-opt-393a4555
 		|   |   |-- etc
@@ -154,7 +161,6 @@ On a pure OSTree system, the filesystem layout will look like this:
 		|       |-- sys
 		|       `-- usr
 		|-- root
-		`-- var
 		
 
 Making this efficient
@@ -204,19 +210,19 @@ can do this because again each checkout is designed to be read-only.
 
 So we mentioned above there are:
 
-		/gnomeos/root-3.0-opt
-		/gnomeos/root-3.2-opt
+		/ostree/gnomeos-3.2-opt-7e9788a2
+		/ostree/gnomeos-3.2-opt-393a4555
 
 There is also a "repository" that looks like this:
 
-		.ht/objects/17/a95e8ca0ba655b09cb68d7288342588e867ee0.file
-		.ht/objects/17/68625e7ff5a8db77904c77489dc6f07d4afdba.meta
-		.ht/objects/17/cc01589dd8540d85c0f93f52b708500dbaa5a9.file
-		.ht/objects/30
-		.ht/objects/30/6359b3ca7684358a3988afd005013f13c0c533.meta
-		.ht/objects/30/8f3c03010cedd930b1db756ce659c064f0cd7f.meta
-		.ht/objects/30/8cf0fd8e63dfff6a5f00ba5a48f3b92fb52de7.file
-		.ht/objects/30/6cad7f027d69a46bb376044434bbf28d63e88d.file
+		/ostree/repo/objects/17/a95e8ca0ba655b09cb68d7288342588e867ee0.file
+		/ostree/repo/objects/17/68625e7ff5a8db77904c77489dc6f07d4afdba.meta
+		/ostree/repo/objects/17/cc01589dd8540d85c0f93f52b708500dbaa5a9.file
+		/ostree/repo/objects/30
+		/ostree/repo/objects/30/6359b3ca7684358a3988afd005013f13c0c533.meta
+		/ostree/repo/objects/30/8f3c03010cedd930b1db756ce659c064f0cd7f.meta
+		/ostree/repo/objects/30/8cf0fd8e63dfff6a5f00ba5a48f3b92fb52de7.file
+		/ostree/repo/objects/30/6cad7f027d69a46bb376044434bbf28d63e88d.file
 
 Each object is either metadata (like a commit or tree), or a hard link
 to a regular file.
@@ -242,27 +248,36 @@ during an upgrade and reboot process, you either get the full new
 system, or the old one.  There is no "Please don't turn off your
 computer".  We do this by simply using a symbolic link like:
 
-/gnomeos -> /gnomeos-e3b0c4429
+/ostree/current -> /ostree/gnomeos-3.4-opt-e3b0c4429
 
-Where /gnomeos-e3b0c4429/ has the full regular filesystem tree with
-usr/ etc/ directories as above.  To upgrade or rollback (there is no
+Where gnomeos-e3b0c4429 has the full regular filesystem tree with usr/
+etc/ directories as above.  To upgrade or rollback (there is no
 difference internally), we simply check out a new tree into
-/gnomeos-b90ae4763 for example, then swap the symbolic link, then
-remove the old tree.
+gnomeos-b90ae4763 for example, then swap the "current" symbolic link,
+then remove the old tree.
 
-But does this mean you have to "reboot" for OS upgrades?  Very likely,
+But does this mean you have to reboot for OS upgrades?  Very likely,
 yes - and this is no different from RPM/deb or whatever.  They just
-typically lie to you about it =) But read on.
+typically lie to you about it =)
 
-Let's consider a security update to a shared library.  We can download
-the update to the repository, build a new tree and atomically swap it
-as above, but what if a process has the old shared library in use?
+A typical model with RPM/deb is to unpack the new files, then use some
+IPC mechanism (SIGHUP, a control binary like /usr/sbin/apachectl) to
+signal the running process to reload.  There are multiple problems
+with this - one is that in the new state, daemon A may depend on the
+updated configuration in daemon B.  This may not be particularly
+common in default configurations, but it's highly likely that that
+some deployments will have e.g. apache talking to a local MySQL
+instance.  So you really want to do is only apply the updated
+configuration when all the files are in place; not after each RPM or
+.deb is installed.
 
-Here's where we will probably need to inspect which processes are
-using the library - if any are, then we need to trigger either a
-logout/login if it's just the desktop shell and/or apps, or a
-"fastboot" if not.  A fastboot is dropping to "init 1" effectively,
-then going back to "init 5".
+What's even harder is the massive set of race conditions that are
+possible while RPM/deb are in the process of upgrading.  Cron jobs are
+very likely to hit this.  If we want the ability to apply updates to a
+live system, we could first pause execution of non-upgrade userspace
+tasks.  This could be done via SIGSTOP for example.  Then, we can swap
+around the filesystem tree, and then finally attempt to apply updates
+via SIGHUP, and if possible, restart processes.
 
 Configuration Management
 ------------------------
diff --git a/gnomeos/README b/gnomeos/README
index 26819564..f05d214d 100644
--- a/gnomeos/README
+++ b/gnomeos/README
@@ -16,19 +16,18 @@ base "runtime", and one "devel" with all of the development tools like
 gcc.  We then import that into an OSTree branch
 e.g. "bases/gnomeos-3.4-yocto-i686-devel".
 
-Now we also assume that you have ostree installed on the host build
-system via e.g. jhbuild or RPM if doing a cross build.  The core
-ostbuild tool can then chroot into a checkout of the Yocto base, and
-start generating artifacts.
+We also have a Yocto recipe "ostree-native" which generates (as you
+might guess) a native binary of ostree.  That binary is used to import
+into an "archive mode" OSTree repository.  You can see it in
+$builddir/tmp/deploy/images/repo.
 
-Each generated artifact is committed to an OSTree branch like
-"artifacts/gnomeos-3.4-i686-devel/libxslt/master/runtime".  Then, a
-"compose" process merges together the individual filesystem trees into
-the final branches (e.g. gnomeos-3.4-i686-devel), and the process
-repeats.
+Now that we have an OSTree repository storing a base filesystem, we
+can use "ostbuild" which uses "linux-user-chroot" to chroot inside,
+run a build on a source tree, and outputs binaries, which we then add
+to the build tree for the next module, and so on.
 
 ostbuild details
--------------------
+----------------
 
 The simple goal of ostbuild is that it only takes as input a
 "manifest" which is basically just a list of components to build.  A
@@ -39,24 +38,13 @@ There is no support for building from "tarballs" - I want the ability
 to review all of the code that goes in, and to efficiently store
 source code updates.
 
-For GNOME, tarballs are mostly pointless - it's easy enough to just
-run autogen.sh.  However there are two challenges:
+The result of a build of a component is an OSTree branch like
+"artifacts/gnomeos-3.4-i686-devel/libxslt/master".  Then, a "compose"
+process merges together the individual filesystem trees into the final
+branches (e.g. gnomeos-3.4-i686-devel).
 
-1) Tarballs for modules which self-build-depend may include
-   pre-generated files.  For example - flex's tarball includes a
-   generated .c file for the parser.  For these, we can either move
-   the module build to the Yocto level (thus giving a convenient way
-   to pull in host files), or possibly add the ability to
-   hardlink/copy in host binaries to ostbuild.
-
-2) Tarballs which include translations pulled from a different
-   location.  For example - bison.  For these, we basically have to
-   maintain our own git repositories.
-
-
-
-building
---------
+Doing a full build on your system
+---------------------------------
 
 srcdir=/src
 builddir=/src/build