[jhbuild/wip/rootowned] rootfs: More DESIGN thoughts

[Colin Walters <walters src gnome org>]

commit 4bd3a6251fbcee193b1a17f5b877a165bc5b270b
Author: Colin Walters <walters verbum org>
Date:   Sat Oct 8 12:04:15 2011 -0400

    rootfs: More DESIGN thoughts

 rootfs-manager/DESIGN |  142 +++++++++++++++++++++++++++++++++++++++++++------
 1 files changed, 125 insertions(+), 17 deletions(-)
---
diff --git a/rootfs-manager/DESIGN b/rootfs-manager/DESIGN
index 5d4041b..397b779 100644
--- a/rootfs-manager/DESIGN
+++ b/rootfs-manager/DESIGN
@@ -9,28 +9,56 @@ a set of software compiled into an alternative root.
 What we're building is components.  A component corresponds to exactly
 one git repository.
 
+== The Root Repository ==
+
+A root repository is simply a git repository:root-
+/gnome/repo.git
+
+Keep reading.
+
 == A Root ==
 
-A root is composed of a set of artifacts, which overlay into a
-filesystem tree.  Roots are actually read-only bind-mounts on top of a
-real writable location, which isn't "public".
+A root is a filesystem tree, stored in the repository using git.  The
+filesystem tree is built using a set of artifacts.  Roots are actually
+read-only bind-mounts on top of a real writable location, which isn't
+"public".
+
+Each root is checkout of a branch in the repo.
+
+TODO can we modify git to add the concept of a "raw" object type only
+used for blobs?  The metadata for it could be stored in extended
+attributes. 
+http://thread.gmane.org/gmane.comp.version-control.git/102001/focus=102094
 
-/gnome/root-3.2-opt/{share,include,lib}
-/gnome/root-3.2-debug/{share,include,lib}
+In GNOME, we will have a root per:
+ - major version (3.0, 3.2)
+ - "runtime", "sdk", and "devel"
+ - Build type (opt, debug)
+ - Architecture (ia32, x86_64)
 
-Roots may need a git style database of sha1sums, in order to do
-efficient updates.  Hmm...should we go full on git style objects/?
+/gnome/root-3.2-runtime-opt-x86_64/{etc,bin,share,usr,lib}
+/gnome/root-3.2-devel-debug-x86_64/{etc,bin,share,usr,lib}
+/gnome/.real/root-3.2-runtime-opt-x86_64
+/gnome/.real/root-3.2-devel-debug-x86_64
+
+A "runtime" root is what's necessary to run applications.  A SDK root
+is that, plus all the command line developer tools (gcc, gdb, make,
+strace).  And finally the "devel" root has all the API-unstable
+headers not necessary for applications (NetworkManager.h etc.)
+
+Hmm, maybe we should punt developer tools into a Unix app framework.
 
 == Artifact ==
 
 An artifact is a binary result of compiling a component (there may be
 multiple).  Think of an artifact as like a Linux distribution
 "package", except there are no runtime dependencies, conflicts, or
-pre/post scripts.  It's basically just a gzipped tarball.
+pre/post scripts.  It's basically just a gzipped tarball, and we
+encode metadata in the filename.
 
 Example:
 
-gdk-pixbuf,o=master,r=3.2-opt,b=opt,v=2.24.0-10-g1d39095,.tar.gz
+gdk-pixbuf-runtime,o=master,r=3.2-opt-x86_64,b=opt,v=2.24.0-10-g1d39095,.tar.gz
 
 This is an artifact from the gdk-pixbuf component.  Here's a decoding of the key/value pairs:
 
@@ -39,6 +67,14 @@ r: The name of the root this artifact was compiled against
 b: The name of the build flavor (known values are "opt" and "debug")
 v: The output of "git describe".
 
+To build a root, we simply unpack the artifacts that compose it, and
+run "git commit".
+
+jhbuild will default to splitting up shared libraries' unversioned .so
+link and header files into -devel, and the rest into -runtime.
+
+All binaries default to runtime.
+
 == Configuration Management ==
 
 Have you ever been a system administrator on a zypper/yum system, done
@@ -71,15 +107,87 @@ supported by the Debian/Fedora's tools at least.
 
 === Updating a root with a new local artifact ===
 
-Since roots are read-only, we can safely "cp -al" them (a fast and
-efficient operation).  Let's walk through this:
+Whenever you install a local artifact, if no "local" branch exists for
+that root, it's created.
+
+Let's say we're debugging gdk-pixbuf, tracking down a memory
+corruption bug.  We've added a few printfs, and want to rerun things.
+GCC optimization is screwing us, so we build it in debug mode (-O0).
+The active root is root-3.2-opt.
+
+$ pwd
+~/src/gdk-pixbufroot
+$ echo $JHBUILD_ROOT
+/gnome/root-3.2-opt
+<hack hack hack>
+$ jhbuild make debug
+<time passes, hopefully not too much>
+$ ls gdk-pixbuf*.tar.gz
+gdk-pixbuf-runtime,o=master,r=3.2-opt,b=debug,v=2.24.0-10-g1d39095,.tar.gz
+gdk-pixbuf-devel,o=master,r=3.2-opt,b=debug,v=2.24.0-10-g1d39095,.tar.gz
+$ jhbuild install gdk-pixbuf*,o=master,r=3.2-opt,b=debug,v=2.24.0-10-g1d39095,.tar.gz
+<policykit auth dialog>
+
+Now here's where the cool stuff happens.  jhbuild takes
+/gnome/root-3.2-opt (the which is given in the r= above), and looks
+for the corresponding git branch (root-3.2-opt).  Now jhbuild notices
+there's no corresponding "local" branch, i.e. local-3.2-opt.  One is
+created and checked out:
+
+# pwd
+/gnome/repo.git
+# git branch local-3.2-opt root-3.2-opt
+# git clone --branch local-3.2-opt /gnome/repo.git /gnome/local-3.2-opt
+
+Now, the artifacts specified are overlaid:
+
+# cd /gnome/local-3.2-opt
+# tar xvf
+
+Ok, now we need to remove old no longer shipped files from the root.
+Thus, we need a list of files corresponding to each original artifact,
+and to know which artifacts are in a root.  This is depressingly close
+to package management =(
+
+But note that we may only need to ship these manifests in the -devel
+root.
+
+# git commit -a -m "Install artifact gdk-pixbuf-runtime,o=master,r=3.2-opt,b=debug,v=2.24.0-10-g1d39095,.tar.gz"
 
-Let's say we build a new local gdk-pixbuf.
-/gnome/root-3.2-opt/ 
+== Updating roots from master ==
 
+We need to split this into two phases - download, and apply.  Luckily,
+git exposes exactly the operations we need, namely "git fetch" and
+"git checkout".
 
-== Updating roots from master ==
+jhbuild fetch
+jhbuild 
+
+== Many roots on build master ==
+
+builds.git
+  Generated after every time an artifact is built.
+fastqa.git
+  After each root is built, a very quick test suite is run in it;
+  probably this is booting to GDM.  If that works, the latest build
+  is committed here.  Hopefully we can get fastqa under 2 minutes.
+dailyqa.git
+  Much more extensive tests, let's say they take 24 hours.
+
+
+== TODO figure out ==
+
+* devel/runtime split for things with binaries (buildapi)
+* xserver uses cpp (ugh)
+ -> try to move artifact split into upstream git?
+* Language packs?
+* what about optional random ui tools that are built with tracker?
+ -> don't build them by default, make them an app
+
+
+== RPM Compatibility ==
 
-Depending on the frequency of update, we could simply compute
-"expected delta" files and apply those.  We should probably look at
-the git protocol here.
+We should be able to install LSB rpms.  This implies providing "rpm".
+The tricky part here is since the OS itself is not assembled via RPMs,
+we need to fake up a database of "provides" as if we were.  This is
+probably not terribly hard, but still.