[ostree] docs: Add a section on writing buildsystems

[Colin Walters <walters src gnome org>]

commit f51d651b10c61ab97039dab4558d0685460af58c
Author: Colin Walters <walters verbum org>
Date:   Thu Mar 3 21:17:53 2016 -0500

    docs: Add a section on writing buildsystems
    
    And add a test that is a demo buildsystem.

 Makefile-tests.am                    |    1 +
 docs/manual/buildsystem-and-repos.md |  180 ++++++++++++++++++++++++++++++++++
 mkdocs.yml                           |    1 +
 tests/test-demo-buildsystem.sh       |  104 +++++++++++++++++++
 4 files changed, 286 insertions(+), 0 deletions(-)
---
diff --git a/Makefile-tests.am b/Makefile-tests.am
index f93b91f..cd686c4 100644
--- a/Makefile-tests.am
+++ b/Makefile-tests.am
@@ -67,6 +67,7 @@ test_scripts = \
        tests/test-auto-summary.sh \
        tests/test-prune.sh \
        tests/test-refs.sh \
+       tests/test-demo-buildsystem.sh \
        $(NULL)
 
 if BUILDOPT_FUSE
diff --git a/docs/manual/buildsystem-and-repos.md b/docs/manual/buildsystem-and-repos.md
new file mode 100644
index 0000000..d418cb0
--- /dev/null
+++ b/docs/manual/buildsystem-and-repos.md
@@ -0,0 +1,180 @@
+# Writing a buildsystem and managing repositories
+
+OSTree is not a package system.  It does not directly support building
+source code.  Rather, it is a tool for transporting and managing
+content, along with package-system independent aspects like bootloader
+management for updates.
+
+We'll assume here that we're planning to generate commits on a build
+server, then have client systems replicate it.  Doing client-side
+assembly is also possible of course, but this discussion will focus
+primarily on server-side concerns.
+
+## Build vs buy
+
+Therefore, you need to either pick an existing tool for writing
+content into an OSTree repository, or to write your own.  An example
+tool is [rpm-ostree](https://github.com/projectatomic/rpm-ostree) - it
+takes as input RPMs, and commits them (currently oriented for a server
+side, but aiming to do client side too).
+
+## Initializing
+
+For this initial discussion, we're assuming you have a single
+`archive-z2` repository:
+
+```
+mkdir repo
+ostree --repo=repo init --mode=archive-z2
+```
+
+You can export this via a static webserver, and configure clients to
+pull from it.
+
+## Writing your own OSTree buildsystem
+
+There exist many, many systems that basically follow this pattern:
+
+```
+$pkg --installroot=/path/to/tmpdir install foo bar baz
+$imagesystem commit --root=/path/to/tmpdir
+```
+
+For various values of `$pkg` such as `yum`, `apt-get`, etc., and
+values of `$imagesystem` could be simple tarballs, Amazon Machine
+Images, ISOs, etc.
+
+Now obviously in this document, we're going to talk about the
+situation where `$imagesystem` is OSTree.  The general idea with
+OSTree is that wherever you might store a series of tarballs for
+applications or OS images, OSTree is likely going to be better.  For
+example, it supports GPG signatures, binary deltas, writing bootloader
+configuration, etc.
+
+OSTree does not include a package/component build system simply
+because there already exist plenty of good ones - rather, it is
+intended to provide an infrastructure layer.
+
+The above mentioned `rpm-ostree compose tree` chooses RPM as the value
+of `$pkg` - so binaries are built as RPMs, then committed as a whole
+into an OSTree commit.
+
+But let's discuss building our own.  If you're just experimenting,
+it's quite easy to start with the command line.  We'll assume for this
+purpose that you have a build process that outputs a directory tree -
+we'll call this tool `$pkginstallroot` (which could be `yum
+--installroot` or `dbootstrap`, etc.).
+
+Your initial prototype is going to look like:
+
+```
+$pkginstallroot /path/to/tmpdir
+ostree --repo=repo commit -s 'build' -b exampleos/x86_64/standard --tree=dir=/path/to/tmpdir
+```
+
+Alternatively, if your build system can generate a tarball, you can
+commit that tarball into OSTree.  For example,
+[OpenEmbedded](http://www.openembedded.org/) can output a tarball, and
+one can commit it via:
+
+```
+ostree commit -s 'build' -b exampleos/x86_64/standard --tree=tar=myos.tar
+```
+
+## Constructing trees from unions
+
+The above is a very simplistic model, and you will quickly notice that
+it's slow.  This is because OSTree has to re-checksum and recompress
+the content each time it's committed.  (Most of the CPU time is spent
+in compression which gets thrown away if the content turns out to be
+already stored).
+
+A more advanced approach is to store components in OSTree itself, then
+union them, and recommit them.  At this point, we recommend taking a
+look at the OSTree API, and choose a programming language supported by
+[GObject Introspection](https://wiki.gnome.org/Projects/GObjectIntrospection)
+to write your buildsystem scripts.  Python may be a good choice, or
+you could choose custom C code, etc.
+
+For the purposes of this tutorial we will use shell script, but it's
+strongly recommended to choose a real programming language for your
+build system.
+
+Let's say that your build system produces separate artifacts (whether
+those are RPMs, zip files, or whatever).  These artifacts should be
+the result of `make install DESTDIR=` or similar.  Basically
+equivalent to RPMs/debs.
+
+Further, in order to make things fast, we will need a separate
+`bare-user` repository in order to perform checkouts quickly via
+hardlinks.  We'll then export content into the `archive-z2` repository
+for use by client systems.
+
+```
+mkdir build-repo
+ostree --repo=build-repo init --mode=bare-user
+```
+
+You can begin committing those as individual branches:
+
+```
+ostree --repo=build-repo commit -b exampleos/x86_64/bash --tree=tar=bash-4.2-bin.tar.gz
+ostree --repo=build-repo commit -b exampleos/x86_64/systemd --tree=tar=systemd-224-bin.tar.gz
+```
+
+Set things up so that whenever a package changes, you redo the
+`commit` with the new package version - conceptually, the branch
+tracks the individual package versions over time, and defaults to
+"latest".  This isn't required - one could also include the version in
+the branch name, and have metadata outside to determine "latest" (or
+the desired version).
+
+Now, to construct our final tree:
+
+```
+rm exampleos-build -rf
+for package in bash systemd; do
+  ostree --repo=build-repo checkout -U --union exampleos/x86_64/${package} exampleos-build
+done
+# Set up a "rofiles-fuse" mount point; this ensures that any processes
+# we run for post-processing of the tree don't corrupt the hardlinks.
+mkdir -p mnt
+rofiles-fuse exampleos-build mnt
+# Now run global "triggers", generate cache files:
+ldconfig -r mnt
+  (Insert other programs here)
+fusermount -u mnt
+ostree --repo=build-repo commit -b exampleos/x86_64/standard --link-checkout-speedup exampleos-build
+```
+
+There are a number of interesting things going on here.  The major
+architectural change is that we're using `--link-checkout-speedup`.
+This is a way to tell OSTree that our checkout is made via hardlinks,
+and to scan the repository in order to build up a reverse `(device,
+inode) -> checksum` mapping.
+
+In order for this mapping to be accurate, we needed the `rofiles-fuse`
+to ensure that any changed files had new inodes (and hence a new
+checksum).
+
+## Migrating content between repositories
+
+Now that we have content in our `build-repo` repository (in
+`bare-user` mode), we need to move the `exampleos/x86_64/standard`
+branch content into the repository just named `repo` (in `archive-z2`
+mode) for export, which will involve zlib compression of new objects.
+We likely want to generate static deltas after that as well.
+
+Let's copy the content:
+
+```
+ostree --repo=repo pull-local build-repo exampleos/x86_64/standard
+```
+
+Clients can now incrementally download new objects - however, this
+would also be a good time to generate a delta from the previous
+commit.
+
+```
+ostree --repo=repo static-delta generate exampleos/x86_64/standard
+```
diff --git a/mkdocs.yml b/mkdocs.yml
index 89211c7..894c17d 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -9,3 +9,4 @@ pages:
       - Atomic Upgrades: 'manual/atomic-upgrades.md'
       - Adapting Existing Systems: 'manual/adapting-existing.md'
       - Formats: 'manual/formats.md'
+      - Build Systems and Repos: 'manual/buildsystem-and-repos.md'
diff --git a/tests/test-demo-buildsystem.sh b/tests/test-demo-buildsystem.sh
new file mode 100755
index 0000000..500eac6
--- /dev/null
+++ b/tests/test-demo-buildsystem.sh
@@ -0,0 +1,104 @@
+#!/bin/bash
+#
+# Copyright (C) 2016 Colin Walters <walters verbum org>
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the
+# Free Software Foundation, Inc., 59 Temple Place - Suite 330,
+# Boston, MA 02111-1307, USA.
+
+set -euo pipefail
+
+if ! fusermount --version >/dev/null 2>&1; then
+    echo "1..0 # SKIP no fusermount"
+    exit 0
+fi
+
+. $(dirname $0)/libtest.sh
+
+echo "1..1"
+
+# Run "triggers" like ldconfig, gtk-update-icon-cache, etc.
+demo_triggers() {
+    root=$1
+    shift
+    mkdir -p ${root}/usr/lib
+    echo updated ldconfig at $(date) > ${root}/usr/lib/ld.so.cache.new
+    mv ${root}/usr/lib/ld.so.cache{.new,}
+}
+
+# Make a binary in /usr/bin/$pkg which contains $version
+exampleos_build_commit_package() {
+    pkg=$1
+    version=$2
+    mkdir -p ${pkg}-package/usr/bin/
+    echo "${pkg}-content ${version}" > ${pkg}-package/usr/bin/${pkg}
+    # Use a dummy subject for this.
+    ostree --repo=build-repo commit -b exampleos/x86_64/${pkg} -s '' --tree=dir=${pkg}-package
+    rm ${pkg}-package -rf
+}
+
+exampleos_recompose() {
+    rm exampleos-build -rf
+    for pkg in ${packages}; do
+       ostree --repo=build-repo checkout -U --union exampleos/x86_64/${pkg} exampleos-build
+    done
+
+    # Now that we have our rootfs, run triggers
+    rofiles-fuse exampleos-build mnt
+    demo_triggers mnt/
+    fusermount -u mnt
+    
+    # Then we commit it, using --link-checkout-speedup to effectively
+    # only re-checksum the ldconfig file.  We also have dummy commit
+    # message here.
+    ostree --repo=build-repo commit -b exampleos/x86_64/standard -s 'exampleos build' 
--link-checkout-speedup exampleos-build
+}
+
+packages="bash systemd"
+
+mkdir build-repo
+ostree --repo=build-repo init --mode=bare-user
+mkdir repo
+ostree --repo=repo init --mode=archive-z2
+# Our FUSE mount point
+mkdir mnt
+
+# "Build" some packages which are really just files with
+# the version number inside.
+exampleos_build_commit_package bash 0.4.7
+exampleos_build_commit_package systemd 224
+
+# Now union the packages and commit
+exampleos_recompose
+
+# This is our first commit - let's publish it.
+ostree --repo=repo pull-local build-repo exampleos/x86_64/standard
+
+# Now, update the bash package - this is a new commit on the branch
+# exampleos/x86_64/bash.
+exampleos_build_commit_package bash 0.5.0
+
+# We now have two commits
+exampleos_recompose
+
+# Publish again:
+ostree --repo=repo pull-local build-repo exampleos/x86_64/standard
+# Optional: Generate a static delta vs the previous build
+ostree --repo=repo static-delta generate exampleos/x86_64/standard
+# Optional: Regenerate the summary file
+ostree --repo=repo summary -u
+
+# Try: ostree --repo=demo-repo ls -R exampleos/x86_64/standard
+
+echo "ok demo buildsystem"