Releasing binaries from BuildStream to the world

[Sam Thursfield <sam thursfield codethink co uk>]

Hello,

I've been doing some work on switching the Baserock project[1] to use
BuildStream as its build tool.

Existing Baserock build tools allow bootstrapping a compiler and sysroot
using tools from the host. BuildStream doesn't allow this, which means
that Baserock needs to start releasing sysroot binaries somewhere that
we can reference in our builds.

I have 2 questions on how best to do this which are relevant to the
wider BuildStream community because deploying binaries is something
we'll pretty much all need to do.

1. How and where should we host these binaries? How do we transfer them
   there once the build completes?

2. How do we record the provenance of those binaries?

Hosting
-------

The sensible options for hosting a sysroot seem to be:

  a. tar.gz on a web server
  b. a tree in an OSTree repo
  c. Docker image in a Docker registry

OSTree is my preferred option as it supports file-level deduplication,
and we already have infrastructure in place to host OSTree repos.

But how do we get the files from BuildStream to the OSTree repository?

One option is to "re-use" our OSTree cache as a releases repo. In fact
I'm proposing that as our interim solution[2] but there are two issues
with it. Firstly it makes the cache no longer be a cache -- we'll need
to ensure that the release artifacts never get deleted and are backed
up and that's much more difficult if they're mixed in with 100s of GBs
of less useful artifacts. Secondly, the BuildStream artifact format is
effectively an implementation detail that may change in future. The ref
names used by BuildStream are not optimized for human readability. An
artifact may not necessarily contain all the files we need. Also,
BuildStream artifacts (in its current format) don't carry full
provenance info -- you can't see what the source repo and source commit
was just by looking at its embedded metadata.

If we don't re-use BuildStream's artifact sharing functionality then
we need some other way of getting the binaries from BuildStream's local
cache to our public releases repo.

We could resurrect "ostree-push"[3] for this perhaps? Our release
workflow would then become:

    bst build bootstrap/stage3-sysroot.bst
    bst checkout bootstrap/stage3-sysroot.bst ./sysroot
    ostree init --repo=./repo
    ostree commit --repo=./repo --branch=stage3-sysroot/x86_64 ./sysroot
    ostree remote-add releases ssh://ostree.baserock.org/releases
    ostree-push --repo=./repo --update releases stage3-sysroot/x86_64

Does that seem sensible? Or is this something we could usefully build
into BuildStream itself? Could we have an `ostree_deploy` element for
example?


Provenance
----------

Baserock takes the stance that you should always know where your
binaries come from, i.e. what was the source code, what arguments
were passed to `configure`, etc.

The way existing Baserock build tools did this was by dumping the cache
key's inputs into the filesystem of each artifact. So you would end up
with a /baserock/*.json directory, and each file would contain stuff
like this:

    {
        "repo-alias": "upstream:make-tarball",
        "repo": "git://git.baserock.org/delta/make-tarball",
        "original_ref": "4978d7129e42340ab9efeb0cb9cae4ad0fa052d4",
        "sha1": "4978d7129e42340ab9efeb0cb9cae4ad0fa052d4",
        "configure-commands": [
            "./configure --prefix=\"$PREFIX\" --disable-nls"
        ],
        ...
    }

BuildStream has all this info available too, but it lacks the ability to
dump it as JSON. I propose that we add a `bst show --mode=json` command
that allows us to dump the cache key inputs into the sysroot before we
push it to the releases repo. So our release process would just gain a
line like:

    bst show --mode=json bootstrap/stage3-sysroot.bst \
        > stage3-sysroot/manifest.json

Does this seem sensible?

I'm interested if anyone else is looking at similar processes, and
how much it makes sense / is possible to automate this stuff with
BuildStream. Perhaps we could/should wrap the release process up in
a `bootstrap/stage3-sysroot-release.bst` script element? Perhaps a
kind of "deployment" base element would be useful in BuildStream??

Thanks
Sam

--
Sam Thursfield, Codethink Ltd.
Office telephone: +44 161 236 5575