GNOME.org
 

Re: Releasing binaries from BuildStream to the world

On 28/09/17 15:19, Tristan Van Berkom wrote:

Things for baserock will look all around nicer when producing the very binaries needed to get off the ground, 
full circle :)


It's turtles all the way down...


But how do we get the files from BuildStream to the OSTree repository?


Using `bst checkout` somewhere and then committing the checkout to an OSTree repository.

Remember we only really mutate data (often by compiling it) but intentionally treat deployment as a separate 
problem space.


OK. So I guess the answer to my later questions...


Perhaps we could/should wrap the release process up in
a `bootstrap/stage3-sysroot-release.bst` script element? Perhaps a
kind of "deployment" base element would be useful in BuildStream??
...is "no, do that stuff outside BuildStream". Which is fine, we can wrap up the release process in a 10 line script. 


One option is to "re-use" our OSTree cache as a releases repo. In fact
I'm proposing that as our interim solution[2] but there are two issues
with it. Firstly it makes the cache no longer be a cache -- we'll need
to ensure that the release artifacts never get deleted and are backed
up and that's much more difficult if they're mixed in with 100s of GBs
of less useful artifacts. Secondly, the BuildStream artifact format is
effectively an implementation detail that may change in future. The ref
names used by BuildStream are not optimized for human readability. An
artifact may not necessarily contain all the files we need. Also,
BuildStream artifacts (in its current format) don't carry full
provenance info -- you can't see what the source repo and source commit
was just by looking at its embedded metadata.

If we don't re-use BuildStream's artifact sharing functionality then
we need some other way of getting the binaries from BuildStream's local
cache to our public releases repo.


I am skeptical of using an artifact cache share as a means of distribution/hosting in this way.

We may end up addressing your concerns of deletion, however what you propose seems to heavily rely on 
buildstream's internal artifact cache structure - I wont even try to guarantee that this structure will never 
change, only that buildstream will continue to be able to interface with its caches and properly 
push/pull/checkout etc.


Sounds like we agree :-)



We could resurrect "ostree-push"[3] for this perhaps? Our release
workflow would then become:

    bst build bootstrap/stage3-sysroot.bst
    bst checkout bootstrap/stage3-sysroot.bst ./sysroot
    ostree init --repo=./repo
    ostree commit --repo=./repo --branch=stage3-sysroot/x86_64 ./sysroot
    ostree remote-add releases ssh://ostree.baserock.org/releases
    ostree-push --repo=./repo --update releases stage3-sysroot/x86_64

Does that seem sensible? Or is this something we could usefully build
into BuildStream itself? Could we have an `ostree_deploy` element for
example?


Internally we are using an ad-hoc fork of the ostree-push code, which is more optimized and fault tollerent, 
a similar script could be fashioned out of buildstream's push/pull code.
Right... I noticed you have some PRs open against the ostree-push repo, are those relevant still or is there quite a bit of a delta between that code and BuildStream's equivalent now? 


Or, one could run a process on the machine hosting the ostree every now and again, to run buildstream to `bst 
pull` the artifacts you want and then checkout and create commits to your ostree repo locally.
This would mean that I have to develop and maintain the release tooling on a remote server rather than on my laptop, and would be less reusable than getting `ostree-push` more up to scratch. 


Provenance
----------


...

BuildStream has all this info available too, but it lacks the ability to
dump it as JSON. I propose that we add a `bst show --mode=json` command
that allows us to dump the cache key inputs into the sysroot before we
push it to the releases repo. So our release process would just gain a
line like:

    bst show --mode=json bootstrap/stage3-sysroot.bst \
        > stage3-sysroot/manifest.json

Does this seem sensible?


Not so much, why would we do this in JSON when we use YAML everywhere else ? I dont think there is a lack of 
YAML parsers out there.
I tend to think "YAML for humans, JSON for machines" and this I'm thinking this as a data dump rather than a nice human readable output (which is something `bst show` already supports). It matters to me very little though -- what's important to me is having a "machine-parsable output" mode. 


There are some things you can already extract with bst show in YAML format, and i'd be happy to look at 
expanding that to cover what you consider is needed.
It seems simplest to just dump the entire unique key for each element in the pipeline into a YAML document. I don't want to try and predict in advance which subset of that info will useful; and it's easy enough to filter the data later on using something like `jq` (or some yaml equivalant) 


Other than that, we are explore a possibility (not sure yet of how/when) of GPG signing the artifacts, 
allowing real assertion of provenance beyond bits of encoded information which you assume comes from a 
trusted source.
We already restrict who can push binaries to the releases repo so it should be a trusted source already. Having people sign the releases with their personal GPG keys or a shared GPG may be useful in future. 

Sam

--
Sam Thursfield, Codethink Ltd.
Office telephone: +44 161 236 5575
[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]