ostree buildsystems, packages and pinning

[Colin Walters <walters verbum org>]

On Mon, Oct 29, 2018, at 1:51 PM, Will Manley wrote:
> We commit a debian package index file[3] as kind of lockfile to our
> source git repository to ensure reproducibility of built ostrees and to
> help manage security updates.

Very cool, thanks for posting this!  It is the kind of discussion I really
like to have here.

> We build ostree images on x86 for deployment to an ARMv7 device. So
> development happens on our developer laptops and it's deployed to the
> device using ostree - even during development. We're not currently using
> ostree admin unlock or the like.

Are you doing this for CI too?  If I were in your case I'd probably try
to support at least doing an x86_64 build too and run through basic sanity tests
in a VM as well.

> * bubblewrap for chroot https://github.com/projectatomic/bubblewrap
> * multistrap to create base images https://wiki.debian.org/Multistrap
> * qemu with binfmt support so we can run ARM binaries (like apt-get
>   install) inside the chroot. https://wiki.debian.org/QemuUserEmulation* 

I need to play with that.

> fakeroot so we don't need to run the build as root but the images will
>   still have the right permissions

See also e.g. https://github.com/projectatomic/rpm-ostree/issues/1011

> To fix this we took a leaf from modern programming language package
> managers.  We use the lockfile concept as used by rust's cargo package
> manager (cargo.lock[1]) or nodejs's npm (package-lock.json[2]).  

This makes a lot of sense. 

Your CI job then is a lot like e.g.:
https://dependabot.com/

Although I think the first time I personally saw the "CI bot updating pinned data"
pattern was in the context of the Cockpit project, which does it for fixed VM images
it uses for testing; a recent example is:
https://github.com/cockpit-project/cockpit/pull/10480

Another thing that's strongly related to this though is that IMO,
classic package metadata (dpkg/rpm) need versioning.  And it'd
probably be nice if e.g. crates.io too had a version number one
could reference in addition to the git sha1.

Using that then one could specify e.g. `apt/yum install --repoversion X.Y.Z`
and also have reproducibility.  The reason I really want this though
is because for rpm-ostree on the client side, one quickly runs
into the fact that ostree has a very nice git-like history model with
clear checksums and versions, and rpm...has no such thing.
https://github.com/projectatomic/rpm-ostree/issues/415

Actually in Fedora today the "pungi" tool does output versioned
directories: https://kojipkgs.fedoraproject.org/compose/updates/
But it's not an API today and nothing in the libdnf ecosystem understands
how to parse it (there's not an index other than the autogenerated
HTML as far as I know, etc.)

But anyways yeah, pinning the packages makes sense.

> I would like to extract each deb immediately after downloading into its
> own ostree 

Yep, rpm-ostree does this, although sadly right now after downloading
all of the packages, it's not interleaved yet.

> This would also save more disk-space: we'd no longer need to
> store the debs themselves, but could refer to the contents by ostree
> ref e.g. the ref dpkg/data/<sha256> might refer to the deb with that
> sha256.  The lockfile has these SHA256s recorded so you'd know which
> ostree refs to use.
> This is of-course a much larger step - you'd still need to handle the
> metadata, pre-inst scripts, etc under control.gz which might be a little
> tricky, but multistrap manages it.

Having a lot of experience with this I can say the benefit and cost
is exactly that: it's a major leap from what apt/yum etc. do today,
with some nice benefits, but one also ends up maintaining a
separate parallel path.  Which so far is definitely worth it I think.

Thanks again for posting this!  If you are able to extract any
of this work and post it, happy to add it to the README.md
as a reference.