Building bootable VM images as part of the edit/compile/test cycle

[Sam Thursfield <sam thursfield codethink co uk>]

Hi all,

This is going to be quite a long mail, perhaps unnecessarily so but I
want to get a bunch of context written up that I'm not sure everyone
involved in BuildStream is aware of.

It's almost a year since we started publicly proposing that
BuildStream should become GNOME's official integration tool[1]. A key
part of our proposal is that developers will be able to not only build
the latest GNOME and whatever local changes they have but will also be
able to easily spin up a VM to test out the fully integrated result.

There has been a huge amount of work on BuildStream's core since then,
and it is now being used for GNOME releases[2], but nobody has really
been resourcing the VM images work. There has been some progress though
so I will highlight what is done already and what remains to be done.
I am unlikely to spend more time on this, hopefully this mail is useful
for whoever eventually picks up the work!

Baserock VM images
------------------

Early on Tristan demonstrated that we can build bootable x86_64 VM
images in BuildStream without requiring root permissions[3]. This was
based on some existing work in the Baserock definitions project[4].

Out of this came the 'x86image' BuildStream element:
https://buildstream.gitlab.io/bst-external/elements/x86image.html#module-elements.x86image

Freedesktop SDK images
----------------------

Another tentacle of the BuildStream world-domination plan is the
Freedesktop SDK 1.8 project. The main goal there is to replace the
Yocto-based sysroot that forms the base of most Flatpak runtimes
with a sysroot that is built using BuildStream and maintained
independently from Yocto. The current "Freedesktop SDK" base is version
1.6, the new BuildStream based one will be API incompatible and is
version 1.8. A first release is already available for testing[5].

The GNOME and KDE Flatpak SDKs are both built on top of the Freedesktop
SDK and they will need to transition from 1.6 to 1.8 at some point.
GNOME currently builds and releases their Flatpak SDKs using the
flatpak-builder tool from metadata in the gnome-sdk-images Git repo[6].
This build process can happily consume the Freedesktop SDK 1.8 releases
just like the 1.6 releases.

However, our goal with moving GNOME to BuildStream is that we want
everything that GNOME releases to be built from the gnome-build-meta[7]
repo using BuildStream, including the Flatpak SDKs[8]. This means we'll
use a 'junction' element to pull the freedesktop-sdk BuildStream project
into the gnome-build-meta project, and use that as the base sysroot
instead of the current Debian base.

Since this work needs to be done anyway, we had the idea of extending
the Freedesktop SDK 1.8 project so that it can used to build bootable
VM images. These could then be used as a basis for GNOME VM images
and would also work for any other project that wants something similar.

We now have basic support in the Freedesktop SDK project for producing
VM images, currently just a minimal root shell is available but more
work is ongoing[9] to integrate systemd, Weston and a few other bits.

Using this for GNOME
--------------------

I have been really keen to get an initial proof of concept VM image up
and running, so we can get a handle on how much work is remaining to
create a proper developer workflow. As more GNOME developers switch to
using BuildStream this becomes higher and higher priority.

So here's how far I got. I looked at building gnome-build-meta on top
of the freedesktop SDK repo, but there is a lot of stuff missing
compared to the Debian base and I didn't finish. Hopefully the work I
did can be built upon; I pushed the work-in-progress branch to here:
https://gitlab.gnome.org/GNOME/gnome-build-meta/merge_requests/9

As a shortcut, I cobbled together a VM image using the kernel, initramfs
and deployment tools from the freedesktop SDK repo plus the Debian
sysroot used by gnome-build-meta. This work is in branch 
sam/debian-vm-hack of gnome-build-meta.git.

Running GNOME inside the image
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This rough image does actually boot into `/bin/sh`, and with some
hackery can get to a usable console session. I ran these commands:

    exec bash  # get useable line editing
    passwd     # set root password
    exec /lib/systemd/systemd

That allowed me to boot to a console login.

I had a go at running `X` and `gnome-shell --wayland --display-server`,
but the Linux config in the freedesktop-SDK repo is just 'defconfig' so
it lacks any of the drivers that might make DRI work. I would try
enabling `CONFIG_DRM_VIRTIO_GPU` to begin with; see:
https://www.kraxel.org/blog/tag/virtio-gpu/

There are probably more drivers that need enabling, we used to have
more stuff enabled but without any log of why all of it was there, so
it was removed in
https://gitlab.com/freedesktop-sdk/freedesktop-sdk/commit/f64c40ab6a09516ed75fecd86792cfa472749c74. 
Someone will need to go and
add back some of these options but with a comment above each one
explaining why it is needed.

More integration work around components such as GDM and PolicyKit
would be needed to get a really working system.

Build speed
~~~~~~~~~~~

There's about 5GB of data in the VM image when using only the 'runtime'
domain. If we included the 'debug', 'devel' and 'docs' domains it would
be much larger (I'm not sure how much). Build times are a major concern
here as we ultimately want this to be part of the normal GNOME
development cycle.

On my build VM I got the following timings:

  compose element: 07:56
  x86image element: 55:47

Tristan ran the same build and got the following:

  compose element: 3:48
  image element: 24:05

The x86image element currently does a needless copy of all the
data, fixing this issue[10] would bring down the build time
dramatically, probably to under 10 minutes in the best case.
This would be a great step forward compared to many other tools
but still not practical for an edit/compile/test cycle. We will
need to think a bit more on how to make it really fast.

One option would be to get the VM to boot directly from a folder on the
host. There are a few options, none of which are ideal. NFS booting is
one option, but it requires the developer to configure NFS on their
host and can be pretty fragile inside the guest. The '9pfs' filesystem
driver would require no special config on the host, but there might
also be issues inside the guest[11].

The GNOME Continuous build system used to produce images which followed
the "OSTree system layout"[12]. These could be upgraded from any state
to latest version just by running `ostree admin upgrade`. They could
also roll back to older versions.

Enabling this requires a special rootfs layout that would be tricky to
achieve with the Debian base, but using the Freedesktop base it should 
be doable.

We could then have a workflow that looked like this:

  * 1. download a base VM image and boot it
  * 2. share a directory on your local machine with the VM, and
       configure it as an OSTree remote inside the VM
  * 3. `bst build gnome-vm.bst` with your latest changes, then
       `bst checkout` to the shared directory
  * 4. run `ostree admin upgrade` in the VM and reboot.

After the first build, steps 3 and 4 are the developer cycle. We avoid
building an image at all for most builds which should save a lot of
time. There will likely be some more general optimization in
BuildStream needed too, but this is already on the roadmap for this
year.

Tracking the work
-----------------

I suggest we use this issue to track all of the above work, breaking
sub-issues out as needed:

  https://gitlab.gnome.org/GNOME/gnome-build-meta/issues/7

Sam

[1]: 
https://mail.gnome.org/archives/desktop-devel-list/2017-April/msg00071.html
[2]: 
https://mail.gnome.org/archives/desktop-devel-list/2018-January/msg00025.html
[3]: See "A bootable system" in
https://blogs.gnome.org/tvb/2017/04/10/buildstream-progress-and-booting-images/
[4]: 
https://gitlab.com/baserock/definitions/blob/master/elements/systems/minimal-system-image-x86_64.bst
[5]:
https://lists.freedesktop.org/archives/freedesktop-sdk/2018-February/000014.html
[6]: https://gitlab.gnome.org/GNOME/gnome-sdk-images
[7]: https://gitlab.gnome.org/GNOME/gnome-build-meta
[8]: https://gitlab.gnome.org/GNOME/gnome-build-meta/issues/16
[9]: https://gitlab.com/freedesktop-sdk/freedesktop-sdk/issues?label_name=VM
[10]: https://gitlab.com/BuildStream/bst-external/merge_requests/16
[11]: 
https://unix.stackexchange.com/questions/90423/can-virtfs-9p-be-used-as-root-file-system
[12]: https://ostree.readthedocs.io/en/latest/manual/adapting-existing/

--
Sam Thursfield, Codethink Ltd.
Office telephone: +44 161 236 5575