Re: [BuildStream] Introduce a docker sandbox

[Chandan Singh <chandan chandansingh net>]

Hi Thomas,

Welcome to the list :)

> I have been looking into running Docker containers from within Buildstream.  I found this traffic in the 
> list archives:
>
> https://mail.gnome.org/archives/buildstream-list/2018-October/msg00081.html

Two important questions that were asked in that thread were about
where this implementation should live and what contracts it should
have. Since then, BuildStream's sandbox story has evolved a bit more.
At this point, it is safe to say that any new sandboxes should be
implemented as `sandbox-run` interfaces. Currently there are a few
examples of this:

* https://gitlab.com/BuildGrid/buildbox/buildbox-run-bubblewrap
* https://gitlab.com/BuildGrid/buildbox/buildbox-run-userchroot
* https://gitlab.com/BuildGrid/buildbox/buildbox-run-hosttools

So, the docker sandbox could become buildbox-run-docker. The positive
part of relying on the buildbox-run interface is that there are no
changes needed on the BuildStream side at all. buildbox-run-docker
would be a drop-in replacement for any other buildbox-run interface,
so long as it implements the same interface.

> I haven't been able to find much around Docker, other than how to run Buildstream within Docker, and I 
> didn't see if there was follow-up to the above noted discussions.

BuildStream can also interface with Docker in other ways. For example,
the bst-plugins-container project
(https://gitlab.com/BuildStream/bst-plugins-container) provides source
and element plugins to interface with Docker images.

The source plugin allows one to import a Docker image as flat files in
the BuildStream universe. This is most useful to bootstrap a project
using a docker image as the base system. It can also be used in other
ways like importing self-contained binaries etc.

The element plugin allows you to layer BuildStream elements as Docker images.

Using the two plugins together, one could - start from a Docker image,
import it in a BuildStream project, do some transformations using
BuildStream, layer these elements back into a Docker image.

> I am working with some people who have constructed a build system within many Docker containers, and I am 
> looking into whether I can run those Docker containers within Buildstream.  Is this even possible?  I've 
> looked at the Alpine based sandbox, and I built my own sandbox runtime with Ubuntu, but it looks like 
> everything is contained within Bubblewrap, making it seem difficult/unpractical to run Docker containers?  
> I would need to both build Docker containers, and fetch from Artifactory for import into Docker containers 
> and/or run them.

None of the current sandboxes can do this, but in theory it should be
possible to have a sandbox that allows for privileged operations like
running Docker containers.

Having said that, running proper docker containers inside the build
sandbox will lead to nested containers, which can get very complicated
very quickly. So, I'd be wary of going down this route. It will also
hurt repeatability of builds, which is one of the key principles of
BuildStream.

> From what I can tell, this probably isn't a good approach anyways - it seems it would be best to run a 
> standard Make, CMake or autotools based build from within Buildstream, and encapsulate the entire build 
> system, including Buildstream, in a Docker container to allow execution under Windows and Linux.

I tend to agree that it would be a better experience for users if
BuildStream took care of storing build configuration at the level of
CMake, autotools etc.

For platforms where a native sandboxing implementation is not
available, there are a few options to run BuildStream:

1. If possible, you can leverage remote execution
(https://docs.buildstream.build/master/format_project.html#project-remote-execution)
by having a remote build farm of the target platform. BuildStream can
already run on Mac and Windows, as a remote execution client.

2. Implementing buildbox-run-docker would also solve this, since that
could integrate nicely with Docker for Mac/Windows. This would also
allow for local builds, if that's a requirement.

3. Wrapping the whole thing in a Docker container, as you suggested
originally. There is some prior art with this approach. We have a
bst-here script which makes it a one-liner to build a BuildStream
project inside Docker. You can find more details about this at
https://buildstream.build/container_install.html.

Hopefully this has answered some of your questions. I'd be curious to
hear which route you decide to go.

Cheers,
Chandan