Re: [BuildStream] Buildstream shell et al

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

On Mon, 2018-08-13 at 13:21 +0100, Sander Striker via BuildStream-list
wrote:
> Hi,
>
> Are we expecting the user to have a full environment, including their
> favorite debugger included in the shell environment?  Given that that
> means first making all these tools available through .bsts somehow,
> realistically the answer to that is no.  The question then becomes,
> how do we bring together build isolation and user convenience and
> control?

Hi Sander,

Thanks for raising this, indeed it is a tricky and interesting
question.

So far the answer has indeed been to just provide all the tooling that
the team wants within the build pipeline elements; and chop out what
you don't need in the deploy stages either by distinguishing build-vs-
runtime dependency, or using split-rules and compose elements.

This does not solve the issue that developers want what they want, and
might not be even allowed to push their tooling of choice into an
upstream maintained BuildStream project.

> Now, I don't think we can solve this for everything.  If you're
> building systems, and bootstrapping from scratch on exotic or
> uncommon environments, the only path you have may be to have all the
> tools be part of the .bst definitions.
> However, when on more mainstream platforms, there is potentially an
> easier path.
>
> The following is meant as food-for-thought, not a fully baked
> proposal.  I'm going to highlight one use case only:  I am a
> developer and my build has produced a binary that fails in a
> seemingly awkward way.  Now, I am proficient with SwankyDebug, but
> that is not available in bst shell.  Now, I also know from experience
> that once I am using debugger, I am going to make code changes and
> need to rebuild.  How do I get access to both my debugger and my
> build environment?
>
> Consider the following just brainstorming...  
>
> If we can actually generate a Docker container which contains the
> entire staged build environment including mounting of workspaces,
> then the user can take it on themselves to extend the docker
> container to have whatever tools desired.
> Note that there currently is no facility to create your entire build
> environment outside of a bst shell, other than by hand walking the
> dependencies and doing bst checkout.  Workspace builds, that is,
> running bst build on a workspace, need to be good enough to not have
> anyone want to do this in the common case.
>
> A naive implementation could be as simple as a special Sandbox* that
> creates a Dockerfile as such:
>
> FROM scratch
> # Staging
> COPY path/to/hashed/content/in/local/cas /actual/path
> ...
> # Workspaces
> VOLUME /path/to/mount/for/workspace
> ...
>
> And generate the proper commandline invocation for it, so the
> relevant open workspaces would be properly mapped to the volume
> locations.
>
> Now the user can choose to exit the docker container.  And write a
> Dockerfile that extends the one above that pulls in SwankyDebug. 
> Launching that container gives me everything I need.
>
> One _could_ argue that doing a Sandbox implementation that uses OCI
> containers for everything instead of bwrap, would give us local
> builds on all platforms where Docker is available.  And would allow
> the user to leverage familiar tools to achieve their needs.

I am not fond of the idea of supporting other formats as first class
citizens in the BuildStream core APIs. E.g., if we choose to leverage
Docker for sandboxing on some platforms, that should remain an internal
implementation detail.

Plugins of course can import from whatever format they can, into the
abstract sandbox and artifact cache.

I think that something that is probably the inverse of what you have in
mind is more feasible, or seems more agreeable for supporting inside a
proper `bst shell`.

Instead of a codepath which uses docker to add the build data to an
image which contains external tools; consider a codepath which allows
the user to run a `bst shell` with some additional data overlayed into
the sandbox environment.

Following our `bst checkout` pattern, it makes sense to only support:

  * The user's filesystem (limited support, the user's filesystem might
    not support all the expected file attributes in the artifacts).

  * Tar format. Here we provide as much support as we can. When
    checking out data, we are not limited by what the user's filesystem
    can support (or by the user's permissions on their filesystem).

We basically choose tar because it is a pretty universal format which
supports just about anything in terms of filesystem data
representation.

Considering what we already do at checkout time, we could provide
something symmetrical for `bst shell`, perhaps an CLI option or two
added to `bst shell`, like:

  o bst shell --overlay <HOST DIRECTORY> <SANDBOX DIRECTORY>

  o bst shell --overlay-tar <HOST TARBALL> <SANDBOX DIRECTORY>


These could potentially be specified multiple times in one invocation
(like the existing `--mount` option), but probably for the debugging
tool use case you want to specify it only once with the SANDBOX
DIRECTORY being "/".

The functionality would be to create the shell environment in the
regular way, but after staging all the dependencies and running
integration commands, we could additionally stage the user specified
payload to the specified sandbox paths, and then run the shell in the
regular way (the reverse could be possible with say, an "--underlay"
option).

This way we keep our sandboxing techniques opaque, and we also provide
a method for the user to stage whatever they want into the sandbox
environment, without the restrictions of the `--mount` option (i.e.
these are staged files, and can be interleaved with the actual build
data payload).

This could also probably be optimized by caching the user specified
input in the underlying CAS used by the artifact cache (specifically
the CAS because, these would not be proper artifacts, but could be
cached in a similar way).

Thoughts ?

Cheers,
    -Tristan