[BuildStream] A Bazel Plugin for BuildStream?

[Thomas Coldrick <thomas coldrick codethink co uk>]

Hi all,

I've recently started work on a bazel plugin for BuildStream, that is a
BuildStream build element-like that builds bazel projects. In this thread I
hope to get some advice and opinions on the implementation, and also 
find out
where the preferred place for development would be.

Implementation Plan
===================

So, currently I have a plugin based roughly on a BuildElement that can build
bazel projects, targeting bst 1.4.1. This plugin is largely a complete and
utter hack, to the extent of allowing the build sandbox access to the 
network.
I'll outline the issues which led to this, and some ideas for how to solve
them.

How did it end up like this?
----------------------------
The main issue was that bazel integrates fetching external dependencies. 
As a
result we need to implement some manner of source transform plugin too. I
created a rough version of such a plugin using the `bazel fetch` 
subcommand as
a quick starter, but it turns out that `bazel fetch foo` does not always 
allow
you to `bazel build foo` while offline. This would have been fine, as 
there's a
way to download all external sources, but bazel *also* creates a bunch of
symlinks to internal tools at fetch time. These could mostly be fixed up 
when I
staged the source, or be overridden using an argument to bazel, but some 
were
just mysteriously missing.

Due to time constraints I gave up on creating such a source transform plugin
temporarily in an attempt to get a working build element.

Bazel also tends to do a lot of work in $HOME, using $HOME/.cache as a 
root for
almost everything. This can be modified using a command line option, but 
also
projects break if we move it to the build-root.

Finally, as a java project bazel relies on /dev/shm, which does not exist in
the bst sandbox (by default at least).

How can we fix it?
------------------
The last problem was a simple fix, I simply added /dev/shm using
Sandbox.mark_directory(), however I get the feeling that this is 
actually bind
mounting the host device into the sandbox. This is not an ideal state of
affairs, in terms of sandboxing. I'm hoping that someone on this list 
knows of
a better way to put /dev/shm in the sandbox without doing this.

Bazel's heavy use of $HOME could be the way to integrate a source 
plugin. When
fetching sources bazel uses $HOME/.cache as the place to put them, in a
directory for the project we're currently building. This might go 
against some
BuildStream design principles, but I think the easiest way out is to 
fetch the
source from inside the sandbox, setting up some $HOME to act as the 
bazel home.
We can then simply stage this into the build sandbox with everything 
downloaded
into the right place and no broken symlinks, as it will be in the exact 
place
it's supposed to be.

Doing this will require us to implement some kind of weird source plugin 
that
does it's thing from inside a sandbox. It will also mean that you will need
a bazel binary inside the sandbox to use the bazel source transform.

As another option, we could use the `distdir` command line option for 
bazel to
provide a location that contains all of the downloaded sources. However it
seems to me that generating a comprehensive list of all of the necessary
external dependencies would be an awkward job, especially if we only want to
find those for a given target. This could be potentially used as an 
alternative
though.

Where should development happen?
================================
I assume that bst-plugins-experimental would be the place for this, but 
given
the immaturity of the plugin it might end up as a very nasty looking MR. If
there are other preferences, please let me know.

Thanks,

Tom