Re: [BuildStream] A Bazel Plugin for BuildStream?

[Thomas Coldrick <thomas coldrick codethink co uk>]

Hello again all,

Since the last mail in this thread I have made some pretty good 
progress, to the
point that I think I can upstream a working pair of bazel plugins, that 
are good
enough for use.

In the last mail I had a build element that had network access, and a source
transform plugin that was failing to combat the symlink hydra produced 
by bazel
fetching external dependencies, and not even all of the ones necessary 
to build
at that.

Now I have removed network access from the build element and created a 
*working*
source transform plugin to fetch the external bazel dependencies. The build
element is essentially the same, only with a more restricted sandbox. The
interesting changes are in the source transform.

Thanks to bschubert on IRC, I was directed to the
`--experimental_repository_resolved_file` option to `bazel query` and it's
derivatives. While this did not simplify the actual sources fetched by 
bazel, it
does produce an easy-to-parse "manifest" of sources (along with some 
bazel internal
stuff that doesn't really matter for our use case). Using this I managed to
download all of the external dependencies required for a target, and add 
them to the
build using the `--distdir` bazel option. With this I successfully built 
bazel inside
BuildStream master, using bazel.

As the source essentially just parses a file in the correct format, 
there is actually
no need to have bazel installed on the host to use the plugin, so long 
as the file is
in place. At the moment the plugin can be configured to fall back to 
host bazel,
should a suitable file not be found, but you can just put one in place 
via a `local`
or `patch` source. This has the added benefit of being more declarative.

You can see the development here [0], with the updated version of the 
plugins in the
only (currently) open WIP MR [1]. After some cleanup I will upstream 
these plugins
to bst-plugins-experimental.

Cheers,

Tom