Remote execution update

[Jim MacArthur <jim macarthur codethink co uk>]

Here's a quick update on what me, Finn Ball and Jürg Billeter have been 
doing towards remote execution with BuildStream recently:

* Jürg's CAS FUSE work is in progress, but usable. I've had some 
problems building on Ubuntu, but Finn Ball has been able to build it 
easily.
* CAS-based Virtual directories are considered reasonably stable. This 
work is at 
https://gitlab.com/BuildStream/buildstream/merge_requests/481. It's 
missing one part which will improve performance but shouldn't otherwise 
affect the functionality.
* BuildGrid (our remote execution server and worker) is available at 
https://gitlab.com/BuildGrid/buildgrid. There's a small demo you can do 
written in the README for this which allows you to start a server and 
send messages to a worker.
* BuildStream execution client: We have a version of BuildStream which 
will pack the input root and command into CAS objects and transmit them 
to the artifact server. It will also send Action messages to the 
BuildGrid server and wait for the worker to complete before continuing. 
We've demonstrated this works by manually unpacking the input root and 
running the commands on the server. This is an unstable branch at the 
moment, jmac/source_pushing_experiments. Jim is working to make this 
into a stable version.
* Finn is working on a worker bot, which will need to pull the objects 
from the artifact server to the worker machine, run the build with 
BuildBox and return the results.

Our current focus is to try and make a demo of remote execution work 
before GUADEC. We've got a lot of outstanding code in branches, which 
we've not made too much noise about yet because we wanted to point to a 
clear benefit first, which hopefully the demo will do. As well as the 
demo, we're in need of some written material, such as blog posts as 
Tristan VB has suggested recently.

We've looked at BuildFarm - not in enough detail yet, but it looks 
similar in structure to the work we're doing. The main difference we can 
see is in the sandbox - so far as I understand it, the current 
Bazel-Buildfarm worker doesn't build in a sandbox. Other than that, we 
expect to be very similar. I had a brief try at running the BuildFarm 
worker with BuildGrid server and vice versa, but they didn't work 
straight away.

I'd like to see BuildFarm and BuildGrid interoperate as much as 
possible. Apart from the sandbox, is there anything else fundamental 
between them that would stop them being replaceable components? That's a 
question for the BuildFarm mailing list more than this one, I think, so 
I'll look into that.

--
Codethink privacy policy: https://www.codethink.co.uk/privacy.html