Re: Proposal for Remote Execution

[Sander Striker <s striker striker nl>]

Hi,

On Wed, Apr 11, 2018 at 10:37 PM Jürg Billeter <j bitron ch> wrote:

[...]

Remote Execution
~~~~~~~~~~~~~~~~
With CAS support in place in the artifact cache, the virtual file system
API, and the FUSE layer, actual remote execution support can be added.
The core part on the client side is to implement a remote execution backend
for the Sandbox class.

As part of the build job, Sandbox.run() will upload missing blobs from local
CAS to remote CAS and submit an action to the Execution service. The output
files will not immediately be downloaded, avoiding unnecessary network
bandwidth, however, the digests will be added to the virtual root directory
of the sandbox as appropriate.

The artifact directory structure (files, meta, logs) will be created locally
using the virtual file system API, and uploaded to the remote CAS as part of
the push job, if push is enabled.

Should assemble still be putting together the artifact directory structure, or do we want to turn the artifact into a proto and store that in the ArtifactCache?  I mentioned this on a separate thread and it's easy to have it be lost: https://mail.gnome.org/archives/buildstream-list/2018-April/msg00038.html

The top level directory then doesn't need to be special cased; we can decide what to download from the ArtifactCache entry rather than the directory node in CAS.  In terms of interface it "feels" cleaner.

Cheers,

Sander

--

Cheers,

Sander