Re: Proposal for Remote Execution

[Jim MacArthur <jim macarthur codethink co uk>]

On 25/04/18 12:54, Jürg Billeter wrote:
> Proposed steps with additional details:
> > > * Create abstract Directory class.
> > > * Implement regular OS file system backend.
> > > * Add get_virtual_directory() to Sandbox class returning a Directory object,
> > >     still using regular OS file system backend for now.
> > > * Add boolean class variable BST_VIRTUAL_DIRECTORY for Element plugins to
> > >     indicate that they only use get_virtual_directory() instead of
> > >     get_directory(). Add error to Sandbox class if get_directory() is used
> > >     even though BST_VIRTUAL_DIRECTORY is set.
> > > * Port users of Sandbox.get_directory() to get_virtual_directory():
> > >     - Element class
> > >     - ScriptElement class
> > >     - Compose plugin
> > >     - Import plugin
> > >     - Stack plugin
> > How do you think we should put sources into this virtual file system
> > before the FUSE is implemented? Source plugins such as `tar` require a
> > directory to pass to the Python tarfile module, so we can either expose
> > the underlying file system directory to the plugin (not ideal) or use
> > tarfile to extract to a temporary directory and then copy/link it into
> > the virtual file system, which might be a performance penalty. Any thoughts?
> My plan was to have the source plugin use a temporary directory and
> then import the contents of that directory (in the future) into CAS
> during the fetch job. With this there should be no additional overhead
> during the build job. And later FUSE will reduce staging cost for the
> build job.
>
> I don't expect source plugins themselves to require any changes, this
> should be handled by BuildStream core code.
>
> Jürg
Indeed, _stage_sources_at handles this temporary directory nicely. 
However, we won't be able to run any commands in the sandbox, and this 
includes integration commands used to stage dependencies, so I don't 
think we will be able to do much with them before the FUSE is 
implemented - unless you were planning to extract to a temporary 
directory to run each sandbox command, which seems impractical.

Jim