Re: Proposal for Remote Execution

[Jürg Billeter <j bitron ch>]

Hi Jim,

On Wed, 2018-04-25 at 12:15 +0100, Jim MacArthur wrote:
> On 11/04/18 21:37, Jürg Billeter wrote:
> > Virtual File System API
> > ~~~~~~~~~~~~~~~~~~~~~~~
> > As a second phase I'm proposing to introduce a virtual file system API that
> > both BuildStream core and element plugins can use in place of path-based
> > file system operations provided by the OS. The goal is to allow transparent
> > and efficient manipulation of Merkle trees.
> >
> > The API consists of an abstract Directory class that supports creating
> > directories, copying/renaming/moving files and whole directory trees, and
> > importing files from a local path. See also the existing `utils` functions
> > `link_files` and `copy_files`, equivalent operations need to be supported.
> >
> > Proposed steps with additional details:
> > * Create abstract Directory class.
> > * Implement regular OS file system backend.
> > * Add get_virtual_directory() to Sandbox class returning a Directory object,
> >    still using regular OS file system backend for now.
> > * Add boolean class variable BST_VIRTUAL_DIRECTORY for Element plugins to
> >    indicate that they only use get_virtual_directory() instead of
> >    get_directory(). Add error to Sandbox class if get_directory() is used
> >    even though BST_VIRTUAL_DIRECTORY is set.
> > * Port users of Sandbox.get_directory() to get_virtual_directory():
> >    - Element class
> >    - ScriptElement class
> >    - Compose plugin
> >    - Import plugin
> >    - Stack plugin
>
> How do you think we should put sources into this virtual file system 
> before the FUSE is implemented? Source plugins such as `tar` require a 
> directory to pass to the Python tarfile module, so we can either expose 
> the underlying file system directory to the plugin (not ideal) or use 
> tarfile to extract to a temporary directory and then copy/link it into 
> the virtual file system, which might be a performance penalty. Any thoughts?

My plan was to have the source plugin use a temporary directory and
then import the contents of that directory (in the future) into CAS
during the fetch job. With this there should be no additional overhead
during the build job. And later FUSE will reduce staging cost for the
build job.

I don't expect source plugins themselves to require any changes, this
should be handled by BuildStream core code.

Jürg