Re: Improving staging performance

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

On Wed, 2017-11-29 at 22:18 +0000, Angelos Evripiotis wrote:
> > I think Sander's idea of stage-on-demand fuse layer might be the most
> > productive so far, but it might take some "actual real work" to get
> > that done, and to get it optimized to a level where the python layer
> > does not add significant enough overhead to cancel out the
> > optimization.
>
> I'm glad you say that, because on a similar note Sander pointed me towards
> sandboxfs. It aims to solve much the same problem for Bazel.
>
> https://blog.bazel.build/2017/08/25/introducing-sandboxfs.html
>
> The idea is that you feed it a list of paths and where they map to in
> the sandbox.
> It provides a fuse layer that you can base your sandbox on.
>
> It's implemented in Go and is currently experimental, I imagine we might take it
> on as an optional dependency.

Sounds like something we might prefer to have our own implementation
of, along with the SafeHardlinks layer and the layer we will inevitably
need for addressing infamous bug #38.

> If I were to put in the effort of making a proof-of-concept patch for
> integrating into
> BuildStream, would you have time to evaluate it?

Of course !

I just can't say when unfortunately - certainly not in 2017.

Cheers,
    -Tristan