Re: [BuildStream] Execution environments

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

On Thu, 2018-11-01 at 23:21 +0000, Sander Striker via BuildStream-list wrote:
> Hi Jürg,

[...]
> This requires a cache key version bump doesn't it?  If we are only assuming
> backward compatibility in terms of consuming from existing caches, we
> would generate the cache key both with the old and the new version.  We
> check if an artifact exists under the new key, if not we check under the old key.
> If not present we build the artifact.  We then store the artifact under the new
> version key.

We currently do not make guarantees about the cache key stability
beyond the promise that cache keys will not change from one major
release to the next (e.g. they have not changed in 1.2, and they will
not change for the duration of 1.4).

There is a sensible plan for supporting cache key stability, however it
requires that we revision the artifact version in such as way as to
cope with artifacts which are formatted differently.

i.e.:

* When implementing features, we necessarily change how an artifact is
  stored (most notably the metadata changes, but introduction of build
  trees is another tangible example).

  BuildStream would have to continue to understand every previous
  artifact version, from the point of implementing this stability
  onwards.

* Projects would need to inform BuildStream which artifact version they
  intend to use.

* BuildStream would also have to lock down some new features to only be
  available since a given artifact version, and error out gracefully.

While this is very much a long term goal for us, I have been advising
against implementing this in the short term due to the immense amount
of churn and feature additions we've been seeing in the last year.

I would advise to wait until the dust settles a bit more, when
BuildStream is less of a fast moving target, we can implement this with
much less overhead.

Cheers,
    -Tristan