Re: [BuildStream] Execution environments

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi Jürg,

On Sat, 2018-11-17 at 14:06 +0100, Jürg Billeter wrote:
> On Thu, 2018-11-01 at 20:28 +0100, Jürg Billeter wrote:
> > Cache key: We already include `os` and `arch` in the cache key
> > calculation, currently using the value from the host. However, as the
> > existing `arch` is based on `uname -m` instead of the proposed list,
> > cache keys will still change.
> >
> > Open issue: The current `arch` project option type is documented and
> > implemented using `uname -m` as default. However, as described above,
> > `uname -m` varies across systems and is thus not suitable. Any
> > suggestions how to handle this with minimal breakage of existing
> > projects?
>
> A possibility to solve this issue is to use the proposed OS-independent 
> arch names for `arch` project options only if the project declares the
> new format-version.
>
> We don't currently use this approach (change behavior based on project
> format-version), however, it might provide a reasonable backward
> compatibility story, as long as we don't overuse it. This approach
> might also be useful when/if we introduce cache key stability.

I quite dislike this, it is a slippery slope towards being more lenient
about breaking changes.

Personally I would rather just document this breaking change on our
wall of shame and live with it, than open the door to BuildStream
behaving differently depending on minimal version requirements.

I see artifact versioning as something completely separate to format
versioning, while it is true that when introducing artifact key
stability, we will need explicit versioning to be mentioned in
project.conf I think this is an artifact version and not a format
version, even if it is true that some BuildStream features (format
related or otherwise) will not be available in older supported artifact
versions.

> Related to this, would it also make sense to report an error when a
> project uses a feature that wasn't available in the format version
> declared in project.conf?

I would very much support this.

>  This would force projects to declare the
> right format version and provide a better error message for users of
> old BuildStream versions. (We should not retroactively do this for past
> format versions, but we could do it for future format version bumps).
>
> On the topic of project options and execution environments, I think it
> would also make sense to introduce a new project option type `os` that
> works similar to the `arch` option type. This would allow multi-
> platform projects to default to the host OS instead of requiring their
> users to explicitly specify the corresponding project option.

Agree.

Cheers,
    -Tristan