[BuildStream] Proposal: Be explicit when overriding junction configuration, or else warn/error

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi all,

I'm posting this proposal as a reply to a previous proposal implemented
by Chandan, as the thread in question holds a lot of relevant context
here.

In the interest of moving forward, I will present a fully specified
proposal which I (currently) think is a decent API for adding the
safeguards I want in place.

Take this with a grain of salt, there may be other ways to implement
this which are better, if you have a different preference then I humbly
request that you propose a (fairly specified) alternative that we can
discuss.


Background
~~~~~~~~~~
Chandan's proposal I'm replying to has resulted in the addition of
"Junction targets"[0], which essentially allow one to specify a
junction from a subproject to inherit configuration from (orthogonal
nitpick, I wonder if we should rename 'target' to 'inherit' here).

This is great for easing the work of synchronizing the configuration of
common junctions where projects depend on common projects.

Regardless of whether you use junction inheritance (or targetting);
defining a junction in a project higher up in the project ancestry
causes that junction to override the configuration of junctions to the
same project in the dependency graph; which is also great for deciding
the ultimate configuration in situations where multiple configurations
might conflict.


Problem statement
~~~~~~~~~~~~~~~~~
When you define a junction from your project, you are implicitly
overriding the junction configuration for any and all junctions to the
same project in the project hierarchy.

Consider the following scenario:

  * You create project 'foo' which junctions project 'bar' and project
    'baz'

                         foo
                        /   \
                      bar   baz

  * Naturally you expect that the artifacts produced by your
    subprojects are exactly the same in your project as they would be,
    had you built these subprojects directly, given the source
    configuration and project options you defined.

  * You regularly `bst track` your junctions to get the latest stable
    releases from these projects and rebuild your artifacts on these.

  * One day project 'bar' decides to also depend on project 'baz'

                         foo
                        /  |
                      bar  |
                        \  |
                         baz

    Project 'bar' now has the reasonable expectation that project 'baz'
    is configured how 'bar' configured it, and your toplevel 'foo'
    still has the same reasonable expectations.

This scenario leads to a situation where project 'foo' has overwritten
the configuration of project 'bar's junction to project 'baz'
implicitly, breaking the expectation that 'bar's artifacts are exactly
the same as they would be, had you built project 'bar' directly.

Further, there could even be a confusing build break, if project 'bar'
is not compatible with project 'foo's configuration of 'baz'.

I know this is confusing, I hope I was able to explain it adequately,
it is also discussed previously in this thread here[1].


Proposed warning/failure
~~~~~~~~~~~~~~~~~~~~~~~~
I basically propose that junctions themselves not implicitly override
all junctions in the project implicitly, but that they be explicit
about which junction configurations they override.

If a junction to a common project arises over time which is unaccounted
for, then we get an error (or at least a warning).


Proposed Implementation
~~~~~~~~~~~~~~~~~~~~~~~
After discussing this on IRC, I am still tending towards my initial
proposal of adding an 'overrides' list type option to the junction
configuration.

This overrides list would specifically address the junctions in
subprojects which it overrides, if there are other occurrences of the
same project which are not explicitly overridden, then an error (or
configurable warning) is raised.

Here is an example of project 'foo's declaration of the 'baz' junction,
if it were to explicitly override 'bar's junction to 'baz':

    kind: junction
    sources:
    - kind: git
      url: ...
      ref: ...

    config:
      overrides:
      - bar.bst:baz.bst

Note that this would work equally well if we were to inherit the
configuration of a junction and explicitly apply it to another
junction.


Possible complimentary changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  - Deep conflicting junctions

    As far as I recall, we don't have a way to recursively address a
    deeply nested junction, I don't think we are allowed to specify a
    junction as:

      foo.bst:bar.bst:baz.bst

    I think that this capability needs to be added separately, so that
    project authors do not need to fork projects they depend on just
    because they are getting warnings/errors about deeply nested
    junctions.

  - Coalesce junctions by project names

    We are still coalescing projects by their junction names, Jürg
    pointed out on IRC that he wished we could stop doing this.

    This is probably the most confusing part of the junction API, and I
    think we should really be using the project name itself rather than
    the name of the element which junctions the project.

    This may be hard to achieve in the current load process but is
    probably ideal, and again I think needs to be set aside as a
    separate issue to the one I'm trying to address in this proposal.


Cheers,
    -Tristan

[0]: https://docs.buildstream.build/master/elements/junction.html#targeting-other-junctions
[1]: https://mail.gnome.org/archives/buildstream-list/2019-April/msg00019.html


PS: Why is this important to me
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[this adds some context as to why I am proposing this specific route]

In the original design, "junctions" are BuildStream's response to the
yocto "layering" concept. With junctions we clearly trade away the
powerful flexibility which yocto offers, and in return we eliminate the
horrible pain we experience when trying to continuously upgrade various
layers in a complicated yocto build.

The vision here is that we have many projects maintained by separate
organizations, and that we have the ability to integrate these "stacks"
into various projects, we can clearly see in the embedded space that
there is a need for clean and frictionless continuous integration of
projects/stacks into a final product.

> From the point of view of a toplevel product, I want to update my
junctions with `bst source track` on a regular basis and validate and
integrate them, I truly hope to not be interrupted just because a
project grew a new junction which I shouldn't need to know about.