Re: [BuildStream] Allowing duplicate junctions [Was: Be explicit when overriding junction configuration, or else warn/error]

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi,

On Fri, 2020-06-05 at 09:49 +0200, Jürg Billeter wrote:
> Hi,
>
> Thanks for moving this discussion forward and writing this up. The new
> plan seems sensible to me overall. A few points/questions inline.

Thanks for going through this :)

> On Thu, 2020-06-04 at 19:27 +0900, Tristan Van Berkom wrote:
> > [...]
> >
> > Allow duplication of projects
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > [...]
> >
> > In addition to the above, we will allow this toplevel project to
> > "duplicate" one the conflicting junctions so that it is allowed to
> > coexist.
> >
> > This will be configured in project.conf right beside the overrides:
> >
> >   # Junction specific configurations here
> >   #
> >   junctions:
> >
> >     # Specify junctions which are allowed to be loaded in parallel,
> >     # even though they were found to conflict with other subprojects
> >     # depending on the same projects which these refer to.
> >     #
> >     duplicate:
> >     - local-junction.bst
> >     - subproject.bst:sub-subproject.bst
> >
> > This whitelisting allows the toplevel project to make the judgment call
> > that it is perfectly okay for the same project to be used twice, if for
> > example they serve different purposes and there will not be any risk of
> > separate instances of the same element being staged at the same
> > location in a given build sandbox (which would cause overlap errors in
> > the end).
>
> I assume only one of the duplicate junctions has to be listed. E.g., if
> two non-internal and non-overridden junctions refer to the same
> project, either one or both have to be listed as duplicates. If three
> junctions to the same project exist, either two or all three have to be
> listed as duplicates. Is this correct?

You are touching upon a flaw I've been thinking of since last night.

It should not be possible for a project to mark *both* conflicting
projects as "duplicate", but only the ability to mark one project as a
duplicate of the other (or possibly multiple projects as a duplicate of
a single one).

Perhaps this can be addressed by causing an error to occur if ever
duplicates are listed for all instances of a project, another approach
might be to change this semantic to use a dictionary instead of a list
(denoting which subproject junction is a duplicate of which, allowing a
bit more validation to catch this before hand).

> Another aspect that is not completely clear to me is how/whether
> duplicate declarations in a subproject affect the top-level project.
> Are they inherited/merged, ignored or something else?

The duplication is inherited, so reverse dependency projects (higher
levels in a project graph) do not have to redundantly redeclare these
duplicates.

However, if reverse dependency projects introduce *new* dependencies on
common projects, they must of course receive the appropriate error and
decide what to do (override, link or declare internal or duplicate).

> If they are inherited/merged (my preference), does this mean the
> behavior would be identical to marking the junction as `internal` as
> long as we don't have any additional dependency checks for internal
> junctions? Or is there a difference I'm not considering?

Essentially it has the same effect of avoiding the error message, and
is also accomplished simply by making a given project name unique in a
global table used while loading projects.

The only substantial difference here is that with duplicates; it is
always a depending project which "decides to accept the second instance
of the same project", whereas with internal projects, the project is
preemptively duplicated in a sense, ensuring it never causes a conflict
with any reverse dependency project.

> If they are ignored, the top-level project will often have to copy
> these declarations, which seems inconvenient and would be contrary to
> the third objective, especially for simple projects junctioning complex
> subprojects.
>
> Regarding naming, I don't think `duplicate` is ideal, especially given
> that the subprojects may not be exact duplicates (different version or
> configuration). However, as you've mentioned, naming is hard.

Yeah, for now I think I'm unblocked to "write code" but lets try to
come up with better words over the weekend before landing all of this.

> > [...]
> >
> > NOTE: The reason we should not place this configuration in the
> > junction
> >       itself, is because junctions can be overridden as previously
> >       described, or they can be linked[4], so it is important to tie
> >       this information to the project where the active junction
> >       configuration resides, and not in the junction declaration
> >       itself.
> >
> >       This information pertains to the relationship of a project and
> >       it's junction, and not the declaration of the junction itself.
>
> I think this makes sense for `duplicate` and `internal`, however, I'm
> not sure about overrides. If you override `subproject.bst:foo.bst`,
> shouldn't this also replace any possible overrides which the subproject
> configured for its `foo.bst` junction? I.e., I consider `overrides` a
> modification or configuration of the junction/subproject itself, not a
> relationship between a project and its junction.

Aha, this is an interesting observation.

I agree: If I override a junction in a subproject, I should be
overriding any expectations which that subproject has about it's
junction (sub-subproject), including any overrides.

Without this, there is no way to unset/disregard the overrides which a
subproject has made on it's subprojects.

It may seem a bit weird to allow full paths for overrides in a
junction, but those will simply have to be project-relative full paths,
so as to support overriding of a sub-sub-sub-sub-subproject junction.

Cheers,
    -Tristan