Re: [BuildStream] Allowing duplicate junctions [Was: Be explicit when overriding junction configuration, or else warn/error]

[Jürg Billeter <j bitron ch>]

Hi Tristan,

On Fri, 2020-05-08 at 16:50 +0900, Tristan Van Berkom wrote:
> [...]
>
>     Cross architecture bootstrapping
>     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>     When bootstrapping a runtime for a different architecture, it can
>     be interesting to use the same toolchain project configured
>     multiple times with different project options defining which host
>     and target architectures to build libc/gcc under.
>
>     When combining this ability with remote execution, we can
>     streamline the process of bootstrapping a system under any
>     architecture which we have runners for on the RE cluster.

A possible solution for this use case is to extend the key used for
conflict detection a bit. Instead of only using the project name as
key, we could include the values of (selected) project options as well.
E.g., the target architecture option would be sensible to include, in
my opinion.

>     Auxiliary projects which provide static build-only dependencies
>     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>     When one project depends on another project for some static data
>     which will be consumed as build-only dependencies, the data
>     from the junctioned project is consumed statically as is, and there
>     is no concern of runtime dependencies being propagated forward to
>     reverse dependency projects which might consume the same junction.

This also includes statically linked libraries where no runtime data is
required (or runtime data is in a private prefix).

"Isolated junctions" seems like a sensible solution for this use case.


>     Separation of tooling and data
>     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>     [...]

I think the ideal solution here would be for BuildStream core to know
when multiple dependency trees are configured to be staged in separate
prefixes/sysroots. BuildStream could then report a conflict if and only
if elements from different versions of the same project are used in the
same prefix/sysroot.

This would probably not be a simple change, though, so we don't want to
block the MR on this but maybe pick this up in the future. A question
is also whether we can perform such dependency-level checks early on
without significant performance penalty.

Deferring the check to assemble/staging time would likely be a lot
simpler and less of a performance risk, however, erroring out that late
is not ideal.

The dependency-based approach might work for the static build-only
dependency case as well, even without the need to explicitly mark
junctions as isolated.

> [...]
>
>   Looking at this email so far, I'm tempted to think that we might have
>   both of these approaches (declaring a junction as 'isolated' can
>   allow hiding a local junction and be more convenient, but failing
>   this we can still whitelist junctions in reverse dependencies).

I suspect we may need more time to get to the best solution for each
use case. We don't want to block !1901 on that, though. Could it make
sense to change the error to a fatal warning to unblock !1901,
deferring more advanced configuration/checks?

I.e., allow projects to make the conflict warning/error non-fatal in
project.conf until we've implemented a better solution for each use
case? To be clear, configuring the conflict warning as non-fatal would
cause the relevant junctions/subprojects to be independent (no
automatic coalescing, unlike master). And if there are real file
conflicts, projects would still get overlap warnings/errors.

Cheers,
Jürg