Recursive pipelines and subprojects

[Jürg Billeter <j bitron ch>]

Hi all,

I've been working on the recursive pipeline feature to support
BuildStream projects that depend on other BuildStream projects in a
practical and convenient manner. There has been some discussion on IRC
but not yet on the mailing list.

The simplest way to combine two BuildStream projects, conceptually, is
to have one project import the output of a single (stack) element from
another project. This is already possible with BuildStream as is,
however, not in a convenient integrated way.

Some scenarios require deeper integration across projects. You may want
to depend on multiple elements of the other project with proper
handling of indirect dependencies.

# Version and option handling for subprojects

For typical cases you want to use a single version and a single set of
options for a subproject. This means that subprojects need to be
explicitly defined with the desired version and options, not just as
part of an element dependency. For this I propose to introduce a new
kind of elements: a junction element.

Junction elements are somewhat special, not all methods of regular
elements are applicable. The source of a junction element is a
BuildStream project and `bst fetch` and `bst track` can be used as with
regular elements. However, a junction element refers to a whole project
and thus, does not produce an artifact. This means that it cannot be
staged or built.

Options are generally project-specific and implicit inheritance across
projects would likely cause problems. I thus propose to always require
junctions to explicitly specify options for subprojects, using an
`options` dict in the junction element config. Options from the top-
level projects can still influence subproject options but the mapping
will be explicit.

A somewhat cumbersome aspect is that subprojects may define junctions
on their own and multiple of those may point to the same project.
However, in general the version and options of these indirect junctions
are not guaranteed to match. I propose possible conflicts of indirect
subprojects to be resolved using a junction in the top-level project.
Source and options will be taken from that top-level junction and
conflicting junctions in subprojects will be ignored.

I'm still not sure about the best approach to match junctions from the
top-level project to junctions in subprojects. Simple matching based on
the name of the junction element is possible. I would like to avoid
matching based on the source URL. I tend towards making this explicit
as well by an additional dict in the junction element config mapping
names of junction elements to be overridden. However, I'm open to
suggestions.

# Implementation

I have a prototype that implements this feature. It's not ready yet for
a merge request but hopefully soon. Despite the feature name being
'recursive pipelines' my implementation uses a single pipeline object
and the support for junctions is in the loader.

As all elements are loaded before the pipeline is created, the loader
also has to fetch junction sources without the use of the pipeline.
This is not ideal but not expected to be a significant issue in
practice. Implicit fetching will be disabled for commands such as `bst
show` to ensure there is no slowdown and no side effects for info
commands.

Any comments or questions on this initial overview?

Jürg