Re: [BuildStream] Specifying default dependencies for element kinds

[Chandan Singh <chandan chandansingh net>]

Hi Tristan,

Thanks for getting back. A few responses inline:

On Tue, Jun 11, 2019 at 7:39 AM Tristan Van Berkom
<tristan vanberkom codethink co uk> wrote:
> On Fri, 2019-06-07 at 19:49 +0100, Chandan Singh wrote:
> > Hi,
> >
> > Some of us have been recently discussing about how to handle default
> > dependencies for certain element kinds in BuildStream, and I thought this
> > might be good topic for the mailing list.
>
> Thanks for raising this Chandan,
>
> I think that this is essentially:
>
>    "It could be nice to infer additional YAML configurations from the plugin"

I think this captures the general sentiment reasonably well, however I
would just like to add a disclaimer that I'm not suggesting that
BuildStream should try to do this with some sort of magic :) I still
want such things to be defined declaratively (same as other things in
BuildStream).

> And I wonder if this extends beyond only the build dependencies on the
> build systems required.

> From my side, I did not have anything else specific in mind, at least
for now :) But, as you say, similar ideas might apply to other things
as well.

<snip>

> In general, I think that I favor inferring more YAML configuration from
> the plugin, and I wonder if ideally that should be deep down in the
> YAML layer somehow - i.e. not a feature tied to such a specific
> construct as "dependencies", but perhaps a hook to allow more
> participation in YAML composition ?

I agree that it should be composition, more on that below.

> That said, one of the obvious problems with this as a feature in
> BuildStream is that BuildStream cannot have intimate knowledge of the
> project that it is building - i.e. BuildStream cannot itself know how
> you decide to provide the tooling required for autotools to work in the
> sandbox - so - the project would have to be the one to somehow make a
> global statement that: "autotools elements depend on autotool.bst"

Completely agree. It should definitely to be declarative.

<snip>

> Interesting, I am quite liking this approach... and I wonder, does this
> not already work all by itself ?
>
> Or are the plugin specific YAML overrides for elements limited to the
> "config", "variables" and "environment" sections ?

This does not work today, as we currently do not support composition
of dependencies. Composition isn't limited to `config` and `variables`
either, we apply it other fields as well, like public data,
environment variables, and even sandbox config. In fact, the only
element fields that do not support composition are: `kind`, `sources`
and dependencies. I am not sure how useful composition will be for
kind and sources as they are, more often than not, different for
different elements. So, dependencies are really the only fields
missing out on composition.

If we just add composition support for dependencies, that would
already solve most of our issues. It would make me extra happy if we
can make the default composition behavior configurable, but that can
be a separate discussion. (The use-case there being that, at least in
case of dependencies, I expect that we'd want to extend the list
majority of the time rather than overwriting it.)

> I.e. I would want it to be done somewhere like here:
>
>
> It also works well with downstream projects, for instance consider:
>
>   o The base project which bootstraps gcc and builds the build systems
>     from the ground up, offers an include file which extends the build
>     elements.
>
>   o Downstream projects which junction this "root" project include
>     those extensions to the build element configurations in their
>     project.conf
>
> This way the information can be maintained in one place and need not be
> repeated in an error prone way across multiple junctioning projects.

Unless I have missed some detail I don't see why such composition
couldn't support this behavior, as other fields already support this.
So, we should be coverted here.

> Cheers,
>     -Tristan

Thanks!

Chandan