Re: [BuildStream] plugin as elements

[Benjamin Schubert <contact benschubert me>]

Hey Tristan, Hey all

Thanks for the detailed reply.

I've got some comments inline, but here is a summary and the implementation plan I think is our agreement.
If that is fine with everyone, I'll create issues about it so we can start implementing all the parts.

Please let me know if something required might not be possible with this solution.

- We can reference plugins with a 'junction' origin. The format is as follow:

```
- origin: junction
  junction: potato-junction.bst
  elements:
    - starch
  sources:
    - shop
```

Plugins referenced like that are loaded in the junction's context. However, all their defaults
are provided by the project that uses them, as is currently the case for the `pip` and `local`
sources. If a project wants to share defaults, they can provide an include file.

Does that seem good to everyone?


‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Monday, April 27, 2020 10:38 AM, Tristan Van Berkom <tristan vanberkom codethink co uk> wrote:

[...]
> Right, I'm worried that involving an element in the operation of this
> makes cross junction plugin usage undesirably complex.
>
> As a user, I would very much like to see this simply as:
>
> "You can load this plugin verbatim from another project"
>
> And not have any elements to declare on either side of the junction.

Good point, after more discussions I believe we should keep it simple and the plan you are proposing
is easier and should satisfy every requirement.

> I would much prefer to prioritize on making the life of plugin
> consumers as easy and simple as possible, even at the nominal expense
> of providing a project.conf on the provider side (there are always more
> consumers than providers, so shifting the work to where there is less
> to be done makes better sense).

Fair point. My concern was if a plugin author didn't want to provide a project.conf. However,
we can compose sources in order to inject one, so the concern is moot.

> > > I would see this working in such a way that we simply redirect the
> > > operation of loading plugins to a subproject Project instance.
> >
> > I don't think it is that simple, if the plugin has defaults that are overriden
> > by the project, then it will be resolved in the subproject's scope.
> > Is that what we really want?
>
> Very interesting and insightful point.
>
> First I think this is orthogonal to whether we involve an element into
> this or not: for elements with default YAML and project.conf we can
> still make this choice, however you are right that we would need to
> load it a bit differently.
>
> I think that the safest thing to do here is:
>
> -   Load the plugin in the context of the project from whence it came
> -   Have the local project.conf additionally apply it's overrides in
>     advance of local element.bst YAML having the final say.
>
> -   Whether the project.conf overrides in the originating project
>     should be observed here is questionable, I would think we shouldnt
>     observe overrides from the originating project.
>
>     My rationale for the above is that on the one hand, we would want to
>     use defaults laid out from the project where the plugin will be used,
>     this does seem desirable.

I am not sure about this one. However, as discussed on irc [0], we can delegate
the "defaults" to an include that the junction would provide, and thus only respect
the local project.conf. This should solve the problem.

> > > [...]
> > > Something which has so far not been discussed is accessibility of
> > > plugins we bless and maintain under BuildStream governance.
> > >
> > >     I would certainly want to support obtaining blessed plugins using both
> > >     the `junction` and `pip` plugin origins, giving users the choice as to
> > >     how they want to obtain them.
> >
> > I think we can discuss this separately?
>
> Well, we can but it is sort of centric to the reason I've been asking
> for this feature; it would be a shame if the most important plugins
> were unusable in this way.

Fair point, however I don't see any opposition in supporting this in BuildStream blessed
plugins if we agree on the functionality?


Cheers,
Ben

[0]: https://irclogs.baserock.org/buildstream/%23buildstream.2020-05-04.log.html#t2020-05-04T09:29:03