Re: [BuildStream] plugin as elements

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi Benjamin,

On Mon, 2020-04-27 at 08:33 +0000, Benjamin Schubert wrote:
> Hey,
[...]
> The idea of having a `bst element` for a plugin brings a few advantages to the table:
>
> - You can import your plugin from wherever you want instead of having it 'local'
> - Possibly more explicit 'patching' mechanism, since you can patch directly the plugin
>   instead of having to patch the junction itself
> - Plugins authors don't need to maintain an additional project.conf if they don't
>   want to (this would still be doable and would still work)
>
> But is also more complex than specifying "use the plugin from this junction".
>
> If this also fit the bill I am fine with implementing it like that.

Right, I'm worried that involving an element in the operation of this
makes cross junction plugin usage undesirably complex.

As a user, I would very much like to see this simply as:

   "You can load this plugin verbatim from another project"

And not have any elements to declare on either side of the junction.

I also expect it to be a fairly common practice to derive plugins from
junctions to projects where you also depend on elements from (as
domains of junctioned projects are intertwined, so are the plugins they
use). It would be quite redundant to have to also define an element in
this case.


That said I can see where you are coming from that:

  (A) You don't need to have a project.conf in the upstream BuildStream
      plugin sets for these to be usable (but I think this is a rather
      trivial requirement to satisfy).

  (B) You can use plugins from anywhere without requiring any upstream
      support there.

I do however think the overall expense to users is greater with your
proposal.

I would much prefer to prioritize on making the life of plugin
consumers as easy and simple as possible, even at the nominal expense
of providing a project.conf on the provider side (there are always more
consumers than providers, so shifting the work to where there is less
to be done makes better sense).

> > I would see this working in such a way that we simply redirect the
> > operation of loading plugins to a subproject Project instance.
>
> I don't think it is that simple, if the plugin has defaults that are overriden
> by the project, then it will be resolved in the subproject's scope.
> Is that what we really want?

Very interesting and insightful point.

First I think this is orthogonal to whether we involve an element into
this or not: for elements with default YAML and project.conf we can
still make this choice, however you are right that we would need to
load it a bit differently.

I think that the safest thing to do here is:

  * Load the plugin in the context of the project from whence it came

  * Have the local project.conf additionally apply it's overrides in
    advance of local element.bst YAML having the final say.

  * Whether the project.conf overrides in the originating project
    should be observed here is questionable, I would think we shouldnt
    observe overrides from the originating project.

My rationale for the above is that on the one hand, we would want to
use defaults laid out from the project where the plugin will be used,
this does seem desirable.

After realizing that we will have similar obstacles for pip distributed
plugins with YAML defaults ("Where do the variables come from ?") I
decided to delete the following three omitted paragraphs from this
email.

I think this is an important topic and that whatever we decide, it will
have little impact on how cross junction loading will work.

> > [...]
> >     Something which has so far not been discussed is accessibility of
> >     plugins we bless and maintain under BuildStream governance.
> >
> >     I would certainly want to support obtaining blessed plugins using both
> >     the `junction` and `pip` plugin origins, giving users the choice as to
> >     how they want to obtain them.
>
> I think we can discuss this separately?

Well, we can but it is sort of centric to the reason I've been asking
for this feature; it would be a shame if the most important plugins
were unusable in this way.

[...]
> >     Another thing which might be relevant (which applies to all origins
> >     equally I think), is that I think it would be interesting one day to
> >     extend the plugin origins in project.conf to support "aliases", which
> >     would allow statements such as:
> >
> >     "Load the 'foo' plugin from the 'bar' pip package, and call it 'baz' locally"
> >
> >     The rationale here is we might one day need two plugins which happen to
> >     have a name clash in the same project... I don't think we need this now
> >     but I'd like to keep the doors open to that.
>
> As I understand it it would indeed be needed for everything. Let's make a new thread
> about it?

Yes this is definitely off topic and not a pressing matter to me (I
think that we can easily implement this in a backwards compatible way
too, so I don't think it needs to be discussed before it becomes really
needed).

I mostly raised this out of confusion around the element based
proposal, and thought this might provide a useful alternative for what
you may have wanted to achieve by involving elements in this process.

Cheers,
    -Tristan