Re: [BuildStream] plugin as elements

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi Benjamin,

Thanks for writing this up :)

Of course I support this feature, I just have some questions and
possible semantic variance I would suggest.

On Fri, 2020-04-24 at 14:48 +0000, Benjamin Schubert wrote:
> Hey everyone
[...]
> The plan would be to add a new plugin `origin:`, which would be `bst`.

I think it would be more explicit and fit better into our terminology
if we called this origin "junction".

> It would take one parameters:
>
> element: which would be the name of the buildstream element containing
> the plugin
>
> The element providing plugins would be expected to have a layout similar to
> the "local" plugin.

This part confuses me, I really don't understand why this loading
mechanism would require an element.

I would see this working in such a way that we simply redirect the
operation of loading plugins to a subproject Project instance.

In this way, we offer just a thin window into a subproject and allow
consuming the plugins it has declared in it's own project.conf
verbatim.

It would impact the loading process such that subprojects referenced by
the project.conf "plugin origins" would be loaded first, so as to
ensure that plugins from that subproject are already available while
loading other elements in the parent project.

An example of what this would look like in project.conf:

  # A statement that we want 'git' source from the project
  # loaded by 'my_junction.bst'
  plugins:
  - origin: junction
    junction: my_junction.bst
    sources:
      git: 0

The my_junction.bst would of course remain unaffected, regardless of
whether I *also* depend on elements through that junction or not.

Is there something that I am missing here which would require something
more complex like unpacking an element from a subproject just to obtain
plugins from that subproject ?


Something which has so far not been discussed is accessibility of
plugins we bless and maintain under BuildStream governance.

I would certainly want to support obtaining blessed plugins using both
the `junction` and `pip` plugin origins, giving users the choice as to
how they want to obtain them.

This would essentially mean keeping a project.conf in the root of the
plugin git repositories which we also use to distribute as `pip`
packages (possibly having tarballs of these published somewhere so that
we can easily bootstrap)... I don't expect this to be difficult to
support, though.

Cheers,
    -Tristan

PS:

Another thing which might be relevant (which applies to all origins
equally I think), is that I think it would be interesting one day to
extend the plugin origins in project.conf to support "aliases", which
would allow statements such as:

  "Load the 'foo' plugin from the 'bar' pip package, and call it 'baz' locally"

The rationale here is we might one day need two plugins which happen to
have a name clash in the same project... I don't think we need this now
but I'd like to keep the doors open to that.