[BuildStream] plugin as elements

[Benjamin Schubert <contact benschubert me>]

Hey everyone

This is a proposal to allow treating plugins as BuildStream elements.

I also propose keeping the `tar` source plugin in core.

This was first mentionned in [0], and then in the BST 2 plan

as 'Allow accessing plugins across junctions ?' [1]

Why ?

Some projects have requested to be able to use 'the exact same plugins' as

projects they junctions. Others have requested to be able to import plugins

without installing them as `pip` packages, but still be able to have a better

flexibility than just local plugins.

This would enable:

- Strict versioning of plugins

- Ensuring everyone in a project uses the exact same version

- Sharing plugins across junctions/projects

This would not:

- Remove pip-installable and local plugins

- Affect anyone not interested

- Installing and handling third party dependencies

The plan would be to add a new plugin `origin:`, which would be `bst`.

It would take one parameters:

element: which would be the name of the buildstream element containing

the plugin

The element providing plugins would be expected to have a layout similar to

the "local" plugin.

What it would change in the core:

- This would affect how we load elements, as we may need to load plugins during

load time. This is not a problem as we already have access to the scheduler

in the loader for junctions.

- Loading the plugin would mean extracting its content in a directory through

  casd and importing it from there. We already do such extraction for junctions.

  As for junction, this plugin would not be buildable and we would use the

  sources as the content.

What advantage does it have over current solutions (local and pip):

Over pip: it is per-project, and versionned with the project, so you are sure

that every user uses the correct version

Over local: no need to use submodules or vendor in the plugins, available

cross junction.

An example would be:

project.conf:

plugins:

  - origin: bst

    path: my_junction:plugins.bst

    sources:

      git: 0

  - origin: bst

    path: internal_plugins.bst

    elements:

      cmake: 0

my_junction:plugins.bst:

kind: plugin

sources:

- kind: tar

  url: gitlab:bst-plugins-experimental/tree/master.tar.gz

In order to allow us to bootstrap projects easily, I would suggest that we

therefore keep the `tar` plugin in core.

The tar plugin can be used to fetch python packages, archives and any other

distribution mechanism for those plugins and has no external dependencies by

default, which makes it convenient to use there. It also is deeply embedded

in our tests.

Any comments, suggestions, problems?

Does anyone foresee any problems that this would bring with regard to

remote execution?

Thanks!

Ben

- [0] https://mail.gnome.org/archives/buildstream-list/2019-April/msg00022.html

- [1] https://mail.gnome.org/archives/buildstream-list/2020-April/msg00013.html