Re: [BuildStream] [Proposal] Plugin fragmentation / Treating Plugins as Sources

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi Adam,

On Mon, 2019-04-15 at 13:57 +0100, Adam Coldrick wrote:
> Hi Tristan,
>
> On Mon, 2019-04-15 at 20:45 +0900, Tristan Van Berkom wrote:
> <snip>
> > Again, I think the idea is "neat", but I should reiterate that
> > external
> > imports is really very limited to a very small amount of plugins who
> > actually import external libraries.
> >
> > There is no reason at all for an element to import anything external,
> > and only a small subset of plugins actually import external
> > libraries.
>
> This is the current situation, but it feels quite restrictive to
> enforce that requirement on anyone considering writing a plugin in the
> future. I don't really have an idea in mind of what kind of plugin may
> want to do that, and maybe this assumption is invalid and nobody will
> ever want external dependencies in a new plugin (I'd be interested to
> hear what authors of plugins such as those in bst-external think of
> this, since I don't have a ton of context really).

So in the above, when I say there is no reason for an Element to import
anything external, I really believe that will always be true.

Essentially Elements have the freedom to use any BuildStream APIs in
order to construct output from input, the implementations of Element
methods should not ever cause any side effects.

In general, we also *want* Plugins (Source or Element) to use
BuildStream provided APIs instead of trying to do things themselves
wherever possible - ensuring that plugins only use abstract interfaces
instead of trying to fiddle with files directly (e.g. now we have the
virtual directories) ensures that we can (re)implement the APIs plugins
call seamlessly, and when we add more platform support or just add new
features, plugins just automatically reap the rewards without needing
to be rewritten or modified.

> From this perspective, it is really desirable that plugins use
BuildStream provided APIs for virtually everything that they do.


For Source plugins specifically the story is of course different, as
far as I can tell the only reason you would ever want to import
something external is if you are implementing a Source plugin and want
to use an external python library for the `Source.fetch()` and
`Source.track()` implementations.

So I guess on the one hand, we don't really *need* to allow this, which
would save us from the complexities of managing venvs, but if we *do*
add this venv plugin origin and pay for it - we would do well to also
convert as many existing Source plugins as possible to use pure python
instead of host tools - so as to get the most bang for our buck.


Another thought I had while writing this is that... in addition to
handling the case where the host python is upgraded and venvs need to
be reconstructed; we probably also want a local PyPI mirror/cache on
the host - i.e. when two venv plugins have any dependencies which can
be shared but need to be installed into separate venvs, we wouldnt
really want to fetch the same dependency from PyPI over the wire more
than once.

Cheers,
    -Tristan