Re: [BuildStream] [Proposal] Plugin fragmentation / Treating Plugins as Sources

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

On Tue, 2019-04-16 at 17:23 +0100, Angelos Evripiotis wrote:
> On Sun, Apr 14, 2019 at 11:39 AM Tristan Van Berkom
> > <tristan vanberkom codethink co uk> wrote:
> > If we can get some prototype which demonstrates the base concept of:
> >
> >   * Installing two separate venvs with different dependency versions
> >   * Running an interpretor which loads a plugin in both separate
> >     venvs (it could even be the same plugin, and need not be a
> >     "BuildStream" plugin, but some python module loaded on demand)
> >   * Prove that we infact have separation (perhaps by having the plugin
> >     just print the versions of it's dependencies).
>
> I've been looking at this today, it hasn't come together in the easy
> way I was hoping for :)

Heh, yeah.

While the implications of all the *other* implementation details,
assuming we can get over this single hurdle, scare me a lot; I still do
think it is a very neat idea.

As I mentioned in Adam's subthread, if we do this, we aught to convert
Source plugins to use pure python dependencies (as much as feasible) in
order for this to be worthwhile (the main benefit I can see here is
easier ramp up time and avoiding the need to install host tool
dependencies at all).

Did you look at the link which Adam raised:

  https://github.com/mitsuhiko/multiversion

Apparently this hack has been tried before, and Adam mentions in his
mails the (few) changes which are needed to make the above trick work
with Python 3:

    https://mail.gnome.org/archives/buildstream-list/2019-April/msg00037.html
    https://mail.gnome.org/archives/buildstream-list/2019-April/msg00038.html

I haven't looked at it so I cannot speak to how safe the afore
mentioned trick actually is.

> I'll spend some time on it again tomorrow; I'm now less optimistic
> though.
>
> If it does come together, I expect some limitations to prevent inconsistencies:
>
> - BuildStream and the plugins should probably only use standard
> library and BuildStream types when calling eachother. This is to avoid
> confusion around mismatched versions.

Good point.

This observation also lead me to consider that any plugins should never
have any expectation of side effects caused by external libraries
persisting in the environment (pip source and element comes to mind as
territory where one should tread carefully).

> - We'll need to wrap calls from BuildStream to plugins with something
> that intercepts inline imports. e.g. like a pluginbase.PluginSource
> context.

I didn't expect this, I was rather expecting that with the import
behavioral overrides one might be able to inform the *module* itself
how import works for the whole module and anything in scope.

> - Plugins should only use public API functions to call eachother, so
> we can intercept inlined imports correctly. Alternatively we can
> introduce proxy objects that do it. Alternatively inline importing
> could be banned, I doubt this would be workable though.

I think this is always true anyway, except for cases where two plugins
which want to talk to eachother are distributed in the same package.

With that aside, it's conceivable that a plugin export some public API
which augments BuildStream's API.

So I think your above statement (if imports really need to be
overridden at the execution stack level and not at the module level)
translates more to: Plugins should only use BuildStream public API
functions to call eachother.


Also, we had a conversation at a previous gathering about plugin
derivation which comes to mind; i.e. it's not really supported right
now, and supporting plugin derivation has a whole other set of
implications.

However, that does not prevent the offerings of base classes from
BuildStream core (like ScriptElement for example), which might have
other helper methods asides from the basic Element methods: In a venv
enabled scenario we are discussing, one might expect to have shared
libraries specifically intended for BuildStream plugins which contain
abstract classes of their own (e.g. the git base class).

Cheers,
    -Tristan