Re: [BuildStream] [Proposal] Plugin fragmentation / Treating Plugins as Sources

[Angelos Evripiotis <angelos evripiotis gmail com>]

On Wed, 17 Apr 2019 at 05:32, Tristan Van Berkom <tristan vanberkom codethink co uk> wrote:

On Tue, 2019-04-16 at 17:23 +0100, Angelos Evripiotis wrote:
> On Sun, Apr 14, 2019 at 11:39 AM Tristan Van Berkom
> > <tristan vanberkom codethink co uk> wrote:
> > If we can get some prototype which demonstrates the base concept of:
> >
> >   * Installing two separate venvs with different dependency versions
> >   * Running an interpretor which loads a plugin in both separate
> >     venvs (it could even be the same plugin, and need not be a
> >     "BuildStream" plugin, but some python module loaded on demand)
> >   * Prove that we infact have separation (perhaps by having the plugin
> >     just print the versions of it's dependencies).
>
> I've been looking at this today, it hasn't come together in the easy
> way I was hoping for :)

Heh, yeah.

Unfortunately I don't think the simple gnat-on-top-of-PluginBase approach I've looked at is going to work out.

I think we'd probably need to take an approach that replaces sys.modules, builtins.__import__, and perhaps also sys.meta_path. Basically the whole import system. My first attempt at that resulted in mysterious results, I'd expect to need to spend a fair bit of time on that. I'm not sure I'd entirely trust it even then.

Here are the details of the approach I looked at, I'll close the MR when there are no more comments on it:

https://gitlab.com/BuildStream/buildstream/merge_requests/1296

I think that's all the time I have to investigate multi-venv unfortunately.

I now think the best way forward would be to stay as we are - buildstream and plugins installed into the same environment, running in the same interpreter instance.

I haven't followed the arguments against the pip source yet, it seems to work pretty well for my use-case though. I'm pretty pro-pip-source :)

Did you look at the link which Adam raised:

  https://github.com/mitsuhiko/multiversion

I did, but stopped looking when I saw that there were extra requirements on the plugin authors, I figured we should be able to do without it.

Perhaps it would have borne fruit:)

Cheers,

Angelos