Re: [BuildStream] [Summary] Plugin fragmentation / Treating Plugins as Sources

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi Adam,

Judging from your reply, I don't believe you've understood the
discussion or summary, or perhaps you have read passed the important
details.

I'm trying to clarify for you, please read carefully and patiently.

On Thu, 2019-04-18 at 15:54 +0100, Adam Jones via buildstream-list
wrote:
> On Thu, 2019-04-18 at 20:05 +0900, Tristan Van Berkom via buildstream-
> list wrote:
> > Hi !
[...]
> >   * This proposal only automates the downloads of plugin files,
> >     and does not automate the installation of third party
> >     python library dependencies.
>
> This is quite a problem for end users, especially if you can't run your
> builds due a plugin not having access to a third party python library,
> that users may or may not be able to install, even then managing to
> install the correct version. Pip handles this for you.

That is really only a problem if plugins do import third party python
libraries instead of restricting themselves to APIs provided by
BuildStream and the standard library.

And as I've pointed out, in the *current* situation, it's not safe to
be doing this.

Chandan's proposal would cause it to be safe, and then, it would *only*
be safe to import external python libraries with Chandan's venv origin.

I don't think you've really understood the problem here so, I'll try to
outline it for you in different words, please try to understand:

The current pip origin (without automating any plugin downloads), which
I've already pointed out is not safe unless API stable, is actually
even not safe *even* if you are only using API stable plugin packages:

  * Project (A) uses API stable plugin package FOO
  * Project (B) uses API stable plugin package BAR
  * Project (A) junctions in project (B)
  * Plugin package FOO depends external library "BAZ == 1.2"
  * Plugin package BAR depends external library "BAZ == 2.0"

The above scenario leaves maintainer of project (A) with no recourse,
they must go to the length of porting either API stable plugin package
FOO or BAR to use the same API of library BAZ.

That is of course, by no means an acceptable workflow; when a project
author decides to use a given plugin library, they should be guaranteed
that it will always "just work", and this end user functionality is
*much* more important than a plugin developer's desire to import
external python libraries.

> <snip>
>
> > And the negative points:
> >
> >   * Most importantly, Angelos has done some research into this
> >     and (as I rather suspected), a solution of this nature does
> >     not appear to be all that viable or trustworthy.
> >
> >     See his posts here[3] and here[4].

Did you read this part ?

Do you understand the experiment which Angelos was attempting ?

> <snip>
> >     Essentially this would be a lot of local state to ensure is
> >     always up to date, not to mention we would have to write a lot
> >     of new code when compared to my original `git` origin proposal.
> >
> >   * The plugin tracking implementation would probably be futile.
> >
> >     We would want to support not only hosting of plugins on PyPI,
> >     but also allow use of any VCS which pip supports (one of the
> >     advantages of this proposal), but this would be very complicated
> >     to support for tracking purposes.
> >
> >     I don't think this is a deal breaker because I don't think
> >     tracking of plugins is a very important part of the solution
> >     anyway.
>
> I think the pip/venv approach is the most *obvious* approach for
> someone who is used to using a Python tool, most new starter guides
> recommend to work inside a virtualenv, so this is not something that is
> uncommon.
>
> I am raising this point as why does BuildStream itself have to carry
> the burden of package versioning, yes it is not nice, but if a project
> requires multiple dependencies with different versions. A good
> development flow would be to use something like a venv or docker, not
> to rely on your build tool to solve this for you.
>
> Python has tools like https://github.com/pypa/pipenv that aim to reduce
> this burden on the users also and very widely used.

I understand you want the world to be a better place than it is, me
too.

The above only solves a problem where *one* application imports a fixed
set of API unstable libraries, by carefully selecting which version of
which library to activate inside a single environment.

Our problem is with plugins, so we need to allow *multiple* plugins
with different requirements on API unstable libraries to execute inside
a single environment.

Do you understand now, why Chandan's proposal is so very difficult to
implement ?

[...]
> Python imports are very common and people expect to be able to import
> foo, bar when needed, trying to stop this will just cause friction.
> <snip>

Please, I am not *trying* to stop this, I am simply stating that as
things currently stand, doing this is just not safe and shouldn't be
encouraged.

The question is not about trying to stop plugins importing from
external libraries (the current climate already dictates this should be
done), rather it is the inverse: whether it is possible to make it safe
to do so, Chandan's very brave proposal sets out to achieve this
safety.

Cheers,
    -Tristan