Re: [BuildStream] [Summary] Plugin fragmentation / Treating Plugins as Sources

[Adam Jones <adam jones codethink co uk>]

On Thu, 2019-04-18 at 20:05 +0900, Tristan Van Berkom via buildstream-
list wrote:
> Hi !
>
> On Wed, 2019-04-17 at 15:57 +0100, Angelos Evripiotis wrote:
> > Dear list,
>
> I think Angelos is right that this deserves a summary, even if the
> clear decisions which often come with a summary are not necessarily
> there yet.

Thanks for suggesting this Angelos.

> It's a bit challenging to summarize all of these details, I'll start
> by
> re-stating my initial objectives, and then I'll try to summarize the
> pros and cons of both suggested approaches.

Thanks for the summary Tristan.

<snip>

> The solution
> ~~~~~~~~~~~~
> I'll try to summarize here what would be common to both `git` and
> `venv` solutions. From what I understand, I think we all agree at
> this
> high level.
>
>   * Plugins would be obtained at project load time (if not already
>     obtained), in an automated fashion, similar to how we currently
>     fetch junctions in an automated fashion.
>
>   * Projects would have the ability to reference an exact version
>     of a given plugin (or a given set of plugins).
>
>     This is particularly relevant for plugins which are not API
>     stable, where it becomes important for a given project to specify
>     exactly which version of a plugin they know to work.
>
>     Please refer to my initial reply to Chandan[1] (near the top) for
>     an explanation of why exactly it is dangerous to use the `pip`
>     plugin origin for any set of plugins that is not maintained in an
>     API stable way.
>
>     NOTE: My initial reply to Chandan misrepresents his `venv`
>         counter proposal, I did not understand that his proposal
>         intended the `venv`s to be isolated on a per plugin basis,
>         because I did not believe that to be possible.
<snip>

>   * This proposal only automates the downloads of plugin files,
>     and does not automate the installation of third party
>     python library dependencies.

This is quite a problem for end users, especially if you can't run your
builds due a plugin not having access to a third party python library,
that users may or may not be able to install, even then managing to
install the correct version. Pip handles this for you.

<snip>

> And the negative points:
>
>   * Most importantly, Angelos has done some research into this
>     and (as I rather suspected), a solution of this nature does
>     not appear to be all that viable or trustworthy.
>
>     See his posts here[3] and here[4].
<snip>
>     Essentially this would be a lot of local state to ensure is
>     always up to date, not to mention we would have to write a lot
>     of new code when compared to my original `git` origin proposal.
>
>   * The plugin tracking implementation would probably be futile.
>
>     We would want to support not only hosting of plugins on PyPI,
>     but also allow use of any VCS which pip supports (one of the
>     advantages of this proposal), but this would be very complicated
>     to support for tracking purposes.
>
>     I don't think this is a deal breaker because I don't think
>     tracking of plugins is a very important part of the solution
>     anyway.

I think the pip/venv approach is the most *obvious* approach for
someone who is used to using a Python tool, most new starter guides
recommend to work inside a virtualenv, so this is not something that is
uncommon.

I am raising this point as why does BuildStream itself have to carry
the burden of package versioning, yes it is not nice, but if a project
requires multiple dependencies with different versions. A good
development flow would be to use something like a venv or docker, not
to rely on your build tool to solve this for you.

Python has tools like https://github.com/pypa/pipenv that aim to reduce
this burden on the users also and very widely used.

> Why use external dependencies ?
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> I think the context of why a plugin *would* want an external python
> library dependency is important to consider in this conversation.
>
> I have a pretty thorough writeup in my reply to Adam Coldrick[5],
> for more details, but to summarize my opinions of that:
>
>   * Really, the only reason is for Sources to talk to external
>     tooling and implement Source.fetch() and Source.track().
>
>   * Elements create output from input, they should not cause
>     any side effects, so there is not much reason to desire
>     an external library here.
Python imports are very common and people expect to be able to import
foo, bar when needed, trying to stop this will just cause friction.
<snip>

> [3]: 
> https://mail.gnome.org/archives/buildstream-list/2019-April/msg00047.html
> [4]: 
> https://mail.gnome.org/archives/buildstream-list/2019-April/msg00051.html
<snip>

Cheers,

Adam