Re: [BuildStream] [Summary] Plugin fragmentation / Treating Plugins as Sources

[Benjamin Schubert <contact benschubert me>]

Hey everyone,

Thanks for the summary Tristan!

Given the lengths and number of responses, here are a few snippets of answers.

But first TLDR:

- I don't disagree with adding git sources for plugins.
  I don't however see any values in it, what's the difference with git
  submodules and local origin?

- I don't disagree with trying a venv implementation for isolating plugins.
  I'm however very worried about it, and think it should be avoided as the
  result would be worse to maintain than having compatibility layers in some
  plugins

- I disagree with removing the pip source. software has to be packageable and
  platform integrators should be able to expect software not to connect randomly
  to the internet to fetch things at runtime.

  Pip is a well known standard for python packaging. Newer version of Python
  even include a "-m ensure_pip" to install it if it is not there.
  Most python developers is used to pip. Even though some people might have
  run into trouble with the pip origin, there is nothing inherently wrong with
  it and we are using it every day to install our BuildStream plugins internally
  very successfully.

- I disagree that plugin incompatibilities with different third party libraries
  is a problem. Most linux distribution include a _lot_ of python software and
  the tools _do_ work all together with minimum friction. Yes it might mean a
  bit of compatibility layering but it's far easier than hacking something
  around venvs, or preventing third party libraries


In the end, as long as I can still have my pip source and _prevent_ BuildStream
from looking for plugins by itself, I'm ok.


===== Other responses =====


Thu, 18 Apr 2019 20:05:37 +0900, Tristan Van Berkom wrote: [0]

>   * Avoids any need for distro packaging of plugin files
>  * Avoids project users needing to know what plugins a project
>    requires and needing to explicitly install those plugins
>    in any way


Avoid? I can live with that.
However, it should still be possible to configure BuildStream to NEVER try
to fetch things by itself concerning plugins. In some constrained environments,
this would not be welcome. We should therefore make sure that we can still
prevent BuildStream from fetching any of its dependencies by itself.

Moreover, python packages and distribution packages are two different things.
There is no inherent need of packaging all python packages in a distro.
As far as I know, most distribution only package a tiny subset of all python
packages.


> As Sam Thursfield pointed out[6], the dependency on the `requests`
> library could instead be satisfied by BuildStream itself providing a
> more efficient API for fetching files, so BuildStream would instead
> depend on the `requests` library, and have the freedom to later
> redefine how this is implemented in the future (I quite like this idea
> in fact).

I do not like this idea, that would transform BuildStream in adding cruft
in the core just because one plugin might need it. If we push that further, we
would end up with git, svn, bazar, requests, and many other things either
exposed or reimplemented here.

Moreover, if we wanted to do that correctly, that would mean that BuildStream
would need to know about anything that a plugin would want to make, and, thus,
would render the use of plugins pointless, since we could just do it in the
core. Plugins are there to allow to add more capabilities to BuildStream.


Thu, 25 Apr 2019 18:21:31 +0900, Tristan Van Berkom wrote: [1]

> It is okay that people might have lower standards or lower expectations
> of what we can achieve with BuildStream, I don't mean to judge, I just
> wanted to prefix this message so people understand where I am coming
> from.

Could we try to avoid passive-aggressive messages in the ML to keep it focused
on our goal: Making BuildStream and Builds in general better?

> This is an issue that is particular to python because python libraries
> do not even make any promise to be API compatible.

Except for the libc API (which had some breakages), can you give me a single
opensource software that never broke an API? This issue is true for every
software. Many good software try to keep things compatible. And python is no
stranger to that. If you look at requests, which was mentioned multiple times
here, Openstack has a dependency on requests >= 1.2.6... more than 20 releases
without breaking the API, over many years. The question here is choosing wisely
packages that do care about that.

> but in the current landscape we don't really know this anymore, as everyone
> is now buying into the "vendoring" approach

Let's take a few big python packages:
- OpenStack doesn't vendor
- Ansible doesn't vendor (And gosh they have so many plugins and dependencies!)
- SaltStack doesn't vendor (Again so many plugins and python dependencies)
- Django doesn't vendor (1k+ plugins on https://djangopackages.org)

I don't know many packages  in Python vendoring. The only one I actually know
is Requests, which vendors urllib3 from the standard library, for a very
specific reason: Python releases being 18 months apart, a vulnerability flaw
in urllib3 might take up to 18 months to be patched and the decision was
discussed heavily before vendoring.

Another one would be pip, which vendors dependencies in order to avoid a
bootstrap problem.

> Conversely on the host, especially when we're talking about the kind of
> extensively scriptable CLI tools that we are likely to use to download
> source code, a great deal of effort is made to ensure those tools
> remain compatible and scripts which use them don't break from one
> version to another, or from one host to another.

Yes, and plugins that are heavily used will very likely have to have some
compatibility layer. I don't see why this would be different with python
dependencies.
Moreover, there is no inherent reason for a cli to be more stable than a Python
API, and thus using cli will not buy us anything (Let's remember the numerous
git bugs found when we added a CentOS runner?)

> This approach also allows projects to declare a sysroot which
> has the tools *they* need, without imposing one on downstream
> junctioning projects.

Allows projects yes, but plugins will still need to be compatible. So that
would not solve any problem or am I missing something?

> I don't think it is unreasonable to have some plugin's be
> incompatible with each other, we should strive to avoid this but do
> we really want to make this our responsibility to make this
> impossible?

I disagree. We should make sure that all _blessed_ plugins are compatible.
We should _not_ care about the rest. Does the kernel care about API breakage
for third party modules? Not at all. Why would we here?


> I don't really understand why this should be such a big issue, I
> understand the *desire* to have that freedom, but I don't see how
> having that freedom buys us much given what plugins actually do,
> especially if any reliability and stability needs to be sacrificed on
> the alter of developer freedom and convenience (that doesn't add up to
> me at all).

Having to reinvent things that exist, are well tested and relied on is more
than an inconvenience, it is of questionable value, and likely requires
investments to get to a similar level of quality than anticipated. On top of
this we would also lose an important aspect, which is familiarity with well
known packages and APIs.
Yes I want to be able to use external tools and libraries when writing plugins.
No I don't want to have to rewrite git in python.


On Tue, 23 Apr 2019 20:01:34 +0100, Thomas Coldrick wrote: [2]

> That said, it looks like this would add a great deal of complexity to
> BuildStream internals, and perhaps be built on barely-maintainable
> hacks, if it's possible at all.

> I think that unless the venv proposal can be done cleanly, then using a
> remote-type source is preferable. There are, however, some caveats to
> this.

I completely agree

> Firstly, I'm not sure if *just* a git type source is suitable.
> Expecting authors of plugins for a python tool to use `setup.py` and
> pip is reasonable, but forcing a version control system seems more
> unreasonable. I'm far from against including a git plugin origin, but I
> don't think it should be the only one. To me using something like the
> `remote` source as a base seems more reasonable, as long as a hash is
> necessary for the loading of such a plugin.

Again, I completely agree, python developers have developed an entire
ecosystem and are used to the tooling around it. We should not prevent them
from leveraging it.


Cheers,
Ben

[0] https://mail.gnome.org/archives/buildstream-list/2019-April/msg00056.html
[1] https://mail.gnome.org/archives/buildstream-list/2019-April/msg00072.html
[2] https://mail.gnome.org/archives/buildstream-list/2019-April/msg00068.html