Re: [BuildStream] [Summary] Plugin fragmentation / Treating Plugins as Sources

[William Salmon <will salmon codethink co uk>]

On 19/04/2019 06:35, Tristan Van Berkom via buildstream-list wrote:

[snip]
I don't think you've really understood the problem here so, I'll try to
outline it for you in different words, please try to understand:

The current pip origin (without automating any plugin downloads), which
I've already pointed out is not safe unless API stable, is actually
even not safe *even* if you are only using API stable plugin packages:

   * Project (A) uses API stable plugin package FOO
   * Project (B) uses API stable plugin package BAR
   * Project (A) junctions in project (B)
   * Plugin package FOO depends external library "BAZ == 1.2"
   * Plugin package BAR depends external library "BAZ == 2.0"

The above scenario leaves maintainer of project (A) with no recourse,
they must go to the length of porting either API stable plugin package
FOO or BAR to use the same API of library BAZ.

That is of course, by no means an acceptable workflow; when a project
author decides to use a given plugin library, they should be guaranteed
that it will always "just work", and this end user functionality is
*much* more important than a plugin developer's desire to import
external python libraries.

Tristan

I don't understand why this is a issue peculiar to python.

If both project A and project B use sub-process to call a external program that should be on the path but different versions of that program have slightly different cli options or behaviours then you could get in to the same problem.

It is true that many python projects are fast and losse with API compatibility but python is not unique in this. Indeed having our CI run lots of OS's often shows inconsistencies with 'host tools'.

I don't think it is unreasonable to have some plugin's be incompatible with each other, we should strive to avoid this but do we really want to make this our responsibility to make this impossible?

If you have very complex plugin requirements that involve the need for different versions of host tools for different projects/plugins then as far as i can see neither venv's or the other proposals on this thread will help?


My summary of this thread would be:

 * Tristan seems to wants all plugins to be compatible, this is a noble goal.

 * But in order to achieve it, we seem to be inventing very complex ways to manage this that seem to need a lot of work to create and maintain.

 * The plugin situation is not the best as it stands, but the suggested solutions are very expensive.

If we remove the requirement to have all plugins be compatible with each other, could we not come up with some much more simple improvements?

I see the need for this thread to explore the cost of having all plugins made compatible. As we can not make a good decision as to if the gain is worth the cost until we under stand the cost. But Tristan's response to Adams email makes it sound like it was decided that all plugins must be compatible, when i am not sure this is the case.

I think Angelos' summary is helpful in pointing out how much extra work there will be in creating a `gold plated` solution but if we are to maintain the solution are we also going to have our CI enforce this? and how much more cognitive load are we going to add to developers and maintainers?

We have agreed that bst 2 will brake compatibility with bst 1.2 but have we agreed that all plugins must be compatible with each other?

Will