Re: [BuildStream] Clean separation of BuildStream 1 & 2

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi Benjamin,

On Fri, 2019-04-05 at 11:45 +0100, Benjamin Schubert wrote:
> Hey,
>
> Sorry I'm a bit late to the party, as I realize a MR is already up for this.
>
> TLDR: I think we should avoid renaming everything, as it will bring
> more pain to us as maintainers and developers of the project, and to users,
> who will need more changes and tweaks than would be necessary when updating.
>
> I think we can keep using standard tools to handle versioning, which will
> prevent headaches for testers and maintainers of projects, as those tools are
> well known and documented.
>
> More details inline.

TLDR; The pain to us as maintainers and developers is largely
      exaggerated, it is a little bit of pain now, most of which is
    
 already taken care of.

      Ensuring clean separation will leave no opportunity at all for
      users to hurt themselves by mixing and matching incompatible
      cores, plugins packages and projects, avoiding a lot of potential
      bugs and noise.

      Further, I don't believe that a hard/atomic switch from
      BuildStream 1 to BuildStream 2 is an option to consider at all.
      We don't control who decides to switch to BuildStream 2 and when,
      forcing an upgrade instead of parallel installing will
      potentially leave users in situations where they don't have a
      BuildStream to use for one or more of the projects they work on.


This has been a hard email to write, because I admit that I have been
frustrated for the duration of this project with the python community
because of this very debate.

Distutils maintainers actually want to deprecate and drop the
data_files from setup.py which is the only (albiet weak) technique
available for installing files into ${prefix}, outside of the python
environment, and actually integrating with the underlying OS.

The various locations of python environments (system, pip, pip user)
often cause problems due to unstable python libraries being mixed and
matched at random, and python developers respond by suggesting to use a
venv, of course using venvs just makes things more difficult for
downstream distro package maintainers who intend to build your package
in a controlled environment without internet access (and basically goes
against the ethos to of having only a single versions of a given
library installed on the system at a time; which I also strongly agree
with, this is not a matter of not having enough disk space, but it is a
matter of upgradability, bugfixability, and generally reducing overall
system complexity which is IMO very important).

To make things worse, virtual environments have just been making it
easier for python library maintainers to break API, while more
responsible/serious maintainers don't generally do this, the landscape
as is limits what libraries I feel it is safe to depend on in the long
term for BuildStream.

Using python was a language choice for BuildStream, not a choice to buy
into distutils and venv ecosystem, or a choice to limit ourselves to
not being able to integrate with the host OS nicely.


The above is not meant to be a rant, I just wanted to first communicate
this frustration - it is why this email was hard to compose, and I
recognize that there is frustration on both sides of this distro vs pip
debate (which people have been having for years), rather I am trying to
sympathize and avoid this being a rant.


Now, for the topic at hand, instead of replying everything inline and
feeling frustrated, I will just provide a single detailed reply after
having read your email multiple times.


BuildStream 2 to replace BuildStream 1
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You are suggesting that BuildStream 2 *replace* BuildStream 1 at the
time that BuildStream 2 is available.

I agree that BuildStream 1 should become obsolete and phased out, and I
strongly believe that this will happen - it is the goal.

That said, upgrading a user's existing installation of BuildStream 1 to
BuildStream 2 is not fair play.

What is going to happen, and we don't have any control over, nor do we
have the right to dictate, is that users will opt into using
BuildStream 2, more precisely; BuildStream project maintainers will
have to make this choice for everyone who participates in the project,
at their own pace. The people who maintain BuildStream projects will
have to have a high enough level of confidence that people who work on
the project are not inconvenienced by converting to a BuildStream 2
project.

This will hopefully happen before BuildStream 2 is completely API
stable but probably wont until we start messaging that it is close.

In any case, BuildStream 1 and BuildStream 2 projects will exist in the
wild, which means that users will need to use both until BuildStream 1
is obsolete.


Renaming everything will bring pain to us maintainers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
I have to say that this is exaggerated. The benefit outweighs the small
cost of admitting that BuildStream 2 is a new, separate API, by a long
shot.

It took me under a day to safely change the core BuildStream namespace
(including all references in documentation etc), the bulk of the work
described here is mostly done, and all that remains to do is to add a
couple safeguards so that proper error messages are communicated when
mixing and matching incompatible components (bst core/plugin
package/project data).

The *only* thing which changes for plugins is that BuildStream 2 facing
plugins must import from `buildstream2` instead of `buildstream`, and
the fact that they need to be renamed is just fact, it is completely
orthogonal to the BuildStream core namespace rename (see "The plugin
story" below for more details on renames).

I want to stress here that this move provides some real security such
that users *cannot* mess up, we do not leave any opportunity for them
to mess up and we do not ask them to setup virtual environments or to
do anything complicated to protect themselves.

And at the same time, the cost to us is virtually nothing, if anything
it is a matter of taste that we might not like to import from
"buildstream2", but this will fade, it costs nothing, and anyway this
is quite standard practice in software.


The plugin story
~~~~~~~~~~~~~~~~
We got external plugins wrong in BuildStream 1, maybe we can obsolete
the "pip" method of BuildStream's plugin lookup mechanism in
BuildStream 2 and fix it.

If we had done plugins by the book, we would be loading plugins from a
directory that the BuildStream installation *owns*, this means:

  * BuildStream installs a buildstream.pc file in ${libdir}/pkgconfig

  * BuildStream encodes it's own ${pkgdatadir}/plugins as a variable
    in it's pkgconfig file at install time.

    This means that if you install BuildStream into ${prefix} = /usr,
    then the "pluginsdir" would be /usr/share/buildstream/plugins

  * Plugins anyway depend on an installation of BuildStream in order
    to install, plugins should load the buildstream.pc at thier
    install time to obtain BuildStream's "pluginsdir"

    Plugins then install their package into our "pluginsdir"

As it stands now, we are looking up a system global plugins package by
name.

If we had done this in the traditional way described above, then a
plugin package like bst-external could just branch for BuildStream 2
support and install under the same name, without needing any rename.

They would just look for the buildstream2.pc file which would safely
relocate their installation into a directory specific to BuildStream 2.


Distributed unstable plugin repositories
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This section strays off topic a bit, but it is an interesting thought
experiment related to trusting virtual envs to solve API compatibility
for us.

Related to the plugin story, I should also point out that it is
*dangerous* to distribute a BuildStream plugins repository on PyPI
unless it is API stable.

It is in fact dangerous to *use* an external plugin repository that is
not API stable *unless* you are using it as a git submodule and loading
it via the `local` method of loading plugins (as we recommended in bst
external, and now continue to recommend in bst-plugins-experimental).

Distributing any API unstable plugin repositories currently means that
there can only be one on your system, as soon as you work on two
projects which use the same unstable plugin repository - you can have
situations where both projects get out of sync on this dependency.

Now, enter virtual envs. You could say that you need 2 separate virtual
envs to work on both of the separate projects which need different
versions of the same plugin repo (I personally think that is going
*way* too far in terms of causing the user pain).

But for arguments sake, lets say that we *did* recommend multiple venvs
for different projects - as soon as those two projects are junctioned,
then BuildStream absolutely needs to load the plugins from the same
python environment, and you have a broken project state which cannot be
built at all.


About virtual envs and users
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In your mail, you mention that users should know the basics of pip, but
it is a non sequitur that they should also manage virtual envs (and
consequently also have to mix and match the correct version of plugin
repository to work with the correct version of BuildStream themselves).

We have always recommended installing with pip:

    https://buildstream.build/source_install.html#install_pypi

I think that we should be doing our very best to ensure that the ramp
up time costs next to nothing for a user to get started using
BuildStream (this is just one aspect of being user friendly, but
really; even if we've got a great tool: If one has to struggle to use
it; we have sort of failed our mission to make lives easier).

I think that asking any user to use a virtualenv and be responsible for
managing what version of BuildStream is in it, and also what version of
a given plugins repository is in it, then we are not being anywhere
near friendly enough to our users.


Cheers,
    -Tristan