Re: [BuildStream] Migration of plugins away from BuildStream core

[Phil Dawson <phil dawson codethink com>]

Tristan

Thanks for the input. See my responses inline, I hope they also go some 
way to addressing the concerns raised in the previous mails.

On 27/02/2019 10:43, Tristan Van Berkom via buildstream-list wrote:
> Hi Jim,
>
> Thanks for raising this.
>
> I think that we would be putting the cart before the horse if we were
> to proceed with moving plugins outside of the core without solving the
> issues around:
>
>    * Making it easier to use plugins outside of the core
>
>      * As you pointed out, while using the `pip` technique does seem to
>        work for me most of the time; when it doesnt work, it fails
>        miserably.
>
>        Incidentally, someone who was present at the meeting about
>        externalizing plugins at BeaverCon in Manchester 2018, had such
>        an issue installing `bst-external`, and we were unable to quickly
>        fix it.
Thanks for pointing this out. As far as I understand this should just 
work, and in my experience has. As you describe it, clearly that is not 
always the case. In light of that I'd be inclined to agree that we need 
to come up with a nice way of solving this.
>      * I had given this some thought last year, and brought up some
>        ideas on how we might be able to treat external plugins as
>        git repositories, and have BuildStream core include a special
>        `git` based Source specifically for obtaining a project's
>        required plugins.
>
>        https://mail.gnome.org/archives/buildstream-list/2018-November/msg00065.html
Interesting.  This seems like a sane approach, but I could do with 
giving it a bit more thought.
>    * Making it easy to maintain plugins
>
>      * Exposing parts of the test suite which make sense.
>
>        E.g., we have a whole battery of tests which run against every
>        "source kind", these are currently scattered across various tests
>        but when combined into one invokable suite, constitute a test
>        case asserting our expectations of abstract sources and how they
>        should behave.
>
>        An external plugin repository should be able to implement a
>        'Repo' scaffold (tests/testutils/repo) and add a one liner
>        to their own test suite in order to run BuildStreams thorough
>        source tests against their sources (asides from additional custom
>        tests which cannot be generalized, which need to have separate
>        tests regardless of where the plugins are hosted).
While we don't yet expose the repo scaffolding for testing sources, we 
have exposed the basic boilerplate which allows easy invocation of 
BuildStream and inspection of it's output.
>      * Expose some API for building plugin documentation.
>
>        We have some custom mechanics for building documentation already,
>        if we can wrap this up in some way which plugins can use, then
>        we can guarantee consistent documentation style across plugin
>        repositories.
>
>        Moreover, we can have the liberty to *change* these mechanics
>        in the future in the main BuildStream repository, so any
>        stylistic changes should propagate naturally to plugin
>        documentation.
>
>        Getting this right at first allows us to dodge a huge mess
>        later down the road, or a tiresome game of whack-a-mole where
>        we make efforts to hunt down plugin maintainers and patch those
>        modules to produce consistent documentation.
>
>    * Solving the CI problem in general
>
>      * With separately moving repositories, we need plugins to test
>        against a stable version of BuildStream, and have a way to
>        upgrade.
>
>        Similarly to how we always control the set of dependencies in
>        a CI environment so as to ensure that tests can only fail because
>        of code changes in the MR under test.
>
>      * Similarly, losing the plugin tests from BuildStream core means
>        that we miss out on plugins which exercise core BuildStream
>        codepaths which still need to be tested as a part of CI.
>
>        So in the same way as plugins need a stable version of
>        BuildStream which is upgraded periodically in CI, BuildStream
>        also needs to install and test against a limited set of plugins
>        known to exercise all of BuildStream's internal codepaths.

This last point should be solved by
https://gitlab.com/BuildStream/buildstream/merge_requests/1158.
As far as the prerequisites you've listed go, what we're missing
right now is a nice method of distribution of external plugins
(Especially as you say pip does not perform reliably here) and an
API for generating plugin documentation.

> On Tue, 2019-02-26 at 18:08 +0000, Jim MacArthur via buildstream-list wrote:
> [...]
> > I've now read all the above sources, and all the documents linked from
> > your first message, but not anything linked further than that; this has
> > taken me about 30 minutes so far.
> >
> > Nothing yet in there says why we're planning to move plugins out of
> > core. The only one that hints at it is Agustin's summary of the 2018
> > Gathering, which quotes:
> >
> > "BuildStream doesn't want to keep all that baggage"
> >
> > ... and that's it, that's all the justification I can find. Note that
> > this document also says:
> >
> > "Second reason is for keeping plugins that are not yet mature enough
> > [...] but eventually want to move into the BuildStream core, e.g.
> > X86Image". This directly contradicts today's plan; we are planning to
> > move mature plugins out of core.
> >
> > Also: "BuildStream cannot maintain every plug-in that people write, we
> > cannot bring them all into core. But we should have a convenient set of
> > plugins distributed with the core"... which I fully agree with. It then
> > explains some of the pain users have already reported from having to use
> > bst-external.
> I think these summaries are incomplete and inaccurate.
>
> We did discuss this at length at the 2018 gathering (in Manchester) at
> length, so I'll try to summarize what I can from memory.
>
> * We want to avoid a situation where everything and the kitchen sink
>    ends up in BuildStream, and encourage more development and
>    maintenance of plugins outside of BuildStream.
>
>    If BuildStream itself becomes a home for "all the plugins" then we
>    sort of defeat the purpose of having plugins (not completely, but I
>    think it would be counter productive to consider BuildStream to be
>    a home for all plugins).
>
> * There is already an existing misconception that plugins should
>    graduate from external projects into BuildStream.
>
>    This is a social problem, there is an unfounded desire to get your
>    plugin accepted into the "golden upstream" set of maintained core
>    plugins.
>
>    This lead to discussion about where we should draw the line,
>    and what the criteria should be for accepting a new plugin into
>    BuildStream core.
>
> * I think that expensive technical solutions to social problems
>    should be considered beneath us as a rule, and I personally
>    do not care in the slightest about hurt feelings of developers
>    who may feel discriminated against just because their plugin is
>    not accepted upstream.
>
>    I would be happy with a statement that the existing plugins
>    included with the core exist for historical reasons, and to
>    encourage people to maintain plugins outside of the BuildStream
>    core, and/or use custom plugins if the core set of plugins
>    do not satisfy a given use case.
>
>    I don't think the majority of contributors in the room at the
>    time agree with my stance on this, though.
>
> * I do agree that in an ideal world, we would have a clearer story
>    about what plugins should be included in the core.
>
>    This led to discussion about what that story would be.
>
>    Incidentally, we also discussed a gstreamer-like approach where
>    we could have -good/-bad/-ugly repositories, however it was also
>    raised that having a "bst-plugins-good" kind of repository would
>    result in the same social problem where different parties are
>    competing to have their plugin "blessed".
>
> * In the end, we did agree to move the majority of plugins outside
>    of the core, excepting only for "special" elements which are either
>    very generic or are special due to internal relationships with the
>    core.
>
>    E.g. `filter` and `junction` elements need to use private API to
>    interact with the core because of their special natures, as such
>    we can never move them outside of the core.

Thanks for the thorough summary. I in hindsight, I should
have really included this at the beginning of my original mail.

> While I do think we were all in agreement about moving these plugins
> outside of the core, I don't recall any agreement about recklessly
> proceeding with this plan in advance of developing the required
> infrastructure for this to be a painless experience for both users and
> plugin repository maintainers.
>
> I do recall explicitly punting this change until after the "next
> release" at the time, *because* of the logistic overhead involved in
> making this painless.
>
> My recommendation would be to create a single test plugins repository,
> and use this as a model to solve all of the issues outlined at the
> beginning of this email. When we are perfectly satisfied that the
> maintenance effort for plugin repositories is sufficiently low, and the
> user experience is not negatively impacted (if not improved) by moving
> plugins around, *that* would be the time to actually start migrating
> plugins around.
>
> Cheers,
>      -Tristan
Given the concerns raised. I propose initially removing the ostree
plugin only, using this as our guinea pig to test and improve the
tooling discussed above. One of the original motivating factors for
migrating plugins away from core was to reduce the number of
none-python dependencies users are required to install ostree being a
case in point here. Thus in addition to helping us get the surrounding
infrastructure up to scratch, this would also allow us to drop ostree
as a dependency of BuildStream's basic install. We can then make any
further removals when we're happy that we're ready to do so.

Cheers,

Phil

--
Phil Dawson, Software Engineer                          Codethink Ltd
https://www.codethink.co.uk/
We respect your privacy. See https://www.codethink.co.uk/privacy.html