Re: [BuildStream] Prepare for possible Python plugin migration in v1.6

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi Chiara,

Thanks for raising this on the list :)

On Wed, 2018-11-07 at 16:44 +0000, Chiara Tolentino via BuildStream-list wrote:
> (Forking off the discussion in [1] as this should be two separate
> conversations really)
>
> I agree that, as Tristan mentioned [2], the workload required for the
> Python plugins
> to move out of core is not feasible for v1.4, but it would probably be best to
> at least start the conversation now about what would need to be discussed and
> agreed upon to prepare to land this change in, hopefully, v1.6. For example:
>
> 1.) What process would we have for telling users in the future about moving
> plugins out of core? Do we have to make a definitive list of plugins moving out
> or a definitive list of plugins that are sure to remain in core? Do we make a
> more vague warning? Do we need to make this announcement at all?

I think that we want a list of plugins which remain in core, not the
other way around.

We certainly want to notify users as strongly and clearly as possible,
so we will want to mention this in the release notes at least.

I fear that we won't have a clear story about how exactly the plugins
will be used and where they will dwell by the time we release 1.4, but
once we do; we should find a way to communicate that in advance.

Ideally, we iron out the whole user story (what does the user have to
do to adapt ?) as quickly as possible in the beginning of the 1.6
cycle, and we find a way to communicate that story to 1.4 users, one
way to do that might be to add a warning message in 1.4 as soon as we
know what will happen (not ideal, but if it is just for one cycle it's
not too bad).

> 2.) As a starting point, what would everyone think about only keeping the
> following plugins in core?
> General elements:
> - import
> - junction
> - compose
> - script
> - junction
> - filter
>
> Build elements:
> - manual
>
> Sources:
> - local
> - remote

This list looks about right to me.

Side note is that we have a goal to expose an abstract class or utility
module to implement `git` oriented sources. It will be better to not
land this in 1.4 and to land that in the plugins package which will
contain the `git` plugin instead.

> 3.) Is there anything else that we would need to start working on now to be
> able to make a v1.6 deadline for this change?

Yes.

  * We need to expose a new stable API in the BuildStream python module
    for testing.

    Mostly the `cli` pytest fixture and it's friends which currently
    dwell in `tests/testutils` need to be moved into buildstream and
    declared stable.

    This new `testing` module will have to be documented in the
    reference manual too.

    This should cover integration testing (testing which requires
    downloading external runtimes and running commands in sandboxed
    environments) as well as the simpler tests.

  * Similarly to the above, we need to share our documentation
    generation related tooling in an API stable way for downstream
    plugin packages.

    We currently have:
      - some magic in our Makefile which generates some rst templates
        for plugins automatically

      - The `bst2html.py` script and makefile rules which generate html
        by running BuildStream with colorization force enabled, from
        the session files in `doc/sessions`

      - The `badges.py` script which updates badges which are used
        by documentation and by gitlab rendered README.rst pages

        If it is interesting to show the latest versions of BuildStream
        in the places where we show our badges, we should presume that
        it will be equally interesting for plugin documentation
        packages.

  * Currently the fact that we have all of the existing plugins in
    BuildStream is a huge advantage for BuildStream's CI. For example,
    the `git` plugin is the only plugin which implements the
    `SourceFetchers` interface, so running the git related tests
    is important to run for every merge request in BuildStream core.

    We don't want to lose this advantage, so we will need some
    strategy for including third party test suites in our main
    core CI.

    E.g., I would envision that BuildStream CI manually updates
    the exact reference to the external plugin package git commits
    and tests a well known, unchanging version of a third party plugin
    repo against the current BuildStream version under test.

    Updating the commit reference to third party plugins requires
    a commit to BuildStream's .gitlab-ci.yml, and so changing to
    a newer version of test plugins is an activity that is also
    run through CI.

    We want to do similar in third party plugin CI setups.

    While this cannot be implemented before moving out plugins,
    the exact strategy for this cross-testing should be well
    thought out and designed before doing any migrations.

Cheers,
    -Tristan