[BuildStream] Proposal: Testing BuildStream against external plugins

[Phil Dawson <phil dawson codethink com>]

Hi all,

Following on from the discussion surrounding plugins at the gathering
I wanted to put forward a proposal for how we might leverage the test
suites of external plugins in BuildStream's CI.

Obtaining the external plugins
------------------------------------------------

We maintain a list of plugin repositories that we wish to test
against. We keep track of a known good version of each plugin which we
use.

My suggestion would be that plugin dependencies be installed at the
beginning of the CI job.  While this will slow down the plugin testing
job compared to pre installing the dependencies in a docker image,
installing them as part of the pipeline will mean that we only need to
maintain one set of information about the plugins we wish to test
against.


Running the tests
---------------------------
As a start, I think we can mandate that plugin repos which are to be
included in BuildStream's CI allow their tests to be exercised via
'setup.py test'

In the future we could perhaps use a BuildStream project to keep track
of these external plugins and time allow them to make use of custom
test commands. However, the former solution has the advantage of
making it trivial to view nicely formatted pytest outputs in the
GitLab.


CI job structure
-------------------------
My initial thought is that we test all the external plugins in a
single CI job in parallel with the main test suite.

Bundling these tests into a single job would have the advantage of not
overly cluttering the CI pipeline, but would make it marginally more
difficult to understand the test logs in the event of a failure.

Given the aim would be to avoid downstream breakages and we would
expect a plugin's cross platform testing will be handled by the
downstream plugin maintainers I think this job need only be run for a
single platform.



Please let me know if you think I've missed something or disagree with
any of the assertions I've made.

Cheers,
Phil

--
Phil Dawson, Software Engineer                          Codethink Ltd
https://www.codethink.co.uk/
We respect your privacy. See https://www.codethink.co.uk/privacy.html