Re: [BuildStream] Proposal: Testing BuildStream against external plugins

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

On Wed, 2019-02-06 at 14:37 +0000, Phil Dawson via BuildStream-list wrote:
> Hi all,

Hi Phil,

[...]
> CI job structure
> -------------------------
> My initial thought is that we test all the external plugins in a
> single CI job in parallel with the main test suite.
>
> Bundling these tests into a single job would have the advantage of not
> overly cluttering the CI pipeline, but would make it marginally more
> difficult to understand the test logs in the event of a failure.
>
> Given the aim would be to avoid downstream breakages and we would
> expect a plugin's cross platform testing will be handled by the
> downstream plugin maintainers I think this job need only be run for a
> single platform.

Disagree on a couple points here.

The aim is not exactly to "avoid downstream breakages", but to avoid
BuildStream breakages.

BuildStream will no longer have enough plugins to exercise it's own
core codepaths in a real test scenario - as such; we need to test the
core against enough plugins which exercise a variety of plugin
scenarios. For example, it is important to test against `git`
particularly because it is the one `Source` implementation which
implements `SourceFetchers`.

On the other hand, it is pointless to test the BuildStream core against
both `zip` and `tar` plugins, as they both necessarily exercise the
core in exactly the same ways, one of these should be enough.

It does not make any sense to treat these core codepaths any
differently from the rest of the core either, if we are testing on 5 or
10 different distros then there is no reason to discriminate against
these codepaths just because they can only be triggered with the right
plugins taking advantage of some feature, i.e. they are just the same
and should be run in every scenario we test.

Cheers,
    -Tristan