Re: [BuildStream] BuildStream tests and external plugins

[William Salmon <will salmon codethink co uk>]

I have copy and pasted my original reply assuming that bens re send is 
basically the same as his first one.

On 25/03/2020 17:39, Benjamin Schubert wrote:
> Hey everyone,
>
> TLDR: I propose to add a `@pytest.mark.plugins` mark on every test 
requiring
> a plugin that is/will be external and run tests in sequence: first 
the ones
> that are entirely self contained ("Buildstream core") then the ones with
> other plugins ("BuildStream integration")

This seems sensible

[...]

> We have three types of tests:
>
> - Core tests, don't depend on a sandbox nor on external plugins
> - Integration tests, don't require external plugins but need a sandbox.
> - Source plugin tests, are run against a set of internal and external 
plugins
>    and are allowed sandbox access.

The `source` plugins can be run with out a sandbox and we have one or 
two tests that sit in your `Core tests` category but that do use source 
plugins that are ready to move out, eg the `tar` source is used for some 
cache key tests. But there are only about 10 out of over 1000.

[...]

> This has a few advantages:
>
> - It is clear which tests require an external system
> - It mixes well with the `integration` mark we already currently have
> - It is minimally invasive

I think this should also be flexible enough to deal with the case I 
mentioned above.

[...]