Re: [BuildStream] Standardized tests - 2

[Darius Makovsky <darius makovsky codethink co uk>]

Hi Ben,

On 06/12/2019 15:23, Benjamin Schubert wrote:
> Hey everyone
>
> TLDR: I propose to add a new optional 'entrypoint' to external plugins to 
> register the sources for the generic source tests.
>
> In [0] I introduced a plan for standardized source tests accross repositories. I 
> have now a slightly different but implemented version I would like to propose.
>
> The relevant MRs are:
>
> - On Buildstream [1]
> - On Bst-plugins-experimental [2]
>
> The plan I propose is:
>
> 1) Each plugin that want to be tested should define a 
> `buildstream.tests.source_plugins` entrypoint, that contains a 
> `register_sources` method, that does the registration of the sources to run the 
> tests.

My concern with that is there would be test infrastructure in the plugin
implementation. Would it make more sense for there to be a collection
point defined in testing for this instead? Generic plugin tests (such as
what they implement) can be generated for every collected plugin at test
time. Additionally if you rely on a plugin to implement the registration
then it's possible some will not be tested. I think it's better to make
the test package responsible for this.

> Q: Should we keep 'buildstream.tests.source_plugins' or just have a 
> 'buildstream.tests', which _can_ contain a 'register_sources' method? This would 
> allow us to add more registration methods in the future with less effort.

I think it makes sense to keep `source_plugins` just because it makes it
easier to understand and reduces bloat in `tests` (especially if we go
with test generators for source plugins).

> I also propose to run a fixed version as part of BuildStream CI, plus one job 
> that runs against master but as an indication only.

of the experimental plugins against master? So would the buildstream ci
test the experimental plugins or is this the experimental plugins
testing against a pinned version of buildstream and also against master?

-- 
Best Regards,
Darius


For Codethink's privacy-policy please see
https://www.codethink.co.uk/privacy.html