Re: [BuildStream] Standardized tests - 2

[Benjamin Schubert <contact benschubert me>]

Hey Darius, sorry for the late reply

> My concern with that is there would be test infrastructure in the plugin
> implementation. Would it make more sense for there to be a collection
> point defined in testing for this instead?

I am not sure what you mean by "in testing". Do you mean by that in the
'tests' folder of the plugin?

If so, I would be against this, since we would not be able to install the
plugins normally anymore, we would also need access to their tests/ folder

> Generic plugin tests (such as
> what they implement) can be generated for every collected plugin at test
> time.

This is already the case, however, we need to know how to generate the 'repo'
for those tests, hence why we need some input from the plugins themselves.

> Additionally if you rely on a plugin to implement the registration
> then it's possible some will not be tested. I think it's better to make
> the test package responsible for this.

Again, I am not sure what you mean by the 'test' package?

> of the experimental plugins against master? So would the buildstream ci
> test the experimental plugins or is this the experimental plugins
> testing against a pinned version of buildstream and also against master?

The BuildStream CI would test the experimental plugins against both a pinned known (last release)
version and the master

The master test would be non-fatal, the other one would be fatal.

Cheers,

Benjamin

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, 11 December 2019 11:23, Darius Makovsky via buildstream-list <buildstream-list gnome org> wrote:

> Hi Ben,
>
> On 06/12/2019 15:23, Benjamin Schubert wrote:
>
> > Hey everyone
> > TLDR: I propose to add a new optional 'entrypoint' to external plugins to
> > register the sources for the generic source tests.
> > In [0] I introduced a plan for standardized source tests accross repositories. I
> > have now a slightly different but implemented version I would like to propose.
> > The relevant MRs are:
> >
> > -   On Buildstream [1]
> > -   On Bst-plugins-experimental [2]
> >
> > The plan I propose is:
> >
> > 1.  Each plugin that want to be tested should define a
> >     `buildstream.tests.source_plugins` entrypoint, that contains a
> >     `register_sources` method, that does the registration of the sources to run the
> >     tests.
>
> My concern with that is there would be test infrastructure in the plugin
> implementation. Would it make more sense for there to be a collection
> point defined in testing for this instead? Generic plugin tests (such as
> what they implement) can be generated for every collected plugin at test
> time. Additionally if you rely on a plugin to implement the registration
> then it's possible some will not be tested. I think it's better to make
> the test package responsible for this.
>
> > Q: Should we keep 'buildstream.tests.source_plugins' or just have a
> > 'buildstream.tests', which can contain a 'register_sources' method? This would
> > allow us to add more registration methods in the future with less effort.
>
> I think it makes sense to keep`source_plugins` just because it makes it
> easier to understand and reduces bloat in `tests` (especially if we go
> with test generators for source plugins).
>
> > I also propose to run a fixed version as part of BuildStream CI, plus one job
> > that runs against master but as an indication only.
>
> of the experimental plugins against master? So would the buildstream ci
> test the experimental plugins or is this the experimental plugins
> testing against a pinned version of buildstream and also against master?
>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Best Regards,
> Darius
>
> For Codethink's privacy-policy please see
> https://www.codethink.co.uk/privacy.html
>
> buildstream-list mailing list
> buildstream-list gnome org
> https://mail.gnome.org/mailman/listinfo/buildstream-list