Re: Proposal: Add support for running tests in BuildStream

[Paul Sherwood <paul sherwood codethink co uk>]

On 2018-07-19 07:52, Tristan Van Berkom via Buildstream-list wrote:
> On Tue, 2018-07-17 at 23:00 -0400, Chandan Singh wrote:
> > Implementation
> > ~~~~~~~~~~~~~~
> >
> > I have not thought of the complete details yet but here's a rough 
> > sketch of
> > the implementation plan:
> >
> > - Add a new type of dependency other than 'build' and 'run' called 
> > 'test',
> >   which will be staged only while running tests.

Seems logical to me. We wouldn't want to drag in test dependencies 
unless they are going to be used.

> > - Allow elements to define 'test-commands', which will define how to 
> > run the
> >   tests.

I agree with this also. Baserock had/has a longer set of set of 
*commands, including test-commands

> > - When constructing the pipeline, queue jobs for running tests if
> >   'test-commands' are defined.
> >
> > - Optionally, provide a way skip running the tests when doing a build.
> >
> >
> > What do people think? Although we still need to finalize the details 
> > of how
> > this would work, are there any other concerns with the high-level 
> > plan?

+1 from me.

Now Tristan's comment...

> From what I understand, your proposal explicitly tries to reduce the
> number of elements in your pipeline - while conversely, the design of
<snip>

While I think it's correct that Chandan has an (implied) interest in 
avoiding too many bst files, I don't think that's the main drive for his 
proposal.

Regarding organisation of elements vs bst files, the approach we ended 
up with in YBD was to separate by convention, not by enforcement in 
code. For example YBD can parse a whole cluster (including set of 
systems, made up of strata, with all component *commands etc) from a 
single file. I tried to make this as permissive as possible for users, 
by scanning the whole directory tree looking for what might count as 
applicable definitions files.

>   o We are currently in the process of adding the "(@)" include
>     directive, which I think could plausibly have a natural extension
>     to a macro directive (e.g. and include with contextual
>     substitutions).

Does bst *have* to introduce more cryptic operators for this? :-)

br
Paul