Proposal: Add support for running tests in BuildStream

[Chandan Singh <chandan chandansingh net>]

Hi,

This was discussed at GUADEC this year, and perhaps also in other settings in
the past. But we did not have a mailing list thread or an issue about it so I
thought I could get the ball rolling.

Overview
========

BuildStream currently does not recognize tests as being special which leads
to some awkward setups. If one wants to run tests as part of BuildStream builds
they generally have to choose one of the following two options:

1. Run the tests as part of the build commands, or
2. Have a separate element that just runs tests

None of these two approaches are ideal - first approach increases the build
times and one has to include the test dependencies as build dependencies,
whereas the second approach doubles the number of elements as well as has the
side effect that the build for the main element doesn't fail when the tests
fail.

Proposal
========

One solution to this issue would be to have BuildStream recognize and handle
tests different than builds. Tests in BuildStream should work something like
this:

- A separate test job for tests is added to the pipeline that should start
  only after the build is finish. This job should not block other build
  jobs from starting but should be able to mark the original build as failed
  if the tests fail.

- It would also make sense to not push any artifacts until all the tests have
  passed.

- As tests may have extra dependencies, there should be some way of declaring
  dependencies that are only needed for running the tests.

Implementation
~~~~~~~~~~~~~~

I have not thought of the complete details yet but here's a rough sketch of
the implementation plan:

- Add a new type of dependency other than 'build' and 'run' called 'test',
  which will be staged only while running tests.

- Allow elements to define 'test-commands', which will define how to run the
  tests.

- When constructing the pipeline, queue jobs for running tests if
  'test-commands' are defined.

- Optionally, provide a way skip running the tests when doing a build.


What do people think? Although we still need to finalize the details of how
this would work, are there any other concerns with the high-level plan?

Cheers,
Chandan