Re: [BuildStream] Proposal: Add support for running tests in BuildStream

[Chandan Singh <chandan chandansingh net>]

Hi Tristan,

> Hi again Chandan,
>
> This is a lot to think about, I took the time to try to break down
> these requirements and brainstorm a bit further...
> Ok so there are a few main points I can glean from this, I'll try to
> condense them here:
>
>  a.) We don't want tests to block reverse dependency builds.
>
>  b.) We want to ensure that tests are passing before deploying
>      something (I think in your case it's a .deb, for other use cases
>      it can be another packaging format, a full system firmware, etc).
>
>  c.) We want to structure things such that we are sure which backing
>      element is associated to a failing test, this should be clearer
>      and stronger than a naming convention.
>
>  d.) We want to minimize the amount of .bst files and the amount of
>      YAML which needs to be maintained.
>
>  e.) Tests often require a different execution environment and set
>      of dependencies
>
>      E.g. many tests found in the modules which make up a typical linux
>      system require external services to run, like an X server or a
>      database.
>
> In addition to your points, I would add:
>
>  f.) We want to be able to run many kinds of tests, e.g. some tests
>      are provided by upstream module maintainers to test a specific
>      module, other tests may involve launching a VM and capturing
>      screenshots.
>
>  g.) With caching of build trees on the horizon, we probably want some
>      semantic allow reuse of that build tree for testing purposes.

Thanks for taking the time to break down my message into more concrete
requirements. (g) is definitely important as otherwise we would waste a
considerable amount of time in re-staging the sources and dependencies for the
tests.

> Let's skip over (a), I think all solutions aside from running tests
> inside the BuildElement as a workaround would cater to (a).
>
> Focusing on (b) and (c), I would imagine that a natural way to
> structure a package deploying project would have a pipeline structured
> similar to this (with (B)uild, (T)est, (P)ackage and (S)tack elements):
>
>      B(foo)     (additional test deps)
>        |  \       /
>        |    \    /
>        |    T(foo)
>        |    /
>        |  /
>      P(foo)
>        |
>        |
>   S(main target)
>
> It appears that satisfying (b) is very easy, we just need to make
> packaging of "foo" contingent not only on building of "foo", but also
> on testing of "foo".
>
> Addressing (c) I think is a matter of having a strong relationship
> between the test and what element that test is testing (although at
> this point there are many ways of looking at the problem).
>
> To brainstorm this, I would suggest that the potential test Element
> plugin have the following properties:
>
>    o Tests a specific element
>    o Can add dependencies for the sake of testing, these must be build
>      only dependencies (so they are not propagated forward).
>    o Can optionally stage the build tree of the built element
>      - needed for `make check` style tests, not necessary for other
>        integration style tests
>      - satisfy (g) as an optimization (no need to rebuild just to run
>        make check)
>      - alternatively without (g), needs to stage the sources of the
>        element it's testing
>    o The output of the test element is exactly the same output
>      as the build element itself (similar to a filter element without
>      the filtering, acts as a "pass through" but with a check/test)
>
> With a test Element designed like this, the dependency graphs can be
> simplified and I think we manage to address (c) by ensuring that
> semantically, the test Element "tests a specific element".
>
>      B(foo)  (additional test deps)
>        |     /
>        |   /
>      T(foo)
>        |
>        |
>      P(foo)
>        |
>        |
>   S(main target)
>
> Orthogonal BuildElements which depend on "foo", can depend directly on
> B(foo), while deployments of "foo" should depend on T(foo), a failure
> of T(foo) is always clearly a failure related to B(foo), there is no
> room for doubt.

I'm not sure about the part where you say that orthogonal build elements should
directly depend on B(foo). If T(foo) fails, how would foo's dependents know
about that? This correctly addresses the problem where P(foo) fails if T(foo)
fails but what about P(baz) that depends on B(baz). How can we ensure that it
also fails when T(foo) fails?

> Unless I'm missing something, this only leave the problem of (d), which
> can at least be thought of as a completely orthogonal problem. Solving
> (d) is of high value on it's own I think as it should let us express
> complex pipelines more easily.
>
> As an example of what could be possible with a little YAML-foo, we
> should be able to express B(foo), T(foo) and P(foo) in the same foo.bst
> file, and it would probably make sense that if a multi element .bst
> file was referred to without a subscript, it should always mean the
> last element. We could even have a grouping mode which implies a series
> of elements which depend on eachother, if that makes sense.
>
> Such a foo.bst file might look like this:
>
>    ===============================================================
>    # Specify the grouping type, lets say `sequence` means that
>    # the elements depend on eachother in a sequence
>    sequence
>
>    --
>    kind: autotools
>    # depend on the successful *build* of bar
>    depends:
>    - bar.bst[0]
>
>    --
>    kind: test
>    # Build depend on additional deps
>    depends:
>    - filename: base/mysql.bst
>      type: build
>
>    config:
>      # Specify the element to test, probably in such a grouping
>      # there is a way to specify this without saying "foo.bst",
>      # perhaps self[0] or such could work.
>      #
>      test-element: foo.bst[0]
>      commands:
>      - make check
>
>    --
>    # In the deploy stage we probably dont need much configuration,
>    # this is driven mostly by public data from the depended upon
>    # elements, and if the test element is a passthrough, we already
>    # implicitly depend on it due to being part of a "sequence".
>    #
>    kind: deb_deploy
>    ===============================================================
>
> If it is the case that a lot of the configuration in these files become
> redundant (i.e. only the "name" of something changes, lots of boiler
> plate), then it would be interesting to employ something like macros,
> turning large parts of boiler plate into one liners.

I like the idea of having the base build definition plus other related elements
like tests, deployable etc in the same yaml file for clarity and readability so
+1 on that.

For tests specifically, my concern is that if we follow this approach, we will
end up with roughly as many 'kind' of test elements as there are build
elements, like 'make-test', 'cmake-test' etc. since most build systems support
some way of running tests using them. This also runs the risk that the build
element and the test element can potentially get out of sync.

In the same way that we want the test element to be strongly connected to a
specific build element, perhaps it would also make sense to somehow strongly
tie the test element types with corresponding build element types. If the same
element plugin was providing both the 'build-commands' and the 'test-commands',
it would have been simpler but I am not sure how that fits into this model.

In the end, as a user, I would like to be able to say "I am an element of kind
make, build and test me using the standard make commands", 'make check' in this
case. The case of 'make' is simpler and may not warrant an additional plugin
but such plugins would be useful for more complex test setups.

> In use cases where you do not deploy a "package", a later "compose" and
> firmware creating element would still depend on the last element of the
> groups.
>
> In the unlikely but possible case where you want to `bst checkout` the
> individual build results of each BuildElement as a part of your
> production pipeline, then doing `bst checkout foo.bst` would still be a
> checkout of the BuildElement content that is contingent on the tests
> having passed.
>
>
> Of course this is not entirely specced out, but do you see anything
> with the general direction which fails to satisfy your use cases, or
> the wider general use cases we should be considering for the tool ?

Thanks! IMHO such an approach could work if we address the above comments.

Cheers,

Chandan