Re: [BuildStream] Proposal: Add support for running tests in BuildStream

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

On Wed, 2018-08-01 at 18:17 +0100, Chandan Singh wrote:
> Hi Tristan,
[...]
> > The point is that:
> >
> > * The existence of the BuildElement's artifact is a statement that the
> >   code has built successfully.
> >
> > * The existence of the testing element's artifact is a statement that
> >   the built code has also been tested.
> >
> > * BuildElements which depend on other BuildElements for the purpose of
> >   building, do not require that other testing elements have also
> >   passed.
> >
> >   Note: this is what affords you the parallelism of builds not blocking
> >         on the testing of elements you depend on.
>
> This is the part where I am not very convinced :) The existence of the
> BuildElement's artifact is surely a statement that the code has been built
> successfully. But as a user what I care about the most is not whether or not
> it is built successfully but rather "is the output integratable?".

Here is the interesting part, "is the output integratable ?"

The design of BuildStream as it stands, without a massive redesign
which I think we don't need; will satisfy this requirement.

You continue to insist however that the output of your pipeline *is
every artifact*, but that runs counter to the design.

Further, even if we baked this whole thing into the base Element, if
you were to start builds in parallel, there would be *something* that
exists which those elements can stage and build things against whilst
the tests are running in parallel: This thing is currently called an
artifact, and the statement that the corresponding tests have passed
should be another artifact.

There most certainly must be a way to express a pipeline which does
what you want, when we ask the question "is the output integratable ?",
we should be talking about a tested artifact, not every single artifact
in the pipeline.

When building a Linux firmware for instance, there is always a stage
after the builds of every element, before the composition stage, where
one can depend on the tested artifacts.

If understand correctly, your use case involves generating packages;
and if I were to spell out the requirement I believe you are hinting at
(but don't think has been adequately described in this conversation) in
my own words, I would say that requirement is:

  o I want the creation of a deployable package artifact to be
    contingent on it's input artifact having passed tests.

  o Further, I want the creation of a deployable package artifact
    to be contingent on the input artifact's dependencies all having
    passed tests also.

Did I accurately describe the problem ?

The graphs in this email do not address the requirement above, and
perhaps there is indeed a need for a core enhancement in order to
achieve this. This core enhancement could be something along the lines
of describing roles and relationships between an element and other
related elements, and these relationships could be leveraged to
automate a transformation on a dependency tree.

For instance, one might have to make the statement that:

  Creating this deployable package depends on the "test" elements
  related to this deployable package's build element, and all of
  it's build dependencies, recursively.

I.e. a way to annotate the role of the relationship (in this case
"test"), and a way to generate a dependency tree which depends on
elements which are related via a role, rather than the elements
themselves, could accomplish this (think of this like depending on a
"test" shadow of the build dependency tree, instead of the dependency
tree itself).

There must still be a few more ways to fry this cat, and seasonings we
can apply to reduce it's chewyness (like the YAML improvements which we
should discuss separately).

The above is one plausible approach, and it is not elaborated to my
satisfaction (it's just an idea at this stage), but I *think* you have
enough to go on, or might come up with another approach for making this
kind of association conveniently ?

Cheers,
    -Tristan