Re: [BuildStream] Preserving artifacts out of workspaces

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

On Mon, 2018-08-13 at 15:47 +0100, William Salmon wrote:
> Hi Tristan
[...]
> I have tried to recreate what you described in 
> https://gitlab.com/BuildStream/buildstream/commit/dab4489569652d8b37ff57b05f1f1e34059114f5
>
> Is this approximately what you meant?

Yes, it appears like both of the examples you whipped up basically do
out of tree builds properly.

> It seems to work ok, although there are a few funnies I am still working 
> through.
>
> If i run `bst build hello.bst`
>
> Then it builds fine and the hello program can be run in the shell or via 
> `bst shell hello.bst -- hello`
>
> But if I make a workspace first and then run `bst build hello.bst` it 
> fails but if I make a build folder in the workspace then `bst build 
> hello.bst` runs.

Right, I can see how this can be confusing.

The reason this is happening afaics, is because the `BuildElement` will
take care of creating the `command-subdir` in the case that it does not
exist at build time, but not at workspace creation time.

This appears to be an edge case bug: We are not properly creating the
workspace of the build tree in exactly the same way that we would
create the build tree without workspaces involved; so any extra steps
we might do in creating the build tree needs to be captured in the
process of creating a workspace, I would say.


> Then I can look in the workspace and see my build artifacts or "Element 
> output"(s) in the build directory.
>
> I can also run `bst shell hello.bst --build` and run the "Element output"
>
> > Does this satisfy the use case sufficiently that we can just close this
> > and move on ?
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> If my example is what you were trying to describe and all the relevant 
> parties have a look and try out my example implementation of your 
> suggestion and are happy with this way of working then I propose:
>
> * Given that we have got this far in to disusing this before you pointed 
> this out I think it might be a good idea to properly document this, 
> maybe a tutorial, and then have the issue closed once the MR adding the 
> docs has been merged?
>
> That dose everyone think?

Honestly, I see this a bit differently.

I think first: We probably need to bake in some optionality in the
BuildElement YAML files for the build systems which potentially support
out of tree builds (like autotools, cmake, meson at least).

Using variables declared in those build element kinds, it should be
fairly straight forward to allow the user to choose whether they want
to perform an out of tree build or not with a simple switch.

Ideally, it would be nice if out of tree builds were the default.

While this would break some existing builds; a lot of the time people
want to run the builds out of tree just as a measure to ensure that
their project conforms to their build tool well (i.e., not all automake
or cmake projects build well out of tree; but their respective
maintainers are usually interested in noticing the breakage early
enough to catch it).

I think whether we make it the default or not, we should definitely
offer the option to do it automatically; and if we do that, we probably
don't need a whole example of how to override the BuildElement YAML to
do it yourself.

Make sense ?

Cheers,
    -Tristan