Re: [BuildStream] Preserving artifacts out of workspaces

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

On Wed, 2018-08-15 at 14:35 +0900, Tristan Van Berkom via BuildStream-list wrote:
> On Mon, 2018-08-13 at 15:47 +0100, William Salmon wrote:
> > Hi Tristan
>
> [...]
> > I have tried to recreate what you described in 
> > https://gitlab.com/BuildStream/buildstream/commit/dab4489569652d8b37ff57b05f1f1e34059114f5
> >
> > Is this approximately what you meant?
>
> Yes, it appears like both of the examples you whipped up basically do
> out of tree builds properly.
>
> > It seems to work ok, although there are a few funnies I am still working 
> > through.
> >
> > If i run `bst build hello.bst`
> >
> > Then it builds fine and the hello program can be run in the shell or via 
> > `bst shell hello.bst -- hello`
> >
> > But if I make a workspace first and then run `bst build hello.bst` it 
> > fails but if I make a build folder in the workspace then `bst build 
> > hello.bst` runs.
>
> Right, I can see how this can be confusing.
>
> The reason this is happening afaics, is because the `BuildElement` will
> take care of creating the `command-subdir` in the case that it does not
> exist at build time, but not at workspace creation time.
>
> This appears to be an edge case bug: We are not properly creating the
> workspace of the build tree in exactly the same way that we would
> create the build tree without workspaces involved; so any extra steps
> we might do in creating the build tree needs to be captured in the
> process of creating a workspace, I would say.
>
>
> > Then I can look in the workspace and see my build artifacts or "Element 
> > output"(s) in the build directory.
> >
> > I can also run `bst shell hello.bst --build` and run the "Element output"
> >
> > > Does this satisfy the use case sufficiently that we can just close this
> > > and move on ?
> > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> > If my example is what you were trying to describe and all the relevant 
> > parties have a look and try out my example implementation of your 
> > suggestion and are happy with this way of working then I propose:
> >
> > * Given that we have got this far in to disusing this before you pointed 
> > this out I think it might be a good idea to properly document this, 
> > maybe a tutorial, and then have the issue closed once the MR adding the 
> > docs has been merged?
> >
> > That dose everyone think?
>
> Honestly, I see this a bit differently.
>
> I think first: We probably need to bake in some optionality in the
> BuildElement YAML files for the build systems which potentially support
> out of tree builds (like autotools, cmake, meson at least).
>
> Using variables declared in those build element kinds, it should be
> fairly straight forward to allow the user to choose whether they want
> to perform an out of tree build or not with a simple switch.
>
> Ideally, it would be nice if out of tree builds were the default.

Actually scratch this idea of making it default; seeing as a project
can determine the default for all of it's elements[0], I don't think
there is any need for such a breaking change.

I would imagine that the gnome-build-meta[1] project will want to
enable out of tree builds as much as possible, since they are
interested in ensuring correctness of their sources; while the
freedesktop-sdk[2] project might not, as they are more interested in
just having stable builds that work all the time.

In any case, a single statement in project.conf can enable it for all
elements of a given `kind`, and the element declarations themselves
(.bst files) can make a final decision as well.

Cheers,
    -Tristan

[0]: http://buildstream.gitlab.io/buildstream/format_project.html#project-overrides
[1]: https://gitlab.gnome.org/GNOME/gnome-build-meta
[2]: https://gitlab.com/freedesktop-sdk/freedesktop-sdk/