Re: [BuildStream] Preserving artifacts out of workspaces

[William Salmon <will salmon codethink co uk>]

On 15/08/18 06:53, Tristan Van Berkom wrote:
> On Wed, 2018-08-15 at 14:35 +0900, Tristan Van Berkom via BuildStream-list wrote:
> > On Mon, 2018-08-13 at 15:47 +0100, William Salmon wrote:
> >
> > [...]
> > > It seems to work ok, although there are a few funnies I am still working
> > > through.
> > >
> > > If i run `bst build hello.bst`
> > >
> > > Then it builds fine and the hello program can be run in the shell or via
> > > `bst shell hello.bst -- hello`
> > >
> > > But if I make a workspace first and then run `bst build hello.bst` it
> > > fails but if I make a build folder in the workspace then `bst build
> > > hello.bst` runs.
> > Right, I can see how this can be confusing.
> >
> > The reason this is happening afaics, is because the `BuildElement` will
> > take care of creating the `command-subdir` in the case that it does not
> > exist at build time, but not at workspace creation time.
> >
> > This appears to be an edge case bug: We are not properly creating the
> > workspace of the build tree in exactly the same way that we would
> > create the build tree without workspaces involved; so any extra steps
> > we might do in creating the build tree needs to be captured in the
> > process of creating a workspace, I would say.
I will add this to my todo for the ticket and when i update the ticket 
with our plan i will add the bug
> > > Then I can look in the workspace and see my build artifacts or "Element
> > > output"(s) in the build directory.
> > >
> > > I can also run `bst shell hello.bst --build` and run the "Element output"
> > >
> > > > Does this satisfy the use case sufficiently that we can just close this
> > > > and move on ?
> > > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > > If my example is what you were trying to describe and all the relevant
> > > parties have a look and try out my example implementation of your
> > > suggestion and are happy with this way of working then I propose:
> > >
> > > * Given that we have got this far in to disusing this before you pointed
> > > this out I think it might be a good idea to properly document this,
> > > maybe a tutorial, and then have the issue closed once the MR adding the
> > > docs has been merged?
> > >
> > > That dose everyone think?
> > Honestly, I see this a bit differently.
Given how nice the `directory` and `command-subdir` "settings" are I 
think it would be nice to have them better documented maybe via the 
example I have, but then lots of examples add to the doc's maintenance 
burden, i am happy to document them as you like but i would like to 
document them or understand were I have missed there documentation.

I think one of the links on 
https://buildstream.gitlab.io/buildstream/core_plugins.html should 
mention the `directory` "setting", maybe adding a item to the list of 
sources that mentions things that are in the base source class might not 
be a bad idea?
> > I think first: We probably need to bake in some optionality in the
> > BuildElement YAML files for the build systems which potentially support
> > out of tree builds (like autotools, cmake, meson at least).
> >
> > Using variables declared in those build element kinds, it should be
> > fairly straight forward to allow the user to choose whether they want
> > to perform an out of tree build or not with a simple switch.
> >
> > [...]
> [...]
>
> In any case, a single statement in project.conf can enable it for all
> elements of a given `kind`, and the element declarations themselves
> (.bst files) can make a final decision as well.
>
> Cheers,
>      -Tristan

As for updating the build elements yaml to make this easy, I am still 
getting my head round what this involves as I presume we will want to 
have as little affect as possible and to be as easy to expand as possible?

eg. we would want the example in 
doc/examples/autotools/elements/hello.bst that already changes 
command-subdir to still work if possible weather or not this new flag is on?

My current working theory on this would be to make it so that the build 
elements work if you only set `directory` and/or `command-subdir` and 
that you don't need to set `autogen` and `conf-cmd` as that would make 
out of source builds much better in and of its self. But then have the 
switch you mentioned to set them to default values eg, `Source` and 
`build`.

But that would brake the example above, i need to think more about this, 
i would like to avoid adding another "setting" in addition to the one 
already proposed if possible but i am not sure i can avoid it.

I guess I need to have a look through the gnome and freedesktopSDK 
builds to see how crazy they get with there build commands, are there 
any other places that might give good edge cases to consider?

Once I have a more thought through proposal I will reply hear, however i 
am keen to get some more feed back and buy in especially from the 
proposers of this issue before i get too far down this road.

Many Thanks
Will