Re: [BuildStream] Preserving artifacts out of workspaces

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

On Wed, 2018-08-15 at 12:10 +0100, William Salmon wrote:
[...]
> Given how nice the `directory` and `command-subdir` "settings" are I 
> think it would be nice to have them better documented maybe via the 
> example I have, but then lots of examples add to the doc's maintenance 
> burden, i am happy to document them as you like but i would like to 
> document them or understand were I have missed there documentation.
>
> I think one of the links on 
> https://buildstream.gitlab.io/buildstream/core_plugins.html should 
> mention the `directory` "setting", maybe adding a item to the list of 
> sources that mentions things that are in the base source class might not 
> be a bad idea?

I've been a bit frustrated about this part of the reference manual as
well.

Currently, we redundantly document the "directory" configuration in
each and every Source plugin; usually with the same copy/paste text:

  # Optionally specify a relative staging directory
  # directory: path/to/stage

See:
  http://buildstream.gitlab.io/buildstream/sources/local.html
  http://buildstream.gitlab.io/buildstream/sources/remote.html
  http://buildstream.gitlab.io/buildstream/sources/tar.html
  ...

We should probably instead have a common ground for documenting
automatic configurations handled by the core `Source` base class, it
would make sense to add a new section at the top, module level of:

  http://buildstream.gitlab.io/buildstream/buildstream.source.html

And link back to that from each `Source` plugin, so we can document it
in one place. The annoying part is that there is only one configuration
that is handled by the `Source` base class, which is the "directory"

Similarly, we don't have much for the "command-subdir" variable and
should probably follow the same pattern here.

As far as I can see it's only documented in plugin authoring facing
documentation at:

   http://buildstream.gitlab.io/buildstream/buildstream.buildelement.html#element-configure-sandbox

Which describes how the `BuildElement` behaves and implements the
`Element` virtual methods.

We should probably add a similar section to the same `BuildElement`
page but for project authors, where we can describe more clearly the
significance of %{command-subdir}, %{build-root}, %{install-root} and
any common ways in which a project author can influence the behavior of
a `BuildElement`, and link back to that section from the individual
`BuildElement` element plugins (we currently have them linking back to
the `BuildElement` page, but it would be better to link to a specific
section for project authors, detailing specifically the YAML API
surface).

> > > I think first: We probably need to bake in some optionality in the
> > > BuildElement YAML files for the build systems which potentially support
> > > out of tree builds (like autotools, cmake, meson at least).
> > >
> > > Using variables declared in those build element kinds, it should be
> > > fairly straight forward to allow the user to choose whether they want
> > > to perform an out of tree build or not with a simple switch.
> > >
> > > [...]
> >
> > [...]
> >
> > In any case, a single statement in project.conf can enable it for all
> > elements of a given `kind`, and the element declarations themselves
> > (.bst files) can make a final decision as well.
> >
> > Cheers,
> >      -Tristan
>
> As for updating the build elements yaml to make this easy, I am still 
> getting my head round what this involves as I presume we will want to 
> have as little affect as possible and to be as easy to expand as possible?
>
> eg. we would want the example in 
> doc/examples/autotools/elements/hello.bst that already changes 
> command-subdir to still work if possible weather or not this new flag is on?
>
> My current working theory on this would be to make it so that the build 
> elements work if you only set `directory` and/or `command-subdir` and 
> that you don't need to set `autogen` and `conf-cmd` as that would make 
> out of source builds much better in and of its self. But then have the 
> switch you mentioned to set them to default values eg, `Source` and 
> `build`.

Yes this is an interesting implementation detail.

While it does sound ideal to have `command-subdir` "work" for out of
tree builds as well as regular ones, I'm not sure if it is meaningful.

Once you have, for example the `autotools` implementing an out of tree
build mode; the `command-subdir` will most probably be the build
directory, regardless of from where it was configured. I think it makes
sense for that directory to be "%{build-root}/_build" under all
circumstances, regardless of whether you are building the toplevel of
the sources in there, or a deep subdirectory of the sources.

At this point, one might instead have the `autotools` element provide a
new variable, say `%{source-subdir}`, which could simply tell the
`autotools` element "where the sources are" inside the `%{build-root}`.

Keep in mind that:

  * The `%{build-root}`, `%{install-root}` and `%{command-subdir}` are
    core features, implemented partly in the `Element` base class and
    partly in the `BuildElement` abstract class.

  * Other plugins sit on top of that core, they can be made to behave
    differently by modifying the variables which influence their parent
    class behavior.

  * Other plugins can also provide their own element specific variables
    and configurations which can be better suited.

I think that in this light, it makes sense to:

  * Make sure that `command-subdir` keeps working for regular build
    elements, when out of tree builds are not enabled.

  * If we can make `command-subdir` meaningful for out of tree builds,
    that might be interesting for the sake of avoiding growing API.

    But if it no longer makes sense (and I think it cannot); it would
    be better to just provide a separate variable, at the individual
    plugin level, like `%{source-subdir}`.


Setting `command-subdir` in a YAML file has consequences which should
be easy enough to predict; and being such a low level variable which
influences directly the $CWD of commands, it is a bit of a sledgehammer
 and should be handled with care when using BuildElement elements other
than `manual`.

Note that this is why the tutorial is designed to talk about manual
elements first, and then grow into other BuildElements; so that the
user should be able to get the hang of predicting how modifying
variables at different levels will effect the final build commands.

Cheers,
    -Tristan