Re: [BuildStream] Preserving artifacts out of workspaces

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

On Thu, 2018-08-09 at 16:29 +0100, William Salmon via BuildStream-list wrote:
> I have been looking at, preserving artifacts out of workspaces, 
> https://gitlab.com/BuildStream/buildstream/issues/512.
>
> Overview
> ========
>
> At the end of the ticket it mentions artifacts appearing in arbitrary 
> location with in the sandbox however this seems to be at odds with, my 
> understanding of, the behavior of the sand box, as at build time most of 
> the sandbox is read only.
>
> This means that the only place that artifacts should be is in 
> subdirectories of the parent of the source directory.

This is incorrect, the parent directory of the source directory is
definitely not writable. Only %{build-root} and %{install-root} are
writable during a BuildElement's assemble phase.


I've been glancing at the chatter on IRC about artifacts and subdirs
and clearly had no idea what anyone was talking about, looking at 512
this seems to be much ado about nothing.

I will copy paste my comment from 512 here for the readership as my
reply to this email, with the intent of stopping the over engineering
train before it leaves the station.


My comment on 512:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
As far as I can see, the following invariants make this a non-issue:

* The BuildElement uses Sandbox.mark_directory() to declare that 
  build-root and install-root will not be readonly during the
  Element.assemble() stage.

* We currently do not support out of tree builds where 
  builddir != srcdir

  * We only support builddir != srcdir where the builddir is a
    subdirectory

* If we want to support completely out of tree builds, this does not
  need to precipitate such a dramatic change


A straight forward way of implementing out-of-tree builds, would be:

* Stage the sources to the source/ subdirectory, using the Source base
  class understood directory configuration

* Set the command-subdir variable to the source/ directory, so that all
  commands are run in the subdir where the sources are staged (this
  will "just work" for any build element).

* Invoke the configure script (or what have you), informing it that the
  build dir is ../build

  * Now the build dir and source dir are both subdirectories of 
   %{build-root}, a nicely writable location.

* Workspaces will "just work", since both the Source.stage() and
  Source.init_workspace() methods already take into account the
  automatic directory configuration which can be set for any Source.

Does this satisfy the use case sufficiently that we can just close this
and move on ?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

> Having Spoken to the originator of the ticket and some of those who
> have 
> already been discussing the ticket, it seems that their most likely
> use 
> case would be to have artifacts in the parent of the source
> directory 
> (or in folders contained with in it).
>
> There for if I make the *assumption*that all artifacts will reside
> in 
> the source directories parent (or folders from it).

It is very confusing that you are talking about "artifacts", an
"artifact" never exists in a sandbox at all.

A crucial portion of the result artifact will be collected from
"${install-root}" after the build, and this crucial part can be
accessed at any time later using `bst checkout`.

If we were to have a glossary somewhere (which was recently suggested),
I would refer to this portion of the artifact as the "Element output".


Anyway, I will snip the rest of this email; and state that I am very
opposed to this seemingly misguided proposal.

The issue report states that it is impossible to do something that is
clearly possible - this does not justify modifying the core or adding
any fancy new features.

Best Regards,
    -Tristan