Re: [BuildStream] Feature proposal: Add --build option to bst checkout

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi Chandan,

On Wed, 2018-09-19 at 22:14 +0100, Chandan Singh wrote:
> Hi,
>
> I was considering adding a `--build` option to `bst checkout` that would alter
> the behavior of this command similar to how `--build` works in `bst shell`.
> So, instead of checking out an element (and its runtime dependencies),
> BuildStream would checkout its build dependencies and sources instead. This
> checkout will look the same as the file system tree of the build sandbox.
>
> `bst shell --build` is very useful in debugging a build but there are some
> scenarios where it is not quite suitable. Here are some examples where
> `checkout --build` might be more useful:
>
> - if BuildStream is running as a remote execution client, `shell --build` will
>   be unusable

How is this a foregone conclusion ?

If we can setup a shell on a remote worker for the purpose of launching
a build, certainly connecting the user's terminal to that remote shell
is not immensely onerous - I would frankly expect this to "just work".

That said, I would also not expect to be able to launch graphical
applications remotely which access my own display server, or audio;
however with a bit of tinkering (forwarding the correct DISPLAY etc),
it should be still possible with some performance degradation.

Also, if you are using remote execution on a host which is compatible
with the target you are building (currently only linux binaries but
this should change with time), you should be able to download the
artifacts and sources from this build network as an initialization step
of `bst shell`, and have the shell execute in a local sandbox even if
the build executed remotely.

> - such a checkout can easily be imported into other systems, like Docker, which
>   can be useful for debugging purposes
>
> - in certain cases, it will allow users to involve host tools for debugging

For the above two points, as already outlined in this email[0], my view
here is that you have this entirely backwards.

The shell is an integral part of BuildStream, we need to work on making
it perform better (for which I have another proposal brewing), instead
of resorting to circumventing it.

> Proposed changes
> ================
>
> - Add `--build` option to `bst checkout`
>
> I think this option should also be mutually exclusive with `--deps [none,run]`
> as that makes sense in the context of checking out built artifacts, whereas
> `--build` makes sense in the context of checking out the sources. We can also
> consider adding `--deps build` option instead but in my opinion that may be
> more confusing for the user.

On the one hand, I feel that your arguments for proposing this in the
first place do not hold much water.

On the other hand, it *seems* that this proposal is very easy to
implement, and should not impose onerous complexity in the tooling
moving forward.

I think I have a counter proposal that I would prefer here, actually:

  * Let's add `--deps build` to `bst checkout`

    This does what you would expect, it only checks out the build
    dependencies.

  * Let's add a *separate* command for checking out sources.

    This would have the advantage of clearer symmetry with
    `bst checkout`, and could offer the same `--deps` options
    and `--except` options which we do in `bst show` and such.

    This way we offer the same API as `bst show` for expressing
    the multiple element from which you might want to checkout
    sources.

With this approach, you would get some added flexibility - there are
other reasons one might want to only checkout the sources (maybe we
want to ship all of the exact sources which went into an appliance,
except for a few proprietary bits, to a customer, for license
compliance ?).

Even if you want to checkout sources in side a `bst checkout` directory,
you might want more sources than just the target of a `bst checkout --build`

Cheers,
    -Tristan

[0]: https://mail.gnome.org/archives/buildstream-list/2018-August/msg00043.html