[BuildStream] Summary of the 'Staging multiple elements in bst shell' discussion at the gathering

[James Ennis <james ennis codethink com>]

Hi all,

Here is a summary of one of the discussions that took place at the 
gathering: 'Staging multiple elements in bst shell'.

I'll briefly introduce the topic and then elaborate on what was 
discussed and then what was decided.

# Introduction of the topic
Pre-gathering, many discussions have been held around how `bst shell` 
should be able to handle multiple elements. One of the key motivations 
for this feature has been the request to be able to include tooling 
(e.g. a debugger) into the shell. Additionally, we should also be able 
to stage the target element(s) on top of a base element without having 
to declare the base runtime.

For this feature request, there already exists a previous discussion on 
the mailing list [0], BuildStream issue #422 [1] and BuildStream merge 
request !909 [2].

# Current progress
!909 [2] was created to address #422 [1] and other discussions held at 
the previous BuildStream gathering. As it currently stands, this MR has 
extended `shell` to allow for multiple elements in both 'normal' and 
build shells. However, to achieve the latter, it has introduced an 
extension to the public API.

# Summary of discussion at the gathering
First of all, we discussed what the desired and viable use-cases were 
here. It was agreed that for now, it makes sense to be able to stage 
multiple runtime elements in the shell so that we could, for example, 
include our favourite debugger/text editor. However, regarding the 
question of staging multiple elements into the build shell - it doesn't 
make sense to have multiple build environments in the same shell. We 
should have exactly one element that sets the build environment and the 
ability to add more runtime elements into the shell.

!909 [2] already contains the logic to handle multiple runtimes in the 
'normal' shell, which should perhaps be split out of this MR.

However, regarding a build shell with multiple (runtime) elements 
introduced, !909 has expanded our API to include a 'visited' dict which 
tracks what has already been staged. It was suggested that we should try 
to think of a 'cleaner' way to handle this as it has made the plugin API 
a bit less straightforward and could perhaps introduce potential 
associated issues.

So, to summarise, there are two use cases:
1. Staging multiple elements (and their runtime dependencies) in a  
‘normal’ shell
  - For example: `bst shell application.bst debugger.bst 
text-editor.bst`
  - Equivalent to creating a stack element which runtime-depends on the 
target elements, building this and then invoking `bst shell` on the 
stack element.
  - The logic for this exists in !909 thus can be split out.

2. Staging a build environment for one (and only one) target element and 
including additional runtime elements
  - For example: `bst shell --build application.bst debugger.bst 
text-editor.bst`
  - Less trivial

Regarding the second use case, it was agreed that it would probably be 
better to virtually stage extra runtime elements (with their 
dependencies) separately and then merge the 'extra runtime tree' (a 
single tree for all extra runtime elements and their dependencies) into 
the staged build environment. We would then try to run the integration 
commands on top of this.

The proposed method should work well in common scenarios, but there may 
exist some situations where this approach does not work. For example, if 
the extra runtime elements depend on a conflicting base system. However, 
it was agreed that there is not really a way to support such a scenario 
so the best we can do is provide good error messages in the case of 
conflicts.

# Further discussion
In addition to this, there was also discussion about how our CLI should 
look, the agreement on this has been demonstrated in the two example use 
cases above.

Chandan also re-raised the point that `bst shell` is still slow. 
However, for now, this is not within the scope of this feature request.

For anyone interested in that discussion, I hope you have found this 
helpful!

Thanks,
James


[0] 
https://mail.gnome.org/archives/buildstream-list/2018-August/msg00047.html
[1] https://gitlab.com/BuildStream/buildstream/issues/422
[2] https://gitlab.com/BuildStream/buildstream/merge_requests/909