[BuildStream] Summary: Workspace related DX features & design

[Tom Pollard <tom pollard codethink co uk>]

Hi all,

We're sending this email out with the aim to consolidate the outcomes of
the discussions around 'Proposal: Workspace related DX features & design'[1]

Part of the outcome of this summary will be the consolidation and
potential creation of new gitlab issues as explained in the original
email. Each issue will take into account the details in any related
original issues along with the discussion generated from the initial
Workspace DX proposal email. This will also accommodate for further
discussion on the specific tasks.

Points to cover:
1. run bst commands in workspace directories, including defaulting
 command to said workspace element.
2. Make workspace creation easier.
       - Default location
       - Automatic names.
       - Making multiple workspaces
       - Relative paths
       - Use of cached builds/buildtrees
       - Moving workspaces
3. Debugging in bst shell, workspace considerations

1. ============

- Using .bstproject.yaml as the workspace fragment, only
contains the path to its parent project (or other projects that have
been forcefully associated to it), non-extensible.

- Support for multiple projects pulling a workspace is not allowed by
default but it can be forced with known side-effects. Default behavior
is to say a workspace already exists at a given location with a given
name, along with what project it belongs to. If forced to be a workspace
for multiple projects then the original parent project takes precedence
by being the first entry in the .bstproject.yml fragment (i.e. is the
default resolution unless overridden with --directory). bst commands
again need to be forced & warn the user that it may affect other
projects consuming the shared workspace.

2. ============

- Default location to be set at user level, with the standard levels of
provenance. Both the default directory & --directory at cli are
referring to the 'root' directory in which directories for workspaces
will be created, with each element's workspace named after the
respective elementname as subdirs.

-  Multiple workspaces:
  ~~~~~~~~~~~~~~~~~~~~
  bst [project options] workspace open [open options] element1.bst
[workspacename] \
          --open [open options] element2.bst [workspace name] \
          --open [open options] element2.bst [workspace name] \
          ...

- [project options] = --directory pathto/project \
                      ...

- [open options] = --directory  pathto/workspace \
                   --no-checkout \
                   -f --force \
                   --track \
                   ...

- [workspace name] =  optional value to override the default behaviour
of naming workspaces as elementname

- Examples:

bst --directory /path/to/project workspace open --directory
~/my_workspaces elementname.bst

This will create a workspace for elementname.bst in ~/workspace i.e
~/workspace/elementname for the project in /path/to/project

bst workspace open elementnameA.bst --open elementnameB.bst workspacebob
--open elementnameC.bst

With a default value for the workspace 'root' directory set in user
configuration to ~/fooworkspaces, this will create
~/fooworkspaces/elementnameA ~/fooworkspaces/workspacebob
~/fooworkspaces/elementnameC

bst workspace open elementnameA.bst

With a default value for the workspace 'root' directory set in user
configuration to ~/fooworkspaces, this will create
~/fooworkspaces/elemenetnameA. If no default has been set this will
create a folder name elementnameA in the CWD (or '.')containing the
workspace output, or the current helpful warning message will be
extended to include the new features.

- For [project options], --directory will be resolved relative to the
CWD, for [open options] any --directory will also be resolved against
the CWD. The --directory option will be taken as the complete path to
the workspace and will override any relevant default paths or cli values
(such as [workspace name]), if either of theses cases are met then the
user will be warned/errored accordingly.

- If you set a default workspace directory that is a relative path then
it will be interpreted relative to the CWD. But the %{project-root}
variable can be used to create defaults relative to the project the
command is running for.

- The opening of workspaces with a cached buildtree (from a remote cache
server or otherwise) including the default behaviour user configuration
& the ability to override this with the addition of a cli flag to bst
workspace open [open options] will be added as a separate task once the
utilisation of buildtrees (such as the optionality to pull them into
cached artifacts) is more mature. These defaults & options can added in
an additive manner to the cli proposal defined above and therefore can
easily be landed separately without a breaking change to the cli api.

- The issue of moving workspaces, or furthermore the moving of projects
with open workspaces which could lead to workspace dissociation was
raised. It highlighted how having relative workspaces to a project could
potentially handle movement, but also the need to handle what happens
when paths get out of sync. The consensus was to not specifically
support the moving of projects/workspaces with a specific command, but
to provide useful documentation / error messages to the user on how to
handle the situation by closing and reopening the dissociated workspaces
with the existing commands. This will require regression testing to
ensure the existing commands capture these usecases. This does not rule
out the possibility of an additive api change to bst workspace to
further streamline this process for the user.

3. ============

- The topic of debugging in the bst shell (including the mounting of all
relelevant workspaces into it) was briefly raised in the initial
proposal to try and identify any knock-on effects this would have on the
workspace cli. However, the consensus gathered was that this would not
be the case. Therefore we will not consider this case anymore as part of
this proposal and as such should be continued in a separate proposal.


Regards,

Tom & Will

[1]
https://mail.gnome.org/archives/buildstream-list/2018-August/msg00078.html
-- 
https://www.codethink.co.uk/privacy.html