GNOME.org
 

Re: [BuildStream] Proposal: Workspace related DX features & design

Hi,

On Fri, Aug 24, 2018 at 2:17 PM William Salmon via BuildStream-list <buildstream-list gnome org> wrote:
Hi all,

As recently raised by Tristan [1], there are numerous related issues & hanging 
merge requests revolving around improvements & new functionality for BuildStream 
workspaces. As such it make sense to address them in a common proposal which 
hopefully addresses & highlights any overlapping or common changes needed to 
meet the respective goals. Tackling & grouping related requests together also 
means that if we have to conclude that a CLI change is needed [2], then it can 
be done once to minimise & hopefully mitigate further changes. It's also 
important to understand how any changes to the CLI could be consumed, in the 
sense of a user (porcelain) or in automation (machine readable) so any feedback 
from those perspectives is welcome.

Overview
========

We want to enhance the usability of workspaces in BuildStream. This includes 
increasing the flexibility of the existing implementation (such as supporting 
new defaults for workspace location & names [3]), QoL changes (such as the 
ability to use relevant bst commands inside workspaces, and bst commands 
targeting multiple elements, [4][5]) and new additions for aiding development
& debugging (such as cached workspaces & multiple workspaces in shell, [6][7]). 
Some of the issues raised in the call for proposal are months old and as such 
some of the included comments have now been rectified or made redundant; as 
such consolidation is needed.

When reviewing the state of the tickets that extend the support for debugging 
& development within the shell, it has been highlighted that cached build trees 
may be the more appropriate option instead of extending workspaces. As such it 
may be more appropriate to create a new bst command for this workflow & consider 
it outside of this scope. Remote execution should also take advantage of build 
trees (for failed build artifact debugging [8]) and not need any new API changes 
to workspaces, however if there is disagreement we should capture those changes 
in this proposal. Once a change has been developed in a workspace it would be 
desirable to have an easy way to get this information into the respective 
upstream/vcs (or even just locally for the projects consuming it); we think this 
may need to expand the workspace API but not change any existing features e.g. 
`bst workspace patch create` (or a documented patch/quilt workflow) or similar.

Proposal
========

These are the workspace related features to be considered together:

 * run bst commands in workspace directories, including defaulting command to 
   said workspace element.
 * Make workspace creation easier.
   - Automatic names.
   - Default location.
   - From cached build.
   - Make multiple workspaces.
   - Relative paths outside of project.
 
 * Debug, workspace/cached build may be used to preserve the required artifacts 
   so including this here.

Implementation
==============

Comands in workspaces
~~~~~~~~~~~~~~~~~~~~

This was started by chandan/cs-shadow in [9] however it leaves some questions.

[9] uses a file in ~/.local/share to achieve this but then it cannot easily 
implement relative workspaces. We propose using a `fragment` file that we will 
create in a .bst/project.yml of a workspace's root directory. This is similar 
to how git submodules identify which directories in the main project are 
subdirectories.

Makes sense.  Not sure about the terminology, but let's leave that as an implementation detail :).
 
Support for multiple projects pulling a workspace is not allowed by default but 
it can be forced with known side-effects. Default behaviour is to say a 
workspace already exists at a given location with a given name. If forced to be 
a workspace for multiple projects then the original parent project takes 
precedence (i.e. is the default resolution unless overridden) (bst commands 
again need to be forced & warn the user that it may affect other projects 
consuming the shared workspace).

I don't know how common this use case of having multiple projects refer to the same workspace is going to be, but it doesn't sound too expensive to support it regardless.

We will add a `fragment` to workspaces that points back to the project which
created it. This will allow for commands run in workspaces to find the project
that they should run from. Also it allows for projects to check for two projects 
using the same workspace.

The latter being important for I assume for refcounting purposes so that a bst workspace close doesn't end up cleaning up a workspace in use by another project?

The fragment will also be relative if the workspace was created as relative to 
allow for moving workspaces.

I am having a bit of trouble parsing this sentence.  Could you expand on this a little?
 
Workspace creation commands
~~~~~~~~~~~~~~~~~~~~~~~~~

Note: Defaults already exist at a project and system level so we will just store 
the default workspace location information alongside it.
 
We will change from:
bst workspace open -element.bst workspacename
to:
bst workspace open --directory /path/to/workspaces element.bst [workspacename]

I'm assuming that the Note you started with regarding defaults makes the --directory optional?  The main reason for double checking is because I want the command to be as simple as possible:
  bst workspace open element.bst

When opening a single element then you can:

bst workspace open --directory /path/to/workspaces --multi elementA.bst elementB.bst

Why do we need the --multi argument?  That doesn't seem intuitive.  If it is to get around the fact that otherwise elementB.bst would be interpreted as the workspacename, then I don't think we should do that.  Otherwise you end up with an open workspace carrying the name of the second element you wanted a workspace open for.
 
The directory option specifies the directory, in which the folder 
containing/(the root of) the workspace will be created. The folder 
containing/(the root of) the workspace is either the "workspacename" from the 
command line or if not given then it is the element name without the trailing
".bst".

eg. `bst workspace open --directory /path/to/workspaces elementA.bst`

would create:
path/to/workspaces/elementA/
as the root or the workspace

and `bst workspace open --directory /path/to/workspaces element.bst workspacename`

would create
path/to/workspaces/workspacename/

If the directory is relative, e.g. not starting with a "/" then we treat it as
relative to the project and the workspaces fragment will be in relative terms,
so you can move both or, more likely, the folder that contains both.

Makes sense.

What is the sensible default location used when no workspace location is defined?  'workspaces' as a subdirectory of the project directory?  Or will not having it set result in a helpful failure message?
 
In anticipation of a feature to allow a workspace to be created from a build 
tree we would also add a pair of exclusive options (--no-cache --use-cache) that 
would override the buildstream/system/project default option for, if a cached 
build should be used. The open command will only accept a single one of these 
options and so if you want some of the elements to be cached and some not, you 
will need to call the command twice - once with the elements to be 
cached and once for those to be created without.  As we propose to land all the 
CLI changes to `workspace open` command in one MR, we may need to land the 
option before the feature is ready. In this case we will have bst issue a "not 
implemented" error if the --use-cache option was used.

Makes sense to me.
 
Moving projects
~~~~~~~~~~~~~~~

This will only be supported if the workspaces are defined with a relative path.

Note: you could fudge by manually changing the fragment or we could add a
 utility to automate this.

This is because otherwise the fragment will get lost and you cannot run bst 
commands from the workspace. If run in the project then the fragment check will 
fail.

/projects/Stuff/ProjectA/
                        buildstreamfolder/project.conf
                        workspaceA/source/etc
                        workspaceB/source/etc

could be moved to:

/projects/better/ProjectA/
                        buildstreamfolder/project.conf
                        workspaceA/source/etc
                        workspaceB/source/etc

What is the recovery mode for a moved workspace that can't find its project?  Can it be re-registered?
 
Debug
~~~~~

Note: while the scope of this task is workspaces, they are clearly an important 
factor in debugging especially when debugging something you are using as a 
workspace, so we need to catch any potental API changes to workspaces. This 
therefore requires some detail of how to debug to ensure that no API changes 
are needed.

We will add an example of using GDB from a binary package to show how a generic
debugger could be used from within the bst shell. However it may well be that 
BuildStream could build the debugger, and it could be added to the shell as a 
runtime dependency. There has been relevant talk surrounding the perceived 
specifics of how to utilise 'SwankyDebug' inside a bst produced shell here [10].

This will then form the reference implementation that we can then talk around 
to implement other debuggers that may have other requirements.

This example will include debugging a library from an app (getting break 
points/messages) from the library when running `gdb app` from the app's shell.

Mounting workspaces vs cached build trees was brought up in [7] for this 
perceived usecase and as such it needs to be clarified whether the commitment 
to implement this feature with the current workspaces is 'worth it', or if the 
work to use cached build trees is ultimately the best long-term path. One other 
point of consideration is do we want to differentiate a flag for runtime 
dependency debugging vs build debugging (which is currently -b), or would 
runtime debugging be the default functionality of 'shell'?

The following must be implemented without affecting the project files (we dont 
want debug options to be accidentally added to the project's vcs) but parts will 
affect the cache key:

- Get list of runtime/build time dependencies
- Reference against open workspaces for the project
- Mount them into the respective buildroots, provide feedback if there are 
  clashes
- Environment is set up for target element, user has to be aware of this if 
  they want to tinker with dependencies within shell.
- Mount in debugger (local element type if not to be built by BuildStream, 
  either as dependency or debug flag)
- Debug, this can include making source changes & recompiling within shell
- Drop out of shell & carry on with bst workflow as appropriate to the 
  debugging outcome.

I feel that half of the proposal is non-contentious and the other half is around Debug.  From the above I am not clear there are going to be any conflicting workspace CLI changes needed, it seems not.  Can we separate out Debug on that basis?

Cheers,

Sander
 
Outstanding issues
==================

Issues to close
~~~~~~~~~~~~~~~

The description has been implemented so this can be closed: 

Creating workspaces should be one ticket setting out a concise API, so we will 
close the related ones and create a new one with a clear description and link 
to this list:


Issues to Update
~~~~~~~~~~~~~~~~

If we accept the fragment idea, then we will add the details and submit a MR 

Issues to create
~~~~~~~~~~~~~~~~

 - A new 'epic' issue which will contain the outcomes of this proposal as well 
   as a link to the original mailing list post, to act as a related ticket for
   all new work and to serve as a point of reference/origin.
- A new ticket for relative workspaces that covers "out of project" workspaces.
- An issue to cover all the changes to the command line CLI for the workspace 
  open command.
- An issue to track the proposed PoC for debugging an app & related libraries 
  from within bst shell with GDB as a generic tool.

Many Thanks
Tom & Will

# Links


_______________________________________________
BuildStream-list mailing list
BuildStream-list gnome org
https://mail.gnome.org/mailman/listinfo/buildstream-list
--

Cheers,

Sander
[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]