Re: BuildStream as a Library

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

On Mon, 2018-04-23 at 16:17 +0000, Benjamin Schubert wrote:
> Hey Everyone,

Hi Benjamin,

> We were wondering if BuildStream is or will ever be considered as usable as a 
> Library? Do you provide a stable API other than plugins and config files,
> or would it be possible to help design an API that could be used by another 
> python program.
>
> Let me explain our use case and why we think that would be beneficial:
>
> We are creating a tool that automates the creation of related pull requests
> accross BuildStream workspaces. In order to get the needed information 
> (Opened workspaces, remote urls, tracking branches, etc) we currently
> need to invoke `bst` quite often and load some of the bst files.
>
> This means that we duplicate some work compared to what BuildStream already
> has done. Moreover, we are not sure our parsing is correct, since BuildStream
> configuration parsing is quite complex.
>
> We would like to be able to avoid that by including directly BuildStream as a
> library in our code, but we need to know if that would be a use case supported
> and which parts of BuildStream would be part of this API.
>
> For our use case, having only the yaml loading and dumping would be enough as
> a start.

First of all I'd like to say that I'm enthusiastic about some of the
things which could be done with BuildStream as a library, and have
designed things such that the frontend code is fairly isolated and
separate from the core.

However, I have to say that I'm highly doubtful that we can offer a
stable API in the short term, maybe this is something we can consider
for 1.4 (the next, next stable series). There is a fair amount of
refactoring I want to get done especially around _pipeline.py, and the
initialization codepaths; also I think we need the freedom to churn the
internals for some time.

That said, I'd like to highlight a few things:

* Your tooling needs to get the original tracking urls and
  repositories related to open workspaces

* Open workspaces risk diverging from the BuildStream project
  definitions, i.e. a user may very well update their project
  through the VCS and still have open workspaces, in which case
  the branches etc related to the open workspace can change.

* The `bst workspace list` command is probably one of the fastest
  running codepaths, as it need not require loading the project,
  it also reports parsable YAML intentionally, which is a stable
  interface (in contrast with the private .bst/workspaces.yml file
  which is not a stable interface).

Seeing as I feel it's unrealistic to open up more public facing python
API at least for a fair amount of time, do you think it would suit your
purposes instead to record and track more information in the workspace
metadata, such that it can be reported though `bst workspace list` ?

In this way, we could potentially record the layouting of sources in a
workspace, their upstream URLs and tracking metadata at the time
someone opens a workspace, such that it could be reported in a stable
way; and also have the benefit of being guaranteed to be the
information which was in use when the workspace was opened.

Cheers,
    -Tristan