Re: [BuildStream] Questiont: How to handle monorepos in buildstream

[Jonathan Maw <jonathan maw codethink co uk>]

On 2019-01-18 18:27, Yates, Matthew (StoreOnce) wrote:
> Apologies if this isn't the correct forum for this type of discussion.
>
> I've been struggling to work out how best to handle monorepos with
> internal multiple projects and build artefacts from within
> buildstream, and would be interested to hear other people's thoughts
> on this.
>
> Just for context, the project I'm building is a family of container
> images containing micro-services that work together as a pod. To build
> I require a few dependencies are repositories that contain multiple
> subproject. For example protobuf
> (https://github.com/protocolbuffers/protobuf) is a single repo, but
> contains the different language bindings as separate internal projects
> and build targets (some of which are required by different
> micro-services as our components are not all implemented in the same
> language).
[snip]
> 3. Have some mechanism to propagate sources from a parent element
> which contains the ref to the monorepo to build elements
>
>                             +---------------------+   +---------------+
>                             |                     |   |               |
>                           +-+ protobuf-python.bst <---+ service-a.bst |
>   +------------------+    | |                     |   |               |
>   |                  |    | +---------------------+   +---------------+
>   | protobuf-src.bst <----+
>   |                  |    | +-------------------+     +---------------+
>   +------------------+    | |                   |     |               |
>                           +-+ protobuf-ruby.bst <-----+ service-b.bst |
>                             |                   |     |               |
>                             +-------------------+     +---------------+
>
>
>     With this method I only have a single ref in my project, and I can
> excluded unnecessary build artefacts from micro-services easily. While
> everything downstream will need built on ref change, at least
> developers can open workspaces on the build elements to test local
> changes without rebuilding unrelated artefacts.
>
> Number 3 might be my favourite at the moment, and I have a POC for it
> where the .bst files would look something like this:
>
> protobuf-src.bst:
> kind: stack
> sources:
> - kind: git
>   url: https://github.com/protocolbuffers/protobuf.git
>   track: v3.6.1.2
>   ref: 9c793fc9e80ce6287bfe6f6cdf349055d1477406
>
> protobuf-python.bst
> kind: distutils
> sources:
> - kind: element
>   filename: protobuf-src.bst
> build-depends:
> - ...
[snip]

Hi Matt, thanks for your interest.

This is a problem that I've considered before, and led to the creation 
of filter elements 
(https://docs.buildstream.build/elements/filter.html#module-elements.filter).

With a filter element, you'd have a structure similar to number 3.

protobuf-src.bst builds all the output that you need in total, and 
defines split-rules that separate out the files.
e.g. it might look like:
```
kind: autotools
sources:
- kind: git
  url: https://github.com/protocolbuffers/protobuf.git
  track: v3.6.1.2
  ref: 9c793fc9e80ce6287bfe6f6cdf349055d1477406
public:
  bst:
    split-rules:
      python:
      - %{includedir}/protobuf-python.h
      ruby:
      - %{includedir}/protobuf-ruby.h
```

You would then have filter elements that select only the split-domains 
you want ("python" and "ruby" respectively).
e.g. protobuf-ruby.bst would look something like:
```
kind: filter
config:
  include:
  - ruby
build-depends:
- protobuf-src.bst
```

And then service-b would depend on protobuf-ruby as normal.

protobuf-src.bst is a build-dependency on protobuf-ruby.bst, so if 
anything runtime-depends on protobuf-ruby, it won't also depend on 
protobuf-src.

I hope this helps, let me know if you have any further questions.

Best regards,

Jonathan
--
Jonathan Maw, Software Engineer, Codethink Ltd.
Codethink privacy policy: https://www.codethink.co.uk/privacy.html