Re: [BuildStream] Questiont: How to handle monorepos in buildstream

[Abderrahim Kitouni <akitouni gnome org>]

Hi Matt,You can
Le sam. 19 janv. 2019 à 00:41, Yates, Matthew (StoreOnce)
<yates hpe com> a écrit :
> I've been struggling to work out how best to handle monorepos with internal multiple projects and build 
> artefacts from within buildstream, and would be interested to hear other people's thoughts on this.
>
> Just for context, the project I'm building is a family of container images containing micro-services that 
> work together as a pod. To build I require a few dependencies are repositories that contain multiple 
> subproject. For example protobuf (https://github.com/protocolbuffers/protobuf) is a single repo, but 
> contains the different language bindings as separate internal projects and build targets (some of which are 
> required by different micro-services as our components are not all implemented in the same language).
> [...]
> 3. Have some mechanism to propagate sources from a parent element which contains the ref to the monorepo to 
> build elements
>
>                             +---------------------+   +---------------+
>                             |                     |   |               |
>                           +-+ protobuf-python.bst <---+ service-a.bst |
>   +------------------+    | |                     |   |               |
>   |                  |    | +---------------------+   +---------------+
>   | protobuf-src.bst <----+
>   |                  |    | +-------------------+     +---------------+
>   +------------------+    | |                   |     |               |
>                           +-+ protobuf-ruby.bst <-----+ service-b.bst |
>                             |                   |     |               |
>                             +-------------------+     +---------------+
>
>
>     With this method I only have a single ref in my project, and I can excluded unnecessary build artefacts 
> from micro-services easily. While everything downstream will need built on ref change, at least developers 
> can open workspaces on the build elements to test local changes without rebuilding unrelated artefacts.

You can achieve this without writing your own plugin, by using the
include directive [1]. The source declaration would not be an element
but rather just a snippet you can include in your other elements. For
example,

protobuf-src.yml:

sources:
- kind: git
  url: https://github.com/protocolbuffers/protobuf.git
  track: 3.6.x
  ref: 66dc42d891a4fc8e9190c524fd67961688a37bbe

protobuf-python.bst:

kind: distutils
(@): protobuf-src.yml
build-depends:
...

Note that the path to the included file is relative to the project
root rather than the element path, so you could put the file either in
the elements directory or outside it.

HTH,
Abderrahim

[1] https://docs.buildstream.build/format_intro.html#include