Re: [BuildStream] Questiont: How to handle monorepos in buildstream

[William Salmon <will salmon codethink co uk>]

On 22/01/2019 09:57, Abderrahim Kitouni wrote:
> Hi Matt,You can
> Le sam. 19 janv. 2019 à 00:41, Yates, Matthew (StoreOnce)
> <yates hpe com> a écrit :
> > I've been struggling to work out how best to handle monorepos with internal multiple projects and build 
> > artefacts from within buildstream, and would be interested to hear other people's thoughts on this.
> >
> > Just for context, the project I'm building is a family of container images containing micro-services that 
> > work together as a pod. To build I require a few dependencies are repositories that contain multiple 
> > subproject. For example protobuf (https://github.com/protocolbuffers/protobuf) is a single repo, but contains 
> > the different language bindings as separate internal projects and build targets (some of which are required 
> > by different micro-services as our components are not all implemented in the same language).
> > [...]
> > 3. Have some mechanism to propagate sources from a parent element which contains the ref to the monorepo to 
> > build elements
> >
> >                              +---------------------+   +---------------+
> >                              |                     |   |               |
> >                            +-+ protobuf-python.bst <---+ service-a.bst |
> >    +------------------+    | |                     |   |               |
> >    |                  |    | +---------------------+   +---------------+
> >    | protobuf-src.bst <----+
> >    |                  |    | +-------------------+     +---------------+
> >    +------------------+    | |                   |     |               |
> >                            +-+ protobuf-ruby.bst <-----+ service-b.bst |
> >                              |                   |     |               |
> >                              +-------------------+     +---------------+
> >
> >
> >      With this method I only have a single ref in my project, and I can excluded unnecessary build artefacts 
> > from micro-services easily. While everything downstream will need built on ref change, at least developers 
> > can open workspaces on the build elements to test local changes without rebuilding unrelated artefacts.
> You can achieve this without writing your own plugin, by using the
> include directive [1]. The source declaration would not be an element
> but rather just a snippet you can include in your other elements. For
> example,
>
> protobuf-src.yml:
>
> sources:
> - kind: git
>    url: https://github.com/protocolbuffers/protobuf.git
>    track: 3.6.x
>    ref: 66dc42d891a4fc8e9190c524fd67961688a37bbe
>
> protobuf-python.bst:
>
> kind: distutils
> (@): protobuf-src.yml
> build-depends:
> ...
>
> Note that the path to the included file is relative to the project
> root rather than the element path, so you could put the file either in
> the elements directory or outside it.
>
> HTH,
> Abderrahim
>
> [1] https://docs.buildstream.build/format_intro.html#include
> _______________________________________________
> BuildStream-list mailing list
> BuildStream-list gnome org
> https://mail.gnome.org/mailman/listinfo/buildstream-list

This is a neat way to do it :) Thanks for the tip!

I have not used the include directive, dose it just copy in the lines 
from protobuf-src.yml in to the protobuf-python.bst and 
protobuf-ruby.bst files? if you ran bst-show would the two elements 
protobuf-python.bst and protobuf-ruby.bst  depend on protobuf-src.yml as 
it is effectively now just a text file that is copied in rather than a 
element that they depend on?

This seems to be a really good solution for Matthew's use case but if 
there was a underlying c++ lib that the python and ruby then used it 
would need to be compiled from scratch for both elements?

Regards

Will