[BuildStream] Questiont: How to handle monorepos in buildstream

["Yates, Matthew (StoreOnce)" <yates hpe com>]

Apologies if this isn't the correct forum for this type of discussion.

I've been struggling to work out how best to handle monorepos with internal multiple projects and build 
artefacts from within buildstream, and would be interested to hear other people's thoughts on this.

Just for context, the project I'm building is a family of container images containing micro-services that 
work together as a pod. To build I require a few dependencies are repositories that contain multiple 
subproject. For example protobuf (https://github.com/protocolbuffers/protobuf) is a single repo, but contains 
the different language bindings as separate internal projects and build targets (some of which are required 
by different micro-services as our components are not all implemented in the same language).

Here the approaches I can think of:

1. Create a single element for the repo and build everything in one go.

                        +---------------+
                        |               |
                      +-+ service-a.bst |
  +--------------+    | |               |
  |              |    | +---------------+
  | protobuf.bst <----+
  |              |    | +---------------+
  +--------------+    | |               |
                      +-+ service-b.bst |
                        |               |
                        +---------------+

    This is the most direct and naive approach and will give me all the artefacts I require, however the 
final image will contain artefacts that aren't necessarily required in the final images (although this could 
be solved with a good set of split-rules), and if edits need to be made to the repo, the shared cache can't 
really be used to speed up builds (i.e. all micro-services would need to be rebuilt, even if the sub-project 
they depend on hasn't really changed).

2. Have a separate element for each build artefact which each has its own ref to the monorepo

  +---------------------+   +---------------+
  |                     |   |               |
  | protobuf-python.bst <---+ service-a.bst |
  |                     |   |               |
  +---------------------+   +---------------+

  +-------------------+     +---------------+
  |                   |     |               |
  | protobuf-ruby.bst <-----+ service-b.bst |
  |                   |     |               |
  +-------------------+     +---------------+

    This solves the two problems above, but I have to make constant efforts or create some tooling to make 
sure all the refs remain in sync.

3. Have some mechanism to propagate sources from a parent element which contains the ref to the monorepo to 
build elements

                            +---------------------+   +---------------+
                            |                     |   |               |
                          +-+ protobuf-python.bst <---+ service-a.bst |
  +------------------+    | |                     |   |               |
  |                  |    | +---------------------+   +---------------+
  | protobuf-src.bst <----+
  |                  |    | +-------------------+     +---------------+
  +------------------+    | |                   |     |               |
                          +-+ protobuf-ruby.bst <-----+ service-b.bst |
                            |                   |     |               |
                            +-------------------+     +---------------+


    With this method I only have a single ref in my project, and I can excluded unnecessary build artefacts 
from micro-services easily. While everything downstream will need built on ref change, at least developers 
can open workspaces on the build elements to test local changes without rebuilding unrelated artefacts.

Number 3 might be my favourite at the moment, and I have a POC for it where the .bst files would look 
something like this:

protobuf-src.bst:
kind: stack
sources:
- kind: git
  url: https://github.com/protocolbuffers/protobuf.git
  track: v3.6.1.2
  ref: 9c793fc9e80ce6287bfe6f6cdf349055d1477406

protobuf-python.bst
kind: distutils
sources:
- kind: element
  filename: protobuf-src.bst
build-depends:
- ...

With the source plugin looking something like:

from buildstream import Source, Consistency
from buildstream._stream import Stream
from datetime import datetime

class ElementSource(Source):
    name = 'element'

    def configure(self, node):
        config_keys = ['filename']
        self.node_validate(node, config_keys + Source.COMMON_CONFIG_KEYS)
        self.element_name = self.node_get_member(node, str, 'filename')

        stream = Stream(self._get_context(), self._get_project(), datetime.now())

        self.elem = stream.load_selection((self.element_name,))[0]

    def preflight(self):
        pass

    def load_ref(self, node):
        pass

    def get_ref(self):
        return None

    def set_ref(self, ref, node):
        pass

    def fetch(self):
        pass

    def get_consistency(self):
        return Consistency.CACHED

    def get_unique_key(self):
        self.elem._update_state()
        return self.elem._get_cache_key()

    def stage(self, directory):
        with self.timed_activity("Checking out source from {} at {}"
                                 .format(self.element_name, directory)):
            self.elem._stage_sources_at(directory)


def setup():
    return ElementSource


Obviously this code has some rough edges, but it basically does what I want (although 'bst track' on elements 
that use this does not work correctly). That said, I know I'm not using the Plugin API in the way it was 
intended, so I'm looking for someone to show me a better way of implementing this or to convince me this is a 
bad idea.

Thanks!

Matt Yates