Re: [BuildStream] Proposal: A small number of subprocesses handling jobs

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi Jonathan,

On Fri, 2019-02-22 at 17:04 +0000, Jonathan Maw via BuildStream-list wrote:
> Hi all.
>
> I've been looking at optimisations, and it seems that there is a 
> significant amount
> of time spent in forking off to new processes. (jennis' profile showed 
> 126s in a 576s
> build).

I'm curious how long the builds themselves took to complete in this
scenario.

I.e. a regular build will result in spawning many processes, surely the
overhead of forking the child task for this activity is rather
negligible in the greater scheme of things.

[...]
> Any information passed to an existing subprocess has to be pickled and
> unpickled, so ideally this would need to be as little as possible.
>
> I am not sure how I would go about providing this information, assuming 
> I can track down and isolate all the parts of the pipeline that change during 
> a build.

A job needs access to:

  * The element being processed
  * All of the element's dependencies

Basically, all of the public API exposed to plugins, and all of the
state of all elements which the element depends on, must be readily
handy for the plugin. This contract is rather inherent due to the fact
that an Element has access to all public API and has access to all of
it's dependencies, combined with the fact that the core cannot have any
knowledge whatsoever of what the plugin will do, asides from calling
some public API.

This is why we use a fork rather than a worker pool, to avoid needing
to serialize/deserialize an entire project data model in order to start
processing (as we already know from observing project load times, this
is an expensive activity).

I think that if we are going to change the model for tasks, it should
be first to change over to a threading model rather than a fork model
first.

Once we have a threading model in place, we can then easily take it a
step further and have a thread worker pool instead of spawning threads
on demand (if that makes any impact on performance).

Cheers,
    -Tristan