Re: [BuildStream] Nuking the pickle jobber.

[Benjamin Schubert <contact benschubert me>]

Hey all,

Most of the content here is from a IRC discussion this morning see [0] for the whole discussion if you want.

TLDR: We should land !1965, to remove the pickling, then see if we can do without having this
fork system (number one solution is no subprocesses, number two would be possibly a workerserver.)

And a longer explanation:

The reason why we are using processes and forking for the plugin-related calls is two-folds:

- It helps with parallelism, as we don't have the GIL
- It ensures that plugins will not 'damage' the global state.

With many of the recent changes, like `buildbox-run` and `buildbox-casd` usage, lots of the time
spent in the python process itself went down, and thus the need for multiprocessing is less.
And we can have all those calls running in parallel.

For the global state safety, plugins are anyways imported in the main process and could do any
changes they want anyways. The only thing that is 'safe' is if they follow the guidelines and access
only public methods of BuildStream, and this is also safe when run in the main process, so we don't
really need that.


Therefore, and given that the multiprocessing is a problem (FWIW, `fork` and asyncio are also not
supported by python upstream, so we are on our own more or less), it is easier to drop the
pickling for now, and try other approaches that require either:

- No multiprocessing
- Less pickling

We'll therefore drop the pickling for now, and if other investigations are not successful will
review how we can add it in a less intrusive way later.

Let me know if you have any questions, I will be looking around an implementation without
multiprocessing to see whether we can get a similarly-performant nicer looking implementation.

Cheers,
Ben

[0] https://irclogs.baserock.orgbuildstream%23buildstream.2020-06-16.log.html#t2020-06-16T07:20:00