What to do now ?
~~~~~~~~~~~~~~~~
I am personally a bit torn here, I want other developers to weigh in on
this.
The facts on the ground are:
* This is not a problem for Mac, we can always forcefully set
multiprocessing to use the "fork" method.
* This is a blocker for win32 native support, and that is huge,
I don't want to lose track of this, bringing it back will also
be difficult after ditching it.
* I don't personally believe this should be a blocker for the
activity of subprocessing the scheduler. We should be able to
get away with a single (possibly expensive on WSL) fork() early
on and be able to do regular IPC over Queues to meet our needs.
This was never expected to be "easy" anyway.
And our options are:
1) Get rid of this serialization stuff
+0. I would certainly be supportive of this, given your description and analysis.
2) Refactor/reimplement the serialization stuff to reduce the cost
of ownership, and reduce the burden of knowledge required to
generally write BuildStream code.
This also requires solving the issue of instantiating plugins
in subprocesses via the "spawn" method (which does not work for
plugins loaded through the new "junction" origin).
If we decide to take this route, I would probably still try to
resolve the plugin instantiation issue first in order to land
long standing !1901 ASAP.
+1. I would consider roadmapping this for 2.0 depending on whether it would break the plugin API.
3) Solve the issue of instantiating plugins in subprocesses via the
"spawn" method without any refactoring of object serialization.
As I've pretty much stated, I feel strongly opposed to this
approach, I don't think the current code is viable.
Agreed on basis of the maintenance burden/complexity.
Keeping in mind that I've been working on !1901[2] for a long time now
(on and off while we finally agree on a design) and that this very high
priority branch is blocking on a decision, where should I be spending
my time ?
My $0.02:
- File an issue to implement serialization as you outlined, tag it 2.0
- Land !1965
- Refocus on !1901
HTH.
Cheers,
Sander
Best Regards,
-Tristan
[0]: https://www.python.org/dev/peps/pep-0484/
[1]: https://gitlab.com/BuildStream/buildstream/-/merge_requests/1463#note_194278834
[2]: https://gitlab.com/BuildStream/buildstream/-/merge_requests/1901/
