Re: [BuildStream] Performance: cython

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi,

On Thu, 2019-05-23 at 08:55 +0000, Benjamin Schubert wrote:
> Hey Tristan,
[...]
> > From what I understand, cpython is the main interpretor which we
> > expect, and probably if there are installation targets which have
> > other
> > python interpretors, these can be installed in parallel.
>
> Yes
>
> > So if I understand correctly, this is only a concern in terms of
> > features we can consume from other interpretors, but it is not a
> > concern in term of being easily installable on any host, correct ?
>
> Not sure what you mean about "features we can consume from other
> interpretors".

I mean for instance, with Jython (apparently, I don't know enough
whether it's desirable or trustworthy), we don't have the pesky GIL:

    https://wiki.python.org/moin/GlobalInterpreterLock

So requiring a specific interpretor as I understand it, locks us into
the CPython interpretor and it's features and behaviors.

> But if the concern is easy installation, I would not expect this to
> be a problem.

Right: To clarify, I am not concerned with trying to run BuildStream on
alternative python interpretors, just wanted to ensure that hard
requiring a specific interpretor was not going to affect
installability.

[...]
> > Just for clarification, this build step will be incorporated into
> > the
> > regular `python3 setup.py build` phase correct ?
>
> Correct, we will however need "Cython" installed in the environment where
> we build if we haven't pre-generated the C files before.
>
> For this, we should publish to pypi source distributions with pre-generated
> files, and on developers machines just require Cython.

I think that we should have some symmetry in how we use generated
source files.

Either:

  A.) We generate everything every time in the build phase

  B.) We commit generated code to the repository

Currently we commit the gprc generated files to the repository (which
is not really a nice thing to do, but I think was done to make
development easier).

Whatever we choose, we should probably be doing the same thing for all
generated files.

[...]
> This will require a few changes in our setup.py. Mainly what I intend to do:
>
> 1) Require Cython when running "setup.py sdist". That way we ensure that our
> source distributions always contain the C files
> 2) When invoking "setup.py build" or "setup.py test" or "setup.py install", check if we have Cython.
>    - If we have Cython, everything works transparently.
>    - If we don't have Cython, check if we have the generated ".c" files.
>       - If we do, use them and hope they are up to date
>       - Otherwise abort and ask user to install cython.
>
> And otherwise, everything should work transparently.
> Does that seem acceptable to you?

It sounds like a fine approach to me, again refer to the above.

What we do for generating the .c files, we should probably also do for
the generated gprc protos.

Cheers,
    -Tristan