[BuildStream] BuildStream import timings

[Angelos Evripiotis <angelos evripiotis gmail com>]

Hello list,

One of the issues raised during the BuildStream 2019 gathering is the startup time

of BuildStream.

A new feature added in Python 3.7 is the '-X importtime' option:

https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPROFILEIMPORTTIME

I didn't find any obvious things to do to improve  the results, I'm posting here

in the hope that someone else may have good  ideas :)

It seems the protos stuff is contributing a bunch of startup time, there seems to be some blame hanging around pkg_resources. 

Apart from that it seems that there are lots of relatively small imports, 465 imports total.

Here are the results when running natively on my Mac, sorted by cumulative

import time:

    time PYTHONPROFILEIMPORTTIME=true bst help 2>&1 | grep '^import time' | sort -rk5 | head

    import time: self [us] | cumulative | imported package

    import time:       301 |     235602 | buildstream._frontend

    import time:       601 |     201806 |   buildstream

    import time:       756 |     148726 |     buildstream.utils

    import time:      2923 |     106247 |       buildstream._protos.build.bazel.remote.execution.v2.remote_execution_pb2

    import time:       389 |      85448 |         google.protobuf.internal

    import time:       712 |      85060 |           google.protobuf

    import time:     19140 |      84348 |             pkg_resources

    import time:       486 |      47073 | buildstream._frontend.app

    import time:       642 |      43145 |   buildstream._project

    (Total is around 361000 us)

    real    0m0.361s

    user    0m0.291s

    sys     0m0.073s

Sorted by time in module being imported:

    time PYTHONPROFILEIMPORTTIME=true bst help 2>&1 | grep '^import time' | sort -rk3 | head

    import time: self [us] | cumulative | imported package

    import time:     19263 |      87465 |             pkg_resources

    import time:     15934 |      16935 |                   pkg_resources._vendor.pyparsing

    import time:      8446 |      27014 |               pkg_resources.extern.packaging.requirements

    import time:      7453 |       7651 |               pkg_resources.extern.packaging.specifiers

    import time:      5057 |       6121 |             jinja2.lexer

    import time:      4949 |       4949 |         _hashlib

    import time:      4324 |      18574 | site

    import time:      4292 |       5609 |           grpc._cython.cygrpc

    import time:      3349 |       5124 |                     ssl

    (Total is around 379000 us)

    real    0m0.379s

    user    0m0.295s

    sys     0m0.080s

Note that if you try this locally, you may find that 'buildstream' shows up as

taking a long time to import by itself.

This may be because you have installed it with 'pip install -e .', in which

case versioneer will be using git to determine the version number. I found that

this accounted for about 0.15s of my startup time in that case:

    import time:    147968 |     348927 |   buildstream

Cheers!

Angelos