Hi Daniel,
I am not opposed to porting parts of the bindings to C for optimization purposes. However, from my experience, coming up with better caching strategies, using lazy evaluation, and really questioning the mechanics from a higher level can sometime be much more effective than just porting from one language to another. I would also like to see analysis showing profiles and attempts at using a number of Python optimizations and movement of smaller discrete functions to C where possible. I don't think the path should be a before and after showing "it's faster when this large amount code is ported to C".
In terms of maintenance cost, it seems we are in a constant battle trying to fix memory and reference leaks on the C side of things [1]. So this definitely biases me towards wanting to keep things in Python and in fact move more C over to Python [2]. Again I am not opposed, I would just like to see it done in such a way that peer reviewable/reproducible analysis is available and an the attempt is made with small patches which can stand on their own to achieve the overall goal. This allows for:
1. Easier review
2. More focus on unit testing
3. Less chance of regression
Additionally it would be helpful to see a larger profile of app launch times in Sugar. Just isolating the importing of Gtk may bias the optimization focus towards a module heavy with Python overrides, which might not be want you want?
Doing a quick profile and taking a look at module.py and types.py, a number of things stick out:
$ cat profile_test.py
from gi.repository import GObject, Gio, GLib, Gtk, Gdk, Pango, Clutter, Gst
$ python -m cProfile -s tottime profile_test.py
93580 function calls (92149 primitive calls) in 0.106 seconds
Ordered by: internal time
ncalls tottime cumtime filename:lineno(function)
46 0.021 0.021 {method 'invoke' of 'gi.CallableInfo' objects}
2436 0.008 0.019 types.py:72(get_callable_info_doc_string)
11293 0.005 0.005 {method 'get_name' of 'gi.BaseInfo' objects}
2436 0.005 0.026 types.py:101(update_func)
185/162 0.004 0.053 module.py:130(__getattr__)
287 0.003 0.003 {method 'get_g_type' of 'gi.RegisteredTypeInfo' objects}
2436 0.003 0.005 types.py:53(split_function_info_args)
131 0.003 0.033 types.py:151(_setup_methods)
2317 0.002 0.027 types.py:110(Function)
2482 0.002 0.002 {method 'get_pytype_hint' of 'gi.ArgInfo' objects}
Line by line, I will be adding the related tickets as dependencies of this tracking ticket [3].
46 0.021 0.021 {method 'invoke' of 'gi.CallableInfo' objects}
This might be a performance regression caused by "type_from_name" being moved from static C bindings to introspection, it is used in the GObject overrides to create the series of pre-defined "TYPE_XYZ" module variables [4]. Simple enough to move type_from_name back to C if it helps. Added ticket [5].
2436 0.008 0.019 types.py:72(get_callable_info_doc_string)
2436 0.003 0.005 types.py:53(split_function_info_args)
This is __doc__ string creation code. This could be moved to C but a much easier optimization would be to make it lazy. So the code is only executed when the __doc__ string is accessed (and then cached). Logged as an addition to [6].
11293 0.005 0.005 {method 'get_name' of 'gi.BaseInfo' objects}
A handful of load time code locations call this. A number of these calls would go away with lazy __doc__ evaluation and additional optimizations described below:
2436 0.005 0.000 0.026 0.000 types.py:101(update_func)
This is used as a decorator for wrapping every function/method brought in by gi. The good news is it can completely go away if we want. By providing the double-under attributes set on function objects directly on the info struct, we would no longer need the wrapped function objects. Consider the following [7]:
Foo.bar = Function(bar_info)
becomes
Foo.bar = bar_info
Not only would this cut the number of function objects in half, it might also optimize call times because we remove an additional indirection of python calls. Logged as [6]
All of this takes us back to __getattr__ being on the top of a profile again. In which case we should ask why is it getting called at all? A simple print statement shows without a doubt every access during module load is due to that attr being overridden. This goes back to the question of using the importing of Gtk as our performance measure as it will bias the optimization effort to a module heavy with overrides. Do Sugar apps also import many non-overridden modules?
There are also some larger sweeping optimizations described in [8] and [9]. Likewise we currently create wrappers for GObject class objects for everything in a module, i.e. WidgetClass, ButtonClass... However, I don't think these are even useable so we should skip them until [10] is resolved (which may also allow avoidance of them).
-Simon
