Re: Slow loading, moving base code to C



Hi Daniel,

I am not opposed to porting parts of the bindings to C for optimization purposes. However, from my experience, coming up with better caching strategies, using lazy evaluation, and really questioning the mechanics from a higher level can sometime be much more effective than just porting from one language to another. I would also like to see analysis showing profiles and attempts at using a number of Python optimizations and movement of smaller discrete functions to C where possible. I don't think the path should be a before and after showing "it's faster when this large amount code is ported to C".

In terms of maintenance cost, it seems we are in a constant battle trying to fix memory and reference leaks on the C side of things [1]. So this definitely biases me towards wanting to keep things in Python and in fact move more C over to Python [2]. Again I am not opposed, I would just like to see it done in such a way that peer reviewable/reproducible analysis is available and an the attempt is made with small patches which can stand on their own to achieve the overall goal. This allows for:

1. Easier review
2. More focus on unit testing
3. Less chance of regression

Additionally it would be helpful to see a larger profile of app launch times in Sugar. Just isolating the importing of Gtk may bias the optimization focus towards a module heavy with Python overrides, which might not be want you want?

Doing a quick profile and taking a look at module.py and types.py, a number of things stick out:

$ cat profile_test.py 
from gi.repository import GObject, Gio, GLib, Gtk, Gdk, Pango, Clutter, Gst

$ python -m cProfile -s tottime profile_test.py
         93580 function calls (92149 primitive calls) in 0.106 seconds

   Ordered by: internal time

   ncalls  tottime  cumtime filename:lineno(function)
       46    0.021    0.021 {method 'invoke' of 'gi.CallableInfo' objects}
     2436    0.008    0.019 types.py:72(get_callable_info_doc_string)
    11293    0.005    0.005 {method 'get_name' of 'gi.BaseInfo' objects}
     2436    0.005    0.026 types.py:101(update_func)
  185/162    0.004    0.053 module.py:130(__getattr__)
      287    0.003    0.003 {method 'get_g_type' of 'gi.RegisteredTypeInfo' objects}
     2436    0.003    0.005 types.py:53(split_function_info_args)
      131    0.003    0.033 types.py:151(_setup_methods)
     2317    0.002    0.027 types.py:110(Function)
     2482    0.002    0.002 {method 'get_pytype_hint' of 'gi.ArgInfo' objects}
...

Line by line, I will be adding the related tickets as dependencies of this tracking ticket [3].

46    0.021    0.021 {method 'invoke' of 'gi.CallableInfo' objects}

   This might be a performance regression caused by "type_from_name" being moved from static C bindings to introspection, it is used in the GObject overrides to create the series of pre-defined "TYPE_XYZ" module variables [4]. Simple enough to move type_from_name back to C if it helps. Added ticket [5].


2436    0.008    0.019 types.py:72(get_callable_info_doc_string)
2436    0.003    0.005 types.py:53(split_function_info_args)

    This is __doc__ string creation code. This could be moved to C but a much easier optimization would be to make it lazy. So the code is only executed when the __doc__ string is accessed (and then cached). Logged as an addition to [6].


11293    0.005    0.005 {method 'get_name' of 'gi.BaseInfo' objects}

    A handful of load time code locations call this. A number of these calls would go away with lazy __doc__ evaluation and additional optimizations described below:


2436    0.005    0.000    0.026    0.000 types.py:101(update_func)

    This is used as a decorator for wrapping every function/method brought in by gi. The good news is it can completely go away if we want. By providing the double-under attributes set on function objects directly on the info struct, we would no longer need the wrapped function objects. Consider the following [7]:

Foo.bar = Function(bar_info)
    becomes
Foo.bar = bar_info

Not only would this cut the number of function objects in half, it might also optimize call times because we remove an additional indirection of python calls. Logged as [6]

All of this takes us back to __getattr__ being on the top of a profile again. In which case we should ask why is it getting called at all? A simple print statement shows without a doubt every access during module load is due to that attr being overridden. This goes back to the question of using the importing of Gtk as our performance measure as it will bias the optimization effort to a module heavy with overrides. Do Sugar apps also import many non-overridden modules?

There are also some larger sweeping optimizations described in [8] and [9]. Likewise we currently create wrappers for GObject class objects for everything in a module, i.e. WidgetClass, ButtonClass... However, I don't think these are even useable so we should skip them until [10] is resolved (which may also allow avoidance of them).

[1] https://bugzilla.gnome.org/show_bug.cgi?id=693111
[2] https://bugzilla.gnome.org/show_bug.cgi?id=685373
[3] https://bugzilla.gnome.org/show_bug.cgi?id=693243
[4] https://git.gnome.org/browse/pygobject/tree/gi/overrides/GObject.py?id=3.8.3#n110
[5] https://bugzilla.gnome.org/show_bug.cgi?id=704023
[6] https://bugzilla.gnome.org/show_bug.cgi?id=704037
[7] https://git.gnome.org/browse/pygobject/tree/gi/types.py?id=3.8.3#n98
[8] https://bugzilla.gnome.org/show_bug.cgi?id=640650
[9] https://bugzilla.gnome.org/show_bug.cgi?id=640649
[10] https://bugzilla.gnome.org/show_bug.cgi?id=685218

-Simon




On Wed, Jul 10, 2013 at 2:50 PM, Daniel Drake <dsd laptop org> wrote:
Hi,

Since Sugar was ported from GTK2/pygtk to GTK3/pygobject3, the desktop
and apps are considerably slower to launch. I'm trying to correct
this. You can see at least part of this slowdown by timing the
following 2 programs:

1. import gtk

2. from gi.repository import Gtk

cProfile shows that the biggest obvious offender is the getattr
implementation of IntrospectionModule, eats up quite a bit of CPU
time.

To get a feel for whether this would speed up if it were written in C,
I started to move the code into the _gi module. After starting to do
this, I think the answer is definitely yes. Most of its work is
getting basic info from GI and in order to that it crosses the
Python-C barrier many times, and all of the GI elements are wrapped in
their own python objects, etc. If it were all done in C and just
called into GI directly things would be faster and more direct.

I would be interested in finishing this work and the translation is
not that difficult, but I want to ask for some opinions first. Would
this kind of code movement be accepted?

Another consideration is how far to go. getattr instantiates
GObjectMeta objects and things like at. At first I would leave
GObjectMeta this in Python but we can measure later, maybe that would
also benefit from being done in C.

Daniel
_______________________________________________
python-hackers-list mailing list
python-hackers-list gnome org
https://mail.gnome.org/mailman/listinfo/python-hackers-list



[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]