Re: lazy loading override classes

[Simon Feltman <s feltman gmail com>]

On Fri, Aug 9, 2013 at 10:23 PM, Tomeu Vizoso <tomeu tomeuvizoso net> wrote:
> Just wanted to point out that I got the Sugar shell running
> (experimentally) on top of pygi a few years ago and startup was much
> faster and far less memory was consumed. This is to say that whatever
> PyGObject is doing today that slows this down isn't really that
> necessary.

Hi Tomeu,

A large portion of the Gtk overrides are probably not necessary. But
in the case of GLib and GObject we've started using overrides as a
place for shims needed after the removal of static bindings. Some of
which are simply for compatibility (and give deprecation warnings) and
others are needed just to get things working.

There have been a few efforts to get rid of overrides which are simply
mechanical and can be generalized. You can see the following patterns
looking at the Gtk overrides:

1. Initializer overrides, these generally add a keyword arguments
which are passed along as a constructor property. I've logged [1] and
advocate for removing them. These account for nearly half of the class
overrides which are what cause a large amount of our performance
problems. For example:
class HScrollbar(Gtk.HScrollbar):
    def __init__(self, adjustment=None, **kwds):
        Gtk.HScrollbar.__init__(self, adjustment=adjustment, **kwds)

This generally buys us nothing except the ability to create the object
without specifying the keyword (and a pydoc):
    scrollbar = HScrollbar(adjust)

2. Tail argument defaults. These are things like user_data to
callbacks which we supply None as a default or simply where the
argument might support allow-none:
    def begins_tag(self, tag=None):
        return super(TextIter, self).begins_tag(tag)

These will go away with [2]

3. Default of -1 for text length:
    def set_text(self, text, length=-1):
        Gtk.TextBuffer.set_text(self, text, length)

I think these are a necessary evil because exposing the length arg is
potentially dangerous and fixing the annotations will break API for
other languages. I had attempted a removal of some of these but it
didn't work out [3].

4. Argument type coercion:
    def get_cell_area(self, path, column=None):
        if not isinstance(path, Gtk.TreePath):
            path = TreePath(path)
        return super(TreeView, self).get_cell_area(path, column)

These are particularly annoying because they definitely fall into the
[lack of] consistency pitfall. e.g. Some functions may coerce tuples
into a TreePath while others don't. Consistency can at least be solved
mechanically and the overrides removed with [4].

5. Stripping boolean return results:
class ComboBox(Gtk.ComboBox, Container):
    get_active_iter = strip_boolean_result(Gtk.ComboBox.get_active_iter)

These are methods which return a boolean result that isn't necessary
and are stripped off the return tuple. These should really be marked
as skip in annotations but again this would break API [5]

6. Pythonic and convenience overrides: These can be anything from
supplying "__iter__" to implementing a vaargs method in Python which
exists in the C API and cannot be exposed through GI. In some cases
the C API might be more convenient than what GI provides or we
basically package a function which everyone would have to write
anyway.

In terms of "Pythonic" overrides, I think they give the Python
bindings an edge over other GI based bindings. Things like decorators,
iterators, and context managers allow us to bridge idiomatic Python to
GI which makes the Python programmer side of me excited about being a
user of the bindings.

> Today, I would suggest again that compatibility overrides should be
> optionally loaded by those users that can afford the penalty, and that
> the supported API should be just what PyGObject can offer by relying
> solely on the typelibs and GType. This is not only because of
> performance, but also because of documentability, consistency and
> maintainability.

I agree in terms of documentation and consistency but the actual code
maintenance of overrides is very small compared to problems like
multiple marshaling paths [6] or the static aspects of GLib and
GObject [7] (which also suffer the same documentation and consistency
problems as overrides).

Having the ability to load typelibs without overrides seems fine. It
also brings up a secondary concern I have with the architecture if we
ever decide to package Gtk overrides separately from PyGObject. The
problem is an app will not know if overrides are installed because
using "from gi.repository import Gtk" may bring in the pure typelib or
the overridden version and things could get ugly without an explicit
import distinction. Basically I think we should do something along
with lines of requiring override packages to use something like
"import Gtk" which registers the overrides with PyGObject and "import
gi.<something>.Gtk" can be guaranteed as a pure typelib. We actually
already have "gi.module.get_introspection_module('Gtk')", but I
believe this will breakdown during marshaling and the overrides will
end up being loaded anyway.

> When I think of the overrides that we ship now as part of PyGObject, I
> cannot but wonder if we have really learned the lessons from the years
> we had to use and maintain the static bindings.

It might sound a little grumpy, but I have honestly never seen a
re-write of anything work out very well and instead we end up with the
usual suspects: a new set of problems specific to the new
implementation, lots of buggy new code, a large amount of
incompatibility, and a lack of maintenance on the old system which
ends up in continued usage anyway. Contrary to this, I think all of
people that have worked on the bindings have done a really good job in
regards to integration and transition into GI, we just need to
continue working things out, deprecating things, refactoring, fixing
performance, testing, etc... I think pygtkcompat would have been a
better approach from the start (as opposed to overrides for PyGTK
compatibility). But as mentioned earlier, overrides are a minor
maintenance burden compared to the C side of the bindings.

-Simon

[1] https://bugzilla.gnome.org/show_bug.cgi?id=705810
[2] https://bugzilla.gnome.org/show_bug.cgi?id=640812
[3] https://bugzilla.gnome.org/show_bug.cgi?id=686264
[4] https://bugzilla.gnome.org/show_bug.cgi?id=686261
[5] https://bugzilla.gnome.org/show_bug.cgi?id=690904
[6] https://bugzilla.gnome.org/show_bug.cgi?id=693405
[7] https://bugzilla.gnome.org/show_bug.cgi?id=685373