Re: lazy loading override classes

[Tomeu Vizoso <tomeu tomeuvizoso net>]

On 12 August 2013 09:40, Simon Feltman <s feltman gmail com> wrote:
> On Fri, Aug 9, 2013 at 10:23 PM, Tomeu Vizoso <tomeu tomeuvizoso net> wrote:
> > Just wanted to point out that I got the Sugar shell running
> > (experimentally) on top of pygi a few years ago and startup was much
> > faster and far less memory was consumed. This is to say that whatever
> > PyGObject is doing today that slows this down isn't really that
> > necessary.
>
> Hi Tomeu,
>
> A large portion of the Gtk overrides are probably not necessary. But
> in the case of GLib and GObject we've started using overrides as a
> place for shims needed after the removal of static bindings. Some of
> which are simply for compatibility (and give deprecation warnings) and
> others are needed just to get things working.

So some could be made redundant by fixes in glib itself?

> There have been a few efforts to get rid of overrides which are simply
> mechanical and can be generalized. You can see the following patterns
> looking at the Gtk overrides:
>
> 1. Initializer overrides, these generally add a keyword arguments
> which are passed along as a constructor property. I've logged [1] and
> advocate for removing them. These account for nearly half of the class
> overrides which are what cause a large amount of our performance
> problems. For example:
> class HScrollbar(Gtk.HScrollbar):
>     def __init__(self, adjustment=None, **kwds):
>         Gtk.HScrollbar.__init__(self, adjustment=adjustment, **kwds)
>
> This generally buys us nothing except the ability to create the object
> without specifying the keyword (and a pydoc):
>     scrollbar = HScrollbar(adjust)

Isn't this desired just for compatibility with PyGTK? Why this would
be otherwise desired is beyond me (saving typing is not a good reason
IMO).

> 2. Tail argument defaults. These are things like user_data to
> callbacks which we supply None as a default or simply where the
> argument might support allow-none:
>     def begins_tag(self, tag=None):
>         return super(TextIter, self).begins_tag(tag)
>
> These will go away with [2]

Great!

> 3. Default of -1 for text length:
>     def set_text(self, text, length=-1):
>         Gtk.TextBuffer.set_text(self, text, length)
>
> I think these are a necessary evil because exposing the length arg is
> potentially dangerous and fixing the annotations will break API for
> other languages. I had attempted a removal of some of these but it
> didn't work out [3].

What about adding more binding-friendly API replacements to the libraries?

> 4. Argument type coercion:
>     def get_cell_area(self, path, column=None):
>         if not isinstance(path, Gtk.TreePath):
>             path = TreePath(path)
>         return super(TreeView, self).get_cell_area(path, column)
>
> These are particularly annoying because they definitely fall into the
> [lack of] consistency pitfall. e.g. Some functions may coerce tuples
> into a TreePath while others don't. Consistency can at least be solved
> mechanically and the overrides removed with [4].

Sounds like papering over deficiencies in the APIs.

> 5. Stripping boolean return results:
> class ComboBox(Gtk.ComboBox, Container):
>     get_active_iter = strip_boolean_result(Gtk.ComboBox.get_active_iter)
>
> These are methods which return a boolean result that isn't necessary
> and are stripped off the return tuple. These should really be marked
> as skip in annotations but again this would break API [5]

More papering over warts in the APIs. Doing this in each binding looks
quite wrong to me.

> 6. Pythonic and convenience overrides: These can be anything from
> supplying "__iter__" to implementing a vaargs method in Python which
> exists in the C API and cannot be exposed through GI. In some cases
> the C API might be more convenient than what GI provides or we
> basically package a function which everyone would have to write
> anyway.
>
> In terms of "Pythonic" overrides, I think they give the Python
> bindings an edge over other GI based bindings. Things like decorators,
> iterators, and context managers allow us to bridge idiomatic Python to
> GI which makes the Python programmer side of me excited about being a
> user of the bindings.

Personally (not that it matters much), I don't care that much about
this pythonness. The positive impact that it may have on my code is
massively negated by the problems that overrides bring.

> > Today, I would suggest again that compatibility overrides should be
> > optionally loaded by those users that can afford the penalty, and that
> > the supported API should be just what PyGObject can offer by relying
> > solely on the typelibs and GType. This is not only because of
> > performance, but also because of documentability, consistency and
> > maintainability.
>
> I agree in terms of documentation and consistency but the actual code
> maintenance of overrides is very small compared to problems like multiple marshaling paths [6]

But you have taken care of this already, right?

> or the static aspects of GLib and
> GObject [7] (which also suffer the same documentation and consistency
> problems as overrides).

Yes, pretty much the same problem, but worst :)

> Having the ability to load typelibs without overrides seems fine. It
> also brings up a secondary concern I have with the architecture if we
> ever decide to package Gtk overrides separately from PyGObject. The
> problem is an app will not know if overrides are installed because
> using "from gi.repository import Gtk" may bring in the pure typelib or
> the overridden version and things could get ugly without an explicit
> import distinction. Basically I think we should do something along
> with lines of requiring override packages to use something like
> "import Gtk" which registers the overrides with PyGObject and "import
> gi.<something>.Gtk" can be guaranteed as a pure typelib. We actually
> already have "gi.module.get_introspection_module('Gtk')", but I
> believe this will breakdown during marshaling and the overrides will
> end up being loaded anyway.

Yes, explicit is good.

> > When I think of the overrides that we ship now as part of PyGObject, I
> > cannot but wonder if we have really learned the lessons from the years
> > we had to use and maintain the static bindings.
>
> It might sound a little grumpy, but I have honestly never seen a
> re-write of anything work out very well and instead we end up with the
> usual suspects: a new set of problems specific to the new
> implementation, lots of buggy new code, a large amount of
> incompatibility, and a lack of maintenance on the old system which
> ends up in continued usage anyway. Contrary to this, I think all of
> people that have worked on the bindings have done a really good job in
> regards to integration and transition into GI, we just need to
> continue working things out, deprecating things, refactoring, fixing
> performance, testing, etc...

Sure, I think users of Python and GNOME are (or at least should be!)
quite happy with how things have turned out and this community hasn't
been as vibrant as today, probably.

> I think pygtkcompat would have been a
> better approach from the start (as opposed to overrides for PyGTK
> compatibility). But as mentioned earlier, overrides are a minor
> maintenance burden compared to the C side of the bindings.

Are you taking into account the maintenance of distro packages? One of
the worst problems with the static bindings approach was the explosive
combination of libraries, bindings and distros. So someone makes an
API addition to a library, and it has to be added to M bindings and
then packaged to N distros.

The result was that users could only rely on that API addition being
available to users after a few years (think of LTS releases).

The main goal of the move to introspection regarding maintenance was
that once library authors added API (with an eye to bindability), it
would be available to all application authors without requiring N x M
tasks.

Documentation also requires N x M tasks if we have overrides.

Keep up the great work,

Tomeu

> -Simon
>
> [1] https://bugzilla.gnome.org/show_bug.cgi?id=705810
> [2] https://bugzilla.gnome.org/show_bug.cgi?id=640812
> [3] https://bugzilla.gnome.org/show_bug.cgi?id=686264
> [4] https://bugzilla.gnome.org/show_bug.cgi?id=686261
> [5] https://bugzilla.gnome.org/show_bug.cgi?id=690904
> [6] https://bugzilla.gnome.org/show_bug.cgi?id=693405
> [7] https://bugzilla.gnome.org/show_bug.cgi?id=685373