Re: Poor performance with large ListStore/TreeView

[Simon Feltman <s feltman gmail com>]

As Christoph's example shows, using insert_with_valuesv should speed
things up (I'm seeing about a 6x gain):
    l.insert_with_valuesv(-1, range(len(items)), items)

Another option is to implement your own TreeModel. It may be a bit
more tedious but it could be faster. There isn't really an insertion
time per-say, but unfortunately setting the model or removing it from
the view (exiting the app) seems to exhaust the "do_iter_next" virtual
function which slows it down more than I'd hoped. I've added a small
demo to PyGObject [1], but you still need to implement the
Gtk.TreeSortable interface if you want sorting. For more complete
examples, you might have a look through Gramps [2]

In either case setting the "fixed-height-mode" of the TreeView will
avoid the view attempting to pull the value and height from each item
in the model to calculate the scroll space [3]. You also need to set
Gtk.TreeViewColumnSizing.FIXED as the column sizing property when
using this.

-Simon

[1] https://git.gnome.org/browse/pygobject/tree/demos/gtk-demo/demos/TreeView/treemodel_large.py
[2] http://sourceforge.net/p/gramps/source/ci/master/tree/gramps/gui/views/treemodels/treebasemodel.py
[3] https://developer.gnome.org/gtk3/3.10/GtkTreeView.html#GtkTreeView--fixed-height-mode

On Mon, Jan 13, 2014 at 9:23 PM, Jonathan Ballet <jon multani info> wrote:
> Hi,
>
> I'm developing an application using PyGObject and Python 3 for the Music
> Player Daemon (https://github.com/multani/sonata/) and I've got several
> users who are complaining about the lack of performance of the
> application when loading lot of song into our "Current" playlist (which
> is a Gtk.TreeView). Especially compared to other MPD clients which are
> nearly instantaneous.
>
> I'm going to describe how the application currently works, the problem
> I'm facing and possible solutions I'm exploring. It's a bit long, so...
>
> tl;dr: how to efficiently display (lot of) information into a TreeView?
>
>
> Current implementation
> ----------------------
>
> * I get a list of Song objects with a number of attributes like .album,
>   .artist, etc.
>
> * a user can configure which columns he wants to see in the player with
>   a formatting mini-language. For example, he can set the formatting to
>   be "%N|%T|%A|%Y - %B - %N" which is going to create a tree view
>   with 4 columns:
>     - the 1st one will have the track number (%N)
>     - the 2nd one will have the track name (%T)
>     - the 3rd one will have the artist name (%A)
>     - the last one will be a string of the form "Year - Album - TrackNumber"
>
> * I have a Gtk.ListStore which is built from the definition of the
>   columns above and which stores the following:
>     - a Song object
>     - for each columns defined above, the song formatted according to
>       the user configuration
>     - an additional column to set a row to bold or not (to see the
>       current song)
>
> Once we get the songs to be displayed, the application loops over all
> the songs and:
>
> * applies a formatting function to the current song which returns a list
>   of formatted values
> * stores in the ListStore the song + the formatted values (+ the
>   "bold" attribute)
>
>
> Current performances
> --------------------
>
> I made a small test case to reproduce the problem (attached to this
> email), and here are the results I've got:
>
> * it takes about 1.5 sec. to feed the ListStore with Python 2 and pygtk,
>   with about half of the time spent into ListStore.append
>
> * it takes about 15 sec. to run the same with Python 2/3 and PyGObject
>   where about more that 90% of the time spent into ListStore.append
>   (78% in TreeModel._convert_value)
>
> [I got those values using:
>
>  * python2 -m cProfile -o profile.profile22 test.py -2
>  * python3 -m cProfile -o profile.profile33 testpy
>
>  and using pyprof2calltree and KCacheGrind to get the percentage]
>
> In the real application, it's actually much longer due to the more
> complex formatting functions and inefficient access to the Song object
> attributes, but PyGObject always comes first with more thatn 60% of the
> time inside.
>
>
> Possible solutions?
> -------------------
>
> I tried several approaches to solve this problem:
>
> * I tried to populate the ListStore by chunks instead of doing it in one
>   block. Although it doesn't block the UI anymore, it takes about 1
>   minute to complete and the experience is not that great. I would also
>   need to have more code to handle the case where the content of the
>   playlist changes while it's still being populated from the previous
>   set.
>
> * I tried to minimize the size of the ListStore by only adding the Song
>   object. The goal here was to reduce the cost of the formatting + the
>   cost of adding "so many" columns to the ListStore. The formatting was
>   then done using `cell_data_func` functions on each of the view's
>   columns. It's much faster to append but noticeably slower to display
>   and to scroll around.
>   I tried to put some cache (either on the formatting side, or by
>   having a larger ListStore with the additional columns set to None,
>   filled and used after the column value has been computed) but it's
>   still not that great. I guess calling the cell_data_func has a high
>   cost after all.
>
>
> I'm a bit out of idea on what else I can try. I can provide more
> information about those numbers, especially if you need some profiling
> data. I'm all in for more PyGObject performances too, although I don't
> where to start in the code base, but I'm willing to try stuff.
>
>  Jonathan
> _______________________________________________
> python-hackers-list mailing list
> python-hackers-list gnome org
> https://mail.gnome.org/mailman/listinfo/python-hackers-list