Re: Poor performance with large ListStore/TreeView



As Christoph's example shows, using insert_with_valuesv should speed
things up (I'm seeing about a 6x gain):
    l.insert_with_valuesv(-1, range(len(items)), items)

Another option is to implement your own TreeModel. It may be a bit
more tedious but it could be faster. There isn't really an insertion
time per-say, but unfortunately setting the model or removing it from
the view (exiting the app) seems to exhaust the "do_iter_next" virtual
function which slows it down more than I'd hoped. I've added a small
demo to PyGObject [1], but you still need to implement the
Gtk.TreeSortable interface if you want sorting. For more complete
examples, you might have a look through Gramps [2]

In either case setting the "fixed-height-mode" of the TreeView will
avoid the view attempting to pull the value and height from each item
in the model to calculate the scroll space [3]. You also need to set
Gtk.TreeViewColumnSizing.FIXED as the column sizing property when
using this.

-Simon

[1] https://git.gnome.org/browse/pygobject/tree/demos/gtk-demo/demos/TreeView/treemodel_large.py
[2] http://sourceforge.net/p/gramps/source/ci/master/tree/gramps/gui/views/treemodels/treebasemodel.py
[3] https://developer.gnome.org/gtk3/3.10/GtkTreeView.html#GtkTreeView--fixed-height-mode

On Mon, Jan 13, 2014 at 9:23 PM, Jonathan Ballet <jon multani info> wrote:
Hi,

I'm developing an application using PyGObject and Python 3 for the Music
Player Daemon (https://github.com/multani/sonata/) and I've got several
users who are complaining about the lack of performance of the
application when loading lot of song into our "Current" playlist (which
is a Gtk.TreeView). Especially compared to other MPD clients which are
nearly instantaneous.

I'm going to describe how the application currently works, the problem
I'm facing and possible solutions I'm exploring. It's a bit long, so...

tl;dr: how to efficiently display (lot of) information into a TreeView?


Current implementation
----------------------

* I get a list of Song objects with a number of attributes like .album,
  .artist, etc.

* a user can configure which columns he wants to see in the player with
  a formatting mini-language. For example, he can set the formatting to
  be "%N|%T|%A|%Y - %B - %N" which is going to create a tree view
  with 4 columns:
    - the 1st one will have the track number (%N)
    - the 2nd one will have the track name (%T)
    - the 3rd one will have the artist name (%A)
    - the last one will be a string of the form "Year - Album - TrackNumber"

* I have a Gtk.ListStore which is built from the definition of the
  columns above and which stores the following:
    - a Song object
    - for each columns defined above, the song formatted according to
      the user configuration
    - an additional column to set a row to bold or not (to see the
      current song)

Once we get the songs to be displayed, the application loops over all
the songs and:

* applies a formatting function to the current song which returns a list
  of formatted values
* stores in the ListStore the song + the formatted values (+ the
  "bold" attribute)


Current performances
--------------------

I made a small test case to reproduce the problem (attached to this
email), and here are the results I've got:

* it takes about 1.5 sec. to feed the ListStore with Python 2 and pygtk,
  with about half of the time spent into ListStore.append

* it takes about 15 sec. to run the same with Python 2/3 and PyGObject
  where about more that 90% of the time spent into ListStore.append
  (78% in TreeModel._convert_value)

[I got those values using:

 * python2 -m cProfile -o profile.profile22 test.py -2
 * python3 -m cProfile -o profile.profile33 testpy

 and using pyprof2calltree and KCacheGrind to get the percentage]

In the real application, it's actually much longer due to the more
complex formatting functions and inefficient access to the Song object
attributes, but PyGObject always comes first with more thatn 60% of the
time inside.


Possible solutions?
-------------------

I tried several approaches to solve this problem:

* I tried to populate the ListStore by chunks instead of doing it in one
  block. Although it doesn't block the UI anymore, it takes about 1
  minute to complete and the experience is not that great. I would also
  need to have more code to handle the case where the content of the
  playlist changes while it's still being populated from the previous
  set.

* I tried to minimize the size of the ListStore by only adding the Song
  object. The goal here was to reduce the cost of the formatting + the
  cost of adding "so many" columns to the ListStore. The formatting was
  then done using `cell_data_func` functions on each of the view's
  columns. It's much faster to append but noticeably slower to display
  and to scroll around.
  I tried to put some cache (either on the formatting side, or by
  having a larger ListStore with the additional columns set to None,
  filled and used after the column value has been computed) but it's
  still not that great. I guess calling the cell_data_func has a high
  cost after all.


I'm a bit out of idea on what else I can try. I can provide more
information about those numbers, especially if you need some profiling
data. I'm all in for more PyGObject performances too, although I don't
where to start in the code base, but I'm willing to try stuff.

 Jonathan
_______________________________________________
python-hackers-list mailing list
python-hackers-list gnome org
https://mail.gnome.org/mailman/listinfo/python-hackers-list


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]