Re: GtkTreeView with huge amount of rows; dynamic data source?

[Greg Breland <gbreland mozillanews org>]

I too am going to need the exact same functionality out of a liststore. 
I've put a lot of thought into my requirements but not much time yet
into solutions.

On Thu, 2005-01-13 at 07:05, Gus Koppel wrote:
> Preparing a GtkListStore (or even GtkTreeStore) in memory and prefill it
> with, say, 100 000 entries doesn't seem to be the right way to me.
> Besides the unacceptable memory consumption it would take a significant
> amount of time to get the table entirely read in and the list prepared.

I agree. While I think the processing time is by far the biggest problem
of pre-filling, having neither the memory or the time overhead would be
ideal.

On my machine, simply appending 100K rows to a list store takes 0.8
seconds.  Appending 100K rows with generated data takes about 2.1
seconds, so there is a large advantage to not populating all the data at
once even if pulling it from the source is instant, which in the case of
a database store it most certainly isn't instant.

> So I'm looking for a more dynamic way. My data source (database) is
> capable of delivering arbitrary rows, in random access manner. My
> concept is to dynamically read and display only small parts of the
> database table, as the user scrolls to see them.

A simpler model would be to fill all rows from the scroll position to
the top.  The would make it easier internally to keep track of which
rows are empty with a single var.  However, I do agree it would be nicer
this way for when the user jumps way down a 100K row list.

Since the list is a linked list, if the user jumps from displaying rows
1-50 to row 80K, you'll have to fill in rows 51-79,999 with blanks in
order to populate rows 80,000-80,050.

> Now, I'm wondering whether this concept is accomplishable using a
> standard GtkTreeView widget? Just filling a GtkTreeModel with those 50
> entries every time doesn't work. The problem is obviously the scrollbar
> of the treeview. Its proportions shouldn't reflect the (limited) amount
> of rows in the underlying GtkTreeModel which are displayed any time, but
> rather the total number of listable database records and the (possibly
> tiny) excerpt of it which is visible.

I agree, the real trick is to translate scrollbar position to row
coordinates.  If you can do this along with having the liststore "fake"
out how many rows it is actually storing your home free.

> Another related problem is how to determine the number of rows a
> GtkTreeView can display in its given space? No simple method or property
> there to query that, if I'm not mistaken?

I don't think there is a simple way to determine how many rows a
liststore is displaying, but this can be solved by populating more rows
than you need.  So figure that the typical row is 20 pixels high, on a
2000x2000 screen that would be 100 rows.  So take the top row position
and make sure that the list store is populated for 200 rows below that. 
Not an ideal solution, but a reasonable workaround.  Even better would
be to base your calculation on the actual size of the TreeView control
to be more efficient.

I do recommend pulling more data than you need because users tend so
when a user scrolls down one row your not having to make a round trip to
the server.  I'm sure there is some sweat spot around 100-200 records
for database on the same LAN segment as the client.

> I'm considering putting a custom, independent scrollbar next to the
> GtkTreeView and use that one to navigate the list. However, at least I'd
> need to know the number of visible rows in the GtkTreeView for that. The
> window containing the GtkTreeView has to be resizable by the user, of
> course, so assuming any fixed sizes / numbers of rows isn't feasible.

It would be much nicer if you could use a standard scrollwindow, have
the liststore "Fake" out 100K rows and then on the scroll event check to
make sure the liststore is populated with data.

> Any idea how to accomplish this by using existing widgets, or would I
> need to write an own sort of list widget? Compatibility to GTK+ 2.0 is
> preferred, if existing GTK+ means were to be used.

The above is all my personal theories from thinking about the problem in
the back of my mind.  The real question is can you have the liststore
fake rows.  If not then you have to fall back to appending blank rows,
which is at least 3x faster than generating the rows and populating them
at once and much less memory.

I'll probably go the blank row route at first and work toward the ideal
code later.