Re: GtkTreeView with huge amount of rows; dynamic data source?

[Gus Koppel <gtk spamkiller bytechase cx>]

Greg Breland wrote:

> I think you mis-understood what I mean by the word "populate"  if your
> database has 100K rows then you are going to create a liststore with
> 100K blank rows and then populate the first 200 rows with data from
> the database.  This allows you to get around the tough problem of
> managing your own scroll bars.

No, it doesn't. If I understand you correctly, you suggest to create
i.e. 100 K rows of which 99.8 K rows would never really be used, i.e.
never be filled with data? Only the first 200 rows would be? I think,
that's an unacceptable strategy as well. Just consider adding another
"0" to the hypothetical total number of rows, and the resulting memory
and preparation time requirements.

Populating only the first 200 rows with real data wouldn't solve the
problem with scrollbar management, either. If the scrollbar gets dragged
to a 50% offset, it's not the first 200 (50) rows of a List/TreeStore
that are displayed, but the rows starting at offset 50 K (in this
example). So I'd have to fill those 200 rows with data as well. And if
the user decides to scroll through the entire list, page by page, I
would end up in having nearly all records being read into memory as
well, consuming lots of RAM and slowing down the treeview a lot. That's
what I originally wanted to avoid by a dynamic solution, which doesn't
store more rows in RAM than are actually being displayed.

> > 3. reading exactly the number of visible rows from my data source,
> >    starting at the appropriate start position, and filling a
> >    GtkListStore or GtkTreeStore with it.
>
> I disagree here.  You should read more rows than you need because of
> the latency involved when pulling information from the Database. 
> Usually it takes 1% more time to pull 10 rows than to pull 1 row. 
> What is going to happen when the user holds the scroll down arrow? 
> Your app is going to send thousands of SELECTs to pull one row at a
> time as fast as it can.

Well, that's just an optimization detail, which is to be addressed after
the main problem has been solved. Your suggestion of readahead bases on
the assumption that users would mainly scroll down the list in a
sequential order. However, users could also decide to mainly scroll up
or just use the scrollbar to make rather random movements in the list.
Readahead would result in a slight performance penalty then and, in case
the data source is non-local, significantly increased network traffic
(by 300%, if 50 rows were visible, 200 rows were read every time and the
user doesn't scroll down sequentially).