Re: [gnome-db] Gradual retrieval of data

["Vivien Malerba" <vmalerba gmail com>]

On 3/28/06, Murray Cumming <murrayc murrayc com> wrote:
> On Tue, 2006-03-28 at 09:25 +0200, Vivien Malerba wrote:
> > On 3/27/06, Murray Cumming <murrayc murrayc com> wrote:
> > > If I do a gda_connection_execute_command() with
> > > "SELECT * FROM sometable"
> > > and sometable has a million records,
> > >
> > > will libgda get all the data for all the records, or will it only get
> > > the data from the database when I actually do
> > > gda_data_model_get_value_at()?
> >
> > That depends on the API DBMS provides. Usually (and it's the case for
> > the Postgres provider), the data resulting from such a query is
> > transfered to the client memory and then the data model only really
> > creates the GdaValues from that data when a
> > gda_data_model_get_value_at() is called.
> >
> > However this is not suitable for very large amounts of data. For this
> > purpose I introduced data models for which data can only be retreived
> > using an iterator. To be more precise, the values stored in a data
> > model can:
> > - either be accessed using only an iterator (no
> > gda_data_model_get_value_at() then)
> > - or both using gda_data_model_get_value_at() or iterators: these are
> > "random access" data models
> >
> > The way a data model can be accessed can be retreived using the
> > gda_data_model_get_access_flags().
> >
> > >
> > > I am interested in the Postgres backend.
> >
> > Now to answer your question properly, the postgres provider always
> > returns random access data models.
> >
> > I'm working on a modification of the API which will allow to specify
> > which kind of data model the user want to be returned, and then small
> > modifications to the providers will be required (using declared
> > cursors in case of the postgres DBMS).
> >
> > The code to specify the requested data model type will be in CVS later
> > this week.
>
> So "random access" means, or causes, "all data transferred over the
> network at once".
> and "iterators" means "data transferred on demand"
> ?

I'd say it this way: for "random access" data models, all the data may
be stored in memory at a given time (which can be a problem), and for
"cursor (or iterator) only" data model, the data may not be all stored
in memory at a given time.

Of course the way the data is managed in memory depends on each data
model implementation.

The rule is: when the expected data model may be "big", prefer an
iterator only data model.

Note that using iterators (one or several) is always possible on
random access data models.

>
> This sounds like a very useful feature of the new API. I will try the
> new API again soon. Many thanks.
>

Cheers,

vivien