Re: [Tracker] Database access abstraction

[Jamie McCracken <jamie mccrack googlemail com>]

On Thu, 2008-11-06 at 16:11 +0100, Philip Van Hoof wrote:
> On Thu, 2008-11-06 at 09:39 -0500, Jamie McCracken wrote:
> > its not that clear cut
> >
> > firstly, pausing the indexer is an async process and does not have to be
> > synchronous at all so there is no round trip with direct access.
>
> Pausing the indexer right now has to be (and is performed) synchronous.

it can and should be async

sqlite will release its locks when its ready so there is no need for it
to be synchronous

> > In most cases the indexer will be idle when querying. A client could 
> > even bypass by checking to see if tracker-indexer is running
>
> That's true and is also being done that way by trackerd. Although the
> client can't as-easy as trackerd can, check whether or not the indexer
> is running (in a reliable way). Unless the client subscribes to a few
> signals that are emitted by tracker-indexer (which is private API atm).

I cant remember if the indexer uses a lock file like trackerd but if it
does then its pretty easy to do the check

> Also note that Garnacho is and/or has changed this infrastructure
> recently. Adding Garnacho in CC.
>
> > dbus has overhead with millions of strings - each one has to be type
> > verified and copied several times. Unless you are testing with huge
> > result sets like fetching 100,000 music files with all metadata then any
> > comparison is invalid. Also as we move to flattened tables the query
> > time will get faster and the this will make the dbus overhead more
> > prominent
>
> Agree with this. But you probably want to fetch pages, not entire
> databases, over D-Bus.

absolutely but some badly designed music clients fetch all up front. We
need to be flexible is all Im saying (i expect clients to abuse us)

> I don't think a client would be developed correctly if it really needs
> to fetch an entire database. Clients should be cursor based consumers,
> that fetch pages of data .. instead of all data.
>
> > ïIn other cases, like say using tracker as a gconf backend you will often
> > get calls to fetch individual keys and these need to be in process to
> > avoid round trips for each key
>
> Tracker as a gconf backend?! Let's just use either DConf or GConf for
> that. I don't really see the point in that.

its merely an exampe of an a use case where direct access would be
beneficial

> > ergo its premature to rule out direct access at this point 
>
> Yet if we want to insert massive amounts of data fast, then we need to
> monopolize the SQLite tables by opening (large) transactions.

well no - direct access must be read only (IE Select statements only)

this is because sqlite locking might not work on NFS mounts and thats
why only tracker-indexer writes to the db.

With direct access you can do all your selects directly and only use the
daemon for notification and updates

jamie