Re: [Tracker] New branch for Xesam support in trackerd

[Martyn Russell <martyn imendio com>]

Jamie McCracken wrote:
> On Fri, 2008-05-02 at 11:58 +0200, Martyn Russell wrote:
> > Jamie McCracken wrote:
> > > On Mon, 2008-04-28 at 11:37 +0200, Philip Van Hoof wrote:
> > > > On Mon, 2008-04-28 at 17:05 +0800, wendy wrote:
> > > > > Hi Philip,
> > > > >
> > > > > I think we should take into account of using the "xesam-glib", which
> > > > > will simplify the implementation of "xesam-support" on tracker. 
> > > > Hey Wendy,
> > > >
> > > > Xesam-glib has a few classes that provide an abstract GLib GObject. You
> > > > can implement it and get everything DBus related for free.
> > > >
> > > > Which is nice, indeed.
> > > >
> > > > However. The DBus related things for the Xesam support for trackerd have
> > > > already been done in a very similar way to how xesam-glib works.
> > > >
> > > > For now ain't the code actually doing things asynchronously because none
> > > > of the existing trackerd infrastructure and code is prepared for
> > > > asynchronous operations (there's insufficient locking and data isolating
> > > > in place, SQLite wouldn't cope with multiple threads accessing it, etc).
> > > sqlite 3.5 + is totally threadsafe and multiple threads can access it
> > > safely
> > That is good to know :)
> >
> > > though I dont think making it  async is necessary? Surely that would
> > > make the code harder to read and maintain?
> > It isn't necessary, just like using threads isn't necessary. I thinking
> > using threads is much harder to read and maintain than coding
> > asynchronously. Understanding the execution process for a threaded
> > application with mutexts blocking in various places, etc is a LOT harder
> > to follow and understand than asynchronous programming.
> >
> > > to keep the tracker daemon resposive to handling search requests we will
> > > need it to have threads with the main thread dedicated to handling dbus
> > > calls (and shutdown) only (as it is in trunk)
> > Making the daemon accept DBus calls asynchronously will not make it
> > unresponsive. In my experience, a daemon handing 2 million telephony
> > events a day on a busy switch can operate perfectly asynchronously, this
> > sort of traffic is not what I would expect on a desktop at all either. I
> > really have a hard time believing that the Tracker daemon would be
> > unresponsive if we didn't use threads. Do you have evidence of this?
> >
> > > also as tracker is more increasingly used we will want multiple threads
> > > handling requests
> > Hmm? Why?
>
> if one client is asking for a huge result set which takes 5 seconds it
> will block other clients for the same time period even if they only want
> to ping it, 

But this is all because the database API/our implementation is
synchronous only. This effectively means we would need a thread in the
indexer and in the daemon to do any database transactions, since I
envisage the indexer having a DBus interface possibly (for shutting down
and doing various non-database things too) - just an idea at this point.

Also, have you done tests to see if there are any circumstances where
the user has to wait 5 seconds or more? (I am just interested to be
honest - most tests I have done here don't take that long at all).

If they have to wait for that period of time for some database lookup in
the first place, it means any other lookups will be delayed too - the
only thing this really changes is how fast non-database requests respond
(if at all since they can timeout).

I have briefly looked at the SQLite3 API and it looks like an async
approach could be written in Tracker using sqlite3_step() and throttling
iterating the main loop occasionally. Also, if it is IO blocking, we can
look into the patch that Mozilla uses for firefox to make IO access
asynchronous.

> if the daemon is totally async then we would:
>
> 1) lose any load balancing of queries so even non-db and non-index
> requests can take a long time (and neither sqlite nor the index is
> async)

Depending on my assumption about our approach to using the SQLite3 API
in an async or sync way, this is either a DB limitation or an
implementation flaw which we can improve on. We should probably look
into this some more and avoid the synchronous approach if we can do that.

> 2) lose ability to shutdown promptly - force killing could corrupt index

Yea, although with the new model we would only be reading in the tracker
daemon not writing so there shouldn't be any corruption right?

> 3) any client which used the sync dbus api (like nautilus and file
> chooser) would be blocked totally

What do you mean? You mean it wouldn't work for them? If so, that's not
the case, DBus supports synchronous APIs for asynchronous interfaces.

> with the index being threadsafe via the api I constructed (using
> mutexes) and sqlite 3.5 being completely threadsafe there would be no
> need for locks of any kind and none of the above would happen. For the
> request daemon the threading is very simple too.

I guess, ultimately if we HAVE no asynchronous approach to the database
we have to use threads. I do wonder how we should approach this in the
new libtracker-db internal library - since some backends will have async
support and some won't. Some may even be async only.

-- 
Regards,
Martyn