Re: [Tracker] New branch for Xesam support in trackerd

[Jamie McCracken <jamiemcc blueyonder co uk>]

On Fri, 2008-05-02 at 11:58 +0200, Martyn Russell wrote:
> Jamie McCracken wrote:
> > On Mon, 2008-04-28 at 11:37 +0200, Philip Van Hoof wrote:
> > > On Mon, 2008-04-28 at 17:05 +0800, wendy wrote:
> > > > Hi Philip,
> > > >
> > > > I think we should take into account of using the "xesam-glib", which
> > > > will simplify the implementation of "xesam-support" on tracker. 
> > > Hey Wendy,
> > >
> > > Xesam-glib has a few classes that provide an abstract GLib GObject. You
> > > can implement it and get everything DBus related for free.
> > >
> > > Which is nice, indeed.
> > >
> > > However. The DBus related things for the Xesam support for trackerd have
> > > already been done in a very similar way to how xesam-glib works.
> > >
> > > For now ain't the code actually doing things asynchronously because none
> > > of the existing trackerd infrastructure and code is prepared for
> > > asynchronous operations (there's insufficient locking and data isolating
> > > in place, SQLite wouldn't cope with multiple threads accessing it, etc).
> >
> > sqlite 3.5 + is totally threadsafe and multiple threads can access it
> > safely
>
> That is good to know :)
>
> > though I dont think making it  async is necessary? Surely that would
> > make the code harder to read and maintain?
>
> It isn't necessary, just like using threads isn't necessary. I thinking
> using threads is much harder to read and maintain than coding
> asynchronously. Understanding the execution process for a threaded
> application with mutexts blocking in various places, etc is a LOT harder
> to follow and understand than asynchronous programming.
>
> > to keep the tracker daemon resposive to handling search requests we will
> > need it to have threads with the main thread dedicated to handling dbus
> > calls (and shutdown) only (as it is in trunk)
>
> Making the daemon accept DBus calls asynchronously will not make it
> unresponsive. In my experience, a daemon handing 2 million telephony
> events a day on a busy switch can operate perfectly asynchronously, this
> sort of traffic is not what I would expect on a desktop at all either. I
> really have a hard time believing that the Tracker daemon would be
> unresponsive if we didn't use threads. Do you have evidence of this?
>
> > also as tracker is more increasingly used we will want multiple threads
> > handling requests
>
> Hmm? Why?

if one client is asking for a huge result set which takes 5 seconds it
will block other clients for the same time period even if they only want
to ping it

if the daemon is totally async then we would:

1) lose any load balancing of queries so even non-db and non-index
requests can take a long time (and neither sqlite nor the index is
async)

2) lose ability to shutdown promptly - force killing could corrupt index

3) any client which used the sync dbus api (like nautilus and file
chooser) would be blocked totally

with the index being threadsafe via the api I constructed (using
mutexes) and sqlite 3.5 being completely threadsafe there would be no
need for locks of any kind and none of the above would happen. For the
request daemon the threading is very simple too

jamie

>