Re: [Tracker] Ready for merge

[Jamie McCracken <jamie mccrack googlemail com>]

On Mon, 2008-08-11 at 10:55 +0100, Martyn Russell wrote:
> Jamie McCracken wrote:
> > On Fri, 2008-08-08 at 12:49 +0100, Martyn Russell wrote:
> > > Laurent Aguerreche wrote:
> > > > Hello,
> > > Hi :)
> > >
> > > > Le jeudi 07 aoÃt 2008 Ã 16:38 +0100, Martyn Russell a Ãcrit :
> > > > > Jamie,
> > > > >
> > > > > I think we are ready for merge if you are happy with the current state
> > > > > of the indexer-split branch.
> > > > >
> > > > > If you find anything you think needs fixing before the merge. Let us know.
> > > > I'm experiencing two issues with the indexer-split branch :
> > > > - if I terminate trackerd with "kill" (to send a SIGTERM signal) sent to
> > > > it, I see tracker-indexer still running. I can't kill tracker-indexer
> > > > with "kill", I have to use "kill -9"
> > > > - tracker kills the performances of my hard-drive while it starts: GUIs
> > > > of my applications get frozen!
> > > >
> > > > Do you intend to fix these issues before the merge (notably the second
> > > > one...)?
> > > We have been fixing a few things up today that I noticed last night.
> > > Issues include:
> > >
> > > 1.) Index being too big that indexing is slowed right down:
> >
> > this is why trunk used index merging so it would flush to separate
> > indexes and then merge them into one when finished.
> >
> > By experimentation 16MB is a good size for an index hence trunk flushed
> > in 16mb lumps
>
> Yea, that would fix things.
>
> Right now we are trying to evaluate if it is worth doing that or just
> upgrading to SQLite with FTS.
>
> What are your thoughts here Jamie. Would you rather have time spent
> doing this QDBM work or the SQLite work?

I am doing the sqlite fts fork work atm

I guess the merging can wait as there is a lot of more important stuff

> I would be inclined to say we should merge and immediately start working
> on the SQLite work.
>
> > doing lots of small writes to a large index causes real IO problems with
> > ext/3 (ext/4 fixes this) - each word being indexed is a separate seek
> > and write on the index so you could easily have 100,000+ small updates
> > being applied. Also when words have hits added the hashtable has to
> > relocate them to free space which also causes fragmentation - the
> > optimize qdbm call recovers this by copying the index into a new one
> > once indexing is finished
> >
> > as you can see index merging eliminates this perfomance overhead or the
> > need to call optimize as the final merged index will have no
> > fragmentation
>
> Yea.
>
> Have you had time to look at the rest of the branch to see if it is
> satisfactory?
unfortunately it is not

Running it I get very poor performance (Im running tracker-applet so
dont know if that is affecting/pausing it)

Does not handle file moves (see trunk for this as its non trivial)

The gio file monitoring is not adequate - inotify needs optimisation

EG for inotify we only trigger a change if FILE_WRITE_CLOSED is sent -
that way we know the file has finished changing and we can safely index.
Currently it indexes whenever the file changes which sucks. FAM would
need a pending list (as per trunk) as it only has the file change and we
need to wait til changes are stabilised before trigger an index

Also we are a differential index so i dont understand why you have
removed the differential update code? What does it do when a file
updates?

Im seeing lots of issues so merge has to wait...still compiling list of
problems

jamie