Re: [Tracker] Ready for merge

[Jamie McCracken <jamie mccrack googlemail com>]

On Fri, 2008-08-08 at 12:49 +0100, Martyn Russell wrote:
> Laurent Aguerreche wrote:
> > Hello,
>
> Hi :)
>
> > Le jeudi 07 aoÃt 2008 Ã 16:38 +0100, Martyn Russell a Ãcrit :
> > > Jamie,
> > >
> > > I think we are ready for merge if you are happy with the current state
> > > of the indexer-split branch.
> > >
> > > If you find anything you think needs fixing before the merge. Let us know.
> >
> > I'm experiencing two issues with the indexer-split branch :
> > - if I terminate trackerd with "kill" (to send a SIGTERM signal) sent to
> > it, I see tracker-indexer still running. I can't kill tracker-indexer
> > with "kill", I have to use "kill -9"
> > - tracker kills the performances of my hard-drive while it starts: GUIs
> > of my applications get frozen!
> >
> > Do you intend to fix these issues before the merge (notably the second
> > one...)?
>
> We have been fixing a few things up today that I noticed last night.
> Issues include:
>
> 1.) Index being too big that indexing is slowed right down:

this is why trunk used index merging so it would flush to separate
indexes and then merge them into one when finished.

By experimentation 16MB is a good size for an index hence trunk flushed
in 16mb lumps

doing lots of small writes to a large index causes real IO problems with
ext/3 (ext/4 fixes this) - each word being indexed is a separate seek
and write on the index so you could easily have 100,000+ small updates
being applied. Also when words have hits added the hashtable has to
relocate them to free space which also causes fragmentation - the
optimize qdbm call recovers this by copying the index into a new one
once indexing is finished

as you can see index merging eliminates this perfomance overhead or the
need to call optimize as the final merged index will have no
fragmentation

jamie