Re: [Tracker] Ready for merge



Jamie McCracken wrote:
On Fri, 2008-08-08 at 12:49 +0100, Martyn Russell wrote:
Laurent Aguerreche wrote:
Hello,
Hi :)

Le jeudi 07 aoÃt 2008 Ã 16:38 +0100, Martyn Russell a Ãcrit :
Jamie,

I think we are ready for merge if you are happy with the current state
of the indexer-split branch.

If you find anything you think needs fixing before the merge. Let us know.
I'm experiencing two issues with the indexer-split branch :
- if I terminate trackerd with "kill" (to send a SIGTERM signal) sent to
it, I see tracker-indexer still running. I can't kill tracker-indexer
with "kill", I have to use "kill -9"
- tracker kills the performances of my hard-drive while it starts: GUIs
of my applications get frozen!

Do you intend to fix these issues before the merge (notably the second
one...)?
We have been fixing a few things up today that I noticed last night.
Issues include:

1.) Index being too big that indexing is slowed right down:

this is why trunk used index merging so it would flush to separate
indexes and then merge them into one when finished.

By experimentation 16MB is a good size for an index hence trunk flushed
in 16mb lumps

Yea, that would fix things.

Right now we are trying to evaluate if it is worth doing that or just
upgrading to SQLite with FTS.

What are your thoughts here Jamie. Would you rather have time spent
doing this QDBM work or the SQLite work?

I would be inclined to say we should merge and immediately start working
on the SQLite work.

doing lots of small writes to a large index causes real IO problems with
ext/3 (ext/4 fixes this) - each word being indexed is a separate seek
and write on the index so you could easily have 100,000+ small updates
being applied. Also when words have hits added the hashtable has to
relocate them to free space which also causes fragmentation - the
optimize qdbm call recovers this by copying the index into a new one
once indexing is finished

as you can see index merging eliminates this perfomance overhead or the
need to call optimize as the final merged index will have no
fragmentation

Yea.

Have you had time to look at the rest of the branch to see if it is
satisfactory?

-- 
Regards,
Martyn



[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]