Re: [Tracker] Releasing Tracker 0.6.90

[Carlos Garnacho <carlos imendio com>]

On jue, 2009-02-05 at 11:09 -0500, Jamie McCracken wrote:
> On Thu, 2009-02-05 at 15:50 +0000, Martyn Russell wrote:
> > Jamie McCracken wrote:
> > > On Thu, 2009-02-05 at 14:03 +0000, Martyn Russell wrote:
> > > > Jamie McCracken wrote:
> > > > > Thanks I will check over weekend and try and fix some outstanding issues
> > > > > with TST
> > > > >
> > > > > If all goes well we can release next week
> > > > Jamie I was planning on releasing tomorrow. If we keep going on like 
> > > > this, we will never release.
> > >
> > > what about the deletion performance issue?
> >
> > TJ recently filed a bug for 0.6.6 where deleting emails in the current 
> > release brings the computer completely to its knees and you can't kill 
> > trackerd either. This is ALREADY much better than it was in the last 
> > release. It is better to be slow and responsive than block and be 
> > unresponsive.
> >
> > Carlos was telling me in a discussion we just had, that the kernel has 
> > 20k source files - that's *just* .[ch]. There are likely to be other 
> > files in there too of course which get indexed. He says this takes 20 
> > minutes (maximum to his recollection) to delete. That's almost 400 Mb of 
> > pure readable data file to index. Removing that from the databases is 
> > going to take some time.
> >
> > The deletion performance is down to QDBM. This *should* be significantly 
> > improved with FTS. We are not doing that before this release either, so 
> > the sooner we release, the sooner we can get on with FTS support and 
> > really try to fix this issue.
>
> we are not deleting from qdbm! If you have changed stuff to do this then
> please revert. O.6.6 never deleted from qdbm and indeed never should as
> its prohibitively expensive
>
> we are only deleting from sqlite. Phantom hits are weeded out by
> checking if service ID exists in sqlite. Hit counts for searches are
> estimates only (same as they are for google) 

I still think there should be some cleanup task or something, as these
estimates can be way off. Anyway, I did this, and got down processing
kernel sources deletion down to 3 minutes, good enough?

> jamie