Re: [Tracker] Releasing Tracker 0.6.90

[Jamie McCracken <jamie mccrack googlemail com>]

On Fri, 2009-01-30 at 12:58 +0100, Carlos Garnacho wrote:
> On jue, 2009-01-29 at 16:44 -0500, Jamie McCracken wrote:
> > On Thu, 2009-01-29 at 21:46 +0100, Michael Biebl wrote:
> > > 2009/1/29 Michael Biebl <mbiebl gmail com>:
> > > > 2009/1/29 Martyn Russell <martyn imendio com>:
> > > > > Jamie McCracken wrote:
> > > > > > > I think the crawling should be made more efficient wrt apps and
> > > > > > > conversations - can you handle this? (IE use subfolder mtime to prevent
> > > > > > > non-folder files from being checked)
> > > > >
> > > > > Carlos found some nasty issues here with the mtime checking. I think he
> > > > > committed a fix this afternoon for that.
> > > > >
> > > > > mbiebl can you check those if you get a chance?
> > > >
> > > > Will give it a try, thanks for looking into it!
> > >
> > > The problems with applications and conversations seem to be fixed for me.
> > >
> > > But just when I was happy about one issue being fixed, I noticed that
> > > performance on removal is awful again:
> > > Unpacking linux-2.6.28.tar.bz2 in $HOME takes tracker (r2862) around
> > > 20 minutes to index.
> > > When I rm -rf linux-2.6.28/, it takes over an hour, with my cpu
> > > constantly busy (i.e. max speed at 100%) :-/
> >
> > yeah thats something that needs investigation prior to release - thanks
> > for spotting it
>
> Looks like we have 2 problems here:
>
> 1) The UTF8 collation function set in the DBs. (This is also the main
> culprit in the case of moving the kernel sources, brought up by
> Tshepang). Couldn't we just store URIs or UTF8 paths so we can perform a
> cheaper string comparison there?

not sure thats a bottleneck?

> 2) In the delete case, unindexing could also be quite expensive, could
> we expect that to improve with sqlite FTS?

you should not be unindexing as qdbm is really slow for that

simply delete the file from the metadata db as per 0.6.6

with FTS it will be able to unindex much faster than qdbm