Re: New module proposal: tracker

[Martyn Russell <martyn lanedo com>]

On 06/11/09 19:22, Alexander Larsson wrote:
> On Fri, 2009-11-06 at 15:01 +0000, Martyn Russell wrote:
> > > The seeking is hard to avoid, but there may be ways to readdir stuff
> > > without having the result be persistant in the buffer cache, although
> > > i'm not sure how posix_fadvise(POSIX_FADV_DONTNEED) can be applied when
> > > reading a directory...
> >
> > Yea. We need to research this a bit more to make sure we are doing
> > everything we can. We did try the posix_fadvise() call and it was fine
> > on the desktop, but for small devices like the n900, it caused a slow
> > down which meant we dropped it.
>
> Hmm, really? In what sense? syscall overhead?

In the end to end testing of crawling->extraction->indexing.

Of course on the n900, the hardware is a lot different, the power 
available is less and the kernel is tailored. We didn't get around to 
finding out exactly why it was slower.

> There is one problem with POSIX_FADV_DONTNEED. If you do it on a file
> the kernel will drop it from its caches. This is generally what you want
> if you just indexed a 100 meg text file that no other app cares about
> atm, since it means we won't throw out 100 megs of otherwise useful
> caches. However, if you're reading a file that some other app actually
> cares about this may be a problem, since you're now ensuring that the
> file has to be re-read the next time that app wants to use the file. Not
> sure if there is a better way though...

Yea. We don't index all 100Mb of the file. We only ever index n words of 
any file. This is configurable in $HOME/.config/tracker/tracker-fts.cfg. 
It is set to 10000 for now.

--
Regards,
Martyn