Re: [Tracker] [Re: tracker in F16]

[Martyn Russell <martyn lanedo com>]

On 20/09/11 12:45, Lennart Poettering wrote:
> On Tue, 20.09.11 09:49, Martyn Russell (martyn lanedo com) wrote:
>
> > > a) tracker uses inotify recursively and creates a massive number of
> > > watches due to that. That is both ugly and doesn't scale. Tracker
> > > apparently tries to not take up the full pool of inotfy handles the
> > > system provides, but that won't help if you have more than one user on
> >
> > Yes, this sucks enormously. When we have a clean shutdown (i.e. no
> > work is being done and we know we're up to date), we can start up
> > with n thousand directories being monitored in<  60 seconds with
> > thousands of directories being crawled under cold cache. Again this
> > depends a little on the hardware and the number of directories, but
> > it's damn fast. It also shouldn't be that noticeable. Aleksander has
> > had some ideas about how to avoid this entirely, but then it's a
> > question of user space vs kernel space. I will let him comment
> > further here.
>
> 60s of disk access is actually quite a lot. We can boot userspace in<
> 1s now, 60s of disk access afterwards due to tracker are actually really
> awful.

:)

Let me be precise. The number 60 I gave there is just an estimate and it 
depends on how much content you have of course. For starting up (after 
initial index, the common scenario), this involves setting up monitors 
and checking all files we know about are up to date according to what we 
have in the database:

* warm cache *

Tracker-INFO: Finished mining in seconds:2.746672, total 
directories:1633, total files:15037

* cold cache *

Tracker-INFO: Finished mining in seconds:20.921365, total 
directories:1633, total files:15038

Incidentally, that's 1410 monitors.

--

This argument could be entirely avoided of course if there was some 
journal or kernel support for offline file change tracking.

> > > the system. The solution here should probably be fanotify, which allows
> > > proper recursive file system watches. So far fanotify has been
> > > accessible to root only, which is presumably why tracker doesn't use
> > > it. However, the solution here cannot be to work around that fact by
> > > using inotify, but must be to invest the necessary kernel work to make
> > > fanotify useful from unprivileged processes.
> >
> > Yes, we were looking on at Eric Paris' work with high expectations,
> > but from what we've discussed internally as a team, it doesn't look
> > to improve our situation nearly enough and really would just assist
> > us (from what I remember). Again Aleksander can comment on this.
>
> The thing is that the kernel is fixable. If it's borked fix it, don't
> work around it. It's C code, like userspace too. It's free, like
> userspace is. Tracker is not an island. We as the community own the
> stack from top to bottom, so we can fix it top to bottom. That's what
> makes us better than Windows.

Is there some reason why you continue to lecture us about this?

Further more, I fail to see the point in working on a new solution at 
the same time as one is being developed upstream by one of Red Hat's 
developers. I believe Jürg and Aleksander already pointed out our stance 
on this and we *have* been seeking alternative solutions.

--
Regards,
Martyn

Founder and CEO of Lanedo GmbH.