Re: New module proposal: tracker

[Martyn Russell <martyn lanedo com>]

On 18/08/09 16:07, Lennart Poettering wrote:
> On Tue, 18.08.09 13:05, Martyn Russell (martyn lanedo com) wrote:
> Hmm. The beef I have with Tracker (and Beagle fwiw) is that they build
> something on infrastructure that currently is not good enough
> to sustain it: inotify. inotify is simply not suitable for recursively
> watching $HOME, but Tracker tries that nonetheless. And that is a big
> big failure, it should not do that.

I agree the situation isn't perfect, but it isn't a BIG failure.

Currently Red Hat's Eric Paris is working on this with fanotify:

  https://fedoraproject.org/wiki/Features/fanotify

There are more links from Google of course.

> There's something I like to call the "tracker paradox": if you have
> a large data set tracker is useless because inotify doesn't scale and
> the database is quickly out-of-date -- and if you have a small data
> set then you don't need a search engine and hence tracker is useless
> too.

Well, that really depends on the user and the data set. Most "normal" 
users don't have 10 versions of the linux kernel checked out causing 
these inotify limits to be reached. With ALL my music and external 
drives I don't have a problem with the limit at all. It seems that only 
people with the whole of GNOME checked out into $HOME seem to run into 
these cases.

If I have a small data set (say 10k images), how do I find all images I 
took in 2006 with tags foo ordered by date? The search engine part is 
not about just finding "where did I put my images", the querying is much 
more sophisticated than that.

> As I see it the usefulness of Tracker stands or falls with the
> scalablity of inotify. As long as inotify does not natively support
> recursive watches tracker is not viable.

Well file monitoring really is just one part of Tracker. If you want to 
keep data from your web applications (say Facebook, etc) in Tracker too, 
it is completely unrelated.

> Right now installing tracker on a non-trivially sized $HOME has the
> effect of taking away all inotify watches and thus making inotify
> unavailable for other applications.

This is also not true. Tracker never uses ALL resources, it always 
leaves 512 available for other desktop applications.

> -- where they usally are more
> appropriately used, such as Nautilus. And all that even without fully
> working properly since if the limit of inotify handles is reached the
> view tracker has on the file system will necessarily become
> out-of-date quickly. Or more drastically spoken: installing Tracker on
> a machine with a non-trivially sized $HOME breaks Nautilus and other
> software.

I would also argue that some file 20 levels deep in a directory that is 
never updated is quite unlikely to change and so the chances are quite 
remote.

We also are using breadth based monitoring so top level folders always 
get priority here.

> And no, increasing the inotify limits is a band-aid, not a fix.

Sure.

> Oh and inotify is not the only area where the Linux file system layer
> is not ready to sustain Tracker, it's just the most obvious. Another
> thing that come to my mind is that we curently lack recursive mtimes
> so that tracker could identify changes on filesystems that happened
> while tracker was unable to access them (i.e. due to reboot, logout,
> hot unplug).

:)

Tell me about it. I would love to know that some file in folder x 
changed due to the mtime change at the top most level. But actually, 
from our experience with the Maemo platform, this is not portable anyway 
and Windows doesn't even update the current parent's mtime when a file 
changes in the folder. I would also stress that this likely varies from 
file system to file system slightly.

> Really guys, before pushing forward with getting this into the desktop
> make sure that your infrastructure is actually suitable for what you
> want to do. And right now it simply is not!

I think you clearly don't understand what Tracker 0.7 offers. It is not 
ALL about the file system, in fact the file system takes a back seat and 
is there as a convenience to provide the database with data. 
Applications also use the data store.

> As I see it Tracker is not ready for inclusion in the desktop. It
> might not be entirely Tracker's fault though, it's more the kernel's
> fault, but that doesn't change the conclusion.

Well, again, that assumes that Tracker is all about file monitoring, 
which it isn't.

> Or in short: just f*ing fix the kernel first.

Are you volunteering? :)

--
Regards,
Martyn