Re: New module proposal: tracker

[Lennart Poettering <mztabzr 0pointer de>]

On Tue, 18.08.09 16:44, Martyn Russell (martyn lanedo com) wrote:

>> As I see it the usefulness of Tracker stands or falls with the
>> scalablity of inotify. As long as inotify does not natively support
>> recursive watches tracker is not viable.
>
> Well file monitoring really is just one part of Tracker. If you want to  
> keep data from your web applications (say Facebook, etc) in Tracker too,  
> it is completely unrelated.

And that exists? What providers are there besides the indexer?

>> Right now installing tracker on a non-trivially sized $HOME has the
>> effect of taking away all inotify watches and thus making inotify
>> unavailable for other applications.
>
> This is also not true. Tracker never uses ALL resources, it always  
> leaves 512 available for other desktop applications.

Ah, interesting. It didn't do that the last time I checked. How do you
verify this? by reading the sysctl and then allocating only 512 less
than that? So running tracker means the sysctl is effectively set to
512 from the perspective of other apps?

This is still an ugly hack, wouldn't you agree?

With a vanilla kernel this leaves 7680 dirs to be watched by
tracker. A single git checkout repo takes away 400 of those here.

Sure, I might not be the typical user, but my music directory already
has > 7000 dirs. My entire $HOME has 82283.

>> -- where they usally are more
>> appropriately used, such as Nautilus. And all that even without fully
>> working properly since if the limit of inotify handles is reached the
>> view tracker has on the file system will necessarily become
>> out-of-date quickly. Or more drastically spoken: installing Tracker on
>> a machine with a non-trivially sized $HOME breaks Nautilus and other
>> software.
>
> I would also argue that some file 20 levels deep in a directory that is  
> never updated is quite unlikely to change and so the chances are quite  
> remote.
>
> We also are using breadth based monitoring so top level folders always  
> get priority here.

That's just trying to make the best of a f*ed up situation. It's not
even a work-around, let alone a fix.

Also I doubt that this logic even ameliorates things. I for one tend
to edit the leaves of my $HOME tree, not the trunk of it. You seem too
assume that this was the other way around for most people. For me at
least the top level directories have few (and seldomly edited) files
in them, the deepest levels most. I mean, do you edit
~/projects/tracker/src/indexer.c more often that ~/indexer.c? (file
names made up, but you get the idea)

> Tell me about it. I would love to know that some file in folder x  
> changed due to the mtime change at the top most level. But actually,  
> from our experience with the Maemo platform, this is not portable anyway  
> and Windows doesn't even update the current parent's mtime when a file  
> changes in the folder. I would also stress that this likely varies from  
> file system to file system slightly.

mjg59 has some ideas about this. Ping him. We were discussing this a
bit at gcds with a couple of folks.

> I think you clearly don't understand what Tracker 0.7 offers. It is not  
> ALL about the file system, in fact the file system takes a back seat and  
> is there as a convenience to provide the database with data.  
> Applications also use the data store.

So you are telling me file system notification does not matter much
for Tracker's usefulness? That's news to me indeed. But of course
leads to the question: what is it then what it offers that desn't need
file system notification? Please explain.

>> As I see it Tracker is not ready for inclusion in the desktop. It
>> might not be entirely Tracker's fault though, it's more the kernel's
>> fault, but that doesn't change the conclusion.
>
> Well, again, that assumes that Tracker is all about file monitoring,  
> which it isn't.

So, what's it about then?

>> Or in short: just f*ing fix the kernel first.
>
> Are you volunteering? :)

I am not hacking on Tracker, am I?

Lennart

-- 
Lennart Poettering                        Red Hat, Inc.
lennart [at] poettering [dot] net
http://0pointer.net/lennart/           GnuPG 0x1A015CC4