Re: New module proposal: tracker

[Martyn Russell <martyn lanedo com>]

On 18/08/09 17:28, Lennart Poettering wrote:
> On Tue, 18.08.09 16:44, Martyn Russell (martyn lanedo com) wrote:
>
> > > As I see it the usefulness of Tracker stands or falls with the
> > > scalablity of inotify. As long as inotify does not natively support
> > > recursive watches tracker is not viable.
> >
> > Well file monitoring really is just one part of Tracker. If you want to
> > keep data from your web applications (say Facebook, etc) in Tracker too,
> > it is completely unrelated.
>
> And that exists? What providers are there besides the indexer?

Yes, there are some in progress right now.

> > > Right now installing tracker on a non-trivially sized $HOME has the
> > > effect of taking away all inotify watches and thus making inotify
> > > unavailable for other applications.
> >
> > This is also not true. Tracker never uses ALL resources, it always
> > leaves 512 available for other desktop applications.
>
> Ah, interesting. It didn't do that the last time I checked. How do you
> verify this? by reading the sysctl and then allocating only 512 less
> than that? So running tracker means the sysctl is effectively set to
> 512 from the perspective of other apps?

It has done this since 0.6.9x was released, so for a long time. I think 
it even did it when Jamie started the code.

> This is still an ugly hack, wouldn't you agree?

Sure, but you can't have everything at once and you have to make the 
best with what you have sometimes.

> With a vanilla kernel this leaves 7680 dirs to be watched by
> tracker. A single git checkout repo takes away 400 of those here.
>
> Sure, I might not be the typical user, but my music directory already
> has>  7000 dirs. My entire $HOME has 82283.

:)

> > We also are using breadth based monitoring so top level folders always
> > get priority here.
>
> That's just trying to make the best of a f*ed up situation. It's not
> even a work-around, let alone a fix.

So we should just tell users, sorry, the kernel is shit and we can't be 
arsed to *try* to make things better for you with what's available?

> Also I doubt that this logic even ameliorates things. I for one tend
> to edit the leaves of my $HOME tree, not the trunk of it. You seem too
> assume that this was the other way around for most people. For me at
> least the top level directories have few (and seldomly edited) files
> in them, the deepest levels most. I mean, do you edit
> ~/projects/tracker/src/indexer.c more often that ~/indexer.c? (file
> names made up, but you get the idea)

Right but source is less important and as I have said earlier in this 
thread (and others agree) we should probably be ignoring source checkouts.

> > Tell me about it. I would love to know that some file in folder x
> > changed due to the mtime change at the top most level. But actually,
> > from our experience with the Maemo platform, this is not portable anyway
> > and Windows doesn't even update the current parent's mtime when a file
> > changes in the folder. I would also stress that this likely varies from
> > file system to file system slightly.
>
> mjg59 has some ideas about this. Ping him. We were discussing this a
> bit at gcds with a couple of folks.

OK, nice thanks.

> > I think you clearly don't understand what Tracker 0.7 offers. It is not
> > ALL about the file system, in fact the file system takes a back seat and
> > is there as a convenience to provide the database with data.
> > Applications also use the data store.
>
> So you are telling me file system notification does not matter much
> for Tracker's usefulness? That's news to me indeed. But of course
> leads to the question: what is it then what it offers that desn't need
> file system notification? Please explain.

Well, for a start, we don't index . prefixed directories, so for 
applications that want to store their bookmarks, contacts, etc they can 
use tracker for that. The Nepomuk ontology describes all of that and how 
the data relates. Those updates do not come from file system updates, 
they come from applications sending us data.

I also mentioned internet data being indexed too (i.e. flickr, facebook, 
etc), we can't use inotify for those can we ;)

> > > Or in short: just f*ing fix the kernel first.
> >
> > Are you volunteering? :)
>
> I am not hacking on Tracker, am I?

And Tracker is not the kernel either, it is also not the only 
application that could benefit :)

--
Regards,
Martyn