Re: New module proposal: tracker

[Jamie McCracken <jamie mccrack googlemail com>]

On Tue, 2009-08-18 at 19:21 +0100, Michael Meeks wrote:
> Hi Jamie,
> 
> On Tue, 2009-08-18 at 11:32 -0400, Jamie McCracken wrote:
> > > Couldn't you just make gio (or gedit or OpenOffice) notify you every
> > > time it closes a file instead of monitoring bazillions of files? I'm
> > > not very likely to search for files I've never opened anyway.
> >
> > we could use the Gtk Recent files stuff for this and that would work for
> > ordinary users but not devs fetching source code or other command line
> > stuff
> 
> 	As of today, I've not seen a good Linux filing system that can cope
> with directory crawling [ particularly for the initial indexing ], even
> at low I/O priority, without unpleasantly degrading system performance.
> I imagine the sheer seek cost of pulling all those dentries, inodes into
> memory, and evicting all the other useful data you had around - is a big
> part of the plague. Hopefully btrfs will improve the situation somewhat
> here, but wrt. inode / dentry management I suspect there is no really
> good solution.
> 
> 	Why does that matter ? well - I am sure that we could make a really
> really good indexing & querying meta-data foo-matic thing, like beagle
> or tracker: that would handle what people want rather well: searching
> and monitoring their documents, contacts, IM logs, E-mail, web history
> and so on.
> 
> 	Unfortunately, as soon as we have this, it is only a small
> feature-creep step to "lets index all .c/.h files to extract comments in
> the API documentation" - which (I suspect) then commits you to the
> disaster of irritating a lot of developers - so they turn it off, and
> getting bogged down indexing things no-one is ever going to want indexed
> by tracker (?).
> 
> 	Personally, I'd start by ignoring any directory tree with a configure*
> script in the top-level, or perhaps a .git / .svn directory - that
> should reduce the inotify pain :-)
> 
> 	So - my point is: are the devs fetching source code at the console -
> that you are concerned about above, really in the target audience for
> tracker ? and if so why ? inevitably they will use ctags or equivalent -
> ie. a tight, generic end-user focused scope is prolly a win.

thanks for the comments

Ideally we would not want a broken system that gets out of sync with
command line usage (whether its checkouts or just moving/deleting files)
or non-glib apps

One issue might be a folder thats used for checkout but later reused for
non-dev stuff (docs, images etc) so we would have to be careful with
blacklisting folders (and maybe recheck them on tracker startup)

I believe Martyn is going to add code to totally ignore source folders
and yes detection of configure* would be a good starting point for that

Now that gio is part of glib there is a good chance that a solution
might be found there although a freedesktop solution to recent file
changes (including moves and deletes) would be ideal

jamie