Re: stuck in file crawl task loop

[Joe Shaw <joeshaw novell com>]

Hi,

On Sun, 2007-01-21 at 16:02 -0500, Brian J. Murrell wrote:
> 20070119 22:38:29.4152 06251 Beagle DEBUG: Running file crawl task
> 20070119 22:38:29.9395 06251 Beagle DEBUG: Done crawling '/data/jennifer_pc/c/Program Files/Browser Mouse/Browser Mouse/1.0'
>
> As you can see this is one per second, over and over again.

Are you running 0.2.14?  A bunch of looping issues were fixed there.

> How can I discover why this is?
> 
> It is worth noting that /data is on an NFS filesystem... being indexed
> on the NFS client.

Do you have any clock skew between your local machine and the NFS
server?  If you touch a (previously non-existent) file there, and
compare the timestamps between "stat <file>" and "date", what's the
difference there?

> On another side note, I know that beagle keeps it's interesting
> information about when files were indexed last and so forth in EAs.  Can
> an EA be added to a file which one does not have any permissions (i.e.
> to read and/or write)?

It can't, no, but Beagle falls back to using an sqlite database for
storing the same information if it can't be written to EAs for whatever
reason, like lack of permissions or no EA support on the filesystem.

> As a side note: it really is a shame that indexing non-local filesystems
> is so sucky.  

I agree.

> Why did FAM get replaced with something inferior (in terms
> of handling remote file modification events) again?  Was FAM really that
> bad?  

My understanding is that FAM is a security disaster for remote
filesystem notification.  No distribution I'm aware of ships FAM with
the remote notification stuff turned on.

A while back most distributions switched to gamin, which provided the
FAM API but as a per-user daemon, which is good for security.  However,
gamin didn't support remote notification either.

Gamin then was more or less abandoned, and a lot of applications just
access the inotify APIs directly, although some people I think are still
using gamin (which can wrap any API, like dnotify or inotify).

> Is there any hope that the file modification library (jeez, I wish
> I could recall it's name) will gain network file modification
> notification abilities?

The right thing to do is for these remote filesystems to implement file
notification themselves.  CIFS (Samba) supports notification at a
protocol level, and I know that Samba (although maybe only Samba 4) uses
inotify for file notification.  I don't know what the state of file
notification is in NFS, although it's probably in NFSv4.  I don't know
what the state of NFSv4 is on Linux.

As for Beagle and FAM, the file system backend has pluggable event
watchers, so there's no probably no inherent reason why it couldn't work
with FAM.  Originally there was a backend against .NET's built-in file
system watcher -- which on Linux used FAM on the backend -- but that API
wasn't rich enough and so things were always broken.  I think that FAM's
own API is rich enough for us to use, it's just a matter of someone
sending a patch for it.

Joe