Re: stuck in file crawl task loop

[Joe Shaw <joeshaw novell com>]

Hey,

On Mon, 2007-01-22 at 18:09 -0500, Brian J. Murrell wrote:
> They are whatever comes in Ubuntu's Feisty.  Not sure if Kevin is the
> maintainer there or not.  

Ah, ok, you're using Feisty packages.  That's fine then, that's the
official 0.2.14 release.

> So 0.2.14 has the looping bugs fixed?  I thought you said that they were
> fixed post 0.2.14.  No worries either way.

They should be fixed in 0.2.14.  So what you're seeing is obviously an
uncaught one.

> bo7X958nykipdPCXJdiA+w|/home/brian|sieve.script|20041206030115|20070116172617|Beagle.Filters.FilterText|0

The columns here are:

        * unique_id - maps to user.Beagle.Uid
        * directory
        * filename
        * last_mtime - Beagle.MTime
        * last_attrtime - Beagle.AttrTime
        * filter_name - Beagle.Filter
        * filter_version - Beagle.Filter
        
For these, the mtime refers to the modification time of the file when it
was indexed.  When rewalking a directory, we simply check whether the
file's actual mtime is different (older or newer) than the last indexed
mtime and reindex if it is.

The attrtime is the last time the file attributes were changed, like
through a chmod.  It maps to the filesystem's ctime.  If the ctime is
newer than attrtime, we reindex the file.  Since setting an extended
attribute changes the ctime of the file, with the extended attribute
version you can look at this as the last time the file was indexed.
There isn't any explicit "indexed at <time>" setting.

> So some files on a local FS are using EAs and some are in the database.
> Any ideas why?  I did start using beagle before I was mounting /home
> with user_xattr, but I thought some code had gone into beagle to convert
> the use of files in the db to using EAs if they later became available.

The main reason why it would store inside the DB instead of an xattr is
permissions.  If the file isn't owned by you, or you don't have write
access to it (even if you do own it), you can't write an xattr.

Beagle will try to convert DB entries to xattrs when it can (removing
them from the DB), but that only happens on a reindex.  If the file
hasn't changed, Beagle won't touch it.

> > If you have all of your files on an NFS server and they don't change
> > much, you could use the beagle-build-index tool to create a static index
> > of the files on the NFS server, and export that to Beagle for searching
> > it on the client.
> 
> Right.  Which precludes $HOME though.

Why, because the files change too often?  You might be able to poke Alex
into polishing up the server-side watcher. :)

> Thanx for all the info and feedback Joe.  You are great!

No problem, let me know if you have any more questions!

Joe