Re: Building off Medusa

[Alexander Larsson <alexl redhat com>]

On Tue, 8 Apr 2003, David C Sterratt wrote:

> >>>>> Seth Nickell writes:
> 
>  >> The incremental live indexing isn't difficult.  Medusa could use
>  >> libfam to gather modifications to files, and reindex them.  I got
>  >> it pretty sorted out, although perhaps a "medusa-modifyd" is
>  >> needed, which puts filesystem change manifests in a FIFO queue,
>  >> and medusa reads the queue and reindexes the filesystem.  This
>  >> also helps for offline searches.  The multiuser thing is crucial
>  >> to a business setting.  The metadata indexing is crucial for me
>  >> (damnit i want to find my MP3).
> 
>  > Just so you know... Incremental indexing won't be possible using
>  > libfam.  FAM will not scale to monitoring over about 500 files, so
>  > you definitely will not be able to get change notification on all
>  > the files on a disk.  I would love a way to register with the
>  > kernel to be notified whenever *any* file changes, but I don't
>  > believe there is such a mechanism.
> 
> Apparrently there is a mechanism available in 2.4.19 kernels and
> higher that can monitor changes in unlimited numbers of directories.
> There is a command line interface to it called dnotify
> (http://www.student.lu.se/~nbi98oli/dnotify.html).  I haven't tried it
> myself, but it looks as though it should be able to monitor a
> directory recursively and tell you when there are any changes to that
> directory, though it can't tell you the name of the file that was
> changed.

It is not at all unlimited. You need to hold open a file descriptor for 
each directory that you want to monitor, and the number of open file 
descriptors are limited (it also keeps you from unmounting things, and 
various other issues). Also, to recursively monitor things you need to 
keep adding monitors to new dirs, which is racy.

Basically dnotify has a horrible interface (signals? WTF?), has a horrible 
implementation and has less-than-perfect semantics.
 
> Another project (that supplies its own kernel module and may overcome
> the file name limitation of dnotify) is changedfiles
> (http://www.bangstate.com/changedfiles/).  Again, I haven't tried this
> out.

I haven't tried this, but depending on anything that is not accepted in 
the official kernel is bad.

Every now and then I talk to kernel developers about this issue, but they 
don't seem to be interested in the problem. 

-- 
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
 Alexander Larsson                                            Red Hat, Inc 
                   alexl redhat com    alla lysator liu se 
He's a superhumanly strong pirate waffle chef possessed of the uncanny powers 
of an insect. She's a transdimensional blonde fairy princess with a song in 
her heart and a spring in her step. They fight crime!