Help! Your beagle likes to eat my file index!

[Martin Soto <soto informatik uni-kl de>]

Hi everyone,

After having tested many versions of Beagle and subsequently dropped
them because of  problems similar to those described in this message,
I'm now trying with 0.2.8 (installed on an otherwise vanilla Ubuntu
Dapper system from the http://us.ubuntu.cafuego.net/ unofficial
repository.)

It works generally well except for a single problem: it seems to feel
like indexing my home directory forever. I have more or less regularly
invoked beagle-index-info to monitor the file index, and, most of the
time, the number of indexed files grows steadily. However, after some
time, it just drops down to zero and the whole process starts again. My
home directory contains about 120.000 files, and since the machine is
not permanently running, indexing them takes quite some time. So, it is
horribly frustrating to see that after two of three days of steady work,
the index dissappears suddenly.

This has the sad consequence of making Beagle sort of unusable for me.
Actually, I never know if I can trust it or not. Sometimes I get pretty
complete results. Sometimes I get nothing at all.

Of course, I've tried to check the logs but they don't say that much to
me, and I haven't been able to find any guide that explains how to use
them. As far as my interpretation goes, beagle-index-helper crashes
(maybe it chokes on some file but, if that's the case, I can't tell
which one it is) and when restarting, it decides to purge the index
because it finds a dangling lock. The first lines of the IndexHelper log
after such a restart look like this:

060829 1603573071 08703 IndexH DEBUG: Starting messaging server
060829 1603574653 08703 IndexH  WARN: Unable to set IO priority for
process to idle
060829 1603574665 08703 IndexH DEBUG: IO priority for process set to
best effort 7
060829 1603574770 08703 IndexH DEBUG: Helper Size: VmRSS=16.5 MB,
size=1.00, 0.0%
060829 1603581588 08703 IndexH DEBUG: Checking for dangling locks...
060829 1603581614 08703 IndexH  WARN: Found a dangling index lock
on /media/hda2/home/lap096/.beagle/Indexes/FileSystemIndex/Locks/lucene-eab5a6ecc0bd7233dbd1618832f0e4bf-write.lock
060829 1603581630 08703 IndexH DEBUG:
Purging /media/hda2/home/lap096/.beagle/Indexes/FileSystemIndex
060829 1603596706 08703 IndexH DEBUG:
-file:///media/hda2/home/lap096/UML/linux-2.4.24/include/asm-sparc64/ioctls.h
060829 1603596750 08703 IndexH DEBUG:
-file:///media/hda2/home/lap096/UML/linux-2.4.24/include/asm-sparc64/keyboard.h
060829 1603596754 08703 IndexH DEBUG:
-file:///media/hda2/home/lap096/UML/linux-2.4.24/include/asm-sparc64/chmctrl.h
06
...

I would gladly file a bug, but I don't even know what to report exactly.
So, some help on tracking this problem down would be very appreciated.
By the way, I really don't know, but is Lucene so lacking in robustness
that you have to completely erase an index that took days to build just
because a process crashed while accessing it?

Thanks,

M. S.