Re: Beagle and Gaim Logs

[Richard Laager <rlaager wiktel com>]

On Mon, 2006-04-10 at 15:58 +0100, Daniel Drake wrote:
> Richard Laager wrote:
> > 1. Old Logs
> > 
> > Really old versions of Gaim (pre 0.70, I think) stored conversations in
> > what we now call "old logs". These files are flat files, which contain
> > all the conversations for a given buddy. They are named things like:
> > ~/.gaim/logs/rlaager.log (for a buddy with the screenname "rlaager").

> We used to index these, but dropped support when we reworked the IM log 
> filtering. To my knowledge, we haven't had any comments about this - 
> i.e. nobody noticed so far ;)

I guess this depends on what you want to use Beagle for. If you only
want it to deal with things in the recent past, then this support is
irrelevant (unless you're one of those people that uses the plugin to
log new things to this format). If you want to use Beagle for searching
old stuff as well, then it would be useful.

> Beagle isn't very good at indexing log files like this, since they just 
> continually grow and grow, and beagle ends up indexing the whole file 
> every time a line is added. This is OK for small per-conversation files, 
> but is a problem otherwise.

I'm not sure if it's helpful, but I implemented conversation "indexing"
on these files for 2.0.0beta3. It creates a ~/.gaim/BUDDY.idx file to
accompany the ~/.gaim/BUDDY.log file, which looks like this:

99      752     1040624978
906     2023    1041205844
2984    337     1041301733

The fields are:
1. File offset to the start of the conversation (excluding the "New
conversation" line).
2. Length (in bytes) of the conversation.
3. Conversation timestamp (obviously, as a time_t).

The fields are tab separated, and a record ends with a newline. For
forwards compatibility, my policy is that parsers MUST ignore additional
fields at the end of the line (which they don't understand).

This allows Gaim to pull up the log viewer without parsing the
entire .log file (which is hideously expensive).

The biggest problem is that you're not guaranteed a .idx file, both from
a backwards compatibility standpoint and from the fact that we only
create the .idx files the first time the corresponding .log file is
read.

You could index all the .log files once, and then monitor the .idx file
instead. By comparing that data to what you've indexed, you could
probably avoid re-indexing all of the conversations.

Richard