Re: Beagle and Gaim Logs

[Joe Shaw <joeshaw novell com>]

Hi,

On Sat, 2006-04-08 at 16:49 -0500, Richard Laager wrote:
> I'm the Gaim developer currently "in charge" of the logging code. I have
> a few questions about Beagle's searching of Gaim logs. Please bear with
> me... I'm new to Beagle. :) It looks really nice so far!

Thanks!  I'm glad to see a Gaim developer pop up, because integration
with Gaim is one of Beagle's strongest points.  I hope we can make it
even better.

> 1. Old Logs
>
> Beagle doesn't seem to index these. In the interest of completeness*,
> I'd like to see these files be indexed, and I'm willing to help in
> whatever way I can. I never used C# before, so I would need a little
> guidance if I was to try to create a patch.

We used to, but we dropped support for them after we were seeing some
looping bugs and after the filtering code was reworked, as Daniel
mentioned.  To the best of my knowledge, we've never gotten a bug filed
asking for those files to be supported again.

As Daniel also mentioned, the format of these files is problematic for
Beagle, because it means that the file has to be entirely refiltered any
time a line is added.  This is the reason why we don't index XChat or
irssi logs right now.  It's not an insurmountable problem, but it's a
problem nonetheless.

To support these, I think we would probably want a separate filter, if
possible.  We would also need a way to support these in our IM log
viewer app.

I'll be happy to help you in any way I can.  I'm usually on #dashboard
on Gimpnet during the weekdays, although I'm often not that chatty.  You
can catch me there or on email.

> 2. Log file naming
>
> First, let me say I'm sorry I didn't notify you about that. 

It's perfectly ok. :)  We're kind of used to file formats changing out
from underneath us.  Often developers don't consider files like this to
be part of an API, so we have to hack around things.

> However, I would like to reserve the ability to add additional data
> before .EXT. So, basically, a filename such as this would be valid:
> 
> 2006-04-04.131115-0500CDT.rlaager-wants-to-make.life.difficult.html
> 
> The timezone abbreviation ends at the first period. So, I won't be
> making filenames like this:
> 
> 2006-04-04.131115-0500CDTWhere_does_the_tz_end_and_this_start.html

Ok.  I *think* our current code will deal fine with this.  Lukas, who
has more or less become the steward of that code, would know better.
Either way, we'll make sure it's adequate.

> 3. Timezones
> 
> You might wonder why on earth I added the timezone information, if there
> was backwards compatibility breakage, etc.
> 
> We display log times like this:
> $ date +%c
> Sat 08 Apr 2006 04:23:11 PM CDT
> 
> I spent a couple months in another timezone. When I came back, all of
> the logs were displayed as being CDT or CST. This was technically
> incorrect and my obsessive nature drove me to find a solution.
>
> [earlier]
>
> `date +%Y-%m-%d.%H%M%S%z%Z.EXT`

The timezone thing is actually kind of an issue for us.  For some
reason, .NET doesn't include a way to parse the "%z" format.  Instead of
"-0500" they want "-05:00", so we have to use nasty hacks to parse it.
We also don't parse the "%Z" at all, because there's no way in .NET to
parse timezone names, from what I can tell.  Ultimately we probably
don't care, since we only ever store stuff in UTC and display in local
time.

> The log reader plugin:
>
> I'd like to see these other logs indexed by Beagle as well. One way to
> do this would be to write a separate Beagle indexing module for them...

Depending on how these are stored on disk, we can probably add them into
the Gaim backend.

> SQL Logging:
> 
> We get a fair number of questions about aggregating logs from multiple
> Gaim instances. While doing synchronization of your ~/.gaim or
> ~/.gaim/logs directory is one way of doing this, there has been some
> interest in having Gaim log to a SQL database. A Gaim plugin would add
> an additional logger to accomplish this.
>
> How could we make Beagle index logs in this case?

Are these instances on separate computers all logging to one remote
database server?  My practicality suspicions aside, the way we've
traditionally been doing indexing of remote data is to build an index on
the server and export it somehow (NFS or Samba) to the client machines.
Alternatively, if all the requisite information (server, db name,
username, password) were stored in a config file somewhere, the Gaim
backend could make the connections itself and build a local index based
on this info.

> It would be nice to have some way that would just work with any new Gaim
> logger that a user had. Perhaps we could make a Beagle indexer that
> worked with Gaim somehow (either a plugin itself, or via DBUS maybe).
> I'd love to hear people's thoughts on this.

This would definitely be an option if there were a reliable way to
quickly get at this information.  That's the biggest strength of these
individual on-disk logs: they're easy to look up.  We could possibly add
a parallel method for more esoteric logging, but my primary concern is
the 98th percentile of users out there, who will never use anything but
the standard logs.

> 5. This one isn't directly related to logs, but... Right now, if I
> search for someone's name, I see a "Contact". If I double-click it,
> nothing happens. Has anyone looked at using gaim-remote in Gaim 1.x
> and/or the DBUS stuff in Gaim 2.0.0 to start a new conversation with
> that user?

The contacts you are seeing are from the Evolution addressbook, not
Gaim's.  I have some code that has been languishing in my local tree to
do additional lookups for Evolution contacts based on IM accounts, and
vice versa.  That way you can send IMs when you get an Evo contact and
send emails when you get an IM conversation back.  The next step from
there is to additionally search IM conversations based on the info in
the contact, but that is part of a larger metadata chaining project that
needs to be developed.

As for starting a conversation, I believe this already works, but only
for AIM accounts.  Other accounts require the newer gaim-remote.  I
believe that the newer gaim-remote isn't backward compatible with the
old one, so we can't solve the issue for both.  I might be wrong about
that, but if it's true, I wouldn't want to switch until Gaim 2.0 is more
widely deployed.

> Alternatively, is this already handled if I had Galago installed?

No, our Galago support was mainly to show the status of the buddy in a
conversation.  When we developed our new UI, that support wasn't moved
over, so we don't really have any Galago support presently.

> 6. Searching from Gaim
> 
> Gaim currently has its own searching of logs, and it sucks majorly. I'd
> love to integrate the Beagle search functionality... to reuse the
> indexing that's already done by Beagle, etc. Any thoughts on that? How
> hard would that be? Is it a good idea? How might I start?
>
> The biggest problem I can think of is that Beagle doesn't support Gaim
> logging plugins (as discussed above). However, it would be nice to use
> the Beagle index when possible and then fallback to doing searching
> ourselves.

You can essentially do what Nautilus and Yelp have done, which is to use
our libbeagle C API to do specific searches, and fall back to a slower,
less rich internal search if it isn't available.  Looking at the Yelp
code for that -- it's all in yelp-search-pager.c -- is a decent example
of that.  There are some simple example programs for libbeagle in the
source tree, and the whole API is documented, although we're lacking a
good Getting Started tutorial on building an app with it.

Sorry for the long email, hope this helps. :)

Joe