Re: beagled and "bushy" IMAP folders

[Joe Shaw <joeshaw novell com>]

Hi,

This is somewhat related to the mail I just sent.

On Wed, 2006-03-22 at 09:58 -0800, Florin Andrei wrote:
> If Evolution is not running, beagle-status reveals that the tasks queue
> gets empty after a while (if I don't use the system much). That, I
> believe, is normal.
> 
> If I do run Evolution, the tasks queue never gets empty. 

Evolution doesn't log or send out any sort of notification when the
status of mails change, and so we have to manually walk its data
whenever someone on disk changes and update our own internal state
accordingly.

The inotify events generated by Evoltuion as it writes out its IMAP
summary to disk is how we notice these changes, but it doesn't
necessarily mean that the data itself has changed.  So every time these
files change, we have to walk the file.

> I am not sure if I'm actually saying this is a bug. On one hand, that's
> a lot of files that get updated every 10 minutes so Beagle seems
> justified to poll them. OTOH, this system is a laptop and Beagle polling
> all those files all the time means the resources get drained faster.

Varadhan and I have come up with a solution that'll go into Evolution
such that the overhead of doing this will be pretty greatly reduced.
We're not going to be able to avoid looking at the files, but we should
be able to do a lot of the operations much faster.

I've also fixed a bug in CVS recently that was causing Beagle to get
into a tighter CPU loop when walking across these files that subverted
our scheduler, so hopefully the impact won't be as noticeable now.

Finally, there has been some talk about maybe suspending indexing while
on battery power -- and there are some bugs open about that -- but there
aren't any concrete plans about that right now.

> Or could it be smarter and figure out somehow which summary files it
> really needs to update but without polling all of them? (but then it
> would need to somehow talk with Evolution, right?)

We might also be able to do this, but right now Evolution isn't sending
notifications about these unless you use a fairly experimental plugin.

> Am I making at least one valid point? :-)

Yes, well said. :)

Thanks,
Joe