[Evolution-hackers] Very poor performance and hangs with ema addressbook factory

[sean finney <seanius seanius net>]

Hi all,

First off, a quick into for those who haven't already met me in IRC:  

I'm consulting with the IT department of a large corporation who are
evaluating evolution-mapi as a basis for a native linux mail client
for use in large scale deployment.  In the past month or two I've been
hanging out on IRC (pestering Milan, mostly :D ), doing a fair amount of
QA and a little bit of hacking when it necessary.  At this point I'm pretty
optomistic that we'll be able to move forward with this, which is awesome,
though there are a couple issues remaining with "show stopper"
status here.


So on that note, I'd like to talk about the awful, awful performance of
the addressbook factory in EMA.  I've filed a bug[1], and was pointed
at the ongoing discussion about "backend caches" on this list, and
thought I'd join in and find out what needs to be done to get this
fixed in a reasonable (and hopefully backportable-to-.32) manner.  Since
this is more architectural in nature, I figure here is a better place
to discuss it rather than there.

For those too lazy to follow the links, this is the issue from what I can
tell so far:

 * evolution+mapi fetches the entire Global Address List (GAL) from Active
   Directory[2], and "caches"[3] it as an XML file
 * while fetching, the entire application will often block/hang
 * while fetching, the addressbook-factory will monopolize one or more
   cores at 100%.
 * on any access to contacts information from this list (contacts pane,
   autocomplete, etc), this data is loaded by the backend factory in its
   entirety.
 * seemingly (haven't read the code well enough), this data is searched
   linearly for matches.  or... it's really slow anyway.
 * if the frontend visits the contacts pane (where ALL contacts will be
   shown), or the "To/CC" type buttons, all information from the GAL is
   loaded *also* into the frontend, doubling memory usage.

Some ballpark idea of the sizes we're talking about here:

~61k contact entries in the GAL
~40MB xml file containing "cached" contacts
~500 MB RAM usage in the addressbook factory backend
~500 MB RAM usage in the evolution frontend
~5-10 minutes of evolution being entirely hung/unresponsive (as in greyed
out by the window manager, even).

I don't have any profiling output but have a very strong suspicion from
just poking via gdb that the majority of the time is spent doing various
things with the in-memory loaded xml file.  I think this would be a great
place to either (a) replace the xml file with an sqlite database, or (b)
split out the xml file into individual xml files and/or vcards and have
a rebuildable sqlite index.

So my question to the list is: is anyone already working on something similar
to this, somewhere else?  otherwise, any opinions for how it ought to be done?


	sean


[1] https://bugzilla.gnome.org/show_bug.cgi?id=644817
[2] the entire fetching is needed for stuff like autocompletion to work, i've
    been told.
[3] the term "cache" is not really appropriate here, as it's not a cache,
    it's a "replica", but i digress...