[Evolution-hackers] Memory usage of CamelFolderSummary and CamelImapFolder

[Philip Van Hoof <spam pvanhoof be>]

I did another round of checking where the memory of Camel is going to in
tinymail's Camel.

I have also tested its Camel with Evolution with success.

Tinymail's Camel has the following features:

  o. The CamelFolderSummary uses mmap. This significantly reduces memory
     usage because an mmap is on-demand paged.

  o. The CamelMessageInfoBase structure is significantly smaller (this
     has been put in #ifdef's so that it can easily be reversed for
     Evolution). This reduces memory usage of the memory that can't be
     mmap()ed.

  o. When new message-info arrives, each 1000th item the
     CamelFolderSummary is dumped to disk and effectively reloaded. The
     pstring_hashtable will also be updated (pstrings that no longer
     occur are unreferenced)

  o. The CamelImapFolder consumes much less bandwidth by asking for a
     lot less headers and by more efficiently forming IMAP commands,
     the procedure has been reimplemented into a cancellable one. By
     that I mean that if a cancellation happens, already received data
     can mostly be recovered (and will be recovered). This without the
     continuations which are only available in IMAP4rev1 (by simply
     storing data more quickly on disk, and starting from the previous
     store point)

  o. When the CamelFolderSummary instance is reloaded, it will reuse
     CamelMessageInfo instances. It will not destroy them unless they
     have been removed during an expunge request.

     If the message-info is available in the mmap, it will unreference
     the pstrings that might have been in use by the message-info and it
     will reassign the struct's char pointers to locations in the mmap.

     Because Camel has a property accessor, this even works on folders
     that are open (messages that are currently visible should not be a
     problem because the tree-view makes copies of the strings when they
     need to become visible -- the GtkTreeView does, I haven't checked
     Evolution's but Evolution didn't crash after I did a a lot basic
     removing, copying, scrolling and moving of messages).

  o. The "this is a non-mmapped message-info instance" flags have been
     put in the "flags" member rather than separate gbooleans which
     consumed another two ints in memory per instance

These four/five/six dots (together) solve all the remaining problems
that the original mmap patch had (the most important issue was that when
new messages arrived, those where not mmap()ed, they are now reloaded
periodically hence will be mmap()ed during such a reload quickly).

The speed of fetching new messages is actually "much" faster than the
original Evolution implementation. That's probably because a lot less
bandwidth is needed. If you would count without the bandwidth
optimisation (on a hypothetical extremely fast IMAP service) you would
see a performance hit compared to the original one. That's mostly
because each 1000th received message-info, a reload happens.


For a massif graph of downloading 3000, 800, 3000, 800, 3000, 800
headers you can check out this message on the tinymail mailing list:

http://mail.gnome.org/archives/tinymail-devel-list/2006-November/msg00111.html


The relevant code:
https://svn.tinymail.org/svn/tinymail/trunk/libtinymail-camel/camel-lite/camel/camel-folder-summary.c
https://svn.tinymail.org/svn/tinymail/trunk/libtinymail-camel/camel-lite/camel/providers/imap/camel-imap-folder.c

If I can assist people with bringing their Camel into this shape, then
please let me know.

-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be
gnome: pvanhoof at gnome dot org
work: vanhoof at x-tend dot be
blog: http://pvanhoof.be/blog