Re: [Evolution-hackers] Memory usage of CamelFolderSummary and CamelImapFolder

[Philip Van Hoof <spam pvanhoof be>]

On Thu, 2006-11-16 at 16:13 +0100, Philip Van Hoof wrote:
> I did another round of checking where the memory of Camel is going to in
> tinymail's Camel.
> 
> I have also tested its Camel with Evolution with success.
> 
> Tinymail's Camel has the following features:
> 

Next to these features, tinymail's Camel also:

  o. Removes all compilation warnings
  o. Uses all of Matthew Barnes's very nice patches (except the
     GStringChunk one) (thanks a lot Matthew)
  o. Uses all my GSlice-and-other patches (check the patches mailing
     list of Evolution to get an overview)
  o. Renames all the library-names to avoid filename conflicts
  o. Removes the libedataserver dependency (a very small libedataserver
     is statically linked)
  o. Has correct linking (the provider's libraries have all incorrect
     LDFLAGS crap, and link with WAY to much libraries) -- I hate it
     when people have no clue about what they are doing in the
     Makefile.am or simply forget to remove old linking flags
  o. Has its own configure.ac and its own build environment, and is
     integrated with tinymail's build and repository
  o. Reimplements OpenSSL support (certificates are on todo)
  o. Still supports all typical features of the normal Camel
     (NSS/NSPR/SMIME/SSL/Kerberos/etc etc)
  o. Removes the folder meta-info patch (unused by Tinymail, and not
     really done very well -- consumes memory, and it looks like a lot,
     but in stead of testing it .. I simply removed it, so I don't know
     for sure)

  o. Eum ...
  o. Did I miss something?


And next to existing features will tinymail's Camel soon:

  o. Support partial message retrieval in IMAP (certain)
  o. Support partial message retrieval in POP (uncertain)
  o. Support summaries in the POP provider (most likely)
  o. Support merging and backup of local cache (certain)
  o. Have correct certificate management for the SSL support (low
     priority)
  o. . . . . (you already know I wont stop, no matter what)



>   o. The CamelFolderSummary uses mmap. This significantly reduces memory
>      usage because an mmap is on-demand paged.
> 
>   o. The CamelMessageInfoBase structure is significantly smaller (this
>      has been put in #ifdef's so that it can easily be reversed for
>      Evolution). This reduces memory usage of the memory that can't be
>      mmap()ed.
> 
>   o. When new message-info arrives, each 1000th item the
>      CamelFolderSummary is dumped to disk and effectively reloaded. The
>      pstring_hashtable will also be updated (pstrings that no longer
>      occur are unreferenced)
> 
>   o. The CamelImapFolder consumes much less bandwidth by asking for a
>      lot less headers and by more efficiently forming IMAP commands,
>      the procedure has been reimplemented into a cancellable one. By
>      that I mean that if a cancellation happens, already received data
>      can mostly be recovered (and will be recovered). This without the
>      continuations which are only available in IMAP4rev1 (by simply
>      storing data more quickly on disk, and starting from the previous
>      store point)
> 
>   o. When the CamelFolderSummary instance is reloaded, it will reuse
>      CamelMessageInfo instances. It will not destroy them unless they
>      have been removed during an expunge request.
> 
>      If the message-info is available in the mmap, it will unreference
>      the pstrings that might have been in use by the message-info and it
>      will reassign the struct's char pointers to locations in the mmap.
> 
>      Because Camel has a property accessor, this even works on folders
>      that are open (messages that are currently visible should not be a
>      problem because the tree-view makes copies of the strings when they
>      need to become visible -- the GtkTreeView does, I haven't checked
>      Evolution's but Evolution didn't crash after I did a a lot basic
>      removing, copying, scrolling and moving of messages).
> 
>   o. The "this is a non-mmapped message-info instance" flags have been
>      put in the "flags" member rather than separate gbooleans which
>      consumed another two ints in memory per instance
> 
> These four/five/six dots (together) solve all the remaining problems
> that the original mmap patch had (the most important issue was that when
> new messages arrived, those where not mmap()ed, they are now reloaded
> periodically hence will be mmap()ed during such a reload quickly).
> 
> The speed of fetching new messages is actually "much" faster than the
> original Evolution implementation. That's probably because a lot less
> bandwidth is needed. If you would count without the bandwidth
> optimisation (on a hypothetical extremely fast IMAP service) you would
> see a performance hit compared to the original one. That's mostly
> because each 1000th received message-info, a reload happens.
> 
> 
> For a massif graph of downloading 3000, 800, 3000, 800, 3000, 800
> headers you can check out this message on the tinymail mailing list:
> 
> http://mail.gnome.org/archives/tinymail-devel-list/2006-November/msg00111.html
> 
> 
> The relevant code:
> https://svn.tinymail.org/svn/tinymail/trunk/libtinymail-camel/camel-lite/camel/camel-folder-summary.c
> https://svn.tinymail.org/svn/tinymail/trunk/libtinymail-camel/camel-lite/camel/providers/imap/camel-imap-folder.c
> 
> If I can assist people with bringing their Camel into this shape, then
> please let me know.
> 
-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be
gnome: pvanhoof at gnome dot org
work: vanhoof at x-tend dot be
blog: http://pvanhoof.be/blog