[Evolution-hackers] Reviewing imap_update_summary

[Philip Van Hoof <spam pvanhoof be>]

Greetings,

imap_update_summary is implemented in three or four steps:

  o. Getting (all) the headers/uids
  o. Finding the ones that we must still fetch
  o. Fetching those (x)
  o. Writing out the summary

The steps each consume memory and reuse some of the memory of the
previous step. Pointers to that memory is stored in GPtrArray's like
fetch_data and messages.

In the code I have found no real reason to why this was done in
separated loops (steps) rather than one step and at the end of the loop,
free the data already. Especially for the third step (x), which seem to
consume most memory while it's happening.

The current implementation requires the data being received from the
IMAP service to be kept in memory until the entire folder has been
received and all steps done. This consumes more than one entire kilobyte
per message. Multiply that with for example 5,000 headers and you'll get
5 MB memory consumption for fetching the new messages of a very small
IMAP folder (in case no other messages had been received before you
first started the procedure).

Multiply that with 50,000 headers and you'll get 50 - 60 MB memory
consumption for a not extremely big but relatively big IMAP folders.

Which will be freed, yes, but nevertheless it's a slowly growing peak
(the speed depends on the connection with your IMAP server) that only
gets deallocated or when pressing cancel or when all messages are
received (which can take a significant amount of time).


The strange part is that if I measure the amount of bytes that I receive
from the IMAP service; I measure far less bytes being transferred than
bytes being consumed in memory. It not only stores all the received
data, it also stores a lot more in memory (probably mostly 4 bytes
pointers and GData stuff).

I wonder whether there was a reason why it was implemented this way? If
not, I'm planning to rewrite imap_update_summary in a different way. For
example by immediately creating a CamelMessageInfo struct or burning it
to the summary file instantly and freeing the GData created from the
stream in-loop.



-- 
Philip Van Hoof, software developer at x-tend 
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
work: vanhoof at x-tend dot be 
http://www.pvanhoof.be - http://www.x-tend.be