Memory testing > 15,000 headers

[Philip Van Hoof <spam pvanhoof be>]

Hey guys,

Here are some memory measurements:

If you use valgrind, you will see the memory of this identified as
heap-admin, posix_memalign and malloc depending on whether or not the
instance of the structure was allocated using gslice or malloc.

I measured with 14,613 random messages of the Xen-devel mailing list:

imap://mail.tinymail.org, U:tinymailunittest, P:unittest
folder: INBOX/15000/mailinglist

You can login using Roundcube webmail at https://mail.tinymail.org

You can repeat this testing using the tests in the tinymail framework:

svn co https://svn.tinymail.org/svn/tinymail && cd tinymail/trunk
./autogen.sh --prefix=/opt/tinymail --enable-tests --with-platform=gpe
make && make install

# Getting the headers locally (this will work out of the box, the
# account information is in the source code indeed)
/opt/tinymail/bin/memory-test --online

valgrind --tool=massif /opt/tiynymail/bin/memory-test -c /tmp/tinymail.0

You will mostly be using this code for this specific test:

http://svn.tinymail.org/svn/tinymail/trunk/tests/memory/memory-test.c

Some certainties on 32bit x86 (with the mmap patch, without, Camel-
MessageInfoBase is ~ two or three bytes smaller per instance)

sizeof (CamelMessageInfo) = 12 - accounts for 175356 bytes (~171.25K)
sizeof (CamelMessageInfoBase) = 80 - accounts for 1169040 bytes (~1141.64K)
sizeof (CamelMessageContentInfo) = 36 - accounts for 526068 bytes (~513.74K)
sizeof (struct _CamelFlag) = 8 - accounts for 116904 bytes (~114.16K)
sizeof (struct _CamelTag) = 12 - accounts for 175356 bytes (~171.25K)

In a first context,

The g_malloc function accounted for 44% of the memory.

42.1% of the memory was needed for e_memchunk_alloc and 29.5% by
e_memchunk_alloc0.

12% of the memory was needed for camel_folder_summary_content_info_new
at the line: ci = e_memchunk_alloc(s->content_info_chunks);

12.3% of the memory was needed for random g_strdups which both Camel and
tinymail use. I will investigate this soon.

6.3% of the memory was needed by message_info_load which is in
camel-folder-summary.c. The g_malloc is the following line:

mi->references = g_malloc(sizeof(*mi->references) + ((count-1) *
sizeof(mi->references->references[0])));

5.1% of the memory was needed for camel_content_type_new.

And 4.5% of it was needed for by camel_header_set_param 

GSlice uses posix_memalign and accounted for 19.4% of the total memory.
These are GObjects being instantiated by tinymail (mostly TnyHeader
instances). Therefore that 19% isn't interesting for Evolution.

In total. tinymail used 4,500K memory. Tinymail used a Camel with the
mmap patch. Without the mmap patch, the memory grows to ~ 10,000K
(because each header-value needs to be stored in the heap).

Note that Evolution keeps *all* headers in memory at all times. I tested
with 15,000 such headers. A typical Evolution will probably have five or
six such folders. This illustrates the importance of keeping the size
per header small.

At this moment that size is ~ (12+80+36+8+12) bytes per header.


-- 
Philip Van Hoof, software developer at x-tend 
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
work: vanhoof at x-tend dot be 
http://www.pvanhoof.be - http://www.x-tend.be