Re: [Evolution-hackers] Moving the struct instance heap space to mmap

["Michael Zucchi" <notzed gmail com>]

Hmm, will it really reduce memory usage though?  I'm not so sure.  Remember that although the summary file contains all strings on disk, in memory they are actually unique-ized.  For a lot of load-types (e.g. mailing lists), this saves a trememdous amount of memory - even after all the overhead of the hashtable to manage it.  Address and subject strings in particular.  And a mmap file isn't terribly different to performance to malloc'd memory - once the latter is setup, in both cases the kernel is the one loading it off disk (swap, or filesystem).

(an mmap file IS just memory, so if "mmap'd file size + in-memory support tables" > "in-memory version", then you haven't generally won - even if 'ps' shows you have).

Note also that all of the summary items, and all of the strings thus stored, are using special allocators which reduce - significantly - the overhead of malloc, on these small allocations.  In earlier versions I tried a different string allocator for each messageinfo which mimicks much of what you're doing here, but in memory.  Strings were stored in an array structure, but I only stored a single pointer to the base of the array, and packed the strings into a NUL separated right-sized buffer, and searched each time I needed to access them, overhead was only about 5 bytes (pointer to the 'array' plus an array 'limit').   Performance was fine, but memory usage was significantly larger than the current model, where strings are 'unique-ized' - like 30% or mo
 re iirc.

Having extra indices on disk - you're really just writing a datbase table - and all of the associated complexity, consistency and coherency issues that implies.  And unless you do something 'proper' like a btree with a unique key, you're going to have to end up storing plenty of extra support stuff in memory anyway, 
e.g. a hashtable to find items by uid, the sorted list of messages for the view, and so on.

We did a lot to reduce per-message overhead at the camel level, but its an area of diminishing returns - the ui table view will still take as much memory, etc.

On 08/09/06, Philip Van Hoof <spam pvanhoof be> wrote:

On Thu, 2006-09-07 at 21:14 +0200, Philip Van Hoof wrote:

> To read message n, you would simply do something like:
>
> from = sstart + *(istart + (sizeof (int) * 4 * n) + 1)
> subject = sstart + *(istart + (sizeof (int) * 4 * n) + 2)
> to = sstart + *(istart + (sizeof (int) * 4 * n) + 3)
> flags = sstart + *(istart + (sizeof (int) * 4 * n) + 4)
>

No no no no no. I meant (something like) this of course:


unsigned char *sstart = (uchar*)mmap(..), *istart = (uchar*)mmap(..);
#define AMOUNT_OF_OFFSETS_PER_RECORD 4
#define AOO AMOUNT_OF_OFFSETS_PER_RECORD

from = sstart + *(istart + ((sizeof (uint32_t) * AOO * n) + sizeof (uint32_t)))
subject = sstart + *(istart + ((sizeof (uint32_t) * AOO * n) + sizeof (uint32_t)))
to = sstart + *(istart + ((sizeof (uint32_t) * AOO * n) + sizeof (uint32_t)))
flags = sstart + *(istart + ((sizeof (uint32_t) * AOO * n) + sizeof (uint32_t)))

To much pointers in my head ;)

--
Philip Van Hoof, software developer at x-tend
home: me at pvanhoof dot be
gnome: pvanhoof at gnome dot org
work: vanhoof at x-tend dot be
http://www.pvanhoof.be - http://www.x-tend.be