Re: Moving the struct instance heap space to mmap

[Philip Van Hoof <spam pvanhoof be>]

On Fri, 2006-09-08 at 11:22 -0500, Federico Mena Quintero wrote:
> On Fri, 2006-09-08 at 10:37 +0200, Philip Van Hoof wrote:

> 1. What is the maximum length of those strings?  What is the maximum
> length of one message header in the summary file?  In the message info
> struct, can you use short ints for offsets instead of full pointers?  Or
> even single bytes for lengths, if the strings are really short?

At this moment is the file written using encoded integers. The algorithm
for that basically looks at the 0x80 bit to see whether or not the
bitfield ends on the current byte. In memory it's expanded to a 4 byte
normale integer (which is a reason why ls summary is smaller than the
amount of memory being used).

I'm sure that indeed an unsigned char can be used in stead. Very few
strings are larger than 256 bytes. And else two unsigned chars or 16
bit. Any string of which the length doesn't fit in 16 bit is probably an
error rather than a summary-string coming from an E-mail.

It does, however, feel a bit like micro optimizations. But if you
multiply it with the amount of headers, it becomes more significant
indeed.

> 2. What are the access patterns?  When the array of summaries gets
> accesed, do you need all the fields, or only some of them?  I.e. you may
> need to access some flags directly, but you may be able to afford
> computing some lenghts by hand if you are doing an uncommon operation.

Only some of them. Removing unneeded pointers from CamelMessageInfoBase
is on my todo list. I tried it once and only partly succeeded (something
became unstable for some strange reason with the NNTP provider).

> 3. Even if some things get accessed frequently, some others may not be.
> All my message folders show the default columns:  Flags, From, Subject,
> Date --- these are all fields in CamelMessageInfoBase.  However, some
> others don't get displayed at all:  mlist, to, cc, etc.  Can we remove
> those pointers from the struct, and compute them on demand?  [Those
> fields may be used when filtering, especially mlist.  But filtering is
> pretty much only done when you fetch new mail (isn't it?), so you can
> maybe compute that field on demand instead of keeping it around.]
> 
> The goal is to reduce the working set.  Moving all that stuff to another
> mmap() cache just moves your working set from the heap to elsewhere :)

Correct. But the kernel does an awesome job (if you don't have a swap
partition -- mobile device --) of swapping it out. And an even better
job if your mmap is read-only (which it is, in this case).

The third idea sounds a little bit like the disk summary idea. Jeffrey
and Michael can probably explain that one better than me.


-- 
Philip Van Hoof, software developer at x-tend 
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
work: vanhoof at x-tend dot be 
http://www.pvanhoof.be - http://www.x-tend.be