Re: Moving the struct instance heap space to mmap

[Philip Van Hoof <spam pvanhoof be>]

On Thu, 2006-09-07 at 21:48 -0500, Federico Mena Quintero wrote:
> On Thu, 2006-09-07 at 21:14 +0200, Philip Van Hoof wrote:
> 
> > The *new*/*extra* idea is to create a second index file which contains
> > the offsets to the pointers in the camel summary file. Then mmap also
> > that file. "Extra" because the idea will build on top of the existing
> 
> Ummm, but this won't reduce your working set by very much, will it?
> 
> I haven't looked at the details, but can't you just keep an array in
> memory with pointers to the *start* of each summary block, and then
> compute the other pointers on demand?

Yes. This sounds possible. Each member of the summary, however, would
need a length (or a lot strlens are needed per access).

The "SUBJECT_OFFSET" would be variable because the other members, in the
summary, aren't fixed-sized. Unless you for example use fixed-sized
strings (which would make the mmaped file grow a lot). Consider that you
would have to support the CC field of spam E-mail: 500 E-mail
addresses :), forcing you to make that CC field +- 2k in size per item.

Or the subject of ~ 200 bytes per item, whereas some subjects would
probably be less than 10 bytes. Even for mmap that would become wastage.

But it would be possible, with the offset to the start of the record, to
strlen the strings and that way calculate the position of the next item.

Or to put the length of the strings in a little sub-index per record (a
little bit like reiserfs, hehe). Or to put the length of the string in
front of the string, like pstrings, to avoid the strlen (note that this
makes each string 32 bits larger, unless you encode the integer which
makes data alignment more difficult).


Thanks a lot for your input Federico.

-- 
Philip Van Hoof, software developer at x-tend 
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
work: vanhoof at x-tend dot be 
http://www.pvanhoof.be - http://www.x-tend.be