Re: [Evolution-hackers] camel-folder-summary.c with mmap

[Philip Van Hoof <spam pvanhoof be>]

On Mon, 2006-07-10 at 09:57 +0200, Philip Van Hoof wrote:
> On Sun, 2006-07-09 at 18:08 +0200, Philip Van Hoof wrote:
> > On Sun, 2006-07-09 at 14:59 +0200, Philip Van Hoof wrote:
> > 
> > > Note about this E-mail. I'm assuming people who will read this, have a
> > > technical in-depth knowledge of Camel and camel-folder-summary.c

SomeExtraNotes

There's still one or two small bugs that cause the file position to be
*sometimes* two bytes incorrect. It's kinda hard to find why because of
the many possibilities and rather difficult to interpret file format
(with its token compression and added, imho mostly unneeded, information
per specific provider implementation).

So if you are going to test this, and you get funny hangs in
decode_uint32: that's probably because the offset in the mmap() addr
pointer got wrong, and it's now hanging in a "00" byte (decoding such a
byte to a uint32 basically means that it'll loop forever over it).

In gdb, use "up" and "print *(char*)s->filepos" to get an idea what it
stored as the last interesting position. It should point at the byte
right after the last thing that got read.

The ideal situation would be developing a new summary format that is
more suitable for mmap(), but not backward compatible. There was way to
many things to fix (like strdup()ing things) for using fopen() in the
new "m" mode. That mode also isn't platform independent, of course.

I tried my new format on an old implementation, and it's not really
working. So I'm soon going to check what's wrong here (can't be very
hard to fix this, probably the pstring length of empty strings or
something like that).

The memory savings are, however, significant. The speed, ie. when
sorting the summary view on tinymail, which causes a lot string-reads in
the mmap, is very very good. Opening a lot file descriptors and seeing
the amount of mmap()'s grow on a Evolution instance ... didn't slowdown
my desktop (it did scale). Overall: I'm extremely satisfied with the
results and I haven't even experimented with posix_madvise().

The load-speed is definitely faster than the current fread/malloc/strdup
implementation. I'm soon going to provide cachegrind results on that.


-- 
Philip Van Hoof, software developer at x-tend 
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
work: vanhoof at x-tend dot be 
http://www.pvanhoof.be - http://www.x-tend.be