Re: [Evolution-hackers] Memory usage of CamelFolderSummary and CamelImapFolder

[Philip Van Hoof <spam pvanhoof be>]

On Thu, 2006-11-16 at 11:41 -0500, Joe Shaw wrote:

Hey Joe,

> On Thu, 2006-11-16 at 16:13 +0100, Philip Van Hoof wrote:
> >   o. The CamelFolderSummary uses mmap. This significantly reduces memory
> >      usage because an mmap is on-demand paged.
> 
> Does the on-disk format of the CamelFolderSummary change much or at all?
> In reading a summary from disk with Beagle, the main problem we've found
> is that it is entirely unsearchable, because records within the file are
> of variable length and there is no end-of-record marker, which means
> that you can't open the file, seek to some random location, and expect
> to find where the next (or previous) message begins.  This means that
> any time the summary changes, we have to walk the whole thing over again
> to see changes.

It does change. Now the files are mmap()able and all strings have end
markers (\0 characters). All strings are also data-padded to four bytes.

So you can mmap() the file and search for strings, once found you can
walk-back to the uid of the E-mail.

The variable-length is still the case. But I can, if necessary, adjust
the summary file format to have end-of-item markers (so that you can
walk the file back until you find that marker, and they you'll know the
exact location of for example the uid).

> There was some work a while back to do a "metasummary", which was
> essentially a summary of the summary for easier searching, but I'm not
> sure what the end result of that was, or if it's in 2.8 or newer.

That is a patch that I have removed from tinymail's Camel (because I
dislike its implementation).


-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog