Re: [Evolution-hackers] Camel mmap summary ideas, proposal for a meeting

[Jeffrey Stedfast <fejj novell com>]

There's a couple problems that I can think of that will need to be
solved in order for mmap to work (at least for Evolution, altho at least
a few of the problems also apply to tnymail)

1. (Evolution only?) fd usage will easily max out on systems like
Solaris and/or installations where the user has a large number of
folders. Each mmap'd file requires a persistantly open file descriptor,
which, especially combined with vFolders, will be difficult to keep
under the system threshold.

tnymail solves this by never allowing more than a single folder to be
opened, but Evolution can't enforce that quite as easily. Even with
the ::open()/::close() idea I proposed a few years ago (or hacking
Evolution to try and do what tnymail does even), vFolders will still be
problematic because they keep all of their source folders opened
(mmap'd) so that the summary info is available.


2. As pvanhoof has discovered, when new messages arrive in an opened
folder, the summary info is just added to the array as malloc'd objects
(well, structs) and are not immediately written to disc.

While, yes, the simple solution is to do what pvanhoof proposes: write
those entries to disc and then re-mmap the file - this is kinda gross
and as Varadhan has pointed out, is not as efficient because we'd have
to reparse the entire summary file again. We also would have the
following problems:

  a. remapping the file also means that all string pointers currently in
use by the application become invalid.

  b. to fix this, you'd need to emit a folder_changed signal letting the
application know that all of the original objects were removed and
replaced by a new set meaning that the message-list would be required to
re-build each time anything changed which is less than ideal. (okay,
yes, I believe it currently mostly does this already because of some
problems with incrememntally adding stuff to ETable but it means that
this problem can NEVER be fixed if we go the mmap route). It would also
mean that those MessageInfo objects wouldn't persist through
folder_changed events which would suck.

  c. you'd also need to sync any changes to flags/tags to disc before
you started appending the new message info's so that you don't lose any
data (this isn't really a problem, just listing it here so people don't
overlook this).


It also seems to me that if we were really going to be serious about
using mmap as a real solution (and not just a hack), we'd have to
redesign the summary files to group all the string data together to try
and keep strings in contiguous pages to keep page swapping to a minimum.
The current file layout is terribly inefficient for this.

-- 
Jeffrey Stedfast
Evolution Hacker - Novell, Inc.
fejj novell com  - www.novell.com