Re: [Evolution-hackers] Camel mmap summary ideas, proposal for a meeting

[Jeffrey Stedfast <fejj novell com>]

On Wed, 2006-07-12 at 20:05 +0200, Philip Van Hoof wrote:
> On Wed, 2006-07-12 at 13:34 -0400, Jeffrey Stedfast wrote:
> 
> > There's a couple problems that I can think of that will need to be
> > solved in order for mmap to work (at least for Evolution, altho at least
> > a few of the problems also apply to tnymail)
> 
> > 1. (Evolution only?) fd usage will easily max out on systems like
> > Solaris and/or installations where the user has a large number of
> > folders. Each mmap'd file requires a persistantly open file descriptor,
> > which, especially combined with vFolders, will be difficult to keep
> > under the system threshold.
> 
> Agree. Each folder keeps one filedescriptor open. On systems that have a
> lot Evolution instances with a lot users that have a lot folders, this
> is not going to be nice.
> 
> However. A kernel that scales can probably easily handle a few thousand
> open file descriptors. And the drawback of closing them (like what the
> current implementation of Camel does) is that you need to put all the
> content in malloc()-s. Which consumes a lot more memory resources than
> an open file descriptor in the kernel would.
> 
> > tnymail solves this by never allowing more than a single folder to be
> > opened, but Evolution can't enforce that quite as easily. Even with
> > the ::open()/::close() idea I proposed a few years ago (or hacking
> > Evolution to try and do what tnymail does even), vFolders will still be
> > problematic because they keep all of their source folders opened
> > (mmap'd) so that the summary info is available.
> 
> Indeed
> 
> > 2. As pvanhoof has discovered, when new messages arrive in an opened
> > folder, the summary info is just added to the array as malloc'd objects
> > (well, structs) and are not immediately written to disc.
> 
> > While, yes, the simple solution is to do what pvanhoof proposes: write
> > those entries to disc and then re-mmap the file - this is kinda gross
> > and as Varadhan has pointed out, is not as efficient because we'd have
> > to reparse the entire summary file again.
> 
> Which however isn't as expensive as the current implementation. If you
> posix_madvise POSIX_MADV_WILLNEED, the kernel will probably in a very
> efficient way copy the content of the file from the filesystem into
> memory (and will most likely not page it out until you advise the kernel
> to use a less gross mapping). I don't know this for sure, I haven't
> checked the Linux kernel implementation (nor the Solaris one) for mmap.
> 
> > We also would have the following problems:
> 
> >   a. remapping the file also means that all string pointers currently in
> > use by the application become invalid.
> 
> Correct. This is a real problem indeed. 
> 
> >   b. to fix this, you'd need to emit a folder_changed signal letting the
> > application know that all of the original objects were removed and
> > replaced by a new set meaning that the message-list would be required to
> > re-build each time anything changed which is less than ideal. (okay,
> > yes, I believe it currently mostly does this already because of some
> > problems with incrememntally adding stuff to ETable but it means that
> > this problem can NEVER be fixed if we go the mmap route). It would also
> > mean that those MessageInfo objects wouldn't persist through
> > folder_changed events which would suck.
> 
> Sounds like some refactoring/rethinking work. But imo not unsolvable.
> Looking at the code in evolution/mail is going to be a good idea anyway.
> 
> Components that expect strings to be there, while the strings weren't
> intended to be in memory for ever anyway, implement a bug imo. If they
> wanted the string to be there until they decide not to use it anymore,
> the object would need reference counting or they should copy the
> information.

CamelMessageInfo's are ref counted.

> 
> Afaics doesn't the CamelMessageInfo type have reference counting. So ..

yes :)

I suppose it could be possible to put signals on the MessageInfo objects
to notify the user of it that the string pointers have changed. would
make MessageInfo's more cumbersome to use, but might "solve" the problem
above.

-- 
Jeffrey Stedfast
Evolution Hacker - Novell, Inc.
fejj novell com  - www.novell.com