Re: [Evolution-hackers] Moving the struct instance heap space to mmap

[Philip Van Hoof <spam pvanhoof be>]

On Mon, 2006-09-11 at 12:07 +0200, Till Adam wrote:

Excluding some people and mailing lists.

> Having attempted various ways of implementing on-disk and in-memory 
> indexing, the last and current being mmap'd binary on-disk structures 
> (with a lovely collection of data integrity, robustness and performance 
> issues, (NFS, anyone?)), and having long battled huge memory footprints 
> in our applications, we (the KDEPIM developers) have decided that a 
> database approach to the indexing (and the caching) problem is probably 
> our best bet. 

> That's the heart of Akonadi, the PIM data storage server we are 
> currently implementing as part of KDE 4.

What I think is interesting about Akonadi for me is whether or not it
will be possible to implement a libtinymail-akonadi when the time is
right for this, and whether or not it would mean that projects like
Qtopia would be interested (because I would and am interested in their
needs when it comes to E-mail client application development).

I'm probably the least religious-about-it person in the G-town,
cooperation would mean benefits for both projects.

> Any and all input from any of you would be very much welcomed. We have a 
> lot of respect for the Evolution team, both present and past, and as 
> someone who's done a lot of work on KMail's IMAP implementation I have 
> particular respect for you, Michael. I feel your pain, dude ;). I've 
> also been following Philip's work on tinymail with interest, because 
> it's the kind of work I would love to do on KMail but lack the time, at 
> present.

While doing tinymail ...

I conclude that the most important part for an E-mail client on a mobile
device is to attempt to consume as few as possible memory per non-
visible summary item. Where "the summary" is the information being shown
to the user when he opens a folder (the from, received, to, subject, the
read-state, indication of attachments and needed meta data for threaded
sorting per message in the folder).

Non-visible data means data that can't be read by the user of the
device/software because the view isn't showing it to the user (it
doesn't fit on the screen or window where the view is).

This of course means that the view should pull what it needs from the
model. The model shouldn't push what it has to the view. For example
each time a row becomes visible, the view would pull the information
from the model (the model being a list-model in this case). A view will
pull row-per-row, not a full list.

It's important that the method for giving the view an item in the model
can be implemented.

A database cursor in the model is suitable for this as I can let the
view tell the model: "I want the next or the previous item". This is
exactly what a cursor is about: iterating the result of a query.

I also need a way to go to the "nth" item while knowing the "current"
location. By that I mean that I can store in the state of the app. what
the "current" location is, and that when I at some point need the "nth"
I can efficiently choose the walking direction (nexts or prevs).

Currently it's implemented by doing sufficient "next" or "prevs" until I
reach the "nth" location. A database cursor can also be used for this.

If I can skip-sequential walk the cursor, it's even better. If the
database has support for traveling a cursor to the "nth" location by
itself, in a very efficient way, then that would be utterly cool.

The current model of tinymail is indeed implemented using an iterator.
It simply stores a position-state, and walks next/prevs to go to an
"nth" location in a list of headers in memory (a doubly linked list with
an external instance keeping the "current", called an iterator).

The view (GtkTreeView) will indeed hammer "give me the nth item" on that
list. It happens hundreds of times per second when scrolling fast and
even more when sorting (the compare function of a qsort asks for nth).

> In short, this discussion reminds me a lot of similar ones we had over 
> in K-town which eventually led to Akonadi, so I couldn't resist to 
> pitch it to you a bit. My apologies for hijacking the thread. If you 
> have more concrete questions as to the kinds of problems we are facing 
> with memory mapped index files, feel free to ask in PM or on 
> evolution-hackers, which I'm subscribed to.


-- 
Philip Van Hoof, software developer at x-tend 
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
work: vanhoof at x-tend dot be 
http://www.pvanhoof.be - http://www.x-tend.be