Re: [Evolution-hackers] =?iso-8859-1?q?Moving_the_struct_instance_hea?= =?iso-8859-1?q?p_space_to=09mmap?=

Heya folks,

On Monday 11 September 2006 11:26, Philip Van Hoof wrote:

> I wouldn't be against using one of those 'embedded' database engines
> like sqlite3*, mysql-embedded or db* and ask for a cursor at the
> model impl of a treeview.
> * sqlite3_prepare(), sqlite3_step(), sqlite3_reset() etc etc
> I fear copying the entire query result in the memory of the
> application (how most developers use databases in their apps)
> wouldn't reduce memory consumption at all. Launching a full new SQL
> query each time a row becomes visible, is going to cost a lot in
> performance. Since sorting would be done at the db, it might indeed
> be 'fast enough'. On a mobile device, screens are small. Amounts of
> visible headers too. 20 queries isn't a lot.
> > Maybe gtktreeview changed since i last looked at it.  It was rather
> > slow too, iirc.  There's only so much virtualisation you can do
> > with tree's anyway, some of the data you need all of the other data
> > to calculate - unless you keep another on-disk index in 'tree sort'
> > order, and virtualise even that.
> Sorting is indeed still a pain. Showing once the model is sorted
> isn't anymore with the current gtktreeview and fixed-row height
> turned on.
> The cursor idea and a db engine might be a quick solution for this.
> Databases are often very good at sorting stuff.

Having attempted various ways of implementing on-disk and in-memory 
indexing, the last and current being mmap'd binary on-disk structures 
(with a lovely collection of data integrity, robustness and performance 
issues, (NFS, anyone?)), and having long battled huge memory footprints 
in our applications, we (the KDEPIM developers) have decided that a 
database approach to the indexing (and the caching) problem is probably 
our best bet. 

That's the heart of Akonadi, the PIM data storage server we are 
currently implementing as part of KDE 4. We are trying hard to make it 
as non-KDE specific as possible, the core data-transfer protocol is 
IMAP (with extensions for non-mail mimetypes), the control protocol is 
DBUS. The client side library implementation uses kdelibs, but a glib 
based implementation (which could then be wrapped for ruby, python, 
etc) is something we would love to see. The server side is implemented 
using SQLite and Qt-Core, but nothing from KDE, and no gui stuff. It's 
portable to all KDE4 platforms, including Windows. The (KDE) library on 
the client side makes heavy use of Qt4's model/view framework, which 
allows load-on-demand, etc. Searches, filters and sorts can happen on 
the server side, sing external search provider processes. Just to give 
you an idea, there's of course much more to it. And no, we are not by 
default storing mail in a db, just metadata. It can cache payloads, 
though, if that is desired.

I've talked to a few of the evolution hackers in Bangalore about this at, when it was still a wild idea in the back of my mind, but 
it has since come to a state where the initial implementation is 
delivering stuff and most of the main pieces are in place. This would 
be an ideal time for any interested party to have a closer look, find 
flaws in our approach, contribute their experience with building 
similar systems (EDS!) and maybe even write a glib client library which 
would open this up to the rest of the freedesktop world, something that 
we would very much like to see. If there is any possibility at all to 
achieve a shared pim data storage backend on the free desktop we (KDE) 
are committed to attempting it.

Any and all input from any of you would be very much welcomed. We have a 
lot of respect for the Evolution team, both present and past, and as 
someone who's done a lot of work on KMail's IMAP implementation I have 
particular respect for you, Michael. I feel your pain, dude ;). I've 
also been following Philip's work on tinymail with interest, because 
it's the kind of work I would love to do on KMail but lack the time, at 

In short, this discussion reminds me a lot of similar ones we had over 
in K-town which eventually led to Akonadi, so I couldn't resist to 
pitch it to you a bit. My apologies for hijacking the thread. If you 
have more concrete questions as to the kinds of problems we are facing 
with memory mapped index files, feel free to ask in PM or on 
evolution-hackers, which I'm subscribed to.


Till Adam

Attachment: pgpMjVlCvoNec.pgp
Description: PGP signature

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]