Heya folks, On Monday 11 September 2006 11:26, Philip Van Hoof wrote: > I wouldn't be against using one of those 'embedded' database engines > like sqlite3*, mysql-embedded or db* and ask for a cursor at the > model impl of a treeview. > > * sqlite3_prepare(), sqlite3_step(), sqlite3_reset() etc etc > > I fear copying the entire query result in the memory of the > application (how most developers use databases in their apps) > wouldn't reduce memory consumption at all. Launching a full new SQL > query each time a row becomes visible, is going to cost a lot in > performance. Since sorting would be done at the db, it might indeed > be 'fast enough'. On a mobile device, screens are small. Amounts of > visible headers too. 20 queries isn't a lot. > > > Maybe gtktreeview changed since i last looked at it. It was rather > > slow too, iirc. There's only so much virtualisation you can do > > with tree's anyway, some of the data you need all of the other data > > to calculate - unless you keep another on-disk index in 'tree sort' > > order, and virtualise even that. > > Sorting is indeed still a pain. Showing once the model is sorted > isn't anymore with the current gtktreeview and fixed-row height > turned on. > > The cursor idea and a db engine might be a quick solution for this. > Databases are often very good at sorting stuff. Having attempted various ways of implementing on-disk and in-memory indexing, the last and current being mmap'd binary on-disk structures (with a lovely collection of data integrity, robustness and performance issues, (NFS, anyone?)), and having long battled huge memory footprints in our applications, we (the KDEPIM developers) have decided that a database approach to the indexing (and the caching) problem is probably our best bet. That's the heart of Akonadi, the PIM data storage server we are currently implementing as part of KDE 4. We are trying hard to make it as non-KDE specific as possible, the core data-transfer protocol is IMAP (with extensions for non-mail mimetypes), the control protocol is DBUS. The client side library implementation uses kdelibs, but a glib based implementation (which could then be wrapped for ruby, python, etc) is something we would love to see. The server side is implemented using SQLite and Qt-Core, but nothing from KDE, and no gui stuff. It's portable to all KDE4 platforms, including Windows. The (KDE) library on the client side makes heavy use of Qt4's model/view framework, which allows load-on-demand, etc. Searches, filters and sorts can happen on the server side, sing external search provider processes. Just to give you an idea, there's of course much more to it. And no, we are not by default storing mail in a db, just metadata. It can cache payloads, though, if that is desired. I've talked to a few of the evolution hackers in Bangalore about this at foss.in/05, when it was still a wild idea in the back of my mind, but it has since come to a state where the initial implementation is delivering stuff and most of the main pieces are in place. This would be an ideal time for any interested party to have a closer look, find flaws in our approach, contribute their experience with building similar systems (EDS!) and maybe even write a glib client library which would open this up to the rest of the freedesktop world, something that we would very much like to see. If there is any possibility at all to achieve a shared pim data storage backend on the free desktop we (KDE) are committed to attempting it. Any and all input from any of you would be very much welcomed. We have a lot of respect for the Evolution team, both present and past, and as someone who's done a lot of work on KMail's IMAP implementation I have particular respect for you, Michael. I feel your pain, dude ;). I've also been following Philip's work on tinymail with interest, because it's the kind of work I would love to do on KMail but lack the time, at present. In short, this discussion reminds me a lot of similar ones we had over in K-town which eventually led to Akonadi, so I couldn't resist to pitch it to you a bit. My apologies for hijacking the thread. If you have more concrete questions as to the kinds of problems we are facing with memory mapped index files, feel free to ask in PM or on evolution-hackers, which I'm subscribed to. Cheers, Till Adam
Attachment:
pgpa995gWnmXx.pgp
Description: PGP signature