Re: Last version of the new summary storage engine

[Philip Van Hoof <spam pvanhoof be>]

ps. I added Jeffrey and Matthew in BCC as I asked their opinion on this
experiment. 

On Mon, 2007-12-10 at 20:33 +0000, Dave Cridland wrote:
> On Mon Dec 10 20:15:51 2007, Philip Van Hoof wrote:
> > 
> > Reference counting fixes. Prepared an API (it has a summary.h now).
> 
> Some comments and questions...
> 
> 1) Why is UID a char *? guint32_t would seem better. uint32_t would  
> seem more portable still.

Because some formats (not IMAP ones) have string UIDs :-(. For example
for Maildir I would like to use the filename. The UIDL of POP is also
not an integer-as-a-string. Etc etc.

Perhaps I can make something that converts a string to an unique UID-as
-integer, though?

Might indeed be more convenient. Pointing to the uid is four bytes plus
the heap admin (gslice, or GStringChunk, though) and the uid itself is
again ~6 bytes plus the '\0' ... So this would indeed make sense.

Right now I'm even storing it twice, once as GHashNode too :(, and it's
a g_strdup too (not necessary, though, I can remove that strdup for the
"uids" hashtable's keys).


> 2) size should also be a uint32_t, since otherwise on 64-bit  
> platforms you'll have a 64-bit size, there, which isn't needed.

Right. Unless you want to support terrabyte-sized E-mails of course. 

Who knows :), 640kb ought to be ...

> 3) What am I meant to call when I get an EXPUNGE or VANISHED?

This is not done yet.

My current idea is that EXPUNGE and VANISHED will merely set a flag. I
still need to implement something that skips EXPUNGED-marked items when
dumping the SummaryBlock and something detects that 50% of the items are
marked as such and that therefore the SummaryBlock needs a rewrite.

Probably something that runs in the background at rare times. This is
why I early-on added the locking.

I might also consider a flock(), although I dislike file locks. The
problem is that multiple instances might try to use the same folder's
cache (although I absolutely don't recommend this with Tinymail, I also
can't control it unless I flock it).

Not sure ... etc etc (I'm concerned about NFS and speed of flock).

> 4) I think I need an API which freezes the summary - when I know I'll  
> be doing multiple changes - so it won't write it to disk until I tell  
> it. So, for example, I'd freeze the summary, issue the EXPUNGE  
> command, process the FETCH and VANISHED I get back, then thaw it,  
> letting it write to disk, when I see the tagged response. Meanwhile,  
> of course, the summary needs to be accurate in as much as if, partway  
> through the expunge, I ask for a message, the sequence number etc  
> will still be accurate.

Sounds like a good idea, indeed. I'll keep this in mind while coding on
this idea further.


-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be