Re: [Evolution-hackers] Moving from the single mbox file format for the local folders

[Jeffrey Stedfast <fejj novell com>]

On 12/15/2009 02:46 PM, Chenthill wrote:
> Hi fellow hackers!!
>     I have been working for a while during last week on one the blockers
> in evolution - https://bugzilla.gnome.org/show_bug.cgi?id=550414 -
> 'Folder and summary mismatch error'(old one -
> https://bugzilla.gnome.org/show_bug.cgi?id=213072). As a matter of fact
> we have been working as a team to get the blockers down. I have not been
> able to reproduce the issue or yet find the exact problematic area.
>
> 	The mismatch in the frompos index in the folder summary may be caused
> by either a threading issue or a crash while storing the indexes. I am
> still investigating it to find the real cause.
>   

I don't think it's a threading or crash issue.

> 	Looking at other issues such as, 
> https://bugzilla.gnome.org/show_bug.cgi?id=522433 - 'Fails opening mbox
>   
>> 2GB', just got a thought if we could solve both the issues by,
>>     
>   

This just means the proper LARGEFILE flags are not being used at compile
time. Either EDS's configure isn't doing proper checks or else Evolution
itself isn't doing proper checks and there is some sort of clash.

An easy way to fix this is to do what I did with GMime, which is to
simply make all public stream APIs that use off_t use goffset instead
(I'm sure Matthew will want to do this anyway). Then the problem is much
simpler to solve - just make sure that Camel uses the proper CFLAGS for
LARGEFILE support (which you can steal from GMime's configure scripts).

> Approach #1,
> migrating local storage from mbox to maildir format. With maildir I have
> heard about two issues,
>
> * Not able to create subfolders under INBOX -
> https://bugzilla.gnome.org/show_bug.cgi?id=536240 .
>   

This is just a bug and it should be fixed regardless.

> Approach #2,
> Migrate from a single mbox file per folder to mbox per email. Srini
> mentioned an advantage that this would avoid the file renames that
> maildir does. I think this is much like how other remote providers in
> evo store the email.
>   

I'm not sure if you mean the CamelImapMessageCache way or CamelDataCache
(as someone else mentioned in this thread).

Either way, it seems a messy way of organizing messages as well as
costly in terms of inodes (and possibly in wasted disk space, although
Meeks' email seems to suggest file-per-message might not be that bad).
Then there's the problem of getting mail into and out of this storage
scheme. (Note: Maildir would be less inode-intensive)

I think that Evolution should always choose to use a standard mailbox
format rather than make up its own. So if the consensus is to move away
from mbox, then my vote would be with Maildir.

We originally chose mbox (when I say originally, I mean for 2.x), it was
largely done because most other popular clients also sued mbox. One of
the things we had to do was to figure out a way to structure an mbox
folder tree, and the way I did that was to mimic Thunderbird's folder
layout (which was quite nice). IMHO, this was an added bonus because I
think that Thunderbird was the most likely candidate for people to be
switching to/from as it was the other major mail client at the time -
making migration as simple as a cp -r (or an mv) is pretty slick.

Performance gotchas with Maildir:

There were some comments earlier in this thread about not wanting to use
Maildir because of performance problems with rename(). It's not the
rename() which is the performance problem, but rather the fact that once
a message file is renamed, the client must scan the directory listing(s)
looking for the new name (strcmp()ing everything up to the ':' iirc). If
the volume of mail gets large enough, this could potentially be
problematic. I believe it was problematic on ext2, but things may have
changed since then. I should point out that this is ONLY a problem if
other clients are involved because Evo could (should) keep track of the
name changes in its summary files.

Hope this helps,

Jeff