Re: [Evolution-hackers] Moving from the single mbox file format for the local folders

[chenthill <pchenthill novell com>]

On Wed, 2009-12-16 at 09:56 -0500, Jeffrey Stedfast wrote:
> On 12/15/2009 02:46 PM, Chenthill wrote:
> > Hi fellow hackers!!
> >     I have been working for a while during last week on one the blockers
> > in evolution - https://bugzilla.gnome.org/show_bug.cgi?id=550414 -
> > 'Folder and summary mismatch error'(old one -
> > https://bugzilla.gnome.org/show_bug.cgi?id=213072). As a matter of fact
> > we have been working as a team to get the blockers down. I have not been
> > able to reproduce the issue or yet find the exact problematic area.
> >
> > 	The mismatch in the frompos index in the folder summary may be caused
> > by either a threading issue or a crash while storing the indexes. I am
> > still investigating it to find the real cause.
> >   
> 
> I don't think it's a threading or crash issue.
Looking through the comments from both the bugs and the fixes that have
gone through, i had this thought. Any other clues ?

Thinking about the amount of time this bug has been there for (primarily
with our mbox implementation) , I thought of making some change which
could benefit more rather than trying to just fix this. This might not
be an ideal way to think though :)


- Chenthill.
> 
> > 	Looking at other issues such as, 
> > https://bugzilla.gnome.org/show_bug.cgi?id=522433 - 'Fails opening mbox
> >   
> >> 2GB', just got a thought if we could solve both the issues by,
> >>     
> >   
> 
> This just means the proper LARGEFILE flags are not being used at compile
> time. Either EDS's configure isn't doing proper checks or else Evolution
> itself isn't doing proper checks and there is some sort of clash.
> 
> An easy way to fix this is to do what I did with GMime, which is to
> simply make all public stream APIs that use off_t use goffset instead
> (I'm sure Matthew will want to do this anyway). Then the problem is much
> simpler to solve - just make sure that Camel uses the proper CFLAGS for
> LARGEFILE support (which you can steal from GMime's configure scripts).
> 
> > Approach #1,
> > migrating local storage from mbox to maildir format. With maildir I have
> > heard about two issues,
> >
> > * Not able to create subfolders under INBOX -
> > https://bugzilla.gnome.org/show_bug.cgi?id=536240 .
> >   
> 
> This is just a bug and it should be fixed regardless.
> 
> > Approach #2,
> > Migrate from a single mbox file per folder to mbox per email. Srini
> > mentioned an advantage that this would avoid the file renames that
> > maildir does. I think this is much like how other remote providers in
> > evo store the email.
> >   
> 
> I'm not sure if you mean the CamelImapMessageCache way or CamelDataCache
> (as someone else mentioned in this thread).
> 
> Either way, it seems a messy way of organizing messages as well as
> costly in terms of inodes (and possibly in wasted disk space, although
> Meeks' email seems to suggest file-per-message might not be that bad).
> Then there's the problem of getting mail into and out of this storage
> scheme. (Note: Maildir would be less inode-intensive)
> 
> I think that Evolution should always choose to use a standard mailbox
> format rather than make up its own. So if the consensus is to move away
> from mbox, then my vote would be with Maildir.
> 
> We originally chose mbox (when I say originally, I mean for 2.x), it was
> largely done because most other popular clients also sued mbox. One of
> the things we had to do was to figure out a way to structure an mbox
> folder tree, and the way I did that was to mimic Thunderbird's folder
> layout (which was quite nice). IMHO, this was an added bonus because I
> think that Thunderbird was the most likely candidate for people to be
> switching to/from as it was the other major mail client at the time -
> making migration as simple as a cp -r (or an mv) is pretty slick.
> 
> Performance gotchas with Maildir:
> 
> There were some comments earlier in this thread about not wanting to use
> Maildir because of performance problems with rename(). It's not the
> rename() which is the performance problem, but rather the fact that once
> a message file is renamed, the client must scan the directory listing(s)
> looking for the new name (strcmp()ing everything up to the ':' iirc). If
> the volume of mail gets large enough, this could potentially be
> problematic. I believe it was problematic on ext2, but things may have
> changed since then. I should point out that this is ONLY a problem if
> other clients are involved because Evo could (should) keep track of the
> name changes in its summary files.
> 
> Hope this helps,
> 
> Jeff