Re: Scaling Evo for large mailsets (was Re: [OT] Re: [Evolution] strategies for handling lots of mail)

[Lee Revell <rlrevell joe-job com>]

On Sat, 2005-06-25 at 07:14 -0400, Jeffrey Stedfast wrote:
> On Sat, 2005-06-25 at 04:23 -0500, Ron Johnson wrote:
> > On Sat, 2005-06-25 at 14:51 +0800, Not Zed wrote:
> > > > There is no exact solution that'll fit to everyone. Let me tell you what
> > > > I'm doing. I have more than 2000 folders and sub/subfolders and it takes
> > > > ~5 min to start Evo on a relevantly powerfull machine (PIV 3000 MHz/512
> > >
> > > Offtopic, i'm just curious ... How can you possibly have (and manage -
> > > and have time left over to do anything other than move mails around)
> > > 2000 folders of critical mail?
> > >
> > > On a more important note, how many total messages is that?  And how many
> > > vfolders do you have?
> > >
> > > I've been radically redesigning the way mail information is stored and I
> > > guess I need to make sure it scales well this far and beyond.
> >
> > Since you asked... ;)
> >
> > Evo+IMAP seems not to scale well when folders get more than 5000ish
> > emails.  Time-to-open a folder takes an IMO inordinate amount of 
> > time scanning all the headers.
>
> the only thing that can be done for this is to populate the message-list
> as the summary information comes in from the server (but this very
> difficult to do since ETable's performance wrt this is horrible). we
> can't make the scan any faster because of all the features users want
> (like attachment icons, mailing-list vfoldering, etc) which requires a
> huge amount of information from the server.
>
> once all the summary information has been "seen" by evolution, it goes a
> lot faster, but still takes a while because we have to ask for the flags
> for all messages to update our summary with any flag changes that may
> have been done since our last folder visit.

Sorry, I don't buy that.  If Outlook can do it fast, we can do it too.

Much of the cost seems to be associated with updating the "Unmatched"
vfolder.  As I pointed out on evolution-hackers a few weeks ago, the
"Updating vfolders..." step during startup is something like O(N!) where
N is the number of folders - as it iterates over each folder it rescans
all headers in each previous folder.

You can demonstrate by hacking the code to disable this update and
startup is 10x faster.

Lee