Re: Scaling Evo for large mailsets (was Re: [OT] Re: [Evolution] strategies for handling lots of mail)

[Jeffrey Stedfast <fejj novell com>]

On Sat, 2005-06-25 at 13:54 -0400, Lee Revell wrote:
> On Sat, 2005-06-25 at 07:14 -0400, Jeffrey Stedfast wrote:
> > On Sat, 2005-06-25 at 04:23 -0500, Ron Johnson wrote:
> > > On Sat, 2005-06-25 at 14:51 +0800, Not Zed wrote:
> > > > > There is no exact solution that'll fit to everyone. Let me tell you what
> > > > > I'm doing. I have more than 2000 folders and sub/subfolders and it takes
> > > > > ~5 min to start Evo on a relevantly powerfull machine (PIV 3000 MHz/512
> > > >
> > > > Offtopic, i'm just curious ... How can you possibly have (and manage -
> > > > and have time left over to do anything other than move mails around)
> > > > 2000 folders of critical mail?
> > > >
> > > > On a more important note, how many total messages is that?  And how many
> > > > vfolders do you have?
> > > >
> > > > I've been radically redesigning the way mail information is stored and I
> > > > guess I need to make sure it scales well this far and beyond.
> > >
> > > Since you asked... ;)
> > >
> > > Evo+IMAP seems not to scale well when folders get more than 5000ish
> > > emails.  Time-to-open a folder takes an IMO inordinate amount of 
> > > time scanning all the headers.
> >
> > the only thing that can be done for this is to populate the message-list
> > as the summary information comes in from the server (but this very
> > difficult to do since ETable's performance wrt this is horrible). we
> > can't make the scan any faster because of all the features users want
> > (like attachment icons, mailing-list vfoldering, etc) which requires a
> > huge amount of information from the server.
> >
> > once all the summary information has been "seen" by evolution, it goes a
> > lot faster, but still takes a while because we have to ask for the flags
> > for all messages to update our summary with any flag changes that may
> > have been done since our last folder visit.
>
> Sorry, I don't buy that.  If Outlook can do it fast, we can do it too.
>
> Much of the cost seems to be associated with updating the "Unmatched"
> vfolder.

*sigh*

will you get off that already? that's not where the slowness is for me.

>   As I pointed out on evolution-hackers a few weeks ago, the
> "Updating vfolders..." step during startup is something like O(N!) where
> N is the number of folders - as it iterates over each folder it rescans
> all headers in each previous folder.

maybe in your case, but not in everyones.

> You can demonstrate by hacking the code to disable this update and
> startup is 10x faster.

doesn't make a lick of difference for me.

but then again, my setup isn't exactly like your setup, so that is to be
expected.

Jeff