Re: [Evolution-hackers] Introduction and Questions

[Jeffrey Stedfast <fejj novell com>]

On Fri, 2007-06-08 at 14:39 +0300, Philip Van Hoof wrote:
> On Thu, 2007-06-07 at 09:25 -0400, Jeffrey Stedfast wrote:
> 
> > On Thu, 2007-06-07 at 10:56 +0300, Philip Van Hoof wrote:
>  
> > > The best way is to ask for the ENVELOPE and the remaining info using the
> > > normal BODY.PEEK method.
> > 
> > Have you actually ever tested this theory? or did you just pull this out
> > of your proverbial?
> 
> I have. The ENVELOPE-way is slightly better because words like "From:"
> and "To:" are not repeated each time. Although with COMPRESS and/or
> support for deflate in the TLS/SSL library will probably compress that
> difference away mostly.

ENVELOPE alone is no doubt as fast or faster (hopefully faster) than
explicit header fetching (I expect most servers probably cache ENVELOPE
info, whereas explicit header fetching would have to be done by
re-parsing the message) and (probably) less processor intensive
server-side.

> 
> Note I didn't test this with compression. I saw a difference of 5% to
> 10% depending on several factors in bandwidth saving. I usually test all
> this with wireshark, which doesn't help a lot with compressed data. It
> would be nice, indeed, to get some analysis of how data is being
> compressed when compression by either COMPRESS or the SSL library is
> being done.
> 
> Note that I haven't yet implemented COMPRESS in camel-lite, but I will
> do this very soon (I have a CamelStreamGzip thingy ready for this).
> 
> On top of bandwidth, on pure speed a lot IMAP servers will return your
> result faster if you only use ENVELOPE. That's because they optimise on
> the fields that are in ENVELOPE (indexes, etc etc).

nod, which is why imap4 used ENVELOPE. I later had to add explicit
fetching of other headers, though, to maintain feature compat with imap.

> 
> But then again, there's not enough in ENVELOPE to support all of
> Evolution's features, so you'll still have to use BODY.PEEK which will
> most likely slowdown everything back again.

indeed, and it doesn't appear to be faster trying to 
use ENVELOPE + explicit header fetching, /probably/ because the server
still has to spend time parsing the actual message off disk, so the
performance gain of using ENVELOPE for the (typically? cached) summary
info is negated - possibly made even slower (server has to hit 2
code-paths instead of just the 1), but bandwidth is still slightly
decreased, possibly making up for the increased processing time.

> 
> > You should note that Thunderbird does not use ENVELOPE, probably because
> > they found it was LESS efficient to fetch the ENVELOPE +
> > BODY.PEEK[HEADER.FIELDS (REFERENCES CONTENT-TYPE)] on many existing IMAP
> > servers in the wild.
> 
> Probably, indeed.
> 
> > Having myself tested this in the past, I don't recall it making much
> > difference one way or the other.
> 
> On bandwidth, it does. Especially when you have a lot of messages in
> some folders and the server is not compressing things (no COMPRESS nor
> TLS/SSL compressing things for you).

ok, fair enough... not sure how much you really cut out tho, address
lists are likely more bandwidth intensive in the ENVELOPE response than
in raw format.

To: Jeffrey Stedfast <fejj novell com>, Miguel de Icaza <miguel ximian com>\r\n

ENVELOPE equiv:

(("Jeffrey Stedfast" NIL "fejj" "novell.com") ("Miguel de Icaza" NIL "miguel" "ximian.com"))

You can see that the ENVELOPE response is more verbose, so the savings
really depends on individual messages.

You do save on the command echoing from the server, tho...


all in all, yea, probably you save some bandwidth. For most users,
ENVELOPE is probably a win as far as bytes-sent-across-the-wire.

The biggest win for ENVELOPE, in my opinion, is not bandwidth but server
processing (measured in both time and resources). Unfortunately, if you
have to explicitly request headers anyway, this is completely negated :(

> 
> > Sadly using ENVELOPE breaks on at least one proprietary IMAP server
> > which occasionally likes to send back responses with NIL as each and
> > every item, at least until a client has issued a BODY.PEEK fetch
> > request.
> 
> Sounds like a bug in that server implementation.

yes, it is.

> 
> > >  It should be possible to specify per folder
> > > which of the headers are to be fetched, to make it really efficient.
> > 
> > how do you propose calculating the list of headers each folder should
> > request? which headers would you eliminate conditionally?
> 
> I'd probably start with the references in case the user doesn't want his
> headers-view to be threaded, and fetch only the sequence number plus the
> references header the first time threaded is enabled on that folder and
> after that (while its enabled) for each summary-item update too.
> 
> A lot mobile devices's screens are simply not suitable for threaded view
> anyway: introduces way to much wasted white space. Why fetch this
> information and waste the user's GPRS bandwidth and therefore often his
> money too? (for a lot people, GPRS bandwidth costs money per downloaded
> megabyte)
> 
> In case the "From" is not displayed, there's no real need to fetch it
> until the first time the column is added. Same for the dates (date sent
> and date received, usually only one of both is used by the user). 
> 
> Etc etc etc.

I think this proposal is overly complex and for desktop users would just
inconvenience them more than help. Also, the data is used for more than
just displaying strings in the message-list (to my recollection).


Perhaps a better approach would just be to allow applications using
Camel to define what info they want cached based on their feature-set. 

Obviously if they don't support mailing-list vfolders, no need for the
mailing-list header info to be cached. Likewise, if the client doesn't
support threading, no need for References/In-Reply-To. If the client
doesn't support showing an attachment icon in the message-list, then
clearly they don't need Content-Type... and so on.

>  
> > > The imap4 implementation seems to have an ENVELOPE parser, so I guess
> > > either it does it already or it will use ENVELOPE in future.
> > 
> > it fetches:
> > 
> > ENVELOPE BODY.PEEK[HEADER.FIELDS (Content-Type References In-Reply-To)]
> > 
> > and optionally the mailing-list headers
> > 
> > I believe it fetches IN-REPLY-TO (which is included in ENVELOPE)
> > explicitly because some IMAP servers tried to be smart and parse the
> > IN-REPLY-TO header for us and didn't handle embedded phrases which
> > certain broken free-software mailers like to send.
> > 
> > > 
> > > For a mobile device, that works over GPRS, you for example are usually
> > > not interested (not at all) in the mailinglist specific headers.
> > 
> > this is mobile-device specific tho, when I was implementing imap4 and
> > people started trying to use it, it was their #1 complaint (which is why
> > imap4 conditionally requests the mlist headers... tho that flag may be
> > hard-coded to TRUE now, I'm not sure)
> 
> But some people don't use it. Although people more often complain about
> missing features and not so much about that their E-mail client is
> wasting their money on GPRS bandwidth, I still believe it's important to
> support low bandwidth modes too. Also for Evolution, as Evolution is
> often used with laptops.

I think the laptop problem is solved with the "basic headers" feature,
at least as far as collecting new summary info is concerned.

Syncing flags is another story, and where the real
slowness/user-frustration lies.

I'm sure David Woodhouse would agree here.

> 
> I do agree, though, that Evolution has a focus difference compared to
> what I'm doing with Camel. In my opinion is this why splitting Camel
> from Evolution might help: we could then more easily introduce features
> for letting Camel switch from low bandwidth mode, to Evolution mode and
> back.

I'm not sure I follow how this would help in this way... tho I'm not
against splitting Camel out, either.

> 
> Right now any change that isn't absolutely 100% super pro and for
> Evolution and Evolution only, gets ignored by you guys at Novell.

I would argue that not to be the case. I would instead argue that the
reason many of your patches haven't been incorporated into Camel is that
they are hackish and/or tailored explicitly for /your/ needs, and not
for the good of the many.

> 
> > > Note that Tinymail's camel-lite implements all the needed solutions to
> > > this bandwidth consumption problem. And that its code is, although a lot
> > > of work because the Evolution maintainers didn't seem interested at the
> > > time it was happening, definitely portable to upstream.
> > > 
> > 
> > some of your changes are broken wrt working with all imap servers in the
> > wild.
> 
> UID SEARCH, although a lot more bandwidth efficient than UID FETCH for
> getting a list of UIDS, doesn't work on some servers. For example the
> Citadel IMAP service used to fail on this, although they have fixed this
> after they contacted me about the problem.

That's not the one I was thinking of, actually... I was more thinking of
the BODY(STRUCTURE) stuff.

At least Courier-IMAP doesn't seem to be able to parse nested multipart
messages reliably and so the BODY responses would be invalid, so piecing
parts back together again obviously became unreliable as well.

> 
> My opinion is that IMAP server developers should ALSO fix their crap.

if only it were this easy...

Even supposing that you can always get the server software developer(s)
to fix their brokenness and release a new bug-fixed version, you still
have to wait for the provider to upgrade their software. If most of
their users are fine with what they currently have running, why upgrade
it? And if it hadn't been working for most of their users, why were they
still using it?

It takes a lot of time for this to happen, and in the meanwhile your
users can't read mail, so they switch to something else that will work.

> 
> If they want to get users with Mobile devices, they'll have to implement
> the new Lemonade features, indeed.

Something tells me that's not a high priority for most of them.

> 
> Although I will put-in some more checks that try to detect whether the
> IMAP server will incorrectly deal with the changes (so that the old
> method can be fallen back to), I will not strongly steer Tinymail in the
> direction of supporting each and every piece of shit IMAP server.

That's where tnymail's path differs from Evolution's, then, I'm afraid.

As much as I'd love to the approach of "if the other software doesn't
follow the standards, screw them!", that's not Novell's stance... they
need their customers to be able to drop Evolution in as a replacement to
whatever mail client they were using before (Outlook?), and have it
"just work" no matter what broken software it has to inter-operate with.

There's a huge number of bugs in bugzilla about inter-operating with
other software (broken mailers, broken servers, broken this/that/etc)

> 
> If they don't get it right then they should not have users via Tinymail.
> That's putting the burden of supporting their server-defects on me.

Then don't expect to have many tnymail users. Users don't distinguish
between your software being broken and everyone elses - if your software
doesn't inter-operate with Broken Mailer v3.0 (or Broken Mail Server
v3.0), then it's YOUR fault, not Broken Mailer v3.0's. It's a very very
frustrating fact, hence why I couldn't take working on mailer software
anymore - there's way too much broken mail software out there.

That said, I agree - it's unfair.

> 
> > > ps. In my opinion is also the imap4 project getting the majority of
> > > its design wrong.
> > 
> > thank you, oh king of software design.
> 
> My opinion on this kind of "messages to each other", Jeffrey, is that in
> stead of trying to fight each other, we'd better cooperate and make
> things better.

Then don't be so negative. I'd hardly say that imap4 has a majority of
its design wrong.

Try some constructive criticism for a change...

> 
> But then again, this is your choice, not mine.
> 
> Welcome back to the Camel team, by the way. The Evolution guys can
> definitely use your experience.

I'm not back, I'm merely commenting :)

The Evolution devs are doing fine without me :)

Jeff