Re: [Evolution-hackers] Introduction and Questions

[Philip Van Hoof <spam pvanhoof be>]

On Thu, 2007-06-07 at 09:25 -0400, Jeffrey Stedfast wrote:

> On Thu, 2007-06-07 at 10:56 +0300, Philip Van Hoof wrote:
 
> > The best way is to ask for the ENVELOPE and the remaining info using the
> > normal BODY.PEEK method.
> 
> Have you actually ever tested this theory? or did you just pull this out
> of your proverbial?

I have. The ENVELOPE-way is slightly better because words like "From:"
and "To:" are not repeated each time. Although with COMPRESS and/or
support for deflate in the TLS/SSL library will probably compress that
difference away mostly.

Note I didn't test this with compression. I saw a difference of 5% to
10% depending on several factors in bandwidth saving. I usually test all
this with wireshark, which doesn't help a lot with compressed data. It
would be nice, indeed, to get some analysis of how data is being
compressed when compression by either COMPRESS or the SSL library is
being done.

Note that I haven't yet implemented COMPRESS in camel-lite, but I will
do this very soon (I have a CamelStreamGzip thingy ready for this).

On top of bandwidth, on pure speed a lot IMAP servers will return your
result faster if you only use ENVELOPE. That's because they optimise on
the fields that are in ENVELOPE (indexes, etc etc).

But then again, there's not enough in ENVELOPE to support all of
Evolution's features, so you'll still have to use BODY.PEEK which will
most likely slowdown everything back again.

> You should note that Thunderbird does not use ENVELOPE, probably because
> they found it was LESS efficient to fetch the ENVELOPE +
> BODY.PEEK[HEADER.FIELDS (REFERENCES CONTENT-TYPE)] on many existing IMAP
> servers in the wild.

Probably, indeed.

> Having myself tested this in the past, I don't recall it making much
> difference one way or the other.

On bandwidth, it does. Especially when you have a lot of messages in
some folders and the server is not compressing things (no COMPRESS nor
TLS/SSL compressing things for you).

> Sadly using ENVELOPE breaks on at least one proprietary IMAP server
> which occasionally likes to send back responses with NIL as each and
> every item, at least until a client has issued a BODY.PEEK fetch
> request.

Sounds like a bug in that server implementation.

> >  It should be possible to specify per folder
> > which of the headers are to be fetched, to make it really efficient.
> 
> how do you propose calculating the list of headers each folder should
> request? which headers would you eliminate conditionally?

I'd probably start with the references in case the user doesn't want his
headers-view to be threaded, and fetch only the sequence number plus the
references header the first time threaded is enabled on that folder and
after that (while its enabled) for each summary-item update too.

A lot mobile devices's screens are simply not suitable for threaded view
anyway: introduces way to much wasted white space. Why fetch this
information and waste the user's GPRS bandwidth and therefore often his
money too? (for a lot people, GPRS bandwidth costs money per downloaded
megabyte)

In case the "From" is not displayed, there's no real need to fetch it
until the first time the column is added. Same for the dates (date sent
and date received, usually only one of both is used by the user). 

Etc etc etc.
 
> > The imap4 implementation seems to have an ENVELOPE parser, so I guess
> > either it does it already or it will use ENVELOPE in future.
> 
> it fetches:
> 
> ENVELOPE BODY.PEEK[HEADER.FIELDS (Content-Type References In-Reply-To)]
> 
> and optionally the mailing-list headers
> 
> I believe it fetches IN-REPLY-TO (which is included in ENVELOPE)
> explicitly because some IMAP servers tried to be smart and parse the
> IN-REPLY-TO header for us and didn't handle embedded phrases which
> certain broken free-software mailers like to send.
> 
> > 
> > For a mobile device, that works over GPRS, you for example are usually
> > not interested (not at all) in the mailinglist specific headers.
> 
> this is mobile-device specific tho, when I was implementing imap4 and
> people started trying to use it, it was their #1 complaint (which is why
> imap4 conditionally requests the mlist headers... tho that flag may be
> hard-coded to TRUE now, I'm not sure)

But some people don't use it. Although people more often complain about
missing features and not so much about that their E-mail client is
wasting their money on GPRS bandwidth, I still believe it's important to
support low bandwidth modes too. Also for Evolution, as Evolution is
often used with laptops.

I do agree, though, that Evolution has a focus difference compared to
what I'm doing with Camel. In my opinion is this why splitting Camel
from Evolution might help: we could then more easily introduce features
for letting Camel switch from low bandwidth mode, to Evolution mode and
back.

Right now any change that isn't absolutely 100% super pro and for
Evolution and Evolution only, gets ignored by you guys at Novell.

Usually the explanation goes like this: "Yeah but Evolution has
priorities this and Evolution is desktop that ..."

The final result of all that this and that is that I just forked it.

I do have to say, though, that I'm strongly noticing that more and more
tasks of the project are being given to the right people. Maybe things
will get better?

I'm for example looking forward to cooperating with Matthew on Camel a
lot more.

> > Note that Tinymail's camel-lite implements all the needed solutions to
> > this bandwidth consumption problem. And that its code is, although a lot
> > of work because the Evolution maintainers didn't seem interested at the
> > time it was happening, definitely portable to upstream.
> > 
> 
> some of your changes are broken wrt working with all imap servers in the
> wild.

UID SEARCH, although a lot more bandwidth efficient than UID FETCH for
getting a list of UIDS, doesn't work on some servers. For example the
Citadel IMAP service used to fail on this, although they have fixed this
after they contacted me about the problem.

My opinion is that IMAP server developers should ALSO fix their crap.

If they want to get users with Mobile devices, they'll have to implement
the new Lemonade features, indeed.

Although I will put-in some more checks that try to detect whether the
IMAP server will incorrectly deal with the changes (so that the old
method can be fallen back to), I will not strongly steer Tinymail in the
direction of supporting each and every piece of shit IMAP server.

If they don't get it right then they should not have users via Tinymail.
That's putting the burden of supporting their server-defects on me.

> > ps. In my opinion is also the imap4 project getting the majority of
> > its design wrong.
> 
> thank you, oh king of software design.

My opinion on this kind of "messages to each other", Jeffrey, is that in
stead of trying to fight each other, we'd better cooperate and make
things better.

But then again, this is your choice, not mine.

Welcome back to the Camel team, by the way. The Evolution guys can
definitely use your experience.


-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog