Re: [Evolution-hackers] Overview of camel-lite

[Philip Van Hoof <spam pvanhoof be>]

On top of all these it will be very interesting to see what the latest
bugfix-style patches to camel-lite were.

The reason is that we are in group coding on Tinymail (and a product
that works on top of Tinymail) for a few weeks.

We have found a lot of problems (while valgrinding a lot of situations)
in both the POP3 and IMAP code of camel-lite. These where not only
problems that are in the code that implements one of these new features.

Of the more serious ones where a complete lack of locking in the POP3
store and folder, whereas this is very much needed for multiple of the
struct members of the types being used.


I have also found a security bug (actually a few, but one is very
serious) a few weeks ago. I first contacted Jeffrey and I think Matthew
too, but I just looked and the bug has not yet been fixed nor have I
seen a lot of action.

Therefore I'm stopping the silence (for security) and making a bug,
marking it as a security one that is critical (as it can be used to
remotely execute code on a victim's computer).

http://bugzilla.gnome.org/show_bug.cgi?id=447414

Please fix this asap and contact distributors of Evolution about it.



On Mon, 2007-06-11 at 10:57 +0300, Philip Van Hoof wrote:
> I'm going to make an overview of camel-lite so that if people read the
> discussions here, that they know what it's all about.
> 
> Camel-lite is the fork of Camel that Tinymail is using internally right
> now. Camel-lite's code is in my opinion not really in a good shape, it
> more or less implements certain features that are necessary for
> Tinymail.
> 
> Bad things:
> 
>  * Some of the features have been done in a hackish way. 
> 
>  * Mostly this had to do with non-blocked reads not being very well
>    supported within the code and design of upstream camel
> 
>  * Removes support for a few for Evolution very important but for
>    Tinymail unused features
> 
>  * It's a fork (effectively, yes)
> 
> Neutral:
> 
>  * I am trying to get some of the changes in upstream Camel. Although
>    I'm selective because I don't want to bore the Evolution team with
>    features that they'll most likely not consider for inclusion anyway.
> 
>    My point of view is that if they show interested, I'll bump the
>    priority of trying to clean it up and getting it upstream
> 
> Support:
> 
>  * Push E-mail through IMAP IDLE, IMAP commands shutdown and restart the
>    IDLE state of the connection. A non-blocking read handles EXISTS,
>    FETCH and EXPUNGE events (you see messages being added almost
>    instantly when they either get added or received)
> 
>  * Better detecting of EXPUNGE through / partly caused by IMAP IDLE (you
>    see your messages getting removed almost instantly if you remove
>    messages from the folder from another E-mail client)
> 
>  * Better detecting and handling of FETCH through / party caused by IMAP
>    IDLE (flag changes are effectively working instantly with for example
>    Cyrus: you see your view changing almost instantly if you work from
>    another E-mail client with your folder)
> 
>  * Better progress indicating. Camel-lite doesn't use percentages but "n
>    of nth" values. For example bytes of a message or items of a summary
>    download
> 
>  * Support for BINARY fetching messages, which is more bandwidth
>    efficient in some situations 
> 
>  * Support for CONDSTORE which is far more bandwidth efficient when
>    synchronising folders
> 
>  * Support for receiving a message while summary is being downloaded.
>    Camel-lite creates for this single-use-case a new socket (this is a
>    very new feature that is being bug-fixed these weeks)
> 
>  * Support for displaying the recently received headers early while
>    folder summary is being downloaded (very neat, and extremely useful
>    with the parallel receiving of a message being functional now)
> 
>  * Stores already-received summary information very early, recovers from
>    already stored information (saves bandwidth in case of a connection
>    failure)
> 
>  * Makes the selection (the amount of columns or fields) of the summary
>    download a lot smaller
> 
>  * Doesn't store a message temporarily in memory while retrieving it:
>    immediately streams from the socket to the file stream in the cache.
>    This is obviously an important memory-usage fix for large messages on
>    mobile devices.
> 
>  * Implements summary support for POP3, makes the POP3 provider of Camel
>    a full and real CamelStore that can be used just like the IMAP one.
>    This, by the way, uses TOP for retrieving and generating the summary
> 
>  * Implements partial message retrieval (only retrieving the readable
>    body mime part of a message) for both IMAP and POP3 (with POP3 the
>    connection is forcefully cut, which might not work well with all POP3
>    servers. Although this is regretfully one of the few only solutions
>    for this type of feature with POP3)
> 
>  * A lot of bug fixes and locking improvements (also a lot of new
>    situations to support and lock correctly, so it can also be seen as a
>    non-improvement depending on your point of view -- stability vs.
>    features --)
> 
>  * A lot of small memory-usage improvements (A lot of g_new's are now
>    GSlice's)
> 
>  * Full replacement (> 95% of the allocations) of EMemChunk with GSlice
> 
>  * The CamelFolderSummary loaded in mmap, in stead of fread'ed into
>    malloc'ed blocks of memory (this is of course a really big memory
>    improvement in terms of allocated heap -- the kernel is really smart
>    about mmap, and it's quite fast too .. indeed (probably faster than
>    fread for loading, and ~ equally fast on a device with few memory
>    resources during usage) --)
> 
>  * "All" compilation warnings fixed (we usually compile with -Wall and
>    -Werror). Fixing this caused four major bug fixes in initialisation
>     of variables in Camel.
> 
> Planned:
> 
>  * Improved CamelFolderSummary implementation (blocks of mmap-ed files
>    and the flags in a different file)
> 
>  * QRESYNC support in parallel with CONDSTORE (Only Isode MBox is soon
>    going to support QRESYNC, but it's cool so .. why not)
> 
>  * Fetching individual mime parts and letting CamelMimePart fetch
>    requested mime parts (in stead of always fetching full messages to
>    the cache)
> 
>  * NOTIFY support (read IMAP/Lemonade drafts)
> 
>  * CONTEXT support (read IMAP/Lemonade drafts)
> 
>  * Support for SEARCH over multiple folders (IMAP drafts, future)
> 
>  * Support for server-side SORT
> 
> 
> Not planned:
> 
>  * Pipelining (maybe some day with or in the IMAP4, or with or in
>    libspruce)
> 
> 
> 
-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://www.pvanhoof.be/blog