Re: [Evolution-hackers] Camel Manifesto

[Jeffrey Stedfast <fejj novell com>]

Matthew Barnes wrote:
> With work on Bonobo removal wrapping up, I've finally started taking a
> closer look at Camel (Evolution's mail storage and networking library)
> and laying out plans for where I'd like it to go over the short and long
> term, with the ultimate goal of splitting it off as a useful standalone
> GNOME library (but we're a long way from that).
>
> As usual I'm taking a bottom-up approach, starting with basic cleanup
> chores (both code and development policies) and building up from there.
> Here's some of my thoughts:
>
>
> Backward Compatibility Policy
> -----------------------------
>
> A reverse dependency search in Debian and Fedora reveals the only
> external projects currently linking to Camel are mail-notification,
> evolution-jescs and Anjal (please correct me if I've missed any).
>
> That tells me that until Camel moves out of its parents' basement and
> matures into an real, honest-to-goodness library, fixing its interface
> is more important than maintaining backward compatibility.  Deprecation
> periods for obsolete API are not necessary, in my opinion.  The few
> external projects linking to Camel will just have to keep up with the
> changes every six months.
>   

I think it's a worthy goal to separate Camel out of Evolution.

> That's not a license to go hog wild though.  Some caveats:
>
> 1) The soname -must- be kept accurate.  If you break the API or ABI,
>    increment the soname when you commit the break.  It doesn't matter if
>    the break doesn't affect external projects, nor even if it's already
>    been incremented since the last point release.  Bump it anyway. 
>    "Always bump" is an easy policy to remember.  It makes our own daily
>    development run smoother, and helps ensure a release doesn't slip out
>    with an inaccurate soname.
>
>    If you're not sure if your patch requires a soname increment, please
>    ask in IRC or Bugzilla.  Patch reviewers should try to remember too.
>
> 2) If you must break the API, try to do so in a way that things will
>    fail noisily at build time rather than mysteriously at run time.  For
>    example, if you want to change the behavior of an existing function,
>    it's better to rename the function or change its parameter list so
>    that stale Evolution code will fail to build.
>
> 3) Camel started life as a general purpose mail library and I'd like to
>    try to get back to that.  Camel has become too Evolution-centric in
>    my view, with too many quick-fix hacks for Evolution bugs that would
>    not be appropriate for a general purpose mail library.  I will clean
>    these up as I find them, but try to keep that in mind when altering
>    the API yourself.
>   

I'm not sure what Evolution-specific quick-fix hacks you mean?

>
> Migrate to GObject
> ------------------
>
> Camel's homegrown type system will be replaced with GObject so that
> introspection and D-Bus + language bindings are possible.  CamelObject
> will remain (for now), but as a subclass of GObject.  The redundant
> parts of CamelObject will be removed.
>   

Worthy goals... there are some things like CamelObjectBag and such that
don't have an equivalent in GObject.

> I'm also following GLib and GTK+'s example of sealing up public instance
> data in private sections and enforcing that only its top-level header
> file be included outside of Camel (including the providers).  Unlike
> GLib and GTK+, there will be no transition period.
>   
Okay.

> This will give us a lot more freedom to reorganize the library and
> refactor code without disturbing the ABI.  Debugging is also easier when
> you can trap data accesses through "get" and "set" functions.
>   

Sure, but I don't think any code actually goes behind any get/set
method's back at the moment (or at least didn't back when I worked on
Camel). Regardless, I'm cool with the proposed changes...

> I've been chipping away at this as a side project for the past year (it
> was a good mindless activity when I got burned out on Bonobo removal),
> and I recently published my results to git.gnome.org as a branch named
> "camel-gobject".  The CamelObject conversion is finished -- including
> all the boilerplate changes in the subclasses -- but I haven't finished
> sealing up the API.
>
> The branch probably won't land until 2.31 at the earliest.  The backward
> compatibility policies I described above would be in effect thereafter.
>
>
> Kill CamelStream
> ----------------
>
> This is a distant future goal and will have to happen gradually, but I
> would like Camel to shift to a single-threaded design where all file and
> network operations directly use or are derived from GIO's asynchronous
> file and stream APIs.  SSL support is currently under development for
> GIO, and that's the only missing piece I see at the platform layer.
>
> I realize this is a drastic course correction and will require rewriting
> all the providers and much of the mailer code in Evolution, but I firmly
> believe that the overuse of threads carries much of the blame for
> Evolution's chronic instability over the years and that reversing that
> trend first and foremost requires minimizing our use of threads for I/O
> and relying more heavily on GLib's main loop.
>
>
> Comments and constructive criticism encouraged.
>   

I agree with Michael Meeks' concerns here. I also think there are much
more important fish to fry which are also far easier to tackle.

An IMAP rewrite (which supports IDLE) seems to be something a lot of
users would be very thankful for, for example. The ancient IMAP code
that is still being used by default (afaik) is probably one of the
biggest sources of frustration for both users and maintainers (it also
has a reputation for being the biggest source of race
conditions/deadlocks in all of Evolution - which seems to be your
primary reason for making Camel async ;-)

The sqlite backend stuff could also use some work. As far as I'm aware,
the tables are non-optimal. At one point I noticed that UIDs (the
primary key in the table) were stored as strings - it would be better to
store them as uint32s (yes, I know the gw and exchange backends do not
use uint32s for UIDs) for performance reasons. It should be possible for
these backends to have a second table which mapped the canonical uint32
UID key to the key used by the servers.

Additionally, the way Camel current works (even now, with the sqlite
backend), when a folder is opened, the entire summary for the folder is
loaded the same as it used to be before the sqlite changes. This seems
rather... wasteful? Kinda defeats the purpose of using sqlite (or any
other database backend). The main problem with the older format is that
it did not allow random-access, which means we really needed to load the
whole thing into memory for a number of reasons:

    1. each record being a different size requires sequential access
from disk
    2. can't sort by sender, subject, date (or whatever) until you have
the entire summary in memory
    3. doing lookups on message UIDs would be prohibitively slow from disk

With a db backend, none of these problems exist any longer. Some Camel
APIs would likely need to change in order to support taking advantage of
this new feature, but it's something that needs to be done.


These things seem like much bigger wins for Camel's (and, by extension,
Evolution's) usability/maintainability than making Camel async and are
also far more trivial to implement.

Just some things to think about...

Jeff