Re: [Evolution-hackers] Camel Manifesto

[Chenthill <pchenthill novell com>]

On Sat, 2009-11-21 at 14:57 +0530, Sankar P wrote:
> On Sat, Nov 21, 2009 at 12:51 AM, Jeffrey Stedfast <fejj novell com>
> wrote:
> > Matthew Barnes wrote:
> >> With work on Bonobo removal wrapping up, I've finally started
> taking a
> >> closer look at Camel (Evolution's mail storage and networking
> library)
> >> and laying out plans for where I'd like it to go over the short and
> long
> >> term, with the ultimate goal of splitting it off as a useful
> standalone
> >> GNOME library (but we're a long way from that).
> >>
> >> As usual I'm taking a bottom-up approach, starting with basic
> cleanup
> >> chores (both code and development policies) and building up from
> there.
> >> Here's some of my thoughts:
> >>
> >>
> >> Backward Compatibility Policy
> >> -----------------------------
> >>
> >> A reverse dependency search in Debian and Fedora reveals the only
> >> external projects currently linking to Camel are mail-notification,
> >> evolution-jescs and Anjal (please correct me if I've missed any).
> >>
> >> That tells me that until Camel moves out of its parents' basement
> and
> >> matures into an real, honest-to-goodness library, fixing its
> interface
> >> is more important than maintaining backward compatibility.
>  Deprecation
> >> periods for obsolete API are not necessary, in my opinion.  The
> few
> >> external projects linking to Camel will just have to keep up with
> the
> >> changes every six months.
> >>
> >
> > I think it's a worthy goal to separate Camel out of Evolution.
> >
> >> That's not a license to go hog wild though.  Some caveats:
> >>
> >> 1) The soname -must- be kept accurate.  If you break the API or
> ABI,
> >>    increment the soname when you commit the break.  It doesn't
> matter if
> >>    the break doesn't affect external projects, nor even if it's
> already
> >>    been incremented since the last point release.  Bump it
> anyway.
> >>    "Always bump" is an easy policy to remember.  It makes our
> own daily
> >>    development run smoother, and helps ensure a release doesn't
> slip out
> >>    with an inaccurate soname.
> >>
> >>    If you're not sure if your patch requires a soname increment,
> please
> >>    ask in IRC or Bugzilla.  Patch reviewers should try to
> remember too.
> >>
> >> 2) If you must break the API, try to do so in a way that things
> will
> >>    fail noisily at build time rather than mysteriously at run
> time.  For
> >>    example, if you want to change the behavior of an existing
> function,
> >>    it's better to rename the function or change its parameter
> list so
> >>    that stale Evolution code will fail to build.
> >>
> >> 3) Camel started life as a general purpose mail library and I'd
> like to
> >>    try to get back to that.  Camel has become too
> Evolution-centric in
> >>    my view, with too many quick-fix hacks for Evolution bugs that
> would
> >>    not be appropriate for a general purpose mail library.  I
> will clean
> >>    these up as I find them, but try to keep that in mind when
> altering
> >>    the API yourself.
> >>
> >
> > I'm not sure what Evolution-specific quick-fix hacks you mean?
> >
> >>
> >> Migrate to GObject
> >> ------------------
> >>
> >> Camel's homegrown type system will be replaced with GObject so that
> >> introspection and D-Bus + language bindings are possible.
>  CamelObject
> >> will remain (for now), but as a subclass of GObject.  The
> redundant
> >> parts of CamelObject will be removed.
> >>
> >
> > Worthy goals... there are some things like CamelObjectBag and such
> that
> > don't have an equivalent in GObject.
> >
> >> I'm also following GLib and GTK+'s example of sealing up public
> instance
> >> data in private sections and enforcing that only its top-level
> header
> >> file be included outside of Camel (including the providers).
>  Unlike
> >> GLib and GTK+, there will be no transition period.
> >>
> > Okay.
> >
> >> This will give us a lot more freedom to reorganize the library and
> >> refactor code without disturbing the ABI.  Debugging is also
> easier when
> >> you can trap data accesses through "get" and "set" functions.
> >>
> >
> > Sure, but I don't think any code actually goes behind any get/set
> > method's back at the moment (or at least didn't back when I worked
> on
> > Camel). Regardless, I'm cool with the proposed changes...
> >
> >> I've been chipping away at this as a side project for the past year
> (it
> >> was a good mindless activity when I got burned out on Bonobo
> removal),
> >> and I recently published my results to git.gnome.org as a branch
> named
> >> "camel-gobject".  The CamelObject conversion is finished --
> including
> >> all the boilerplate changes in the subclasses -- but I haven't
> finished
> >> sealing up the API.
> >>
> >> The branch probably won't land until 2.31 at the earliest.  The
> backward
> >> compatibility policies I described above would be in effect
> thereafter.
> >>
> >>
> >> Kill CamelStream
> >> ----------------
> >>
> >> This is a distant future goal and will have to happen gradually,
> but I
> >> would like Camel to shift to a single-threaded design where all
> file and
> >> network operations directly use or are derived from GIO's
> asynchronous
> >> file and stream APIs.  SSL support is currently under development
> for
> >> GIO, and that's the only missing piece I see at the platform layer.
> >>
> >> I realize this is a drastic course correction and will require
> rewriting
> >> all the providers and much of the mailer code in Evolution, but I
> firmly
> >> believe that the overuse of threads carries much of the blame for
> >> Evolution's chronic instability over the years and that reversing
> that
> >> trend first and foremost requires minimizing our use of threads for
> I/O
> >> and relying more heavily on GLib's main loop.
> >>
> >>
> >> Comments and constructive criticism encouraged.
> >>
> >
> > I agree with Michael Meeks' concerns here. I also think there are
> much
> > more important fish to fry which are also far easier to tackle.
> >
> > An IMAP rewrite (which supports IDLE) seems to be something a lot of
> > users would be very thankful for, for example. The ancient IMAP code
> > that is still being used by default (afaik) is probably one of the
> > biggest sources of frustration for both users and maintainers (it
> also
> > has a reputation for being the biggest source of race
> > conditions/deadlocks in all of Evolution - which seems to be your
> > primary reason for making Camel async ;-)
> >
> 
I have taken the imapx implementation by Notzed and currently porting to
the db summary. With this support like IDLE should be pretty simple.
Once the features are in parity with the default imap, we would make
imapx as the default. Am working on this for evolution 2.30.


> I absolutely agree. I am reminded of some of the code comments that
> used to exist like: "There are so many broken IMAP servers out there",
> "Hack to handle the broken Exchange 2005 IMAP access" etc.  Evolution
> now has far more providers to support and far less people to work on,
> than say two years ago.  So, I might actually suggest that we rely on
> something like "offlineimap" to do the networking part. Evolution can
> then just work on the maildir provided by the mail fetching daemon.
> since offlineimap supports signals, you can map "get-mail" events
> also. the only problem is that the service will be disconnected and
> all mails will be fetched.
> 

It is clear that we have less active developers at the moment than
before and the efforts have been carefully planned in selected areas.
offlineimap may not be a solution for users who do not want to
cache the mails. A typical use case for mobile based apps. So our target
is both desktop and mobile based apps.

Perhaps I had thought about in-corporating the gnio changes and making
the imap implementation alone a bit outside of camel of other apps also
to make use. But ssl support in gnio made me drop that plan. I am
strongly in favor of migrating from evolution specific libraries to glib
based ones. 

OTOH I am impressed with the amount of choices which linux provides,
with so many email clients who keeps competing and improving.

> Though everyone likes to write new code, I am sure it will cause a lot
> of maintenance problems with so many providers to support. Camel can
> just be mail library that runs as a service and provides a single
> point of api for other tools (Tracker/Beagle, OpenOffice, whatever).
> Internally it makes use of offlineimap, smtp, GW-MUA for whatever mail
> transport.
So a good news,  Travis Reitter (treitter),  Jonathon Jongsma (jonner)
have started the work on mailer-to-eds :-)

> 
> > The sqlite backend stuff could also use some work. As far as I'm
> aware,
> > the tables are non-optimal. At one point I noticed that UIDs (the
> > primary key in the table) were stored as strings - it would be
> better to
> > store them as uint32s (yes, I know the gw and exchange backends do
> not
> > use uint32s for UIDs) for performance reasons. It should be possible
> for
> > these backends to have a second table which mapped the canonical
> uint32
> > UID key to the key used by the servers.
> >
> 
> Yes. when Srini and I started with the sqlite summary work, we planned
> to implement the basic workflow first and then to improve the database
> in more effective ways.  But sadly none of us work anymore on
> Evolution directly and we weren't aware of these surprises when we
> implemented the initial version of sqlite summary.
> 
> One important area that could improve immensely is Search. We now do a
> "Like '<string>'" query on database often leading to slowed search
> results. We can create more indices and once the user starts typing in
> the text box, we can autocomplete adn do a EXACT Search in teh
> database often leading to faster results. For instance, if I search
> for Fejj now the query that  goes to db is LIKE  FEJJ. Whereas, if we
> can autocomplete during the search and show fejj novell com, then in
> the database it is faster to search with a EXACT match. this may not
> look like a big improvement in theory but I believe it will be a big
> usability improvement and can also of be use for other applications
> that will rely on us, when camel becomes a service etc. Sadly, I was
> never able to make the management understand the need for this when I
> was in the team and also we never had the time to complete the
> disk-summary migration and other related tasks etc.
> 
> > Additionally, the way Camel current works (even now, with the sqlite
> > backend), when a folder is opened, the entire summary for the folder
> is
> > loaded the same as it used to be before the sqlite changes. This
> seems
> > rather... wasteful? Kinda defeats the purpose of using sqlite (or
> any
> > other database backend). The main problem with the older format is
> that
> > it did not allow random-access, which means we really needed to load
> the
> > whole thing into memory for a number of reasons:
> >
> >    1. each record being a different size requires sequential
> access
> > from disk
> >    2. can't sort by sender, subject, date (or whatever) until you
> have
> > the entire summary in memory
> >    3. doing lookups on message UIDs would be prohibitively slow
> from disk
> >
> > With a db backend, none of these problems exist any longer. Some
> Camel
> > APIs would likely need to change in order to support taking
> advantage of
> > this new feature, but it's something that needs to be done.
> >
> 
> IIRC, For showing in message list, we already fetch only whatever
> records are needed and do not load the full summary onto the memory.
> For instance, if you have selected "Unread mail" in the QuickShow
> combo box, only unread mails' summary items are loaded.
> 
> >
> > These things seem like much bigger wins for Camel's (and, by
> extension,
> > Evolution's) usability/maintainability than making Camel async and
> are
> > also far more trivial to implement.
> >
> > Just some things to think about...
> >
> 
> I tend to agree with Fejj on this and I am big fan of threads. But I
> no longer work on evo and knowing Matthew  I believe he knows what is
> the best. All the best :-)
We are all part of evolution community :) So would be good not to say,
"am not longer working on evo". Sharing opinions/ideas is a big
contribution in it-self :) It helps people who contribute through code
to do the right things. just a humble thought.


- Chenthill.
>