Re: [Evolution] IMAP rewrite

[Not Zed <notzed ximian com>]

On Tue, 2003-01-28 at 16:13, Timo Sirainen wrote:
> I remember reading something of IMAP rewrite, how far did that get? Is
> there still a chance of getting that soon?

I wrote a ton of code, but didn't quite get to the implementing a folder
stage, close though.  It needed a few more tweaks to the design, and i
got bored with it.  It basically used a sort of state machine/operation
queue that could support multiple outstanding requests and handling
replies at any time.  Requests could be built on the fly (rather than
building huge strings in memory), and also read/processed on the fly via
a simple reentrant parser.  I was also going to implement a CamelStream
which would download big data in small chunks, etc.

I didn't quite get to working out the exact mechanics of reserving the
connection for multiple per-folder ops and a few other 'hairy' details
required to complete the implementation.

> I'm thinking of fixing the IMAP code myself soon, unless there's some
> larger changes coming out soon. Was there some plans to change the Camel
> API? I remember reading something about it.. Anyway, what I want fixed:
>
> - Currently it's just "write command", "wait for reply". It should
> rather be waiting for IMAP replies all the time. Server can send them at
> any point, Evolution actually breaks currently if that's done before
> APPEND. I want to use my server's ability to notify of new mails
> _immediately_ after they come to selected mailbox.

This can't really be done with the current codebase.  For one, it isn't
linked into the glib main loop (and can't be), and there isn't a
separate thread managing stuff like this all the time.  At best it could
take new replies whenever it tries something else, or enters the imap
i/o loop.

> - Send the flag changes to the server immediately. I think it now waits
> until closing the folder. Also keep the flags synced with the server all
> the time, if I change them elsewhere I expect Evolution to update them
> internally as well.

They are two different problems.  Evolution ignores unsolicited FETCH
responses, so flags chagned elsewhere aren't updated.  Updating flags
immediately will need more work than just "updating them immediately",
so that multiple flag changes (which are done one at a time via the api)
aren't done one at a time via the connection (== very slow).

Flags are synced when the folder is 'synced' (which is managed by the
mailer, generally when you change folders).

> - Labels should be sent to server using custom message flags, assuming
> the server supports them.

This is part of the mailer, not the imap code.  It has to work with all
backends too (which it should already anyway, including IMAP i think).

> - If IMAP connection hangs to one server, it shouldn't affect the rest
> of them. And in general there seems to be several annoying problems when
> one or all connections die, I just can't remember what exactly now. Oh,
> and send NOOP to server once in a while, if it can't get a reply soon,
> reconnect (to work around hanged TCP connections). If server kills the
> connection, silently reconnect instead of complaining.

This doesn't have anything to do with camel.  This is because certain
slow-running operations are put in a single-consumer queue.  If an
operation in that queue blocks forever, so does anything else on that
queue ...  Obviously something that needs addressing though.  The thread
sheduling all over needs some work.

The write's should have timeouts (via select(io write)), but i think
they're missing from various places.

> - All other kind of smaller things that don't come to my mind right now.
> As an author of IMAP server, I hate seeing almost all clients implement
> it wrong or badly. One of my main motivations to get the thing written
> was to be able to use Evolution.

> So .. Any comments on how easy it would be to fix the problems above? I
> haven't studied too deeply Evolution's code yet. The first thing worries
> me a bit as it might require larger changes.. How is the IMAP code run
> now? Is it in it's own thread? Or is it just called whenever something
> is clicked and it blocks while waiting for reply? Could I just register
> the connections fds into GLIB's main loop and make the code asynchronous
> in general?

As above.  It isn't in its own thread.  Its basically pretty crap,
pretty simple, request-wait-response code.  But you can't really link it
into glib main loops because it needs to be multithreaded.  It would
probably be easiest to do something like it is now, and maybe add a
thread or something that handles unsolicited incoming data.

Also, you just dont want to use the existing code at all.  Its a mess,
and few bits of the code are even re-usable in a different design.

What IMAP server are you using anyway?  The ones I test against are
pretty limited and dont really suffer a lot from evolution's poor IMAP
implementation (e.g. they dont do responses without requests, etc).

 Z