Re: [Evolution] IMAP rewrite

[Timo Sirainen <tss iki fi>]

On Wed, 2003-01-29 at 18:45, Jeffrey Stedfast wrote:
> > - Currently it's just "write command", "wait for reply". It should
> > rather be waiting for IMAP replies all the time.
>
> afaik, IMAP cannot send untagged responses "whenever it wants to". It
> may only send them as part of a response to a command.
>
> if you can point me to the location in rfc2060 that says otherwise,
> please do so.

Hmm. I don't see it clearly mentioned, but there's in section 7: "The
client MUST be prepared to accept any response at all times." So that
means the server is allowed to send them and smart client would notice
the changes immediately. For example "* BYE" with disconnection isn't
now noticed at all with evolution until something is tried to be done.

Supporting IDLE-extension would be nice too, that would require just
waiting for replies.

> >  Server can send them at
> > any point, Evolution actually breaks currently if that's done before
> > APPEND.
>
> oh?

Hmm. I thought I had added it to bugzilla. Can't find it though.
Communication anyway goes like this:

S: * 10 EXISTS
C: APPEND sent-mail {10000}
S: + OK
C: ...

But Evolution sees the EXISTS only after it's sent the APPEND command,
so I think it expects "+ OK" but sees it instead and aborts. But the
server still expects the 10000 characters and the connection gets messed
up.

> >  I want to use my server's ability to notify of new mails
> > _immediately_ after they come to selected mailbox.
>
> I presume you are talking about that fam-like extension?

Something like that.. Though I only do stat()ing once every few seconds
now, but I was going to add support for Linux's dnotify and *bsd's
kqueue(). Every few seconds isn't bad either, much better than the every
few minutes wait now. Normally it isn't that bad, but when I know that a
mail should come any second now, it's annoying to keep clicking the
send/receive button.

> > - Send the flag changes to the server immediately.
>
> the problem here is that camel_folder_set_message_flags() doesn't take
> an exception argument (ie, it is not allowed to fail - thus it *cannot*
> talk to the server).

Does it matter? It can try to send the flags, if it fails keep the
dirty-flag on and do it later like now.

> 3) NOOP timeout (if there are flags to sync, it will sync them rather
> than doing a NOOP)

If it syncs only flags, that has the annoying effect of not noticing if
any mails are expunged (EXPUNGEs can't be sent to FETCH or STORE
commands).

> >  Also keep the flags synced with the server all
> > the time, if I change them elsewhere I expect Evolution to update them
> > internally as well.
>
> yes, this would be nice... 
>
> to do this right, what we'll need to do is store a copy of what the
> server's flags are for the message and a copy of what the current state
> of the flags are as displayed by evolution. when we are notified by the
> server that the flags for that message have changed, we assign that
> value to the store_flags and the display_flags *if and only if* the
> display_flags are the same as the store_flags (ie, they have not been
> changed by the user in this client).

Hmm. Would be better if it would change those flags that haven't been
touched in Evolution. Maybe don't remember server flags, but have
display_flags and dirty_flags? Then change display_flags unless the flag
is in the dirty_flags.

> >  Or is it just called whenever something
> > is clicked and it blocks while waiting for reply?
>
> pretty much this is the way it works, although it only "blocks" in
> another thread.

Are all connections handled in one thread? It at least seems so, since
if one connection gets stuck I can't do anything with others either.

> >  Could I just register
> > the connections fds into GLIB's main loop and make the code asynchronous
> > in general?
>
> no, camel MUST remain threadsafe and well, glib gio-channels aren't.

So each connection should be handled in it's own thread, if they can't
be made asynchronous?

> Now, my idea (and I think NotZed agreed?) was to take CamelImapEngine a
> step farther and have CamelImapDriver - a thin abstraction on top of
> CamelImapEngine which brokers CamelImapCommands to the best
> CamelImapEngine for the job. Basically, it would be responsible for
> multiple CamelImapEngines - this would allow us to have multiple
> connections tot he server so that we could parallelise some commands.

Multiple connection support would be nice, just remember that one
mailbox isn't necessarily possible to be selected in multiple
connections (eg. UW imapd). APPENDs and STATUSes could be done anyway in
separate connections.

Some other things that come to my mind:

Saving attachments is sometimes _slow_ for some reason, even if the
attachment is only a few kilobytes it can take almost 10 seconds to
save. Any ideas why? Larger attachments anyway will block the UI until
it's downloaded, and the save dialog is only then shown. Dialog should
really be shown first, then the downloading done in background.

Large fetches could be done in smaller (configurable) parts so in the
middle of the fetches some smaller commands could be done, or the larger
fetching possibly be aborted (if another message was selected, or the
message was deleted, no need to continue with it).

Even if Evolution noticed server's EXISTS replies immediately, that
wouldn't help with other folders. I've been thinking of how I could make
it check them all fast. One way that would work with all servers is to
open a separate IMAP connection for all folders and wait there. Another
way would be if server sent STATUS-replies whenever it noticed something
changed. Two ways to do that: configure the server to do that always, or
create a new IMAP extension: STATUS-WATCH (MESSAGES UNSEEN) INBOX
evolution list2 list3.