Re: Rescanning IMAP folders

[Dave Cridland <dave cridland net>]

On Tue Feb 13 15:30:16 2007, Philip Van Hoof wrote:
> On Tue, 2007-02-13 at 14:47 +0000, Dave Cridland wrote:
>
> > Good - you can get EXPUNGE only when commands that don't have > 
> sequence numbers in the arguments are running.
> > > EXISTS and FETCH can come at any time at all, even when no 
> commands > are running. (I think Oryx's server, Archiveopteryx, 
> does this, but > it's probably the only one.)
>
> I noticed that some places in the old Camel code deal with such 
> things
> by simply ... ignoring them or setting some old variables right if 
> they
> are at hand. *gihmrrl*
>
> (yes, insert a pause here, as I'm afraid of changing those Camel 
> pieces)
Yes, that doesn't sound too good.


> > From an IMAP perspective, these things are essentially 
> synchronized, > and the client can only break this if it uses 
> sequence numbers in a > command other than at the beginning of a 
> pipeline.
>
> Ok, afaik this ain't happening in camel's code.
No, because there's no pipelining.

> > Yes, that's right. Even with NOTIFY, you'll only get information 
> that > some messages have been expunged in other folders, not 
> which. (But a > server doing NOTIFY probably does QRESYNC, which 
> hands you a list of > messages which have been expunged when you 
> SELECT).
>
> Okay, NOTIFY is for some other time. Hah :). And most likely going 
> to be
> for when my hero fejj has finished his libspruce implementation.
You might as well wait until at least one server actually implements 
it, anyway. :-)


> I mean, I need to consolidate Camel a little bit too i think. It's
> design really isn't fit for the really cool new things. Although 
> indeed
> I can get quite a lot of it right and better ... step by step.
>
> For example .. I truly miss pipelining or, a more event-based 
> design.
> For events I need to non-blocking read the socket. All others are
> blocking reads.
>
> I'd prefer to always non-blocking read whatever I get in a poll 
> loop,
> and on detected responses process what I got ... and callback to the
> caller with the results. Something libspruce's IMAP code is more or 
> less
> doing.
The IPL does both - you can wait() for a command to complete, or you 
can get a callback when it does, and intermediate responses (stuff 
like FETCH during a FETCH command) are treated the same as truly 
unsolicited responses (like EXISTS during a FETCH command) - they 
just update the local cache.


> People have been telling me to launch a bunch of threads and open
> connections like a crazy person. But actually with pipelining, 
> poll()
> and some clever design ain't this necessary at all. 
> Right?
Indeed. The IPL can use a single additional thread, and Polymer does 
this, because it uses wxPython, and there's no interface into the 
networking areas of that. But Telomer, which is pyGTK, uses glib's 
I/O eventing. (PyS60 uses a thread, too. You set these things up by 
arcana, which is a bit of a problem.)


> > You can use multiple connections to monitor multiple mailboxes, 
> or > else you can just get really good at synchronization.
>
> Then I'd prefer to become good at synchronization. Note that mobile
> devices, which is tinymail's focus, "often" lose the connectivity 
> with
> the network. Nor is the connectivity something you can "assume" at 
> any
> time.
>
> My opinion is that being good at synchronization is always a good 
> idea
> then. Because the connection will be up and down .. constantly 
> changing.
Yes, I agree. It's also good because if you can sync fast, then 
accessing a rarely used, but huge, mailbox becomes practical. (I can, 
for example, read 100k message mailboxes with Polymer, and it just 
works. PyGTK makes it a bit painful on Telomer, but it still mostly 
works).


> Launching both a bunch of threads and opening a bunch of connections
> isn't going to make locking data more easy, ain't going to make 
> managing
> all those connection and dealing with errors more easy and since a
> mobile device is rather going to be user for one action at the time 
> (for
> example: the user will look at at most one folder) I really don't 
> see
> the big deal with multiple connections and multiple threads.
No. It's only useful if your UI can explicitly show more than one 
mailbox. Polymer doesn't, it shows all mailboxes in a Notebook, and 
whichever the last one shown on a server is the SELECTed one.


> I did consider it for IDLE though. But so-far I prefer the current
> solution over a second connection.
All the IPL's IDLE code is lines 1534 to 1628, and most of that is 
concerned with figuring out how to turn it off. The rest is concerned 
with syncing the cache files, etc. No new threads, nothing very 
complex at all.

> Is UIDNEXT something that I can be sure of? Or are there IMAP 
> servers
> that might get this wrong or incompatible with your technique?
Well, you'll see in the code that old versions of MDaemon don't give 
you UIDNEXT, Courier didn't last time I looked, and so on. Also, 
UIDNEXT needn't increase by one each time, it can increase by 2, or 
4, or 90 - it just has to increase. The code doesn't explicitly 
handle these cases, it just assumes that expunges might have happened.

> Thanks a lot Dave. If you have any other suggestion, please do 
> bother me
> with it :)

Throw away Camel and start over? :-)

Dave.
--
Dave Cridland - mailto:dave cridland net - xmpp:dwd jabber org
 - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
 - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade