Re: Rescanning IMAP folders

[Philip Van Hoof <spam pvanhoof be>]

On Tue, 2007-02-13 at 14:47 +0000, Dave Cridland wrote:

> Good - you can get EXPUNGE only when commands that don't have 
> sequence numbers in the arguments are running.
> 
> EXISTS and FETCH can come at any time at all, even when no commands 
> are running. (I think Oryx's server, Archiveopteryx, does this, but 
> it's probably the only one.)

I noticed that some places in the old Camel code deal with such things
by simply ... ignoring them or setting some old variables right if they
are at hand. *gihmrrl*

(yes, insert a pause here, as I'm afraid of changing those Camel pieces)

> From an IMAP perspective, these things are essentially synchronized, 
> and the client can only break this if it uses sequence numbers in a 
> command other than at the beginning of a pipeline.

Ok, afaik this ain't happening in camel's code.

> So if a message is EXPUNGEd, and you give the server no opportunity 
> to tell you (ie, you don't use IDLE and you don't issue a command), 
> then you can still refer to that message with its sequence number. 
> (You'll quite possibly find an empty message, but you can change its 
> flags quite happily).

Okay.

 
> Yes, that's right. Even with NOTIFY, you'll only get information that 
> some messages have been expunged in other folders, not which. (But a 
> server doing NOTIFY probably does QRESYNC, which hands you a list of 
> messages which have been expunged when you SELECT).

Okay, NOTIFY is for some other time. Hah :). And most likely going to be
for when my hero fejj has finished his libspruce implementation.

I mean, I need to consolidate Camel a little bit too i think. It's
design really isn't fit for the really cool new things. Although indeed
I can get quite a lot of it right and better ... step by step.

For example .. I truly miss pipelining or, a more event-based design.
For events I need to non-blocking read the socket. All others are
blocking reads.

I'd prefer to always non-blocking read whatever I get in a poll loop,
and on detected responses process what I got ... and callback to the
caller with the results. Something libspruce's IMAP code is more or less
doing.

People have been telling me to launch a bunch of threads and open
connections like a crazy person. But actually with pipelining, poll()
and some clever design ain't this necessary at all. 

Right?

> You can use multiple connections to monitor multiple mailboxes, or 
> else you can just get really good at synchronization.

Then I'd prefer to become good at synchronization. Note that mobile
devices, which is tinymail's focus, "often" lose the connectivity with
the network. Nor is the connectivity something you can "assume" at any
time.

My opinion is that being good at synchronization is always a good idea
then. Because the connection will be up and down .. constantly changing.

Launching both a bunch of threads and opening a bunch of connections
isn't going to make locking data more easy, ain't going to make managing
all those connection and dealing with errors more easy and since a
mobile device is rather going to be user for one action at the time (for
example: the user will look at at most one folder) I really don't see
the big deal with multiple connections and multiple threads.

I did consider it for IDLE though. But so-far I prefer the current
solution over a second connection.

 
> This is post-SELECT synchronization, so deals with the last two 
> questions. IDLE has nothing at all to do with this.

Aha, in that case I'll in-depth read your code and explanation.

> > > If (A - D + C) == (B - E), then we know that no messages have 
> > been > expunged. We'll make this condition G.
> > > > So, if F is true, then we'll move onto looking at 
> > HIGHESTMODSEQ, and > if we've seen those changes, we can avoid 
> > doing a FETCH at all. Neat, > eh?
> > 
> > Yes, neat. But I wont get EXPUNGEs for just any folder... so what 
> > about
> > C for those? :-)
> > 
> > 
> There's another value here - the unwitnessed expunges. If F or G is 
> true, none have happened.

Is UIDNEXT something that I can be sure of? Or are there IMAP servers
that might get this wrong or incompatible with your technique?

Sorry for the massive cutting, just did that to keep things readable. I
read all your stuff of course and I will be rereading it several times
for the next weeks too :-)


Thanks a lot Dave. If you have any other suggestion, please do bother me
with it :)