Rescanning IMAP folders

[Philip Van Hoof <spam pvanhoof be>]

Rescanning an IMAP folder has two purposes:

-> Updating the flags
-> Removing the expunged messages from both the cache and the summary

The second part is implemented badly. That's because IMAP has two things
to identify one message:

Its sequence number, and its UID.

The sequence number MUST be the index of the array of the summary. The
UID is a field like any other per item.

An expunge will happen like this: "* 2 EXPUNGE". This means that the
message with sequence 2 in the current folder has been expunged.

What will happen with the Push-Email implementation (the IMAP IDLE) is
that simply message #2 will be removed from the summary.

But if there was no IMAP IDLE for that folder, the only way to
synchronize both is to check one by one whether the sequence (the index
of the array + 1, as the IMAP sequence starts at 1 and C arrays start at
0) and the UID match.

The current solution is to simply remove any non-match and then to
request them from the IMAP server again. This consumes bandwidth.

And that consumption is not always necessary.

Imagine an expunge of message #1. What will happen:

SEQUENCE: 1 2 3 4 5
INDEX:    0 1 2 3 4
UID:      1 2 4 5 6

Server expunged #1

SEQUENCE: 1 2 3 4
INDEX:    0 1 2 3 4
UID:      1 2 4 5 6

Synchronization (imap_rescan):

SEQUENCE: 1 2 3 4
INDEX:    
UID:

Refetch (imap_update_summary):

SEQUENCE: 1 2 3 4
INDEX:    0 1 2 3
UID:      2 4 5 6


This happened because if "(1, 0, 1)" gets removed (SEQUENCE, INDEX, UID)
at the remote server, then of course is (2, 1, "2") not the right
combination anymore. It had to be (2, 1, "4"). So what will happen is
that that one will get removed by imap_rescan. 

Right, repeat this for index 3, 4 and 5 and you see that all will be
removed and all will have to be refetched in imap_update_summary.

But was this really necessary? Not really, we can re-calculate the local
summary to avoid the refetching of all in imap_update_summary, right?

( This is a real question Dave ;-) )

This isn't easy though.

Again I can use assistance. The code is in camel-imap-folder.c at
imap_rescan.

If any expunge was detected, the condstore implementation isn't used.
The condstore implementation doesn't cope with expunges. So it ain't
used for those situations. Which is why imap_rescan is ALWAYS important
enough to be *very* optimized.

The detection of whether expunges happened checks the last local UID
with "UIDNEXT - 1" and compares the EXISTS with the local size. Only if
CONDSTORE is activated, "LAST LOCAL UID == REMOTE UIDNEXT - 1" and
"LOCAL SIZE == EXISTS" do I assume that condstore can be used. Else
imap_rescan is used. Is that a correct assumption?

(Also a real question)

Note that I do process "* seq EXPUNGE" in IDLE (they'll simply get
removed from the local summary by using the sequence-1 as array index,
and this seems to work correctly).


I added the author of Polymer/Telomer in CC (Dave). That's because I
will most likely take a look at his source code for this too.