Re: Modest/TinyMail problems (continue from the blog comments)

[Philip Van Hoof <spam pvanhoof be>]

Hi there,

Problem is that for IMAP you want consistency with the UID and the
sequence number to check whether or not a synchronization is needed.

So you do actually need to build a storage of UIDs versus sequence
numbers (that's the least you need, to ensure that you synchronize
correctly). 

At least ... that's for an IMAP server that has neither CONDSTORE nor
QRSYNC (like most IMAP servers in the field).

What we could do, however, is receive just the sequence number and the
UID, not the rest of the envelope headers, for all items later than n.

The other problem however is that "what is later"? IMAP servers are
'recommended' but I don't think that they are required to have a sorting
order on "date received".

Most IMAP servers only increment the UID for their NEXTUID, but I don't
think this is a requirement (not to fill up holes). 

Fair enough, I already hear you think, so we'll just use SORT and SEARCH
then? That's another problem, IMAP servers are not required to implement
either the SORT nor the SEARCH capability.

So we end up with:

1) Just download the UID, not the rest of the envelope headers, but then
we are not really certain that when we ask "the bottom last 200 items"
that those are the 200 last received ones.

2) To solve that we can SORT/SEARCH, but we might not have that
capability at all.

3) If we can't solve the request, we could fall back to just downloading
all of the envelopes, like what is the only supported current behavior
of Tinymail right now.


Sounds good for me ... I await your patches (for Tinymail) :)


There are also other solutions for handling really large folders of more
than 15,000 items. The problem is not so much network bandwidth of the
envelopes. The problem is rather that each 5000th item the complete
summary storage file is rewritten on a relatively slow flash storage.

That's a file of by that time 12 MB. If any application during that
write causes a fsync() call, that'll indeed hang your device for several
minutes.

A solution for that is to either append instead of rewrite the file, or
rewrite the summary storage solution to use multiple smaller mapped
files. We have an experimental summary storage like that in our code,
but it's currently disabled because we didn't want to release untested
code in the device's Modest instance.

I can point to it and help you out with testing the implementation if
you join the Tinymail mailing list ... because going in detail is going
to be offtopic on this mailing list.

http://mail.gnome.org/mailman/listinfo/tinymail-devel-list


On Tue, 2008-06-24 at 23:24 +0200, Andrea Grandi wrote:
> Hi all,
> 
> I continue here a discussion you can find in this post:
> http://pvanhoof.be/blog/index.php/2008/06/24/big-day-for-modest-and-tinymail
> (Just read the comments).
> 
> This is my idea: I don't want to be able to download 15.000+ messages.
> I just want to download ONLY last 50, 100, 200 at maximum. Modest
> doesn't behave in this way now. Even if you set the option "just
> download ONLY 50 items". Maybe a bug? Don't know...
> 
> I think the minimum number of messages should be the default value,
> then it should let the user keep a maximum of x messages (where x is a
> value that user can choose). When 200th message arrives, just drop the
> first one and so on... (just like a FIFO queue with a maximum number
> of items). Sounds good my idea? I think that Modest could manage 200
> messages quite good.
> 
> Please let me know what do you think about it.
> 
> Best regards,

-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be