Re: [Evolution-hackers] Reviewing imap_update_summary

[Philip Van Hoof <spam pvanhoof be>]

On Mon, 2006-10-23 at 04:02 -0600, Sankar P wrote:

Hey Sankar,

Thanks for the response.

> I rewrote this behavior in camel-GroupWise to fetch data in iterations,
> so that the memory requirement remains a constant k instead of O(n), n
> being the number of messages. I expected it work better. (GW and IMAP
> code are similar in this aspect)
> 
> However, when I tested it, as expected, the memory requirement came down
> but the number of disk-access increased and hence it became slow. So I
> reverted it to the old behavior. 

I acknowledge that the time increases. For me not significant but it
does indeed take longer. Doing nothing is of course faster than dumping
something to disk.

> You can try rewriting this in IMAP and you will find out that the time
> taken to complete the sync will increase in case of large folders.

This is correct and the case. On a mobile device, however, the time it
takes is (imo) less important than the possibility to do it. Maybe I
shouldn't use the "1, 2, 4, 8, etc" serie but do something like "1, 10,
100, 1000" to decrease the amount of to-disk dumps? What do you think?

At this moment it's the provider-code itself that determines this nth.

When talking about 'design', it might indeed be better if the camel-
folder-summary.c implements that decision. For now, the experiment
itself is implemented as a hack (I agree).

I would be interested in trying to improve it in a design-sense if
there's interest (and discussion) from the Evolution team in something
like this.

Being a hack, once supported by all providers, I am probably going to
use it in tinymail. I would of course prefer that Camel itself would
someday enjoy the same improvements too.

But it's not something I require nor will push for. I "touch evolution",
if you want it..tell me :)

> I tested the changes that I made (in camel-GW) and found that in actual
> deployment scenario, people prefer having an occasional memory-shootup
> (which will come down as soon as mail-fetch is complete) rather than
> having so many disk-access, that will eventually make operations longer
> to complete.

For a desktop user I can imagine that, yes.

For a mobile device, the memory shootup means that it's impossible to
support such large folders and that during the download, the device
becomes unusable for almost-large folders (that I definitely do want to
support) sized around 20,000 hdrs.

This is perhaps why a different design would be a good idea?: delegate
the decision to a more specific plugin or piece of code that is global
for all providers (rather than a patch-per-provider for each target).
 
> If you want the memory usage to be a constant value, the best solution
> is to make the folder_sync function fetch summaries iteratively from a
> database back-end (not from flat-files or mmap). Perhaps this should be
> done at a higher (camel) level so that all the providers can make use of
> it; Means rewriting parts of the camel folder, summary etc.

I agree

> Anyway, all this is already covered under "On-Disk-Summaries". If you
> are so obsessed about memory usage, go ahead and give it a try. 

I should. But I need something that works today ;).

I have already, however, repeatedly stated that I'm very interested in
the on-disk-summary concepts and that if a team would start with this
work, I'd definitely join that team.

I'm not naive to think that I could do it on my own. I ACK that
implementing the concepts takes a team of smart people. I think that
libspruce and GMime would be a nice (fresh) starting point for such an
implementation.

If it would happen, tinymail would probably become one of the first
E-mail framework(/client) to actively use the disk-summary concepts and
ideas.

Absolutely :)

> Some times, the customer's needs are different from what the hacker may
> perceive to be the most important thing. Evolution's periodic memory
> shootup (and subsequent coming down) is considered to be normal by the
> customers than things like Proxy-authentication-failure (libsoup), EDS
> crashes etc, that we have been working on.

> It is an interesting work but we (the Evolution team) have got other
> priorities driven based on actual customer deployment needs. These are
> the most important things that Evolution (and indirectly Camel also)
> should address to become a stable enterprise-level GNOME app. 

No problem. Yet I hope my experiments might someday be useful for the
Evolution project. Until then, I'll use tinymail as host for them.


-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be
gnome: pvanhoof at gnome dot org
work: vanhoof at x-tend dot be
blog: http://pvanhoof.be/blog