[evolution-patches] Initial download of a folder

[Philip Van Hoof <spam pvanhoof be>]

Hi there,

This patch is useless without the mmap one. Nevertheless it might be
interesting, for Evolution, as an experiment. It's not yet finished and
support for it in other providers is needed (which I will work on).

Sidenote for, for example, Brutus and other 3th party developers who
would like to get support for their service in tinymail (in case of
Brutus, I also want support for Exchange, so chances are high that I
will integrate it at some point anyway).

Each nth (in my experiment that's each 1000th) header, the method that
receives message-headers (and uses camel_folder_summary_add or an
equivalent) should issue a new API "camel_folder_summary_dump_mmap" if
it wants to support this hack.

The query also shouldn't ask "for all message headers" but rather "for
the next n message headers" and put the entire thing in a "while there
are still headers to fetch, loop". I haven't looked at the Brutus's GPL
Camel provider, so at this moment I don't know what it will do or if it
looks at all like the default imap one of Evolution.

The memory consumption of the default camel-imap-folder.c, 35M for
getting 30,000 headers, is obviously unacceptable for tinymail. Because
this basically means that such large folders can't be initially
downloaded to a typical mobile device (not even in steps, as a
cancellation means that no summary info is written in the default or
current implementation). Most such devices simply can't handle 35M. Not
even if you stand on your head and start dancing for it.

The attached patch makes it possible to both in steps and in one go do
an initial download of a large IMAP folder on a device with few memory
resources.


More information here

http://pvanhoof.be/blog/index.php/2006/10/22/initial-download-of-a-large-imap-folder-on-tinymail

and here http://tinymail.org/trac/tinymail/wiki/MemoryStats


ps. I'll ask again: if any Evolution developer feels that this type of
messages shouldn't not be posted here: tell me. If nobody does, it's my
opinion that it's related to Evolution and that maybe Evolution could
benefit from experiments like this one.

-- 
Philip Van Hoof, software developer
home: me at pvanhoof dot be
gnome: pvanhoof at gnome dot org
work: vanhoof at x-tend dot be
blog: http://pvanhoof.be/blog

Index: camel/providers/imap/camel-imap-folder.c
===================================================================
--- camel/providers/imap/camel-imap-folder.c	(revision 1046)
+++ camel/providers/imap/camel-imap-folder.c	(working copy)
@@ -2348,7 +2348,8 @@
 	CamelStream *stream;
 	char *uid, *resp;
 	GData *data;
-	
+	gboolean more = TRUE;
+
 	if (store->server_level >= IMAP_LEVEL_IMAP4REV1)
 		header_spec = "HEADER.FIELDS (" CAMEL_MESSAGE_INFO_HEADERS MAILING_LIST_HEADERS ")";
 	else
@@ -2360,7 +2361,12 @@
 	
 	if( g_getenv ("EVO_IMAP_FETCH_ALL_HEADERS") )
 		header_spec = "HEADER";
-	
+
+while (more)
+{
+	gint count = 0;
+	more = FALSE;
+
 	/* Figure out if any of the new messages are already cached (which
 	 * may be the case if we're re-syncing after disconnected operation).
 	 * If so, get their UIDs, FLAGS, and SIZEs. If not, get all that
@@ -2374,25 +2380,34 @@
 		camel_message_info_free(&mi->info);
 	} else
 		uidval = 0;
-	
+
 	size = (exists - seq) * (IMAP_PRETEND_SIZEOF_FLAGS + IMAP_PRETEND_SIZEOF_SIZE + IMAP_PRETEND_SIZEOF_HEADERS);
 	got = 0;
 	if (!camel_imap_command_start (store, folder, ex,
-				       "UID FETCH %d:* (FLAGS RFC822.SIZE INTERNALDATE BODY.PEEK[%s])",
-				       uidval + 1, header_spec))
+				       "UID FETCH %d:%d (FLAGS RFC822.SIZE INTERNALDATE BODY.PEEK[%s])",
+				       uidval + 1, uidval + 1 + 1000, header_spec))
 		return;
+		
+
+	printf ("Getting more %d:%d\n", uidval + 1, uidval + 1 + 1000);
+
 	camel_operation_start (NULL, _("Fetching summary information for new messages in %s"), folder->name);
 	
 	/* Parse the responses. We can't add a message to the summary
 	 * until we've gotten its headers, and there's no guarantee
 	 * the server will send the responses in a useful order...
 	 */
+
 	fetch_data = g_ptr_array_new ();
 	messages = g_ptr_array_new ();
+
 	while ((type = camel_imap_command_response (store, &resp, ex)) ==
-	       CAMEL_IMAP_RESPONSE_UNTAGGED) {
+	       CAMEL_IMAP_RESPONSE_UNTAGGED) 
+	{
+		more = TRUE;
 		data = parse_fetch_response (imap_folder, resp);
 		g_free (resp);
+
 		if (!data)
 			continue;
 		
@@ -2401,7 +2416,7 @@
 			g_datalist_clear (&data);
 			continue;
 		}
-		
+
 		if (g_datalist_get_data (&data, "FLAGS"))
 			got += IMAP_PRETEND_SIZEOF_FLAGS;
 		if (g_datalist_get_data (&data, "RFC822.SIZE"))
@@ -2416,43 +2431,46 @@
 			add_message_from_data (folder, messages, first, data);
 			g_datalist_set_data (&data, "BODY_PART_STREAM", NULL);
 		}
-		
-		camel_operation_progress (NULL, got * 100 / size);
+
+		if (size > 0)
+			camel_operation_progress (NULL, got * 100 / size);
 		g_ptr_array_add (fetch_data, data);
 	}
 	camel_operation_end (NULL);
-	
+
 	if (type == CAMEL_IMAP_RESPONSE_ERROR)
 		goto lose;
-	
+
 	/* Free the final tagged response */
 	g_free (resp);
-	
+
 	/* Figure out which headers we still need to fetch. */
 	needheaders = g_ptr_array_new ();
 	size = got = 0;
-	for (i = 0; i < fetch_data->len; i++) {
+
+	for (i = 0; i < fetch_data->len; i++) 
+	{
 		data = fetch_data->pdata[i];
 		if (g_datalist_get_data (&data, "BODY_PART_LEN"))
 			continue;
-		
+
 		uid = g_datalist_get_data (&data, "UID");
 		if (uid) {
 			g_ptr_array_add (needheaders, uid);
 			size += IMAP_PRETEND_SIZEOF_HEADERS;
 		}
 	}
-	
+
 	/* And fetch them */
-	if (needheaders->len) {
+	if (needheaders->len) 
+	{
 		char *uidset;
 		int uid = 0;
-		
+
 		qsort (needheaders->pdata, needheaders->len,
 		       sizeof (void *), uid_compar);
-		
+
 		camel_operation_start (NULL, _("Fetching summary information for new messages in %s"), folder->name);
-		
 		while (uid < needheaders->len) {
 			uidset = imap_uid_array_to_set (folder->summary, needheaders, uid, UID_SET_LIMIT, &uid);
 			if (!camel_imap_command_start (store, folder, ex,
@@ -2464,7 +2482,7 @@
 				goto lose;
 			}
 			g_free (uidset);
-			
+
 			while ((type = camel_imap_command_response (store, &resp, ex))
 			       == CAMEL_IMAP_RESPONSE_UNTAGGED) {
 				data = parse_fetch_response (imap_folder, resp);
@@ -2476,7 +2494,8 @@
 				if (stream) {
 					add_message_from_data (folder, messages, first, data);
 					got += IMAP_PRETEND_SIZEOF_HEADERS;
-					camel_operation_progress (NULL, got * 100 / size);
+					if (size > 0)
+						camel_operation_progress (NULL, got * 100 / size);
 				}
 				g_datalist_clear (&data);
 			}
@@ -2491,7 +2510,7 @@
 		g_ptr_array_free (needheaders, TRUE);
 		camel_operation_end (NULL);
 	}
-	
+
 	/* Now finish up summary entries (fix UIDs, set flags and size) */
 	for (i = 0; i < fetch_data->len; i++) {
 		data = fetch_data->pdata[i];
@@ -2501,7 +2520,7 @@
 			g_datalist_clear (&data);
 			continue;
 		}
-		
+
 		mi = messages->pdata[seq - first];
 		if (mi == NULL) {
 			CamelMessageInfo *pmi = NULL;
@@ -2535,9 +2554,10 @@
 		
 		uid = g_datalist_get_data (&data, "UID");
 
-		/* This strdup() might not get freed since the mmap() patch! */
-		if (uid)
+		if (uid) {
 			mi->info.uid = g_strdup (uid);
+			mi->info.uid_needs_free = TRUE;
+		}
 
 		flags = GPOINTER_TO_INT (g_datalist_get_data (&data, "FLAGS"));
 		if (flags) {
@@ -2548,26 +2568,29 @@
 			mi->info.flags |= flags;
 			flags_to_label(folder, mi);
 		}
-		
+
 #ifdef NON_TINYMAIL_FEATURES
 		size = GPOINTER_TO_INT (g_datalist_get_data (&data, "RFC822.SIZE"));
 		if (size)
 			mi->info.size = size;
 #endif
+
 		g_datalist_clear (&data);
 	}
 	g_ptr_array_free (fetch_data, TRUE);
-	
+
 	/* And add the entries to the summary, etc. */
 	for (i = 0; i < messages->len; i++) {
 		mi = messages->pdata[i];
+
 		if (!mi) {
-			g_warning ("No information for message %d", i + first);
+			/* g_warning ("No information for message %d", i + first);
 			camel_exception_setv (ex, CAMEL_EXCEPTION_SYSTEM,
 					      _("Incomplete server response: no information provided for message %d"),
-					      i + first);
-			break;
+					      i + first); */
+			continue;
 		}
+
 		uid = (char *)camel_message_info_uid(mi);
 		if (uid[0] == 0) {
 			g_warning("Server provided no uid: message %d", i + first);
@@ -2599,15 +2622,17 @@
 			camel_folder_change_info_recent_uid(changes, camel_message_info_uid (mi));
 	}
 
+	camel_folder_summary_dump_mmap (folder->summary);
+
 	for ( ; i < messages->len; i++) {
 		if ((mi = messages->pdata[i]))
 			camel_message_info_free(&mi->info);
 	}
-	
 	g_ptr_array_free (messages, TRUE);
-	
-	return;
-	
+
+
+	goto endbmore;
+
  lose:
 	if (fetch_data) {
 		for (i = 0; i < fetch_data->len; i++) {
@@ -2623,6 +2648,12 @@
 		}
 		g_ptr_array_free (messages, TRUE);
 	}
+
+ endbmore:
+ count = 0;
+
+ } /* more */
+
 }
 
 /* Called with the store's connect_lock locked */
Index: camel/providers/Makefile.am
===================================================================
--- camel/providers/Makefile.am	(revision 1046)
+++ camel/providers/Makefile.am	(working copy)
@@ -1,18 +1,20 @@
 ## Process this file with automake to produce Makefile.in
 
+
+SUBDIRS = pop3 smtp imap local  
+
 if ENABLE_NNTP
-NNTP_DIR=nntp
+SUBDIRS += nntp 
 endif
 
 if ENABLE_IMAPP
-IMAPP_DIR=imapp
+SUBDIRS += imapp 
 endif
 
+
 if OS_WIN32
 else
-SENDMAIL_DIR=sendmail
+SUBDIRS += sendmail 
 endif
 
-SUBDIRS = pop3 $(SENDMAIL_DIR) smtp imap $(NNTP_DIR) local $(IMAPP_DIR) 
 
-
Index: camel/camel-folder.c
===================================================================
--- camel/camel-folder.c	(revision 1046)
+++ camel/camel-folder.c	(working copy)
@@ -2116,7 +2116,7 @@
 	struct _CamelFolderChangeInfoPrivate *p;
 	GPtrArray *olduids;
 	char *olduid;
-	
+
 	g_assert(info != NULL);
 	
 	p = info->priv;
Index: camel/camel-folder-summary.c
===================================================================
--- camel/camel-folder-summary.c	(revision 1046)
+++ camel/camel-folder-summary.c	(working copy)
@@ -113,6 +113,10 @@
 static void camel_folder_summary_init       (CamelFolderSummary *obj);
 static void camel_folder_summary_finalize   (CamelObject *obj);
 
+
+static void camel_folder_summary_mmap_add(CamelFolderSummary *s, CamelMessageInfo *info);
+static void camel_folder_summary_unload_mmap (CamelFolderSummary *s);
+
 static CamelObjectClass *camel_folder_summary_parent;
 
 static void
@@ -161,6 +165,26 @@
 	g_slice_free1 ((gint)user_data, data);
 }
 
+static gboolean always_true (gpointer key, gpointer value, gpointer gp)
+{
+	return TRUE;
+}
+
+static void 
+camel_folder_summary_unload_mmap (CamelFolderSummary *s)
+{
+	camel_folder_summary_clear(s);
+	if (s->file)
+		g_mapped_file_free (s->file);
+	s->file = NULL;
+
+	g_ptr_array_foreach (s->messages, foreach_msginfo, (gpointer)s->message_info_size);
+	if (s->messages->len > 0)
+		g_ptr_array_remove_range (s->messages, 0, s->messages->len-1);
+	g_hash_table_foreach_remove (s->messages_uid, always_true, NULL);
+
+}
+
 static void
 camel_folder_summary_finalize (CamelObject *obj)
 {
@@ -177,6 +201,7 @@
 	g_ptr_array_foreach (s->messages, foreach_msginfo, (gpointer)s->message_info_size);
 	g_ptr_array_free(s->messages, TRUE);
 	g_hash_table_destroy(s->messages_uid);
+	s->messages_uid = NULL;
 
 	g_hash_table_foreach(p->filter_charset, free_o_name, 0);
 	g_hash_table_destroy(p->filter_charset);
@@ -591,7 +616,7 @@
 			}
 		}
 
-		camel_folder_summary_add(s, mi);
+		camel_folder_summary_mmap_add(s, mi);
 	}
 
 
@@ -807,6 +832,17 @@
 }
 
 
+
+void 
+camel_folder_summary_dump_mmap (CamelFolderSummary *s)
+{
+	camel_folder_summary_save (s);
+	camel_folder_summary_unload_mmap (s);
+	camel_folder_summary_load(s);
+
+	return;
+}
+
 /**
  * camel_folder_summary_add:
  * @summary: a #CamelFolderSummary object
@@ -845,7 +881,25 @@
 	CAMEL_SUMMARY_UNLOCK(s, summary_lock);
 }
 
+static void
+camel_folder_summary_mmap_add(CamelFolderSummary *s, CamelMessageInfo *info)
+{
+	CAMEL_SUMMARY_LOCK(s, summary_lock);
 
+/* unnecessary for pooled vectors */
+#ifdef DOESTRV
+	/* this is vitally important, and also if this is ever modified, then
+	   the hash table needs to be resynced */
+	info->strings = e_strv_pack(info->strings);
+#endif
+
+	g_ptr_array_add(s->messages, info);
+	g_hash_table_insert(s->messages_uid, (char *)camel_message_info_uid(info), info);
+	s->flags |= CAMEL_SUMMARY_DIRTY;
+
+	CAMEL_SUMMARY_UNLOCK(s, summary_lock);
+}
+
 /**
  * camel_folder_summary_add_from_header:
  * @summary: a #CamelFolderSummary object
@@ -1090,9 +1144,9 @@
 void
 camel_folder_summary_touch(CamelFolderSummary *s)
 {
-	CAMEL_SUMMARY_LOCK(s, summary_lock);
+	//CAMEL_SUMMARY_LOCK(s, summary_lock);
 	s->flags |= CAMEL_SUMMARY_DIRTY;
-	CAMEL_SUMMARY_UNLOCK(s, summary_lock);
+	//CAMEL_SUMMARY_UNLOCK(s, summary_lock);
 }
 
 
Index: camel/camel-folder-summary.h
===================================================================
--- camel/camel-folder-summary.h	(revision 1046)
+++ camel/camel-folder-summary.h	(working copy)
@@ -308,6 +308,8 @@
 CamelType			 camel_folder_summary_get_type	(void);
 CamelFolderSummary      *camel_folder_summary_new	(struct _CamelFolder *folder);
 
+void camel_folder_summary_dump_mmap (CamelFolderSummary *s);
+
 unsigned char*
 decode_uint32 (unsigned char *start, guint32 *dest, gboolean is_string);