GNOME.org
 

[Evolution-hackers] [PATCH] offline improvements/features

Hi guys,

I've come up with a patch which changes the way imap offline works.

Basically it does two things:
 - instead of syncing only read messages, it always syncs all messages
when you go offline
 - adds a server option ("receiving options") to enable automatic,
background downloading of new messages as they arrive.

It's still a little bit 'work in progress', and I think most of the
logic should probably lie in the back-end instead, but it should be
fully functional.

Some issues:
 - clicking on an unread message wont force a check for new messages
anymore, as the message is already local and no chance to get updates
from the server (if you're using the automatic option).
 - does the sync code conflict/interact badly with filtering?
 - does the sync code interact badly with reading mail?

I'm cross-posting this to evolution@ in case people are building from
source, and aren't on evolution-hackers@, but might want to try this
patch.  I probably wont do that again, so join -hackers@ if you want to
get involved.

It's against the (roughly) 1.4.3 source tree.

 Michael


Index: camel/ChangeLog
===================================================================
RCS file: /cvs/gnome/evolution/camel/ChangeLog,v
retrieving revision 1.1836.2.6
diff -u -3 -r1.1836.2.6 ChangeLog
--- camel/ChangeLog	29 Jul 2003 13:53:28 -0000	1.1836.2.6
+++ camel/ChangeLog	31 Jul 2003 19:13:53 -0000
@@ -1,3 +1,9 @@
+2003-07-30  Not Zed  <NotZed Ximian com>
+
+	* providers/imap/camel-imap-provider.c: Added offline_sync, to
+	automatically sync offline folders.  Currently used by the mailer,
+	but will move here eventually.
+
 2003-07-25  Jeffrey Stedfast  <fejj ximian com>
 
 	* camel-mime-utils.c (header_decode_word): Revert NotZed's fix for
Index: camel/providers/imap/camel-imap-provider.c
===================================================================
RCS file: /cvs/gnome/evolution/camel/providers/imap/camel-imap-provider.c,v
retrieving revision 1.24
diff -u -3 -r1.24 camel-imap-provider.c
--- camel/providers/imap/camel-imap-provider.c	22 May 2002 20:17:48 -0000	1.24
+++ camel/providers/imap/camel-imap-provider.c	31 Jul 2003 19:13:53 -0000
@@ -55,6 +55,8 @@
 	{ CAMEL_PROVIDER_CONF_SECTION_END },
 	{ CAMEL_PROVIDER_CONF_CHECKBOX, "filter", NULL,
 	  N_("Apply filters to new messages in INBOX on this server"), "0" },
+	{ CAMEL_PROVIDER_CONF_CHECKBOX, "offline_sync", NULL,
+	  N_("Automatically synchronize offline mailboxes to local storage"), "0" },
 	{ CAMEL_PROVIDER_CONF_END }
 };
 
Index: mail/ChangeLog
===================================================================
RCS file: /cvs/gnome/evolution/mail/ChangeLog,v
retrieving revision 1.2761.2.7
diff -u -3 -r1.2761.2.7 ChangeLog
--- mail/ChangeLog	28 Jul 2003 21:01:33 -0000	1.2761.2.7
+++ mail/ChangeLog	31 Jul 2003 19:13:54 -0000
@@ -1,3 +1,17 @@
+2003-07-30  Not Zed  <NotZed Ximian com>
+
+	* mail-folder-cache.c: Added handler for automatically
+	synchronising offline folders in the background.
+
+	* mail-folder-cache.c (setup_folder): tell the offline code a new
+	folder is available.
+	(mail_note_store_remove): tell offline code the store has gone.
+	(mail_note_get_store_by_account): function to find a camelstore
+	based on an evolution account name.
+
+	* mail-ops.c (prep_offline_do): change to sync all messages, not
+	just unread ones, by default.
+
 2003-07-25  Jeffrey Stedfast  <fejj ximian com>
 
 	* message-browser.c (on_key_press): New callback function to
Index: mail/component-factory.c
===================================================================
RCS file: /cvs/gnome/evolution/mail/component-factory.c,v
retrieving revision 1.328.4.1
diff -u -3 -r1.328.4.1 component-factory.c
--- mail/component-factory.c	15 Jul 2003 19:47:57 -0000	1.328.4.1
+++ mail/component-factory.c	31 Jul 2003 19:13:54 -0000
@@ -782,7 +782,9 @@
 	
 	for (i = 0; i < sizeof (standard_folders) / sizeof (standard_folders[0]); i++)
 		*standard_folders[i].uri = g_strdup_printf ("file://%s/local/%s", evolution_dir, standard_folders[i].name);
-	
+
+	em_offline_init();
+
 	vfolder_load_storage(corba_shell);
 	
 	accounts = mail_config_get_accounts ();
@@ -815,7 +817,7 @@
 		
 		rule_context_load (search_context, system, user);
 	}
-	
+
 	if (mail_config_is_corrupt ()) {
 		GtkWidget *dialog;
 		
Index: mail/mail-folder-cache.c
===================================================================
RCS file: /cvs/gnome/evolution/mail/mail-folder-cache.c,v
retrieving revision 1.69
diff -u -3 -r1.69 mail-folder-cache.c
--- mail/mail-folder-cache.c	2 Jun 2003 17:50:25 -0000	1.69
+++ mail/mail-folder-cache.c	31 Jul 2003 19:13:54 -0000
@@ -51,6 +51,10 @@
 
 #define w(x) 
 #define d(x) /*(printf("%s(%d):%s: ",  __FILE__, __LINE__, __PRETTY_FUNCTION__), (x))*/
+#define o(x) /* offline debug */
+
+static void em_offline_check_folder(CamelStore *store, const char *account_name, const char *folder_name);
+static void em_offline_remove_store(const char *account_name);
 
 /* note that many things are effectively serialised by having them run in
    the main loop thread which they need to do because of corba/gtk calls */
@@ -358,6 +362,7 @@
 		up->path = g_strdup(mfi->path);
 		if (si->storage != NULL) {
 			up->name = g_strdup(fi->name);
+			em_offline_check_folder(si->store, evolution_storage_get_name(si->storage), fi->full_name);
 		}
 		up->uri = g_strdup(fi->url);
 		up->unread = (fi->unread_message_count==-1)?0:fi->unread_message_count;
@@ -665,6 +670,7 @@
 {
 	struct _update_data *ud;
 	struct _store_info *si;
+	char *name = NULL;
 
 	g_assert(CAMEL_IS_STORE(store));
 
@@ -677,6 +683,9 @@
 	if (si) {
 		g_hash_table_remove(stores, store);
 
+		if (si->storage)
+			name = g_strdup(evolution_storage_get_name(si->storage));
+
 		camel_object_unhook_event(store, "folder_created", store_folder_created, NULL);
 		camel_object_unhook_event(store, "folder_deleted", store_folder_deleted, NULL);
 		camel_object_unhook_event(store, "folder_renamed", store_folder_renamed, NULL);
@@ -701,6 +710,11 @@
 		g_free(si);
 	}
 	UNLOCK(info_lock);
+
+	if (name) {
+		em_offline_remove_store(name);
+		g_free(name);
+	}
 }
 
 static void
@@ -930,3 +944,410 @@
 
 	return fi.fi != NULL;
 }
+
+/* could just maintain another hashtable ... */
+struct _get_store_info {
+	const char *name;
+	CamelStore *store;
+};
+
+static void
+emfc_get_store_cb(void *key, struct _store_info *si, struct _get_store_info *gsi)
+{
+	const char *name;
+
+	if (gsi->store == NULL
+	    && si->storage != NULL
+	    && (name = evolution_storage_get_name(si->storage)) != NULL
+	    && strcmp(name, gsi->name) == 0) {
+		gsi->store = si->store;
+		camel_object_ref(si->store);
+	}
+}
+
+/* by account name */
+static CamelStore *
+mail_note_get_store_by_account(const char *name)
+{
+	struct _get_store_info gsi = { name, NULL };
+
+	/* this could cause a race with opening of stores vs folders (?) */
+	if (stores == NULL)
+		return NULL;
+
+	LOCK(info_lock);
+	g_hash_table_foreach(stores, (GHFunc)emfc_get_store_cb, &gsi);
+	UNLOCK(info_lock);
+
+	return gsi.store;
+}
+
+/* stuff for offline mode handling */
+
+/* FIXME: should it just be a list/flags stored on and handled by the store? */
+static pthread_mutex_t offline_lock = PTHREAD_MUTEX_INITIALIZER;
+
+static GSList *offline_paths;
+
+/* the store for these folders isn't active yet, so wait till it is */
+static EDList offline_pending_folders = E_DLIST_INITIALISER(offline_pending_folders);
+/* the store for these folders is active, these folders are to be synced */
+static EDList offline_sync_folders = E_DLIST_INITIALISER(offline_sync_folders);
+/* used for all other folders, so each folder info is always in some list */
+static EDList offline_idle_folders = E_DLIST_INITIALISER(offline_idle_folders);
+
+static EDList * const offline_lists[] = { &offline_pending_folders, &offline_sync_folders, &offline_idle_folders };
+
+static volatile int offline_sync_id = -1;
+
+struct _offline_folder_info {
+	struct _offline_folder_info *next, *prev;
+
+	char *shell_path;
+	const char *folder_path; /* note: folder_path points inside of shell_path */
+	CamelStore *store;
+	CamelFolder *folder;
+	int changed_id;
+
+	CamelFolderChangeInfo *changes;
+};
+
+static void emo_flush(void);
+static void emo_free_folder_info(struct _offline_folder_info *ofi);
+
+static char *emo_sync_desc(struct _mail_msg *mm, int done)
+{
+	return g_strdup(_("Retrieving remote mail"));
+}
+
+static void
+emo_folder_changed(CamelFolder *folder, CamelFolderChangeInfo *changes, struct _offline_folder_info *ofi)
+{
+	o(printf("folder changed '%s'\n", ofi->shell_path));
+
+	if (changes->uid_added->len > 0) {
+		LOCK(offline_lock);
+		camel_folder_change_info_cat(ofi->changes, changes);
+		e_dlist_remove((EDListNode *)ofi);
+		e_dlist_addtail(&offline_sync_folders, (EDListNode *)ofi);
+		UNLOCK(offline_lock);
+		emo_flush();
+	}
+}
+
+static void
+emo_sync_do(struct _mail_msg *mm)
+{
+	struct _offline_folder_info *ofi;
+
+	LOCK(offline_lock);
+
+	while ((ofi = (struct _offline_folder_info *)e_dlist_remhead(&offline_sync_folders))) {
+		CamelFolder *folder = ofi->folder;
+
+		e_dlist_addtail(&offline_idle_folders, (EDListNode *)ofi);
+
+		o(printf("offline syncing folder '%s'\n", ofi->folder_path));
+		if (folder == NULL) {
+			/* This could be slow ... deadlocks? */
+			folder = camel_store_get_folder(ofi->store, ofi->folder_path, 0, &mm->ex);
+			if (folder) {
+				if (!CAMEL_IS_DISCO_FOLDER(folder)) {
+					g_warning("folder '%s' is not able to work in offline mode, ignored", ofi->shell_path);
+					emo_free_folder_info(ofi);
+					continue;
+				}
+				ofi->folder = folder;
+				ofi->changed_id = camel_object_hook_event(folder, "folder_changed", (CamelObjectEventHookFunc)emo_folder_changed, ofi);
+			}
+		}
+
+		if (folder != NULL) {
+			camel_object_ref(folder);
+			if (ofi->changes->uid_added->len == 0) {
+				UNLOCK(offline_lock);
+				camel_disco_folder_prepare_for_offline((CamelDiscoFolder *)folder,
+								       "(match-all)",
+								       &mm->ex);
+				camel_object_unref(folder);
+			} else {
+				int i;
+				GPtrArray *uid_list = g_ptr_array_new();
+
+				for (i=0;i<ofi->changes->uid_added->len;i++)
+					g_ptr_array_add(uid_list, g_strdup(ofi->changes->uid_added->pdata[i]));
+				camel_folder_change_info_clear(ofi->changes);
+				UNLOCK(offline_lock);
+
+				for (i=0;camel_exception_get_id(&mm->ex) == 0 && i<uid_list->len;i++) {
+					char *uid = uid_list->pdata[i];
+
+					o(printf("syncing uid '%s'\n", uid));
+					camel_disco_folder_cache_message((CamelDiscoFolder *)ofi->folder,
+									 uid,
+									 &mm->ex);
+					g_free(uid);
+				}
+				g_ptr_array_free(uid_list, TRUE);
+				camel_object_unref(folder);
+			}
+		} else {
+			g_warning("cannot open offline folder '%s'\n", ofi->shell_path);
+		}
+
+		if (camel_exception_get_id(&mm->ex) == CAMEL_EXCEPTION_USER_CANCEL)
+			break;
+		camel_exception_clear(&mm->ex);
+		LOCK(offline_lock);
+	}
+
+	UNLOCK(offline_lock);
+
+	offline_sync_id = -1;
+}
+
+/* we use this to get the cancellation stuff for 'free' */
+static struct _mail_msg_op emo_sync_op = {
+	emo_sync_desc,
+	emo_sync_do,
+	NULL,
+	NULL,
+};
+
+static void
+emo_flush(void)
+{
+	if (offline_sync_id == -1 && !e_dlist_empty(&offline_sync_folders)) {
+		struct _mail_msg *m;
+
+		o(printf("launching thread to sync offline folders\n"));
+		m = mail_msg_new(&emo_sync_op, NULL, sizeof(*m));
+		offline_sync_id = m->seq;
+		e_thread_put(mail_thread_new, (EMsg *)m);
+	}
+}
+
+/* must have info_lock */
+static void
+emo_add_path(const char *path)
+{
+	struct _offline_folder_info *ofi;
+	char *store_name, *folder_name;
+
+	if (path[0] == 0)
+		return;
+
+	store_name = alloca(strlen(path)+1);
+	strcpy(store_name, path);
+	folder_name = strchr(store_name+1, '/');
+	if (folder_name == NULL)
+		return;
+	*folder_name++ = 0;
+	store_name++;
+
+	o(printf("Adding offline check path '%s'\n", path));
+
+	ofi = g_malloc0(sizeof(*ofi));
+	ofi->changes = camel_folder_change_info_new();
+	ofi->shell_path = g_strdup(path);
+	ofi->folder_path = ofi->shell_path + (folder_name - store_name + 1);
+	ofi->store = mail_note_get_store_by_account(store_name);
+	if (ofi->store == NULL)
+		e_dlist_addtail(&offline_pending_folders, (EDListNode *)ofi);
+	else
+		e_dlist_addtail(&offline_sync_folders, (EDListNode *)ofi);
+}
+
+static void
+em_offline_check_folder(CamelStore *store, const char *account_name, const char *folder_name)
+{
+	struct _offline_folder_info *ofw, *ofn;
+	char *path;
+
+	if (!CAMEL_IS_DISCO_STORE(store)
+	    || camel_url_get_param(((CamelService *)store)->url, "offline_sync") == NULL)
+		return;
+
+	LOCK(offline_lock);
+
+	path = g_strdup_printf("/%s/%s", account_name, folder_name);
+	o(printf("new folder available '%s'\n", path));
+	ofw = (struct _offline_folder_info *)offline_pending_folders.head;
+	ofn = ofw->next;
+	while (ofn) {
+		o(printf("path '%s'\n", ofw->shell_path));
+		if (!strcmp(ofw->shell_path, path)) {
+			o(printf("This is an offline folder ... processing it\n"));
+			ofw->store = store;
+			camel_object_ref(store);
+			e_dlist_remove((EDListNode *)ofw);
+			e_dlist_addtail(&offline_sync_folders, (EDListNode *)ofw);
+		}
+
+		ofw = ofn;
+		ofn = ofn->next;
+	}
+
+	g_free(path);
+
+	UNLOCK(offline_lock);
+
+	emo_flush();
+}
+
+static void
+emo_free_folder_info(struct _offline_folder_info *ofi)
+{
+	g_free(ofi->shell_path);
+	if (ofi->store)
+		camel_object_unref(ofi->store);
+	if (ofi->folder) {
+		camel_object_remove_event(ofi->folder, ofi->changed_id);
+		camel_object_unref(ofi->folder);
+	}
+	e_dlist_remove((EDListNode *)ofi);
+	g_free(ofi);
+}
+
+static void
+emo_remove_old(char *key, struct _offline_folder_info *ofi, void *data)
+{
+	o(printf("path removed, removing '%s'\n", ofi->shell_path));
+
+	emo_free_folder_info(ofi);
+}
+
+static void
+em_offline_remove_store(const char *account_name)
+{
+	char *path;
+	struct _offline_folder_info *ofw, *ofn;
+	int i;
+
+	/* store goes inactive, move any matching folders back to pending w/o a store/folder set */
+
+	LOCK(offline_lock);
+	path = g_strdup_printf("/%s/", account_name);
+	for (i=1;i<sizeof(offline_lists)/sizeof(offline_lists[0]);i++) {
+		ofw = (struct _offline_folder_info *)offline_lists[i]->head;
+		ofn = ofw->next;
+		while (ofn) {
+			if (!strncmp(ofw->shell_path, path, strlen(path))) {
+				o(printf("Removing folder '%s'\n", ofw->shell_path));
+				if (ofw->store) {
+					camel_object_unref(ofw->store);
+					ofw->store = NULL;
+				}
+				if (ofw->folder) {
+					camel_object_remove_event(ofw->folder, ofw->changed_id);
+					camel_object_unref(ofw->folder);
+					ofw->folder = NULL;
+				}
+				e_dlist_remove((EDListNode *)ofw);
+				e_dlist_addtail(&offline_pending_folders, (EDListNode *)ofw);
+			}
+			ofw = ofn;
+			ofn = ofn->next;
+		}
+	}
+	UNLOCK(offline_lock);
+
+	emo_flush();
+}
+
+static void
+emo_path_changed_cb(GConfClient *gconf, guint cnxn_id, GConfEntry *entry, gpointer user_data)
+{
+	GSList *l;
+	GHashTable *all_paths;
+	int i;
+
+	if (strcmp(entry->key, "/apps/evolution/shell/offline/folder_paths") != 0)
+		return;
+
+	o(printf("paths have changed, rechecking\n"));
+
+	g_slist_foreach(offline_paths, (GFunc)g_free, NULL);
+	g_slist_free(offline_paths);
+	offline_paths = gconf_client_get_list(gconf, "/apps/evolution/shell/offline/folder_paths", GCONF_VALUE_STRING, NULL);
+
+	LOCK(offline_lock);
+
+	all_paths = g_hash_table_new(g_str_hash, g_str_equal);
+	for (i=0;i<sizeof(offline_lists)/sizeof(offline_lists[0]);i++) {
+		struct _offline_folder_info *ofw, *ofn;
+
+		ofw = (struct _offline_folder_info *)offline_lists[i]->head;
+		ofn = ofw->next;
+		while (ofn) {
+			g_hash_table_insert(all_paths, ofw->shell_path, ofw);
+			ofw = ofn;
+			ofn = ofn->next;
+		}
+	}
+
+	for (l = offline_paths; l != NULL; l = l->next) {
+		char *path = l->data;
+
+		o(printf("path '%s'\n", path));
+		if (g_hash_table_lookup(all_paths, path) == NULL)
+			emo_add_path(path);
+		else
+			g_hash_table_remove(all_paths, path);
+	}
+
+	g_hash_table_foreach(all_paths, (GHFunc)emo_remove_old, NULL);
+	g_hash_table_destroy(all_paths);
+
+	UNLOCK(offline_lock);
+
+	emo_flush();
+}
+
+static void
+emo_account_changed_cb(EAccountList *eal, EAccount *ea)
+{
+	/* HACK: the way account_changed is called, it will automatically
+	   drop/restart the connection, and re-match any folders implicitly */
+	/* TODO: This doesn't actually work anyway since we end up just
+	   getting the same (old) store again afterwards */
+	em_offline_remove_store(ea->name);
+}
+
+static void
+emo_account_removed_cb(EAccountList *eal, EAccount *ea)
+{
+	em_offline_remove_store(ea->name);
+}
+
+void
+em_offline_init(void)
+{
+	GSList *l;
+	GConfClient *gconf = mail_config_get_gconf_client();
+	EAccountList *eal;
+
+	eal = mail_config_get_accounts();
+	g_signal_connect(eal, "account_changed", G_CALLBACK(emo_account_changed_cb), NULL);
+	g_signal_connect(eal, "account_removed", G_CALLBACK(emo_account_removed_cb), NULL);
+
+	/*offline_notify_id = */
+	gconf_client_add_dir(gconf, "/apps/evolution/shell/offline", GCONF_CLIENT_PRELOAD_ONELEVEL, NULL);
+	gconf_client_notify_add(gconf, "/apps/evolution/shell/offline", emo_path_changed_cb, NULL, NULL, NULL);
+	offline_paths = gconf_client_get_list(gconf, "/apps/evolution/shell/offline/folder_paths", GCONF_VALUE_STRING, NULL);
+	for (l = offline_paths; l != NULL; l = l->next)
+		emo_add_path((char *)l->data);
+
+	emo_flush();
+}
+
+#if 0
+/* tells the offline code what the currently displayed folder is, so it
+   can ensure that folder is downloaded first? */
+void
+em_offline_current_folder(CamelFolder *folder)
+{
+	/* could take move any sync job for that folder to the front of the list */
+	/* noop */
+}
+#endif
Index: mail/mail-folder-cache.h
===================================================================
RCS file: /cvs/gnome/evolution/mail/mail-folder-cache.h,v
retrieving revision 1.11
diff -u -3 -r1.11 mail-folder-cache.h
--- mail/mail-folder-cache.h	27 Oct 2001 00:47:21 -0000	1.11
+++ mail/mail-folder-cache.h	31 Jul 2003 19:13:54 -0000
@@ -47,4 +47,7 @@
    to a (referenced) copy of the folder if it has already been opened */
 int mail_note_get_folder_from_uri(const char *uri, CamelFolder **folderp);
 
+/* auto-download offline code */
+void em_offline_init(void);
+
 #endif
Index: mail/mail-ops.c
===================================================================
RCS file: /cvs/gnome/evolution/mail/mail-ops.c,v
retrieving revision 1.392
diff -u -3 -r1.392 mail-ops.c
--- mail/mail-ops.c	2 Jun 2003 17:50:25 -0000	1.392
+++ mail/mail-ops.c	31 Jul 2003 19:13:54 -0000
@@ -2145,7 +2145,7 @@
 	if (folder) {
 		if (CAMEL_IS_DISCO_FOLDER(folder)) {
 			camel_disco_folder_prepare_for_offline((CamelDiscoFolder *)folder,
-							       "(match-all (or (not (system-flag \"Seen\")) (system-flag \"Flagged\")))",
+							       "(match-all)",
 							       &mm->ex);
 		}
 		/* prepare_for_offline should do this? */
Index: shell/evolution-storage.c
===================================================================
RCS file: /cvs/gnome/evolution/shell/evolution-storage.c,v
retrieving revision 1.95
diff -u -3 -r1.95 evolution-storage.c
--- shell/evolution-storage.c	28 Apr 2003 20:22:12 -0000	1.95
+++ shell/evolution-storage.c	31 Jul 2003 19:13:54 -0000
@@ -872,6 +872,14 @@
 	/* FIXME: Implement me */
 }
 
+const char *
+evolution_storage_get_name(EvolutionStorage *storage)
+{
+	EvolutionStoragePrivate *priv = storage->priv;
+
+	return priv->name;
+}
+
 EvolutionStorageResult
 evolution_storage_register (EvolutionStorage *evolution_storage,
 			    GNOME_Evolution_StorageRegistry corba_storage_registry)
Index: shell/evolution-storage.h
===================================================================
RCS file: /cvs/gnome/evolution/shell/evolution-storage.h,v
retrieving revision 1.36
diff -u -3 -r1.36 evolution-storage.h
--- shell/evolution-storage.h	14 Mar 2003 18:13:58 -0000	1.36
+++ shell/evolution-storage.h	31 Jul 2003 19:13:54 -0000
@@ -143,6 +143,8 @@
 void                    evolution_storage_rename               (EvolutionStorage                *storage,
 								const char                      *new_name);
 
+const char 	       *evolution_storage_get_name	       (EvolutionStorage		*storage);
+
 EvolutionStorageResult  evolution_storage_register             (EvolutionStorage                *storage,
 								GNOME_Evolution_StorageRegistry  corba_registry);
 EvolutionStorageResult  evolution_storage_register_on_shell    (EvolutionStorage                *evolution_storage,
[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]