[Evolution-hackers] A Camel API to get the filename of the cache, also a proposal to have one format to rule them all

[Philip Van Hoof <spam pvanhoof be>]

Hi there evos,

For an EPlugin that I'm working on I will need a Camel API to get the
filename of the cache.

I will attach a patch that adds this API. The EPlugin that I'm developing is
available at Bug# 565091 and more information about it can be found at

http://live.gnome.org/Evolution/Metadata.


I added a bug for tracking this request:

http://bugzilla.gnome.org/show_bug.cgi?id=566279

I know that for maildir (cur, tmp, new) and mbox (seek position) it's a
little bit controversial to return a filename. For maildir I always use
the cur-file one and for mbox I added "/!seek_pos" to the end of the
returned filename. 

The reason why I need this is that for indexing already cached E-mails,
Tracker will MIME parse what we can MIME parse. For example filenames
and Exif data of attached images is stolen out of the cached items, to
be made searchable.

We don't want to require Evolution to eat all the code involved in
indexing massive amounts of file formats. Best thing we can do right now
is to simply pass the filenames over IPC.

We STRONGLY recommend to the Evolution team to:

a) migrate away the IMAP specific data cache (see c to store separate parts)
b) migrate away the mbox data cache (the all-in-one file crap)

And to

c) invent a better storage format that doesn't store the attachments in
server's (usually) Base64 encoding. The one format to rule them all.

Instead store the encoded attachments in decoded format (original file
format). This will reduce diskspace (encoding increases diskspace usage)
and will make it more easy to scan the original file for XMP and Exif
information. Don't try to gzip or whatever anything. None of that makes
any sense (original files are usually compressed ideally already).

For example: devices that want to compress have filesystems that do this
for you. Don't be silly trying to do this yourself.

By storing the encoded version the only thing you currently gain is that
the feature "view E-mail source" doesn't need to recode the attachments.

This ain't a much-used feature. It doesn't have to be fast, at all.

No it doesn't. Really it doesn't.

For Maildir I recommend wasting diskspace by storing both the original
Maildir format and in parallel store the attachments separately.

Maildir ain't accessible by current Evolution's UI, by the way.

For MBox I recommend TO STOP USING THIS BROKEN FORMAT. It's insane with
today's mailboxes that easily grow to 3 gigabytes in size per user.


Once all start using the CamelDataCache API, implementing that new
format and implementing converters wont be very hard. 

For existing CamelDataCache users it's just one format to convert. For
IMAP, mbox, Maildir and mh it's indeed a few extra formats to handle
using a conversion. Wont kill you to implement that, and,  I'll help.


I know c) is a controversial proposal. But the current situation really
makes NO sense. Just go look at the ugly ugly code in Camel, think about
it for a second or two, and you too will see that it just ain't making
any sense.

Not for indexing-engines like either Beagle or Tracker nor is the
needless redundancy in implementations making any sense for Evolution
itself. It's rather a maintenance burden and it's making Evolution far
less agile for supporting new capabilities (like getting its cached data
indexed by softwares that focus on search capabilities).


-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be

Index: camel/providers/nntp/camel-nntp-folder.c
===================================================================
--- camel/providers/nntp/camel-nntp-folder.c	(revision 9848)
+++ camel/providers/nntp/camel-nntp-folder.c	(working copy)
@@ -123,6 +123,25 @@
         return ((CamelFolderClass *) folder_class)->set_message_flags (folder, uid, flags, set);
 }
 
+static char*
+nntp_get_filename (CamelFolder *folder, const char *uid, CamelException *ex)
+{
+	CamelNNTPStore *nntp_store = (CamelNNTPStore *) folder->parent_store;
+	char *article, *msgid;
+
+	article = alloca(strlen(uid)+1);
+	strcpy(article, uid);
+	msgid = strchr (article, ',');
+	if (msgid == NULL) {
+		camel_exception_setv (ex, CAMEL_EXCEPTION_SYSTEM,
+				      _("Internal error: UID in invalid format: %s"), uid);
+		return NULL;
+	}
+	*msgid++ = 0;
+
+	return camel_data_cache_get_filename (nntp_store->cache, "cache", msgid, ex);
+}
+
 static CamelStream *
 nntp_folder_download_message (CamelNNTPFolder *nntp_folder, const char *id, const char *msgid, CamelException *ex)
 {
@@ -483,6 +502,7 @@
 	camel_folder_class->count_by_expression = nntp_folder_count_by_expression;
 	camel_folder_class->search_by_uids = nntp_folder_search_by_uids;
 	camel_folder_class->search_free = nntp_folder_search_free;
+	camel_folder_class->get_filename = nntp_get_filename;
 }
 
 CamelType
Index: camel/providers/pop3/camel-pop3-folder.c
===================================================================
--- camel/providers/pop3/camel-pop3-folder.c	(revision 9848)
+++ camel/providers/pop3/camel-pop3-folder.c	(working copy)
@@ -54,6 +54,7 @@
 static GPtrArray *pop3_get_uids (CamelFolder *folder);
 static CamelMimeMessage *pop3_get_message (CamelFolder *folder, const char *uid, CamelException *ex);
 static gboolean pop3_set_message_flags (CamelFolder *folder, const char *uid, guint32 flags, guint32 set);
+static char* pop3_get_filename (CamelFolder *folder, const char *uid, CamelException *ex);
 
 static void
 camel_pop3_folder_class_init (CamelPOP3FolderClass *camel_pop3_folder_class)
@@ -69,7 +70,8 @@
 	camel_folder_class->get_message_count = pop3_get_message_count;
 	camel_folder_class->get_uids = pop3_get_uids;
 	camel_folder_class->free_uids = camel_folder_free_shallow;
-	
+	camel_folder_class->get_filename = pop3_get_filename;
+
 	camel_folder_class->get_message = pop3_get_message;
 	camel_folder_class->set_message_flags = pop3_set_message_flags;
 }
@@ -515,6 +517,23 @@
 	fi->stream = NULL;
 }
 
+static char*
+pop3_get_filename (CamelFolder *folder, const char *uid, CamelException *ex)
+{
+	CamelPOP3Folder *pop3_folder = (CamelPOP3Folder *)folder;
+	CamelPOP3Store *pop3_store = (CamelPOP3Store *)((CamelFolder *)pop3_folder)->parent_store;
+	CamelPOP3FolderInfo *fi;
+
+	fi = g_hash_table_lookup(pop3_folder->uids_uid, uid);
+	if (fi == NULL) {
+		camel_exception_setv (ex, CAMEL_EXCEPTION_FOLDER_INVALID_UID,
+				      _("No message with UID %s"), uid);
+		return NULL;
+	}
+
+	return camel_data_cache_get_filename (pop3_store->cache, "cache", fi->uid, NULL);
+}
+
 static CamelMimeMessage *
 pop3_get_message (CamelFolder *folder, const char *uid, CamelException *ex)
 {
Index: camel/providers/groupwise/camel-groupwise-folder.c
===================================================================
--- camel/providers/groupwise/camel-groupwise-folder.c	(revision 9848)
+++ camel/providers/groupwise/camel-groupwise-folder.c	(working copy)
@@ -89,6 +89,7 @@
 static void groupwise_msg_set_recipient_list (CamelMimeMessage *msg, EGwItem *item);
 static void gw_update_cache ( CamelFolder *folder, GList *item_list, CamelException *ex, gboolean uid_flag);
 static CamelMimeMessage *groupwise_folder_item_to_msg ( CamelFolder *folder, EGwItem *item, CamelException *ex );
+static char* groupwise_get_filename (CamelFolder *folder, const char *uid, CamelException *ex);
 
 
 #define d(x)  
@@ -96,6 +97,15 @@
 const char * GET_ITEM_VIEW_WITH_CACHE = "peek default recipient threading attachments subject status priority startDate created delivered size recurrenceKey message notification";
 const char * GET_ITEM_VIEW_WITHOUT_CACHE = "peek default recipient threading hasAttachment subject status priority startDate created delivered size recurrenceKey";
 
+static char* 
+groupwise_get_filename (CamelFolder *folder, const char *uid, CamelException *ex)
+{
+	CamelGroupwiseFolder *gw_folder = CAMEL_GROUPWISE_FOLDER(folder);
+
+	return camel_data_cache_get_filename (gw_folder->cache, "cache", uid, ex);
+}
+
+
 static CamelMimeMessage *
 groupwise_folder_get_message( CamelFolder *folder, const char *uid, CamelException *ex )
 {
@@ -2485,6 +2495,7 @@
 	camel_folder_class->sync = groupwise_sync;
 	camel_folder_class->expunge = groupwise_expunge;
 	camel_folder_class->transfer_messages_to = groupwise_transfer_messages_to;
+	camel_folder_class->get_filename = groupwise_get_filename;
 }
 
 static void
Index: camel/providers/imap/camel-imap-folder.c
===================================================================
--- camel/providers/imap/camel-imap-folder.c	(revision 9848)
+++ camel/providers/imap/camel-imap-folder.c	(working copy)
@@ -102,6 +102,7 @@
 static void imap_expunge (CamelFolder *folder, CamelException *ex);
 //static void imap_cache_message (CamelDiscoFolder *disco_folder, const char *uid, CamelException *ex);
 static void imap_rename (CamelFolder *folder, const char *new);
+static char* imap_get_filename (CamelFolder *folder, const char *uid, CamelException *ex);
 
 /* message manipulation */
 static CamelMimeMessage *imap_get_message (CamelFolder *folder, const gchar *uid,
@@ -167,6 +168,7 @@
 	camel_folder_class->sync= imap_sync;
 	camel_folder_class->append_message = imap_append_online;
 	camel_folder_class->transfer_messages_to = imap_transfer_online;
+	camel_folder_class->get_filename = imap_get_filename;
 }
 
 static void
@@ -446,6 +448,14 @@
 	g_free(imap_folder->priv);
 }
 
+static char*
+imap_get_filename (CamelFolder *folder, const char *uid, CamelException *ex)
+{
+	CamelImapFolder *imap_folder = (CamelImapFolder*) folder;
+
+	return camel_imap_message_cache_get_filename (imap_folder->cache, uid, "", ex);
+}
+
 static int
 imap_getv(CamelObject *object, CamelException *ex, CamelArgGetV *args)
 {
Index: camel/providers/imap/camel-imap-message-cache.c
===================================================================
--- camel/providers/imap/camel-imap-message-cache.c	(revision 9848)
+++ camel/providers/imap/camel-imap-message-cache.c	(working copy)
@@ -438,7 +438,36 @@
 	}
 }
 
+/**
+ * camel_imap_message_cache_get_filename:
+ * @cache: the cache
+ * @uid: the UID of the data to get
+ * @part_spec: the part_spec of the data to get
+ * @ex: exception
+ *
+ * Return value: the filename of a cache item
+ **/
+char*
+camel_imap_message_cache_get_filename (CamelImapMessageCache *cache,
+				       const char *uid,
+				       const char *part_spec,
+				       CamelException *ex)
+{
+	char *path;
+	
+	if (uid[0] == 0)
+		return NULL;
+	
+#ifdef G_OS_WIN32
+	/* See comment in insert_setup() */
+	if (!*part_spec)
+		part_spec = "~";
+#endif
+	path = g_strdup_printf ("%s/%s.%s", cache->path, uid, part_spec);
 
+	return path;
+}
+
 /**
  * camel_imap_message_cache_get:
  * @cache: the cache
Index: camel/providers/imap/camel-imap-message-cache.h
===================================================================
--- camel/providers/imap/camel-imap-message-cache.h	(revision 9848)
+++ camel/providers/imap/camel-imap-message-cache.h	(working copy)
@@ -86,6 +86,11 @@
 					      const char *part_spec,
 					      CamelException *ex);
 
+char*       camel_imap_message_cache_get_filename (CamelImapMessageCache *cache,
+					      const char *uid,
+					      const char *part_spec,
+					      CamelException *ex);
+
 void         camel_imap_message_cache_remove (CamelImapMessageCache *cache,
 					      const char *uid);
 
Index: camel/providers/imap4/camel-imap4-folder.c
===================================================================
--- camel/providers/imap4/camel-imap4-folder.c	(revision 9848)
+++ camel/providers/imap4/camel-imap4-folder.c	(working copy)
@@ -78,8 +78,8 @@
 static GPtrArray *imap4_search_by_expression (CamelFolder *folder, const char *expr, CamelException *ex);
 static GPtrArray *imap4_search_by_uids (CamelFolder *folder, const char *expr, GPtrArray *uids, CamelException *ex);
 static void imap4_search_free (CamelFolder *folder, GPtrArray *uids);
+static char* imap4_get_filename (CamelFolder *folder, const char *uid, CamelException *ex);
 
-
 static CamelOfflineFolderClass *parent_class = NULL;
 
 
@@ -139,6 +139,7 @@
 	folder_class->search_by_expression = imap4_search_by_expression;
 	folder_class->search_by_uids = imap4_search_by_uids;
 	folder_class->search_free = imap4_search_free;
+	folder_class->get_filename = imap4_get_filename;
 }
 
 static void
@@ -170,6 +171,14 @@
 	g_free (folder->cachedir);
 }
 
+static char*
+imap4_get_filename (CamelFolder *folder, const char *uid, CamelException *ex)
+{
+	CamelIMAP4Folder *imap4_folder = (CamelIMAP4Folder *) folder;
+
+	return camel_data_cache_get_filename (imap4_folder->cache, "cache", uid, ex);
+}
+
 static int
 imap4_getv (CamelObject *object, CamelException *ex, CamelArgGetV *args)
 {
Index: camel/providers/local/camel-mbox-folder.c
===================================================================
--- camel/providers/local/camel-mbox-folder.c	(revision 9848)
+++ camel/providers/local/camel-mbox-folder.c	(working copy)
@@ -31,6 +31,7 @@
 #include <unistd.h>
 #include <sys/stat.h>
 #include <sys/types.h>
+#include <inttypes.h>
 
 #include <glib.h>
 #include <glib/gi18n-lib.h>
@@ -67,6 +68,7 @@
 static void mbox_append_message(CamelFolder *folder, CamelMimeMessage * message, const CamelMessageInfo * info,	char **appended_uid, CamelException *ex);
 static CamelMimeMessage *mbox_get_message(CamelFolder *folder, const gchar * uid, CamelException *ex);
 static CamelLocalSummary *mbox_create_summary(CamelLocalFolder *lf, const char *path, const char *folder, CamelIndex *index);
+static char* mbox_get_filename (CamelFolder *folder, const char *uid, CamelException *ex);
 
 static void mbox_finalise(CamelObject * object);
 
@@ -83,6 +85,7 @@
 	/* virtual method overload */
 	camel_folder_class->append_message = mbox_append_message;
 	camel_folder_class->get_message = mbox_get_message;
+	camel_folder_class->get_filename = mbox_get_filename;
 
 	lclass->create_summary = mbox_create_summary;
 	lclass->lock = mbox_lock;
@@ -315,6 +318,54 @@
 	}
 }
 
+static char* 
+mbox_get_filename (CamelFolder *folder, const char *uid, CamelException *ex)
+{
+	CamelLocalFolder *lf = (CamelLocalFolder *)folder;
+	CamelMboxMessageInfo *info;
+	off_t frompos;
+	char *filename = NULL;
+
+	d(printf("Getting message %s\n", uid));
+
+	/* lock the folder first, burn if we can't, need write lock for summary check */
+	if (camel_local_folder_lock(lf, CAMEL_LOCK_WRITE, ex) == -1)
+		return NULL;
+
+	/* check for new messages always */
+	if (camel_local_summary_check((CamelLocalSummary *)folder->summary, lf->changes, ex) == -1) {
+		camel_local_folder_unlock(lf);
+		return NULL;
+	}
+	
+	/* get the message summary info */
+	info = (CamelMboxMessageInfo *) camel_folder_summary_uid(folder->summary, uid);
+
+	if (info == NULL) {
+		camel_exception_setv(ex, CAMEL_EXCEPTION_FOLDER_INVALID_UID,
+				     _("Cannot get message: %s from folder %s\n  %s"),
+				     uid, lf->folder_path, _("No such message"));
+		goto fail;
+	}
+
+	if (info->frompos == -1) {
+		camel_message_info_free((CamelMessageInfo *)info);
+		goto fail;
+	}
+
+	frompos = info->frompos;
+	camel_message_info_free((CamelMessageInfo *)info);
+
+
+	filename = g_strdup_printf ("%s%s!%" PRId64, lf->folder_path, G_DIR_SEPARATOR_S, (long long) frompos);
+
+fail:
+	/* and unlock now we're finished with it */
+	camel_local_folder_unlock(lf);
+
+	return filename;
+}
+
 static CamelMimeMessage *
 mbox_get_message(CamelFolder *folder, const gchar * uid, CamelException *ex)
 {
Index: camel/providers/local/camel-maildir-folder.c
===================================================================
--- camel/providers/local/camel-maildir-folder.c	(revision 9848)
+++ camel/providers/local/camel-maildir-folder.c	(working copy)
@@ -56,6 +56,7 @@
 
 static void maildir_append_message(CamelFolder * folder, CamelMimeMessage * message, const CamelMessageInfo *info, char **appended_uid, CamelException * ex);
 static CamelMimeMessage *maildir_get_message(CamelFolder * folder, const gchar * uid, CamelException * ex);
+static char* maildir_get_filename (CamelFolder *folder, const char *uid, CamelException *ex);
 
 static void maildir_finalize(CamelObject * object);
 
@@ -102,6 +103,7 @@
 
 	camel_folder_class->append_message = maildir_append_message;
 	camel_folder_class->get_message = maildir_get_message;
+	camel_folder_class->get_filename = maildir_get_filename;
 
 	lclass->create_summary = maildir_create_summary;
 }
@@ -232,6 +234,28 @@
 	g_free (dest);
 }
 
+static char* 
+maildir_get_filename (CamelFolder *folder, const char *uid, CamelException *ex)
+{
+	CamelLocalFolder *lf = (CamelLocalFolder *)folder;
+	CamelMaildirMessageInfo *mdi;
+	CamelMessageInfo *info;
+
+	/* get the message summary info */
+	if ((info = camel_folder_summary_uid(folder->summary, uid)) == NULL) {
+		camel_exception_setv(ex, CAMEL_EXCEPTION_FOLDER_INVALID_UID,
+				     _("Cannot get message: %s from folder %s\n  %s"),
+				     uid, lf->folder_path, _("No such message"));
+		return NULL;
+	}
+
+	mdi = (CamelMaildirMessageInfo *)info;
+
+	/* what do we do if the message flags (and :info data) changes?  filename mismatch - need to recheck I guess */
+	return g_strdup_printf("%s/cur/%s", lf->folder_path, camel_maildir_info_filename(mdi));
+}
+
+
 static CamelMimeMessage *
 maildir_get_message(CamelFolder * folder, const gchar * uid, CamelException * ex)
 {
Index: camel/providers/local/camel-mh-folder.c
===================================================================
--- camel/providers/local/camel-mh-folder.c	(revision 9848)
+++ camel/providers/local/camel-mh-folder.c	(working copy)
@@ -55,6 +55,7 @@
 
 static void mh_append_message(CamelFolder * folder, CamelMimeMessage * message, const CamelMessageInfo *info, char **appended_uid, CamelException * ex);
 static CamelMimeMessage *mh_get_message(CamelFolder * folder, const gchar * uid, CamelException * ex);
+static char* mh_get_filename (CamelFolder *folder, const char *uid, CamelException *ex);
 
 static void mh_finalize(CamelObject * object);
 
@@ -70,6 +71,7 @@
 	/* virtual method overload */
 	camel_folder_class->append_message = mh_append_message;
 	camel_folder_class->get_message = mh_get_message;
+	camel_folder_class->get_filename = mh_get_filename;
 
 	lclass->create_summary = mh_create_summary;
 }
@@ -187,6 +189,14 @@
 	g_free (name);
 }
 
+static char* mh_get_filename (CamelFolder *folder, const char *uid, CamelException *ex)
+{
+	CamelLocalFolder *lf = (CamelLocalFolder *)folder;
+
+	return g_strdup_printf("%s/%s", lf->folder_path, uid);
+}
+
+
 static CamelMimeMessage *mh_get_message(CamelFolder * folder, const gchar * uid, CamelException * ex)
 {
 	CamelLocalFolder *lf = (CamelLocalFolder *)folder;
Index: camel/camel-folder.c
===================================================================
--- camel/camel-folder.c	(revision 9848)
+++ camel/camel-folder.c	(working copy)
@@ -121,6 +121,10 @@
 static gboolean        folder_changed        (CamelObject *object,
 					      gpointer event_data);
 
+static char*           get_filename          (CamelFolder *folder, 
+					      const char *uid, 
+					      CamelException *ex);
+
 static CamelFolderQuotaInfo *get_quota_info  (CamelFolder *folder);
 
 static void
@@ -166,6 +170,7 @@
 	camel_folder_class->thaw = thaw;
 	camel_folder_class->is_frozen = is_frozen;
 	camel_folder_class->get_quota_info = get_quota_info;
+	camel_folder_class->get_filename = get_filename;
 
 	/* virtual method overload */
 	camel_object_class->getv = folder_getv;
@@ -233,7 +238,20 @@
 	return camel_folder_type;
 }
 
+static char*
+get_filename (CamelFolder *folder, const char *uid, CamelException *ex)
+{
+	w(g_warning ("CamelFolder::get_filename not implemented for '%s'",
+		     camel_type_to_name (CAMEL_OBJECT_GET_TYPE (folder))));
+	return g_strdup ("/dev/null");
+}
 
+char *
+camel_folder_get_filename (CamelFolder *folder, const char *uid, CamelException *ex)
+{
+	return CF_CLASS (folder)->get_filename (folder, uid, ex);
+}
+
 /**
  * camel_folder_construct:
  * @folder: a #CamelFolder object to construct
Index: camel/camel-folder.h
===================================================================
--- camel/camel-folder.h	(revision 9848)
+++ camel/camel-folder.h	(working copy)
@@ -211,6 +211,7 @@
 	
 	CamelFolderQuotaInfo * (*get_quota_info) (CamelFolder *folder);
 	guint32	(*count_by_expression) (CamelFolder *, const char *, CamelException *);
+	char * (*get_filename) (CamelFolder *, const char *uid, CamelException *);
 } CamelFolderClass;
 
 /* Standard Camel function */
@@ -353,6 +354,8 @@
 void camel_folder_free_shallow (CamelFolder *folder, GPtrArray *array);
 void camel_folder_free_deep    (CamelFolder *folder, GPtrArray *array);
 
+char * camel_folder_get_filename (CamelFolder *folder, const char *uid, CamelException *ex);
+
 /* update functions for change info */
 CamelFolderChangeInfo *	camel_folder_change_info_new		(void);
 void			camel_folder_change_info_clear		(CamelFolderChangeInfo *info);
Index: camel/camel-data-cache.c
===================================================================
--- camel/camel-data-cache.c	(revision 9848)
+++ camel/camel-data-cache.c	(working copy)
@@ -350,7 +350,29 @@
 	return stream;
 }
 
+
 /**
+ * camel_data_cache_get_filename:
+ * @cdc: A #CamelDataCache
+ * @path: Path to the (sub) cache the item exists in.
+ * @key: Key for the cache item.
+ * @ex: 
+ * 
+ * Lookup the filename for an item in the cache
+ * 
+ * Return value: The filename for a cache item
+ **/
+gchar *
+camel_data_cache_get_filename (CamelDataCache *cdc, const char *path, const char *key, CamelException *ex)
+{
+	char *real;
+
+	real = data_cache_path(cdc, FALSE, path, key);
+
+	return real;
+}
+
+/**
  * camel_data_cache_remove:
  * @cdc: A #CamelDataCache
  * @path: 
Index: camel/camel-data-cache.h
===================================================================
--- camel/camel-data-cache.h	(revision 9848)
+++ camel/camel-data-cache.h	(working copy)
@@ -85,6 +85,9 @@
 int             camel_data_cache_clear(CamelDataCache *cache,
 				       const char *path, CamelException *ex);
 
+gchar *         camel_data_cache_get_filename(CamelDataCache *cdc, 
+					      const char *path, const char *key, CamelException *ex);
+
 /* Standard Camel function */
 CamelType camel_data_cache_get_type (void);