[evolution-patches] new feature, vfolder w/ included conversations

[Not Zed <notzed ximian com>]

Big patch, actually took f-all to write except a good 3-4 hours wasted tracking down a bug in CamelFolderThread not creating a fully valid tree :(

Provides a new vfolder option "In conversation", which includes thread-related messages, there are three types you can ask for:
- all messages in the whole thread 'tree', from the original post down
- the subtree of only this message and all of its replies
- the subtree of only this message and all of its replies, and its direct parents to the original post

Issues:
- performance, it can need to do a LOT of work.  There's no real way around this, it needs to access the whole folder tree to look for any possible threads.  Big source folders will get the whole thread tree recalculated for each vfolder that uses it.
- providers need to use the new (simpler) camel_folder_search_search api (i've fixed all in the public cvs).
- the option once set, applies globally to the search, rather than just for the context in which the search is applied, if that makes sense.  basically means if you vfolder (or search) a vfolder which matches conversations, your vfolder will also implictly match conversations in the same way even if you didn't ask for it.  Its sort of tricky to fix; need to keep track of/apply the option for each paranthesised level of the s-exp.  A bit painful.
- i don't like the option names much - *shrug*.

hmm, what happened to my signature setting??  oh its getting it from the wrong from account.  wtf.

!Z

Michael Zucchi <notzed ximian com> Ximian Evolution and Free Software Developer

Novell, Inc.

Index: camel/ChangeLog
===================================================================
RCS file: /cvs/gnome/evolution/camel/ChangeLog,v
retrieving revision 1.2120
diff -u -3 -r1.2120 ChangeLog
--- camel/ChangeLog	6 May 2004 03:58:05 -0000	1.2120
+++ camel/ChangeLog	6 May 2004 09:28:32 -0000
@@ -1,5 +1,34 @@
 2004-05-06  Not Zed  <NotZed Ximian com>
 
+	* camel-digest-folder.c (digest_search_by_expression) 
+	(digest_search_by_uids): 
+	* providers/nntp/camel-nntp-folder.c (nntp_folder_search_by_expression)
+	(nntp_folder_search_by_uids): 
+	* providers/imap/camel-imap-folder.c (imap_search_by_expression) 
+	(imap_search_by_uids):
+	* providers/local/camel-local-folder.c (local_search_by_expression) 
+	(local_search_by_uids): use camel_folder_search_search & some minor cleanups.
+
+	* camel-folder-search.c (search_threads): keep track of the match
+	threads option for this search.
+	(camel_folder_search_match_expression): Removed, not used anymore.
+	(camel_folder_search_search): new api entry point for searching, a
+	bit easier to use and needed for thread matching.
+
+	* camel-folder-thread.h: make re a bitfield, saves 4 bytes/node.
+
+	* camel-folder-thread.c (thread_summary): properly setup parent
+	pointers when we remove phantom nodes.
+	(camel_folder_thread_messages_new): check wanted before destroying
+	it.
+	(get_root_subject): set re directly, not from address.
+
+	* camel-folder-search.c (camel_folder_search_search): new search
+	api entry point, take a full summary and optionally a subset of
+	uids to match against.
+	(search_match_all): use the uids' passed in to only search a
+	subset of uid's.
+
 	* providers/imap/camel-imap-store.c (connect_to_server): set
 	nodelay and keepalive on the socket.
 
Index: camel/camel-digest-folder.c
===================================================================
RCS file: /cvs/gnome/evolution/camel/camel-digest-folder.c,v
retrieving revision 1.17
diff -u -3 -r1.17 camel-digest-folder.c
--- camel/camel-digest-folder.c	23 Oct 2003 19:57:58 -0000	1.17
+++ camel/camel-digest-folder.c	6 May 2004 09:28:32 -0000
@@ -315,31 +315,22 @@
 	return message;
 }
 
-
 static GPtrArray *
 digest_search_by_expression (CamelFolder *folder, const char *expression, CamelException *ex)
 {
 	CamelDigestFolder *df = (CamelDigestFolder *) folder;
-	CamelFolderSearch *search;
-	GPtrArray *summary, *matches;
-	
-	summary = camel_folder_get_summary (folder);
+	GPtrArray *matches;
 	
 	CAMEL_DIGEST_FOLDER_LOCK (folder, search_lock);
 	
 	if (!df->priv->search)
 		df->priv->search = camel_folder_search_new ();
 	
-	search = df->priv->search;
-	camel_folder_search_set_folder (search, folder);
-	camel_folder_search_set_summary (search, summary);
-	
-	matches = camel_folder_search_execute_expression (search, expression, ex);
+	camel_folder_search_set_folder (df->priv->search, folder);
+	matches = camel_folder_search_search(df->priv->search, expression, NULL, ex);
 	
 	CAMEL_DIGEST_FOLDER_UNLOCK (folder, search_lock);
 	
-	camel_folder_free_summary (folder, summary);
-	
 	return matches;
 }
 
@@ -347,38 +338,20 @@
 digest_search_by_uids (CamelFolder *folder, const char *expression, GPtrArray *uids, CamelException *ex)
 {
 	CamelDigestFolder *df = (CamelDigestFolder *) folder;
-	CamelFolderSearch *search;
-	GPtrArray *summary, *matches;
-	int i;
-	
-	summary = g_ptr_array_new ();
-	for (i = 0; i < uids->len; i++) {
-		CamelMessageInfo *info;
-		
-		info = camel_folder_get_message_info (folder, uids->pdata[i]);
-		if (info)
-			g_ptr_array_add (summary, info);
-	}
-	
-	if (summary->len == 0)
-		return summary;
-	
+	GPtrArray *matches;
+
+	if (uids->len == 0)
+		return g_ptr_array_new();
+
 	CAMEL_DIGEST_FOLDER_LOCK (folder, search_lock);
 	
 	if (!df->priv->search)
 		df->priv->search = camel_folder_search_new ();
 	
-	search = df->priv->search;
-	camel_folder_search_set_folder (search, folder);
-	camel_folder_search_set_summary (search, summary);
-	
-	matches = camel_folder_search_execute_expression (search, expression, ex);
+	camel_folder_search_set_folder (df->priv->search, folder);
+	matches = camel_folder_search_search(df->priv->search, expression, NULL, ex);
 	
 	CAMEL_DIGEST_FOLDER_UNLOCK (folder, search_lock);
-	
-	for (i = 0; i < summary->len; i++)
-		camel_folder_free_message_info (folder, summary->pdata[i]);
-	g_ptr_array_free (summary, TRUE);
 	
 	return matches;
 }
Index: camel/camel-folder-search.c
===================================================================
RCS file: /cvs/gnome/evolution/camel/camel-folder-search.c,v
retrieving revision 1.64
diff -u -3 -r1.64 camel-folder-search.c
--- camel/camel-folder-search.c	4 Feb 2004 04:07:39 -0000	1.64
+++ camel/camel-folder-search.c	6 May 2004 09:28:32 -0000
@@ -36,6 +36,7 @@
 #include <glib.h>
 
 #include "camel-folder-search.h"
+#include "camel-folder-thread.h"
 
 #include "camel-exception.h"
 #include "camel-medium.h"
@@ -48,9 +49,15 @@
 #define d(x) 
 #define r(x) 
 
+#define MATCH_THREADS_ALL (1<<0)
+#define MATCH_THREADS_REPLIES (1<<1)
+#define MATCH_THREADS_PARENTS (1<<2)
+
 struct _CamelFolderSearchPrivate {
 	GHashTable *mempool_hash;
 	CamelException *ex;
+
+	int match_threads;
 };
 
 #define _PRIVATE(o) (((CamelFolderSearch *)(o))->priv)
@@ -72,6 +79,7 @@
 static ESExpResult *search_get_current_date(struct _ESExp *f, int argc, struct _ESExpResult **argv, CamelFolderSearch *s);
 static ESExpResult *search_get_size(struct _ESExp *f, int argc, struct _ESExpResult **argv, CamelFolderSearch *s);
 static ESExpResult *search_uid(struct _ESExp *f, int argc, struct _ESExpResult **argv, CamelFolderSearch *s);
+static ESExpResult *search_threads(struct _ESExp *f, int argc, struct _ESExpResult **argv, CamelFolderSearch *s);
 
 static ESExpResult *search_dummy(struct _ESExp *f, int argc, struct _ESExpResult **argv, CamelFolderSearch *search);
 
@@ -103,6 +111,7 @@
 	klass->get_current_date = search_get_current_date;
 	klass->get_size = search_get_size;
 	klass->uid = search_uid;
+	klass->threads = search_threads;
 }
 
 static void
@@ -205,6 +214,7 @@
 	{ "get-current-date", CAMEL_STRUCT_OFFSET(CamelFolderSearchClass, get_current_date), 1 },
 	{ "get-size", CAMEL_STRUCT_OFFSET(CamelFolderSearchClass, get_size), 1 },
 	{ "uid", CAMEL_STRUCT_OFFSET(CamelFolderSearchClass, uid), 1 },
+	{ "threads", CAMEL_STRUCT_OFFSET(CamelFolderSearchClass, threads), 1 },
 };
 
 void
@@ -408,36 +418,176 @@
 	return matches;
 }
 
+static void
+fill_thread_table(struct _CamelFolderThreadNode *root, GHashTable *id_hash)
+{
+	while (root) {
+		g_hash_table_insert(id_hash, (char *)camel_message_info_uid(root->message), root);
+		if (root->child)
+			fill_thread_table(root->child, id_hash);
+		root = root->next;
+	}
+}
+
+static void
+add_thread_results(struct _CamelFolderThreadNode *root, GHashTable *result_hash)
+{
+	while (root) {
+		g_hash_table_insert(result_hash, (char *)camel_message_info_uid(root->message), GINT_TO_POINTER (1));
+		if (root->child)
+			add_thread_results(root->child, result_hash);
+		root = root->next;
+	}
+}
+
 /**
- * camel_folder_search_match_expression:
+ * camel_folder_search_search:
  * @search: 
  * @expr: 
- * @info: 
+ * @uids: to search against, NULL for all uid's.
  * @ex: 
  * 
- * Returns #TRUE if the expression matches the specific message info @info.
- * Note that the folder and index may need to be set for body searches to
- * operate as well.
+ * Run a search.  Search must have had Folder already set on it, and
+ * it must implement summaries.
  * 
  * Return value: 
  **/
-gboolean
-camel_folder_search_match_expression(CamelFolderSearch *search, const char *expr, const CamelMessageInfo *info, CamelException *ex)
+GPtrArray *
+camel_folder_search_search(CamelFolderSearch *search, const char *expr, GPtrArray *uids, CamelException *ex)
 {
-	GPtrArray *uids;
-	int ret = FALSE;
+	ESExpResult *r;
+	GPtrArray *matches = NULL, *summary_set;
+	int i;
+	GHashTable *results;
+	EMemPool *pool;
+	struct _CamelFolderSearchPrivate *p = _PRIVATE(search);
+
+	g_assert(search->folder);
+
+	p->ex = ex;
+	p->match_threads = 0;
 
-	search->current = (CamelMessageInfo *)info;
+	/* setup our search list, summary_hash only contains those we're interested in */
+	search->summary = camel_folder_get_summary(search->folder);
+	search->summary_hash = g_hash_table_new(g_str_hash, g_str_equal);
 
-	uids = camel_folder_search_execute_expression(search, expr, ex);
 	if (uids) {
-		if (uids->len == 1)
-			ret = TRUE;
-		camel_folder_search_free_result(search, uids);
+		GHashTable *uids_hash = g_hash_table_new(g_str_hash, g_str_equal);
+
+		summary_set = search->summary_set = g_ptr_array_new();
+		for (i=0;i<uids->len;i++)
+			g_hash_table_insert(uids_hash, uids->pdata[i], uids->pdata[i]);
+		for (i=0;i<search->summary->len;i++)
+			if (g_hash_table_lookup(uids_hash, camel_message_info_uid(search->summary->pdata[i])))
+				g_ptr_array_add(search->summary_set, search->summary->pdata[i]);
+	} else {
+		summary_set = search->summary;
+	}
+
+	for (i=0;i<summary_set->len;i++)
+		g_hash_table_insert(search->summary_hash, (char *)camel_message_info_uid(summary_set->pdata[i]), summary_set->pdata[i]);
+
+	/* only re-parse if the search has changed */
+	if (search->last_search == NULL
+	    || strcmp(search->last_search, expr)) {
+		e_sexp_input_text(search->sexp, expr, strlen(expr));
+		if (e_sexp_parse(search->sexp) == -1) {
+			camel_exception_setv(ex, 1, _("Cannot parse search expression: %s:\n%s"), e_sexp_error(search->sexp), expr);
+			goto fail;
+		}
+
+		g_free(search->last_search);
+		search->last_search = g_strdup(expr);
+	}
+	r = e_sexp_eval(search->sexp);
+	if (r == NULL) {
+		if (!camel_exception_is_set(ex))
+			camel_exception_setv(ex, 1, _("Error executing search expression: %s:\n%s"), e_sexp_error(search->sexp), expr);
+		goto fail;
 	}
+
+	matches = g_ptr_array_new();
+
+	/* now create a folder summary to return?? */
+	if (r->type == ESEXP_RES_ARRAY_PTR) {
+		d(printf("got result ...\n"));
+
+		/* we use a mempool to store the strings, packed in tight as possible, and freed together */
+		/* because the strings are often short (like <8 bytes long), we would be wasting appx 50%
+		   of memory just storing the size tag that malloc assigns us and alignment padding, so this
+		   gets around that (and is faster to allocate and free as a bonus) */
+		pool = e_mempool_new(512, 256, E_MEMPOOL_ALIGN_BYTE);
+		/* reorder result in summary order */
+		results = g_hash_table_new(g_str_hash, g_str_equal);
+		for (i=0;i<r->value.ptrarray->len;i++) {
+			d(printf("adding match: %s\n", (char *)g_ptr_array_index(r->value.ptrarray, i)));
+			g_hash_table_insert(results, g_ptr_array_index(r->value.ptrarray, i), GINT_TO_POINTER (1));
+		}
+
+		/* Handle thread match, after all other processing, not very cheap! */
+		if (p->match_threads && search->folder) {
+			CamelFolderThread *threads = camel_folder_thread_messages_new(search->folder, NULL, TRUE);
+			GHashTable *threads_hash = g_hash_table_new(g_str_hash, g_str_equal);
+
+			fill_thread_table(threads->tree, threads_hash);
+			for (i=0;i<r->value.ptrarray->len;i++) {
+				struct _CamelFolderThreadNode *node, *scan;
+
+				node = g_hash_table_lookup(threads_hash, (char *)g_ptr_array_index(r->value.ptrarray, i));
+				if (node == NULL) /* this shouldn't happen but why cry over spilt milk */
+					continue;
+
+				/* select messages in thread according to search criteria */
+				if ((p->match_threads & (MATCH_THREADS_ALL|MATCH_THREADS_PARENTS)) == MATCH_THREADS_PARENTS) {
+					scan = node;
+					while (scan && scan->parent) {
+						scan = scan->parent;
+						g_hash_table_insert(results, (char *)camel_message_info_uid(scan->message), GINT_TO_POINTER(1));
+					}
+				} else if (p->match_threads & MATCH_THREADS_ALL) {
+					while (node && node->parent)
+						node = node->parent;
+				}
+				g_hash_table_insert(results, (char *)camel_message_info_uid(node->message), GINT_TO_POINTER(1));
+				if (node->child && (p->match_threads & (MATCH_THREADS_ALL|MATCH_THREADS_REPLIES)))
+					add_thread_results(node->child, results);
+			}
+			
+			camel_folder_thread_messages_unref(threads);
+			g_hash_table_destroy(threads_hash);
+		}
+
+		for (i=0;i<summary_set->len;i++) {
+			CamelMessageInfo *info = g_ptr_array_index(summary_set, i);
+			char *uid = (char *)camel_message_info_uid(info);
+			if (g_hash_table_lookup(results, uid))
+				g_ptr_array_add(matches, e_mempool_strdup(pool, uid));
+		}
+		g_hash_table_destroy(results);
+
+		/* instead of putting the mempool_hash in the structure, we keep the api clean by
+		   putting a reference to it in a hashtable.  Lets us do some debugging and catch
+		   unfree'd results as well. */
+		g_hash_table_insert(p->mempool_hash, matches, pool);
+	} else {
+		g_warning("Search returned an invalid result type");
+	}
+
+	e_sexp_result_free(search->sexp, r);
+fail:
+	if (search->summary_set)
+		g_ptr_array_free(search->summary_set, TRUE);
+	g_hash_table_destroy(search->summary_hash);
+
+	camel_folder_free_summary(search->folder, search->summary);
+	search->folder = NULL;
+	search->summary = NULL;
+	search->summary_hash = NULL;
+	search->summary_set = NULL;
 	search->current = NULL;
+	search->body_index = NULL;
 
-	return ret;
+	return matches;
 }
 
 void camel_folder_search_free_result(CamelFolderSearch *search, GPtrArray *result)
@@ -457,9 +607,6 @@
 	g_ptr_array_free(result, TRUE);
 }
 
-
-
-
 /* dummy function, returns false always, or an empty match array */
 static ESExpResult *
 search_dummy(struct _ESExp *f, int argc, struct _ESExpResult **argv, CamelFolderSearch *search)
@@ -516,7 +663,7 @@
 				for (i=0;i<v->len;i++)
 					g_hash_table_insert(have, s[i], s[i]);
 
-				v = search->summary;
+				v = search->summary_set?search->summary_set:search->summary;
 				m = (CamelMessageInfo **)v->pdata;
 				for (i=0;i<v->len;i++) {
 					char *uid = (char *)camel_message_info_uid(m[i]);
@@ -548,6 +695,7 @@
 {
 	int i;
 	ESExpResult *r, *r1;
+	GPtrArray *v;
 
 	if (argc>1) {
 		g_warning("match-all only takes a single argument, other arguments ignored");
@@ -585,21 +733,25 @@
 		return r;
 	}
 
-	/* TODO: Could make this a bit faster in the uncommon case (of match-everything) */
-	for (i=0;i<search->summary->len;i++) {
-		search->current = g_ptr_array_index(search->summary, i);
+	v = search->summary_set?search->summary_set:search->summary;
+	for (i=0;i<v->len;i++) {
+		const char *uid;
+
+		search->current = g_ptr_array_index(v, i);
+		uid = camel_message_info_uid(search->current);
+
 		if (argc>0) {
 			r1 = e_sexp_term_eval(f, argv[0]);
 			if (r1->type == ESEXP_RES_BOOL) {
 				if (r1->value.bool)
-					g_ptr_array_add(r->value.ptrarray, (char *)camel_message_info_uid(search->current));
+					g_ptr_array_add(r->value.ptrarray, (char *)uid);
 			} else {
 				g_warning("invalid syntax, matches require a single bool result");
 				e_sexp_fatal_error(f, _("(match-all) requires a single bool result"));
 			}
 			e_sexp_result_free(f, r1);
 		} else {
-			g_ptr_array_add(r->value.ptrarray, (char *)camel_message_info_uid(search->current));
+			g_ptr_array_add(r->value.ptrarray, (char *)uid);
 		}
 	}
 	search->current = NULL;
@@ -917,8 +1069,10 @@
 		
 		g_ptr_array_free(indexed, TRUE);
 	} else {
-		for (i=0;i<search->summary->len;i++) {
-			CamelMessageInfo *info = g_ptr_array_index(search->summary, i);
+		GPtrArray *v = search->summary_set?search->summary_set:search->summary;
+
+		for (i=0;i<v->len;i++) {
+			CamelMessageInfo *info = g_ptr_array_index(v, i);
 			const char *uid = camel_message_info_uid(info);
 			
 			if (match_words_message(search->folder, uid, words, ex))
@@ -966,8 +1120,10 @@
 		r->value.ptrarray = g_ptr_array_new();
 
 		if (argc == 1 && argv[0]->value.string[0] == 0) {
-			for (i=0;i<search->summary->len;i++) {
-				CamelMessageInfo *info = g_ptr_array_index(search->summary, i);
+			GPtrArray *v = search->summary_set?search->summary_set:search->summary;
+
+			for (i=0;i<v->len;i++) {
+				CamelMessageInfo *info = g_ptr_array_index(v, i);
 
 				g_ptr_array_add(r->value.ptrarray, (char *)camel_message_info_uid(info));
 			}
@@ -1168,6 +1324,38 @@
 			if (argv[i]->type == ESEXP_RES_STRING)
 				g_ptr_array_add(r->value.ptrarray, argv[i]->value.string);
 		}
+	}
+
+	return r;
+}
+
+static ESExpResult *
+search_threads(struct _ESExp *f, int argc, struct _ESExpResult **argv, CamelFolderSearch *search)
+{
+	ESExpResult *r;
+	struct _CamelFolderSearchPrivate *p = search->priv;
+	int i;
+
+	/* This works sort of 'funny', we just flag it, then add the threads after we're done */
+	if (argc == 0)
+		p->match_threads = MATCH_THREADS_ALL;
+	else for (i=0;i<argc;i++) {
+		if (argv[i]->type == ESEXP_RES_STRING) {
+			if (!strcmp(argv[i]->value.string, "all"))
+				p->match_threads |= MATCH_THREADS_ALL;
+			else if (!strcmp(argv[i]->value.string, "replies"))
+				p->match_threads |= MATCH_THREADS_REPLIES;
+			else if (!strcmp(argv[i]->value.string, "replies_parents"))
+				p->match_threads |= MATCH_THREADS_REPLIES|MATCH_THREADS_PARENTS;
+		}
+	}
+
+	if (search->current) {
+		r = e_sexp_result_new(f, ESEXP_RES_BOOL);
+		r->value.bool = FALSE;
+	} else {
+		r = e_sexp_result_new(f, ESEXP_RES_ARRAY_PTR);
+		r->value.ptrarray = g_ptr_array_new();
 	}
 
 	return r;
Index: camel/camel-folder-search.h
===================================================================
RCS file: /cvs/gnome/evolution/camel/camel-folder-search.h,v
retrieving revision 1.27
diff -u -3 -r1.27 camel-folder-search.h
--- camel/camel-folder-search.h	3 Sep 2003 18:05:54 -0000	1.27
+++ camel/camel-folder-search.h	6 May 2004 09:28:32 -0000
@@ -50,6 +50,7 @@
 	/* these are only valid during the search, and are reset afterwards */
 	CamelFolder *folder;	/* folder for current search */
 	GPtrArray *summary;	/* summary array for current search */
+	GPtrArray *summary_set;	/* subset of summary to actually include in search */
 	GHashTable *summary_hash; /* hashtable of summary items */
 	CamelMessageInfo *current; /* current message info, when searching one by one */
 	CamelMimeMessage *current_message; /* cache of current message, if required */
@@ -112,18 +113,23 @@
 
 	/* (uid "uid" ...) True if the uid is in the list */
 	ESExpResult * (*uid)(struct _ESExp *f, int argc, struct _ESExpResult **argv, CamelFolderSearch *s);
+
+	/* (threads) True if message is in thread of message already matched */
+	ESExpResult * (*threads)(struct _ESExp *f, int argc, struct _ESExpResult **argv, CamelFolderSearch *s);
 };
 
 CamelType		camel_folder_search_get_type	(void);
 CamelFolderSearch      *camel_folder_search_new	(void);
 void camel_folder_search_construct (CamelFolderSearch *search);
 
+/* This stuff currently gets cleared when you run a search ... what on earth was i thinking ... */
 void camel_folder_search_set_folder(CamelFolderSearch *search, CamelFolder *folder);
 void camel_folder_search_set_summary(CamelFolderSearch *search, GPtrArray *summary);
 void camel_folder_search_set_body_index(CamelFolderSearch *search, CamelIndex *index);
+/* this interface is deprecated */
 GPtrArray *camel_folder_search_execute_expression(CamelFolderSearch *search, const char *expr, CamelException *ex);
-gboolean camel_folder_search_match_expression(CamelFolderSearch *search, const char *expr,
-					      const CamelMessageInfo *info, CamelException *ex);
+
+GPtrArray *camel_folder_search_search(CamelFolderSearch *search, const char *expr, GPtrArray *uids, CamelException *ex);
 void camel_folder_search_free_result(CamelFolderSearch *search, GPtrArray *);
 
 #ifdef __cplusplus
Index: camel/camel-folder-thread.c
===================================================================
RCS file: /cvs/gnome/evolution/camel/camel-folder-thread.c,v
retrieving revision 1.18
diff -u -3 -r1.18 camel-folder-thread.c
--- camel/camel-folder-thread.c	18 Jun 2003 15:13:12 -0000	1.18
+++ camel/camel-folder-thread.c	6 May 2004 09:28:32 -0000
@@ -37,7 +37,8 @@
 #include "camel-folder-thread.h"
 #include "e-util/e-memory.h"
 
-#define d(x)
+#define d(x) 
+#define m(x) x
 
 /*#define TIMEIT*/
 
@@ -114,6 +115,7 @@
 			if (c->child == NULL) {
 				d(printf("removing empty node\n"));
 				lastc->next = c->next;
+				m(memset(c, 0xfe, sizeof(*c)));
 				e_memchunk_free(thread->node_chunks, c);
 				continue;
 			}
@@ -151,13 +153,13 @@
 }
 
 static char *
-get_root_subject(CamelFolderThreadNode *c, int *re)
+get_root_subject(CamelFolderThreadNode *c)
 {
 	char *s, *p;
 	CamelFolderThreadNode *scan;
 	
 	s = NULL;
-	*re = FALSE;
+	c->re = FALSE;
 	if (c->message)
 		s = (char *)camel_message_info_subject(c->message);
 	else {
@@ -183,7 +185,7 @@
 				while (isdigit(*p) || (ispunct(*p) && (*p != ':')))
 					p++;
 				if (*p==':') {
-					*re = TRUE;
+					c->re = TRUE;
 					s = p+1;
 				} else
 					break;
@@ -235,7 +237,7 @@
 	clast = (CamelFolderThreadNode *)cp;
 	c = clast->next;
 	while (c) {
-		c->root_subject = get_root_subject(c, &c->re);
+		c->root_subject = get_root_subject(c);
 		if (c->root_subject) {
 			container = g_hash_table_lookup(subject_table, c->root_subject);
 			if (container == NULL
@@ -264,6 +266,7 @@
 					scan = scan->next;
 				scan->next = c->child;
 				clast->next = c->next;
+				m(memset(c, 0xee, sizeof(*c)));
 				e_memchunk_free(thread->node_chunks, c);
 				continue;
 			} if (c->message == NULL && container->message != NULL) {
@@ -521,17 +524,25 @@
 		child = c->next;
 		if (child->message == NULL) {
 			newtop = child->child;
+			newtop->parent = NULL;
 			/* unlink pseudo node */
 			c->next = newtop;
 
-			/* link its siblings onto the end of its children */
+			/* link its siblings onto the end of its children, fix all parent pointers */
 			scan = (CamelFolderThreadNode *)&newtop->child;
-			while (scan->next)
+			while (scan->next) {
 				scan = scan->next;
+			}
 			scan->next = newtop->next;
+			while (scan->next) {
+				scan = scan->next;
+				scan->parent = newtop;
+			}
+
 			/* and link the now 'real' node into the list */
 			newtop->next = child->next;
 			c = newtop;
+			m(memset(child, 0xde, sizeof(*child)));
 			e_memchunk_free(thread->node_chunks, child);
 		} else {
 			c = child;
@@ -544,6 +555,8 @@
 		c = c->next;
 		if (c->message == NULL)
 			g_warning("threading missed removing a pseudo node: %s\n", c->root_subject);
+		if (c->parent != NULL)
+			g_warning("base node has a non-null parent: %s\n", c->root_subject);
 	}
 
 	thread->tree = head;
@@ -616,7 +629,8 @@
 
 	thread_summary(thread, summary);
 
-	g_hash_table_destroy(wanted);
+	if (wanted)
+		g_hash_table_destroy(wanted);
 
 	return thread;
 }
Index: camel/camel-folder-thread.h
===================================================================
RCS file: /cvs/gnome/evolution/camel/camel-folder-thread.h,v
retrieving revision 1.8
diff -u -3 -r1.8 camel-folder-thread.h
--- camel/camel-folder-thread.h	27 Aug 2002 21:45:22 -0000	1.8
+++ camel/camel-folder-thread.h	6 May 2004 09:28:32 -0000
@@ -36,8 +36,8 @@
 		*child;
 	const CamelMessageInfo *message;
 	char *root_subject;	/* cached root equivalent subject */
-	int re;			/* re version of subject? */
-	int order;
+	guint32 order:31;
+	guint32 re:1;			/* re version of subject? */
 } CamelFolderThreadNode;
 
 typedef struct _CamelFolderThread {
Index: camel/providers/imap/camel-imap-folder.c
===================================================================
RCS file: /cvs/gnome/evolution/camel/providers/imap/camel-imap-folder.c,v
retrieving revision 1.331
diff -u -3 -r1.331 camel-imap-folder.c
--- camel/providers/imap/camel-imap-folder.c	5 May 2004 13:27:33 -0000	1.331
+++ camel/providers/imap/camel-imap-folder.c	6 May 2004 09:28:33 -0000
@@ -1574,7 +1574,7 @@
 imap_search_by_expression (CamelFolder *folder, const char *expression, CamelException *ex)
 {
 	CamelImapFolder *imap_folder = CAMEL_IMAP_FOLDER (folder);
-	GPtrArray *matches, *summary;
+	GPtrArray *matches;
 
 	/* we could get around this by creating a new search object each time,
 	   but i doubt its worth it since any long operation would lock the
@@ -1582,14 +1582,10 @@
 	CAMEL_IMAP_FOLDER_LOCK(folder, search_lock);
 
 	camel_folder_search_set_folder (imap_folder->search, folder);
-	summary = camel_folder_get_summary(folder);
-	camel_folder_search_set_summary(imap_folder->search, summary);
-	matches = camel_folder_search_execute_expression (imap_folder->search, expression, ex);
+	matches = camel_folder_search_search(imap_folder->search, expression, NULL, ex);
 
 	CAMEL_IMAP_FOLDER_UNLOCK(folder, search_lock);
 
-	camel_folder_free_summary(folder, summary);
-
 	return matches;
 }
 
@@ -1597,38 +1593,17 @@
 imap_search_by_uids(CamelFolder *folder, const char *expression, GPtrArray *uids, CamelException *ex)
 {
 	CamelImapFolder *imap_folder = CAMEL_IMAP_FOLDER(folder);
-	GPtrArray *summary, *matches;
-	int i;
-
-	/* NOTE: could get away without the search lock by creating a new
-	   search object each time */
-	
-	qsort (uids->pdata, uids->len, sizeof (void *), uid_compar);
-	
-	summary = g_ptr_array_new();
-	for (i=0;i<uids->len;i++) {
-		CamelMessageInfo *info;
-
-		info = camel_folder_get_message_info(folder, uids->pdata[i]);
-		if (info)
-			g_ptr_array_add(summary, info);
-	}
+	GPtrArray *matches;
 
-	if (summary->len == 0)
-		return summary;
+	if (uids->len == 0)
+		return g_ptr_array_new();
 
 	CAMEL_IMAP_FOLDER_LOCK(folder, search_lock);
 
 	camel_folder_search_set_folder(imap_folder->search, folder);
-	camel_folder_search_set_summary(imap_folder->search, summary);
-
-	matches = camel_folder_search_execute_expression(imap_folder->search, expression, ex);
+	matches = camel_folder_search_search(imap_folder->search, expression, uids, ex);
 
 	CAMEL_IMAP_FOLDER_UNLOCK(folder, search_lock);
-
-	for (i=0;i<summary->len;i++)
-		camel_folder_free_message_info(folder, summary->pdata[i]);
-	g_ptr_array_free(summary, TRUE);
 
 	return matches;
 }
Index: camel/providers/local/camel-local-folder.c
===================================================================
RCS file: /cvs/gnome/evolution/camel/providers/local/camel-local-folder.c,v
retrieving revision 1.48
diff -u -3 -r1.48 camel-local-folder.c
--- camel/providers/local/camel-local-folder.c	29 Mar 2004 20:13:47 -0000	1.48
+++ camel/providers/local/camel-local-folder.c	6 May 2004 09:28:33 -0000
@@ -575,10 +575,7 @@
 local_search_by_expression(CamelFolder *folder, const char *expression, CamelException *ex)
 {
 	CamelLocalFolder *local_folder = CAMEL_LOCAL_FOLDER(folder);
-	GPtrArray *summary, *matches;
-
-	/* NOTE: could get away without the search lock by creating a new
-	   search object each time */
+	GPtrArray *matches;
 
 	CAMEL_LOCAL_FOLDER_LOCK(folder, search_lock);
 
@@ -587,15 +584,10 @@
 
 	camel_folder_search_set_folder(local_folder->search, folder);
 	camel_folder_search_set_body_index(local_folder->search, local_folder->index);
-	summary = camel_folder_get_summary(folder);
-	camel_folder_search_set_summary(local_folder->search, summary);
-
-	matches = camel_folder_search_execute_expression(local_folder->search, expression, ex);
+	matches = camel_folder_search_search(local_folder->search, expression, NULL, ex);
 
 	CAMEL_LOCAL_FOLDER_UNLOCK(folder, search_lock);
 
-	camel_folder_free_summary(folder, summary);
-
 	return matches;
 }
 
@@ -603,23 +595,10 @@
 local_search_by_uids(CamelFolder *folder, const char *expression, GPtrArray *uids, CamelException *ex)
 {
 	CamelLocalFolder *local_folder = CAMEL_LOCAL_FOLDER(folder);
-	GPtrArray *summary, *matches;
-	int i;
-
-	/* NOTE: could get away without the search lock by creating a new
-	   search object each time */
+	GPtrArray *matches;
 
-	summary = g_ptr_array_new();
-	for (i=0;i<uids->len;i++) {
-		CamelMessageInfo *info;
-
-		info = camel_folder_get_message_info(folder, uids->pdata[i]);
-		if (info)
-			g_ptr_array_add(summary, info);
-	}
-
-	if (summary->len == 0)
-		return summary;
+	if (uids->len == 0)
+		return g_ptr_array_new();
 
 	CAMEL_LOCAL_FOLDER_LOCK(folder, search_lock);
 
@@ -628,15 +607,9 @@
 
 	camel_folder_search_set_folder(local_folder->search, folder);
 	camel_folder_search_set_body_index(local_folder->search, local_folder->index);
-	camel_folder_search_set_summary(local_folder->search, summary);
-
-	matches = camel_folder_search_execute_expression(local_folder->search, expression, ex);
+	matches = camel_folder_search_search(local_folder->search, expression, uids, ex);
 
 	CAMEL_LOCAL_FOLDER_UNLOCK(folder, search_lock);
-
-	for (i=0;i<summary->len;i++)
-		camel_folder_free_message_info(folder, summary->pdata[i]);
-	g_ptr_array_free(summary, TRUE);
 
 	return matches;
 }
Index: camel/providers/nntp/camel-nntp-folder.c
===================================================================
RCS file: /cvs/gnome/evolution/camel/providers/nntp/camel-nntp-folder.c,v
retrieving revision 1.44
diff -u -3 -r1.44 camel-nntp-folder.c
--- camel/providers/nntp/camel-nntp-folder.c	21 Apr 2004 06:52:05 -0000	1.44
+++ camel/providers/nntp/camel-nntp-folder.c	6 May 2004 09:28:34 -0000
@@ -287,7 +287,7 @@
 nntp_folder_search_by_expression (CamelFolder *folder, const char *expression, CamelException *ex)
 {
 	CamelNNTPFolder *nntp_folder = CAMEL_NNTP_FOLDER (folder);
-	GPtrArray *matches, *summary;
+	GPtrArray *matches;
 	
 	CAMEL_NNTP_FOLDER_LOCK(nntp_folder, search_lock);
 	
@@ -295,15 +295,10 @@
 		nntp_folder->search = camel_folder_search_new ();
 	
 	camel_folder_search_set_folder (nntp_folder->search, folder);
-	summary = camel_folder_get_summary (folder);
-	camel_folder_search_set_summary (nntp_folder->search, summary);
-	
-	matches = camel_folder_search_execute_expression (nntp_folder->search, expression, ex);
+	matches = camel_folder_search_search(nntp_folder->search, expression, NULL, ex);
 	
 	CAMEL_NNTP_FOLDER_UNLOCK(nntp_folder, search_lock);
 	
-	camel_folder_free_summary (folder, summary);
-	
 	return matches;
 }
 
@@ -311,22 +306,10 @@
 nntp_folder_search_by_uids (CamelFolder *folder, const char *expression, GPtrArray *uids, CamelException *ex)
 {
 	CamelNNTPFolder *nntp_folder = (CamelNNTPFolder *) folder;
-	GPtrArray *summary, *matches;
-	int i;
-	
-	/* NOTE: could get away without the search lock by creating a new
-	   search object each time */
-	
-	summary = g_ptr_array_new ();
-	for (i = 0; i < uids->len; i++) {
-		CamelMessageInfo *info;
-		
-		if ((info = camel_folder_get_message_info (folder, uids->pdata[i])))
-			g_ptr_array_add (summary, info);
-	}
-	
-	if (summary->len == 0)
-		return summary;
+	GPtrArray *matches;
+
+	if (uids->len == 0)
+		return g_ptr_array_new();
 	
 	CAMEL_NNTP_FOLDER_LOCK(folder, search_lock);
 	
@@ -334,16 +317,9 @@
 		nntp_folder->search = camel_folder_search_new ();
 	
 	camel_folder_search_set_folder (nntp_folder->search, folder);
-	camel_folder_search_set_summary (nntp_folder->search, summary);
-	
-	matches = camel_folder_search_execute_expression (nntp_folder->search, expression, ex);
+	matches = camel_folder_search_search(nntp_folder->search, expression, uids, ex);
 	
 	CAMEL_NNTP_FOLDER_UNLOCK(folder, search_lock);
-	
-	for (i = 0; i < summary->len; i++)
-		camel_folder_free_message_info (folder, summary->pdata[i]);
-	
-	g_ptr_array_free (summary, TRUE);
 	
 	return matches;
 }
Index: filter/ChangeLog
===================================================================
RCS file: /cvs/gnome/evolution/filter/ChangeLog,v
retrieving revision 1.404
diff -u -3 -r1.404 ChangeLog
--- filter/ChangeLog	3 May 2004 14:54:24 -0000	1.404
+++ filter/ChangeLog	6 May 2004 09:28:34 -0000
@@ -1,3 +1,7 @@
+2004-05-06  Not Zed  <NotZed Ximian com>
+
+	* vfoldertypes.xml: added stuff for match on thread.
+
 2004-05-03  Jeffrey Stedfast  <fejj ximian com>
 
 	* vfoldertypes.xml: Added the Junk status flag to the xml
Index: filter/libfilter-i18n.h
===================================================================
RCS file: /cvs/gnome/evolution/filter/libfilter-i18n.h,v
retrieving revision 1.55
diff -u -3 -r1.55 libfilter-i18n.h
--- filter/libfilter-i18n.h	26 Mar 2004 05:10:04 -0000	1.55
+++ filter/libfilter-i18n.h	6 May 2004 09:28:34 -0000
@@ -1,5 +1,7 @@
 /* Automatically generated. Do not edit. */
 char *s = N_("Adjust Score");
+char *s = N_("All related conversations");
+char *s = N_("All replies and parent messages");
 char *s = N_("Assign Color");
 char *s = N_("Assign Score");
 char *s = N_("Attachments");
@@ -24,6 +26,7 @@
 char *s = N_("Expression");
 char *s = N_("Follow Up");
 char *s = N_("Important");
+char *s = N_("In Conversation");
 char *s = N_("is");
 char *s = N_("is after");
 char *s = N_("is before");
@@ -47,6 +50,7 @@
 char *s = N_("Recipients");
 char *s = N_("Regex Match");
 char *s = N_("Replied to");
+char *s = N_("Replies to messages");
 char *s = N_("returns");
 char *s = N_("returns greater than");
 char *s = N_("returns less than");
Index: filter/vfoldertypes.xml
===================================================================
RCS file: /cvs/gnome/evolution/filter/vfoldertypes.xml,v
retrieving revision 1.28
diff -u -3 -r1.28 vfoldertypes.xml
--- filter/vfoldertypes.xml	3 May 2004 14:54:24 -0000	1.28
+++ filter/vfoldertypes.xml	6 May 2004 09:28:34 -0000
@@ -420,5 +420,29 @@
    <input type="string" name="mlist"/>
  </part>
 
+ <part name="threads">
+  <title>In Conversation</title>
+  <input type="optionlist" name="thread-type">
+   <option value="all">
+    <title>All related conversations</title>
+    <code>
+     (threads "all")
+    </code>
+   </option>
+   <option value="replies">
+    <title>Replies to messages</title>
+    <code>
+     (threads "replies")
+    </code>
+   </option>
+   <option value="replies_parents">
+    <title>All replies and parent messages</title>
+    <code>
+     (threads "replies_parents")
+    </code>
+   </option>
+  </input>
+ </part>
+
 </partset>
 </filterdescription>