"clue seen by backend X" approach to performance tuning

[DJ Adams <dj adams pobox com>]

Not wanting to let the possibility of me appearing stupid get in the way, 
herewith a couple of thoughts on the approach of marking clues as "seen 
by backend X" with performance/tuning in mind ...

I've been trying to figure out why I'm getting seemingly random results 
from my initial implementation of getting the engine (specifically the
cluepacket-manager) to mark the clues in a cluepacket (well, in a
querycontext) as having been seen by backends as the backends return.

Basically, what's happening is that in the case where there's more than
one cluechainer (I have textchain and bookchain running side by side), 
and multiple new clues being generated, meaning RunQuery being called
multiple times with ever-increasing clue counts (this in itself is
actually correct behaviour), is that sometimes backends are getting 
invoked with a clue that *should* have been marked as already having 
been seen by that backend but *isn't*. 

And I think it's inherent in the async/threading nature of the design; 
while a clue(packet) is being looked at by backend X, i.e. *before*
that backend returns, another cluepacket is generated (without the
seen-by mark) -- on the basis of new clues returned from another
backend -- ready to be sprayed over the backends ... so the clue is
going to hit backend X again without being marked as seen.

I think. Then again, I could be totally wrong.

Sorry to foist this ramble upon you, but I need to get it out of my 
head (edd listened on IRC just now - thanks edd) before I go mad.

Comments / flames welcome

dj
http://www.pipetree.com/qmacro