Re: suggest: make searches efficient

["Mikkel Kamstrup Erlandsen" <mikkel kamstrup gmail com>]

On 09/01/2008, Debajyoti Bera <dbera web gmail com> wrote:
> > Ie cluster/facet extraction on the result set? It is far from trvial
> > to do in an efficient and scalable way, but it can be done... The
> > website I linked to have 10M items in the index.
>
> Its easy to do such fancy data-mining tricks for a webserver. On a desktop,
> such fancy things might cause annoying CPU spikes. But still this needs to be
> implemented to see how far can it work.

The example I sent you uses an external structure we call the "facet
map" to look up clusters on the fly[1]. It is an addition to the
Lucene index that can be updated at reasonably little overhead. -
Which still might be too much for a desktop box of course.

> Aside the question of scalability, I am wondering how to display this in the
> beagle-search GUI ? Without cluttering the interface.

Well; I suggest showing results like in that link :-)

Keep a flat list of all results that can then be refined by clicking
on the clusters.

Cheers,
Mikkel

[1]: I can provide a pointer to the code if anybody wants to look at
it. It is Java though.