Re: GSoC Weekly Report

[Debajyoti Bera <dbera web gmail com>]

> completely sure that such a loose typing system will greatly benefit
> us when working with TEXT/STRING types, however, the gzipped blobs
> might benefit from less disk usage thanks to being stored in a single
> file, in addition, I know that incremental i/o is a possibility with
> blobs in sqlite 3.4, which could potentially be utilized to optimize
> work like this.
>
> Anyways, please send a patch to the list if thats not too much to ask,
> or just give us an update as to how things are going.

I and Arun had some discussion about this and we were trying to balance the 
performance and size issues. He already has the sqlite-idea implemented; 
however I would also like to see how a hybrid idea works i.e. store the huge 
number of extremely small files in sqlite and store the really large ones on 
the disk. Implementing this is tricky (*).

- dBera

(*) One of my recent efforts has been to add language detection support (based 
on a patch in bugzilla). This will enable us to use the right stemmers and 
analyzers depending on the language. The hard part is stealing some initial 
text for language detection and doing it in a transparent way. Incidentally, 
one implementation of the hybird approach mentioned above and the language 
detection crosses path. I am waiting for some free time to get going after 
them.

-- 
-----------------------------------------------------
Debajyoti Bera @ http://dtecht.blogspot.com
beagle / KDE fan
Mandriva / Inspiron-1100 user