[Tracker] FTS4 branch review

[Martyn Russell <martyn lanedo com>]

Hello all,

So Carlos recently finished the fts4 branch for review. For those who 
don't know what this is, there is a nice blog from Carlos here:

http://blogs.gnome.org/carlosg/2013/01/28/snippets-in-trackers-full-text-search-results/

Briefly, this branch is about updating the Full Text Search support from 
version 3 to version 4. Upstream SQLite provides a module and we've 
always used that and adapted it for our needs. There are 2 things which 
make this branch different to any other upstream syncing we've done in 
the past:

1. We can now use the upstream FTS4 with no changes from our side.
2. We now have support for fts:snippet() and fts:offset() properly (and 
with multivalued properties I believe which we didn't have before).

I've been playing with this and I've tested it a lot. It seems to work 
quite well. I've not had a chance to do any performance reviews yet.

There are some additions I would like to put in place over time after 
this branch is in master, namely improving tracker-needle to *highlight* 
the text we found. You would think this was easy, but the more you get 
into it, the more you realise it's a quick hack. Mostly this is because 
there is escaping, so using fts:offsets() would make more sense, but 
then you can see offsets for ALL properties and you don't need 
nie:title's offsets if you're showing that already, etc, etc. So it 
needs some thinking. There is already the possibility to use 
fts:snippet('foo','<b>','</b>') in SPARQL, but this also has issues in 
tracker-needle, (in that we escape all text first).

That aside, i've actually put my updates into the branch already. I 
didn't see anything fundamentally wrong. A large chunk of the changes is 
to the embedded fts sources that we added a long time ago.

Jürg, given you were quite involved in this work a while back, I wanted 
to check if you had any comment or wanted to review this before it hit 
master?

The branch is here:

  http://git.gnome.org/browse/tracker/log/?h=fts4

Thanks again for the work here Carlos,

--
Regards,
Martyn

Founder and CEO of Lanedo GmbH.