Re: [Tracker] indexer-split branch missing lots of important stuff

[Jamie McCracken <jamie mccrack googlemail com>]

On Mon, 2008-07-28 at 10:33 +0100, Martyn Russell wrote:
> Jamie McCracken wrote:
> > Compared to trunk, an awful lot of essential functionality has been
> > removed in the indexer-split.
>
> Yes, a lot of stuff had to be disabled to make the split possible.
>
> > Apologies if they are elsewhere and I have missed them in the code but
> > if not these need to be restored form trunk before i give approval for
> > merge
>
> We understand.
>
> > These are:
> >
> > 1) temp blacklisting of files that change frequently - see revision 1149
> > and 1169 in trunk
>
> Easily readded. I think it makes sense to have this black list in the
> tracker-monitor.c file so it can check each time we get a monitor event.
>
> Jamie as I recall, this was a per-session black list right, not a saved
> database black list. If so, that makes our job a lot easier.
>
> > 2) low disk space limit (cant see this used anywhere)
>
> Yes, I looked into this last week, I honestly wasn't sure how it was
> best to do this. The Indexer is the one populating disk space, I wonder
> if we should just add something in there? At the same time, it would be
> nice if the daemon knew why it wasn't indexing any more. So perhaps the
> daemon should just monitor the disk usage and disable the indexing until
> there is more space available.

yes best done in daemon

> > 3) stop words in tracker-parser.c - why did you remove this?
>
> If you mean tracker->stop_words then it wasn't used as far as I can see.
> If you mean tracker-parser.c, it is now in src/libtracker-common.

http://svn.gnome.org/viewvc/tracker/trunk/src/trackerd/tracker-parser.c?view=markup

this uses stop words - the one in libtracker-common does not

pls make sure you load the appropriate stop word list for the stemming
langauge - see trunk for this

> > 4) index merging - where did this go? Without this indexing wont scale
> > up (its slow updating a large index) + you will suffer from
> > fragmentation and wasted disk space due to hashtable resize relocations
> > (these are never recovered) - merging solves both these issues
>
> I will look into this.
>
> > 5) evolution on startup should scan any changed summary files for new
> > junk or deleted items and remove them from index - see trunk for this
>
> I will look into this.
>
> Thanks for your quick review Jamie.

np - im on holiday next few days but will look to review again next
weekend

jamie
>