Re: [Tracker] status of 0.6.90?

[Jamie McCracken <jamie mccrack googlemail com>]

On Mon, 2008-10-13 at 12:43 +0100, Martyn Russell wrote:
> Jamie McCracken wrote:
> > There are also a load of other issues that need correcting:
> >
> > 1) enumerating and crawling directories needs to be done in the indexer
> > (and pass directories to watch back to the daemon). Daemon can then run
> > as nice 0 and normal ionice instead of nice 19 as only cpu/io heavy ops
> > will be searches and queries which need to be fast as possible
> >
> > 2) indexing needs to do all files in a directory before indexing the
> > directory itself to prevent files not getting indexed if daemon is
> > stopped in mid-index
> >
> > 3) Needs to be fully backward compatible with API and config options.
> > Its likely that we will have to force a reindex for 0.6.6 as Im not sure
> > db will work with old versions. I would rather have sqlite fts and xesam
> > db support as well as flattened tables in to prevent the next few
> > versions forcing a reindex after each upgrade if that is the case (IE I
> > would rather do the reindex once if possible)
> >
> > 4) Db connections should use  sqlite3_soft_heap_limit call to limit heap
> > usage of sqlite (sqlite will runaway and eat memory while indexing if
> > you dont - these are not leaks and will not show up in valgrind!)
> >
> > see http://sqlite.org/c3ref/soft_heap_limit.html and note following:
> >
> > A negative or zero value for N in a call to sqlite3_soft_heap_limit(N)
> > means that there is no soft heap limit and sqlite3_release_memory() will
> > only be called when memory is completely exhausted. The default value
> > for the soft heap limit is zero. 
> >
> > Ergo sqlite will happily eat all memory until you run out before
> > attempting to free a single byte unless we set a value for above
> >
> > 5) probably some fine tuning and default settings for throttle might
> > need to be adjusted
> >
> > 6) indexing email attachments?  Might be a regression if we dont as
> > 0.6.6 did. I need to think how this fits in with xesam though as we dont
> > want to add them to email or files index as they are of different source
> > (probably we will have a separate index/db for each source -
> > archive,attachment et al as they are not files or emails as such)
> >
> > 7) email optimisations - really slow for large mboxes. mbox could be
> > optimised to store last known offset and record details  to prevent full
> > scan when new emails are appended. needs to be done smartly (IE verify
> > last record structure and UID at known offset) so we dont screw up if
> > mbox was compacted or changed beyond recognition. i will likely restore
> > the junkemail table to speed up junk checking for mbox too.
> >
> > I will try and do 4, 5 and 7 above over the weekend. I think martyn said
> > he will do 1,2,3 soon.
>
> Hi :)
>
> Jamie, can you update us with the status of SQLite FTS3 support? The
> reason I ask is we want to really get the MMC support working nicely so
> we enable/disable hits based on the MMC being mounted or not and of
> course this depends on QDBM which holds statistics for the data we are
> searching.

I hope to have fts ready in a week or two (only have time at weekends
atm due to work)

> Do you have any ideas on how to do this too? I am guessing the way to do
> it is to hinge everything on the Enable column in the services table and
> link that to the search we do for _get_all_hits() in the new SQLite QDBM
> replacement DB.

we have a volumes table that can store HAL UID against mounted path

each file can have a volume ID in the services table (I think its called
auxilaryID)

the idea is that we do a file move if volume is mounted against a new
path otherwise enabled = 0 for each file with that volumeID if its not
present

never got round to getting this to work in trunk but should be straight forward to do (i think)

jamie