Re: [Tracker] status of 0.6.90?

[Martyn Russell <martyn imendio com>]

Jamie McCracken wrote:
> On Mon, 2008-10-13 at 12:43 +0100, Martyn Russell wrote:
> > Jamie McCracken wrote:
> > > There are also a load of other issues that need correcting:
> > >
> > > 1) enumerating and crawling directories needs to be done in the indexer
> > > (and pass directories to watch back to the daemon). Daemon can then run
> > > as nice 0 and normal ionice instead of nice 19 as only cpu/io heavy ops
> > > will be searches and queries which need to be fast as possible
> > >
> > > 2) indexing needs to do all files in a directory before indexing the
> > > directory itself to prevent files not getting indexed if daemon is
> > > stopped in mid-index
> > >
> > > 3) Needs to be fully backward compatible with API and config options.
> > > Its likely that we will have to force a reindex for 0.6.6 as Im not sure
> > > db will work with old versions. I would rather have sqlite fts and xesam
> > > db support as well as flattened tables in to prevent the next few
> > > versions forcing a reindex after each upgrade if that is the case (IE I
> > > would rather do the reindex once if possible)
> > >
> > > 4) Db connections should use  sqlite3_soft_heap_limit call to limit heap
> > > usage of sqlite (sqlite will runaway and eat memory while indexing if
> > > you dont - these are not leaks and will not show up in valgrind!)
> > >
> > > see http://sqlite.org/c3ref/soft_heap_limit.html and note following:
> > >
> > > A negative or zero value for N in a call to sqlite3_soft_heap_limit(N)
> > > means that there is no soft heap limit and sqlite3_release_memory() will
> > > only be called when memory is completely exhausted. The default value
> > > for the soft heap limit is zero. 
> > >
> > > Ergo sqlite will happily eat all memory until you run out before
> > > attempting to free a single byte unless we set a value for above
> > >
> > > 5) probably some fine tuning and default settings for throttle might
> > > need to be adjusted
> > >
> > > 6) indexing email attachments?  Might be a regression if we dont as
> > > 0.6.6 did. I need to think how this fits in with xesam though as we dont
> > > want to add them to email or files index as they are of different source
> > > (probably we will have a separate index/db for each source -
> > > archive,attachment et al as they are not files or emails as such)
> > >
> > > 7) email optimisations - really slow for large mboxes. mbox could be
> > > optimised to store last known offset and record details  to prevent full
> > > scan when new emails are appended. needs to be done smartly (IE verify
> > > last record structure and UID at known offset) so we dont screw up if
> > > mbox was compacted or changed beyond recognition. i will likely restore
> > > the junkemail table to speed up junk checking for mbox too.
> > >
> > > I will try and do 4, 5 and 7 above over the weekend. I think martyn said
> > > he will do 1,2,3 soon.
> > Hi :)
> >
> > Jamie, can you update us with the status of SQLite FTS3 support? The
> > reason I ask is we want to really get the MMC support working nicely so
> > we enable/disable hits based on the MMC being mounted or not and of
> > course this depends on QDBM which holds statistics for the data we are
> > searching.
>
> I hope to have fts ready in a week or two (only have time at weekends
> atm due to work)
>
> > Do you have any ideas on how to do this too? I am guessing the way to do
> > it is to hinge everything on the Enable column in the services table and
> > link that to the search we do for _get_all_hits() in the new SQLite QDBM
> > replacement DB.
>
> we have a volumes table that can store HAL UID against mounted path
>
> each file can have a volume ID in the services table (I think its called
> auxilaryID)
>
> the idea is that we do a file move if volume is mounted against a new
> path otherwise enabled = 0 for each file with that volumeID if its not
> present
>
> never got round to getting this to work in trunk but should be straight forward to do (i think)

Yea it looks straight forward.

One question I have is, do we really need the "enabled" column if it
only pertains to the Volumes table - can we not just have 1 column and 1
row in the Volumes table for "enabled" and do some nice SQL to know if
content is enabled based on purely the auxiliary ID in the Services
table. Or is the "enabled" column needed for other things.

Something else I have noticed. When it comes to deleting a file, it is
really quite nasty because we have to delete in several places. It would
be much nicer to delete the service id and that trigger deletes for
every other place that relationally depends on it (like
content/metadata/etc).

-- 
Regards,
Martyn