Re: [Tracker] status of 0.6.90?

[Jamie McCracken <jamie mccrack googlemail com>]

There are also a load of other issues that need correcting:

1) enumerating and crawling directories needs to be done in the indexer
(and pass directories to watch back to the daemon). Daemon can then run
as nice 0 and normal ionice instead of nice 19 as only cpu/io heavy ops
will be searches and queries which need to be fast as possible

2) indexing needs to do all files in a directory before indexing the
directory itself to prevent files not getting indexed if daemon is
stopped in mid-index

3) Needs to be fully backward compatible with API and config options.
Its likely that we will have to force a reindex for 0.6.6 as Im not sure
db will work with old versions. I would rather have sqlite fts and xesam
db support as well as flattened tables in to prevent the next few
versions forcing a reindex after each upgrade if that is the case (IE I
would rather do the reindex once if possible)

4) Db connections should use  sqlite3_soft_heap_limit call to limit heap
usage of sqlite (sqlite will runaway and eat memory while indexing if
you dont - these are not leaks and will not show up in valgrind!)

see http://sqlite.org/c3ref/soft_heap_limit.html and note following:

A negative or zero value for N in a call to sqlite3_soft_heap_limit(N)
means that there is no soft heap limit and sqlite3_release_memory() will
only be called when memory is completely exhausted. The default value
for the soft heap limit is zero. 

Ergo sqlite will happily eat all memory until you run out before
attempting to free a single byte unless we set a value for above

5) probably some fine tuning and default settings for throttle might
need to be adjusted

6) indexing email attachments?  Might be a regression if we dont as
0.6.6 did. I need to think how this fits in with xesam though as we dont
want to add them to email or files index as they are of different source
(probably we will have a separate index/db for each source -
archive,attachment et al as they are not files or emails as such)

7) email optimisations - really slow for large mboxes. mbox could be
optimised to store last known offset and record details  to prevent full
scan when new emails are appended. needs to be done smartly (IE verify
last record structure and UID at known offset) so we dont screw up if
mbox was compacted or changed beyond recognition. i will likely restore
the junkemail table to speed up junk checking for mbox too.

I will try and do 4, 5 and 7 above over the weekend. I think martyn said
he will do 1,2,3 soon.

jamie



On Sat, 2008-10-04 at 01:04 +0200, Michael Biebl wrote:
> 2008/10/3 Tshepang Lekhonkhobe <tshepang gmail com>:
> > Hi,
> >
> > It's been quite a while since 0.6.6 and wonder what's still left
> > before releasing, besides:
> > * TST searches killing trackerd and tracker-indexer
>
> FWIW, I can reproduce this bug. Imo this is the most important
> showstopper for me atm.
>
> > * memory leak in tracker-indexer
> > * tracker-preferences work (hopefully let it use tracker-config)
>
> In addition, I noticed that tracker-preferences tries to respawn
> /usr/bin/trackerd, but the trackerd binary is now in libexecdir.
>
> Other points:
> * config file of tracker-applet is ~/.config/trackertracker-applet.cfg
> * duplicated code in src/libtracker-common/tracker-config(uration).[ch]
> * API/ABI break in libtracker-gtk
> * The frequency of notifications via tracker-applet has increased a
> lot. I constantly get
> notifications that tracker has finished indexing, whenever a file
> changes in my home dir.
> IIRC in 0.6.6 I only got this notification after the initial indexing run.
>
>
> Cheers,
> Michael