Re: [Tracker] Re-index/re-scan on each restart?

[Martyn Russell <martyn lanedo com>]

On 02/09/10 04:41, Michael Steiner wrote:
> Following additional observations after having enabled increased
> logging and doing some work towards using 0.9.  Assuming that the
> information below might be too much from the mailing list, in
> particular with attached logs, i send it only to you. Tell me if i
> better should keep the ML in the loop.

I think it is fine to CC the ML.

> As mentioned, i did change the log-files, all to at least verbosity=1
> and the miner one to 3, but didn't immediately restart.
>
> Some parallel work and a suspend/resume later, i noticed that
> tracker-store had crashed but unfortunately,

Ah, that's a shame, if you can reproduce it would like to see the crash 
of course.

> i forgot to copy the
> log-file and my subsequent tracker-control --start removed it, so i
> can't tell you what the exact exception was.  Unfortunately, i
> realized this only later so even a subsequent Tracker-Critical timeout
> exception in the miner seemingly caused by a suspend/resume cycle got
> lost due to a crash where my machine got unresponsive due to too
> probably much IO activity swapping out X or alike (tracker contributed
> to that but there were other jobs like a huge recursive diff which
> might have contributed).

Looking at the logs would help understand why this happens definitely. 
It's not something we have had reported so far as I recall.

> Anyway, after the corresponding restart, i noticed that
> tracker-miner-fs had strange errros, namely on
>
>     01 Sep 2010, 20:05:46: Tracker-Critical **: Could not obtain the mounted volumes: Method "SparqlQuery" with signature 
> "s" on interface "org.freedesktop.Tracker1.Resources" doesn't exist
>
> and tons
>
>     01 Sep 2010, 20:05:46: Tracker-Critical **: Could not execute cache query: Method "SparqlQuery" with signature 
> "s" on interface "org.freedesktop.Tracker1.Resour
> ces" doesn't exist
>
> Thinking that his was maybe related to the crash, i stopped tracker
> with
>
>    tracker-control --terminate
>
> save logs as logs.saved/1 (see attached) and restarted with
>
>    tracker-control --start
>
>
> Now i noticed now plenty of logs in tracker-store:  First about 250 of the form
>
>     01 Sep 2010, 21:06:39: Tracker-Critical **: Creating SQL table failed: table "XYZ" already exists

Generally, SQL errors about tables or database structure are caused by 
the database not being correctly set up. This can happen for a number of 
reasons and usually a reindex is required.

> and then about 2400 of the form
>
>     01 Sep 2010, 21:06:39: Tracker-Critical **: constraint failed

Yea, these are all related to the same problem. Incorrectly set up 
database, possibly caused by corruption.

> in tracker-miner-fs, i see DBUS timeout errors and again tons of the
>
>     Tracker-Critical **: Could not execute sparql: Unable to insert multiple values for ...

Yea, that's another side effect.

> At that point, i've stopped tracker again with --terminate to save the
> the log-files (in logs.saved/2). As mentioend this was vanilla tracker
> 0.8.16 compiled/installed as
>
>       wget http://ftp.gnome.org/pub/GNOME/sources/tracker/0.8/tracker-0.8.16.tar.bz2
>       tar -jxf tracker-0.8.16.tar.bz2
>       cd tracker-0.8.16
>       ./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var # default from README
>       make
>       cd `/bin/pwd` # some tests are picky ...
>       vi ./tests/libtracker-miner/Makefile
>       # disable in TEST_PROGS ``tracker-password-provider-test'' test which doesn't run ..
>       make check # fails in middle but ignore
>       make install
>
>
>
> In parallel, i was also upgrading the build tools for 0.9/git master
> which were not available out-of-the-box with ubuntu (turned out it was
> only vala 0.9.7 instead of 0.8.0, sqlite3 3.7.2 instead of 3.6.22 plus
> some dependencies to build them, although, all of them in ubuntu
> repos) and then build it.

Yes, we are quite close to bleeding edge requirements for master/0.9.

:)

> - i had some problems first with the configuration of the
>    video-extractor: the config resulted in a '?' and the compile failed
>    with some mplayer problem, probably related to me having mplayer
>    libraries but not the mplayer binary itself. Installing gstreamer
>    dev libraries and --enable-video-extractor=gstreamer worked around
>    that problem

Yes, I know how to fix this. I am going to do that today.

> - i also have to explicitly say --disable-miner-evolution as
>    auto-detection didn't seem to work properly

Oh?

> after that a make worked fine. In make check, i run again into the
> known password-provider-test problem. So i disabled it in
> ./tests/libtracker-miner/Makefile but contrary to 0.8 where the rest
> of checks worked fine, i got following problem
>
>     make[3]: Entering directory 
> `/home/msteiner/no-backup/packages/system/tracker/git/tracker/tests/tracker-steroids'
>     TEST: tracker-test... (pid=25302)
>       /steroids/tracker/tracker_sparql_query_iterate:                      **
>     Tracker:ERROR:tracker-test.c:180:query_and_compare_results: assertion failed: 
> (!tracker_sparql_cursor_next (cursor_fd, NULL, NULL))
>     FAIL
>     GTester: last random seed: R02S44097a348efd74202bfa039bb84cd5b6
>     /bin/sh: line 1: 25301 Terminated              gtester --verbose tracker-test
>
>
> Interestingly, when i tried subsequently the 0.9.18 release version, i
> also run into the sparql_query_iterate problem. However, for 0.9.18
> the tracker-password-provider-test worked?! (maybe some regression
> problem?)

Oh, this should work, I fixed a number of issues in this area recently.

> I didn't install it yet for two reasons:
>
> (1) i didn't know whether you wanted me to do some additional tests
>      with 0.8, e.g., to restart from scratch with tracker-control
>      --soft-reset or --hard-reset

Yes, I think so given you database errors.

> (2) i do not really want to firmly install test software into the main
>      system. So far i've installed new sqlite3, valac and tracker 0.9
>      in separate test directories.  However, the problem i face is i
>      have no idea how (and whether at all) i can start a dbus/gnome
>      application from a non-standard directory.  Are there some
>      environment variables (and maybe some dbus commands) to
>      (temporarily) allow tracker to run without having to be installed
>      into the main system/standard paths?  Clearly, just defining
>      LD_LIBRARY_PATH and alike is not sufficient.

There are quite some environment variables you can use with Tracker, the 
man pages cover a lot of them, but generally, greping the code base for 
getenv is another way to find out.

Usually installing to /usr/local is good enough here, I do have git Vala 
installed there (not packages), for SQLite I use 3.7.x packages and I am 
still using old d-bus (< 1.3.1) - new d-bus works for the other guys in 
the team though, I believe Aleksander has a PPA set up for that somewhere.

> Anyway, i hope above information give some useful feedback and let me
> know what additional debugging i can/should do.

I think just rerunning with tracker-control -rs should be enough.

--
Regards,
Martyn