Re: [Tracker] Re-index/re-scan on each restart?

On 02/09/10 04:41, Michael Steiner wrote:
Following additional observations after having enabled increased
logging and doing some work towards using 0.9.  Assuming that the
information below might be too much from the mailing list, in
particular with attached logs, i send it only to you. Tell me if i
better should keep the ML in the loop.

I think it is fine to CC the ML.

As mentioned, i did change the log-files, all to at least verbosity=1
and the miner one to 3, but didn't immediately restart.

Some parallel work and a suspend/resume later, i noticed that
tracker-store had crashed but unfortunately,

Ah, that's a shame, if you can reproduce it would like to see the crash of course.

i forgot to copy the
log-file and my subsequent tracker-control --start removed it, so i
can't tell you what the exact exception was.  Unfortunately, i
realized this only later so even a subsequent Tracker-Critical timeout
exception in the miner seemingly caused by a suspend/resume cycle got
lost due to a crash where my machine got unresponsive due to too
probably much IO activity swapping out X or alike (tracker contributed
to that but there were other jobs like a huge recursive diff which
might have contributed).

Looking at the logs would help understand why this happens definitely. It's not something we have had reported so far as I recall.

Anyway, after the corresponding restart, i noticed that
tracker-miner-fs had strange errros, namely on

    01 Sep 2010, 20:05:46: Tracker-Critical **: Could not obtain the mounted volumes: Method "SparqlQuery" with signature 
"s" on interface "org.freedesktop.Tracker1.Resources" doesn't exist

and tons

    01 Sep 2010, 20:05:46: Tracker-Critical **: Could not execute cache query: Method "SparqlQuery" with signature 
"s" on interface "org.freedesktop.Tracker1.Resour
ces" doesn't exist

Thinking that his was maybe related to the crash, i stopped tracker

   tracker-control --terminate

save logs as logs.saved/1 (see attached) and restarted with

   tracker-control --start

Now i noticed now plenty of logs in tracker-store:  First about 250 of the form

    01 Sep 2010, 21:06:39: Tracker-Critical **: Creating SQL table failed: table "XYZ" already exists

Generally, SQL errors about tables or database structure are caused by the database not being correctly set up. This can happen for a number of reasons and usually a reindex is required.

and then about 2400 of the form

    01 Sep 2010, 21:06:39: Tracker-Critical **: constraint failed

Yea, these are all related to the same problem. Incorrectly set up database, possibly caused by corruption.

in tracker-miner-fs, i see DBUS timeout errors and again tons of the

    Tracker-Critical **: Could not execute sparql: Unable to insert multiple values for ...

Yea, that's another side effect.

At that point, i've stopped tracker again with --terminate to save the
the log-files (in logs.saved/2). As mentioend this was vanilla tracker
0.8.16 compiled/installed as

      tar -jxf tracker-0.8.16.tar.bz2
      cd tracker-0.8.16
      ./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var # default from README
      cd `/bin/pwd` # some tests are picky ...
      vi ./tests/libtracker-miner/Makefile
      # disable in TEST_PROGS ``tracker-password-provider-test'' test which doesn't run ..
      make check # fails in middle but ignore
      make install

In parallel, i was also upgrading the build tools for 0.9/git master
which were not available out-of-the-box with ubuntu (turned out it was
only vala 0.9.7 instead of 0.8.0, sqlite3 3.7.2 instead of 3.6.22 plus
some dependencies to build them, although, all of them in ubuntu
repos) and then build it.

Yes, we are quite close to bleeding edge requirements for master/0.9.


- i had some problems first with the configuration of the
   video-extractor: the config resulted in a '?' and the compile failed
   with some mplayer problem, probably related to me having mplayer
   libraries but not the mplayer binary itself. Installing gstreamer
   dev libraries and --enable-video-extractor=gstreamer worked around
   that problem

Yes, I know how to fix this. I am going to do that today.

- i also have to explicitly say --disable-miner-evolution as
   auto-detection didn't seem to work properly


after that a make worked fine. In make check, i run again into the
known password-provider-test problem. So i disabled it in
./tests/libtracker-miner/Makefile but contrary to 0.8 where the rest
of checks worked fine, i got following problem

    make[3]: Entering directory 
    TEST: tracker-test... (pid=25302)
      /steroids/tracker/tracker_sparql_query_iterate:                      **
    Tracker:ERROR:tracker-test.c:180:query_and_compare_results: assertion failed: 
(!tracker_sparql_cursor_next (cursor_fd, NULL, NULL))
    GTester: last random seed: R02S44097a348efd74202bfa039bb84cd5b6
    /bin/sh: line 1: 25301 Terminated              gtester --verbose tracker-test

Interestingly, when i tried subsequently the 0.9.18 release version, i
also run into the sparql_query_iterate problem. However, for 0.9.18
the tracker-password-provider-test worked?! (maybe some regression

Oh, this should work, I fixed a number of issues in this area recently.

I didn't install it yet for two reasons:

(1) i didn't know whether you wanted me to do some additional tests
     with 0.8, e.g., to restart from scratch with tracker-control
     --soft-reset or --hard-reset

Yes, I think so given you database errors.

(2) i do not really want to firmly install test software into the main
     system. So far i've installed new sqlite3, valac and tracker 0.9
     in separate test directories.  However, the problem i face is i
     have no idea how (and whether at all) i can start a dbus/gnome
     application from a non-standard directory.  Are there some
     environment variables (and maybe some dbus commands) to
     (temporarily) allow tracker to run without having to be installed
     into the main system/standard paths?  Clearly, just defining
     LD_LIBRARY_PATH and alike is not sufficient.

There are quite some environment variables you can use with Tracker, the man pages cover a lot of them, but generally, greping the code base for getenv is another way to find out.

Usually installing to /usr/local is good enough here, I do have git Vala installed there (not packages), for SQLite I use 3.7.x packages and I am still using old d-bus (< 1.3.1) - new d-bus works for the other guys in the team though, I believe Aleksander has a PPA set up for that somewhere.

Anyway, i hope above information give some useful feedback and let me
know what additional debugging i can/should do.

I think just rerunning with tracker-control -rs should be enough.


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]