Re: [Tracker] Re-index/re-scan on each restart?

[Michael Steiner <michisteiner verizon net>]

Martyn,

> > > > > On Thu, 02 Sep 2010 10:26:44 +0100, Martyn Russell <martyn lanedo com> said:

   MR> I see. We do 2 things here. We store the file and its mtime in the
   MR> database. We also use inotify to monitor changes during the day to day
   MR> running of the computer. When the miner-fs starts, it sets up these
   MR> monitors and checks mtimes against the database to make sure nothing
   MR> changed. If the mtime is different, we reindex a file or
   MR> directory. The inotify part is possibly one of the weakest parts of
   MR> Tracker right now, as it doesn't scale with large numbers of
   MR> directories and uses a lot of resources to work with any success
   MR> generally. With FANotify coming up in newer kernels, we are looking to
   MR> improve this situation in the near future.

Looking at the log-files i also started to realize that crawling is
necssary. Thanks for your detailed explanation which resolved any
remaining questiosn.



> > > > > On Thu, 02 Sep 2010 11:27:30 +0100, Martyn Russell <martyn lanedo com> said:


   MR> Looking at the logs would help understand why this happens
   MR> definitely. It's not something we have had reported so far as I
   MR> recall.

Yep, i try to save the logs (as i did for the latter problems i
mentioned for which i did attach the logfile).

However, i noticed now another issue with retaining log-files: It
seems tracker does truncate log-files (tracker-miner-fs) and/or delete
them on restarts, the latter particularly an issue as some processes
(e.g., tracker-extract) seem to get automatically restarted, probably
due to some watchdog. Or put differently, i did start yesterday night
a re-index run and this morning i noticed that the log-files didn't
cover the whole time but semed to be truncated from time to time.  I
looked in the wiki and in the config files but couldn't see any config
option related to log-file backups or log-file truncation.  What am i
supposed to do to not loose log info?


   >> - i also have to explicitly say --disable-miner-evolution as
   >> auto-detection didn't seem to work properly
   MR> 
   MR> Oh?

I did not have evolution dev libraries installed and configure aborted
complaining about missing evolution data server and evolution plugin
packages missing (rather just assuming --disable-miner-evolution as i
would have assumed).  Anyway, not really a big deal: i've installed
now the libraries and then it works



   >> (2) i do not really want to firmly install test software into the main
   >> system. So far i've installed new sqlite3, valac and tracker 0.9
   >> in separate test directories.  However, the problem i face is i
   >> have no idea how (and whether at all) i can start a dbus/gnome
   >> application from a non-standard directory.  Are there some
   >> environment variables (and maybe some dbus commands) to
   >> (temporarily) allow tracker to run without having to be installed
   >> into the main system/standard paths?  Clearly, just defining
   >> LD_LIBRARY_PATH and alike is not sufficient.
   MR> 
   MR> There are quite some environment variables you can use with Tracker,
   MR> the man pages cover a lot of them, but generally, greping the code
   MR> base for getenv is another way to find out.

My main concern was things like interaction with dbus-1 / bonobo/
nautilus which i assumed would read things like
../share/dbus-1/services/org.freedesktop.Tracker1* from the standard
file locations and wouldn't find them if i install tracker to
non-standard locations (at least that matches my experience that if i
install tracker via ./configure --prefix=foobar i can start tracker
tools but they complain about dbus not knowing certain services.


   MR> Usually installing to /usr/local is good enough here, 

In any case, i did uninstall now 0.8.16 (thanks for the make
uninstall!) from /usr and did install 0.8.17 into a version-specific
subdirectory via --prefix and then sym-linked the various
subdirectories to the appropriate place in /usr/local. That way i can
quickly experiment with different versions.

With 0.8.17 installed i did a --hard-reset and restarted indexing with
increased verbosity ...

   >> Anyway, i hope above information give some useful feedback and let me
   >> know what additional debugging i can/should do.

... to hopefully give you more feedback on the issues I had (assuming
they didn't disappear with 0.8.17). However, to really analyze the
log-files, there is still the truncation issue i managed above which
makes that a bit hard ... :-(


-michael-

FYI: the tracker-password-provider-test still failed in 0.8.17 but
does works now in git master (where it yesterday failed). git master,
though, still fails in tracker-test /
/steroids/tracker/tracker_sparql_query_iterate as yesterday.
Interestingly, 0.9.19 proceeds slightly further in test-tracker by
passing /steroids/tracker/tracker_sparql_query_iterate but failed
later in

  /steroids/tracker/tracker_sparql_query_iterate_async:                **
Tracker:ERROR:tracker-test.c:420:async_query_cb: assertion failed: (!tracker_sparql_cursor_next (cursor_glib, 
NULL, NULL))
FAIL
GTester: last random seed: R02Sa67dba50a911fff2bf1695c05b7722f1