Re: [Tracker] miner-fs: Placing monitors on directories takes way too much time

[Martyn Russell <martyn lanedo com>]

On 03/09/10 03:10, Chen, Zhenqiang wrote:
> But my test shows tracker related modules take ~150% CPU, especially
> tracker-store.
>
> Test HW: ASPIREone (netbook)  SW: MeeGo, tracker-0.9.16, sqlite-3.6.
>
> tracker-control -r tracker-control -s top

You can use tracker-control -rs btw ;)

> tracker-store: ~95% dbus-daemon:   ~22% tracker-miner-fs: ~18%
> tracker-extractor: ~12%

The store is expected to take almost ALL the time here as it does the 
most IO operations.

That sounds high for d-bus, are you using d-bus >= 1.3.1? That should 
improve performance here.

> As I understand most time consuming work are IO opertions, which
> should not take much CPU. As comparison, I cp all the files, it takes
> only ~30% CPU.

Sure, but there's a difference between copying data which is quite a 
straight forward operation and extracting data and putting it into a 
database where relationships between the data are described. Not to 
mention the redundancy we add in case there is corruption, etc, etc.

> The data for dbus-daemon, tracker-miner-fs and tracker-extractor seam
> reasonable. Do you think it is reasonable for tracker-store?

Totally. It has the most work to do.

> Profiles show "sqlite3_step" indirectly called in "update_sparql"
> takes most of the time.

I would expect that.

> Any comments on how to improve it?

Are you using glib unicode parsing? We changed configure to require 
!glib yesterday, the indexing performance difference is incredible, 
solved with:

  sudo apt-get install libunistring-dev
  ./autogen.sh

Upgrading to D-Bus 3.7 and getting the latest 0.9.x release will make 
quite a difference too (with direct-access in there too the miner-fs 
communication is faster) and signals emitted are more efficient too 
since yesterday's release.

> Thanks! -Zhenqiang

No problem.

--
Regards,
Martyn