Re: [Tracker] miner-fs: Placing monitors on directories takes way too much time

[Adrien Bustany <abustany gnome org>]

Le Thu, 9 Sep 2010 17:04:06 +0800,
"Chen, Zhenqiang" <zhenqiang chen intel com> a écrit :

>  
> > > I will try d-bus-1.3 and latest tracker.
> >
> > Opps, yea, I meant 1.3.1, not 3.7. Also, note, 1.3.0 is buggy so you 
> > will need 1.3.1. This should avoid quite some memory copies 
> > when indexing.
>
> I tried tracker git code (master, last update Sep 7). But test
> results show it is ~15% slower than tracker-0.9.16. And I tried
> d-bus-1.3.1. It did not help in my case. 
>
> dbus is one of the bottlenecks. Tests show 1000 continuous INSERTs
> will block the d-bus in my system. 
If you use DBus 1.3.1 or higher (1.3 is broken), you'll benefit from a
new update mechanism that greatly offloads the load on DBus, and makes
things much (much) faster. And if you combine that with SQLite unstable
(3.7+), you also get direct access, which makes allows a new query
mechanism that bypasses DBus totally.

> To reduce the dbus overhead, I tried to group the UPDATE for files of
> a dir into one update. Tests show it is ~2X faster. Here are the
> logs: (Notes: among the 10693 files, 10654 files are photos.) 
>
> tracker git master:
> Finished mining in seconds:232.713859, total directories:354, total
> files:10693
>
> tracker-0.9.16
> Finished mining in seconds:200.689239, total directories:354, total
> files:10693
>
> tracker-0.9.16 with group update (testing code segment is attached.)
> Finished mining in seconds:92.813985, total directories:354, total
> files:10693
>
> What do you think about the idea "grouping the updates for files of
> one dir into one update"? 
>
> Thanks!
> -Zhenqiang