Re: [Tracker] miner-fs: Placing monitors on directories takes way too much time

[Martyn Russell <martyn lanedo com>]

On 02/09/10 10:52, Mildred Ki'Lya wrote:
> Hi,

Hi,

> I decided I wanted to monitor recursively my whole $HOME directory, and
> it takes more than two hours for tracker-miner-fs to place monitors on
> each directory recursively, until I made it stop.

Which version of Tracker are you using? There was a bug fixed in this 
area recently.

> Is that normal?

No.

> At 11:30 I was fed up and decided to stop it from indexing $HOME and
> start index where I am working on my tracker extractor plugin (where I
> have example files I want tracked).
>
> I can understand it might take some time, considering I have many
> folders and many subfolders, way too many git repositories and old
> projects, but still ... (only 72 git repos)
>
> It takes approximately 2 seconds for tracker to place a monitor on a
> directory and do what needs to be done within, I thought it could be
> quicker.

That's not normal either.

> Attached, a snip of my tracker-fs log file

Is this a first time index? It looks like it.

> For a smaller hierarchy, where there are only 334 directrories (229 from
> tracker-miner-fs log) and 3688 files it took 5 minutes. I have a
> checkout of the tracker git repository, and perhaps two other git
> repositories (libtinymail and camel-lite which was part of
> evolution-data-server)

At the moment, there is a huge difference between 0.8 and 0.9 and 
depending on what technologies you use (without considering your 
hardware or content being indexed). For example:

  DBus > 1.3.1?
  SQLite > 3.6?
  libunistring/libicu/glib unicode support?
  D-Bus/direct access use?
  ...

Any one of these changes can make a huge difference to performance.


For me, an initial index takes:

  Finished mining in seconds:303.820123, total directories:2224, total 
files:19550

For crawling subsequent times takes:

  Finished mining in seconds:26.631053, total directories:2224, total 
files:19550

That's over n partitions, some USB disks too, etc. with D-Bus < 1.3.1, 
SQLite 3.7, libunistring and direct-access used in part (in 0.9/master).

--
Regards,
Martyn