Re: [Tracker] Controlling tracker priority and settings on tracked files

[Martyn Russell <martyn lanedo com>]

On 07/09/12 16:21, Richard Shephard wrote:
> Hi,

Hello Richard,

> I know that I have seen something like this before in the docs, I just
> can’t find it again.
>
> Our application is using tracker in more or less the standard
> configuration. However, due to processor consumption, we have reduced
> the miner indexing speed to 19, however, tracker-store still consumes
> 90+% of CPU power. Tracker-miner is reduced in priority, but not store.

If you're using Tracker >= 0.14.0, we use SCHED_IDLE to help here. We 
also have the throttle which is what you're referring to above. What 
version are you using?

> Is there a nice way of reducing tracker-store’s CPU usage? (other than
> renice-ing it from the command line)

Hmm, we could patch tracker-store if that's what you mean. Though it 
shouldn't be necessary because the traffic is driven from the miners 
unless you're adding to that with client requests (queries/updates)

We decided not to nice() the store because it is client driven and if 
you need the information fast from application A you don't want it 
restricted because of nice() due to application B being a constant hog.

> Also, I know that the miners can be told what directories and files to
> ignore, but is there a way of telling it what files only to include?

From the config, you can specify directories yes, but not files.

You can specify files to index by the dbus interface, but that's for 
applications usually. It sounds like this wouldn't be useful to you if 
you want to index all files of a specific type.

> i.e. we only need multimedia file types, so we only want to index files
> like mp4,mp3, etc. – not the entire array of possible files.

I see, that's an interesting approach to indexing.

The closest mechanism we have to that is:

  $prefix/share/tracker/extract-rules/*.rule

For example:

  /usr/share/tracker/extract-rules/10-mp3.rule

This allows you to specify which mime types will be covered by the 
extractor modules installed. This would certainly allow you to control 
the data extracted and could improve performance. This would work 
because only the MP3 extractor would ever be found for all files we 
check. So you would have file data in the database on a basic level 
(name, size, etc) but for MP3 files you would have full class 
information too.

To be able to specify mime types to explicitly allow would need some 
patching in Tracker but it wouldn't be too hard to add that.

> The “problem” we have is that one aspect of the project includes a SD
> card with a full Linux install on it. The code mounts this because it
> thinks it is media (and there well maybe), but tracker goes banana’s
> trying to index the full system. This makes the database file grow past
> 600MB. There is not 600MB worth of multimedia files to store, but mostly
> Linux fs style files we have no need of.

I see. Not long ago we disabled (by default) indexing removable devices 
when they're inserted. Depending on your version and config, this might 
be happening and could easily be turned off.

The GNOLME docs application wanted something similar here, they wanted 
to be able to index content on demand for removable devices because it's 
just too hard to generalise about when you need that content and when 
you don't.

If you always mount that media in the same place and you know where the 
media files are located, you could add that directory specifically to 
the config and remove indexing removable media in the preferences. That 
should work.

HTH,

--
Regards,
Martyn

Founder and CEO of Lanedo GmbH.