Re: [Tracker] Excessive memory use by tracker-store

[Philip Van Hoof <spam pvanhoof be>]

On Sun, 2010-05-16 at 10:27 -0400, Calvin Walton wrote:
> On Sun, 2010-05-16 at 13:37 +0200, Philip Van Hoof wrote:
> > On Sun, 2010-05-02 at 22:11 -0400, Calvin Walton wrote:
> > > In the GNOME System Monitor, tracker-store is currently reported as
> > > using 9.9 GB of memory.
> >
> > We recently found a memory leak. Can you retry your experiments using
> > the most recent commit in master?
>
> I've been using the stable releases, so master could be a big jump, but
> sure; I'll try it out.

OK, let us know

> > Also, please give us what cat /proc/`pidof tracker-store`/status | grep
> > VmRSS gives you. GNOME System Monitor isn't really a very accurate tool
> > for this (although +9GB sure sounds bad still).
>
> I double-checked using VmRSS now; it is showing pretty much the same
> picture. (Checking on my laptop, with the same tracker version 0.8.6)
> VmRSS rises by about 1MB/s until tracker-store starts swapping heavily;
> I killed it on my laptop once VmRSS hit 2GB (at about 50% through
> indexing a rather smaller /home).

OK

> <after installing and testing tracker git master...>
>
> Looks like the development branch fixes the problems I was having.
> tracker-store's memory usage is insignificant (~14MB RSS, growing by
> 100k every 5 minutes or so), CPU usage is lower (previously
> tracker-store was pegging a CPU at 100%), and indexing speed is up.

OK, these are acceptable numbers indeed.

> I'll let it run until completion to see if anything else pops up, but
> it's looking good so far.
>
> Are there plans to backport these fixes onto the 0.8 branch, or should I
> attempt to pressure my distro into packaging the 0.9 branch?

There have been some issues with what we implemented in 0.9 that might
or is related to the fixes responsible for the memory improvements.

In particular the concurrent query support.

Indexing speed is up due to an improvement for the query to get the
modification times of files. In your case is this query in a non-linear
way becoming worse as the amount of resources increases (and so does its
memory consumption non-linearly increase). This has been fixed in
master, and is likely portable (as soon as coping with ontology changes,
property changes, is backported).

We are still in the process of testing these improvements and getting
them to behave "stable" (= several weeks of testing without a problem).
Eventually it's likely that they'll be backported to 0.8, but short term
(less then a month) it's not very likely.

For now our own advise for distributions is still to continue using 0.8.

To summarize, scalability to terrabytes of data is being addressed, and
will eventually hit a stable branch.


Cheers,


Philip

-- 


Philip Van Hoof
freelance software developer
Codeminded BVBA - http://codeminded.be