Re: [Tracker] Excessive memory use by tracker-store

[Philip Van Hoof <spam pvanhoof be>]

On Sun, 2010-05-16 at 10:43 -0400, Calvin Walton wrote:
> On Sun, 2010-05-16 at 10:27 -0400, Calvin Walton wrote:
> > On Sun, 2010-05-16 at 13:37 +0200, Philip Van Hoof wrote:
> > > On Sun, 2010-05-02 at 22:11 -0400, Calvin Walton wrote:
> > > > In the GNOME System Monitor, tracker-store is currently reported as
> > > > using 9.9 GB of memory.
> > >
> > > We recently found a memory leak. Can you retry your experiments using
> > > the most recent commit in master?
>
> > Looks like the development branch fixes the problems I was having.
> > tracker-store's memory usage is insignificant (~14MB RSS, growing by
> > 100k every 5 minutes or so), CPU usage is lower (previously
> > tracker-store was pegging a CPU at 100%), and indexing speed is up.
> >
> > I'll let it run until completion to see if anything else pops up, but
> > it's looking good so far.
>
> Well, tracker-store appears to have been fixed, but now tracker-extract
> is acting up. Within 20 minutes of starting indexing, it was up to 16GB
> virtual (10GB RSS) memory usage) and was swapping out all my apps, I had
> to kill it to get my desktop responsive again.

OK, tracker-extract is not as bad as that process can be shut down
periodically, allowing the kernel to free up all resources each time.

Of course ain't tracker-extract being shut down periodically yet.

However, the proper fix is of course to fix the issue.

It's likely that there's a serious memory leak or serious memory
fragmentation in one of the extractor modules.

To find which one and where the leak is, it would be very useful if
you'd do something along the lines of this:

tracker-control -r

export G_SLICE=always-malloc
export G_DEBUG=gc-friendly

[Terminal 1]
tracker-store

[Terminal 2]
valgrind --tool=massif tracker-extract

[Terminal 3 - shortly after Terminal 2]
tracker-miner-fs

(note that the processes are in libexec, you might have to put the
libexec dir of your system, usually /usr/lib/tracker-0.9/, in your PATH)

> Any new ideas?

Once you have reached the excessive memory usage (hard to know, because
valgrind adds a significant overhead itself too), CTRL+C Terminal 2 and
look for a logfile. Compress this logfile and make it available for us
somehow (note, IRC channel is #tracker on GimpNET).


Cheers,

Philip

-- 


Philip Van Hoof
freelance software developer
Codeminded BVBA - http://codeminded.be