Re: [Tracker] Memory usage for tracker-extract and tracker-miner-fs

[Carlos Garnacho <carlosg gnome org>]

Hi Joe,

On Mon, Jan 18, 2016 at 2:02 AM, Joe Rhodes <lists joerhodes com> wrote:
> Carlos:
>
> I just tried the 1.7.2 tarball.  I think we may still have a leak.  The two processes were heading up over 
> 600 MB's.
>
> Here are the valgrind outputs.  I hope this helps:
>
> https://www.dropbox.com/s/gyndn7ithpge6nc/valgrind-tracker-extract-log-1.7.2.gz?dl=0
> https://www.dropbox.com/s/h250z3lh9mvr2vv/valgrind-tracker-miner-fs-log-1.7.2.gz?dl=0

As expected, the valgrind logs are a lot more favorable.
Tracker-extract reports:

==21759==    definitely lost: 168 bytes in 1 blocks
==21759==    indirectly lost: 6,046 bytes in 2 blocks

And those seem to be a memory pool in libjpeg that's mistakenly marked
as "lost". In tracker-miner-fs logs we see:

==21669==    definitely lost: 0 bytes in 0 blocks
==21669==    indirectly lost: 0 bytes in 0 blocks

So I think we're now fine there. Besides these, there's some memory
reported as "possibly lost", but that's the usual glib noise (more
specifically, GObject doesn't deinitialize registered classes/types).
I can also see some "conditional jump depends on uninitialized values"
warnings from the HTML extractor, which I can also reproduce, but need
some more investigation in libxml.

As for the memory used, I must confess 600MB in ps/top seem more
reasonable to me (as opposed to the several GBs you reported
initially). In tracker-miner-fs that sounds like a reasonable number
given your workload (and the fact that we need to keep an in-memory
representation of the directory tree). In tracker-extract sounds a bit
on the high side, but could be explained by memory fragmentation,
because tracker-extract may potentially initialize a lot of libraries
(that may also create their own runtime caches), and those don't/can't
get deinitialized after they are no longer used.

Although keep into account that top/ps is not the best tool to measure
memory usage. A more faithful number is given by the valgrind logs
themselves, eg. for your case in tracker-miner-fs it is:

==21669==    still reachable: 143,987,764 bytes in 3,356,013 blocks

So at the time of you hitting Ctrl-C, tracker-miner-fs had allocated
~140MB. That sounds indeed more like it, although I'm not sure if you
let tracker-miner-fs run in its entirety under valgrind (that is,
until all folders were indexed).

Cheers,
  Carlos