Re: [Tracker] Memory usage for tracker-extract and tracker-miner-fs

[Joe Rhodes <lists joerhodes com>]

Indeed.  I was mistaken about the memory usage.   Tracker-miner-fs has now stabilized at about 265 MB of RAM. 
 I force a re-crawl once every two hours to update the index.

I've disabled the content searching for now by removing all the rules.  Nothing to do with Tracker.  It just 
took too long for the search results to populate on the Mac clients over Netatalk.  In this case, I *know* 
the bottleneck is in Netatalk.

So for my purposes, 1.7.2 is looking great.   Thanks!

Cheers!
-Joe Rhodes

> On Jan 18, 2016, at 6:57 AM, Carlos Garnacho <carlosg gnome org> wrote:
>
> Hi Joe,
>
> On Mon, Jan 18, 2016 at 2:02 AM, Joe Rhodes <lists joerhodes com> wrote:
> > Carlos:
> >
> > I just tried the 1.7.2 tarball.  I think we may still have a leak.  The two processes were heading up over 
> > 600 MB's.
> >
> > Here are the valgrind outputs.  I hope this helps:
> >
> > https://www.dropbox.com/s/gyndn7ithpge6nc/valgrind-tracker-extract-log-1.7.2.gz?dl=0
> > https://www.dropbox.com/s/h250z3lh9mvr2vv/valgrind-tracker-miner-fs-log-1.7.2.gz?dl=0
>
> As expected, the valgrind logs are a lot more favorable.
> Tracker-extract reports:
>
> ==21759==    definitely lost: 168 bytes in 1 blocks
> ==21759==    indirectly lost: 6,046 bytes in 2 blocks
>
> And those seem to be a memory pool in libjpeg that's mistakenly marked
> as "lost". In tracker-miner-fs logs we see:
>
> ==21669==    definitely lost: 0 bytes in 0 blocks
> ==21669==    indirectly lost: 0 bytes in 0 blocks
>
> So I think we're now fine there. Besides these, there's some memory
> reported as "possibly lost", but that's the usual glib noise (more
> specifically, GObject doesn't deinitialize registered classes/types).
> I can also see some "conditional jump depends on uninitialized values"
> warnings from the HTML extractor, which I can also reproduce, but need
> some more investigation in libxml.
>
> As for the memory used, I must confess 600MB in ps/top seem more
> reasonable to me (as opposed to the several GBs you reported
> initially). In tracker-miner-fs that sounds like a reasonable number
> given your workload (and the fact that we need to keep an in-memory
> representation of the directory tree). In tracker-extract sounds a bit
> on the high side, but could be explained by memory fragmentation,
> because tracker-extract may potentially initialize a lot of libraries
> (that may also create their own runtime caches), and those don't/can't
> get deinitialized after they are no longer used.
>
> Although keep into account that top/ps is not the best tool to measure
> memory usage. A more faithful number is given by the valgrind logs
> themselves, eg. for your case in tracker-miner-fs it is:
>
> ==21669==    still reachable: 143,987,764 bytes in 3,356,013 blocks
>
> So at the time of you hitting Ctrl-C, tracker-miner-fs had allocated
> ~140MB. That sounds indeed more like it, although I'm not sure if you
> let tracker-miner-fs run in its entirety under valgrind (that is,
> until all folders were indexed).
>
> Cheers,
>  Carlos