Re: [Tracker] trackerd memory efficiency

[Carlos Garnacho <carlos imendio com>]

On jue, 2009-02-05 at 11:24 +0200, Tshepang Lekhonkhobe wrote:
> 2009/2/5 Martyn Russell <martyn imendio com>:
> > Tshepang Lekhonkhobe wrote:
> > > > What I meant was the RES column in top, just to be clear.
> > > >
> > > > I anyways ran $ valgrind --leak-check=full --show-reachable=yes
> > > > --leak-resolution=high -v /usr/libexec/trackerd
> > > >
> > > > and go the following output, and I don't know what it says:
> > > >
> > > > ==6792== LEAK SUMMARY:
> > > > ==6792==    definitely lost: 36 bytes in 1 blocks.

These are the bytes that valgrind consider leaked for sure

> > > > ==6792==    indirectly lost: 120 bytes in 10 blocks.

And this is not freed memory pointed by freed/leaked pointers.

Actually, these 2 "leaks" aren't such, since it's the XDG directories
cache in glib.

> > > > ==6792==      possibly lost: 43,872 bytes in 84 blocks.

Valgrind couldn't guess whether 43Kb were leaked or not, it could be
interesting knowing where does that come from, but it's probably all
these operations with pointers in parsing/stemming, and inside
sqlite/qdbm which are confusing valgrind.

> > > > ==6792==    still reachable: 6,253,647 bytes in 26,607 blocks.

And this is the legitimate memory that trackerd had allocated, around
6MB

> > > > ==6792==         suppressed: 4,496 bytes in 110 blocks.
> > > > --6792--  memcheck: sanity checks: 29089 cheap, 220 expensive
> > > > --6792--  memcheck: auxmaps: 0 auxmap entries (0k, 0M) in use
> > > > --6792--  memcheck: auxmaps_L1: 0 searches, 0 cmps, ratio 0:10
> > > > --6792--  memcheck: auxmaps_L2: 0 searches, 0 nodes
> > > > --6792--  memcheck: SMs: n_issued      = 856 (13696k, 13M)
> > > > --6792--  memcheck: SMs: n_deissued    = 7 (112k, 0M)
> > > > --6792--  memcheck: SMs: max_noaccess  = 65535 (1048560k, 1023M)
> > > > --6792--  memcheck: SMs: max_undefined = 35 (560k, 0M)
> > > > --6792--  memcheck: SMs: max_defined   = 1348 (21568k, 21M)
> > > > --6792--  memcheck: SMs: max_non_DSM   = 849 (13584k, 13M)
> > > > --6792--  memcheck: max sec V bit nodes:    1581 (80k, 0M)
> > > > --6792--  memcheck: set_sec_vbits8 calls: 51138 (new: 1581, updates: 49557)
> > > > --6792--  memcheck: max shadow mem size:   13968k, 13M
> > > > --6792-- translate:            fast SP updates identified: 48,718 ( 88.1%)
> > > > --6792-- translate:   generic_known SP updates identified: 4,920 (  8.8%)
> > > > --6792-- translate: generic_unknown SP updates identified: 1,647 (  2.9%)
> > > > --6792--     tt/tc: 32,955,230 tt lookups requiring 50,735,061 probes
> > > > --6792--     tt/tc: 32,955,230 fast-cache updates, 10 flushes
> > > > --6792--  transtab: new        35,700 (767,467 -> 11,066,732; ratio
> > > > 144:10) [0 scs]
> > > > --6792--  transtab: dumped     0 (0 -> ??)
> > > > --6792--  transtab: discarded  355 (7,022 -> ??)
> > > > --6792-- scheduler: 2,835,614,942 jumps (bb entries).
> > > > --6792-- scheduler: 29,089/43,763,705 major/minor sched events.
> > > > --6792--    sanity: 29090 cheap, 220 expensive checks.
> > > > --6792--    exectx: 12,289 lists, 11,820 contexts (avg 0 per list)
> > > > --6792--    exectx: 9,597,286 searches, 9,605,875 full compares (1,000 per 1000)
> > > > --6792--    exectx: 0 cmp2, 189 cmp4, 30,892,554 cmpAll
> > > > --6792--  errormgr: 1,500 supplist searches, 190,814 comparisons during search
> > > > --6792--  errormgr: 83 errlist searches, 199 comparisons during search
> >
> > Tshepang, as stated in one of the other emails related to the upcoming release this week (assuming we get 
> > issue #2 done), there will be NO improvements here before the release and after the release when we move 
> > the crawler to the indexer, we should see significant changes.
>
> All I wanted was someone to analyse the valgrind output. This
> originated from me wanting to change the README to truthfully
> represent Tracker's memory consumption. I promise that's all :-)