Re: [xslt] xsltproc memory consumption w/ large DTD / docbook

[Michael Weiser <mweiser fachschaft imn htwk-leipzig de>]

On Wed, 29 Jan 2003, Daniel Veillard wrote:

>   Please try the enclosed patch, and report improvement or crashes.
> This is a relatively dangerous change and I want to get feedback before
> commiting any such change in CVS,
Thanks a lot! I was just finished digging through the sources for the
xsltLoadDocument function and preparing an answer to your original mail
requesting help when your patch came in.

I tried it and nothing broke, first of all. I used a small test stylesheet
on a single file and according to dmalloc memory consumption went down by
about 16MB. I've included the two log files at the bottom of this mail.

Used with the original problematic stylesheet and small documents its
growing as fast as the unpatched version and gets killed at the same
point. For further debugging I've included only about 20 documents in
processing so that it "only" consumes about 170MB and finishes correctly.
On this I ran dmalloc and produced the second set of logs at the bottom of
this mail. Here memory consumption is almost exactly the same again.

Does the patch perhaps only affect the first load and not the actual
document() call? I'll try to provide small testcase.

test stylesheets versions 2.0.46/1.0.22 without patch:

Dmalloc version '4.8.2' from 'http://dmalloc.com/'
[...]
alloc calls: malloc 525591, calloc 156, realloc 421, free 525747
alloc calls: recalloc 0, memalign 0, valloc 0
 total memory allocated: 33577265 bytes (526168 pnts)
 max in use at one time: 8534051 bytes (290858 pnts)
max alloced with 1 call: 49152 bytes
max alloc rounding loss: 7048157 bytes (45%)
max memory space wasted: 27648 bytes (0%)
final user memory space: basic 67, divided 3744, 15609856 bytes
 final admin overhead: basic 91, divided 2592, 10989568 bytes (41%)
 final external space: 0 bytes (0 blocks)
top 10 allocations:
 total-size  count in-use-size  count  source
   16104000   4026           0      0  ra=0x40104a0c
    2875912  55306           0      0  ra=0x40100fa6
    1730664    156           0      0  ra=0x400b1a00
    1538264  54938           0      0  ra=0x401117dc
    1439672  27686           0      0  ra=0x40100b8d
    1159500   3865           0      0  ra=0x400ed2dd
     939239  70198           0      0  ra=0x400fe70a
     638976     39           0      0  ra=0x400acfea
     558494  63221           0      0  ra=0x400ee161
   33577265 526168           0      0  Total of 133
dumping not-freed pointers changed since 0:
 memory table is empty

test stylesheet current cvs with patch:

[...]
alloc calls: malloc 525603, calloc 156, realloc 421, free 525759
alloc calls: recalloc 0, memalign 0, valloc 0
 total memory allocated: 17910246 bytes (526180 pnts)
 max in use at one time: 8548091 bytes (290865 pnts)
max alloced with 1 call: 49152 bytes
max alloc rounding loss: 7034533 bytes (45%)
max memory space wasted: 31328 bytes (0%)
final user memory space: basic 68, divided 3744, 15613952 bytes
 final admin overhead: basic 91, divided 2586, 10964992 bytes (41%)
 final external space: 0 bytes (0 blocks)
top 10 allocations:
 total-size  count in-use-size  count  source
    2875912  55306           0      0  ra=0x400fb93a
    2549601 243692           0      0  ra=0x400eb25d
    1730664    156           0      0  ra=0x400aea00
    1538264  54938           0      0  ra=0x40109a2c
    1439672  27686           0      0  ra=0x400fb40c
    1159500   3865           0      0  ra=0x400eae25
     638976     39           0      0  ra=0x400a9fea
     391488   6117           0      0  ra=0x4010abd2
     390656   6976           0      0  ra=0x400e89a5
   17910246 526180           0      0  Total of 120
dumping not-freed pointers changed since 0:
 memory table is empty

autolayout.xsl versions 2.0.46/1.0.22 without patch:

[...]
alloc calls: malloc 3949980, calloc 2676, realloc 7772, free 3952656
alloc calls: recalloc 0, memalign 0, valloc 0
 total memory allocated: 169751834 bytes (3960428 pnts)
 max in use at one time: 53997469 bytes (2008804 pnts)
max alloced with 1 call: 49152 bytes
max alloc rounding loss: 63941571 bytes (17%)
max memory space wasted: 5280 bytes (0%)
final user memory space: basic 769, divided 28026, 117944320 bytes
 final admin overhead: basic 673, divided 18930, 80293888 bytes (18%)
 final external space: 0 bytes (0 blocks)
top 10 allocations:
 total-size  count in-use-size  count  source
   29687544   2676           0      0  ra=0x400aea00
   29228808 1043886           0      0  ra=0x40109a2c
   23291961 2085252           0      0  ra=0x400eb25d
   22024800  73416           0      0  ra=0x400eae25
   10960896    669           0      0  ra=0x400a9fea
    7440192 116253           0      0  ra=0x4010abd2
    7419664 132494           0      0  ra=0x400e89a5
    6013500  60135           0      0  ra=0x400ec90c
    5848000    711           0      0  ra=0x400ff8bd
  169751834 3960428           0      0  Total of 117
dumping not-freed pointers changed since 0:
 memory table is empty

autolayout.xsl current cvs with patch:

[...]
alloc calls: malloc 3950361, calloc 2676, realloc 7794, free 3953037
alloc calls: recalloc 0, memalign 0, valloc 0
 total memory allocated: 170290373 bytes (3960831 pnts)
 max in use at one time: 53756943 bytes (2008890 pnts)
max alloced with 1 call: 49152 bytes
max alloc rounding loss: 64188017 bytes (18%)
max memory space wasted: 3456 bytes (0%)
final user memory space: basic 769, divided 28027, 117948416 bytes
 final admin overhead: basic 673, divided 18938, 80326656 bytes (18%)
 final external space: 0 bytes (0 blocks)
top 10 allocations:
 total-size  count in-use-size  count  source
   29687544   2676           0      0  ra=0x400b1a00
   29228808 1043886           0      0  ra=0x401117dc
   22024800  73416           0      0  ra=0x400ed2dd
   13873850 1236515           0      0  ra=0x400fe70a
   10960896    669           0      0  ra=0x400acfea
    7440192 116253           0      0  ra=0x40112a7c
    7420112 132502           0      0  ra=0x400eaa01
    6742419 587137           0      0  ra=0x400ee161
    6013500  60135           0      0  ra=0x400ef13c
  170290373 3960831           0      0  Total of 129
dumping not-freed pointers changed since 0:
 memory table is empty
-- 
Thanks in advance,
Michael
Sauced it has to be, sauced!