Re: [xml] Memory leak problem

["Eric Eberhard" <flash vicsmba com>]

Thank you much.  I am on AIX (IBM).  Performance is a non-issue.  We process 2-4 million XML documents per 
day (in and out) using HTTP POST.  On the main server which handles 20,000+ orders per day on automated lines 
that give 2-3 seconds to update inventory, A/R, check credit, rate shop up to 5 carriers, collect credit card 
auths, etc.  This has never been a problem as in forever.  Where I am having the problem is sending real-time 
inventory to thousands of customers where an XML file is created, written, and "freed" -- always worked until 
a customer reached a certain number if SKUs or perhaps customers that need an update.  Some memory somewhere 
is not being freed.  It could be my code.  Or libxlml2.  I am certainly not infallible!  

I really appreciate your giving me those function names -- I will Google them and see if they work on AIX.  

I can't use other tools -- it is integrated with our database manager.  So C is my only option.  I don't know 
if it is something in my code, something in libxml2 -- which is truly excellent -- maybe I should first 
update to the latest version and see?  And then try debugging.  I am pretty careful on my own memory 
management but ... I still make mistakes and I thank you for the pointers.

Grrrr -- you'd think after 40 years of C coding -- and since 1988 on AIX -- I would know these things, but - 
there is always something else to learn.  I contribute a lot to help other people on this forum and I am glad 
that people like you are willing to help me!  I need it!

AIX has "dbx" as the debugger .. being old I am not good at it and tend to use debug code rather than depend 
on the dbx.  I don't even know how to ask dbx to tell me what my memory usage is.  

I will report back to the group when done and solved.  

Have a GREAT Thanksgiving,

E

-----Original Message-----
From: Liam R E Quin [mailto:liam holoweb net] 
Sent: Tuesday, November 26, 2019 7:17 PM
To: Eric Eberhard <flash vicsmba com>; 'BR Chrisman' <brchrisman gmail com>
Cc: xml gnome org
Subject: Re: [xml] Memory leak problem

On Tue, 2019-11-26 at 11:47 -0700, Eric Eberhard wrote:
> Is there a C call to see how much memory one is consuming?  I could 
> likely put that in to try and find it.

Depends on your operating system - there are also environment variables you can se that affect the bahaviour 
of malloc() in various ways, and that may help you. Try malloc_info() maybe,
   int malloc_info(int options, FILE *stream); where options must be 0 (duh).

If you don't have that, malloc_stats() prints to stderr, and if you don't ahve that, mallinfo() returns a 
struct.

On some systems, getrusage() will tell you about your process or thread's memory usage. Some other systems 
have vtimes() instead.

There's a variety of C debugging tools, but it depends on the operating system and environment as to which 
will be available.

It might be that a change in some external library has moved something around in memory and exposed a bug in 
your code (or in someone else's).

Incidentally, for production work, it's worth considering moving to an XSLT 3 engine - whether Saxon 9 (Java, 
C# and C) or another one.

In a recent class exercise (in a course i was running on XSLT 3) people reported times of 6 seconds or so to 
write 10,000 HTIL files from an XML input document, and 10 seconds to do the same in streaming mode.
This was using Saxon EE in Java, for what it's worth. So it might be acceptably fast that you can consider 
moving to XSLT 3 and get the engineering benefits of type checking, too.

Liam

--
Liam Quin, https://www.delightfulcomputing.com/
Available for XML/Document/Information Architecture/XSLT/ XSL/XQuery/Web/Text Processing/A11Y training, work 
& consulting.
Barefoot Web-slave, antique illustrations:  http://www.fromoldbooks.org