Re: [xml] libxml2, python, reducing memory consumption?

Daniel Veillard wrote:
On Tue, Dec 23, 2003 at 11:26:34AM -0800, Robin Dunn wrote:

I have a tool that reads a number of input documents, does some processing and then builds and writes a new document, but it uses about 500MB of RAM when it runs so I am trying to find ways to reduce the memory footprint. I know that using SAX instead of DOM would do it for me, but the nature of what needs to be done just screams for DOM and XPath do that is the way I have done it.

  Try the xmlReader and expand only the subparts where your really need to
do XPath queries. See the end of

The total size of the input documents is about 100M and each one is about 4-8M. The total size of the output document is about 1.5M.

I added a call to doc.freeDoc after I am done with each input document before loading the next one, but that doesn't seem to affect memory consumption at all.

  Then something is wrong...

It turns out that updating the libxml2 RPMs to 2.6.3 solves the problem. (I should have thought of that before since that is usually the first thing I tell wxPython users to do when they are more than one release behind.) I was originally using 2.5.11 from Mandrake 9.2. Then I rebuilt the 2.6.3 SRPM from Mandrake Cooker and now my latest version of the script fluctuates between 40 and 60M of RAM.

Robin Dunn
Software Craftsman  Java give you jitters?  Relax with wxPython!

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]