Re: [xml] libxml2, python, reducing memory consumption?

[Robin Dunn <robin alldunn com>]

Daniel Veillard wrote:
> On Tue, Dec 23, 2003 at 11:26:34AM -0800, Robin Dunn wrote:
>
> > I have a tool that reads a number of input documents, does some 
> > processing and then builds and writes a new document, but it uses about 
> > 500MB of RAM when it runs so I am trying to find ways to reduce the 
> > memory footprint.  I know that using SAX instead of DOM would do it for 
> > me, but the nature of what needs to be done just screams for DOM and 
> > XPath do that is the way I have done it.
>
>
>   Try the xmlReader and expand only the subparts where your really need to
> do XPath queries. See the end of http://xmlsoft.org/xmlreader.html#Mixing
>
>
> > The total size of the input documents is about 100M and each one is 
> > about 4-8M.  The total size of the output document is about 1.5M.
> >
> > I added a call to doc.freeDoc after I am done with each input document 
> > before loading the next one, but that doesn't seem to affect memory 
> > consumption at all.
>
>
>   Then something is wrong...


It turns out that updating the libxml2 RPMs to 2.6.3 solves the problem. 
 (I should have thought of that before since that is usually the first 
thing I tell wxPython users to do when they are more than one release 
behind.)  I was originally using 2.5.11 from Mandrake 9.2.  Then I 
rebuilt the 2.6.3 SRPM from Mandrake Cooker and now my latest version of 
the script fluctuates between 40 and 60M of RAM.

--
Robin Dunn
Software Craftsman
http://wxPython.org  Java give you jitters?  Relax with wxPython!