[xml] libxml memory consumption

[Gavin Sherry <swm linuxworld com au>]

Hi all,

I've been writing some data distributions systems for some collegues who
are dumping large amounts of data out of databases into XML
documents. The simplicity of the documents is such that they only stress
one area of libxml - memory consumption. Many of the documents are
hundreds of MBs.

I have not noticed any reference in the documentation nor (from the skim I
gave it) anything in the source which suggests that libxml can deal with
just localised parts of the XML document. That is, to take a basic
example, if I have a document such as:

<doc>
   <info>
        <data>123</data>
        <data>456></data>
   </info>
   <otherinfo>
        <data>abc</data>
        ...
   </otherinfo>
</doc>

and the size of the tree under 'otherinfo' is 200 MB but I want only to
look at the tree under 'info' then I am using a very large amount of
memory to do it. Has anyone considered building a 'contextually' (for lack
of another word) sensitive read ahead system into libxml which could read
only the section of the tree you are interested in into memory or is this
also featured in some part of the library I am yet to explore?

Thanks

Gavin