Re: [xml] Parsing XML in embedded environment

[Daniel Veillard <veillard redhat com>]

On Sat, Jun 11, 2011 at 01:28:37PM +0200, David Kubicek wrote:
> On 06/11/2011 01:13 PM, David Kubicek wrote:
> > On 06/11/2011 05:58 AM, Liam R E Quin wrote:
> > > One way might be to override the entity resolver.
> >
> > Never mind. I found an open source app that uses the external
> > entity loader and learned from there how to setup a custom global
> > context.
> >
> > The rest I already had in place (custom function
> > xmlNewInputFromMem(), etc.) started working after this.
>
> Perhaps one more question. This is going to be hard to find for me.
> Now, with every document I parse, a new context is created before
> and freed after "xmlParseDocument(ctx)".
>
> Also, my external loader is called for every document, every time
> creating a new xmlParserInput for the same DTD. Using Valgrind, I
> can see that without repeatedly creating xmlParserInput, the app
> only eats about 7MB of heap after processing all documents. When I
> enable the external loader, it consumes over 57MB and runs visibly
> slower (it's a weak HW).
>
> How do I make this work without repeating the xmlParserInput
> allocation for every document (every external entity loader
> invocation)?

  Impossible to answer correctly. A parser input when attached to a
parser context will be freed when parsing finishes as part of the
context freeing work.
  If your memory augment, either your set of documents augments or
you forgot to free some of the parser contexts, or ...
  But if you are parsing N document you will need at least N
xmlParserInput one per entity parsed. You can't "reuse" them as they
are freed.

  Debugging can't be done out of a cristal ball... take your code,
compile and run it on a platform where you have something like valgrind
and see where your memory go.

  There are things that are available for limiting memory usage
for parsing and parsed document, like using the new APIs xmlRead...
which build a dictionary of shared strings for the document(s)
You can also reuse parsing context and there are tricks to reuse the
same dictionary for multiple documents.

Daniel

-- 
Daniel Veillard      | libxml Gnome XML XSLT toolkit  http://xmlsoft.org/
daniel veillard com  | Rpmfind RPM search engine http://rpmfind.net/
http://veillard.com/ | virtualization library  http://libvirt.org/