Re: [xml] Unable to create huge text nodes with libxml2 2.7.7

[Daniel Veillard <veillard redhat com>]

On Sun, Feb 27, 2011 at 02:15:19PM +0100, Michael Kwasigroch wrote:
> Hi list,
>
> I (actually a very important customer ;-) was bit by the 10MB size limit 
> introduced in libxml2 2.7.3. Actually, we are using 2.7.7...
>
> The code basically does the following (error handling omitted for clarity, 
> actual code is more complex):
>
> ----- snip -----
> xmlDocPtr        doc    = NULL;
> xmlNodePtr       node   = NULL;
> xmlTextWriterPtr writer = NULL;
> char             huge[ 100*1024*1024 ] = { 'a' };
> int              written;
>
> doc = xmlNewDoc( (xmlChar *) "1.0" );
> node = xmlNewDocNode( doc, NULL, (xmlChar *)"node", NULL );
> xmlDocSetRootElement( doc, node );
> writer = xmlNewTextWriterTree( doc, node, 0 );
> xmlTextWriterStartElement( writer, BAD_CAST "huge" );
> written = xmlTextWriterWriteBinHex( writer, huge, 0, sizeof( huge ) );
> ----- snip -----
>
> With libxml2 V2.7.7 xmlTextWriterWriteBinHex() always returns -1 and error 
> analysis reveals that the SAX2 parser hit the 10MB limit.

At first I didn't understood as xmlTextWriter is about generating a
serializer buffer output, but then I realized the problem comes from
the use of xmlNewTextWriterTree().

> I was bypassing this by eliminating the size limit check in SAX2.c and 
> recompiling libxml2 but I do not really like this "solution".
>
> So the question is if there is a standard way of passing the 
> XML_PARSE_HUGE option to the SAX2 parser used under the hood of the 
> xmlTextWriter-family functions.

  IMHO xmlNewTextWriterTree is a hack, it forces a reparsing of the data
while it's not really needed, plus contrary to all other APIs invoking a
parser, there is no way of passing any options like you noticed.

> I have done extensive research but was not able to find anything useful. I 
> have checked but did not find a compile-time option to eliminate the 10MB 
> size limit.

  My suggestion would be to directly create the parent node and its
children via the tree API instead of giving it to an intermediary
parser.

Daniel

-- 
Daniel Veillard      | libxml Gnome XML XSLT toolkit  http://xmlsoft.org/
daniel veillard com  | Rpmfind RPM search engine http://rpmfind.net/
http://veillard.com/ | virtualization library  http://libvirt.org/