Re: [xml] libxml2 very slow on big data dump

[Alexandre Macard <amacard arkeia com>]

Alexandre Macard a écrit :
> Stefan Behnel a écrit :
>   
>> Alexandre Macard wrote:
>>   
>>     
>>> Stefan Behnel a écrit :
>>>     
>>>       
>>>> Alexandre Macard wrote:
>>>>       
>>>>         
>>>>> I try dump a node from a big xml (near 7mo), and the libxml2 is very
>>>>> slow to respond.
>>>>>
>>>>> I tried to trace the problem and it seems to take all it's time into
>>>>> the
>>>>> function: xmlOutputBufferWriteEscape.
>>>>> I do not need to escape data because I use a base64 encoding.
>>>>>
>>>>>         
>>>>>           
>>>> You didn't write which version of libxml2 you are using, but there was a
>>>> bug in an older version that could lead to horrible performance when
>>>> serialising character entities.
>>>>
>>>> Try upgrading your library.
>>>>       
>>>>         
>>> Sorry I forgot to precise this information. I am using the last version
>>> 2.7.2.
>>>     
>>>       
>> So maybe it's a similar bug, but for a different encoding (I think it was
>> related to the ASCII encoding at the time).
>>
>> Could you provide the code snippet that you use for serialisation? I.e.
>> what parameters you pass into what function?
>>
>> Stefan
>>
>>
>>   
>>     
> This little test code make 15secs to exit.
> The journal.xml size is 7.1Mo.
>
> int main() {
>     xmlDocPtr doc;
>     xmlNodePtr cur;
>     xmlBufferPtr buf;
>
>     doc = xmlParseFile("./journal.xml");
>    
>     if (doc == NULL ) {
>         fprintf(stderr,"Document not parsed successfully. \n");
>         return (0);
>     }
>     cur = xmlDocGetRootElement(doc);
>
>     if (cur == NULL) {
>         fprintf(stderr,"empty document\n");
>         xmlFreeDoc(doc);
>         return (0);
>     }
>
>     buf = xmlBufferCreate();
>    
>     xmlNodeDump(buf, doc, cur, 1, 1);
>
>     xmlFree(buf);
>     xmlFreeDoc(doc);
>
>     return (0);
> }
>
> I will try to add later a script to generate a similar xml.
>
> Thanks.
> _______________________________________________
> xml mailing list, project page  http://xmlsoft.org/
> xml gnome org
> http://mail.gnome.org/mailman/listinfo/xml
>
>   
I forgot to precise that all the time is passed into function xmlNodeDump.

At the end you find a script that generate similar xml. I used this xml
to test and I had to wait 22secs for my program to exit.

usage: script.sh > journal.xml


#!/bin/bash

#Header
echo -n '<SOAP-ENV:Envelope
xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/";
xmlns:xsi="http://www.w3.org/1999/XMLSchema-instance";
xmlns:xsd="http://www.w3.org/1999/XMLSchema";
SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/";>
<SOAP-ENV:Header/> <SOAP-ENV:Body>  <m:arkws_methodResponse
xmlns:m="urn:arkeia">'

echo -n '<m:list0 xsi:type="xsd:list"><m:last
xsi:type="xsd:integer">1</m:last><m:param0
xsi:type="xsd:integer">0</m:param0><m:base64_param1
xsi:type="xsd:string">MjAwOC8xMi8xNiAxNjo0NzoxMyBJMDAxMTAwMDAgMDFUUF9MSVNUX0FMTDogWW91IGhhdmUgc3VjY2Vzc2Z1bGx5IGxvYWRlZCB0aGUgbGlzdCBvZiB0YXBlcyE=</m:base64_param1><m:param2
xsi:type="xsd:list">'

i=0
while [ $i -lt 15000 ] ; do
    echo -n '<m:item xsi:type="xsd:list"><m:base64_RDATE
xsi:type="xsd:string">MTIzMDkxMDAyNQ==</m:base64_RDATE><m:base64_NUM
xsi:type="xsd:string">MDAwMDE=</m:base64_NUM><m:base64_OWNER
xsi:type="xsd:string">cm9vdA==</m:base64_OWNER><m:base64_THREAD
xsi:type="xsd:string">MDAx</m:base64_THREAD><m:base64_PLID
xsi:type="xsd:string">NDczODVhMWY=</m:base64_PLID><m:base64_CID
xsi:type="xsd:string">NDkzNjlmZjA=</m:base64_CID><m:base64_TPID
xsi:type="xsd:string">NDc1NThlZjM=</m:base64_TPID><m:base64_VOLTAG
xsi:type="xsd:string">L2JhY2t1cHMvZmlsZQ==</m:base64_VOLTAG><m:base64_NAME
xsi:type="xsd:string">dGFwZV9maWxl</m:base64_NAME></m:item>'
    i=`expr $i + 1`
done

echo -n '</m:param2></m:list0>'

#Footer
echo '</m:arkws_methodResponse> </SOAP-ENV:Body></SOAP-ENV:Envelope>'