Re: [xml] Support for really large XML documents



On Wed, May 23, 2012 at 05:55:01PM +0200, Vit Zikmund wrote:
Greetings libxml gurus!
We are using XMLSec library built on top of libxml2 to process some large 
XML files, however it doesn't seem to work for files >2GB, which is 
unfortunately what we need.

I'd like to ask if the library should support processing that large files 
(otherwise, this might be a bug).

It seems there's a limitation in the struct _xmlOutputBuffer, that stores 
written bytes in a signed int - therefore the max limit is 2GB.
Here it is: 
http://git.gnome.org/browse/libxml2/tree/include/libxml/xmlIO.h#n141

We'd really like if the library could support 64 bit sizes and I see the 
struct _xmlParserInputBuffer, that's nearby, does. It uses unsigned long 
that's 64bit for x86_64 architecture, we are building for.
It might really help us if someone here could know what else will need to 
be fixed for the whole thing to work. If it's going to be a patch or a 
full scale project.

  Could you try again with the upcoming 2.9.0 release candidate 1
(or rc2 which i will probably make real soon now!).
  I explicitely changed xmlOutputBuffer to use size_t indexes and
by checking c14n code you should now be only limited by the size of
your memory space, can you confirm, preferably this week before the
final release,

 thanks,

Daniel

-- 
Daniel Veillard      | libxml Gnome XML XSLT toolkit  http://xmlsoft.org/
daniel veillard com  | Rpmfind RPM search engine http://rpmfind.net/
http://veillard.com/ | virtualization library  http://libvirt.org/



[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]