Re: [xml] Support for really large XML documents

[Daniel Veillard <veillard redhat com>]

On Wed, May 23, 2012 at 05:55:01PM +0200, Vit Zikmund wrote:
> Greetings libxml gurus!
> We are using XMLSec library built on top of libxml2 to process some large 
> XML files, however it doesn't seem to work for files >2GB, which is 
> unfortunately what we need.
>
> I'd like to ask if the library should support processing that large files 
> (otherwise, this might be a bug).
>
> It seems there's a limitation in the struct _xmlOutputBuffer, that stores 
> written bytes in a signed int - therefore the max limit is 2GB.
> Here it is: 
> http://git.gnome.org/browse/libxml2/tree/include/libxml/xmlIO.h#n141
>
> We'd really like if the library could support 64 bit sizes and I see the 
> struct _xmlParserInputBuffer, that's nearby, does. It uses unsigned long 
> that's 64bit for x86_64 architecture, we are building for.
> It might really help us if someone here could know what else will need to 
> be fixed for the whole thing to work. If it's going to be a patch or a 
> full scale project.

  Could you try again with the upcoming 2.9.0 release candidate 1
(or rc2 which i will probably make real soon now!).
  I explicitely changed xmlOutputBuffer to use size_t indexes and
by checking c14n code you should now be only limited by the size of
your memory space, can you confirm, preferably this week before the
final release,

 thanks,

Daniel

-- 
Daniel Veillard      | libxml Gnome XML XSLT toolkit  http://xmlsoft.org/
daniel veillard com  | Rpmfind RPM search engine http://rpmfind.net/
http://veillard.com/ | virtualization library  http://libvirt.org/