[xml] (no subject)

[Josiah Johnston <siah nih gov>]

I am implementing an XML file standard for microscope data. ( 
http://openmicroscopy.org/ ) One issue I've been struggling with is 
extraction of pixel data. Pixel data is very large (500 MB is not 
unusual) and can be embedded in the XML document. I do not want this to 
be loaded into RAM.

I am looking for a method available using some component of libxml to 
extract this massive data without loading it into RAM. I've been 
playing with embedding the pixel data in a CDATA section and using a 
little c program to preprocess the file, piping the pixel data to a 
file, and replacing it with a file path. Using <![CDATA[...]]> as a 
file position marker is not an optimal solution.

Does anyone have suggestions for dealing with this? I'd rather use an 
existing XML tool and stick to standard methods of dealing with this 
than develop my own method of addressing this problem. Unfortunately, I 
haven't been able to find any threads that deal with gigantic XML 
nodes. All the memory issues I've seen address gigantic documents.

Thanks in advance,
Josiah