Re: [xml] Unexpected Nodeset Limit for XPath

["Eberhard" <flash vicsmba com>]

Just comments of no help I suspect.

Why can't the limit be soft coded?  I run machines with 100s of gig of memory and don't need a limit.  IBM 
AIX which handles memory very well.

Some day I would like to discuss with someone who looks over this project (which I have in the past 
contributed to) about memory management.

In the old days it was "very bad" to use memory and not return it.  Memory was at a premium.  It was usually 
called "memory leaks."

In the current world machines have more memory than they need (or can get it for few dollars).  With 
multi-cores and multi-processors people don't realize (this is over simplified) that in order to take memory 
or return memory from the O/S requires a "context switch" which is more or less a term to mean the machine 
must switch to single user mode to make the memory operations.

All of my code recently deliberately uses memory leaks.  If I have a program running that will process 1,000s 
of XML files with 1,000 of nodes ... there is zero need to get a tiny slip of memory ofr some purpose and 
return it.  Make the pointer static and if it exists, reuse it.  To do this well, keep a static for the size 
as well and if too small use realloc().  Eventually it will be big enough.

This can provide very dramatic performance increases on a machine that has thousands of processes parsing 
XML.  It could easily be implemented as an option (by having free() set the pointer to NULL if it really does 
a free, and if it does not make the free() a noop.  Not releasing the memory is in fact a memory leak ... but 
if you are repeating the creation/return of memory a lot it is not a big deal for memory usage, and it is a 
big deal for a large multi-user system (my largest is 16,000 users on one box -- a big IBM).  Imaging 16,000 
users stuck single threaded because of memory management.  

Just a thought.  Many machines do not handle the creation/destruction of memory all that well and this could 
allow you to set your own parameters -- perhaps on maximum memory used (although the O/S usually has soft and 
hard limits for that) or perhaps on number of nodes or ...

Eric



VICS, LLC
Eric S Eberhard
2933 W Middle Verde Rd
Camp Verde, AZ  86322

928-567-3727            (land line)
928-301-7537            (cell phone)

http://www.vicsmba.com
https://www.facebook.com/groups/286143052248115


-----Original Message-----
From: xml <xml-bounces gnome org> On Behalf Of Nick
Sent: Wednesday, December 1, 2021 9:53 PM
To: xml gnome org
Subject: [xml] Unexpected Nodeset Limit for XPath

I have a large XML file, but a fairly basic and flat layout. Depending on the number of elements, I'm hitting 
a nodeset limit.

I'm surprised to hit the nodeset limit because it seems like the xpath can ignore (ie. not store) all of the 
"Term" elements--which are causing the error.

If I reduce the number of "Term" elements then it works fine.

libxml2 version 2.9.12

I can repro the issue with xmllint

Here's a basic XML to demonstrate the behavior (this example is stripped down for demonstration purposes, the 
XPath makes more sense in the real-world use case):


<?xml version="1.0" encoding="UTF-8"?>
<Elements id="5M_Elements">
     <Group>
         <Term>1</Term>
         <Term>2</Term>
         <Term>3</Term>
         <!-- 5 Million more Term elements... -->
     </Group>
</Elements>


Here's what I get for the XPath:

xmllint --xpath '//*[@id="5M_Elements"]' ./LibXml_NodeSetLimit.xml  >/dev/null XPath error : Memory 
allocation failed : growing nodeset hit limit

growing nodeset hit limit

^
XPath evaluation failure


Can the XPath code not count all of these elements toward the node limit?


Thanks,

Nick


_______________________________________________
xml mailing list, project page  http://xmlsoft.org/ xml gnome org https://mail.gnome.org/mailman/listinfo/xml