Re: [xml] Unexpected Nodeset Limit for XPath

Just comments of no help I suspect.

Why can't the limit be soft coded?  I run machines with 100s of gig of memory and don't need a limit.  IBM 
AIX which handles memory very well.

Some day I would like to discuss with someone who looks over this project (which I have in the past 
contributed to) about memory management.

In the old days it was "very bad" to use memory and not return it.  Memory was at a premium.  It was usually 
called "memory leaks."

In the current world machines have more memory than they need (or can get it for few dollars).  With 
multi-cores and multi-processors people don't realize (this is over simplified) that in order to take memory 
or return memory from the O/S requires a "context switch" which is more or less a term to mean the machine 
must switch to single user mode to make the memory operations.

All of my code recently deliberately uses memory leaks.  If I have a program running that will process 1,000s 
of XML files with 1,000 of nodes ... there is zero need to get a tiny slip of memory ofr some purpose and 
return it.  Make the pointer static and if it exists, reuse it.  To do this well, keep a static for the size 
as well and if too small use realloc().  Eventually it will be big enough.

This can provide very dramatic performance increases on a machine that has thousands of processes parsing 
XML.  It could easily be implemented as an option (by having free() set the pointer to NULL if it really does 
a free, and if it does not make the free() a noop.  Not releasing the memory is in fact a memory leak ... but 
if you are repeating the creation/return of memory a lot it is not a big deal for memory usage, and it is a 
big deal for a large multi-user system (my largest is 16,000 users on one box -- a big IBM).  Imaging 16,000 
users stuck single threaded because of memory management.  

Just a thought.  Many machines do not handle the creation/destruction of memory all that well and this could 
allow you to set your own parameters -- perhaps on maximum memory used (although the O/S usually has soft and 
hard limits for that) or perhaps on number of nodes or ...


Eric S Eberhard
2933 W Middle Verde Rd
Camp Verde, AZ  86322

928-567-3727            (land line)
928-301-7537            (cell phone)

-----Original Message-----
From: xml <xml-bounces gnome org> On Behalf Of Nick
Sent: Wednesday, December 1, 2021 9:53 PM
To: xml gnome org
Subject: [xml] Unexpected Nodeset Limit for XPath

I have a large XML file, but a fairly basic and flat layout. Depending on the number of elements, I'm hitting 
a nodeset limit.

I'm surprised to hit the nodeset limit because it seems like the xpath can ignore (ie. not store) all of the 
"Term" elements--which are causing the error.

If I reduce the number of "Term" elements then it works fine.

libxml2 version 2.9.12

I can repro the issue with xmllint

Here's a basic XML to demonstrate the behavior (this example is stripped down for demonstration purposes, the 
XPath makes more sense in the real-world use case):

<?xml version="1.0" encoding="UTF-8"?>
<Elements id="5M_Elements">
         <!-- 5 Million more Term elements... -->

Here's what I get for the XPath:

xmllint --xpath '//*[@id="5M_Elements"]' ./LibXml_NodeSetLimit.xml  >/dev/null XPath error : Memory 
allocation failed : growing nodeset hit limit

growing nodeset hit limit

XPath evaluation failure

Can the XPath code not count all of these elements toward the node limit?



xml mailing list, project page xml gnome org

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]