Re: [xml] quite huge memory consumption

[Grzegorz JaÅkiewicz <gryzman gmail com>]

2009/7/27 Daniel Veillard <veillard redhat com>:

> ÂAnd what did you do with all the previous root subtree ?
> Well apparently you leaked it since I don't see any code destroying
> it...

Well, including at the bottom another version of stripped down code.
This time, everything that was there is there (except for some data
retrieval, which doesn't leak as is)
So you saying that I have to free whatever xmlDocSetRootElement is
returning, which I presume, is the 'old' document ?


> ÂWell I don't know what sdoc and snode are but obviously you're hiding
> most of the informations on what you program actually does. Anyway this
> list isn't the place to debug fractional pieces of code, it's trees
> and look at the function xmlCleanupParser()documentation it says
> explicitely that it doesn't free any existing document.

Code:

xmlInitParser();

sDoc = xmlParseFile(filename);
if (!sDoc)
{
  return;
}

context = xmlXPathNewContext(sDoc);

if (!context)
{
  xmlFreeDoc(sDoc);
  return;
}

sc = xmlXPathEvalExpression(sTag, context);

int count = sc->nodesetval->nodeNr;

for(int j=0; j < sc->nodesetval->nodeNr; j++)
{
  if(NULL != sc->nodesetval->nodeTab)
  {
    if(NULL!=(sNode = sc->nodesetval->nodeTab[j]))
    {
      xmlDocSetRootElement(sDoc, sNode);

      std::string etag = "//s[ sId='" + getNextSid() + "']/e";
      const xmlChar * eTag = (const xmlChar *) etag.c_str();
      es = xmlXPathEvalExpression(eTag, context);

      for(int k=0; k < es->nodesetval->nodeNr; k++)
      {
        if(NULL != es->nodesetval->nodeTab)
        {
          if(NULL !=(eNode = es->nodesetval->nodeTab[k]))
          {
            ...
            //move the root node
            xmlDocSetRootElement(sDoc, eNode);
            ...

            std::string ttag = "//e[ id='" + id + "']/t";
            const xmlChar * tTag = (const xmlChar *) ttag.c_str();
            t = xmlXPathEvalExpression(tTag, context);

            if(0 != t->nodesetval->nodeNr)
            {
              if(NULL != t->nodesetval->nodeTab)
              {
                if(NULL !=(tNode = t->nodesetval->nodeTab[0]))
                {
                  std::string ratingtag = "//e[ id='" + id + "']/t/r";
                  const xmlChar * ratingTag = (const xmlChar *)
ratingtag.c_str();
                  rating = xmlXPathEvalExpression(ratingTag, context);
                  for(int m=0; m < rating->nodesetval->nodeNr; m++)
                  {
                    if(NULL != rating->nodesetval->nodeTab)
                    {
                      if(NULL !=(ratingNode = rating->nodesetval->nodeTab[m]))
                      {
                          ... // parse props
                      }
                    }
                  }

                  xmlXPathFreeObject(rating);

                }
              }
            }
            xmlXPathFreeObject(t);
          }
        }
        xmlXPathFreeObject(es);
      }
    }
  }
}
xmlXPathFreeObject(sc);
xmlXPathFreeContext(context);
xmlFreeDoc(sDoc);

xmlCleanupParser();


Here are backtraces from valgrind, running that bit of code (again,
version 2.6.23-1 on centos 4)




 40 bytes in 2 blocks are still reachable in loss record 2 of 12
    at 0x4004405: malloc (vg_replace_malloc.c:149)
    by 0x406645A: xmlNewNs (tree.c:713)
    by 0x410E15B: xmlSAX2StartElementNs (SAX2.c:2183)
    by 0x40571C3: xmlParseStartTag2 (parser.c:8106)
    by 0x406318B: xmlParseElement (parser.c:8437)
    by 0x4063946: xmlParseDocument (parser.c:9129)
    by 0x4063F0D: xmlSAXParseFileWithData (parser.c:12167)
    by 0x4064014: xmlSAXParseFile (parser.c:12208)
    by 0x406406E: xmlParseFile (parser.c:12239)



 50 bytes in 6 blocks are possibly lost in loss record 3 of 12
    at 0x4004405: malloc (vg_replace_malloc.c:149)
    by 0x40BAF0E: xmlStrndup (xmlstring.c:45)
    by 0x40BAF89: xmlStrdup (xmlstring.c:71)
    by 0x40679B5: xmlNewNode (tree.c:2132)
    by 0x406A427: xmlNewDocNode (tree.c:2208)
    by 0x410E5CF: xmlSAX2StartElementNs (SAX2.c:2159)
    by 0x40571C3: xmlParseStartTag2 (parser.c:8106)
    by 0x406318B: xmlParseElement (parser.c:8437)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4063946: xmlParseDocument (parser.c:9129)
    by 0x4063F0D: xmlSAXParseFileWithData (parser.c:12167)
    by 0x4064014: xmlSAXParseFile (parser.c:12208)
    by 0x406406E: xmlParseFile (parser.c:12239)





 16,586,686 bytes in 3,227,125 blocks are still reachable in loss
record 4 of 12
    at 0x4004405: malloc (vg_replace_malloc.c:149)
    by 0x40BAF0E: xmlStrndup (xmlstring.c:45)
    by 0x410DFE4: xmlSAX2TextNode (SAX2.c:1812)
    by 0x410E398: xmlSAX2StartElementNs (SAX2.c:1941)
    by 0x40571C3: xmlParseStartTag2 (parser.c:8106)
    by 0x406318B: xmlParseElement (parser.c:8437)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4063946: xmlParseDocument (parser.c:9129)
    by 0x4063F0D: xmlSAXParseFileWithData (parser.c:12167)
    by 0x4064014: xmlSAXParseFile (parser.c:12208)
    by 0x406406E: xmlParseFile (parser.c:12239)




 24,037,152 bytes in 500,774 blocks are still reachable in loss record
5 of 12
    at 0x4004405: malloc (vg_replace_malloc.c:149)
    by 0x406A85D: xmlNewPropInternal (tree.c:1744)
    by 0x410E7FB: xmlSAX2StartElementNs (SAX2.c:1925)
    by 0x40571C3: xmlParseStartTag2 (parser.c:8106)
    by 0x406318B: xmlParseElement (parser.c:8437)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4063946: xmlParseDocument (parser.c:9129)
    by 0x4063F0D: xmlSAXParseFileWithData (parser.c:12167)
    by 0x4064014: xmlSAXParseFile (parser.c:12208)
    by 0x406406E: xmlParseFile (parser.c:12239)


 32,559,248 (1,288,500 direct, 31,270,748 indirect) bytes in 21,475
blocks are definitely lost in loss record 6 of 12
    at 0x4004405: malloc (vg_replace_malloc.c:149)
    by 0x4067991: xmlNewNode (tree.c:2124)
    by 0x406A427: xmlNewDocNode (tree.c:2208)
    by 0x410E5CF: xmlSAX2StartElementNs (SAX2.c:2159)
    by 0x40571C3: xmlParseStartTag2 (parser.c:8106)
    by 0x406318B: xmlParseElement (parser.c:8437)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4063946: xmlParseDocument (parser.c:9129)
    by 0x4063F0D: xmlSAXParseFileWithData (parser.c:12167)
    by 0x4064014: xmlSAXParseFile (parser.c:12208)
    by 0x406406E: xmlParseFile (parser.c:12239)




 2,641,628 bytes in 515,046 blocks are indirectly lost in loss record
7 of 12
    at 0x4004405: malloc (vg_replace_malloc.c:149)
    by 0x40BAF0E: xmlStrndup (xmlstring.c:45)
    by 0x40BAF89: xmlStrdup (xmlstring.c:71)
    by 0x40679B5: xmlNewNode (tree.c:2132)
    by 0x406A427: xmlNewDocNode (tree.c:2208)
    by 0x410E5CF: xmlSAX2StartElementNs (SAX2.c:2159)
    by 0x40571C3: xmlParseStartTag2 (parser.c:8106)
    by 0x406318B: xmlParseElement (parser.c:8437)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4063946: xmlParseDocument (parser.c:9129)
    by 0x4063F0D: xmlSAXParseFileWithData (parser.c:12167)
    by 0x4064014: xmlSAXParseFile (parser.c:12208)
    by 0x406406E: xmlParseFile (parser.c:12239)


 3,942,960 bytes in 82,145 blocks are indirectly lost in loss record 8
of 12
    at 0x4004405: malloc (vg_replace_malloc.c:149)
    by 0x406A85D: xmlNewPropInternal (tree.c:1744)
    by 0x410E7FB: xmlSAX2StartElementNs (SAX2.c:1925)
    by 0x40571C3: xmlParseStartTag2 (parser.c:8106)
    by 0x406318B: xmlParseElement (parser.c:8437)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4063946: xmlParseDocument (parser.c:9129)
    by 0x4063F0D: xmlSAXParseFileWithData (parser.c:12167)
    by 0x4064014: xmlSAXParseFile (parser.c:12208)
    by 0x406406E: xmlParseFile (parser.c:12239)





 5,765,340 bytes in 96,089 blocks are indirectly lost in loss record 9
of 12
    at 0x4004405: malloc (vg_replace_malloc.c:149)
    by 0x4067991: xmlNewNode (tree.c:2124)
    by 0x406A427: xmlNewDocNode (tree.c:2208)
    by 0x410E5CF: xmlSAX2StartElementNs (SAX2.c:2159)
    by 0x40571C3: xmlParseStartTag2 (parser.c:8106)
    by 0x406318B: xmlParseElement (parser.c:8437)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4063946: xmlParseDocument (parser.c:9129)
    by 0x4063F0D: xmlSAXParseFileWithData (parser.c:12167)
    by 0x4064014: xmlSAXParseFile (parser.c:12208)
    by 0x406406E: xmlParseFile (parser.c:12239)




 18,920,820 bytes in 315,347 blocks are indirectly lost in loss record
10 of 12
    at 0x4004405: malloc (vg_replace_malloc.c:149)
    by 0x410DF53: xmlSAX2TextNode (SAX2.c:1773)
    by 0x410E398: xmlSAX2StartElementNs (SAX2.c:1941)
    by 0x40571C3: xmlParseStartTag2 (parser.c:8106)
    by 0x406318B: xmlParseElement (parser.c:8437)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4063946: xmlParseDocument (parser.c:9129)
    by 0x4063F0D: xmlSAXParseFileWithData (parser.c:12167)
    by 0x4064014: xmlSAXParseFile (parser.c:12208)
    by 0x406406E: xmlParseFile (parser.c:12239)




 44,473,080 bytes in 741,218 blocks are still reachable in loss record
11 of 12
    at 0x4004405: malloc (vg_replace_malloc.c:149)
    by 0x4067991: xmlNewNode (tree.c:2124)
    by 0x406A427: xmlNewDocNode (tree.c:2208)
    by 0x410E5CF: xmlSAX2StartElementNs (SAX2.c:2159)
    by 0x40571C3: xmlParseStartTag2 (parser.c:8106)
    by 0x406318B: xmlParseElement (parser.c:8437)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4063946: xmlParseDocument (parser.c:9129)
    by 0x4063F0D: xmlSAXParseFileWithData (parser.c:12167)
    by 0x4064014: xmlSAXParseFile (parser.c:12208)
    by 0x406406E: xmlParseFile (parser.c:12239)



 119,107,500 bytes in 1,985,125 blocks are still reachable in loss
record 12 of 12
    at 0x4004405: malloc (vg_replace_malloc.c:149)
    by 0x410DF53: xmlSAX2TextNode (SAX2.c:1773)
    by 0x410E398: xmlSAX2StartElementNs (SAX2.c:1941)
    by 0x40571C3: xmlParseStartTag2 (parser.c:8106)
    by 0x406318B: xmlParseElement (parser.c:8437)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4061A35: xmlParseContent (parser.c:8361)
    by 0x40631EE: xmlParseElement (parser.c:8521)
    by 0x4063946: xmlParseDocument (parser.c:9129)
    by 0x4063F0D: xmlSAXParseFileWithData (parser.c:12167)
    by 0x4064014: xmlSAXParseFile (parser.c:12208)
    by 0x406406E: xmlParseFile (parser.c:12239)


so as you can see, all these leaks come from stuff that was allocated
in xmlParseFile()..


Thanks Daniel, looking forward to your response.



-- 
GJ