[xml] Multithreaded library, xmlInitParser and valgrind

[Christian Ferrari <camauz yahoo com>]

Dear All,
I'm developing a multithreaded library and I'm using libxml2.
The code is worked as designed, but valgrind tool detects a small memory leak (24 byte) if 
xmlInitParser()/xmlCleanupParser() is called more than once from threaded functions.

This is the description of the environment:

uname -a
Linux ubuntu 2.6.24-28-server

libxml2 version: 2.6.31.dfsg-2ubuntu1.5

valgrind version: valgrind-3.3.0-Debian

This code does NOT exploit the memory leak and this is in accordance to the note explained at this URL: 
http://xmlsoft.org/threads.html

#include <stdio.h>
#include <string.h>
#include <pthread.h>
#include <unistd.h>
#include <libxml/parser.h>

char *XML_STRING="<?xml version=\"1.0\" encoding=\"UTF-8\" ?><foo></foo>";

void *thread1(void *parm);
void *thread2(void *parm);

int main(int argc, char *argv[])
{
    pthread_t t1, t2;

    xmlInitParser();
    pthread_create(&t1, NULL, thread1, NULL);
    pthread_create(&t2, NULL, thread2, NULL);
    pthread_join(t1, NULL);
    pthread_join(t2, NULL);

    pthread_create(&t1, NULL, thread1, NULL);
    pthread_create(&t2, NULL, thread2, NULL);
    pthread_join(t1, NULL);
    pthread_join(t2, NULL);
    xmlCleanupParser();

    return 0;
}

void *thread1(void *parm)
{
    xmlDocPtr doc;
    printf("[1] --- begin ---\n");
    printf("[1] xmlReadMemory()\n");
    doc = xmlReadMemory(XML_STRING, strlen(XML_STRING), "buffer.xml", NULL, 0);
    printf("[1] xmlFreeDoc()\n");
    xmlFreeDoc(doc);
    printf("[1] sleep()\n");
    sleep(2);
    printf("[1] --- end ---\n");
    return NULL;
}

void *thread2(void *parm)
{
    xmlDocPtr doc;
    printf("[2] --- begin ---\n");
    printf("[2] sleep()\n");
    sleep(1);
    printf("[2] xmlReadMemory()\n");
    doc = xmlReadMemory(XML_STRING, strlen(XML_STRING), "buffer.xml", NULL, 0);
    printf("[2] xmlFreeDoc()\n");
    xmlFreeDoc(doc);
    printf("[2] --- end ---\n");
    return NULL;
}


valgrind --leak-check=full --show-reachable=yes --num-callers=1000 a.out

==21489== Memcheck, a memory error detector.
==21489== Copyright (C) 2002-2007, and GNU GPL'd, by Julian Seward et al.
==21489== Using LibVEX rev 1804, a library for dynamic binary translation.
==21489== Copyright (C) 2004-2007, and GNU GPL'd, by OpenWorks LLP.
==21489== Using valgrind-3.3.0-Debian, a dynamic binary instrumentation framework.
==21489== Copyright (C) 2000-2007, and GNU GPL'd, by Julian Seward et al.
==21489== For more details, rerun with: -v
==21489==
[1] --- begin ---
[2] --- begin ---
[2] sleep()
[1] xmlReadMemory()
[1] xmlFreeDoc()
[1] sleep()
[2] xmlReadMemory()
[2] xmlFreeDoc()
[2] --- end ---
[1] --- end ---
[1] --- begin ---
[1] xmlReadMemory()
[1] xmlFreeDoc()
[1] sleep()
[2] --- begin ---
[2] sleep()
[2] xmlReadMemory()
[2] xmlFreeDoc()
[2] --- end ---
[1] --- end ---
==21489==
==21489== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 21 from 1)
==21489== malloc/free: in use at exit: 0 bytes in 0 blocks.
==21489== malloc/free: 130 allocs, 130 frees, 52,826 bytes allocated.
==21489== For counts of detected errors, rerun with: -v
==21489== All heap blocks were freed -- no leaks are possible.


Moving xmlInitParser() and xmlCleanupParser() inside a thread, EXPLOITS the memory leak IF AND ONLY IF the 
xmlInitParser() is called after xmlCleanupParser().


This code DOES exploit the memory leak:

#include <stdio.h>
#include <string.h>
#include <pthread.h>
#include <unistd.h>
#include <libxml/parser.h>

char *XML_STRING="<?xml version=\"1.0\" encoding=\"UTF-8\" ?><foo></foo>";

void *thread1(void *parm);
void *thread2(void *parm);

int main(int argc, char *argv[])
{
    pthread_t t1, t2;

    pthread_create(&t1, NULL, thread1, NULL);
    pthread_create(&t2, NULL, thread2, NULL);
    pthread_join(t1, NULL);
    pthread_join(t2, NULL);

    pthread_create(&t1, NULL, thread1, NULL);
    pthread_create(&t2, NULL, thread2, NULL);
    pthread_join(t1, NULL);
    pthread_join(t2, NULL);

    return 0;
}

void *thread1(void *parm)
{
    xmlDocPtr doc;
    printf("[1] --- begin ---\n");
    printf("[1] xmlInitParser()\n");
    xmlInitParser();
    printf("[1] xmlReadMemory()\n");
    doc = xmlReadMemory(XML_STRING, strlen(XML_STRING), "buffer.xml", NULL, 0);
    printf("[1] xmlFreeDoc()\n");
    xmlFreeDoc(doc);
    printf("[1] sleep()\n");
    sleep(2);
    printf("[1] xmlCleanupParser()\n");
    xmlCleanupParser();
    printf("[1] --- end ---\n");
    return NULL;
}

void *thread2(void *parm)
{
    xmlDocPtr doc;
    printf("[2] --- begin ---\n");
    printf("[2] sleep()\n");
    sleep(1);
    printf("[2] xmlReadMemory()\n");
    doc = xmlReadMemory(XML_STRING, strlen(XML_STRING), "buffer.xml", NULL, 0);
    printf("[2] xmlFreeDoc()\n");
    xmlFreeDoc(doc);
    printf("[2] --- end ---\n");
    return NULL;
}


valgrind --leak-check=full --show-reachable=yes --num-callers=1000 a.out

[1] --- begin ---
[1] xmlInitParser()
[1] xmlReadMemory()
[1] xmlFreeDoc()
[1] sleep()
[2] --- begin ---
[2] sleep()
[2] xmlReadMemory()
[2] xmlFreeDoc()
[2] --- end ---
[1] xmlCleanupParser()
[1] --- end ---
==21513==
==21513== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 21 from 1)
==21513== malloc/free: in use at exit: 24 bytes in 1 blocks.
==21513== malloc/free: 149 allocs, 148 frees, 52,380 bytes allocated.
==21513== For counts of detected errors, rerun with: -v
==21513== searching for pointers to 1 not-freed blocks.
==21513== checked 129,132 bytes.
==21513==
==21513== 24 bytes in 1 blocks are definitely lost in loss record 1 of 1
==21513==    at 0x4022AB8: malloc (vg_replace_malloc.c:207)
==21513==    by 0x40D8534: xmlNewMutex (in /usr/lib/libxml2.so.2.6.31)
==21513==    by 0x40D7EC6: xmlInitGlobals (in /usr/lib/libxml2.so.2.6.31)
==21513==    by 0x40D80C5: xmlInitializeGlobalState (in /usr/lib/libxml2.so.2.6.31)
==21513==    by 0x40D85FD: xmlGetGlobalState (in /usr/lib/libxml2.so.2.6.31)
==21513==    by 0x40D75BB: __xmlGenericError (in /usr/lib/libxml2.so.2.6.31)
==21513==    by 0x406CCA4: xmlInitParser (in /usr/lib/libxml2.so.2.6.31)
==21513==    by 0x8048773: thread1 (case0033.c:61)
==21513==    by 0x40314FA: start_thread (in /lib/tls/i686/cmov/libpthread-2.7.so)
==21513==    by 0x423BF5D: clone (in /lib/tls/i686/cmov/libc-2.7.so)
==21513==
==21513== LEAK SUMMARY:
==21513==    definitely lost: 24 bytes in 1 blocks.
==21513==      possibly lost: 0 bytes in 0 blocks.
==21513==    still reachable: 0 bytes in 0 blocks.
==21513==         suppressed: 0 bytes in 0 blocks.



Removing the second call to xmlInitParser() and xmlCleanupParser() removes the memory leak.
This code does NOT exploit the memory leak:


#include <stdio.h>
#include <string.h>
#include <pthread.h>
#include <unistd.h>
#include <libxml/parser.h>

char *XML_STRING="<?xml version=\"1.0\" encoding=\"UTF-8\" ?><foo></foo>";

void *thread1(void *parm);
void *thread2(void *parm);

int main(int argc, char *argv[])
{
    pthread_t t1, t2;

    pthread_create(&t1, NULL, thread1, NULL);
    pthread_create(&t2, NULL, thread2, NULL);
    pthread_join(t1, NULL);
    pthread_join(t2, NULL);

    return 0;
}

void *thread1(void *parm)
{
    xmlDocPtr doc;
    printf("[1] --- begin ---\n");
    printf("[1] xmlInitParser()\n");
    xmlInitParser();
    printf("[1] xmlReadMemory()\n");
    doc = xmlReadMemory(XML_STRING, strlen(XML_STRING), "buffer.xml", NULL, 0);
    printf("[1] xmlFreeDoc()\n");
    xmlFreeDoc(doc);
    printf("[1] sleep()\n");
    sleep(2);
    printf("[1] xmlCleanupParser()\n");
    xmlCleanupParser();
    printf("[1] --- end ---\n");
    return NULL;
}

void *thread2(void *parm)
{
    xmlDocPtr doc;
    printf("[2] --- begin ---\n");
    printf("[2] sleep()\n");
    sleep(1);
    printf("[2] xmlReadMemory()\n");
    doc = xmlReadMemory(XML_STRING, strlen(XML_STRING), "buffer.xml", NULL, 0);
    printf("[2] xmlFreeDoc()\n");
    xmlFreeDoc(doc);
    printf("[2] --- end ---\n");
    return NULL;
}


valgrind --leak-check=full --show-reachable=yes --num-callers=1000 a.out

==21540== Memcheck, a memory error detector.
==21540== Copyright (C) 2002-2007, and GNU GPL'd, by Julian Seward et al.
==21540== Using LibVEX rev 1804, a library for dynamic binary translation.
==21540== Copyright (C) 2004-2007, and GNU GPL'd, by OpenWorks LLP.
==21540== Using valgrind-3.3.0-Debian, a dynamic binary instrumentation framework.
==21540== Copyright (C) 2000-2007, and GNU GPL'd, by Julian Seward et al.
==21540== For more details, rerun with: -v
==21540==
[1] --- begin ---
[2] --- begin ---
[2] sleep()
[1] xmlInitParser()
[1] xmlReadMemory()
[1] xmlFreeDoc()
[1] sleep()
[2] xmlReadMemory()
[2] xmlFreeDoc()
[2] --- end ---
[1] xmlCleanupParser()
[1] --- end ---
==21540==
==21540== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 21 from 1)
==21540== malloc/free: in use at exit: 0 bytes in 0 blocks.
==21540== malloc/free: 75 allocs, 75 frees, 26,314 bytes allocated.
==21540== For counts of detected errors, rerun with: -v
==21540== All heap blocks were freed -- no leaks are possible.


A 24 bytes memory leak is not a big issue, but it is annoying me I am not able to remove the leak: my library 
is passive and I have no "clean way" to force the developer to call xmlInitParser() and xmlCleanupParser() in 
its main program because I am implementing "Distributed Transaction Processing: the TX (Transaction 
Demarcation) specification" and there is no a good hook for a main based initialization: the library 
initialization may happen from any thread.

Any hint is welcomed.
Regards
Ch.F.

-------------------------------------------------------------------
Decent workarounds outperform poor solutions