[libxml2.wiki] Create Memory management

[Nick Wellnhofer <nwellnhof src gnome org>]

commit 46a7a6e8e7dcd66594fe2058056843a4093fdb17
Author: Nick Wellnhofer <wellnhofer aevum de>
Date:   Sat Feb 12 18:09:36 2022 +0000

    Create Memory management

 Memory-management.md | 62 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 62 insertions(+)
---
diff --git a/Memory-management.md b/Memory-management.md
new file mode 100644
index 0000000..78a37f0
--- /dev/null
+++ b/Memory-management.md
@@ -0,0 +1,62 @@
+### General overview
+
+The module [`xmlmemory.h`](http://xmlsoft.org/html/libxml-xmlmemory.html) provides the interfaces to the 
libxml2 memory system:
+
+* libxml2 does not use the libc memory allocator directly but xmlFree(), xmlMalloc() and xmlRealloc()
+* those routines can be reallocated to a specific set of routine, by default the libc ones i.e. free(), 
malloc() and realloc()
+* the xmlmemory.c module includes a set of debugging routine
+
+### Setting libxml2 set of memory routines
+
+It is sometimes useful to not use the default memory allocator, either for debugging, analysis or to 
implement a specific behaviour on memory management (like on embedded systems). Two function calls are 
available to do so:
+
+* [xmlMemGet ()](http://xmlsoft.org/html/libxml-xmlmemory.html) which return the current set of functions in 
use by the parser
+* [xmlMemSetup()](http://xmlsoft.org/html/libxml-xmlmemory.html) which allow to set up a new set of memory 
allocation functions
+
+Of course a call to xmlMemSetup() should probably be done before calling any other libxml2 routines (unless 
you are sure your allocations routines are compatibles).
+
+### Cleaning up after using the library
+
+Libxml2 is not stateless, there is a few set of memory structures needing allocation before the parser is 
fully functional (some encoding structures for example). This also mean that once parsing is finished there 
is a tiny amount of memory (a few hundred bytes) which can be recollected if you don't reuse the library or 
any document built with it:
+
+* [xmlCleanupParser ()](http://xmlsoft.org/html/libxml-parser.html) is a centralized routine to free the 
library state and data. Note that it won't deallocate any produced tree if any (use the xmlFreeDoc() and 
related routines for this). This should be called only when the library is not used anymore.
+* [xmlInitParser ()](http://xmlsoft.org/html/libxml-parser.html) is the dual routine allowing to preallocate 
the parsing state which can be useful for example to avoid initialization reentrancy problems when using 
libxml2 in multithreaded applications
+
+Generally xmlCleanupParser() is safe assuming no parsing is ongoing and no document is still being used, if 
needed the state will be rebuild at the next invocation of parser routines (or by xmlInitParser()), but be 
careful of the consequences in multithreaded applications.
+
+### Debugging routines
+
+When configured using --with-mem-debug flag (off by default), libxml2 uses a set of memory allocation 
debugging routines keeping track of all allocated blocks and the location in the code where the routine was 
called. A couple of other debugging routines allow to dump the memory allocated infos to a file or call a 
specific routine when a given block number is allocated:
+
+* [xmlMallocLoc()](http://xmlsoft.org/html/libxml-xmlmemory.html) 
[xmlReallocLoc()](http://xmlsoft.org/html/libxml-xmlmemory.html) and 
[xmlMemStrdupLoc()](http://xmlsoft.org/html/libxml-xmlmemory.html) are the memory debugging replacement 
allocation routines
+* [xmlMemoryDump ()](http://xmlsoft.org/html/libxml-xmlmemory.html) dumps all the information about the 
allocated memory block lefts in the `.memdump` file
+
+When developing libxml2 memory debug is enabled, the tests programs call xmlMemoryDump () and the "make 
test" regression tests will check for any memory leak during the full regression test sequence, this helps a 
lot ensuring that libxml2 does not leak memory and bullet proof memory allocations use (some libc 
implementations are known to be far too permissive resulting in major portability problems!).
+
+If the .memdump reports a leak, it displays the allocation function and also tries to give some information 
about the content and structure of the allocated blocks left. This is sufficient in most cases to find the 
culprit, but not always. Assuming the allocation problem is reproducible, it is possible to find more easily:
+
+1. write down the block number xxxx not allocated
+2. export the environment variable XML_MEM_BREAKPOINT=xxxx , the easiest when using GDB is to simply give 
the command
+
+   `set environment XML_MEM_BREAKPOINT xxxx`
+
+   before running the program.
+3. run the program under a debugger and set a breakpoint on xmlMallocBreakpoint() a specific function called 
when this precise block is allocated
+4. when the breakpoint is reached you can then do a fine analysis of the allocation an step to see the 
condition resulting in the missing deallocation.
+
+I used to use a commercial tool to debug libxml2 memory problems but after noticing that it was not 
detecting memory leaks that simple mechanism was used and proved extremely efficient until now. Lately I have 
also used [valgrind](http://developer.kde.org/\~sewardj/) with quite some success, it is tied to the i386 
architecture since it works by emulating the processor and instruction set, it is slow but extremely 
efficient, i.e. it spot memory usage errors in a very precise way.
+
+### General memory requirements
+
+How much libxml2 memory require ? It's hard to tell in average it depends of a number of things:
+
+* the parser itself should work in a fixed amount of memory, except for information maintained about the 
stacks of names and entities locations. The I/O and encoding handlers will probably account for a few KBytes. 
This is true for both the XML and HTML parser (though the HTML parser need more state).
+* If you are generating the DOM tree then memory requirements will grow nearly linear with the size of the 
data. In general for a balanced textual document the internal memory requirement is about 4 times the size of 
the UTF8 serialization of this document (example the XML-1.0 recommendation is a bit more of 150KBytes and 
takes 650KBytes of main memory when parsed). Validation will add a amount of memory required for maintaining 
the external Dtd state which should be linear with the complexity of the content model defined by the Dtd
+* If you need to work with fixed memory requirements or don't need the full DOM tree then using the 
[xmlReader interface](http://xmlsoft.org/xmlreader.html) is probably the best way to proceed, it still allows 
to validate or operate on subset of the tree if needed.
+* If you don't care about the advanced features of libxml2 like validation, DOM, XPath or XPointer, don't 
use entities, need to work with fixed memory requirements, and try to get the fastest parsing possible then 
the SAX interface should be used, but it has known restrictions.
+
+### Returning memory to the kernel
+
+You may encounter that your process using libxml2 does not have a reduced memory usage although you freed 
the trees. This is because libxml2 allocates memory in a number of small chunks. When freeing one of those 
chunks, the OS may decide that giving this little memory back to the kernel will cause too much overhead and 
delay the operation. As all chunks are this small, they get actually freed but not returned to the kernel. On 
systems using glibc, there is a function call "malloc_trim" from malloc.h which does this missing operation 
(note that it is allowed to fail). Thus, after freeing your tree you may simply try "malloc_trim(0);" to 
really get the memory back. If your OS does not provide malloc_trim, try searching for a similar function.
+
+Daniel Veillard
\ No newline at end of file