Re: [xml] Suggesting something like malloc_trim after freeing

[Daniel Veillard <veillard redhat com>]

On Fri, Feb 01, 2008 at 04:58:51PM +0100, Wolfram Sang wrote:
> Hello,

  Hi Wolfram,

> we had the problem, that an embedded system ran out of memory after
> parsing a huge XML-file. The problem was, that most of the memory was
> only returned to the kernel after the process was terminated. When I
> searched the xmlsoft-site, I found people having similar problems,
> only a solution could not be provided (Example:
> http://mail.gnome.org/archives/xml/2005-March/msg00022.html )
>
> Of course, there is NO memory leak and references to OS memory
> management are correct. Still, there is a way (at least with glibc) to
> enforce giving back the memory: malloc_trim
>
> I am very sceptical if this functions is portable enough to be used
> within libxml2.

 Actually portability wouldn't concern me that much, that could be tested
automatically, what makes it not acceptable is the amount of inpredictability
this would bring to the API. memory compation can be fairly expensive, and
calling it explicitely within one of libxml2 entry point could turn into
a serious problem. This is best left to the application programmer when
it actually needs it.

> Still, I would suggest mentioning it in the docs
> ("Memory management"?), as this might point users into the right
> direction. A paragraph could look like this:

  Now, that's a very good point. This wasn't raised that many time but
i can understand how it could be useful to some users, agreed !

> ===
>
> Returning memory to the kernel
>
> You may encounter that your process using libxml2 does not have a
> reduced memory usage although you freed the tree. This is because
> libxml2 allocates memory in a number of small chunks. When freeing one
> of those chunks, the OS may decide that giving this little memory back
> to the kernel will cause too much overhead and delay the operation. As
> all chunks are this small, they get actually freed but not returned to
> the kernel. On systems using glibc, there is a function call
> "malloc_trim" from malloc.h which does this missing operation (note that
> it is allowed to fail). Thus, after freeing your tree you may simply try
> "malloc_trim(0);" to really get the memory back. If your OS does not
> provide malloc_trim, try searching for a similar function.
>
> ===

  Great !
I am adding it to SVN, it should show up on the web site soon,
thanks a lot for the suggestion and the wording,

Daniel

-- 
Red Hat Virtualization group http://redhat.com/virtualization/
Daniel Veillard      | virtualization library  http://libvirt.org/
veillard redhat com  | libxml GNOME XML XSLT toolkit  http://xmlsoft.org/
http://veillard.com/ | Rpmfind RPM search engine  http://rpmfind.net/