Re: [xml] Memory not going away..

["David W. Bauer Jr." <bauerd cs rpi edu>]

One thing that I should mention is that the memory is NOT being released.
I know this for a fact becuase my application goes on (afer freeing the
xmlDoc) and allocates until the 3GB limit is reached.

So, what I see is this:

        read / parse the file

                --> memory consumed goes to 2.9GB

        free the xmlDoc

                --> memory consumption reported is still the same

        alloc .1 GB more data structures, then malloc fails due to
        process memory limit.

So, the xmlDoc is NOT being freed in any way.

The purpose of the test case was to determine if the system would report
the memory being freed with _fewer_ calls to the memory library, which is
a valid approach since it isolates only the problem being addressed.  If I
were to read/free the doc, then attempt to allocate more memory, it fails
since the doc is still in memory.

One major difference is that I am now getting info in the .memdump file,
which I did not get in my test script:

      04:32:22 PM

      MEMORY ALLOCATED : 32092, MAX was 212704970
BLOCK  NUMBER   SIZE  TYPE
0      12474291     40 malloc()  in none(0) pointer to #12403446 at index
4
1      12403491   2560 realloc() in none(0)
2      12403446     12 malloc()  in none(0) pointer to #12403491 at index
8
3      12400556     40 malloc()  in none(0) pointer to #12395663 at index
4
4      12395663     12 malloc()  in none(0)
5      12395280     40 malloc()  in none(0) pointer to #12180892 at index
4
6      12181821   5120 realloc() in none(0)
7      12180892     12 malloc()  in none(0) pointer to #12181821 at index
8
8      12175534     40 malloc()  in none(0) pointer to #12159688 at index
4
9      12159767    640 realloc() in none(0)
10     12159688     12 malloc()  in none(0) pointer to #12159767 at index
8
11     12159018     40 malloc()  in none(0) pointer to #12028340 at index
4
12     12028402  10240 realloc() in none(0)
13     12028340     12 malloc()  in none(0) pointer to #12028402 at index
8
14     12022826     40 malloc()  in none(0) pointer to #12014020 at index
4
15     12014405    160 realloc() in none(0)
16     12014020     12 malloc()  in none(0) pointer to #12014405 at index
8
17     12013492     40 malloc()  in none(0) pointer to #12004089 at index
4
18     12004115   1280 realloc() in none(0)
19     12004089     12 malloc()  in none(0) pointer to #12004115 at index
8
20     12003623     40 malloc()  in none(0) pointer to #11804873 at index
4
21     11804899  10240 realloc() in none(0)
22     11804873     12 malloc()  in none(0) pointer to #11804899 at index
8
23     11798553     40 malloc()  in none(0) pointer to #11792193 at index
4
24     11792391    640 realloc() in none(0)
25     11792193     12 malloc()  in none(0) pointer to #11792391 at index
8
26     11791913     40 malloc()  in none(0) pointer to #11779712 at index
4
27     11779995    640 realloc() in none(0)
28     11779712     12 malloc()  in none(0) "V"
29     9566790     40 malloc()  in none(0) pointer to #9566785 at index 4
30     9566785     12 malloc()  in none(0)

David Bauer

Ph.D Candidate, Rensselaer Polytechnic Institute
Computer Science Department

Office: Amos Eaton 205
Phone : (518) 276 2657
URI   : http://www.cs.rpi.edu/~bauerd

On Tue, 8 Mar 2005, Daniel Veillard wrote:

> On Tue, Mar 08, 2005 at 05:32:20PM -0500, David W. Bauer Jr. wrote:
> > Yes, but then I should be able to get that memory back in future calls to
> > (m/c)alloc.  The real problem I am fighting with here is that my XML files
> > are large (10^8 bytes), and I am unable to parse the entire file with
> > xmlParseFile _and_ create my data structures.  I realize that I should be
> > using some form of reader, but all I really need is for the xmlDoc to
> > release the memory on calls to xmlFreeDoc.. which, it is still not doing
> > after all this time.
>
>    it *does* !!!! Take a debugger and put a breakpoint in your free()
> routines, you will see that it is called :-(
>
> > I wrote a test script which tests the basic behaviour, and the doc is
>
>    The testing methodology is flawed, the libc memory allocation routine
> may just compact and free the extra heap because you just added more calls
> to malloc() or free().
>
> > freed.. but when I carry over the solution to my code, the file will not
> > go away.  The .memdump IS listing items now, and I am struggling to figure
> > out why these objects are not being removed.
> >
> > Coincidentally, if I have system with 3 GB of RAM, and I allocate and free
> > 3GB of ram in my process, but then do not exit.. any other processes would
> > be forced into swap waiting for my process to complete and free up
> > resources (namely, 3GB of RAM).  In Linux, calls to free in the process
> > return the memory to the OS.
>
>    not instantly, this is wrong, you don't get a syscall per call to free().
> There is something else going on, I can't tell what, and I suspect your
> testing methodology based on OS lookup to not match what happen at the
> libc interface.
>
> Daniel
>
> --
> Daniel Veillard      | Red Hat Desktop team http://redhat.com/
> veillard redhat com  | libxml GNOME XML XSLT toolkit  http://xmlsoft.org/
> http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/