Substantial memory reduction

[Joe Shaw <joeshaw novell com>]

Hi,

I just checked in the fruits of my labor from the Mono Summit.  With the
help of Ben Maurer, I identified a number of places where we were doing
some stupid things in Beagle, and with some additional info on some of
the quirkier aspects of the runtime, I was able to reduce the startup
memory usage of the main Beagle process by about 40%.

The quirkier aspects:

        * Objects of type System.Type are never collected by the GC.
        That's because these objects represent the low-level type system
        and have to be stable inside the runtime.  The dynamic nature of
        many aspects of Beagle caused thousands of these objects to be
        created.
        
        * Our default policy has been to run the Mono runtime with
        --debug.  The main Mono JIT guy, Paolo Molaro, made it clear to
        me that we shouldn't be running this as the default and that
        there aren't any guarantees this doesn't leak memory.  From some
        quick testing by Ben and myself, we seem to believe that it does
        in fact leak.
        
The simple first step is to turn off --debug to the Mono runtime.  The
downside to this is that we will no longer get line numbers in exception
logs by default, but at this point Beagle is generally stable enough
that I am willing to make this tradeoff.  (Besides, it can now be turned
back on by passing --mono-debug to the command-line.)

The second step was more involved.  Beagle dynamically loads a number of
classes to make it highly pluggable.  These include backends, filters,
and the message passing system.  To load these classes previously, we
would walk all the assemblies loaded into memory (for messages) or load
assemblies from well-known locations on disk and then walk those (for
backends and filters).  This is what created the thousands of types I
mentioned before.

Now, we have assembly attributes (essentially assembly metadata) which
specifies the classes that were previously picked out of thousands.
This metadata is usually set in an AssemblyInfo.cs file for the
assembly, and is required for these classes to be loaded.  Think of it
as registering a class.  This is necessary for filters, backends, and
message passing elements.  It's also used in the Thunderbird backend.

The memory savings are substantial.  With a non-existent ~/.beagle and
all 16 backends started:

        old: Debug: Memory usage: VmSize=46.0 MB, VmRSS=19.8 MB,  GC.GetTotalMemory=1499136
        new: Debug: Memory usage: VmSize=34.7 MB, VmRSS=12.9 MB,  GC.GetTotalMemory=1343488
        
That's 7 meg reduction in RSS size from the old to the new.  From some
heap-buddy statistics:

        old:
                         Filename: outfile.beagled
                  Allocated Bytes: 4.6M
                Allocated Objects: 80669
                              GCs: 16
                          Resizes: 10
                  Final heap size: 2.1M
                
                   Distinct Types: 375
                       Backtraces: 16768
                
                12306 live objects using 466k
                
        new:
                         Filename: outfile.beagled
                  Allocated Bytes: 2.8M
                Allocated Objects: 57490
                              GCs: 13
                          Resizes: 9
                  Final heap size: 1.5M
                
                   Distinct Types: 370
                       Backtraces: 12029
                
                8368 live objects using 423k
                
You can see we allocated 1.8 megabytes less, allocated 23k fewer
objects, resized the heap one less time, and GCed two times less.  At
shutdown time, we had 4000 fewer objects -- this number is particularly
relevant because nearly all of those 4000 objects were System.Type
instances.  The other 19,000 objects were probably duplicate instances
of the same types.

We see similar reductions in the index helper process.  I've yet to do
any long-term tests, but I'll do a full index on my system overnight and
see where we are tomorrow.  I'd like to ask people to build and run from
CVS and watch for bugs, and to keep an eye out to see to what extent
memory usage has improved.

Thanks,
Joe