Re: Memory statistics

[David Turner <dturner soremo com>]

Federico Mena Quintero a écrit :

On Sat, 2006-04-22 at 00:16 +0200, David Turner wrote:
  

Yeah :(  Andrea's patch kills most of .data from libpng, but the final
VMA is only slightly smaller.

  

I forgot to tell about the PIC tables (e.g. GOT and PLT), which generally require at least one
4-KB dirty page when linking to a library. The GOT can easily be shared (it's just a bunch of
constant pointers, after all), but the PLT is *much* more problematic because:

-  each PLT slot it is made of code that is updated lazily on usage by the dynamic linker
    (i.e. during program execution, not at load time). If you want to share them, you'd
    better be sure that this is done in a way that is safe from concurrent updates

- due to the way ELF linking works, two identical PLT entries from two distinct processes
   will not necessarily resolve to the same code point ! Poof goes our sharing !!

Oh, and I may be forgetting some other PIC tables, it's been some time since I didn't grab
all the details related to dynamic linking... :-)

... or on OpenOffice.  Michael "I want to be like him when I grow up -
oh, wait, he's younger than me" Meeks has been kicking some ass in that
front.

  

I thought his job was about speeding the symbol lookup process, not the page sharing bit. If not, that
would be a good surprise.

It'd be nice if the dynamic linker was able to automatically share the
updated pointers between
processes which map the same libraries to the same virtual addresses,
but I don't think this has been
implemented yet, since it would require a new ELF section, as well as
modifications to the
compilers.
    

Thinking about it, the compilers already have to place the constant pointers in a specific section, so
changing them shouldn't be too difficult. The linkers are different beasts though.

Wouldn't something like this need to happen:

	1. program gets run; linker does the fixups

	2. linker writes "libfoo.so got put at 0x12345678, here is a 
           list of resolved relocations: <...>"

	3. another program gets run.

	4. linker looks in its archive from (2) to see if it had
	   already resolved libfoo.so at some address.

Sounds like pain.

  

Well, the concept isn't terribly difficult, I guess the implementation shall reveal lots of hideous
detail though ;-)

How do they handle the problem of agreeing on the base addresses for
various libraries?  We have a million libraries in Linux :)

  

First of all, there aren't a million libraries, and there are also tons of DLLs in Windows.

Second, you don't need to use a single address space to rebase all libraries, since you
won't find a program that needs to link to all of them. What I mean is that it's possible to
rebase OpenOffice, FireFox or X.Org shared libraries to the same virtual address, since the chance
of having them linked by the same process is extremely low.

So you end up doing some profiling to see what the "core" shared libraries are, and group them
together for the rebasing operation.

Apart from that, you'll get relocations.

By the way, I'm not saying that this scheme would solve all problems, I'm just trying to provide
directions and ideas. I'm interested however in any specific point that would show why this couldn't
be implemented for specific projects like OLPC, which probably don't need all the whiz-bang
features (and misfeatures) of ELF dynamic linking.

Regards,

- David

  Federico