pango opentype - huge memory waste (DejaVu font)

[nsf <no smile face gmail com>]

Hi. I was experimenting with gtk hello world app and notices that it
eats a lot of memory.

--------------------------------------------------------------------
[nsf @ ~]$ pmap -d `pidof hello`
4397:   ./hello
Address   Kbytes Mode  Offset           Device    Mapping
08048000       4 r-x-- 0000000000000000 003:00004 hello
08049000       4 rwx-- 0000000000000000 003:00004 hello
08192000    1456 rwx-- 0000000008192000 000:00000   [ anon ]
b6be7000    6240 rwx-- 00000000b6be7000 000:00000   [ anon ]
b71ff000     608 r-x-- 0000000000000000 003:00003 DejaVuSans.ttf
b7297000       8 r-x-- 0000000000000000 003:00003 pango-basic-fc.so
< ... skipped ... >
mapped: 22608K    writeable/private: 8088K    shared: 764K
--------------------------------------------------------------------

Notice this anonymous memory blob which eats 6 megs. I started to
look for a problem. And memory allocs profiling got me to "_hb_alloc"
function with that kind of stack:

--------------------------------------------------------------------
Leaked 0xb703a008 (262140 bytes)
	_hb_alloc()
	_HB_OPEN_Load_ClassDefinition()
	_HB_OPEN_Load_EmptyOrClassDefinition()
	Load_ChainContextSubst2()
	.L1434()
	_HB_OPEN_Load_LookupList()
	HB_Load_GSUB_Table()
	pango_ot_info_get_gsub()
	get_tables()
	pango_ot_info_find_script()
	pango_ot_ruleset_new_for()
	pango_ot_ruleset_new_from_description()
	pango_ot_ruleset_get_for_description()
	basic_engine_shape()
	_pango_engine_shape_shape()
	pango_shape()
	shape_run()
	process_item()
	pango_layout_check_lines()
< ... skipped ... >
	main()
	__libc_start_main()
	_start()
--------------------------------------------------------------------

There were few memory allocs with size of 262140 bytes, which is 
(4 * 65535) exactly. I kept digging and found out where it was:
pango/opentype/harfbuzz-open.c
pango/opentype/harfbuzz-gsub.c
pango/opentype/harfbuzz-gpos.c

The problem is that there is an allocation of memory "enough" for
loading something (see function Load_ChainContextSubst2()), but there is
no realloc later. I checked the real usage of memory and it was
something like: 400 bytes out of total 262140! I mean literally it's a
waste of 256 kilobytes per such table alloc (or whatever it is). 

So, I did a quick fix (patch attached) in function
_HB_OPEN_Load_ClassDefinition(). I added a realloc with a real used
size of array. And puff.. the problem is gone. I did LD_PRELOAD and
here is the results:

--------------------------------------------------------------------
[nsf @ ~]$ pmap -d `pidof hello`
21033:   ./hello
Address   Kbytes Mode  Offset           Device    Mapping
08048000       4 r-x-- 0000000000000000 003:00004 hello
08049000       4 rwx-- 0000000000000000 003:00004 hello
0888c000    1432 rwx-- 000000000888c000 000:00000   [ anon ]
b71ab000     608 r-x-- 0000000000000000 003:00003 DejaVuSans.ttf
b7243000       8 r-x-- 0000000000000000 003:00003 pango-basic-fc.so
< ... skipped ... >
mapped: 16348K    writeable/private: 1824K    shared: 764K
--------------------------------------------------------------------

No more strange anonymous bloated chunks. After that I did few tests of
course. I tried different apps like: linuxdcpp, dia, sonata,
claws-mail, midori (webkit-gtk) and everything works. I tried to reboot
and load few programs with patch and then reboot again and do the same 
without patch, memory usage was something like 130 megs versus 200 megs
(~30%). And I couldn't believe, but!

Later I found out that this issue is valid only for DejaVuSans.ttf font
for some reason (I didn't check other fonts). For example Bitstream
Vera Sans doesn't cause that bloat.

Also I'm not sure is my hack correct and bug free. But it clearly works
on my machine. Patch was made against SVN version of pango and affects
only opentype component (one file, 3 functions).

Well, I don't know, maybe someone can comment that issue? :)
Thanks for attention.

nsf





--- pango-old/pango/opentype/harfbuzz-open.c	2009-01-22 12:55:54.000000000 +0500
+++ pango/pango/opentype/harfbuzz-open.c	2009-01-22 12:52:52.000000000 +0500
@@ -934,7 +934,7 @@
 /* ClassDefFormat1 */
 
 static HB_Error  Load_ClassDef1( HB_ClassDefinition*  cd,
-				 HB_UShort             limit,
+				 HB_UShort*            limit,
 				 HB_Stream             stream )
 {
   HB_Error   error;
@@ -970,21 +970,26 @@
   d   = cd->Defined;
   cva = cdf1->ClassValueArray;
 
+  HB_UShort max = 0;
+
   if ( ACCESS_Frame( count * 2L ) )
     goto Fail;
 
   for ( n = 0; n < count; n++ )
   {
     cva[n] = GET_UShort();
-    if ( cva[n] >= limit )
+    if ( cva[n] >= *limit )
     {
       error = ERR(HB_Err_Invalid_SubTable);
       goto Fail;
     }
     d[cva[n]] = TRUE;
+    if (max < cva[n])
+      max = cva[n];
   }
 
   FORGET_Frame();
+  *limit = max;
 
   return HB_Err_Ok;
 
@@ -1004,7 +1009,7 @@
 /* ClassDefFormat2 */
 
 static HB_Error  Load_ClassDef2( HB_ClassDefinition*  cd,
-				 HB_UShort             limit,
+				 HB_UShort*            limit,
 				 HB_Stream             stream )
 {
   HB_Error   error;
@@ -1038,6 +1043,8 @@
   if ( ACCESS_Frame( count * 6L ) )
     goto Fail;
 
+  HB_UShort max = 0;
+
   for ( n = 0; n < count; n++ )
   {
     crr[n].Start = GET_UShort();
@@ -1047,7 +1054,7 @@
     /* sanity check */
 
     if ( crr[n].Start > crr[n].End ||
-	 crr[n].Class >= limit )
+	 crr[n].Class >= *limit )
     {
       /* XXX
        * Corrupt entry.  Skip it.
@@ -1056,13 +1063,17 @@
        n--;
        count--;
     }
-    else
+    else {
       d[crr[n].Class] = TRUE;
+      if (max < crr[n].Class) 
+        max = crr[n].Class;
+    }
   }
 
   FORGET_Frame();
 
   cdf2->ClassRangeCount = count;
+  *limit = max;
 
   return HB_Err_Ok;
 
@@ -1100,14 +1111,17 @@
 
   switch ( cd->ClassFormat )
   {
-  case 1:  error = Load_ClassDef1( cd, limit, stream ); break;
-  case 2:  error = Load_ClassDef2( cd, limit, stream ); break;
+  case 1:  error = Load_ClassDef1( cd, &limit, stream ); break;
+  case 2:  error = Load_ClassDef2( cd, &limit, stream ); break;
   default: error = ERR(HB_Err_Invalid_SubTable_Format);	break;
   }
 
   if ( error )
     goto Fail;
 
+  if ( REALLOC_ARRAY( cd->Defined, limit, HB_Bool ) )
+    goto Fail;
+
   cd->loaded = TRUE;
 
   return HB_Err_Ok;