glib memory allocation problems

[Dimitrios Apostolou <jimis gmx net>]

Hello glib developers, 

I have been hitting an sshfs crash very often and according to Miklos Szeredi 
(of FUSE and sshfs fame) it is very probable that glib is responsible, and in 
particular the g_slice allocation routines when multiple threads are 
involved. Unfortunately the crashes don't always take place at the same 
place. 

Anyway, trying to produce some useful debugging info for you, I compiled glib 
2.13.1 with debugging symbols. Please find attached the backtrace of a crash. 
I have kept the core file in case you need me to run specific gdb commands. 

FYI the crash takes some time to reproduce, it happens after about 12-24 hours 
of continuous full load operation of sshfs. However I have been running 
exactly the same test case as the attached file, with the slight difference 
of G_SLICE=always-malloc for more than 48 hours and the crash hasn't occured 
yet. Nevertheless, I think that another problem has surfaced: running sshfs 
that way shows possible memory leaks in glib. The VSZ of the process has 
surpassed 350MB and keeps growing at a constant rate, which didn't happen 
before. Is it normal for glib to allocate that much more memory when using 
G_SLICE=always-malloc? 

I think the next step would be to run sshfs with G_SLICE=debug-blocks. Before 
doing that and because it takes some time to reproduce, is the backtrace 
attached useful to you? 


Thanks in advance, 
Dimitris

*** glibc detected *** /usr/bin/sshfs: free(): invalid pointer: 0xb1dd3130 ***
[New Thread -1280468080 (LWP 23439)]
[New Thread -1299186800 (LWP 23440)]
[New Thread -1236694128 (LWP 23441)]
======= Backtrace: =========
/lib/libc.so.6[0xb7d67cd0]
/lib/libc.so.6(__libc_free+0x89)[0xb7d69369]
/usr/lib/libglib-2.0.so.0(g_free+0x35)[0xb7e81491]
/usr/lib/libglib-2.0.so.0[0xb7e69deb]
/usr/lib/libglib-2.0.so.0[0xb7e698c3]
/usr/lib/libglib-2.0.so.0(g_hash_table_foreach_remove+0x95)[0xb7e696e3]
/usr/bin/sshfs[0x804f052]
/usr/bin/sshfs[0x804f29d]
/usr/bin/sshfs[0x804f4f4]
/usr/lib/libfuse.so.2[0xb7f1feba]
/usr/lib/libfuse.so.2[0xb7f2019e]
/usr/lib/libfuse.so.2[0xb7f20348]
/usr/lib/libfuse.so.2[0xb7f26091]
/usr/lib/libfuse.so.2[0xb7f26aed]
/usr/lib/libfuse.so.2(fuse_session_process+0x26)[0xb7f28326]
/usr/lib/libfuse.so.2[0xb7f2471b]
/lib/libpthread.so.0[0xb7e344a2]
/lib/libc.so.6(__clone+0x5e)[0xb7dc352e]
======= Memory map: ========
08048000-08053000 r-xp 00000000 03:41 854742     /usr/bin/sshfs
08053000-08054000 rw-p 0000a000 03:41 854742     /usr/bin/sshfs
08054000-094e9000 rw-p 08054000 00:00 0          [heap]
adcfe000-adcff000 ---p adcfe000 00:00 0
adcff000-ae4ff000 rw-p adcff000 00:00 0
ae4ff000-ae500000 ---p ae4ff000 00:00 0
ae500000-aed21000 rw-p ae500000 00:00 0
aed21000-aee00000 ---p aed21000 00:00 0
aee00000-aeef1000 rw-p aee00000 00:00 0
aeef1000-aef00000 ---p aeef1000 00:00 0
aef00000-aeffe000 rw-p aef00000 00:00 0
aeffe000-af000000 ---p aeffe000 00:00 0
af100000-af200000 rw-p af100000 00:00 0
af300000-af500000 rw-p af300000 00:00 0
af500000-af5fb000 rw-p af500000 00:00 0
af5fb000-af600000 ---p af5fb000 00:00 0
af700000-af7ff000 rw-p af700000 00:00 0
af7ff000-af800000 ---p af7ff000 00:00 0
af900000-afb00000 rw-p af900000 00:00 0
afb00000-afbf6000 rw-p afb00000 00:00 0
afbf6000-afc00000 ---p afbf6000 00:00 0
afc00000-afcf1000 rw-p afc00000 00:00 0
afcf1000-afd00000 ---p afcf1000 00:00 0
afd00000-afdfc000 rw-p afd00000 00:00 0
afdfc000-afe00000 ---p afdfc000 00:00 0
aff00000-afff9000 rw-p aff00000 00:00 0
afff9000-b0000000 ---p afff9000 00:00 0
b0000000-b0100000 rw-p b0000000 00:00 0
b0100000-b01e9000 rw-p b0100000 00:00 0
b01e9000-b0200000 ---p b01e9000 00:00 0
b0200000-b02f6000 rw-p b0200000 00:00 0
b02f6000-b0300000 ---p b02f6000 00:00 0
b0300000-b03fe000 rw-p b0300000 00:00 0
b03fe000-b0400000 ---p b03fe000 00:00 0
b04ff000-b0500000 ---p b04ff000 00:00 0
b0500000-b0dfd000 rw-p b0500000 00:00 0
b0dfd000-b0e00000 ---p b0dfd000 00:00 0
b0eff000-b0f00000 ---p b0eff000 00:00 0
b0f00000-b17f9000 rw-p b0f00000 00:00 0
b17f9000-b1800000 ---p b17f9000 00:00 0
b1b00000-b1bce000 rw-p b1b00000 00:00 0
b1bce000-b1c00000 ---p b1bce000 00:00 0
b1d00000-b1de6000 rw-p b1d00000 00:00 0
b1de6000-b1e00000 ---p b1de6000 00:00 0
b1e00000-b1f00000 rw-p b1e00000 00:00 0
b1f00000-b1ff2000 rw-p b1f00000 00:00 0
b1ff2000-b2000000 ---p b1ff2000 00:00 0
b20ff000-b2100000 ---p b20ff000 00:00 0
b2100000-b29f4000 rw-p b2100000 00:00 0
b29f4000-b2a00000 ---p b29f4000 00:00 0
b2ab6000-b2ab7000 ---p b2ab6000 00:00 0
b2ab7000-b32b7000 rw-p b2ab7000 00:00 0
b32d9000-b32da000 ---p b32d9000 00:00 0
b32da000-b3ada000 rw-p b32da000 00:00 0
b3afc000-b3afd000 ---p b3afc000 00:00 0
b3afd000-b42fd000 rw-p b3afd000 00:00 0
b42fd000-b42fe000 ---p b42fd000 00:00 0
b42fe000-b4afe000 rw-p b42fe000 00:00 0
b4afe000-b4aff000 ---p b4afe000 00:00 0
b4aff000-b52ff000 rw-p b4aff000 00:00 0
b52ff000-b5300000 ---p b52ff000 00:00 0
b5300000-b5bfe000 rw-p b5300000 00:00 0
b5bfe000-b5c00000 ---p b5bfe000 00:00 0
b5c98000-b5c99000 ---p b5c98000 00:00 0
b5c99000-b6499000 rw-p b5c99000 00:00 0
b64bb000-b64bc000 ---p b64bb000 00:00 0
b64bc000-b6cbc000 rw-p b64bc000 00:00 0
b6cbc000-b6cbd000 ---p b6cbc000 00:00 0
b6cbd000-b74bd000 rw-p b6cbd000 00:00 0
b74df000-b74e0000 ---p b74df000 00:00 0
b74e0000-b7ce0000 rw-p b74e0000 00:00 0
b7cf7000-b7d01000 r-xp 00000000 03:41 849539     /usr/lib/libgcc_s.so.1
b7d01000-b7d02000 rw-p 00009000 03:41 849539     /usr/lib/libgcc_s.so.1
b7d02000-b7d04000 rw-p b7d02000 00:00 0
b7d04000-b7e29000 r-xp 00000000 03:41 130450     /lib/libc-2.5.so
b7e29000-b7e2a000 r--p 00125000
Program received signal SIGABRT, Aborted.
[Switching to Thread -1219769456 (LWP 23271)]
0xb7f36410 in __kernel_vsyscall ()


(gdb) bt           
#0  0xb7f36410 in __kernel_vsyscall ()
#1  0xb7d2c721 in raise () from /lib/libc.so.6
#2  0xb7d2def8 in abort () from /lib/libc.so.6
#3  0xb7d6205b in __libc_message () from /lib/libc.so.6
#4  0xb7d67cd0 in malloc_printerr () from /lib/libc.so.6
#5  0xb7d69369 in free () from /lib/libc.so.6
#6  0xb7e81491 in IA__g_free (mem=0xb1dd3130) at gmem.c:187
#7  0xb7e69deb in g_hash_node_destroy (hash_node=0xb1f6c080, key_destroy_func=0x80499e4 <g_free plt>, 
    value_destroy_func=0x804ef9c <free_node>) at ghash.c:845
#8  0xb7e698c3 in g_hash_table_foreach_remove_or_steal (hash_table=0x8055220, func=0x804efc3 <cache_clean_entry>, user_data=0xb74bc144, 
    notify=1) at ghash.c:637
#9  0xb7e696e3 in IA__g_hash_table_foreach_remove (hash_table=0x8055220, func=0x804efc3 <cache_clean_entry>, user_data=0xb74bc144)
    at ghash.c:576
#10 0x0804f052 in cache_clean () at cache.c:74
#11 0x0804f29d in cache_add_attr (path=0xb1fbb0c0 "/blahblah/blahblah_08-19_[blahblue-blahs]", stbuf=0xb74bc26c) at cache.c:152
#12 0x0804f4f4 in cache_getattr (path=0xb1fbb0c0 "/blahblah/blahblah_08-19_[blahblue-blahs]", stbuf=0xb74bc26c) at cache.c:220
#13 0xb7f1feba in fuse_do_getattr () from /usr/lib/libfuse.so.2
#14 0xb7f2019e in lookup_path () from /usr/lib/libfuse.so.2
#15 0xb7f20348 in fuse_lookup () from /usr/lib/libfuse.so.2
#16 0xb7f26091 in do_lookup () from /usr/lib/libfuse.so.2
#17 0xb7f26aed in fuse_ll_process () from /usr/lib/libfuse.so.2
#18 0xb7f28326 in fuse_session_process () from /usr/lib/libfuse.so.2
#19 0xb7f2471b in fuse_do_work () from /usr/lib/libfuse.so.2
#20 0xb7e344a2 in start_thread () from /lib/libpthread.so.0
#21 0xb7dc352e in clone () from /lib/libc.so.6


(gdb) bt full
#0  0xb7f36410 in __kernel_vsyscall ()
No symbol table info available.
#1  0xb7d2c721 in raise () from /lib/libc.so.6
No symbol table info available.
#2  0xb7d2def8 in abort () from /lib/libc.so.6
No symbol table info available.
#3  0xb7d6205b in __libc_message () from /lib/libc.so.6
No symbol table info available.
#4  0xb7d67cd0 in malloc_printerr () from /lib/libc.so.6
No symbol table info available.
#5  0xb7d69369 in free () from /lib/libc.so.6
No symbol table info available.
#6  0xb7e81491 in IA__g_free (mem=0xb1dd3130) at gmem.c:187
No locals.
#7  0xb7e69deb in g_hash_node_destroy (hash_node=0xb1f6c080, key_destroy_func=0x80499e4 <g_free plt>, 
    value_destroy_func=0x804ef9c <free_node>) at ghash.c:845
No locals.
#8  0xb7e698c3 in g_hash_table_foreach_remove_or_steal (hash_table=0x8055220, func=0x804efc3 <cache_clean_entry>, user_data=0xb74bc144, 
    notify=1) at ghash.c:637
        node = (GHashNode *) 0xb1f6c080
        prev = (GHashNode *) 0x0
        i = 1837
        deleted = 4867
#9  0xb7e696e3 in IA__g_hash_table_foreach_remove (hash_table=0x8055220, func=0x804efc3 <cache_clean_entry>, user_data=0xb74bc144)
    at ghash.c:576
        __PRETTY_FUNCTION__ = "IA__g_hash_table_foreach_remove"
#10 0x0804f052 in cache_clean () at cache.c:74
        now = 1178589578
#11 0x0804f29d in cache_add_attr (path=0xb1fbb0c0 "/blahblah/blahblah_08-19_[blahblue-blahs]", stbuf=0xb74bc26c) at cache.c:152
        node = (struct node *) 0x8e4b368
        now = 1178589578
#12 0x0804f4f4 in cache_getattr (path=0xb1fbb0c0 "/blahblah/blahblah_08-19_[blahblue-blahs]", stbuf=0xb74bc26c) at cache.c:220
        err = 0
#13 0xb7f1feba in fuse_do_getattr () from /usr/lib/libfuse.so.2
No symbol table info available.
#14 0xb7f2019e in lookup_path () from /usr/lib/libfuse.so.2
No symbol table info available.
#15 0xb7f20348 in fuse_lookup () from /usr/lib/libfuse.so.2
No symbol table info available.
#16 0xb7f26091 in do_lookup () from /usr/lib/libfuse.so.2
No symbol table info available.
#17 0xb7f26aed in fuse_ll_process () from /usr/lib/libfuse.so.2
No symbol table info available.
#18 0xb7f28326 in fuse_session_process () from /usr/lib/libfuse.so.2
No symbol table info available.
#19 0xb7f2471b in fuse_do_work () from /usr/lib/libfuse.so.2
No symbol table info available.
#20 0xb7e344a2 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#21 0xb7dc352e in clone () from /lib/libc.so.6
No symbol table info available.