asterisk users - Sep 2012 - Repeated Asterisk 10.7.0 crashes

I'm getting cycles of repeated crashes which occur and then stop occurring.
Looking at the dumps via gdb shows that something peculiar is happening
that looks like memory corruption:

Program terminated with signal 6, Aborted.
#0  0x0000003686e30285 in raise () from /lib64/libc.so.6
(gdb) up
#1  0x0000003686e31d30 in abort () from /lib64/libc.so.6
(gdb) up
#2  0x0000003686e6971b in __libc_message () from /lib64/libc.so.6
(gdb) up
#3  0x0000003686e71e7e in _int_malloc () from /lib64/libc.so.6
(gdb) up
#4  0x0000003686e7382d in calloc () from /lib64/libc.so.6
(gdb) up
#5  0x000000000054a2a0 in _ast_calloc (num_structs=1, struct_size=88, 
    field_mgr_offset=64, field_mgr_pool_offset=16, pool_size=128, 
    file=0x101010101010101 <Address 0x101010101010101 out of bounds>, 
    lineno=1235, func=0x58af9e "ast_log")
    at /usr/src/asterisk-10.7.1/include/asterisk/utils.h:495
495     AST_INLINE_API(

Once this starts happening, it seems to keep happening, but Asterisk
seems to stay up for hours between the cycles, which I can't reliably
stop from cycling.

Does anybody have any ideas how to debug this?

I suspect it may have something to do with either res_speech_lumenvox
(which I got from Lumenvox) or res_speech_unimrcp (which looks to be
extremely buggy).

----- Original Message -----
> From: "Richard Kenner" <kenner at gnat.com>
> To: asterisk-users at lists.digium.com
> Sent: Tuesday, September 4, 2012 5:56:37 PM
> Subject: [asterisk-users] Repeated Asterisk 10.7.0 crashes
> 
> I'm getting cycles of repeated crashes which occur and then stop
> occurring.
> Looking at the dumps via gdb shows that something peculiar is
> happening
> that looks like memory corruption:
> 
> Program terminated with signal 6, Aborted.
> #0  0x0000003686e30285 in raise () from /lib64/libc.so.6
> (gdb) up
> #1  0x0000003686e31d30 in abort () from /lib64/libc.so.6
> (gdb) up
> #2  0x0000003686e6971b in __libc_message () from /lib64/libc.so.6
> (gdb) up
> #3  0x0000003686e71e7e in _int_malloc () from /lib64/libc.so.6
> (gdb) up
> #4  0x0000003686e7382d in calloc () from /lib64/libc.so.6
> (gdb) up
> #5  0x000000000054a2a0 in _ast_calloc (num_structs=1, struct_size=88,
>     field_mgr_offset=64, field_mgr_pool_offset=16, pool_size=128,
>     file=0x101010101010101 <Address 0x101010101010101 out of bounds>,
>     lineno=1235, func=0x58af9e "ast_log")
>     at /usr/src/asterisk-10.7.1/include/asterisk/utils.h:495
> 495     AST_INLINE_API(
> 
> Once this starts happening, it seems to keep happening, but Asterisk
> seems to stay up for hours between the cycles, which I can't reliably
> stop from cycling.
> 
> Does anybody have any ideas how to debug this?
That definitely has the potential to be a memory corruption.  You'll need to
use a memory profiling tool to determine where the memory corruption is
happening.

There's a page on running Asterisk under valgrind on the wiki here:

https://wiki.asterisk.org/wiki/display/AST/Valgrind

Note that it will severely restrict performance of your system.  You'll want
attempt this on a test system and try to emulate the behavior leading up to
the crash on the test system.  Asterisk log files from the system that crashed
can be helpful here, as they can help you re-create the calls leading up to
the memory corruption.

Given the nature of these things, however, memory corruptions can be very
difficult to debug and resolve.  It may at least point a finger towards the
culprit.

--
Matthew Jordan
Digium, Inc. | Engineering Manager
445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
Check us out at: http://digium.com & http://asterisk.org