asterisk users - Feb 2017 - Advices when Asterisk segfaults and nothing useful in logs

On Fri, Feb 17, 2017 at 5:17 AM, Olivier <oza.4h07 at gmail.com> wrote:
> Hi George,
>
> How does ast_coredumper compare to ast_grab_core ) ?
> Is it worth learning to use both or shall favor one ?
>
> PS: As I don't know either program, yet, my question may seem silly.
> Please, forgive me for this
>
Not silly at all.

ast_grab_core actually kills asterisk to get the core file while
ast_coredumper dumps the core file and let's asterisk continue.  If
asterisk is truly deadlocked this may not matter but in some situations you
might not want to kill asterisk.

If asterisk was compiled with DEBUG_THREADS, ast_coredumper dumps the locks
table which is critical in debugging deadlock scenarios.  It does this from
the coredump, as opposed to running "core show locks", which makes the
locks table consistent with the rest of the backtraces.

ast_coiredumper can also find and process existing coredumps and it can
create a tarball containing the backtraces and lock tables.


>
> 2017-02-14 22:52 GMT+01:00 George Joseph <gjoseph at digium.com>:
>
>>
>>
>> On Tue, Feb 14, 2017 at 2:51 PM, George Joseph <gjoseph at
digium.com>
>> wrote:
>>
>>>
>>>
>>> On Tue, Feb 14, 2017 at 10:21 AM, Olivier <oza.4h07 at
gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> I've got a 13.13.1 system using PJSIP stack on debian
Jessie.
>>>> It runs from 50 to 100 simultaneous calls (so 100 to 200  PJSIP
>>>> channels) all day long.
>>>> From time to time, roughly meaning once a month, it segfaults 
with
>>>> lines (from dmesg -T output) like this:
>>>> asterisk[1160]: segfault at 7efffffe ip 00000000005881d6 sp
>>>> 00007fec95c33910 error 4 in asterisk[400000+2a2000]
>>>>
>>>>
>>>> Debug level was unfortunately not set in asterisk.conf but
verbose
>>>> level was set to 5.
>>>> Asterisk runs with:
>>>> /usr/sbin/asterisk -U asterisk -G asterisk -g
>>>>
>>>> Asterisk is compiled with DONT_OPTIMIZE and BETTER_BACKTRACES
options.
>>>>
>>>> "core show settings" outputs:
>>>> * Directories
>>>>   -------------
>>>>   Configuration file:
>>>>   Configuration directory:     /etc/asterisk
>>>>   Module directory:            /usr/lib/asterisk/modules
>>>>   Spool directory:             /var/spool/asterisk
>>>>   Log directory:               /var/log/asterisk
>>>>   Run/Sockets directory:       /var/run/asterisk
>>>>   PID file:                    /var/run/asterisk/asterisk.pid
>>>>   VarLib directory:            /var/lib/asterisk
>>>>   Data directory:              /var/lib/asterisk
>>>>   ASTDB:                       /var/lib/asterisk/astdb
>>>>   IAX2 Keys directory:         /var/lib/asterisk/keys
>>>>
>>>>
>>>>
>>>> 1. Am I correct to expect a coredump file to be produced
anytime
>>>> asterisk segfaults ?
>>>>
>>>
>>> Yes if -g is set and the user that's running asterisk has
permissions to
>>> set ulimit -c.
>>>
>>>
>>>>
>>>> 2. Does Asterisk prints any WARNING or ERROR message whenever
it
>>>> detects, at startup preferably, that it has not required
permissions to
>>>> write a coredump file ?
>>>>
>>>
>>> No because it's the system that determines where a coredump
goes and
>>> actually writes it, not asterisk.
>>> It's the sysctl kernel.core_pattern setting.
>>>
>>>
>>>>
>>>> 3. Among above directories, which one is choosen to save
coredump files
>>>> ? Is it something that can/should be configured in
/etc/asterisk (I've seen
>>>> related options in some debian  /etc/default/asterisk files but
I would be
>>>> curious to know if such things exist
>>>>
>>>
>>> See above.
>>>
>>>
>>>>
>>>> 4. Is there anything useful I can do with a line such as :
>>>> asterisk[1160]: segfault at 7efffffe ip 00000000005881d6 sp
>>>> 00007fec95c33910 error 4 in asterisk[400000+2a2000] ? Any
pointer ?
>>>>
>>>
>>> Nope.  Not a thing.  Sorry.
>>>
>>>
>>>
>>>>
>>>> 5. Suggestions ?
>>>>
>>>
>>> If you can at least get the system to write a coredump file, there
are
>>> new utilities in /var/lib/asterisk/scripts, namely ast_coredumper
which can
>>> help create the backtraces if it can at least find the core file. 
Just run
>>> "./ast_coredumper --help" for more info.   You should
also be able to use
>>> those utilities with earlier Asterisk 13 versions.
>>>
>>>
>>>
>>
>> Oh yeah, and it's on my list to publish instructions on how ot use
those
>> utilities but they were just released yesterday.
>>
>>
>>
>>>
>>>> Best regards
>>>>
>>>>
>>>> --
>>>>
_____________________________________________________________________
>>>> -- Bandwidth and Colocation Provided by
http://www.api-digital.com --
>>>>
>>>> Check out the new Asterisk community forum at:
>>>> https://community.asterisk.org/
>>>>
>>>> New to Asterisk? Start here:
>>>>      
https://wiki.asterisk.org/wiki/display/AST/Getting+Started
>>>>
>>>> asterisk-users mailing list
>>>> To UNSUBSCRIBE or update options visit:
>>>>    http://lists.digium.com/mailman/listinfo/asterisk-users
>>>>
>>>
>>>
>>>
>>> --
>>> George Joseph
>>> Digium, Inc. | Software Developer
>>> 445 Jan Davis Drive NW - Huntsville, AL 35806 - US
>>> Check us out at: www.digium.com & www.asterisk.org
>>>
>>>
>>
>>
>> --
>> George Joseph
>> Digium, Inc. | Software Developer
>> 445 Jan Davis Drive NW - Huntsville, AL 35806 - US
>> Check us out at: www.digium.com & www.asterisk.org
>>
>>
>> --
>> _____________________________________________________________________
>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>
>> Check out the new Asterisk community forum at:
>> https://community.asterisk.org/
>>
>> New to Asterisk? Start here:
>>       https://wiki.asterisk.org/wiki/display/AST/Getting+Started
>>
>> asterisk-users mailing list
>> To UNSUBSCRIBE or update options visit:
>>    http://lists.digium.com/mailman/listinfo/asterisk-users
>>
>
>
> --
> _____________________________________________________________________
> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>
> Check out the new Asterisk community forum at: https://community.asterisk.
> org/
>
> New to Asterisk? Start here:
>       https://wiki.asterisk.org/wiki/display/AST/Getting+Started
>
> asterisk-users mailing list
> To UNSUBSCRIBE or update options visit:
>    http://lists.digium.com/mailman/listinfo/asterisk-users
>


-- 
George Joseph
Digium, Inc. | Software Developer
445 Jan Davis Drive NW - Huntsville, AL 35806 - US
Check us out at: www.digium.com & www.asterisk.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.digium.com/pipermail/asterisk-users/attachments/20170217/35e4b6a2/attachment-0001.html>

2017-02-17 14:39 GMT+01:00 George Joseph <gjoseph at digium.com>:
>
>
>
> If asterisk was compiled with DEBUG_THREADS,
>
Would you then  advise to run an Asterisk server in production with
DEBUG_THREADS enabled ?
Page [1] does not mention to do so nor to avoid it.

In the production environment I'm thinking about, Asterisk is compiled with
DONT_OPTIMIZE and BETTER_BACKTRACES
hoping that if Asterisk ever fails, sysadmins would have everything needed
to track the issue no matter if those would increase resources needed to
run Asterisk properly.


[1]
https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace#GettingaBacktrace-GettingInformationForADeadlock
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.digium.com/pipermail/asterisk-users/attachments/20170217/7a8571cb/attachment.html>

On Fri, Feb 17, 2017 at 8:30 AM, Olivier <oza.4h07 at gmail.com> wrote:
>
>
> 2017-02-17 14:39 GMT+01:00 George Joseph <gjoseph at digium.com>:
>
>>
>>
>>
>> If asterisk was compiled with DEBUG_THREADS,
>>
>
> Would you then  advise to run an Asterisk server in production with
> DEBUG_THREADS enabled ?
>
No, you should not leave DEBUG_THREADS on in a normal production
environment.  Only enable it when you
are actually hunting for a deadlock.  DEBUG_THREADS causes a noticeable
drop in performance.

Richard
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.digium.com/pipermail/asterisk-users/attachments/20170217/3988eb5e/attachment.html>