George Joseph
2017-Feb-14 21:52 UTC
[asterisk-users] Advices when Asterisk segfaults and nothing useful in logs
On Tue, Feb 14, 2017 at 2:51 PM, George Joseph <gjoseph at digium.com> wrote:> > > On Tue, Feb 14, 2017 at 10:21 AM, Olivier <oza.4h07 at gmail.com> wrote: > >> Hello, >> >> I've got a 13.13.1 system using PJSIP stack on debian Jessie. >> It runs from 50 to 100 simultaneous calls (so 100 to 200 PJSIP channels) >> all day long. >> From time to time, roughly meaning once a month, it segfaults with lines >> (from dmesg -T output) like this: >> asterisk[1160]: segfault at 7efffffe ip 00000000005881d6 sp >> 00007fec95c33910 error 4 in asterisk[400000+2a2000] >> >> >> Debug level was unfortunately not set in asterisk.conf but verbose level >> was set to 5. >> Asterisk runs with: >> /usr/sbin/asterisk -U asterisk -G asterisk -g >> >> Asterisk is compiled with DONT_OPTIMIZE and BETTER_BACKTRACES options. >> >> "core show settings" outputs: >> * Directories >> ------------- >> Configuration file: >> Configuration directory: /etc/asterisk >> Module directory: /usr/lib/asterisk/modules >> Spool directory: /var/spool/asterisk >> Log directory: /var/log/asterisk >> Run/Sockets directory: /var/run/asterisk >> PID file: /var/run/asterisk/asterisk.pid >> VarLib directory: /var/lib/asterisk >> Data directory: /var/lib/asterisk >> ASTDB: /var/lib/asterisk/astdb >> IAX2 Keys directory: /var/lib/asterisk/keys >> >> >> >> 1. Am I correct to expect a coredump file to be produced anytime asterisk >> segfaults ? >> > > Yes if -g is set and the user that's running asterisk has permissions to > set ulimit -c. > > >> >> 2. Does Asterisk prints any WARNING or ERROR message whenever it detects, >> at startup preferably, that it has not required permissions to write a >> coredump file ? >> > > No because it's the system that determines where a coredump goes and > actually writes it, not asterisk. > It's the sysctl kernel.core_pattern setting. > > >> >> 3. Among above directories, which one is choosen to save coredump files ? >> Is it something that can/should be configured in /etc/asterisk (I've seen >> related options in some debian /etc/default/asterisk files but I would be >> curious to know if such things exist >> > > See above. > > >> >> 4. Is there anything useful I can do with a line such as : >> asterisk[1160]: segfault at 7efffffe ip 00000000005881d6 sp >> 00007fec95c33910 error 4 in asterisk[400000+2a2000] ? Any pointer ? >> > > Nope. Not a thing. Sorry. > > > >> >> 5. Suggestions ? >> > > If you can at least get the system to write a coredump file, there are new > utilities in /var/lib/asterisk/scripts, namely ast_coredumper which can > help create the backtraces if it can at least find the core file. Just run > "./ast_coredumper --help" for more info. You should also be able to use > those utilities with earlier Asterisk 13 versions. > > >Oh yeah, and it's on my list to publish instructions on how ot use those utilities but they were just released yesterday.> >> Best regards >> >> >> -- >> _____________________________________________________________________ >> -- Bandwidth and Colocation Provided by http://www.api-digital.com -- >> >> Check out the new Asterisk community forum at: >> https://community.asterisk.org/ >> >> New to Asterisk? Start here: >> https://wiki.asterisk.org/wiki/display/AST/Getting+Started >> >> asterisk-users mailing list >> To UNSUBSCRIBE or update options visit: >> http://lists.digium.com/mailman/listinfo/asterisk-users >> > > > > -- > George Joseph > Digium, Inc. | Software Developer > 445 Jan Davis Drive NW - Huntsville, AL 35806 - US > Check us out at: www.digium.com & www.asterisk.org > >-- George Joseph Digium, Inc. | Software Developer 445 Jan Davis Drive NW - Huntsville, AL 35806 - US Check us out at: www.digium.com & www.asterisk.org -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.digium.com/pipermail/asterisk-users/attachments/20170214/9ce3fd86/attachment.html>
Olivier
2017-Feb-17 12:17 UTC
[asterisk-users] Advices when Asterisk segfaults and nothing useful in logs
Hi George, How does ast_coredumper compare to ast_grab_core ) ? Is it worth learning to use both or shall favor one ? PS: As I don't know either program, yet, my question may seem silly. Please, forgive me for this 2017-02-14 22:52 GMT+01:00 George Joseph <gjoseph at digium.com>:> > > On Tue, Feb 14, 2017 at 2:51 PM, George Joseph <gjoseph at digium.com> wrote: > >> >> >> On Tue, Feb 14, 2017 at 10:21 AM, Olivier <oza.4h07 at gmail.com> wrote: >> >>> Hello, >>> >>> I've got a 13.13.1 system using PJSIP stack on debian Jessie. >>> It runs from 50 to 100 simultaneous calls (so 100 to 200 PJSIP >>> channels) all day long. >>> From time to time, roughly meaning once a month, it segfaults with >>> lines (from dmesg -T output) like this: >>> asterisk[1160]: segfault at 7efffffe ip 00000000005881d6 sp >>> 00007fec95c33910 error 4 in asterisk[400000+2a2000] >>> >>> >>> Debug level was unfortunately not set in asterisk.conf but verbose level >>> was set to 5. >>> Asterisk runs with: >>> /usr/sbin/asterisk -U asterisk -G asterisk -g >>> >>> Asterisk is compiled with DONT_OPTIMIZE and BETTER_BACKTRACES options. >>> >>> "core show settings" outputs: >>> * Directories >>> ------------- >>> Configuration file: >>> Configuration directory: /etc/asterisk >>> Module directory: /usr/lib/asterisk/modules >>> Spool directory: /var/spool/asterisk >>> Log directory: /var/log/asterisk >>> Run/Sockets directory: /var/run/asterisk >>> PID file: /var/run/asterisk/asterisk.pid >>> VarLib directory: /var/lib/asterisk >>> Data directory: /var/lib/asterisk >>> ASTDB: /var/lib/asterisk/astdb >>> IAX2 Keys directory: /var/lib/asterisk/keys >>> >>> >>> >>> 1. Am I correct to expect a coredump file to be produced anytime >>> asterisk segfaults ? >>> >> >> Yes if -g is set and the user that's running asterisk has permissions to >> set ulimit -c. >> >> >>> >>> 2. Does Asterisk prints any WARNING or ERROR message whenever it >>> detects, at startup preferably, that it has not required permissions to >>> write a coredump file ? >>> >> >> No because it's the system that determines where a coredump goes and >> actually writes it, not asterisk. >> It's the sysctl kernel.core_pattern setting. >> >> >>> >>> 3. Among above directories, which one is choosen to save coredump files >>> ? Is it something that can/should be configured in /etc/asterisk (I've seen >>> related options in some debian /etc/default/asterisk files but I would be >>> curious to know if such things exist >>> >> >> See above. >> >> >>> >>> 4. Is there anything useful I can do with a line such as : >>> asterisk[1160]: segfault at 7efffffe ip 00000000005881d6 sp >>> 00007fec95c33910 error 4 in asterisk[400000+2a2000] ? Any pointer ? >>> >> >> Nope. Not a thing. Sorry. >> >> >> >>> >>> 5. Suggestions ? >>> >> >> If you can at least get the system to write a coredump file, there are >> new utilities in /var/lib/asterisk/scripts, namely ast_coredumper which can >> help create the backtraces if it can at least find the core file. Just run >> "./ast_coredumper --help" for more info. You should also be able to use >> those utilities with earlier Asterisk 13 versions. >> >> >> > > Oh yeah, and it's on my list to publish instructions on how ot use those > utilities but they were just released yesterday. > > > >> >>> Best regards >>> >>> >>> -- >>> _____________________________________________________________________ >>> -- Bandwidth and Colocation Provided by http://www.api-digital.com -- >>> >>> Check out the new Asterisk community forum at: >>> https://community.asterisk.org/ >>> >>> New to Asterisk? Start here: >>> https://wiki.asterisk.org/wiki/display/AST/Getting+Started >>> >>> asterisk-users mailing list >>> To UNSUBSCRIBE or update options visit: >>> http://lists.digium.com/mailman/listinfo/asterisk-users >>> >> >> >> >> -- >> George Joseph >> Digium, Inc. | Software Developer >> 445 Jan Davis Drive NW - Huntsville, AL 35806 - US >> Check us out at: www.digium.com & www.asterisk.org >> >> > > > -- > George Joseph > Digium, Inc. | Software Developer > 445 Jan Davis Drive NW - Huntsville, AL 35806 - US > Check us out at: www.digium.com & www.asterisk.org > > > -- > _____________________________________________________________________ > -- Bandwidth and Colocation Provided by http://www.api-digital.com -- > > Check out the new Asterisk community forum at: https://community.asterisk. > org/ > > New to Asterisk? Start here: > https://wiki.asterisk.org/wiki/display/AST/Getting+Started > > asterisk-users mailing list > To UNSUBSCRIBE or update options visit: > http://lists.digium.com/mailman/listinfo/asterisk-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.digium.com/pipermail/asterisk-users/attachments/20170217/9a971a9d/attachment.html>
George Joseph
2017-Feb-17 13:39 UTC
[asterisk-users] Advices when Asterisk segfaults and nothing useful in logs
On Fri, Feb 17, 2017 at 5:17 AM, Olivier <oza.4h07 at gmail.com> wrote:> Hi George, > > How does ast_coredumper compare to ast_grab_core ) ? > Is it worth learning to use both or shall favor one ? > > PS: As I don't know either program, yet, my question may seem silly. > Please, forgive me for this >Not silly at all. ast_grab_core actually kills asterisk to get the core file while ast_coredumper dumps the core file and let's asterisk continue. If asterisk is truly deadlocked this may not matter but in some situations you might not want to kill asterisk. If asterisk was compiled with DEBUG_THREADS, ast_coredumper dumps the locks table which is critical in debugging deadlock scenarios. It does this from the coredump, as opposed to running "core show locks", which makes the locks table consistent with the rest of the backtraces. ast_coiredumper can also find and process existing coredumps and it can create a tarball containing the backtraces and lock tables.> > 2017-02-14 22:52 GMT+01:00 George Joseph <gjoseph at digium.com>: > >> >> >> On Tue, Feb 14, 2017 at 2:51 PM, George Joseph <gjoseph at digium.com> >> wrote: >> >>> >>> >>> On Tue, Feb 14, 2017 at 10:21 AM, Olivier <oza.4h07 at gmail.com> wrote: >>> >>>> Hello, >>>> >>>> I've got a 13.13.1 system using PJSIP stack on debian Jessie. >>>> It runs from 50 to 100 simultaneous calls (so 100 to 200 PJSIP >>>> channels) all day long. >>>> From time to time, roughly meaning once a month, it segfaults with >>>> lines (from dmesg -T output) like this: >>>> asterisk[1160]: segfault at 7efffffe ip 00000000005881d6 sp >>>> 00007fec95c33910 error 4 in asterisk[400000+2a2000] >>>> >>>> >>>> Debug level was unfortunately not set in asterisk.conf but verbose >>>> level was set to 5. >>>> Asterisk runs with: >>>> /usr/sbin/asterisk -U asterisk -G asterisk -g >>>> >>>> Asterisk is compiled with DONT_OPTIMIZE and BETTER_BACKTRACES options. >>>> >>>> "core show settings" outputs: >>>> * Directories >>>> ------------- >>>> Configuration file: >>>> Configuration directory: /etc/asterisk >>>> Module directory: /usr/lib/asterisk/modules >>>> Spool directory: /var/spool/asterisk >>>> Log directory: /var/log/asterisk >>>> Run/Sockets directory: /var/run/asterisk >>>> PID file: /var/run/asterisk/asterisk.pid >>>> VarLib directory: /var/lib/asterisk >>>> Data directory: /var/lib/asterisk >>>> ASTDB: /var/lib/asterisk/astdb >>>> IAX2 Keys directory: /var/lib/asterisk/keys >>>> >>>> >>>> >>>> 1. Am I correct to expect a coredump file to be produced anytime >>>> asterisk segfaults ? >>>> >>> >>> Yes if -g is set and the user that's running asterisk has permissions to >>> set ulimit -c. >>> >>> >>>> >>>> 2. Does Asterisk prints any WARNING or ERROR message whenever it >>>> detects, at startup preferably, that it has not required permissions to >>>> write a coredump file ? >>>> >>> >>> No because it's the system that determines where a coredump goes and >>> actually writes it, not asterisk. >>> It's the sysctl kernel.core_pattern setting. >>> >>> >>>> >>>> 3. Among above directories, which one is choosen to save coredump files >>>> ? Is it something that can/should be configured in /etc/asterisk (I've seen >>>> related options in some debian /etc/default/asterisk files but I would be >>>> curious to know if such things exist >>>> >>> >>> See above. >>> >>> >>>> >>>> 4. Is there anything useful I can do with a line such as : >>>> asterisk[1160]: segfault at 7efffffe ip 00000000005881d6 sp >>>> 00007fec95c33910 error 4 in asterisk[400000+2a2000] ? Any pointer ? >>>> >>> >>> Nope. Not a thing. Sorry. >>> >>> >>> >>>> >>>> 5. Suggestions ? >>>> >>> >>> If you can at least get the system to write a coredump file, there are >>> new utilities in /var/lib/asterisk/scripts, namely ast_coredumper which can >>> help create the backtraces if it can at least find the core file. Just run >>> "./ast_coredumper --help" for more info. You should also be able to use >>> those utilities with earlier Asterisk 13 versions. >>> >>> >>> >> >> Oh yeah, and it's on my list to publish instructions on how ot use those >> utilities but they were just released yesterday. >> >> >> >>> >>>> Best regards >>>> >>>> >>>> -- >>>> _____________________________________________________________________ >>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com -- >>>> >>>> Check out the new Asterisk community forum at: >>>> https://community.asterisk.org/ >>>> >>>> New to Asterisk? Start here: >>>> https://wiki.asterisk.org/wiki/display/AST/Getting+Started >>>> >>>> asterisk-users mailing list >>>> To UNSUBSCRIBE or update options visit: >>>> http://lists.digium.com/mailman/listinfo/asterisk-users >>>> >>> >>> >>> >>> -- >>> George Joseph >>> Digium, Inc. | Software Developer >>> 445 Jan Davis Drive NW - Huntsville, AL 35806 - US >>> Check us out at: www.digium.com & www.asterisk.org >>> >>> >> >> >> -- >> George Joseph >> Digium, Inc. | Software Developer >> 445 Jan Davis Drive NW - Huntsville, AL 35806 - US >> Check us out at: www.digium.com & www.asterisk.org >> >> >> -- >> _____________________________________________________________________ >> -- Bandwidth and Colocation Provided by http://www.api-digital.com -- >> >> Check out the new Asterisk community forum at: >> https://community.asterisk.org/ >> >> New to Asterisk? Start here: >> https://wiki.asterisk.org/wiki/display/AST/Getting+Started >> >> asterisk-users mailing list >> To UNSUBSCRIBE or update options visit: >> http://lists.digium.com/mailman/listinfo/asterisk-users >> > > > -- > _____________________________________________________________________ > -- Bandwidth and Colocation Provided by http://www.api-digital.com -- > > Check out the new Asterisk community forum at: https://community.asterisk. > org/ > > New to Asterisk? Start here: > https://wiki.asterisk.org/wiki/display/AST/Getting+Started > > asterisk-users mailing list > To UNSUBSCRIBE or update options visit: > http://lists.digium.com/mailman/listinfo/asterisk-users >-- George Joseph Digium, Inc. | Software Developer 445 Jan Davis Drive NW - Huntsville, AL 35806 - US Check us out at: www.digium.com & www.asterisk.org -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.digium.com/pipermail/asterisk-users/attachments/20170217/35e4b6a2/attachment-0001.html>