George Joseph
2016-Sep-06 12:37 UTC
[asterisk-users] [SOLVED] Re: Feature Request: what about "core stop panic" ?
On Tue, Sep 6, 2016 at 1:55 AM, Olivier <oza.4h07 at gmail.com> wrote:> Hello, > > After testing "pkill -SEGV -f /usr/sbin/asterisk" on Debian Jessie > platform, I've got several questions : > > > 1. When I issue a "cd /tmp; asterisk -cvvvvvvvvvvvg -U asterisk -G > asterisk" command, and then issue a "pkill -SEGV asterisk" command, > I can read on Asterisk console: > *CLI> Segmentation error (core dumped) > (*) I translated into english the exact output I got. > > I see a /tmp/core file exists. > > My asterisk.conf includes: > full = verbose,notice,warning,error,debug > > After Asterisk's stop, my /var/log:asterisk/full doesn't include any > mention to Segmentation error. > I cannot find anything in dmesg, /var/log/messages or /var/log/kern.log as > I did when a real segmentation fault occurred on production system. > > I would expect Asterisk to log something. > Is this expected behaviour ? >Yes. Asterisk can't really log anything when it ends abruptly because of critical signals like SEGV and since it was user initiated, rather than an actual SEGV, there won't be a message in the kernel log buffer.> > > 2. When I issue a "cd /tmp; systemctl start asterisk" command, and then > issue a "pkill -SEGV asterisk" command, > I can read on Asterisk console: > Disconnected from Asterisk server > Asterisk cleanly ending (0). > Executing last minute cleanups > > I cannot find any core file anyware. > > File /var/log:asterisk/full doesn't include any mention to Segmentation > error. > > Where should core file be created when Asterisk is run as a daemon by > asterisk user and group ? > Is there a setting I can use to specify the directory used (so that we can > make sure appropriate ownership is set) ? >"$ sysctl kernel.core_pattern" will show you where core files are written. For Asterisk to produce the core file, it has to be started with the '-g' option so make sure your asterisk.service file is adding the option.> > Cheers > > > 2016-09-05 13:36 GMT+02:00 Olivier <oza.4h07 at gmail.com>: > >> >> >> 2016-09-02 20:40 GMT+02:00 George Joseph <gjoseph at digium.com>: >> >>> >>> >>> On Fri, Sep 2, 2016 at 9:34 AM, Olivier <oza.4h07 at gmail.com> wrote: >>> >>>> Hello, >>>> >>>> I had a recent case where Asterisk stopped due to a segfault. >>>> This reminded me that being sure that whenever such issue occurs, it's >>>> useful to have a core file or various data at hand to analyze and exchange >>>> with support teams. >>>> >>>> How can you double check a running Asterisk system would produce such >>>> data if a segfault arrises ? >>>> >>>> 1. To my knowledge, no tool exists for this. Is it correct ? >>>> 2. If such tool do not exist, what would you say about something like >>>> "core stop panic", "core stop segfault" or whatever ? >>>> >>> >>> >>> How about just running "pkill -SEGV -f /usr/sbin/asterisk" from the >>> shell? >>> >> >> It seems to comply to what I'm after. >> Thank you very for sharing this here. >> >> >> >> >>> Works for me. >>> >>> >>> >>>> >>>> Best Regards >>>> >>>> >>>> >>>> -- >>>> _____________________________________________________________________ >>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com -- >>>> >>>> Join the Asterisk Community at the 13th AstriCon, September 27-29, 2016 >>>> http://www.asterisk.org/community/astricon-user-conference >>>> >>>> New to Asterisk? Start here: >>>> https://wiki.asterisk.org/wiki/display/AST/Getting+Started >>>> >>>> asterisk-users mailing list >>>> To UNSUBSCRIBE or update options visit: >>>> http://lists.digium.com/mailman/listinfo/asterisk-users >>>> >>> >>> >>> >>> -- >>> George Joseph >>> Digium, Inc. | Software Developer >>> 445 Jan Davis Drive NW - Huntsville, AL 35806 - US >>> Check us out at: www.digium.com & www.asterisk.org >>> >>> >>> -- >>> _____________________________________________________________________ >>> -- Bandwidth and Colocation Provided by http://www.api-digital.com -- >>> >>> Join the Asterisk Community at the 13th AstriCon, September 27-29, 2016 >>> http://www.asterisk.org/community/astricon-user-conference >>> >>> New to Asterisk? Start here: >>> https://wiki.asterisk.org/wiki/display/AST/Getting+Started >>> >>> asterisk-users mailing list >>> To UNSUBSCRIBE or update options visit: >>> http://lists.digium.com/mailman/listinfo/asterisk-users >>> >> >> > > -- > _____________________________________________________________________ > -- Bandwidth and Colocation Provided by http://www.api-digital.com -- > > Join the Asterisk Community at the 13th AstriCon, September 27-29, 2016 > http://www.asterisk.org/community/astricon-user-conference > > New to Asterisk? Start here: > https://wiki.asterisk.org/wiki/display/AST/Getting+Started > > asterisk-users mailing list > To UNSUBSCRIBE or update options visit: > http://lists.digium.com/mailman/listinfo/asterisk-users >-- George Joseph Digium, Inc. | Software Developer 445 Jan Davis Drive NW - Huntsville, AL 35806 - US Check us out at: www.digium.com & www.asterisk.org -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.digium.com/pipermail/asterisk-users/attachments/20160906/58b1b85d/attachment.html>
Tzafrir Cohen
2016-Sep-06 15:48 UTC
[asterisk-users] [SOLVED] Re: Feature Request: what about "core stop panic" ?
On Tue, Sep 06, 2016 at 06:37:52AM -0600, George Joseph wrote:> On Tue, Sep 6, 2016 at 1:55 AM, Olivier <oza.4h07 at gmail.com> wrote:> > Where should core file be created when Asterisk is run as a daemon by > > asterisk user and group ? > > Is there a setting I can use to specify the directory used (so that we can > > make sure appropriate ownership is set) ? > > > > "$ sysctl kernel.core_pattern" will show you where core files are written. > For Asterisk to produce the core file, it has to be started with the '-g' > option so make sure your asterisk.service file is adding the option.Specifically, if the first character of core_pattern is '!', the rest should be an executable, to which the core file is handled. IIRC Centos7 had something of that type installed by default. On Debian Stable you have the package corekeeper (or maybe also systemd-coredump from backports). I haven't tried any of those. -- Tzafrir Cohen icq#16849755 jabber:tzafrir.cohen at xorcom.com +972-50-7952406 mailto:tzafrir.cohen at xorcom.com http://www.xorcom.com
Steve Edwards
2016-Sep-06 16:29 UTC
[asterisk-users] [SOLVED] Re: Feature Request: what about "core stop panic" ?
> On Tue, Sep 06, 2016 at 06:37:52AM -0600, George Joseph wrote: >> >> "$ sysctl kernel.core_pattern" will show you where core files are >> written. For Asterisk to produce the core file, it has to be started >> with the '-g' option so make sure your asterisk.service file is adding >> the option.On Tue, 6 Sep 2016, Tzafrir Cohen wrote:> Specifically, if the first character of core_pattern is '!', the rest > should be an executable, to which the core file is handled. IIRC Centos7 > had something of that type installed by default. On Debian Stable you > have the package corekeeper (or maybe also systemd-coredump from > backports). I haven't tried any of those.I think that should be a 'pipe,' not an exclamation mark to hand off the core to an executable. -- Thanks in advance, ------------------------------------------------------------------------- Steve Edwards sedwards at sedwards.com Voice: +1-760-468-3867 PST https://www.linkedin.com/in/steve-edwards-4244281
Olivier
2016-Sep-07 17:03 UTC
[asterisk-users] [SOLVED] Re: Feature Request: what about "core stop panic" ?
2016-09-06 14:37 GMT+02:00 George Joseph <gjoseph at digium.com>:> > > On Tue, Sep 6, 2016 at 1:55 AM, Olivier <oza.4h07 at gmail.com> wrote: > >> Hello, >> >> After testing "pkill -SEGV -f /usr/sbin/asterisk" on Debian Jessie >> platform, I've got several questions : >> >> >> 1. When I issue a "cd /tmp; asterisk -cvvvvvvvvvvvg -U asterisk -G >> asterisk" command, and then issue a "pkill -SEGV asterisk" command, >> I can read on Asterisk console: >> *CLI> Segmentation error (core dumped) >> (*) I translated into english the exact output I got. >> >> I see a /tmp/core file exists. >> >> My asterisk.conf includes: >> full = verbose,notice,warning,error,debug >> >> After Asterisk's stop, my /var/log:asterisk/full doesn't include any >> mention to Segmentation error. >> I cannot find anything in dmesg, /var/log/messages or /var/log/kern.log >> as I did when a real segmentation fault occurred on production system. >> >> I would expect Asterisk to log something. >> Is this expected behaviour ? >> > > Yes. Asterisk can't really log anything when it ends abruptly because of > critical signals like SEGV and since it was user initiated, rather than an > actual SEGV, there won't be a message in the kernel log buffer. > > >> >> >> 2. When I issue a "cd /tmp; systemctl start asterisk" command, and then >> issue a "pkill -SEGV asterisk" command, >> I can read on Asterisk console: >> Disconnected from Asterisk server >> Asterisk cleanly ending (0). >> Executing last minute cleanups >> >> I cannot find any core file anyware. >> >> File /var/log:asterisk/full doesn't include any mention to Segmentation >> error. >> >> Where should core file be created when Asterisk is run as a daemon by >> asterisk user and group ? >> Is there a setting I can use to specify the directory used (so that we >> can make sure appropriate ownership is set) ? >> > > "$ sysctl kernel.core_pattern" will show you where core files are > written. For Asterisk to produce the core file, it has to be started with > the '-g' option so make sure your asterisk.service file is adding the > option. > >My system shows: # ps aux | grep asteri asterisk 429 7.3 2.4 59468 25088 ? Ssl 18:47 0:03 /usr/sbin/asterisk -U asterisk -G asterisk -g ... # sysctl kernel.core_pattern kernel.core_pattern = core Option -g is present. So either, the "pkill -SEGV asterisk" command doesn't trigger any coredump "Asterisk cleanly ending (0)", either the core dump is not produced (find / -name "core.*" doesn't find anything) for some reason.> > >> >> Cheers >> >> >> 2016-09-05 13:36 GMT+02:00 Olivier <oza.4h07 at gmail.com>: >> >>> >>> >>> 2016-09-02 20:40 GMT+02:00 George Joseph <gjoseph at digium.com>: >>> >>>> >>>> >>>> On Fri, Sep 2, 2016 at 9:34 AM, Olivier <oza.4h07 at gmail.com> wrote: >>>> >>>>> Hello, >>>>> >>>>> I had a recent case where Asterisk stopped due to a segfault. >>>>> This reminded me that being sure that whenever such issue occurs, it's >>>>> useful to have a core file or various data at hand to analyze and exchange >>>>> with support teams. >>>>> >>>>> How can you double check a running Asterisk system would produce such >>>>> data if a segfault arrises ? >>>>> >>>>> 1. To my knowledge, no tool exists for this. Is it correct ? >>>>> 2. If such tool do not exist, what would you say about something like >>>>> "core stop panic", "core stop segfault" or whatever ? >>>>> >>>> >>>> >>>> How about just running "pkill -SEGV -f /usr/sbin/asterisk" from the >>>> shell? >>>> >>> >>> It seems to comply to what I'm after. >>> Thank you very for sharing this here. >>> >>> >>> >>> >>>> Works for me. >>>> >>>> >>>> >>>>> >>>>> Best Regards >>>>> >>>>> >>>>> >>>>> -- >>>>> _____________________________________________________________________ >>>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com -- >>>>> >>>>> Join the Asterisk Community at the 13th AstriCon, September 27-29, 2016 >>>>> http://www.asterisk.org/community/astricon-user-conference >>>>> >>>>> New to Asterisk? Start here: >>>>> https://wiki.asterisk.org/wiki/display/AST/Getting+Started >>>>> >>>>> asterisk-users mailing list >>>>> To UNSUBSCRIBE or update options visit: >>>>> http://lists.digium.com/mailman/listinfo/asterisk-users >>>>> >>>> >>>> >>>> >>>> -- >>>> George Joseph >>>> Digium, Inc. | Software Developer >>>> 445 Jan Davis Drive NW - Huntsville, AL 35806 - US >>>> Check us out at: www.digium.com & www.asterisk.org >>>> >>>> >>>> -- >>>> _____________________________________________________________________ >>>> -- Bandwidth and Colocation Provided by http://www.api-digital.com -- >>>> >>>> Join the Asterisk Community at the 13th AstriCon, September 27-29, 2016 >>>> http://www.asterisk.org/community/astricon-user-conference >>>> >>>> New to Asterisk? Start here: >>>> https://wiki.asterisk.org/wiki/display/AST/Getting+Started >>>> >>>> asterisk-users mailing list >>>> To UNSUBSCRIBE or update options visit: >>>> http://lists.digium.com/mailman/listinfo/asterisk-users >>>> >>> >>> >> >> -- >> _____________________________________________________________________ >> -- Bandwidth and Colocation Provided by http://www.api-digital.com -- >> >> Join the Asterisk Community at the 13th AstriCon, September 27-29, 2016 >> http://www.asterisk.org/community/astricon-user-conference >> >> New to Asterisk? Start here: >> https://wiki.asterisk.org/wiki/display/AST/Getting+Started >> >> asterisk-users mailing list >> To UNSUBSCRIBE or update options visit: >> http://lists.digium.com/mailman/listinfo/asterisk-users >> > > > > -- > George Joseph > Digium, Inc. | Software Developer > 445 Jan Davis Drive NW - Huntsville, AL 35806 - US > Check us out at: www.digium.com & www.asterisk.org > > > -- > _____________________________________________________________________ > -- Bandwidth and Colocation Provided by http://www.api-digital.com -- > > Join the Asterisk Community at the 13th AstriCon, September 27-29, 2016 > http://www.asterisk.org/community/astricon-user-conference > > New to Asterisk? Start here: > https://wiki.asterisk.org/wiki/display/AST/Getting+Started > > asterisk-users mailing list > To UNSUBSCRIBE or update options visit: > http://lists.digium.com/mailman/listinfo/asterisk-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.digium.com/pipermail/asterisk-users/attachments/20160907/e22734f1/attachment.html>
Olivier
2016-Sep-07 17:15 UTC
[asterisk-users] [SOLVED] Re: Feature Request: what about "core stop panic" ?
2016-09-06 17:48 GMT+02:00 Tzafrir Cohen <tzafrir.cohen at xorcom.com>:> On Tue, Sep 06, 2016 at 06:37:52AM -0600, George Joseph wrote: > > On Tue, Sep 6, 2016 at 1:55 AM, Olivier <oza.4h07 at gmail.com> wrote: > > > > Where should core file be created when Asterisk is run as a daemon by > > > asterisk user and group ? > > > Is there a setting I can use to specify the directory used (so that we > can > > > make sure appropriate ownership is set) ? > > > > > > > "$ sysctl kernel.core_pattern" will show you where core files are > written. > > For Asterisk to produce the core file, it has to be started with the '-g' > > option so make sure your asterisk.service file is adding the option. >My asterisk.service is : # cat /run/systemd/generator.late/asterisk.service # Automatically generated by systemd-sysv-generator [Unit] SourcePath=/etc/init.d/asterisk Description=LSB: Asterisk PBX Before=runlevel2.target runlevel3.target runlevel4.target runlevel5.target shutdown.target After=network-online.target systemd-journald-dev-log.socket nss-lookup.target local-fs.target remote-fs.target dahdi.service misdn.service lcr.service wanrouter.service mysql.service postgresql.service Wants=network-online.target Conflicts=shutdown.target [Service] Type=forking Restart=no TimeoutSec=5min IgnoreSIGPIPE=no KillMode=process GuessMainPID=no RemainAfterExit=yes SysVStartPriority=2 ExecStart=/etc/init.d/asterisk start ExecStop=/etc/init.d/asterisk stop ExecReload=/etc/init.d/asterisk reload My /etc/init.d/asterisk file is sourcing my /etc/default/asterisk which includes an (uncommented) line COREDUMP=yes. I also have # grep core /etc/asterisk/asterisk.conf dumpcore = yes make menuselect shows: [*] DONT_OPTIMIZE [*] COMPILE_DOUBLE [ ] DEBUG_THREADS [*] LOADABLE_MODULES [ ] DEBUG_FD_LEAKS [*] BETTER_BACKTRACES [ ] LOTS_OF_SPANS [ ] MALLOC_DEBUG [ ] DEBUG_CHAOS [*] BUILD_NATIVE --- Extended --- [ ] REF_DEBUG [ ] AO2_DEBUG [ ] STATIC_BUILD XXX REBUILD_PARSERS [ ] LOW_MEMORY [ ] DISABLE_INLINE [*] OPTIONAL_API XXX USE_HOARD_ALLOCATOR [ ] RADIO_RELAX [ ] G711_NEW_ALGORITHM < > G711_REDUCED_BRANCHING < > TEST_CODING_TABLES < > TEST_TANDEM_TRANSCODING [ ] ADDRESS_SANITIZER [ ] THREAD_SANITIZER [ ] LEAK_SANITIZER [ ] UNDEFINED_SANITIZER [ ] BUSYDETECT_TONEONLY [ ] BUSYDETECT_COMPARE_TONE_AND_SILENCE [ ] BUSYDETECT_DEBUG [ ] INTEGER_CALLERID Is there a way to read Compiler Flags from a running system without looking at source file directory ? If someone forgets to select appropriate Compiler Flags, does it prevent Asterisk to produce a core file (even an empty one) ?> Specifically, if the first character of core_pattern is '!', the rest > should be an executable, to which the core file is handled. IIRC Centos7 > had something of that type installed by default. On Debian Stable you > have the package corekeeper (or maybe also systemd-coredump from > backports). I haven't tried any of those. > > -- > Tzafrir Cohen > icq#16849755 jabber:tzafrir.cohen at xorcom.com > +972-50-7952406 mailto:tzafrir.cohen at xorcom.com > http://www.xorcom.com > > -- > _____________________________________________________________________ > -- Bandwidth and Colocation Provided by http://www.api-digital.com -- > > Join the Asterisk Community at the 13th AstriCon, September 27-29, 2016 > http://www.asterisk.org/community/astricon-user-conference > > New to Asterisk? Start here: > https://wiki.asterisk.org/wiki/display/AST/Getting+Started > > asterisk-users mailing list > To UNSUBSCRIBE or update options visit: > http://lists.digium.com/mailman/listinfo/asterisk-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.digium.com/pipermail/asterisk-users/attachments/20160907/4da71d59/attachment.html>
George Joseph
2016-Sep-07 19:41 UTC
[asterisk-users] [SOLVED] Re: Feature Request: what about "core stop panic" ?
On Wed, Sep 7, 2016 at 11:03 AM, Olivier <oza.4h07 at gmail.com> wrote:> > > 2016-09-06 14:37 GMT+02:00 George Joseph <gjoseph at digium.com>: > >> >> >> On Tue, Sep 6, 2016 at 1:55 AM, Olivier <oza.4h07 at gmail.com> wrote: >> >>> Hello, >>> >>> After testing "pkill -SEGV -f /usr/sbin/asterisk" on Debian Jessie >>> platform, I've got several questions : >>> >>> >>> 1. When I issue a "cd /tmp; asterisk -cvvvvvvvvvvvg -U asterisk -G >>> asterisk" command, and then issue a "pkill -SEGV asterisk" command, >>> I can read on Asterisk console: >>> *CLI> Segmentation error (core dumped) >>> (*) I translated into english the exact output I got. >>> >>> I see a /tmp/core file exists. >>> >>> My asterisk.conf includes: >>> full = verbose,notice,warning,error,debug >>> >>> After Asterisk's stop, my /var/log:asterisk/full doesn't include any >>> mention to Segmentation error. >>> I cannot find anything in dmesg, /var/log/messages or /var/log/kern.log >>> as I did when a real segmentation fault occurred on production system. >>> >>> I would expect Asterisk to log something. >>> Is this expected behaviour ? >>> >> >> Yes. Asterisk can't really log anything when it ends abruptly because of >> critical signals like SEGV and since it was user initiated, rather than an >> actual SEGV, there won't be a message in the kernel log buffer. >> >> >>> >>> >>> 2. When I issue a "cd /tmp; systemctl start asterisk" command, and then >>> issue a "pkill -SEGV asterisk" command, >>> I can read on Asterisk console: >>> Disconnected from Asterisk server >>> Asterisk cleanly ending (0). >>> Executing last minute cleanups >>> >>> I cannot find any core file anyware. >>> >>> File /var/log:asterisk/full doesn't include any mention to Segmentation >>> error. >>> >>> Where should core file be created when Asterisk is run as a daemon by >>> asterisk user and group ? >>> Is there a setting I can use to specify the directory used (so that we >>> can make sure appropriate ownership is set) ? >>> >> >> "$ sysctl kernel.core_pattern" will show you where core files are >> written. For Asterisk to produce the core file, it has to be started with >> the '-g' option so make sure your asterisk.service file is adding the >> option. >> >> > My system shows: > # ps aux | grep asteri > asterisk 429 7.3 2.4 59468 25088 ? Ssl 18:47 0:03 > /usr/sbin/asterisk -U asterisk -G asterisk -g > ... > # sysctl kernel.core_pattern > kernel.core_pattern = core >Since "core" is a relative file name, the file will be in whatever the working directory is for the process. You may have to hunt it down. For better debugging, you might want to set core_pattern to something like " /tmp/core-%e-%t". That way all core files will have a name like "/tmp/core-asterisk-1473164587.7705". "man core" should give you more info on constructing the file name.> > Option -g is present. > > So either, the "pkill -SEGV asterisk" command doesn't trigger any > coredump "Asterisk cleanly ending (0)", > either the core dump is not produced (find / -name "core.*" doesn't find > anything) for some reason. > > >> >>-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.digium.com/pipermail/asterisk-users/attachments/20160907/14e76887/attachment.html>