Jon, We are only using FastAgi. On the second system (running Asterisk 16) there are no agi's running (just some bash scripts on call hangup). I did add some hackey code (netstat -nua | grep -v 'udp 0 0' | grep -v udp6 | grep -v ' 0 0.0.0.0' | grep udp) to my bash script to check out the packet queue (with the help of https://serverfault.com/questions/623377/is-there-a-way-to-get-current-txqueue-utilization-on-e-g-eth0-or-tun-device) and it *seems there may be a corelation between packets in the queue and the load on Asterisk [root at a30 ~]# /tmp/x.sh CPU USAGE 393.8 Active calls 317 CPU USAGE 117.6 Active calls 318 CPU USAGE 143.8 Active calls 318 CPU USAGE 305.9 Active calls 318 udp 0 768 0.0.0.0:14510 0.0.0.0:* CPU USAGE 123.5 Active calls 316 CPU USAGE 137.5 Active calls 314 CPU USAGE 329.4 Active calls 314 CPU USAGE 312.5 Active calls 317 CPU USAGE 125.0 Active calls 317 CPU USAGE 137.5 Active calls 316 CPU USAGE 123.5 Active calls 316 CPU USAGE 117.6 Active calls 315 CPU USAGE 117.6 Active calls 314 CPU USAGE 135.3 Active calls 312 CPU USAGE 131.2 Active calls 309 CPU USAGE 131.2 Active calls 308 CPU USAGE 118.8 Active calls 309 CPU USAGE 331.2 Active calls 310 CPU USAGE 111.8 Active calls 309 udp 0 1280 0.0.0.0:15598 0.0.0.0:* CPU USAGE 517.6 Active calls 309 CPU USAGE 137.5 Active calls 308 CPU USAGE 143.8 Active calls 308 CPU USAGE 137.5 Active calls 307 CPU USAGE 311.8 Active calls 308 CPU USAGE 137.5 Active calls 308 CPU USAGE 125.0 Active calls 307 CPU USAGE 156.2 Active calls 307 udp 0 1280 0.0.0.0:19148 0.0.0.0:* CPU USAGE 135.3 Active calls 305 CPU USAGE 305.9 Active calls 307 CPU USAGE 156.2 Active calls 309 CPU USAGE 450.0 Active calls 311 CPU USAGE 137.5 Active calls 310 CPU USAGE 137.5 Active calls 310 CPU USAGE 256.2 Active calls 311 CPU USAGE 150.0 Active calls 308 CPU USAGE 281.2 Active calls 308 CPU USAGE 281.2 Active calls 307 CPU USAGE 137.5 Active calls 304 udp 0 1280 0.0.0.0:11284 0.0.0.0:* CPU USAGE 135.3 Active calls 306 CPU USAGE 150.0 Active calls 309 CPU USAGE 331.2 Active calls 310 CPU USAGE 125.0 Active calls 308 CPU USAGE 117.6 Active calls 307 CPU USAGE 135.3 Active calls 306 udp 0 1280 0.0.0.0:18494 0.0.0.0:* CPU USAGE 137.5 Active calls 307 CPU USAGE 141.2 Active calls 306 CPU USAGE 137.5 Active calls 307 CPU USAGE 562.5 Active calls 308 CPU USAGE 862.5 Active calls 311 CPU USAGE 164.7 Active calls 310 udp 0 768 0.0.0.0:17890 0.0.0.0:* udp 0 1280 0.0.0.0:11504 0.0.0.0:* [root at a30 ~]# On Tue, Oct 27, 2020 at 1:21 PM Jon Bonilla (Manwe) <manwe at aholab.ehu.es> wrote:> El Tue, 27 Oct 2020 12:52:47 -0400 > Dovid Bender <dovid at telecurve.com> escribió: > > > Hi, > > > > Sorry in advance that I am emailing the users list and not the biz list I > > think I will find my target audience here. We are looking to hire a > > consultant to help us figure out an issue. We are having what seems are > > "random load" issues with bare metal boxes that are dedicated to Asterisk > > and a few Perl AGI's. We went after all the usual suspects (CPU IDLE, > > memory usage etc.). I wrote the simple bash script below to show me total > > calls and CPU usage of Asterisk > > #! /bin/bash > > > > > > Did you try FastAGI? Asterisk doesn't handle spawning many processes very > well. > It's a bottleneck. > > > > > -- > PekePBX, the multitenant PBX solution > https://pekepbx.com >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.digium.com/pipermail/asterisk-users/attachments/20201027/f61c403a/attachment.html>
Anyone have any other ideas? On Tue, Oct 27, 2020 at 1:27 PM Dovid Bender <dovid at telecurve.com> wrote:> Jon, > > We are only using FastAgi. On the second system (running Asterisk 16) > there are no agi's running (just some bash scripts on call hangup). I did > add some hackey code (netstat -nua | grep -v 'udp 0 0' | grep > -v udp6 | grep -v ' 0 0.0.0.0' | grep udp) to my bash script to check out > the packet queue (with the help of > https://serverfault.com/questions/623377/is-there-a-way-to-get-current-txqueue-utilization-on-e-g-eth0-or-tun-device) > and it *seems there may be a corelation between packets in the queue and > the load on Asterisk > > [root at a30 ~]# /tmp/x.sh > CPU USAGE 393.8 Active calls 317 > CPU USAGE 117.6 Active calls 318 > CPU USAGE 143.8 Active calls 318 > CPU USAGE 305.9 Active calls 318 > udp 0 768 0.0.0.0:14510 0.0.0.0:* > > CPU USAGE 123.5 Active calls 316 > CPU USAGE 137.5 Active calls 314 > CPU USAGE 329.4 Active calls 314 > CPU USAGE 312.5 Active calls 317 > CPU USAGE 125.0 Active calls 317 > CPU USAGE 137.5 Active calls 316 > CPU USAGE 123.5 Active calls 316 > CPU USAGE 117.6 Active calls 315 > CPU USAGE 117.6 Active calls 314 > CPU USAGE 135.3 Active calls 312 > CPU USAGE 131.2 Active calls 309 > CPU USAGE 131.2 Active calls 308 > CPU USAGE 118.8 Active calls 309 > CPU USAGE 331.2 Active calls 310 > CPU USAGE 111.8 Active calls 309 > udp 0 1280 0.0.0.0:15598 0.0.0.0:* > > CPU USAGE 517.6 Active calls 309 > CPU USAGE 137.5 Active calls 308 > CPU USAGE 143.8 Active calls 308 > CPU USAGE 137.5 Active calls 307 > CPU USAGE 311.8 Active calls 308 > CPU USAGE 137.5 Active calls 308 > CPU USAGE 125.0 Active calls 307 > CPU USAGE 156.2 Active calls 307 > udp 0 1280 0.0.0.0:19148 0.0.0.0:* > > CPU USAGE 135.3 Active calls 305 > CPU USAGE 305.9 Active calls 307 > CPU USAGE 156.2 Active calls 309 > CPU USAGE 450.0 Active calls 311 > CPU USAGE 137.5 Active calls 310 > CPU USAGE 137.5 Active calls 310 > CPU USAGE 256.2 Active calls 311 > CPU USAGE 150.0 Active calls 308 > CPU USAGE 281.2 Active calls 308 > CPU USAGE 281.2 Active calls 307 > CPU USAGE 137.5 Active calls 304 > udp 0 1280 0.0.0.0:11284 0.0.0.0:* > > CPU USAGE 135.3 Active calls 306 > CPU USAGE 150.0 Active calls 309 > CPU USAGE 331.2 Active calls 310 > CPU USAGE 125.0 Active calls 308 > CPU USAGE 117.6 Active calls 307 > CPU USAGE 135.3 Active calls 306 > udp 0 1280 0.0.0.0:18494 0.0.0.0:* > > CPU USAGE 137.5 Active calls 307 > CPU USAGE 141.2 Active calls 306 > CPU USAGE 137.5 Active calls 307 > CPU USAGE 562.5 Active calls 308 > CPU USAGE 862.5 Active calls 311 > CPU USAGE 164.7 Active calls 310 > udp 0 768 0.0.0.0:17890 0.0.0.0:* > > udp 0 1280 0.0.0.0:11504 0.0.0.0:* > > [root at a30 ~]# > > > > On Tue, Oct 27, 2020 at 1:21 PM Jon Bonilla (Manwe) <manwe at aholab.ehu.es> > wrote: > >> El Tue, 27 Oct 2020 12:52:47 -0400 >> Dovid Bender <dovid at telecurve.com> escribió: >> >> > Hi, >> > >> > Sorry in advance that I am emailing the users list and not the biz list >> I >> > think I will find my target audience here. We are looking to hire a >> > consultant to help us figure out an issue. We are having what seems are >> > "random load" issues with bare metal boxes that are dedicated to >> Asterisk >> > and a few Perl AGI's. We went after all the usual suspects (CPU IDLE, >> > memory usage etc.). I wrote the simple bash script below to show me >> total >> > calls and CPU usage of Asterisk >> > #! /bin/bash >> > >> > >> >> Did you try FastAGI? Asterisk doesn't handle spawning many processes very >> well. >> It's a bottleneck. >> >> >> >> >> -- >> PekePBX, the multitenant PBX solution >> https://pekepbx.com >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.digium.com/pipermail/asterisk-users/attachments/20201029/0ad5fa49/attachment.html>
Hi Dovid, unfortunately I do not. We have seen similar issues when using Perl AGI Scripts when using Asterisk16/17. But since we moved everything to Java FastAGI we haven't seen any load issues anymore. Are you doing anything from fastagi that takes a lot of time? Something like a Playback? We Don't. We tried to do everything that takes some time in the Dialplan. Maybe you could try to kill the Asterisk with "kill -9" to see in the backtrace what the Asterisk is doing when it has such a high load. In the Backtrace you should be able to have a look at all processes and what they are doing. I guess i do not have to say "DON'T do it on the production system" :) BR Jöran On Thu, Oct 29, 2020 at 7:06 PM Dovid Bender <dovid at telecurve.com> wrote:> Anyone have any other ideas? > > > On Tue, Oct 27, 2020 at 1:27 PM Dovid Bender <dovid at telecurve.com> wrote: > >> Jon, >> >> We are only using FastAgi. On the second system (running Asterisk 16) >> there are no agi's running (just some bash scripts on call hangup). I did >> add some hackey code (netstat -nua | grep -v 'udp 0 0' | grep >> -v udp6 | grep -v ' 0 0.0.0.0' | grep udp) to my bash script to check out >> the packet queue (with the help of >> https://serverfault.com/questions/623377/is-there-a-way-to-get-current-txqueue-utilization-on-e-g-eth0-or-tun-device) >> and it *seems there may be a corelation between packets in the queue and >> the load on Asterisk >> >> [root at a30 ~]# /tmp/x.sh >> CPU USAGE 393.8 Active calls 317 >> CPU USAGE 117.6 Active calls 318 >> CPU USAGE 143.8 Active calls 318 >> CPU USAGE 305.9 Active calls 318 >> udp 0 768 0.0.0.0:14510 0.0.0.0:* >> >> CPU USAGE 123.5 Active calls 316 >> CPU USAGE 137.5 Active calls 314 >> CPU USAGE 329.4 Active calls 314 >> CPU USAGE 312.5 Active calls 317 >> CPU USAGE 125.0 Active calls 317 >> CPU USAGE 137.5 Active calls 316 >> CPU USAGE 123.5 Active calls 316 >> CPU USAGE 117.6 Active calls 315 >> CPU USAGE 117.6 Active calls 314 >> CPU USAGE 135.3 Active calls 312 >> CPU USAGE 131.2 Active calls 309 >> CPU USAGE 131.2 Active calls 308 >> CPU USAGE 118.8 Active calls 309 >> CPU USAGE 331.2 Active calls 310 >> CPU USAGE 111.8 Active calls 309 >> udp 0 1280 0.0.0.0:15598 0.0.0.0:* >> >> CPU USAGE 517.6 Active calls 309 >> CPU USAGE 137.5 Active calls 308 >> CPU USAGE 143.8 Active calls 308 >> CPU USAGE 137.5 Active calls 307 >> CPU USAGE 311.8 Active calls 308 >> CPU USAGE 137.5 Active calls 308 >> CPU USAGE 125.0 Active calls 307 >> CPU USAGE 156.2 Active calls 307 >> udp 0 1280 0.0.0.0:19148 0.0.0.0:* >> >> CPU USAGE 135.3 Active calls 305 >> CPU USAGE 305.9 Active calls 307 >> CPU USAGE 156.2 Active calls 309 >> CPU USAGE 450.0 Active calls 311 >> CPU USAGE 137.5 Active calls 310 >> CPU USAGE 137.5 Active calls 310 >> CPU USAGE 256.2 Active calls 311 >> CPU USAGE 150.0 Active calls 308 >> CPU USAGE 281.2 Active calls 308 >> CPU USAGE 281.2 Active calls 307 >> CPU USAGE 137.5 Active calls 304 >> udp 0 1280 0.0.0.0:11284 0.0.0.0:* >> >> CPU USAGE 135.3 Active calls 306 >> CPU USAGE 150.0 Active calls 309 >> CPU USAGE 331.2 Active calls 310 >> CPU USAGE 125.0 Active calls 308 >> CPU USAGE 117.6 Active calls 307 >> CPU USAGE 135.3 Active calls 306 >> udp 0 1280 0.0.0.0:18494 0.0.0.0:* >> >> CPU USAGE 137.5 Active calls 307 >> CPU USAGE 141.2 Active calls 306 >> CPU USAGE 137.5 Active calls 307 >> CPU USAGE 562.5 Active calls 308 >> CPU USAGE 862.5 Active calls 311 >> CPU USAGE 164.7 Active calls 310 >> udp 0 768 0.0.0.0:17890 0.0.0.0:* >> >> udp 0 1280 0.0.0.0:11504 0.0.0.0:* >> >> [root at a30 ~]# >> >> >> >> On Tue, Oct 27, 2020 at 1:21 PM Jon Bonilla (Manwe) <manwe at aholab.ehu.es> >> wrote: >> >>> El Tue, 27 Oct 2020 12:52:47 -0400 >>> Dovid Bender <dovid at telecurve.com> escribió: >>> >>> > Hi, >>> > >>> > Sorry in advance that I am emailing the users list and not the biz >>> list I >>> > think I will find my target audience here. We are looking to hire a >>> > consultant to help us figure out an issue. We are having what seems are >>> > "random load" issues with bare metal boxes that are dedicated to >>> Asterisk >>> > and a few Perl AGI's. We went after all the usual suspects (CPU IDLE, >>> > memory usage etc.). I wrote the simple bash script below to show me >>> total >>> > calls and CPU usage of Asterisk >>> > #! /bin/bash >>> > >>> > >>> >>> Did you try FastAGI? Asterisk doesn't handle spawning many processes >>> very well. >>> It's a bottleneck. >>> >>> >>> >>> >>> -- >>> PekePBX, the multitenant PBX solution >>> https://pekepbx.com >>> >> -- > _____________________________________________________________________ > -- Bandwidth and Colocation Provided by http://www.api-digital.com -- > > Check out the new Asterisk community forum at: > https://community.asterisk.org/ > > New to Asterisk? Start here: > https://wiki.asterisk.org/wiki/display/AST/Getting+Started > > asterisk-users mailing list > To UNSUBSCRIBE or update options visit: > http://lists.digium.com/mailman/listinfo/asterisk-users-- Jöran Vinzens - vinzens at sipgate.de Telefon: +49 211-63 55 56-21 Telefax: +49 211-63 55 55-22 sipgate GmbH - Gladbacher Str. 74 - 40219 Düsseldorf HRB Düsseldorf 39841 - Geschäftsführer: Thilo Salmon, Tim Mois Steuernummer: 106/5724/7147, Umsatzsteuer-ID: DE219349391 www.sipgate.de - www.sipgate.co.uk -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.digium.com/pipermail/asterisk-users/attachments/20201029/70d654e3/attachment.html>