Hi, we are experiencing a strange issue and I am hoping someone can point me to the right direction or help out with some pointers. We have asterisk 1.6.0.6 with a sangome a104DE card. We have basically 4 T1's for a total of DAHDI 96 channels. We have an agi application (php) that acts as a kind of a calling card application. All users are SIP users that make calls and asterisk then bridges the calls over T1 to the PSTN (after running agi script). Codec between SIP clients and Asterisk is G729 (96 G729 licenses installed on Asterisk). System has 12 GB of RAM. What we are seeing is the following: 1) All is fine till about 70 calls (70 active calls as displayed by core show channels). But as soon as traffic increases, we see the the a lot of AGI Processes are running and the *number of active calls *keeps on GROWING to 120, 130, ... . THis makes no sense since we can have a maximum of 96 active calls (4 T1). People calling start experiencing delays (10-15 seconds before calls are answered or connected, etc.). Upon doing a "ps -ef" we see a lot of AGI processes running that seem to keep growing 2) At that state, if we "Decrease" traffic, the *number of active calls*starts going down and the system becomes "normal" again. 3) What we are seeing are NOT Zombie processes though. It is "as-if" AGI processes are not exiting when traffic increases. But as soon as traffic decreases things become normal. 4) We did follow standard suggestions for AGI scripts (exit, Softhangup, etc) 5) We checked and the CPU utilization seems to be 25% idle at high loads.. so it does not seem like a CPU issue. We have a dual core 3.0GHz intel processor and based upon benchmarking info, we should be easily able to handle 96 calls (with transcoding). 6) We see that each AGI process takes up 149 MB of Virtual Memory when it is forked for each call. 7) Asterisk is taking up 500 MB Of VM. *Any* suggestions/help will be appreciated and I'd be happy to even pay if someone is ready to help us resolve this issue we are having. Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.digium.com/pipermail/asterisk-users/attachments/20090604/ed61478d/attachment.htm
You "are" keeping in mind that each call is actually two calls (SIP - CM --> CM - Asterisk/T1)? _____ From: asterisk-users-bounces at lists.digium.com [mailto:asterisk-users-bounces at lists.digium.com] On Behalf Of Deepak Sent: Thursday, June 04, 2009 11:15 AM To: Asterisk Users Mailing List - Non-Commercial Discussion Subject: [asterisk-users] Asterisk AGI issues (at high load) Hi, we are experiencing a strange issue and I am hoping someone can point me to the right direction or help out with some pointers. We have asterisk 1.6.0.6 with a sangome a104DE card. We have basically 4 T1's for a total of DAHDI 96 channels. We have an agi application (php) that acts as a kind of a calling card application. All users are SIP users that make calls and asterisk then bridges the calls over T1 to the PSTN (after running agi script). Codec between SIP clients and Asterisk is G729 (96 G729 licenses installed on Asterisk). System has 12 GB of RAM. What we are seeing is the following: 1) All is fine till about 70 calls (70 active calls as displayed by core show channels). But as soon as traffic increases, we see the the a lot of AGI Processes are running and the number of active calls keeps on GROWING to 120, 130, ... . THis makes no sense since we can have a maximum of 96 active calls (4 T1). People calling start experiencing delays (10-15 seconds before calls are answered or connected, etc.). Upon doing a "ps -ef" we see a lot of AGI processes running that seem to keep growing 2) At that state, if we "Decrease" traffic, the number of active calls starts going down and the system becomes "normal" again. 3) What we are seeing are NOT Zombie processes though. It is "as-if" AGI processes are not exiting when traffic increases. But as soon as traffic decreases things become normal. 4) We did follow standard suggestions for AGI scripts (exit, Softhangup, etc) 5) We checked and the CPU utilization seems to be 25% idle at high loads.. so it does not seem like a CPU issue. We have a dual core 3.0GHz intel processor and based upon benchmarking info, we should be easily able to handle 96 calls (with transcoding). 6) We see that each AGI process takes up 149 MB of Virtual Memory when it is forked for each call. 7) Asterisk is taking up 500 MB Of VM. *Any* suggestions/help will be appreciated and I'd be happy to even pay if someone is ready to help us resolve this issue we are having. Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.digium.com/pipermail/asterisk-users/attachments/20090604/8a4f09a9/attachment.htm
On Thu, Jun 4, 2009 at 12:15 PM, Deepak <dlalmtl at gmail.com> wrote:> Hi, we are experiencing a strange issue and I am hoping someone can point me > to the right direction or help out with some pointers. > > We have asterisk 1.6.0.6 with a sangome a104DE card. We have basically 4 > T1's for a total of DAHDI 96 channels. > > We have an agi application (php) that acts as a kind of a calling card > application. > > All users are SIP users that make calls and asterisk then bridges the calls > over T1 to the PSTN (after running agi script). Codec between SIP clients > and Asterisk is G729 (96 G729 licenses installed on Asterisk). > System has 12 GB of RAM. > > > What we are seeing is the following: > > 1) All is fine till about 70 calls (70 active calls as displayed by core > show channels). But as soon as traffic increases, we see the the a lot of > AGI Processes are running and the number of active calls keeps on GROWING to > 120, 130, ... . THis makes no sense since we can have a maximum of 96 active > calls (4 T1). People calling start experiencing delays (10-15 seconds before > calls are answered or connected, etc.). Upon doing a "ps -ef" we see a lot > of AGI processes running that seem to keep growingThere could be a number of external constraints that are causing this. One that comes to mind to is something waiting for a database connection, then all the other processes that are waiting for the database pile up and can't exit until they get their turn. Do you have a max number of connections on your database? How does the AGI know what to do based on the 'calling card information'? I'm guessing you store all that in a database, then deduct what the caller uses, etc. You may also enjoy http://us3.php.net/set_time_limit
Jeff LaCoursiere
2009-Jun-04 16:53 UTC
[asterisk-users] Asterisk AGI issues (at high load)
On Thu, 4 Jun 2009, Deepak wrote:> Hi, we are experiencing a strange issue and I am hoping someone can point me > to the right direction or help out with some pointers. > > We have asterisk 1.6.0.6 with a sangome a104DE card. We have basically 4 > T1's for a total of DAHDI 96 channels. > > We have an agi application (php) that acts as a kind of a calling card > application. > > All users are SIP users that make calls and asterisk then bridges the calls > over T1 to the PSTN (after running agi script). Codec between SIP clients > and Asterisk is G729 (96 G729 licenses installed on Asterisk). > > System has 12 GB of RAM. > > > What we are seeing is the following: > > 1) All is fine till about 70 calls (70 active calls as displayed by core > show channels). But as soon as traffic increases, we see the the a lot of > AGI Processes are running and the *number of active calls *keeps on GROWING > to 120, 130, ... . THis makes no sense since we can have a maximum of 96 > active calls (4 T1). People calling start experiencing delays (10-15 seconds > before calls are answered or connected, etc.). Upon doing a "ps -ef" we see > a lot of AGI processes running that seem to keep growing > > 2) At that state, if we "Decrease" traffic, the *number of active > calls*starts going down and the system becomes "normal" again. > > 3) What we are seeing are NOT Zombie processes though. It is "as-if" AGI > processes are not exiting when traffic increases. But as soon as traffic > decreases things become normal. > > 4) We did follow standard suggestions for AGI scripts (exit, Softhangup, > etc) > > 5) We checked and the CPU utilization seems to be 25% idle at high loads.. > so it does not seem like a CPU issue. We have a dual core 3.0GHz intel > processor and based upon benchmarking info, we should be easily able to > handle 96 calls (with transcoding). > > 6) We see that each AGI process takes up 149 MB of Virtual Memory when it is > forked for each call. > > 7) Asterisk is taking up 500 MB Of VM. > > > *Any* suggestions/help will be appreciated and I'd be happy to even pay if > someone is ready to help us resolve this issue we are having. >If each of 96 processes takes up 150MB you would need 15GB of RAM! This is only because you are trying to do this with PHP, which is crazy. Rewrite your AGI in C. My guess is you are dealing with large swap issues, which is degenerating into thrashing your disk. I also run a calling card operation... my AGI is in C :) j
On Thu, 4 Jun 2009, Deepak wrote:> Hi, we are experiencing a strange issue and I am hoping someone can > point me to the right direction or help out with some pointers. > > We have asterisk 1.6.0.6 with a sangome a104DE card. We have basically 4 > T1's for a total of DAHDI 96 channels. > > We have an agi application (php) that acts as a kind of a calling card > application. > > All users are SIP users that make calls and asterisk then bridges the > calls over T1 to the PSTN (after running agi script). Codec between SIP > clients and Asterisk is G729 (96 G729 licenses installed on Asterisk). > > System has 12 GB of RAM. > > What we are seeing is the following: > > 1) All is fine till about 70 calls (70 active calls as displayed by core > show channels). But as soon as traffic increases, we see the the a lot > of AGI Processes are running and the *number of active calls *keeps on > GROWING to 120, 130, ... . THis makes no sense since we can have a > maximum of 96 active calls (4 T1). People calling start experiencing > delays (10-15 seconds before calls are answered or connected, etc.). > Upon doing a "ps -ef" we see a lot of AGI processes running that seem to > keep growing > > 2) At that state, if we "Decrease" traffic, the *number of active > calls*starts going down and the system becomes "normal" again. > > 3) What we are seeing are NOT Zombie processes though. It is "as-if" AGI > processes are not exiting when traffic increases. But as soon as traffic > decreases things become normal. > > 4) We did follow standard suggestions for AGI scripts (exit, Softhangup, > etc) > > 5) We checked and the CPU utilization seems to be 25% idle at high > loads.. so it does not seem like a CPU issue. We have a dual core 3.0GHz > intel processor and based upon benchmarking info, we should be easily > able to handle 96 calls (with transcoding). > > 6) We see that each AGI process takes up 149 MB of Virtual Memory when > it is forked for each call. > > 7) Asterisk is taking up 500 MB Of VM. > > *Any* suggestions/help will be appreciated and I'd be happy to even pay > if someone is ready to help us resolve this issue we are having. > > ThanksI'm just a 1.2 Luddite, but here's my two cents: A couple of observations... 1) PHP would not get my vote for a "high load" environment. You can execute xxx AGI's in C before a single PHP script loads the interpreter and parses. Once you get over the startup "overhead," the difference will not be so significant. 2) 12GB strikes me as just a ton of RAM. I have a "chat" system that processes several times that number of simultaneous callers, running dozens of AGIs (in C) per call. The Asterisk process (according to "top") consumes less than 200mb. The AGI's (when I can "catch" one on top -- they execute so quickly) take less than 10mb. The host only has 2GB and doesn't swap. 3) 500 MB for Asterisk seems huge. 4) 149 MB for an AGI seems huge. Are you reading a bunch of stuff from the database? 5) Are you processing the "HUP" signal correctly in your AGI? I'm guessing you have database issues. I think your AGI is reading too much data from the database (149MB). I think your database design is not handling locking well and could benefit from a bit of re-design, indexing, pruning, etc. What can you tell us about the database? What database is it? (MySQL?) Is it on the same host? What statistics have you collected from the database? How many rows are in your tables? What does your AGI do? Thanks in advance, ------------------------------------------------------------------------ Steve Edwards sedwards at sedwards.com Voice: +1-760-468-3867 PST Newline Fax: +1-760-731-3000
AMEN!!! - How you got that far is nothing short of miraculous. Was Moses your designer? -----Original Message----- From: asterisk-users-bounces at lists.digium.com [mailto:asterisk-users-bounces at lists.digium.com] On Behalf Of Jeff LaCoursiere Sent: Thursday, June 04, 2009 12:23 PM To: Asterisk Users Mailing List - Non-Commercial Discussion Subject: Re: [asterisk-users] Asterisk AGI issues (at high load) On Thu, 4 Jun 2009, Deepak wrote:> BTW,we are using an ODBC connection to Microsoft SQL Server. > We are not using MySQL. > > Would that be a possible cause? >Good lord who designed this mess? Start over. j _______________________________________________ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
VinÃcius Fontes
2009-Jun-04 19:13 UTC
[asterisk-users] Asterisk AGI issues (at high load)
I agree when people say that your problem is at the database and not Asterisk/AGI. Another thing came into my mind, thought: you're using G729 to your SIP peers, and that means Asterisk is doing all the transcoding. How much idle CPU % top reports at high load? If this is too high (like 90%) you would need a dedicated transcoder: http://www.digium.com/en/products/voice/tc400b.php Vin?cius Fontes www.asteriskforum.com.br - Informa??es e discuss?o sobre Asterisk e telefonia IP ----- "Deepak" <dlalmtl at gmail.com> escreveu:> Hi, we are experiencing a strange issue and I am hoping someone can > point me to the right direction or help out with some pointers. > > We have asterisk 1.6.0.6 with a sangome a104DE card. We have basically > 4 T1's for a total of DAHDI 96 channels. > > We have an agi application (php) that acts as a kind of a calling card > application. > > All users are SIP users that make calls and asterisk then bridges the > calls over T1 to the PSTN (after running agi script). Codec between > SIP clients and Asterisk is G729 (96 G729 licenses installed on > Asterisk). > > System has 12 GB of RAM. > > > What we are seeing is the following: > > 1) All is fine till about 70 calls (70 active calls as displayed by > core show channels). But as soon as traffic increases, we see the the > a lot of AGI Processes are running and the number of active calls > keeps on GROWING to 120, 130, ... . THis makes no sense since we can > have a maximum of 96 active calls (4 T1). People calling start > experiencing delays (10-15 seconds before calls are answered or > connected, etc.). Upon doing a "ps -ef" we see a lot of AGI processes > running that seem to keep growing > > 2) At that state, if we "Decrease" traffic, the number of active calls > starts going down and the system becomes "normal" again. > > 3) What we are seeing are NOT Zombie processes though. It is "as-if" > AGI processes are not exiting when traffic increases. But as soon as > traffic decreases things become normal. > > 4) We did follow standard suggestions for AGI scripts (exit, > Softhangup, etc) > > 5) We checked and the CPU utilization seems to be 25% idle at high > loads.. so it does not seem like a CPU issue. We have a dual core > 3.0GHz intel processor and based upon benchmarking info, we should be > easily able to handle 96 calls (with transcoding). > > 6) We see that each AGI process takes up 149 MB of Virtual Memory when > it is forked for each call. > > 7) Asterisk is taking up 500 MB Of VM. > > > *Any* suggestions/help will be appreciated and I'd be happy to even > pay if someone is ready to help us resolve this issue we are having. > > Thanks > > > > > > > _______________________________________________ > -- Bandwidth and Colocation Provided by http://www.api-digital.com -- > > asterisk-users mailing list > To UNSUBSCRIBE or update options visit: > http://lists.digium.com/mailman/listinfo/asterisk-users
<html><body><span style="font-family:Verdana; color:#000000; font-size:10pt;"><br> <blockquote webmail="1" style="border-left: 2px solid blue; margin-left: 8px; padding-left: 8px; font-size: 10pt; color: black; font-family: verdana;"><div ><br>> BTW,we are using an ODBC connection to Microsoft SQL Server.<br> > We are not using MySQL.<br> ><br> > Would that be a possible cause?<br> ><br><br> Good lord who designed this mess? Start over.<br> <br> </div></blockquote><br>I feel the cold evil darkness spread through our lands.. <br><div ><br></div><blockquote webmail="1" style="border-left: 2px solid blue; margin-left: 8px; padding-left: 8px; font-size: 10pt; color: black; font-family: verdana;"><div > </div> </blockquote></span></body></html>