Hi all, we have just upgraded from Asterisk 11 to Asterisk 16. After porting all the config to 16 we figured out some major load problems. the majority running of our Asterisk instances is still having Asterisk 11 so we can compare load handling on both versions. On the same hardware configuration we see load differences that Asterisk 16 takes four times the load as Asterisk 11 (on 11 we see load 0.5, on 16 we see something around 2). Our asterisk is only handling Calls, so there are no Subscription no Registration etc. After some testing we figured out if we eliminate AGI Apps from Dialplan we reduce the load significantly. At the moment we have 6 AGI calls for one single call. If we eliminate 3 of them we reduce the load by half. I also tried to have the AGI replaced by some fake AGI which returns immediately to make sure it is not related to long running AGI Scripts but this made no difference. I tried to tweaked a bit manipulating the thread limitations in stasis.conf but it had no significant effect on load. Also I tried to "decline" all non AGI messages in stasis.conf. It seems it has very little effect. Does anyone have similar issues or a solution? Is there anyone who calls AGI several times during call establishment? any hin and help would be very much appreciated! I am happy to share more config and information if it helps to find a solution. -- Jöran Vinzens - vinzens at sipgate.de sipgate GmbH - Gladbacher Str. 74 - 40219 Düsseldorf HRB Düsseldorf 39841 - Geschäftsführer: Thilo Salmon, Tim Mois Steuernummer: 106/5724/7147, Umsatzsteuer-ID: DE219349391 www.sipgate.de - www.sipgate.co.uk -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.digium.com/pipermail/asterisk-users/attachments/20190920/25404641/attachment.html>
On Fri, Sep 20, 2019, at 1:46 PM, Jöran Vinzens wrote:> Hi all, > > we have just upgraded from Asterisk 11 to Asterisk 16. > After porting all the config to 16 we figured out some major load problems. > > the majority running of our Asterisk instances is still having Asterisk > 11 so we can compare load handling on both versions. > On the same hardware configuration we see load differences that > Asterisk 16 takes four times the load as Asterisk 11 (on 11 we see load > 0.5, on 16 we see something around 2). > > Our asterisk is only handling Calls, so there are no Subscription no > Registration etc. > > After some testing we figured out if we eliminate AGI Apps from > Dialplan we reduce the load significantly. > At the moment we have 6 AGI calls for one single call. If we eliminate > 3 of them we reduce the load by half. > I also tried to have the AGI replaced by some fake AGI which returns > immediately to make sure it is not related to long running AGI Scripts > but this made no difference. > I tried to tweaked a bit manipulating the thread limitations in > stasis.conf but it had no significant effect on load. > Also I tried to "decline" all non AGI messages in stasis.conf. It seems > it has very little effect. > > Does anyone have similar issues or a solution? > Is there anyone who calls AGI several times during call establishment? > > any hin and help would be very much appreciated! > > I am happy to share more config and information if it helps to find a solution.Is this AGI or FastAGI? If AGI then moving to FastAGI will likely improve things. This is because when executing an AGI script the kernel has to fork the process which depending on memory consumption can take some time and drive up load. FreePBX ran into this problem[1] and it additionally caused audio issues in certain cases for them. [1] https://www.freepbx.org/performance-improvements-in-freepbx/ -- Joshua C. Colp Digium - A Sangoma Company | Senior Software Developer 445 Jan Davis Drive NW - Huntsville, AL 35806 - US Check us out at: www.digium.com & www.asterisk.org
Hello;
What programming language are you using? If you are using Perl, then I can
suggest this.
(1) You are going to have to profile your scripts. This is a must. Without
profiling, you don’t know what the problem is. In my opinion, Devel::NYTProf is
the king of that hill. It could be as simple as optimizing a single subroutine.
Then you will know exactly what the problem is.
(2) See which version of Perl are you running and see if upgrading it solves
your problems. The easiest way is to download the newest Perl they support from
ActiveState.com. It creates a completely independent installation in /opt which
will not interfere with your system Perl. You can also compile and install the
newest Perl from source completely separate from your system Perl. That’s what I
do. I have a couple of scripts to automate that process. If you first get a list
of your installed modules using ‘perlmod’, you can clean up the output a bit and
pipe that to cpanm to make sure that you have all the modules you need.
If you are using PHP, then I’m sure that the above still applies, but PHP is not
my area of expertise.
Regards;
John V.
From: asterisk-users <asterisk-users-bounces at lists.digium.com> On
Behalf Of Jöran Vinzens
Sent: Friday, September 20, 2019 12:47 PM
To: Asterisk Users Mailing List - Non-Commercial Discussion <asterisk-users
at lists.digium.com>
Subject: [asterisk-users] Load issues using AGI
Hi all,
we have just upgraded from Asterisk 11 to Asterisk 16.
After porting all the config to 16 we figured out some major load problems.
the majority running of our Asterisk instances is still having Asterisk 11 so we
can compare load handling on both versions.
On the same hardware configuration we see load differences that Asterisk 16
takes four times the load as Asterisk 11 (on 11 we see load 0.5, on 16 we see
something around 2).
Our asterisk is only handling Calls, so there are no Subscription no
Registration etc.
After some testing we figured out if we eliminate AGI Apps from Dialplan we
reduce the load significantly.
At the moment we have 6 AGI calls for one single call. If we eliminate 3 of them
we reduce the load by half.
I also tried to have the AGI replaced by some fake AGI which returns immediately
to make sure it is not related to long running AGI Scripts but this made no
difference.
I tried to tweaked a bit manipulating the thread limitations in stasis.conf but
it had no significant effect on load.
Also I tried to "decline" all non AGI messages in stasis.conf. It
seems it has very little effect.
Does anyone have similar issues or a solution?
Is there anyone who calls AGI several times during call establishment?
any hin and help would be very much appreciated!
I am happy to share more config and information if it helps to find a solution.
--
Jöran Vinzens - vinzens at sipgate.de <mailto:vinzens at sipgate.de>
sipgate GmbH - Gladbacher Str. 74 - 40219 Düsseldorf
HRB Düsseldorf 39841 - Geschäftsführer: Thilo Salmon, Tim Mois
Steuernummer: 106/5724/7147, Umsatzsteuer-ID: DE219349391
www.sipgate.de <http://www.sipgate.de> - www.sipgate.co.uk
<http://www.sipgate.co.uk>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.digium.com/pipermail/asterisk-users/attachments/20190920/cda444a2/attachment.html>
One other thing. I use a package called AGISpeedy. Its available for both Perl
and PHP and I’ve used it for years without any problems. Its supposedly an order
of magnitude faster than regular FastAGI scripts. The only downside is that it
hasn’t been maintained a while, but the package is solid and really doesn’t need
much maintaining. I’ll probably write the author and see if I can keep it
updated for him.
Regards;
John
From: asterisk-users <asterisk-users-bounces at lists.digium.com> On
Behalf Of Tech Support
Sent: Friday, September 20, 2019 1:37 PM
To: 'Asterisk Users Mailing List - Non-Commercial Discussion'
<asterisk-users at lists.digium.com>
Subject: Re: [asterisk-users] Load issues using AGI
Hello;
What programming language are you using? If you are using Perl, then I can
suggest this.
(1) You are going to have to profile your scripts. This is a must. Without
profiling, you don’t know what the problem is. In my opinion, Devel::NYTProf is
the king of that hill. It could be as simple as optimizing a single subroutine.
Then you will know exactly what the problem is.
(2) See which version of Perl are you running and see if upgrading it solves
your problems. The easiest way is to download the newest Perl they support from
ActiveState.com. It creates a completely independent installation in /opt which
will not interfere with your system Perl. You can also compile and install the
newest Perl from source completely separate from your system Perl. That’s what I
do. I have a couple of scripts to automate that process. If you first get a list
of your installed modules using ‘perlmod’, you can clean up the output a bit and
pipe that to cpanm to make sure that you have all the modules you need.
If you are using PHP, then I’m sure that the above still applies, but PHP is not
my area of expertise.
Regards;
John V.
From: asterisk-users <asterisk-users-bounces at lists.digium.com
<mailto:asterisk-users-bounces at lists.digium.com> > On Behalf Of
Jöran Vinzens
Sent: Friday, September 20, 2019 12:47 PM
To: Asterisk Users Mailing List - Non-Commercial Discussion <asterisk-users
at lists.digium.com <mailto:asterisk-users at lists.digium.com> >
Subject: [asterisk-users] Load issues using AGI
Hi all,
we have just upgraded from Asterisk 11 to Asterisk 16.
After porting all the config to 16 we figured out some major load problems.
the majority running of our Asterisk instances is still having Asterisk 11 so we
can compare load handling on both versions.
On the same hardware configuration we see load differences that Asterisk 16
takes four times the load as Asterisk 11 (on 11 we see load 0.5, on 16 we see
something around 2).
Our asterisk is only handling Calls, so there are no Subscription no
Registration etc.
After some testing we figured out if we eliminate AGI Apps from Dialplan we
reduce the load significantly.
At the moment we have 6 AGI calls for one single call. If we eliminate 3 of them
we reduce the load by half.
I also tried to have the AGI replaced by some fake AGI which returns immediately
to make sure it is not related to long running AGI Scripts but this made no
difference.
I tried to tweaked a bit manipulating the thread limitations in stasis.conf but
it had no significant effect on load.
Also I tried to "decline" all non AGI messages in stasis.conf. It
seems it has very little effect.
Does anyone have similar issues or a solution?
Is there anyone who calls AGI several times during call establishment?
any hin and help would be very much appreciated!
I am happy to share more config and information if it helps to find a solution.
--
Jöran Vinzens - vinzens at sipgate.de <mailto:vinzens at sipgate.de>
sipgate GmbH - Gladbacher Str. 74 - 40219 Düsseldorf
HRB Düsseldorf 39841 - Geschäftsführer: Thilo Salmon, Tim Mois
Steuernummer: 106/5724/7147, Umsatzsteuer-ID: DE219349391
www.sipgate.de <http://www.sipgate.de> - www.sipgate.co.uk
<http://www.sipgate.co.uk>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.digium.com/pipermail/asterisk-users/attachments/20190920/1a38b8d7/attachment.html>
Hi, @josh, We are using AGI. It is a very simple perl script. If we need to move to FastAGI we most likely would port it to ARI instead. But I let you know. @john, we using Perl. To see if it is a problem with the perl i had put an "exit 0" just at the first lines so there is no logic done at the AGI. It's only the start up and return from AGI what produces the most of the load. Nevertheless, we will try what you just posted. thanks for your help. I will keep you updated! BR Jöran On Fri, Sep 20, 2019 at 7:38 PM Tech Support <asterisk at voipbusiness.us> wrote:> Hello; > > What programming language are you using? If you are using Perl, then I > can suggest this. > > (1) You are going to have to profile your scripts. This is a must. > Without profiling, you don’t know what the problem is. In my opinion, > Devel::NYTProf is the king of that hill. It could be as simple as > optimizing a single subroutine. Then you will know exactly what the problem > is. > > (2) See which version of Perl are you running and see if upgrading it > solves your problems. The easiest way is to download the newest Perl they > support from ActiveState.com. It creates a completely independent > installation in /opt which will not interfere with your system Perl. You > can also compile and install the newest Perl from source completely > separate from your system Perl. That’s what I do. I have a couple of > scripts to automate that process. If you first get a list of your installed > modules using ‘perlmod’, you can clean up the output a bit and pipe that to > cpanm to make sure that you have all the modules you need. > > > > If you are using PHP, then I’m sure that the above still applies, but PHP > is not my area of expertise. > > > > Regards; > > John V. > > > > > > *From:* asterisk-users <asterisk-users-bounces at lists.digium.com> *On > Behalf Of *Jöran Vinzens > *Sent:* Friday, September 20, 2019 12:47 PM > *To:* Asterisk Users Mailing List - Non-Commercial Discussion < > asterisk-users at lists.digium.com> > *Subject:* [asterisk-users] Load issues using AGI > > > > Hi all, > > > > we have just upgraded from Asterisk 11 to Asterisk 16. > > After porting all the config to 16 we figured out some major load problems. > > > > the majority running of our Asterisk instances is still having Asterisk 11 > so we can compare load handling on both versions. > > On the same hardware configuration we see load differences that Asterisk > 16 takes four times the load as Asterisk 11 (on 11 we see load 0.5, on 16 > we see something around 2). > > > > Our asterisk is only handling Calls, so there are no Subscription no > Registration etc. > > > > After some testing we figured out if we eliminate AGI Apps from Dialplan > we reduce the load significantly. > > At the moment we have 6 AGI calls for one single call. If we eliminate 3 > of them we reduce the load by half. > > I also tried to have the AGI replaced by some fake AGI which returns > immediately to make sure it is not related to long running AGI Scripts but > this made no difference. > > I tried to tweaked a bit manipulating the thread limitations in > stasis.conf but it had no significant effect on load. > > Also I tried to "decline" all non AGI messages in stasis.conf. It seems it > has very little effect. > > > > Does anyone have similar issues or a solution? > > Is there anyone who calls AGI several times during call establishment? > > > > any hin and help would be very much appreciated! > > > > I am happy to share more config and information if it helps to find a > solution. > > > > > > -- > > Jöran Vinzens - vinzens at sipgate.de > > > sipgate GmbH - Gladbacher Str. 74 - 40219 Düsseldorf > HRB Düsseldorf 39841 - Geschäftsführer: Thilo Salmon, Tim Mois > Steuernummer: 106/5724/7147, Umsatzsteuer-ID: DE219349391 > > www.sipgate.de - www.sipgate.co.uk > > -- > _____________________________________________________________________ > -- Bandwidth and Colocation Provided by http://www.api-digital.com -- > > Check out the new Asterisk community forum at: > https://community.asterisk.org/ > > New to Asterisk? Start here: > https://wiki.asterisk.org/wiki/display/AST/Getting+Started > > asterisk-users mailing list > To UNSUBSCRIBE or update options visit: > http://lists.digium.com/mailman/listinfo/asterisk-users-- Jöran Vinzens - vinzens at sipgate.de Telefon: +49 211-63 55 56-21 Telefax: +49 211-63 55 55-22 sipgate GmbH - Gladbacher Str. 74 - 40219 Düsseldorf HRB Düsseldorf 39841 - Geschäftsführer: Thilo Salmon, Tim Mois Steuernummer: 106/5724/7147, Umsatzsteuer-ID: DE219349391 www.sipgate.de - www.sipgate.co.uk -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.digium.com/pipermail/asterisk-users/attachments/20190920/c7068c20/attachment.html>