asterisk users - Feb 2015 - constantly increasing load in Asterisk 11.14

Hi,

we have quite a few Asterisk machines running and try to keep them on a
current version of the Asterisk 11 branch. But since we upgraded to 11.14.0
a couple weeks ago, we have to restart the Asterisk process every week
because the load gets too high and our monitoring complains.

Those machines are doing only SIP-to-SIP call relay, the dialplan is quite
complex, transcoding is done only on a few percent of the calls processed.
During the daytime, there are at max around 200 SIP channels (100 calls)
running at the same time. After one week, one machine has processed about
170k calls.

I have uploaded a comparison of cacti load graphs for one week of a machine
running with 11.14.0 and one running with 11.6.0: http://pbrd.co/1v0SO3R

As you can see, after a restart, both machines have about the same load.
But after the really quiet weekend, the 11.14 Asterisk starts the new week
with a much higer load than the 11.6 Asterisk, where it stays constant.
We've had an 11.5.1 machine running for about half a year without the need
of restarting, but right now, this is not possible.

Has anyone seen this before? Or does anyone know a reason, what change
somewhere between 11.6 and 11.14 could cause this behaviour? It looks like
we have to go back to 11.6.

Best Regards,
Sebastian
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.digium.com/pipermail/asterisk-users/attachments/20150205/16291fe7/attachment.html>

On 05/02/15 10:53, Sebastian Damm wrote:
> Hi,
>
> we have quite a few Asterisk machines running and try to keep them on 
> a current version of the Asterisk 11 branch. But since we upgraded to 
> 11.14.0 a couple weeks ago, we have to restart the Asterisk process 
> every week because the load gets too high and our monitoring complains.
>
> Those machines are doing only SIP-to-SIP call relay, the dialplan is 
> quite complex, transcoding is done only on a few percent of the calls 
> processed. During the daytime, there are at max around 200 SIP 
> channels (100 calls) running at the same time. After one week, one 
> machine has processed about 170k calls.
>
> I have uploaded a comparison of cacti load graphs for one week of a 
> machine running with 11.14.0 and one running with 11.6.0: 
> http://pbrd.co/1v0SO3R
>
> As you can see, after a restart, both machines have about the same 
> load. But after the really quiet weekend, the 11.14 Asterisk starts 
> the new week with a much higer load than the 11.6 Asterisk, where it 
> stays constant. We've had an 11.5.1 machine running for about half a 
> year without the need of restarting, but right now, this is not possible.
>
> Has anyone seen this before? Or does anyone know a reason, what change 
> somewhere between 11.6 and 11.14 could cause this behaviour? It looks 
> like we have to go back to 11.6.
>
>
>
We see it in the certified version which is currently 11.6. We have 
about 50% higher call volume on our servers than yourselves and it does 
take about 3-4 months for the load to reach about 5. I don't know what 
causes it but we take the servers out of our load balancing and restart 
asterisk every few months and this returns the load to normal.

Can you tell me if the memory usage by Asterisk is also increasing with
load over time?

On Thu, Feb 5, 2015 at 4:53 AM, Sebastian Damm <damm at sipgate.de> wrote:
> Hi,
>
> we have quite a few Asterisk machines running and try to keep them on a
> current version of the Asterisk 11 branch. But since we upgraded to 11.14.0
> a couple weeks ago, we have to restart the Asterisk process every week
> because the load gets too high and our monitoring complains.
>
> Those machines are doing only SIP-to-SIP call relay, the dialplan is quite
> complex, transcoding is done only on a few percent of the calls processed.
> During the daytime, there are at max around 200 SIP channels (100 calls)
> running at the same time. After one week, one machine has processed about
> 170k calls.
>
> I have uploaded a comparison of cacti load graphs for one week of a
> machine running with 11.14.0 and one running with 11.6.0:
> http://pbrd.co/1v0SO3R
>
> As you can see, after a restart, both machines have about the same load.
> But after the really quiet weekend, the 11.14 Asterisk starts the new week
> with a much higer load than the 11.6 Asterisk, where it stays constant.
> We've had an 11.5.1 machine running for about half a year without the
need
> of restarting, but right now, this is not possible.
>
> Has anyone seen this before? Or does anyone know a reason, what change
> somewhere between 11.6 and 11.14 could cause this behaviour? It looks like
> we have to go back to 11.6.
>
> Best Regards,
> Sebastian
>
> --
> _____________________________________________________________________
> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
> New to Asterisk? Join us for a live introductory webinar every Thurs:
>                http://www.asterisk.org/hello
>
> asterisk-users mailing list
> To UNSUBSCRIBE or update options visit:
>    http://lists.digium.com/mailman/listinfo/asterisk-users
>


-- 
[image: Digium logo]
Scott Griepentrog
Digium, Inc ? Software Developer
445 Jan Davis Drive NW ? Huntsville, AL 35806 ? US
direct/fax: +1 256 428 6239 ? mobile: +1 256 580 6090
Check us out at: http://digium.com ? http://asterisk.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.digium.com/pipermail/asterisk-users/attachments/20150205/a2f603d3/attachment.html>

On 05/02/15 15:56, Scott Griepentrog wrote:
> Can you tell me if the memory usage by Asterisk is also increasing 
> with load over time?
Just for reference in 11.6 we don't see any memory usage increase but we 
do see system (as opposed to user) cpu increase from 27% to 84% as the 
load increases from 1% to around 4%.

Would be interesting to see if the original posters experience is the 
same but more accelerated on their system.

Hi,

On Thu, Feb 5, 2015 at 4:56 PM, Scott Griepentrog <sgriepentrog at
digium.com>
wrote:
> Can you tell me if the memory usage by Asterisk is also increasing with
> load over time?

Yes, the memory usage does rise a bit.

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
11.14
root     26047  8.9  5.0 1302036 407640 ?      Sl   Jan28 1201:26 asterisk
-vvvgf
11.6
root     36336  9.9  1.1 881684 90360 ?        Sl   Jan28 1328:19 asterisk
-vvvgf

Interestingly, the CPU graphs show no visible increase.

If you need any more information, just let me know.

Gareth, the slow increase of load over months was there with 11.5 or 11.6
already, but I can live with a restart every couple month. Once every week
is too much.
Here is another overview of the load of one machine: http://pbrd.co/1zeFwBy
You can see, the Asterisk was running from April to September without much
change in load, then was restarted. In November, we updated to 11.14, and
from that time, it looks a bit different (and Asterisk needed a lot more
restarts).

Best Regards,
Sebastian
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.digium.com/pipermail/asterisk-users/attachments/20150206/128c0fb7/attachment.html>