asterisk users - Apr 2020 - Asterisk 13.22.0 under very high load conditions - freezes in H exten and blocks new calls

Hi all

I'm running an Asterisk on an Intel XEON E5-2660 virtual with Centos 7 -
32GB RAM. 
When I approach about 320 channels, I -sometimes- get thousands of these
messages suddenly streamed in the CLI / Asterisk log:

WARNING[60753][C-00022cb9] channel.c: Exceptionally long voice queue length
queuing to Local/xxxxxxxxxx at local-0002dbea;2
WARNING[71993][C-00022dcc] channel.c: Exceptionally long voice queue length
queuing to Local/xxxxxxxxxx at local-0002dd10;2
WARNING[72848][C-00022dd1] channel.c: Exceptionally long voice queue length
queuing to Local/xxxx at local-0002dd30;1

Top reports system load as 30.0, but the box is still very responsive over
SSH. Asterisk is shown as consuming 600% CPU when NOT in this state (but
close to it) and doesn't consume more as this state comes into effect.

Top also shows that HT CPU instances are idle 40% of their time.

What then happens when these messages appear is Asterisk stops accepting
originates on the AMI, and that no CDRs write through to the CDR DB anymore
- MySQL itself is idle and has no locks or running inserts indicated in
"show processlist" in MySQL CLI.

The Asterisk instance will then effectively freeze, never hanging up
terrminated calls, and never allowing new cals to start, keeping around 320
channels open, forever.

Thing is I can sometimes peak at over 350 channels, then come back down as
answers, hangups, etc. take place - no problem. But -sustaining- more than
320 channels over a two or three minutes seems to reliably cause the above
to happen.

Asterisk has to be kill -9'ed to get it restarted, it ignores kill -1 /
sighup / cli "core stop now"

CLI does work and is still repsonsive in this state, until I do a "core
stop
now", which doesn't stop the instance even after 20 to 30 minutes.

Asterisk max load is set to 40 and max calls to 350.

Any idea what I can look at? Any tips for running specifically 13 under high
load?

Thanks

Stefan

Are you using NFS? Any ODBC connections?

On Tue, Apr 21, 2020 at 10:23 AM Stefan Viljoen <viljoens at
verishare.co.za>
wrote:
> Hi all
>
> I'm running an Asterisk on an Intel XEON E5-2660 virtual with Centos 7
-
> 32GB RAM.
> When I approach about 320 channels, I -sometimes- get thousands of these
> messages suddenly streamed in the CLI / Asterisk log:
>
> WARNING[60753][C-00022cb9] channel.c: Exceptionally long voice queue length
> queuing to Local/xxxxxxxxxx at local-0002dbea;2
> WARNING[71993][C-00022dcc] channel.c: Exceptionally long voice queue length
> queuing to Local/xxxxxxxxxx at local-0002dd10;2
> WARNING[72848][C-00022dd1] channel.c: Exceptionally long voice queue length
> queuing to Local/xxxx at local-0002dd30;1
>
> Top reports system load as 30.0, but the box is still very responsive over
> SSH. Asterisk is shown as consuming 600% CPU when NOT in this state (but
> close to it) and doesn't consume more as this state comes into effect.
>
> Top also shows that HT CPU instances are idle 40% of their time.
>
> What then happens when these messages appear is Asterisk stops accepting
> originates on the AMI, and that no CDRs write through to the CDR DB anymore
> - MySQL itself is idle and has no locks or running inserts indicated in
> "show processlist" in MySQL CLI.
>
> The Asterisk instance will then effectively freeze, never hanging up
> terrminated calls, and never allowing new cals to start, keeping around 320
> channels open, forever.
>
> Thing is I can sometimes peak at over 350 channels, then come back down as
> answers, hangups, etc. take place - no problem. But -sustaining- more than
> 320 channels over a two or three minutes seems to reliably cause the above
> to happen.
>
> Asterisk has to be kill -9'ed to get it restarted, it ignores kill -1 /
> sighup / cli "core stop now"
>
> CLI does work and is still repsonsive in this state, until I do a
"core
> stop
> now", which doesn't stop the instance even after 20 to 30 minutes.
>
> Asterisk max load is set to 40 and max calls to 350.
>
> Any idea what I can look at? Any tips for running specifically 13 under
> high
> load?
>
> Thanks
>
> Stefan
>
>
>
>
>
> --
> _____________________________________________________________________
> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>
> Check out the new Asterisk community forum at:
> https://community.asterisk.org/
>
> New to Asterisk? Start here:
>       https://wiki.asterisk.org/wiki/display/AST/Getting+Started
>
> asterisk-users mailing list
> To UNSUBSCRIBE or update options visit:
>    http://lists.digium.com/mailman/listinfo/asterisk-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.digium.com/pipermail/asterisk-users/attachments/20200421/577bc875/attachment.html>

Hi Dovid

No NFS connections are in use. Directly mounted EXT4 SSD (emulated, this is in a
virtual environment).

Yes, I'm using ODBC to connect to the MySQL instance that records my CDRs.

Thanks for the reply!

Stefan

---
From: Dovid Bender <dovid at telecurve.com> 
Sent: Tuesday, 21 April 2020 16:53
To: viljoens at verishare.co.za; Asterisk Users Mailing List - Non-Commercial
Discussion <asterisk-users at lists.digium.com>
Subject: Re: [asterisk-users] Asterisk 13.22.0 under very high load conditions -
freezes in H exten and blocks new calls

Are you using NFS? Any ODBC connections?

On Tue, Apr 21, 2020 at 10:23 AM Stefan Viljoen <mailto:viljoens at
verishare.co.za> wrote:
Hi all

I'm running an Asterisk on an Intel XEON E5-2660 virtual with Centos 7 -
32GB RAM. 
When I approach about 320 channels, I -sometimes- get thousands of these
messages suddenly streamed in the CLI / Asterisk log:

WARNING[60753][C-00022cb9] channel.c: Exceptionally long voice queue length
queuing to Local/xxxxxxxxxx at local-0002dbea;2
WARNING[71993][C-00022dcc] channel.c: Exceptionally long voice queue length
queuing to Local/xxxxxxxxxx at local-0002dd10;2
WARNING[72848][C-00022dd1] channel.c: Exceptionally long voice queue length
queuing to Local/xxxx at local-0002dd30;1

Top reports system load as 30.0, but the box is still very responsive over
SSH. Asterisk is shown as consuming 600% CPU when NOT in this state (but
close to it) and doesn't consume more as this state comes into effect.

Top also shows that HT CPU instances are idle 40% of their time.

What then happens when these messages appear is Asterisk stops accepting
originates on the AMI, and that no CDRs write through to the CDR DB anymore
- MySQL itself is idle and has no locks or running inserts indicated in
"show processlist" in MySQL CLI.

The Asterisk instance will then effectively freeze, never hanging up
terrminated calls, and never allowing new cals to start, keeping around 320
channels open, forever.

Thing is I can sometimes peak at over 350 channels, then come back down as
answers, hangups, etc. take place - no problem. But -sustaining- more than
320 channels over a two or three minutes seems to reliably cause the above
to happen.

Asterisk has to be kill -9'ed to get it restarted, it ignores kill -1 /
sighup / cli "core stop now"

CLI does work and is still repsonsive in this state, until I do a "core
stop
now", which doesn't stop the instance even after 20 to 30 minutes.

Asterisk max load is set to 40 and max calls to 350.

Any idea what I can look at? Any tips for running specifically 13 under high
load?

Thanks

Stefan





-- 
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com --

Check out the new Asterisk community forum at: https://community.asterisk.org/

New to Asterisk? Start here:
      https://wiki.asterisk.org/wiki/display/AST/Getting+Started

asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
   http://lists.digium.com/mailman/listinfo/asterisk-users

Hello,

Hard to tell but from [1], " Exceptionally long voice queue length
queuing" should not happen.

1. Are you seeing WARNING or ERROR instances in log before this
"Exceptionally long voice" WARNING ?

2. If possible, I would try to load test a similar setup with SIPp and see
if I can reach or exceed this 320 channels/160 calls limit.

[1]
https://community.asterisk.org/t/exceptionally-long-voice-queue-length-queuing/70456/15

Le mar. 21 avr. 2020 à 16:23, Stefan Viljoen <viljoens at verishare.co.za>
a
écrit :
> Hi all
>
> I'm running an Asterisk on an Intel XEON E5-2660 virtual with Centos 7
-
> 32GB RAM.
> When I approach about 320 channels, I -sometimes- get thousands of these
> messages suddenly streamed in the CLI / Asterisk log:
>
> WARNING[60753][C-00022cb9] channel.c: Exceptionally long voice queue length
> queuing to Local/xxxxxxxxxx at local-0002dbea;2
> WARNING[71993][C-00022dcc] channel.c: Exceptionally long voice queue length
> queuing to Local/xxxxxxxxxx at local-0002dd10;2
> WARNING[72848][C-00022dd1] channel.c: Exceptionally long voice queue length
> queuing to Local/xxxx at local-0002dd30;1
>
> Top reports system load as 30.0, but the box is still very responsive over
> SSH. Asterisk is shown as consuming 600% CPU when NOT in this state (but
> close to it) and doesn't consume more as this state comes into effect.
>
> Top also shows that HT CPU instances are idle 40% of their time.
>
> What then happens when these messages appear is Asterisk stops accepting
> originates on the AMI, and that no CDRs write through to the CDR DB anymore
> - MySQL itself is idle and has no locks or running inserts indicated in
> "show processlist" in MySQL CLI.
>
> The Asterisk instance will then effectively freeze, never hanging up
> terrminated calls, and never allowing new cals to start, keeping around 320
> channels open, forever.
>
> Thing is I can sometimes peak at over 350 channels, then come back down as
> answers, hangups, etc. take place - no problem. But -sustaining- more than
> 320 channels over a two or three minutes seems to reliably cause the above
> to happen.
>
> Asterisk has to be kill -9'ed to get it restarted, it ignores kill -1 /
> sighup / cli "core stop now"
>
> CLI does work and is still repsonsive in this state, until I do a
"core
> stop
> now", which doesn't stop the instance even after 20 to 30 minutes.
>
> Asterisk max load is set to 40 and max calls to 350.
>
> Any idea what I can look at? Any tips for running specifically 13 under
> high
> load?
>
> Thanks
>
> Stefan
>
>
>
>
>
> --
> _____________________________________________________________________
> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>
> Check out the new Asterisk community forum at:
> https://community.asterisk.org/
>
> New to Asterisk? Start here:
>       https://wiki.asterisk.org/wiki/display/AST/Getting+Started
>
> asterisk-users mailing list
> To UNSUBSCRIBE or update options visit:
>    http://lists.digium.com/mailman/listinfo/asterisk-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.digium.com/pipermail/asterisk-users/attachments/20200427/80574415/attachment.html>