asterisk users - Mar 2015 - Problems with the voice quality under load

B.H.

Hello, all :-)

We have a cluster of Asterisk (v. 11.9) servers that host IVR applications.
The servers work behind SIP proxy (kamailio) for load balancing.

All servers are in 2 processor configuration, 8-10 cores per CPU.

When a particular server gets about 500 concurrent calls, the sound quality
begins to degrade, the sound plays slowly and with clicks. As far as i
understand, it's because asterisk is unable to send the voice stream in
time i.e. the server is overloaded.

What i don't understand is, at the time that the server appears to be
overloaded and the audio quality is bad, actual server's load is no more
than 30-40% (60-70% idle CPU on average). IMHO, this indicates that for
some reason the server is unable to use it's CPU capacity efficiently. May
be because of some kind of thread contention inside asterisk?

I have read blogs that advice to divide physical server into several VMs
and they claim that this will improve the total capacity. In my own
experience, this did not work very well and seems like the visualization
actually made the quality worse.

Do you have any advice for me (other than purchasing more servers ;-) ?

Thanks!

-- 
???? NOW!
Moshiach is coming very soon, prepare yourself!
??? ?????? ?????? ?????? ??? ????? ????? ???!
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.digium.com/pipermail/asterisk-users/attachments/20150302/2cba9b4d/attachment.html>

Have you done the math for the network connections? BTF and external
What bit rates for the sound?
What codecs?
How are calls coming in - SIP - analogue

Disks OK(low IO per second)? Caching working OK?

CPU may not be the problem if your CPU utilization is really that low.

Ron

On 02/03/2015 10:26 AM, Mordechay Kaganer wrote:
> B.H.
>
> Hello, all :-)
>
> We have a cluster of Asterisk (v. 11.9) servers that host IVR 
> applications. The servers work behind SIP proxy (kamailio) for load 
> balancing.
>
> All servers are in 2 processor configuration, 8-10 cores per CPU.
>
> When a particular server gets about 500 concurrent calls, the sound 
> quality begins to degrade, the sound plays slowly and with clicks. As 
> far as i understand, it's because asterisk is unable to send the voice 
> stream in time i.e. the server is overloaded.
>
> What i don't understand is, at the time that the server appears to be 
> overloaded and the audio quality is bad, actual server's load is no 
> more than 30-40% (60-70% idle CPU on average). IMHO, this indicates 
> that for some reason the server is unable to use it's CPU capacity 
> efficiently. May be because of some kind of thread contention inside 
> asterisk?
>
> I have read blogs that advice to divide physical server into several 
> VMs and they claim that this will improve the total capacity. In my 
> own experience, this did not work very well and seems like the 
> visualization actually made the quality worse.
>
> Do you have any advice for me (other than purchasing more servers ;-) ?
>
> Thanks!
>
> -- 
> ???? NOW!
> Moshiach is coming very soon, prepare yourself!
> ??? ?????? ?????? ?????? ??? ????? ????? ???!
>
>

-- 
Ron Wheeler
President
Artifact Software Inc
email: rwheeler at artifact-software.com
skype: ronaldmwheeler
phone: 866-970-2435, ext 102

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.digium.com/pipermail/asterisk-users/attachments/20150302/cfb1c898/attachment.html>

On Monday 02 Mar 2015, Mordechay Kaganer wrote:
> When a particular server gets about 500 concurrent calls, the sound quality
> begins to degrade, the sound plays slowly and with clicks. As far as i
> understand, it's because asterisk is unable to send the voice stream in
> time i.e. the server is overloaded.
> 
> What i don't understand is, at the time that the server appears to be
> overloaded and the audio quality is bad, actual server's load is no
more
> than 30-40% (60-70% idle CPU on average). IMHO, this indicates that for
> some reason the server is unable to use it's CPU capacity efficiently.
May
> be because of some kind of thread contention inside asterisk?
> 
> Do you have any advice for me (other than purchasing more servers ;-) ?
Perhaps it's not CPU that is blocking things, but I/O?  It should be
visually
and audibly obvious if this is the case -- the disk activity lights will be 
illuminated, and the HDDs will be making noises.

If you can spare the RAM, consider setting up a tmpfs to hold your IVR prompts 
in memory.

-- 
AJS

Note:  Originating address only accepts e-mail from list!  If replying off-
list, change address to asterisk1list at earthshod dot co dot uk .

On Mon, 2 Mar 2015, Mordechay Kaganer wrote:
> When a particular server gets about 500 concurrent calls, the sound 
> quality begins to degrade, the sound plays slowly and with clicks. As 
> far as i understand, it's because asterisk is unable to send the voice 
> stream in time i.e. the server is overloaded.
> 
> What i don't understand is, at the time that the server appears to be 
> overloaded and the audio quality is bad, actual server's load is no
more
> than 30-40% (60-70% idle CPU on average). IMHO, this indicates that for 
> some reason the server is unable to use it's CPU capacity efficiently. 
> May be because of some kind of thread contention inside asterisk?
You may be using a feature (i.e., meetme) that is single threaded. If you 
view the system using 'htop' instead of 'top' (or, press
'1' while running
top) you may see that a single CPU is maxed out while the others are 
relatively idle.

-- 
Thanks in advance,
-------------------------------------------------------------------------
Steve Edwards       sedwards at sedwards.com      Voice: +1-760-468-3867 PST
Newline                                              Fax: +1-760-731-3000

B.H.

Hi, really thanks for all the relies :-)

Here's my answers:

On Mon, Mar 2, 2015 at 5:56 PM, Ron Wheeler <rwheeler at
artifact-software.com>
 wrote:
> Have you done the math for the network connections? BTF and external
> What bit rates for the sound?
> What codecs?
> How are calls coming in - SIP - analogue
>
> Disks OK(low IO per second)? Caching working OK?
>
There's no problem with network speed - we have 3 port bonding of 1G
connections on each server and 1G external cross connect to the SIP
providers (which are located in the same server farm). Even single 1G link
is sufficient for more than 5000 calls with G711.

All disk IO is off-loaded to the central server with NFS and as far as i
can see, the load is low. In most cases, the people listen to the same
files and caching is OK - each server has 32GB of RAM most of it is used
for caching.

We are connected to several SIP trunk providers. All the setup is SIP only.
The codec is G711.

On Mon, Mar 2, 2015 at 6:02 PM, A J Stiles <asterisk_list at
earthshod.co.uk>
 wrote:
>
> Perhaps it's not CPU that is blocking things, but I/O?  It should be
> visually
> and audibly obvious if this is the case -- the disk activity lights will be
> illuminated, and the HDDs will be making noises.
>
> If you can spare the RAM, consider setting up a tmpfs to hold your IVR
> prompts
> in memory.
>
The prompts are anyway cached in in memory by the disk cache so i don't
think it's the issue in this case. As stated above, i don't see much of
disk load on the NFS server as well.

On Mon, Mar 2, 2015 at 6:14 PM, Steve Edwards <asterisk.org at
sedwards.com>
wrote:
> You may be using a feature (i.e., meetme) that is single threaded. If you
> view the system using 'htop' instead of 'top' (or, press
'1' while running
> top) you may see that a single CPU is maxed out while the others are
> relatively idle.

I did check the CPU load and it is distributed evenly between all the
cores. Most of the load is in "system", probably network handling in
the
kernel.

We do use ConfBridge a lot, but all conferences are on a single server and
the issue is on all the servers.

What other features are single-threaded?

Asterisk is almost the only thing that runs on the PBX servers. There are
also some PHP AGI scripts (5-10 per minute per server on average) and most
of the AGI functions are off-loaded to a java application on the central
server with fast AGI. But there is still a lot of dialplan logic that runs
on the PBX's. Maybe, huge dialplan is the key for the issue?

-- 
???? NOW!
Moshiach is coming very soon, prepare yourself!
??? ?????? ?????? ?????? ??? ????? ????? ???!
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.digium.com/pipermail/asterisk-users/attachments/20150302/55788425/attachment.html>