asterisk users - Aug 2019 - Amazon AWS question

We are running load capacity tests using Amazon AWS configurations.

For the tests, we are basically scaling up calls to a second Asterisk box. 
First box that is calling the second box plays music on hold for 60 seconds,
then hangs up the call.
My initial thought was jitter problems, but that doesn't seem to be the
case.

I believe I found the cause while looking at the asterisk logs.  I am finding
hundreds of entries like this when they report audio problems.

[08/19 15:04:17.176] DEBUG[4062][C-00000076] res_timing_timerfd.c: Expected to
acknowledge 1 ticks but got 4 instead
[08/19 15:04:17.173] DEBUG[4008][C-0000005b] res_timing_timerfd.c: Expected to
acknowledge 1 ticks but got 3 instead
[08/19 15:04:17.180] DEBUG[4008][C-0000005b] res_timing_timerfd.c: Expected to
acknowledge 1 ticks but got 2 instead
[08/19 15:04:17.174] DEBUG[3933][C-00000038] res_timing_timerfd.c: Expected to
acknowledge 1 ticks but got 2 instead

I see references where people say to make sure to set VMWare ESXi Latency
Sensitivity to high for issues like this.

However, I am told there is no such setting with Amazon AWS.  It sounds like the
settings are all based on the type of configuration you buy.

Does anyone have any experience running Asterisk on an Amazon AWS system?
If so, what type of Amazon AWS configuration are you using?
If you have the potential to be playing prompts and/or music on hold for all
channels at once, what is you channel capacity?

Is it recommended to run on dedicated hardware when encountering issues like
this?

Have a great day!
Dan
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.digium.com/pipermail/asterisk-users/attachments/20190821/5cbb5b5f/attachment.html>

Dan,

I don't run Asterisk on AWS, but I do on ESXi.  Are you running a version of
Asterisk before 13?  Newer versions Asterisk handle timing better that don't
require a hardware timing source.

I'm running Asterisk 13 on a small 60 phone system without issues under ESXi
6.0

Doug

On Wed, Aug 21, 2019, at 1:54 PM, Doug Lytle wrote:
> Dan,
> 
> I don't run Asterisk on AWS, but I do on ESXi.  Are you running a 
> version of Asterisk before 13?  Newer versions Asterisk handle timing 
> better that don't require a hardware timing source.
> 
> I'm running Asterisk 13 on a small 60 phone system without issues under
ESXi 6.0
Ultimately it still does rely on timing and depending on the conditions it can
have an impact on things. If timing is poor and the jitterbuffer on an endpoint
can't cope then you will have audio problems. AWS doesn't guarantee this
and depending on the host node problems can occur. It will come up when playing
back audio files, using ConfBridge, and music on hold.

-- 
Joshua C. Colp
Digium - A Sangoma Company | Senior Software Developer
445 Jan Davis Drive NW - Huntsville, AL 35806 - US
Check us out at: www.digium.com & www.asterisk.org

Thanks Doug.

We are running Asterisk 16.3.0 so I think we're on a pretty good version for
the timing.

We have Asterisk running on ESXi here and it's running at several customer
sites in various VM environments.
Ironically, none of them have the latency sensitivity set to high.  This is
something I didn't realize may be needed until encountering the issue during
load testing this week.
I would guess our largest at present has around 100 simultaneous calls and we
haven't run into problems.

At 200 calls, the Amazon AWS VM has 0 issues with audio.
I believe it even was good at 250.
It was when the tester went to 300 calls that I am told audio is often choppy.

I tried performing the noload on res_timing_timerfd.so and the person testing
indicated it did not help.

I forgot to mention, this is on an Ubuntu 16 OS with the latest packages.

Our second Asterisk box receives the call and bridges them with another endpoint
(allowing us to hear it on various phones)
The debugging on this box isn't indicating any issues (at least from what I
can tell).
Obviously a lot is going on (basically 600 calls).

Neither box is indicating jitter issues.  

Dan

-----Original Message-----
From: asterisk-users <asterisk-users-bounces at lists.digium.com> On
Behalf Of Doug Lytle
Sent: Wednesday, August 21, 2019 11:54 AM
To: Asterisk Users Mailing List - Non-Commercial Discussion <asterisk-users
at lists.digium.com>
Subject: Re: [asterisk-users] Amazon AWS question

Dan,

I don't run Asterisk on AWS, but I do on ESXi.  Are you running a version of
Asterisk before 13?  Newer versions Asterisk handle timing better that don't
require a hardware timing source.

I'm running Asterisk 13 on a small 60 phone system without issues under ESXi
6.0

Doug

-- 
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com --

Check out the new Asterisk community forum at: https://community.asterisk.org/

New to Asterisk? Start here:
      https://wiki.asterisk.org/wiki/display/AST/Getting+Started

asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
   http://lists.digium.com/mailman/listinfo/asterisk-users