asterisk users - Oct 2013 - Dahdi_dummy is more accurate than core timer?

Hi,

I have some servers that are dedicated to do meetme conferencing. From 
some previous test i concluded that I need to use dahdi_dummy as it is 
more accurate.

If I did use the core timers in dahdi (not loading dahdi_dummy) I got 
bad quality in the conferences and dahdi_test showed 99.6% as worst.

I thought maybe the issue as bad hardware for the timing or something 
else. But today I re-ran these tests on another server showing the same 
thing.

- Can anybody comment on why DAHDI with core timers drop down to 99.6% 
occasionally?
- Is a hardware-card for timing the most efficient way to get timing 
even if I just use the card for the timing?

Below is some stats (trimmed, but you get the idea).

Thanks!

/Johan


** With Dahdi 2.7.0.1, and core timers:

99.998% 99.611% 99.615% 99.997% 99.993% 99.997% 99.996% 99.608%
99.999% 99.612% 99.607% 99.613% 99.999% 99.998% 99.994% 99.609%

--- Results after 177 passes ---
Best: 100.000% -- Worst: 99.604% -- Average: 99.901099%
Cummulative Accuracy (not per pass): 99.998


** With Dahdi 2.7.0.1, and dahdi_dummy loaded:

99.993% 99.998% 99.998% 99.993% 99.996% 99.998% 99.996% 99.998%
99.998% 99.997% 99.999% 99.998% 99.996% 99.998% 99.999% 99.997%

--- Results after 177 passes ---
Best: 100.000% -- Worst: 99.993% -- Average: 99.997738%
Cummulative Accuracy (not per pass): 99.998


** With Dahdi 2.7.0.1, and Wildcard TE220 providing timing

99.981% 99.983% 99.983% 99.982% 99.983% 99.982% 99.984% 99.981%
99.982% 99.983% 99.984% 99.981% 99.980% 99.984% 99.983% 99.983%

--- Results after 177 passes ---
Best: 99.996% -- Worst: 99.974% -- Average: 99.982104%
Cummulative Accuracy (not per pass): 99.982

Kernel: 2.6.32-5-openvz-amd64

On 02/10/13 12:17, Johan Wilfer wrote:
> Hi,
>
> I have some servers that are dedicated to do meetme conferencing. From 
> some previous test i concluded that I need to use dahdi_dummy as it is 
> more accurate.
>
> If I did use the core timers in dahdi (not loading dahdi_dummy) I got 
> bad quality in the conferences and dahdi_test showed 99.6% as worst.
>
> I thought maybe the issue as bad hardware for the timing or something 
> else. But today I re-ran these tests on another server showing the 
> same thing.
>
> - Can anybody comment on why DAHDI with core timers drop down to 99.6% 
> occasionally?
> - Is a hardware-card for timing the most efficient way to get timing 
> even if I just use the card for the timing?
>
>
Its a little different when you are using meetme as its an application 
built into dahdi itself and not a native asterisk application. It will 
therefore always use dahdi for its timing. If dahdi doesnt have a 
hardware interface (sangoma sell a usb based timing source if you want a 
hardware source) then it will use a software timing source of some form. 
I dont know what method it uses.

Asterisk itself will use dahdi for timing but if res_timerfd is 
available it will use that itself. With your kernel version timerfd 
should be available.
So if you run the following command asterisk will perform 1024 ticks 
over 1000ms which is equivalent to the dahdi test of 8192 over 8000ms 
(if you run it a few times). You can see in my case asterisk is using 
timerfd and no issues at all with the timing.

 > timing test 1024
Attempting to test a timer with 1024 ticks per second.
Using the 'timerfd' timing module for this test.
It has been 1000 milliseconds, and we got 1024 timer ticks

We are developing a conferencing feature and rather than use meetme with 
its dahdi requirement we are using confbridge10 instead so we dont have 
to have dahdi installed and it will use the better timerfd timing source.

More information about timing sources can be found at :-
https://wiki.asterisk.org/wiki/display/AST/Timing+Interfaces

On Wed, Oct 02, 2013 at 01:17:15PM +0200, Johan Wilfer
wrote:
> Hi,
> 
> I have some servers that are dedicated to do meetme conferencing.
> From some previous test i concluded that I need to use dahdi_dummy
> as it is more accurate.
> 
> If I did use the core timers in dahdi (not loading dahdi_dummy) I
> got bad quality in the conferences and dahdi_test showed 99.6% as
> worst.
Hmm...this is the first report I've heard of dahdi_dummy being more
performant than the core timer.

I wonder if this has something to do with the fact that you're
running under 2.6.32-5-openvz-amd64 which might be doing more work
in the system timer (which is where the standard core timer work is
processed).

If you update to the latest 2.6.32-openvz kernel do you still have
the audio problems in conferneces?

After a quick search I found this stackoverflow question which talks
about too much time spent in software interrupt context on
2.6.32-5-openvz being resolved by a kernel update. This could
definitely be related:

http://serverfault.com/questions/399837/software-interrupts-cpu-time-is-high-and-keeps-growing
> I thought maybe the issue as bad hardware for the timing or
> something else. But today I re-ran these tests on another server
> showing the same thing.
> 
> - Can anybody comment on why DAHDI with core timers drop down to
>   99.6% occasionally?
This is because when using the core timer, the timer is only
scheduled to fire ever 4ms. The differences in each *individual*
measurement you see is due to timer jitter + the increased interval
leaking more of the slight jitter up to userspace. However, this
isn't typically a problem when mixing audio in 20ms chunks by
default as is typically done when you're using meetme conferences.

The number that is generally more interesting is the "Cummulative
Accuracy" which shows over the entire dahdi_test how close DAHDI was
to processing the expected amount of audio.

If you run dahdi_test with the -vv flag you can see how sometimes it's a
little
over and sometimes a little under. This is running under virtual box on a
system configured with 4ms ticks and NO_HZ:

  # grep HZ /boot/config-$(uname -r)
  CONFIG_RCU_FAST_NO_HZ=y
  CONFIG_NO_HZ=y
  # CONFIG_HZ_100 is not set
  CONFIG_HZ_250=y
  # CONFIG_HZ_300 is not set
  # CONFIG_HZ_1000 is not set
  CONFIG_HZ=250
  CONFIG_MACHZ_WDT=m
  # dahdi_test -vv -c 10
  Opened pseudo dahdi interface, measuring accuracy...
  
  8192 samples in 8191.312 system clock sample intervals (99.992%)
  8192 samples in 8198.448 system clock sample intervals (99.921%)
  8192 samples in 8184.913 system clock sample intervals (99.913%)
  8192 samples in 8191.640 system clock sample intervals (99.996%)
  8192 samples in 8191.720 system clock sample intervals (99.997%)
  8192 samples in 8192.128 system clock sample intervals (99.998%)
  8192 samples in 8190.824 system clock sample intervals (99.986%)
  8192 samples in 8192.256 system clock sample intervals (99.997%)
  8192 samples in 8191.576 system clock sample intervals (99.995%)
  8192 samples in 8191.631 system clock sample intervals (99.995%)
  --- Results after 10 passes ---
  Best: 99.998% -- Worst: 99.913% -- Average: 99.978985%
  Cummulative Accuracy (not per pass): 99.996

On another system with 10 ms timer ticks the jitter is increased, but even this
system does not have any problems mixing audio in meetme conferences:

  $ zcat /proc/config.gz | grep HZ
  # CONFIG_RCU_FAST_NO_HZ is not set
  CONFIG_NO_HZ=y
  CONFIG_HZ_100=y
  # CONFIG_HZ_250 is not set
  # CONFIG_HZ_300 is not set
  # CONFIG_HZ_1000 is not set
  CONFIG_HZ=100
  CONFIG_MACHZ_WDT=m
  $ sudo dahdi_test -vv -c 10
  Opened pseudo dahdi interface, measuring accuracy...
  
  8192 samples in 8159.872 system clock sample intervals (99.608%)
  8192 samples in 8159.400 system clock sample intervals (99.602%)
  8192 samples in 8239.680 system clock sample intervals (99.418%)
  8192 samples in 8159.800 system clock sample intervals (99.607%)
  8192 samples in 8239.576 system clock sample intervals (99.419%)
  8192 samples in 8159.752 system clock sample intervals (99.606%)
  8192 samples in 8159.848 system clock sample intervals (99.608%)
  8192 samples in 8239.601 system clock sample intervals (99.419%)
  8192 samples in 8159.624 system clock sample intervals (99.605%)
  8192 samples in 8239.144 system clock sample intervals (99.425%)
  --- Results after 10 passes ---
  Best: 99.608% -- Worst: 99.418% -- Average: 99.531611%
  Cummulative Accuracy (not per pass): 99.995

When you explictly load the dahdi_dummy module, your results can
change in a couple of ways.  1) dahdi_dummy tries to always schedule
the system timer to fire at 1ms intervals (which it only will if the
system is configured for CONFIG_HZ=1000).  2) If on a newer kernel,
dahdi dummy will use kernel high resolution timers to increase the
precision of the timer.  However this shouldn't be necessary since
the jitter in the normal kernel timer should be small compared to
all the other jitter in a voip system.
> - Is a hardware-card for timing the most efficient way to get timing
>   even if I just use the card for the timing?
I personally do not think so. The most efficient way should just be
to allow the normal kernel timers to also provide timing to your
asterisk system without the overhead of processing another
interrupt.

[snip]


So I guess I would be interested to hear if a kernel update is all
you need to have good results with the core timer.

If not...it might be worth investigating and measuring adding
hrtimer support into the core timer.

Cheers,
Shaun

-- 
Shaun Ruffell
Digium, Inc. | Linux Kernel Developer
445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
Check us out at: www.digium.com & www.asterisk.org