asterisk users - Jun 2006 - Audio problems on Zap & SIP, local network, not IRQ related?

I am trying to get to the bottom of audio clicks, pops, dropouts with my 
Asterisk server. These occur even when the system is under minimal load 
(e.g. 1 Zap device in a queue being played music on hold) and occurs with 
both Zap and Sip devices so isn't network related. The audio problems occur 
at the same time on all channels and seems to be when Asterisk "gets
busy"
and uses >50% CPU e.g. just after an announcement ("you are first in the
queue"). However running ztspeed (which takes CPU usage to 100%) seems to 
have no impact on zttest numbers or the audio.

I have the following setup:
Fedora Core 4 2.6.16-1.2111 smp kernel
TDM400 with 2 FXO and 2 FXS modules
Various SIP devices
SCSI hard drives
2 x P3 processors 500 MHz
1GB RAM

Based on what I've read I would have thought that this setup could easily 
handle one call being played music on hold in a queue.

I have read 
http://www.asteriskguru.com/tutorials/pci_irq_apic_tdm_ticks_te410p_te405p_noise.html
and various other postings to try and resolve this issue.

When I run zttest I get 99.987793% most of the time but occasionally it 
drops to 99.926758% which often corresponds with the audio degradation. I 
have, however, noticed audio degradation at 99.987793% and good audio at 
<99.87%.

I have followed the instructions on disabling the Linux frame buffer 
http://www.voip-info.org/wiki/index.php?page=Asterisk+disable+frame+buffer. 
(There was no graphic on boot and no vga entry in /boot/grub/menu.lst

I am not running X windows.

The processors are not hyperthreading.

I have disabled ACPI by adding acpi=off to /etc/grub.conf

cat /proc/interrupts indicates no shared IRQs
           CPU0       CPU1
  0:   21418593   21478150    IO-APIC-edge  timer
  1:        142         62    IO-APIC-edge  i8042
  8:          1          0    IO-APIC-edge  rtc
  9:          0          0   IO-APIC-level  acpi
 16:      60618      60379   IO-APIC-level  megaraid
 17:    4185189    4271566   IO-APIC-level  aic7xxx, aic7xxx
 18:    1589117         12   IO-APIC-level  eth0
 19:          0          0   IO-APIC-level  uhci_hcd:usb1
 20:   85250811   86274087   IO-APIC-level  wctdm
NMI:          0          0
LOC:   42898234   42898233
ERR:          0
MIS:          0

lspci -v
00:0a.0 Communication controller: Tiger Jet Network Inc. Tiger3XX Modem/ISDN 
interface
        Subsystem: Unknown device b100:0003
        Flags: bus master, medium devsel, latency 32, IRQ 20
        I/O ports at d800 [size=256]
        Memory at fe100000 (32-bit, non-prefetchable) [size=4K]
        Capabilities: [40] Power Management version 2

lspci -vb
00:0a.0 Communication controller: Tiger Jet Network Inc. Tiger3XX Modem/ISDN 
interface
        Subsystem: Unknown device b100:0003
        Flags: bus master, medium devsel, latency 32, IRQ 11
        I/O ports at d800
        Memory at fe100000 (32-bit, non-prefetchable)
        Capabilities: [40] Power Management version 2

No other device is using IRQ 20 (or 11)

Running zttool shows no alarms, IRQ misses on the TDM400P

cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 7
model name      : Pentium III (Katmai)
stepping        : 3
cpu MHz         : 498.847
cache size      : 512 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov 
pat pse36 mmx fxsr sse
bogomips        : 999.41

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 7
model name      : Pentium III (Katmai)
stepping        : 3
cpu MHz         : 498.847
cache size      : 512 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov 
pat pse36 mmx fxsr sse
bogomips        : 997.59

Does anyone have any further suggestions? I would really appreciate any 
other pointers.

Regards

Cameron

I have made the following additional changes:
- enabling MMX extensions in the Asterisk Makefile and remade, installed
- disabled parallel, serial, and mouse ports in the BIOS
- reenabled ACPI as I was getting errors in the log file

The audio problems still exist. Any further advice on how to improve the 
audio quality would be greatly appreciated.

Regards

Cameron
----- Original Message ----- 
From: "kjcsb" <kjcsb@orcon.net.nz>
To: "Asterisk Users Mailing List - Non-Commercial Discussion" 
<asterisk-users@lists.digium.com>
Sent: Friday, June 02, 2006 9:31 PM
Subject: Audio problems on Zap & SIP, local network, not IRQ related?

>I am trying to get to the bottom of audio clicks, pops, dropouts with my 
>Asterisk server. These occur even when the system is under minimal load 
>(e.g. 1 Zap device in a queue being played music on hold) and occurs with 
>both Zap and Sip devices so isn't network related. The audio problems
occur
>at the same time on all channels and seems to be when Asterisk "gets
busy"
>and uses >50% CPU e.g. just after an announcement ("you are first in
the
>queue"). However running ztspeed (which takes CPU usage to 100%) seems
to
>have no impact on zttest numbers or the audio.
>
> I have the following setup:
> Fedora Core 4 2.6.16-1.2111 smp kernel
> TDM400 with 2 FXO and 2 FXS modules
> Various SIP devices
> SCSI hard drives
> 2 x P3 processors 500 MHz
> 1GB RAM
>
> Based on what I've read I would have thought that this setup could
easily
> handle one call being played music on hold in a queue.
>
> I have read 
>
http://www.asteriskguru.com/tutorials/pci_irq_apic_tdm_ticks_te410p_te405p_noise.html
> and various other postings to try and resolve this issue.
>
> When I run zttest I get 99.987793% most of the time but occasionally it 
> drops to 99.926758% which often corresponds with the audio degradation. I 
> have, however, noticed audio degradation at 99.987793% and good audio at 
> <99.87%.
>
> I have followed the instructions on disabling the Linux frame buffer 
> http://www.voip-info.org/wiki/index.php?page=Asterisk+disable+frame+buffer.
> (There was no graphic on boot and no vga entry in /boot/grub/menu.lst
>
> I am not running X windows.
>
> The processors are not hyperthreading.
>
> I have disabled ACPI by adding acpi=off to /etc/grub.conf
>
> cat /proc/interrupts indicates no shared IRQs
>           CPU0       CPU1
>  0:   21418593   21478150    IO-APIC-edge  timer
>  1:        142         62    IO-APIC-edge  i8042
>  8:          1          0    IO-APIC-edge  rtc
>  9:          0          0   IO-APIC-level  acpi
> 16:      60618      60379   IO-APIC-level  megaraid
> 17:    4185189    4271566   IO-APIC-level  aic7xxx, aic7xxx
> 18:    1589117         12   IO-APIC-level  eth0
> 19:          0          0   IO-APIC-level  uhci_hcd:usb1
> 20:   85250811   86274087   IO-APIC-level  wctdm
> NMI:          0          0
> LOC:   42898234   42898233
> ERR:          0
> MIS:          0
>
> lspci -v
> 00:0a.0 Communication controller: Tiger Jet Network Inc. Tiger3XX 
> Modem/ISDN interface
>        Subsystem: Unknown device b100:0003
>        Flags: bus master, medium devsel, latency 32, IRQ 20
>        I/O ports at d800 [size=256]
>        Memory at fe100000 (32-bit, non-prefetchable) [size=4K]
>        Capabilities: [40] Power Management version 2
>
> lspci -vb
> 00:0a.0 Communication controller: Tiger Jet Network Inc. Tiger3XX 
> Modem/ISDN interface
>        Subsystem: Unknown device b100:0003
>        Flags: bus master, medium devsel, latency 32, IRQ 11
>        I/O ports at d800
>        Memory at fe100000 (32-bit, non-prefetchable)
>        Capabilities: [40] Power Management version 2
>
> No other device is using IRQ 20 (or 11)
>
> Running zttool shows no alarms, IRQ misses on the TDM400P
>
> cat /proc/cpuinfo
> processor       : 0
> vendor_id       : GenuineIntel
> cpu family      : 6
> model           : 7
> model name      : Pentium III (Katmai)
> stepping        : 3
> cpu MHz         : 498.847
> cache size      : 512 KB
> fdiv_bug        : no
> hlt_bug         : no
> f00f_bug        : no
> coma_bug        : no
> fpu             : yes
> fpu_exception   : yes
> cpuid level     : 2
> wp              : yes
> flags           : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca 
> cmov pat pse36 mmx fxsr sse
> bogomips        : 999.41
>
> processor       : 1
> vendor_id       : GenuineIntel
> cpu family      : 6
> model           : 7
> model name      : Pentium III (Katmai)
> stepping        : 3
> cpu MHz         : 498.847
> cache size      : 512 KB
> fdiv_bug        : no
> hlt_bug         : no
> f00f_bug        : no
> coma_bug        : no
> fpu             : yes
> fpu_exception   : yes
> cpuid level     : 2
> wp              : yes
> flags           : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca 
> cmov pat pse36 mmx fxsr sse
> bogomips        : 997.59
>
> Does anyone have any further suggestions? I would really appreciate any 
> other pointers.
>
> Regards
>
> Cameron

> I've read your post on the asterisk mailing list. Agree that the specs
> of that box should easily handle one call with decent quality. The only
> thing I can think of right now is to start using the IRQ affinity stuff
> to move the scsi & ethernet modules over to e.g. CPU2 and let the wctdm
> driver stick to CPU1. You may also want to read the part about setting
> the latency of the card. Iirc the higher the value the better it is for
> a Digium card but check it in the docs at
>
http://www.asteriskguru.com/tutorials/pci_irq_apic_tdm_ticks_te410p_te405p_noise.html
> or the attached file.
>
I have attempted to change the SMP affinity so that the TDM400 is handled by 
CPU1:
echo 2 > /proc/irq/20/smp_affinity
cat /proc/irq/20/smp_affinity
00000002

However I can see the interrupts for IRQ 20 are still incrementing on both 
CPUs. Indeed, after about 10 seconds...
cat /proc/irq/20/smp_affinity
00000001


On further investigation, the SMP affinity on ALL of the IRQs is set to 
00000001. This implies that everything is handled by CPU0, which it clearly 
is not!

I'll do some more research on this but in the meantime if anyone has any 
advice on this issue I would appreciate it.
> I'm not familiar with the megaraid module. You seem to have an adaptec
> scsi controller. Why need the megaraid module? And if you don't need
the
> usb uhci module, disable usb in the bios. Come to think of it, since you
> use scsi, disable the IDE ports too.
Megaraid is Dell's SCSI hardware RAID controller (this is a Dell Poweredge 
2300). I believe the Adaptec stuff is what the Tape, CDROM(!) etc run off. 
There are no options in the BIOS to disable USB or IDE.
> Just saw your other posting to the asterisk mailing list. Afaik it is
> not advisable to use MMX so disable it in the Makefile again and
> recompile. I don't know which distro you use but if it's CentOS,
RHEL or
> Fedora Core 4 or 5 you can use the rpms at http://laimbock.com/asterisk/
>
> Regards,
> Patrick
>
I'm running FC4 but some files are patched for New Zealand so I don't
think
I can use these RPMs.

Regards

Cameron