asterisk users - Nov 2005 - Missing audio from Zaptel channels

I have cross-posted this all over the place, and sent a copy directly to digium
support, in the hope of getting to the bottom of a problem that has me pulling
my hair out.

I currently have 2 production PSTN gateway servers, running asterisk 1.2beta and
TE406P cards (upgraded 405 cards, with hardware echo cancelers that we recently
purchased on recommendation). We went to the beta version after installing the
cancelers, as 1.0.9 kept segfaulting with the cancelers installed. Our PRIs
terminate on a DMS100, at the same premises where our servers are co-located.

Also in my farm, I have a dedicated IVR server, a VOIP gateway (SIP/IAX/H.323)
and clustered MySQL servers running as FastAGI servers, to remove processor load
from the PSTN servers. All servers are connected via gigabit Ethernet, and
use IAX trunking for inter-server communications.

I have been through _everything_ possible to be sure that I don't have any
zaptel timing/irq problems (framebuffer, apic, acpi, smp irq affinity, irq
latency, etc. etc) and have good zttest results with no frame slips, pops or
clicks.

After my PSTN gateway servers have been running for a few hours, I notice that
some missing audio creeps into the start of each call (makes no difference if
the call is ZAP-ZAP native bridge or ZAP-IAX). At best, you miss the first
syllable of the first word. At worst, you can miss the first 3 or 4 seconds of
audio. Further investigation shows that asterisk is lagging after the second leg
of the call is answered (i.e. the time taken to bridge the channels gets 
longer). If the resultant call is a Zaptel native bridge, then the remaining 
audio is fine. If the resultant call is not zaptel natively bridged (eg. call is
routed via another server, or asterisk remains in the media stream for another 
reason) then significant delay exists from one end of the call to another 
(simply put, asterisk seems to slow down).

If I restart asterisk (even without removing and reloading zaptel drivers), 
calls are OK again for a period (typically around 12 hours). A workaround is to 
simply to install a cron job that periodically restarts asterisk when it's
idle,
  but this is a less than ideal solution from my perspective.

Something is definitely changing over time. A memory leak? Runaway process? I
really need help in trying to troubleshoot this, as I've run completely out
of
both patience and ideas.


-- 
=========================================Rod Bacon
Empowered Communications
Ground Floor, 102 York St. South Melbourne
Victoria, Australia. 3205
Phone: +613 99401600 Fax: +613 99401650
FWD: 512237 ICQ: 5662270
==========================================

For those who are interested, the problem appears to NOT exist in 1.2Beta2.

=========================================Rod Bacon
Empowered Communications
Ground Floor, 102 York St. South Melbourne
Victoria, Australia. 3205
Phone: +613 99401600    Fax: +613 99401650
FWD: 512237                   ICQ: 5662270
=========================================

Rod Bacon wrote:
> I have cross-posted this all over the place, and sent a copy directly to 
> digium
> support, in the hope of getting to the bottom of a problem that has me 
> pulling
> my hair out.
> 
> I currently have 2 production PSTN gateway servers, running asterisk 
> 1.2beta and
> TE406P cards (upgraded 405 cards, with hardware echo cancelers that we 
> recently
> purchased on recommendation). We went to the beta version after 
> installing the
> cancelers, as 1.0.9 kept segfaulting with the cancelers installed. Our PRIs
> terminate on a DMS100, at the same premises where our servers are 
> co-located.
> 
> Also in my farm, I have a dedicated IVR server, a VOIP gateway 
> (SIP/IAX/H.323)
> and clustered MySQL servers running as FastAGI servers, to remove 
> processor load
> from the PSTN servers. All servers are connected via gigabit Ethernet, and
> use IAX trunking for inter-server communications.
> 
> I have been through _everything_ possible to be sure that I don't have
any
> zaptel timing/irq problems (framebuffer, apic, acpi, smp irq affinity, irq
> latency, etc. etc) and have good zttest results with no frame slips, 
> pops or clicks.
> 
> After my PSTN gateway servers have been running for a few hours, I 
> notice that
> some missing audio creeps into the start of each call (makes no 
> difference if
> the call is ZAP-ZAP native bridge or ZAP-IAX). At best, you miss the first
> syllable of the first word. At worst, you can miss the first 3 or 4 
> seconds of
> audio. Further investigation shows that asterisk is lagging after the 
> second leg
> of the call is answered (i.e. the time taken to bridge the channels gets 
> longer). If the resultant call is a Zaptel native bridge, then the 
> remaining audio is fine. If the resultant call is not zaptel natively 
> bridged (eg. call is routed via another server, or asterisk remains in 
> the media stream for another reason) then significant delay exists from 
> one end of the call to another (simply put, asterisk seems to slow down).
> 
> If I restart asterisk (even without removing and reloading zaptel 
> drivers), calls are OK again for a period (typically around 12 hours). A 
> workaround is to simply to install a cron job that periodically restarts 
> asterisk when it's idle,  but this is a less than ideal solution from
my
> perspective.
> 
> Something is definitely changing over time. A memory leak? Runaway 
> process? I
> really need help in trying to troubleshoot this, as I've run completely
> out of
> both patience and ideas.
> 
>