SYNOPSIS
Erratic red alarm T1 PRI on asterisk, but zttool running concurrently
during alarm shows no errors, irq misses, or alarms, on any span.
Using asterisk and quad Digium T405P, configured as follows:
Span 1 connects to ISDN PRI (fractional 8 B channels, D channel 24).
Span 2 connects to T1 Mux and analog stations.
Span 3 connects to ISDN PRI Nortel BCM hybrid key system digital trunk.
Span 4 is not configured.
Host system is Athlon > 1.8GHz, 512MB running Gentoo Linux 2.4.20
vanilla kernel, RAID 1 (software Linux), 2 x IDE 80GB. T1 card shares
no interrupt.
After 12 hours to several days, asterisk detects a red alarm on the
configured channels 1 through 8 of span 1 (ISDN PRI.) Concurrently run
zttool shows no alarms, no irq misses, and no errors on any span.
When the alarm is first detected, it "bounces" several times, then
quits
for 6 hours or so, then recurs. Suspecting a timing problem, the
configuration has been repeatedly checked for proper clocking (to CO
span 1) and this is also verified by zttool which shows sync source as
"Card 0, span1" on all configured spans. The original Digium TE410P
Quad card was replaced with a Digium T405P with no improvement. Telco
has replaced both Adtran HDSL2 smartjack and repeater cards. Problem is
independent of traffic and occurs late night, early morning, weekends or
during load. Simulated load of CPU does not produce any failure and CPU
load is otherwise negligible. The utility zttest has been run on the
new T405P card with no errors. Zttool shows red alarm if a span is
disconnected, LB when under loopback test, but otherwise shows no errors
on any span.
Span 1 connects to telco Adtran HDSL2-R. Numerous loop around tests and
30m span pattern tests were performed over several days, and the span
tested clear in both directions through to CSU with no errors.
Span 2 connects to a Zhone T1 mux and shows no alarms or errors, either
from asterisk console or messages (zap channels 25 -48).
Span 3 connects to Nortel BCM PRI (fractional 8 channels, plus D on
24(channels 9-23 are deprovisioned); it likewise shows no errors.
Span 4 is not connected and not configured. It has been swapped with
Span 1 on occasion with no improvement.
Upon occurrence of an red alarm condition on Span 1, all calls are
dropped. Successive call attempts during the red alarm condition
encounter 120IPM congestion tone. However, calls can still be made
between BMC (Span 3) and the T1 (Span 2) when the red alarm is reported
by the asterisk console on Span 1. Specifically, during a red alarm,
only service through Span 1 is affected.
Here is an example from the messages log (grepped for "channel 1:
Red"):
Jun 10 04:02:29 WARNING[163851]: Detected alarm on channel 1: Red Alarm
Jun 12 21:02:17 WARNING[163851]: Detected alarm on channel 1: Red Alarm
Jun 12 21:05:12 WARNING[163851]: Detected alarm on channel 1: Red Alarm
Jun 12 21:08:23 WARNING[163851]: Detected alarm on channel 1: Red Alarm
Jun 12 21:47:01 WARNING[163851]: Detected alarm on channel 1: Red Alarm
Jun 12 21:58:15 WARNING[163851]: Detected alarm on channel 1: Red Alarm
... the alarm continues bouncing, then eventually abates...
Jun 13 01:44:57 WARNING[163851]: Detected alarm on channel 1: Red Alarm
Jun 13 01:52:19 WARNING[163851]: Detected alarm on channel 1: Red Alarm
Jun 13 12:15:15 WARNING[163851]: Detected alarm on channel 1: Red Alarm
Jun 13 12:20:32 WARNING[163851]: Detected alarm on channel 1: Red Alarm
Jun 13 12:25:45 WARNING[163851]: Detected alarm on channel 1: Red Alarm
Jun 13 12:40:02 WARNING[163851]: Detected alarm on channel 1: Red Alarm
Jun 13 12:57:49 WARNING[163851]: Detected alarm on channel 1: Red Alarm
.. it then subsides for several hours, e.g., none further as of Sun Jun
13 17:39:39 EDT 200.
Debug output is apparently is suppressed during alarm. (This has been
reported on bug report separately).
Configuration (/etc/zapata.conf)
span=1,1,3,esf,b8zs
span=2,0,0,esf,b8zs
span=3,0,0,esf,b8zs
#span=4,1,3,esf,b8zs
NOTES
All cables, protectors, and the like, have been verified, reterminated
or swapped. However, these types of errors should have been discovered
via a short-term error test.
The server and all connected telephony equipment is powered by two
separate UPS's, and report no downtime during the red alarm events.
Except during artificial load tests, the server's CPU usage has never
risen above 1%.
Error #500 errors occur sporadically, but without any clear relationship
to the red alarms. UDMA was reduced from UDMA 5 to UDMA 3, but this
failed to correct the error #500 problem.
Debug and warning message logs have been retained. Inexplicably, the red
alarms appear to occur with no external stimulus.
QUESTIONS
1. How can the asterisk messages log show a red alarm, yet the zttool
utility (running concurrently and watched during alarm transition) shows
no red alarm?
2. What are the conditions that asterisk uses to declare red alarms? How
does this differ from the zttool utility?
3. Any other ideas?
