asterisk users - Mar 2005 - Intermittent volume deterioration in conferences

I wonder if anyone can suggest ways to diagnose an infuriating problem
being experienced by customers of a company I did a large Asterisk
project for.

First some background:

The system is a conferencing system using a modified MeetMe. There are
seven Asterisk boxes (we call them bridges) each with four T1 PRIs into a
TE405P. No VoIP is involved. A conference is always local to a single
bridge.  The conference leader has a control screen and may dial into the
bridge, or may instruct the bridge to dial him/her. Once the leader is in
the conference, they instruct the bridge to dial each other participant.
Each conference is recorded locally in the Asterisk system. The bridges
are in Oklahoma and all the leaders and most of the participants are all
over Texas.

The problem:

For the first three or four months of operation everything went very
well, but from early February the customer started reporting problems
with the volume of audio. Initially the reports seemed to be localized
to a particular area of Texas, and to be small in number. Over time,
they have increased in frequency and been reported from different areas.

Sometimes one participant can't be heard very well by the others, and is
also faint on the recording. Other times a participant has trouble
hearing the others, but the others are ok on the recording. There does
not seem to be any significant distortion, just faint volume.

It sounds to me like a phone network issue, but proving that is turning
out to be a nightmare. The fact that it is not confined to one bridge
but is randomly spread across them would seem to suggest it is not a
bridge hardware problem, because it is unlikely to happen in them all.
No changes were made to the hardware, Zaptel drivers or Asterisk on the
bridges since installation.

A day or so ago we disabled echo cancellation on the zap channels, to see
if that would make a difference, but it doesn't seem to have. It still
wouldn't explain why the problem did not previously exist, and started
happening spontaneously.

Sometimes if it's really difficult for people to hear, the leader closes
the conference and reverts to their older conferencing system (that our
system replaced), and reports that the volume is then fine. I don't know
where the older system is located, but I believe it is more local. This is
obviously a worrying scenario.

If anyone can suggest any ideas of ways to tackle the problem, and to
determine whether it really is the Asterisk bridges or the phone systems,
I would be very, very grateful, as it is turning into a nightmare!

Cheers
Tony
-- 
Tony Mountifield
Work: tony@softins.co.uk - http://www.softins.co.uk
Play: tony@mountifield.org - http://tony.mountifield.org