asterisk users - Jun 2004 - FWIW- Cisco 1750 dropped packets and choppy audio

This email is intended to document an issue for anyone searching the archives.

We had a problem yesterday with _all_ iax2 and sip sessions; no reasonable
conversation could be established due to extremely choppy audio in one
direction only (outbound from * to distant sip phones and distant * boxes).
We were running HEAD from June 8th.

While diagnosing the root cause, we monitored bandwidth utilization at the
asterisk-connected managed-switch as well as at the Cisco 1750 Internet
interface. We observed consistent/even data flows to/from the * box, however
the outbound Cisco interface indicated more inbound traffic than outbound
traffic by a considerable/noticeable amount. Both iax2 and sip sessions were 
impacted exactly the same regardless of the codec being used.

In the haste to identify the root-cause, the Cisco 1750 was rebooted
(Version 12.2(4)T7) and the problem disappeared. A Service-Policy had been
applied to the outbound interface for QoS purposes. Removing the policy
while a poor quality session was in progress had zero impact. Unfortunitly, 
no other Cisco data was gathered before the reboot. We're waiting for
reoccurrence to gather additional doc. We are 100% confident this is a
Cisco issue as opposed to * or any other resource. (Someone, maybe Eric, 
mentioned a Cisco QoS bug previously on this list. Indications are this 
might be the bug that person had mentioned.)

The Cisco had been in use for a couple of years and we've never seen this
issue arise prior to yesterday. The * box had been in semi-production since
late last year and has been stable (given the expected issues associated
with using HEAD as opposed to Stable).

There were no logged messages from the Cisco even though syslog messages are
normally monitored closely.

Rich

On Thu, 2004-06-10 at 23:27, Rich Adamson wrote:
> This email is intended to document an issue for anyone searching the
archives.
> 
> We had a problem yesterday with _all_ iax2 and sip sessions; no reasonable
> conversation could be established due to extremely choppy audio in one
> direction only (outbound from * to distant sip phones and distant * boxes).
> We were running HEAD from June 8th.
> 
> While diagnosing the root cause, we monitored bandwidth utilization at the
> asterisk-connected managed-switch as well as at the Cisco 1750 Internet
> interface. We observed consistent/even data flows to/from the * box,
however
> the outbound Cisco interface indicated more inbound traffic than outbound
> traffic by a considerable/noticeable amount. Both iax2 and sip sessions
were
> impacted exactly the same regardless of the codec being used.
> 
> In the haste to identify the root-cause, the Cisco 1750 was rebooted
> (Version 12.2(4)T7) and the problem disappeared. A Service-Policy had been
> applied to the outbound interface for QoS purposes. Removing the policy
> while a poor quality session was in progress had zero impact. Unfortunitly,
> no other Cisco data was gathered before the reboot. We're waiting for
> reoccurrence to gather additional doc. We are 100% confident this is a
> Cisco issue as opposed to * or any other resource. (Someone, maybe Eric, 
> mentioned a Cisco QoS bug previously on this list. Indications are this 
> might be the bug that person had mentioned.)
> 
> The Cisco had been in use for a couple of years and we've never seen
this
> issue arise prior to yesterday. The * box had been in semi-production since
> late last year and has been stable (given the expected issues associated
> with using HEAD as opposed to Stable).
> 
> There were no logged messages from the Cisco even though syslog messages
are
> normally monitored closely.
This may or may not be related, I have a Cisco 8xx (something, ADSL
router) and had very poor audio in both directions, I was seeing very
large number of packets/sec in both directions. I solved it by turning
off all the debugging on the cisco. (I had debug ip packet turned on for
packets matching a specific access-list).

As a side note, I first noticed the CPU load on the cisco sitting on
100% during an attempted phone call.

Re-booting your router would have also disabled all debugging, but that
may not have been your issue, it is difficult to say.

Hope this helps someone else go DoH! like I did :)

Regards,
Adam