asterisk users - Jan 2008 - inbound Audio problems probably not NAT related?

Hello all,
Was hoping to get a sanity check along with a question.  Below is the
output from top run with normal defaults, except to show both CPU's, on
a SuSE 10.2 box with Asterisk v1.4.15.

top - 10:00:58 up 3 days, 5:54, 4 users, load average: 0.15, 0.05, 0.01
Tasks: 110 total, 2 running, 108 sleeping,   0 stopped,   0 zombie
Cpu0 : 0.2%us, 0.2%sy, 0.0%ni, 97.3%id, 2.2%wa, 0.1%hi, 0.0%si, 0.0%st
Cpu1 : 0.3%us, 0.0%sy, 0.0%ni, 99.6%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st
Mem:   4052276k total,  2586128k used,  1466148k free,   389208k buffers
Swap:  4200956k total,        0k used,  4200956k free,  1929952k cached

from show channels:(was the same before and after top was run)
12 active channels
6 active calls

Would any of the guru's here say that this was good, bad, middle of the
road, not enough info to tell?  At the time I copied this there were 5
active calls in show channels.

This server is exhibiting some strange behavior and I was starting to
think it may be system overload.  I find this hard to accept given the
specs but, hey I don't know everything!
some info from /proc/cpuinfo:
vendor_id       : AuthenticAMD
cpu family      : 15
model           : 35
model name      : Dual Core AMD Opteron(tm) Processor 180
stepping        : 2
cpu MHz         : 2411.130
cache size      : 1024 KB

some info from /proc/meminfo:
MemTotal:      4052276 kB
MemFree:       1469356 kB
Buffers:        388196 kB
Cached:        1927548 kB
SwapCached:          0 kB
Active:         893644 kB
Inactive:      1523168 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:      4052276 kB
LowFree:       1469356 kB
SwapTotal:     4200956 kB
SwapFree:      4200956 kB
Dirty:             228 kB
Writeback:           0 kB

Hardware RAID 5
on-motherboard gigE
connected through Cisco switch

On inbound calls I lose the incoming audio after a couple minutes,
outbound audio is always good, then after a while inbound audio
magically starts up again. this happens on maybe 10% of calls at its
worst. I have looked at the possibility of NAT issues and do not believe
that to be the case.

I have noticed that the memory usage climbs steadily but I believe that
is the kernel as top show no process with more than 0.4% memory usage.
Although when I rebooted (yes, an act of desperation) over the weekend
the amount of calls with this problem dropped dramatically along with
total memory usage which is slowly climbing again.  Started at about
1gig on Saturday morning and is now at the 2.6gig shown above in top.

This box typically does around 35,000 minutes of calls each month with a
couple "busy" periods each day during weekdays.  Normally no more than
10 to 12 calls at one time.

provider-->T1 to Cisco router-->Asterisk-->phones

The router is doing NAT and routing all traffic from a specific IP to
the asterisk box and dropping everything from any other IP.
canreinvite is set to no on the sip trunk and all the phones.

One thing that may be related is that when I ssh into this box it takes
a full minute respond after the pass phrase is typed in.  Could this be 
related or am I just grasping at straws?

Any Ideas?

--
JohnM


-------------- next part --------------
A non-text attachment was scrubbed...
Name: jmillican.vcf
Type: text/x-vcard
Size: 265 bytes
Desc: not available
Url :
http://lists.digium.com/pipermail/asterisk-users/attachments/20080115/4334bd08/attachment.vcf

On Jan 15, 2008 8:12 PM, John Millican
<jmillican at sentinelcommunications.com> wrote:
> Hello all,
> Was hoping to get a sanity check along with a question.  Below is the
> output from top run with normal defaults, except to show both CPU's, on
> a SuSE 10.2 box with Asterisk v1.4.15.
>
[snip massive hardware spec]

We have locations which run 120 simultaneous PRI calls on less than
half of the specification you gave :) I don't think that 6-12 calls
will overload it!
> On inbound calls I lose the incoming audio after a couple minutes,
> outbound audio is always good, then after a while inbound audio
> magically starts up again. this happens on maybe 10% of calls at its
> worst. I have looked at the possibility of NAT issues and do not believe
> that to be the case.
First place to look here is for a duplicate IP address on the network.
An arp cache timeout (usually about 10 minutes) can caused the voice
packets to start going to the wrong place temporarily.
> I have noticed that the memory usage climbs steadily but I believe that
> is the kernel as top show no process with more than 0.4% memory usage.
> Although when I rebooted (yes, an act of desperation) over the weekend
> the amount of calls with this problem dropped dramatically along with
> total memory usage which is slowly climbing again.  Started at about
> 1gig on Saturday morning and is now at the 2.6gig shown above in top.
LOL. If I am reading it correctly, 2.4Gb of that is cache and buffers,
and therefore "does not count". This is an example of a Linux kernel
using memory effectively to improve performance.
> This box typically does around 35,000 minutes of calls each month with a
> couple "busy" periods each day during weekdays.  Normally no more
than
> 10 to 12 calls at one time.
>
> provider-->T1 to Cisco router-->Asterisk-->phones
If you are using a Cisco switch, check that all of the silly Cisco
trunking modes are disabled on the port(s) used by Asterisk and the
phones, and ensure that fast-start is enabled for those ports too.
IMHO Cisco nearly always set the defaults for these features back to
front!

Check the keepalive period on the firewall for NAT sessions, and
perhaps disabling silence supression if it is enabled will help to
keep the NAT connection open.
> The router is doing NAT and routing all traffic from a specific IP to
> the asterisk box and dropping everything from any other IP.
> canreinvite is set to no on the sip trunk and all the phones.
>
> One thing that may be related is that when I ssh into this box it takes
> a full minute respond after the pass phrase is typed in.  Could this be
> related or am I just grasping at straws?
This usually means that you have not configured DNS correctly either
at the server end, or the DNS records for the client end are somehow
"lacking"
>
> Any Ideas?
That is a quick random braindump. Perhaps some of it will be useful :)

Steve