asterisk users - Sep 2008 - Possible Packet loss but need an opinion

I have an Asterisk box at a clients site, and they are having quality
issues, they are using g711 for the codec and I think it might be that
asterisk is just overloaded.

Upstream I did a packet capture. The capture showed traffic going to
the asterisk box from the ISP is clean, but traffic from the asterisk
box to the ISP has packet loss. So we moved farther and farther down
the chain until we got to the box itself. We then did a capture on the
asterisk box directly using tcpdump. We found that it showed the same
results. Packetloss leaving the box, no packet loss coming into the
box.

What would this suggest? At most we've had 150 concurrent streams, We
are running the Business version.

Anyone have any ideas?

Thanks

Andrew