asterisk users - May 2003 - SIP phone audio quality in conference bridge

Hello,

I have an asterisk box (Debian, asterisk cvs, Duron 600mhz) and a couple
of SIP phones connecting to it.  Audio quality between the two SIP phones
is fine, being native bridged ulaw, but when we both call into a bridge
on the asterisk box the audio is cutting in and out and is generally unusable.

My SIP phone is local to the box, the other is ~130-170ms away over an IPSEC
vpn.

Is there anything obvious I should look at?  I have a fairly basic config
at this point.  I swapped out an older Fast Ethernet card for an Intel
in case it was an interrupt or driver issue.

I have a X100P also in the box and a Quicknet Internet PhoneJack.  I can
test with those removed if it is likely to be related.

Anyway, I'm just looking for some ideas on what might cause this, or whether
it is expected, etc.
Thanks.

-Andrew