asterisk users - Sep 2006 - Garbled (quality probs) IAX2 & SIP calls Asterisk-to-Asterisk

I'm an almost 3 year Asterisk user now, since the pre-0.9 days, and I 
administer two Asterisk boxes.  One of them is a small office with 16 
users mostly using ATA phones hooked into Asterisk via a Rhino 
channelbank and a T1 card and the other is an even smaller office (mine) 
with just a couple of us on a small Asterisk box.

These two locations are connected over a DSL OpenVPN connection. 
Naturally, a couple of years ago I thought to try an IAX2 connection 
between the two locations to save on toll charges, etc., and just to 
experiment with Asterisk's capabilities in this area.

I seem to remember for the first several months or so, things worked 
pretty well.  But then, starting perhaps 1 - 2 years ago, we started 
noticing quality problems from time to time in which one or both sides 
of the conversation would have what I call "garble", which is
basically
what I'm assuming is dropped packets or some other (probably common) 
VOIP problem.  And when it happens, it's normally bad, so bad that the 
one side really can't hear most of the conversation.

Initially, I turned to some QoS types of solutions (attempting to 
implement this on my Linux router box) which didn't really seem to do 
much good and were never fully implemented anyway.

Later, I picked up a couple of Polycom SoundPoints (a 600 and a few 
501s) which I installed on both sides of the VPN.  At some point, I 
realized I could dial between those phones directly (yet over the same 
VPN) using their native SIP protocol.  I later determined that whenever 
we'd experience the quality problem on an Asterisk <-> Asterisk (via 
IAX, or even SIP which I later tried) call, if I immediately switched to 
an SIP <-> SIP call directly between the two phones, there was no 
quality problem.  I've done this enough times to conclude that whatever 
is causing our loss of quality on the Asterisk calls does not affect the 
hardware-to-hardware calls.

I've read a bit about jitterbuffers in the past couple days and some new 
implementation that's available in the 1.2.x branch.  So I started 
playing with those settings in the past couple of days, and this really 
hasn't seemed to solve the problem either.

So for those of you who've made it this far in my narrative (I apologize 
for its length), what are your best guesses as to tests & fixes I could 
continue with given my symptoms?  Especially considering that this is 
evidently a problem that is only affecting Asterisk speaking IAX2 or SIP 
over a VPN connection to another Asterisk box.  SIP-to-SIP calls placed 
directly over the same network do not seem to experience this quality 
loss at all.

I've measured the latency on the VPN with simple ping tests and it 
*normally* is about 50ms, but sometimes spikes up to around 100 - 150ms 
(note, this was not done very rigorously, but sporadically over a few 
days), and from what I understand, that should not cause at least a 
single VOIP conversation to have significant quality problems.

What might I be doing wrong w/ my Asterisk installation(s)?

Also, thanks to Digium and all of the developers, users and community 
that have made Asterisk such a great offering over the past few years.

Thanks,

jl