asterisk users - Apr 2008 - One-way audio after music on hold

I'm running Connected to Asterisk 1.4.18-1.
We have a T1 PRI and a SIP trunk coming in from our IVR.
When people call in on the SIP trunk, I experience the following
problem:
I put the caller on hold, MOH starts.  MOH quits after approx ten
seconds.  When I pick the call back up off of hold, they can't hear me
for about 20 seconds (sometimes longer or not at all) but I can hear
them.

This doesn't happen with calls on the T1.

Has anyone seen this before?  Any solutions out there?

Thanks,
Ben W