Jean-Marc Valin wrote:
>>OK, I'm actually about ready to start working on this now.
>>
>>If people in the speex community are interested in working with me on
>>this, I can probably start with the speex buffer, but I imagine
>>there's going to be a lot more work needed to get this where I'd
like
>>it to go.
>>
>>
>
>And where would you like it to go? ;-)
>
>
Heh. I guess after playing with different jitter buffers long enough,
I've realized that there's always situations that you haven't
properly
accounted for when designing one.
>>At the API level, It seems pretty easy to make the speex
>>implementation become speex-independent. Instead of having
>>speex_jitter_get call any particular speex_decode or speex_read_bits,
>>etc functions, it could instead just return the "thing" it
got, and a
>>flag. I.e.
>>
>>
>
>It's not as simple as it may look -- otherwise that's what I would
have
>done. These are some of the things that you can't do easily if you
"just
>return the thing":
>- Allow more than one frame per packet, especially if the frames don't
>end on a byte boundary
>- Let the jitter buffer drop/interpolate frames during silence periods
>- Anything that requires the jitter buffer to know about what is being
>decoded.
>
>
I think the only difficult part here that you do is dealing with
multiple frames per packet, without that information being available to
the jitter buffer. If the jitter buffer can be told when a packet is
added that the packet contains Xms of audio, then the jitter buffer
won't have a problem handling this.
This is something I've encountered in trying to make a particular
asterisk application handle properly IAX2 frames which contain either
20ms of 40ms of speex data. For a CBR case, where the bitrate is known,
this is fairly easy to do, especially if the frames _do_ always end on
byte boundaries. For a VBR case, it is more difficult, because it
doesn't look like there's a way to just parse the speex bitstream and
break it up into the constituent 20ms frames.
The problem isn't so much that the jb can't return the right thing, but
that internally it can't know if it just passed back a packet that
contained 40ms of data or 20ms of data, so later it can't know if it's
lost a frame or not.
The other things can be handled based on the return value of the _get
method: dropping frames, interpolating, etc.
>>We could then have a second-level API wrapper around this for speex
>>which would then call speex_decode, etc, as necessary.
>>
>>Basically, I'd like the jitterbuffer to do all the history, length,
>>etc calculations, but not actually know anything about what it's
>>managing.
>>
>>
>
>I would suggest the opposite. You can just think of the current
>implementation as being callback-based. If you look at the
>implementation of speex_decode, it merely looks for a callback function
>in a struct (mode definition). It would not be very hard to provide
>similar (callback structures) wrappers for other codecs. I'm willing to
>modify the current implementation to make that easier (though it's
>already not very hard).
>
>
I can see how you'd do that, but I don't think that would work for me.
I really don't want the jitterbuffer to handle decoding at all, because
in some cases, I want to dejitter the stream, but not decode it.
For example, I will be running this in front of a conferencing
application. This conferencing application handles participants, each
of which can use a different codec. Often, we "optimize" the path
through the conferencing application by passing the encoded stream
straight-through to listeners when there is only one speaker, and the
speaker and participant use the same codec(*). In this case, I want to
pass back the actual encoded frame, and also the information about what
to do with it, so that I can pass along the frame to some participants,
and decode (and possibly transcode) it for others.
(*) (Yes, I do understand that this violates the expectations of the
codecs, but so far, it seems to work well for GSM, and for speex, since
the changes generally occur surrounded by silence, and there is a huge
benefit in scalability, and also in clarity because there is no
generational loss; the only drawback is that there is _sometimes_ some
distortion when the contexts change).
>>In asterisk and iaxclient (my project), the things I'd pass into the
>>jitterbuffer would be different kinds of structures. Some of these
>>may be audio frames, some might be control frames (DTMF, etc) that we
>>want synchronized with the audio stream, etc. In the future, we'd
>>also want to have video frames thrown in there, which would need to be
>>synchronized.
>>
>>
>
>I'm not sure of the best way to do that. Audio has different constraints
>as video when you're doing jitter buffering. For example, it's much
>easier (in terms of perceptual degradation) to skip frames with video
>than with audio, which means that the algorithm to handle that optimally
>may be quite different. Don't you think?
>
>
Yes. While I don't plan on doing video in the first pass, the API for
this would be that when you pass in your "thing", you also pass along:
a) The timestamp for that "thing"
b) Some flags for the "thing". This might be a "stream
number" for the
thing, or it might just be a flag saying this "thing" is audio, or
this
"thing" is a control frame which must never be dropped, etc, or this
is
a video frame (and maybe a keyframe), etc.
A jitterbuffer for an audio/video system needs to be integrated, of
course, so that the audio and video are synchronized. In my first
implementation, I haven't decided if I want to do more than video, but
presently, in asterisk, all frames go through the jitterbuffer, and it
makes some sense to include, e.g. DTMF frames in there, HANGUP frames,
etc. (imagine leaving a voicemail, if DTMF or HANGUP isn't
jitterbuffered, and your have a _lot_ of jitter, you could lose part of
your message. If DTMF frames don't go through there, they may be
processed out-of-order, etc.).
>>So, I guess my questions (for Jean-Marc mostly, but others as well):
>>
>>1) Is it OK with you to add this extra abstraction layer to your
>>jitter buffer?
>>
>>
>
>I think there might be better ways to abstract the codec out of that
>(callbacks and all).
>
>
callbacks or flags, I think, would work the same way, except for the
above. I think it could work for your use as well, though, if you had
some way to tell the jb out-of-band how long the "packet" you sent
along
was (how many milliseconds).
-SteveK
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.xiph.org/pipermail/speex-dev/attachments/20041116/95e9f96a/attachment.htm