Benjamin, Thanks for the prompt response. Are there other recommended methods to verify encoder implementations? Regards, Chris From: benjamin.m.schwartz at gmail.com [mailto:benjamin.m.schwartz at gmail.com] On Behalf Of Benjamin Schwartz Sent: Sunday, November 10, 2013 1:53 PM To: Wang, Chris Cc: opus at xiph.org Subject: Re: [opus] Questions Regarding Opus Test Vectors On Sun, Nov 10, 2013 at 10:41 AM, Wang, Chris <cwang at sonusnet.com<mailto:cwang at sonusnet.com>> wrote: I downloaded test vectors from opus-codec.org<http://opus-codec.org> site. The site indicated that the test files are intended for verifying that the Opus decoders are operating properly. I assume I could also use them to verify my encoder implementation, right? For example, use testvector01.dec as input to my encoder and compare its output bitstream with the corresponding testvector01.bit bitstream file. No. Opus is not reversible. The encoder does not invert the decoder. Any idea what configuration parameters (bandwidth, bitrate, complexity, stereo mode, dtx, etc) I should use on my encoder. Or a more general question: what encoding parameters were used to generate testvectorxx.bit from testvectorxx.dec? Thank you, Chris _______________________________________________ opus mailing list opus at xiph.org<mailto:opus at xiph.org> http://lists.xiph.org/mailman/listinfo/opus -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.xiph.org/pipermail/opus/attachments/20131110/32a4d455/attachment.htm
On 2013-11-11 7:57 AM, Wang, Chris wrote:> Thanks for the prompt response. Are there other recommended methods to > verify encoder implementations?Human listening tests are the best way to verify encoder quality. -r
Wang, Chris wrote:> Benjamin, > > Thanks for the prompt response. Are there other recommended methods to > verify encoder implementations?There are few if any requirements placed on the encoder by the actual RFC, however there are a number of things you can do which provide various amounts of assurance your encoder is working as expected. The first test I would run is to write out the final range coder state of each packet, as is done in the file format used by opus_demo, and then use opus_demo to decode the files. It will check to make sure the final range coder state of the decoder matches. With high probability this will tell you if the decoder is decoding the same bitstream symbols the encoder thought it encoded. Slightly more complicated is to have the encoder resynthesize the coded audio from its internal state as it encodes, and compare that with what the decoder produces. In libopus this behavior can be enabled via the RESYNTH #define, at least for the CELT layer. http://www.ietf.org/proceedings/80/slides/codec-4.pdf covers a number of additional software tests we run. Determining the quality of a correctly-functioning encoder requires subjective (listening tests) and/or objective metrics (PEAQ, PESQ, etc.).
Timothy , I appreciate your feedback. So It looks like there are no standardized test vectors that I could used to verify whether my encoder implementation is bit-exact with the open source standard version. To verify decoder I could use testvectorxx.bit as input to my decoder and see whether the output would match testvectorxx.dec accordingly. Chris -----Original Message----- From: opus-bounces at xiph.org [mailto:opus-bounces at xiph.org] On Behalf Of Timothy B. Terriberry Sent: Sunday, November 10, 2013 5:33 PM To: opus at xiph.org Subject: Re: [opus] Questions Regarding Opus Test Vectors Wang, Chris wrote:> Benjamin, > > Thanks for the prompt response. Are there other recommended methods > to verify encoder implementations?There are few if any requirements placed on the encoder by the actual RFC, however there are a number of things you can do which provide various amounts of assurance your encoder is working as expected. The first test I would run is to write out the final range coder state of each packet, as is done in the file format used by opus_demo, and then use opus_demo to decode the files. It will check to make sure the final range coder state of the decoder matches. With high probability this will tell you if the decoder is decoding the same bitstream symbols the encoder thought it encoded. Slightly more complicated is to have the encoder resynthesize the coded audio from its internal state as it encodes, and compare that with what the decoder produces. In libopus this behavior can be enabled via the RESYNTH #define, at least for the CELT layer. http://www.ietf.org/proceedings/80/slides/codec-4.pdf covers a number of additional software tests we run. Determining the quality of a correctly-functioning encoder requires subjective (listening tests) and/or objective metrics (PEAQ, PESQ, etc.). _______________________________________________ opus mailing list opus at xiph.org http://lists.xiph.org/mailman/listinfo/opus