We've made a little change that changed the output of vorbis is there a way to check the accuracy of the output against the original output? I also asked this question in the irc ... Thanks, Tal.
On Sun, 23 Jan 2005 08:51:08 +0200, Tal <shabi@t2.technion.ac.il> wrote:> > We've made a little change that changed the output of vorbis > is there a way to check the accuracy of the output against the original > output? > > I also asked this question in the irc ...It isn't really clear what you're asking here, perhaps if you gave more details we'd be able to help? What is it that you've changed? The vorbis reference encoder? The reference decoder (or the other xiph.org decoder, Tremor)? If you've changed the encoder, then you'd have to give a more precise definition of "accuracy", because there are a number of things that might refer to (how close your encoder is to the reference one, how close the result is to the original input, how good the result sounds, etc.) If you've changed the decoder, then... well, it shouldn't be changing the output at all (unless you're deliberately producing a low-accuracy decoder for some reason?). Mike
Hi, Thanks for the quick response, We did some optimizations that deliberately cut some corners. Is there a more accurate test to see if the the output is not hurt? Tal. P.S. I mailed it directly to you by mistake :( Thanks for catching it. -----Original Message----- From: Michael Smith [mailto:mlrsmith@gmail.com] Sent: Sunday, January 23, 2005 10:46 AM To: Tal Subject: Re: [Vorbis-dev] Checking accuracy of the output On Sun, 23 Jan 2005 10:11:42 +0200, Tal <shabi@t2.technion.ac.il> wrote:> Hi, > We've changed the encoder. > We've made all sorts of optimizations. > How can we check that issues that you've brought up? "(how close your > encoder is to the reference one, how close the result is to the > original input, how good the result sounds, etc.)"Well, if what you've done is 'just' optimisation, then the encoder output should be identical. In that case, you only need to check that the produced bitstreams are identical (note that most encoders (such as oggenc) use a random serial number - you'll need to set that to the same number for both encodes, oggenc has an option for doing that). If you've made changes that produce (deliberately) different output, then it's much, much more complex. You really need to do double-blind listening tests to see whether your improvements are, in fact, improvements. Mike p.s. Did you mean to keep this on the mailing list? You mailed me directly...
nagaraja.sreerama@wipro.com
2005-Jan-28 03:39 UTC
[Vorbis-dev] Checking accuracy of the output
Hi, I agree with you that it would be better to run the compliance tests similar to MP3. I also agree with you that given the lack of any reference test vectors, we may have to depend on the Vorbis reference Encoder and Vorbis reference Decoder (both floating point implementations) as the reference for accuracy check of given decoder using the above tests. But did anyone tried this test on the tremor code ? Tremor code being a Fixed point code, might have the loss of some bits compared to the original implementation in floating point arithmetic. I did a random check (of course not with any reference sine wave or so, but with the input file as some audio song) and the result was shocking when I tried the above test. The procedure was like this: I converted the song to a vorbis file using the reference vorbis encoder. I passed the same file to (a) reference vorbis decoder and (b) tremor code and got the outputs. Then tried to find the RMS difference as per the procedure above (the same as the MP3 test suits). Surprising to me, I got only a SNR of about 3 bits !! Probably something is wrong in getting the results ?? Another observation is that through listening test I could not make out the difference in quality (at least not much, though I'm not a serious listner). Can anybody check up this point w.r.t. tremor code accuracy ? Best regards, Nags.. -----Original Message----- From: vorbis-dev-bounces@xiph.org [mailto:vorbis-dev-bounces@xiph.org] On Behalf Of John Ripley Sent: Sunday, January 23, 2005 8:57 PM To: Tal Cc: vorbis-dev@xiph.org Subject: Re: [Vorbis-dev] Checking accuracy of the output Michael Smith wrote: >>If you've made changes that produce (deliberately) different output,>>then it's much, much more complex. You really need to do double-blind >>listening tests to see whether your improvements are, in fact, >>improvements.Tal wrote:> Hi, > Thanks for the quick response, > We did some optimizations that deliberately cut some corners. > Is there a more accurate test to see if the the output is not hurt?I'd run a set of tests similar to the MP3 compliance tests used for MAD: http://www.underbit.com/resources/mpeg/audio/compliance/ Take the difference signal of your output and the "ideal" output, then calculate RMS and peak. A subtle thing here is that the ISO MP3 test streams are not created by an encoder - the coefficients are directly generated. You'll have to compare the accuracy of your decoder against the accuracy of the reference Vorbis encoder + decoder combination. Personally, I think the "fully compliant" criteria aren't nasty enough. I would only accept an error due to truncation (+/-1 max). On the other hand, I agree with the "limited accuracy" criteria: you can get away with about 12 bit SNR without double blind tests picking up on the difference. Just don't say that within earshot of a so called "audiophile" :) (It's also a nice sanity check for work-in-progress code. At one point while tweaking the MDCT, I'd accidentally reduced the quality of my decoder to 8 bit SNR and removed everything from 11kHz to 22kHz. It took me 2 weeks to notice...) Are you reducing quality in order to save CPU/battery, or memory usage? - John Ripley. _______________________________________________ Vorbis-dev mailing list Vorbis-dev@xiph.org http://lists.xiph.org/mailman/listinfo/vorbis-dev Confidentiality Notice The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain confidential or privileged information. If you are not the intended recipient, please notify the sender at Wipro or Mailadmin@wipro.com immediately and destroy all copies of this message and any attachments.
nagaraja.sreerama@wipro.com
2005-Feb-01 04:05 UTC
[Vorbis-dev] Checking accuracy of the output
Further to the tests, this time I used a sine sweep at 48000Hz and
encoded using the oggenc1.0.1, to get a ogg bitstream (attached). The
same ogg bitstream is passed to both oggdec(v1.0.1) and tremor code.
When compared the RMS value, the result is as follows:
File name 1: test2_sine_sweep_48k_tremor.pcm (16-bit output)---> tremor
File name 2: test2_sine_sweep_ref.pcm (16-bit output) ---> oggdec 1.0.1
Total number of samples = 1440037
RMS error = 2.132247e-005
Allowed: Full Accuracy (16-bit SNR) = 8.81e-006
Limited accuracy (12-bit SNR) = 1.409e-004
SNR = 14.724785
Peak error = 3.051758e-005 (allowed[2^-14]=6.1035e-005)
Peak error = 256 levels (out of 24-bit)
Please note that according to the tests the 16-bits output should be
converted to 24-bit by forcing the last 8-bits to 0. The same is done
for both the reference output and the tremor output. Now its clear that
the tremor code is not fully compliant but with limited accuracy.
The same test is performed with two other .ogg songs. While one of them
gave almost similar results, the other song yielded drastic changes like
rms error of around 2.5e-1 etc.
Can anyone please comment on this ?
Best regards,
Nags..
-----Original Message-----
From: vorbis-dev-bounces@xiph.org [mailto:vorbis-dev-bounces@xiph.org]
On Behalf Of nagaraja.sreerama@wipro.com
Sent: Friday, January 28, 2005 5:09 PM
Hi,
But did anyone tried this test on the tremor code ?
I did a random check (of course not with any reference sine wave or so,
but with the input file as some audio song) and the result was shocking
when I tried the above test.
Can anybody check up this point w.r.t. tremor code accuracy ?
Best regards,
Nags..
-----Original Message-----
From: vorbis-dev-bounces@xiph.org [mailto:vorbis-dev-bounces@xiph.org]
On Behalf Of John Ripley
Sent: Sunday, January 23, 2005 8:57 PM
To: Tal
Cc: vorbis-dev@xiph.org
Subject: Re: [Vorbis-dev] Checking accuracy of the output
I'd run a set of tests similar to the MP3 compliance tests used for MAD:
http://www.underbit.com/resources/mpeg/audio/compliance/
Take the difference signal of your output and the "ideal" output, then
calculate RMS and peak. A subtle thing here is that the ISO MP3 test
streams are not created by an encoder - the coefficients are directly
generated. You'll have to compare the accuracy of your decoder against
the accuracy of the reference Vorbis encoder + decoder combination.
Personally, I think the "fully compliant" criteria aren't nasty
enough.
I would only accept an error due to truncation (+/-1 max). On the other
hand, I agree with the "limited accuracy" criteria: you can get away
with about 12 bit SNR without double blind tests picking up on the
difference. Just don't say that within earshot of a so called
"audiophile" :)
- John Ripley.
Confidentiality Notice
The information contained in this electronic message and any attachments to this
message are intended
for the exclusive use of the addressee(s) and may contain confidential or
privileged information. If
you are not the intended recipient, please notify the sender at Wipro or
Mailadmin@wipro.com immediately
and destroy all copies of this message and any attachments.
nagaraja.sreerama@wipro.com
2005-Feb-09 16:56 UTC
[Vorbis-dev] Checking accuracy of the output
Further to the tests, this time I used a sine sweep at 48000Hz and
encoded using the oggenc1.0.1, to get a ogg bitstream (attached).
The same ogg bitstream is passed to both oggdec(v1.0.1) and tremor code.
When compared the RMS value, the result is as follows:
File name 1: test2_sine_sweep_48k_tremor.pcm (16-bit output)---> tremor
File name 2: test2_sine_sweep_ref.pcm (16-bit output) ---> oggdec 1.0.1
Total number of samples = 1440037
RMS error = 2.132247e-005
Allowed: Full Accuracy (16-bit SNR) = 8.81e-006
Limited accuracy (12-bit SNR) = 1.409e-004
SNR = 14.724785
Peak error = 3.051758e-005 (allowed[2^-14]=6.1035e-005)
Peak error = 256 levels (out of 24-bit)
Please note that according to the tests the 16-bits output should be
converted to 24-bit by forcing the last 8-bits to 0. The same is done
for both the reference output and the tremor output. Now its clear that
the tremor code is not fully compliant but with limited accuracy.
The same test is performed with two other .ogg songs. While one of them
gave almost similar results, the other song yielded drastic changes like
rms error of around 2.5e-1 etc.
Can anyone please comment on this ?
Best regards,
Nags..
-----Original Message-----
From: vorbis-dev-bounces@xiph.org [mailto:vorbis-dev-bounces@xiph.org]
On Behalf Of nagaraja.sreerama@wipro.com
Sent: Friday, January 28, 2005 5:09 PM
To: jripley@rioaudio.com; shabi@t2.technion.ac.il
Cc: vorbis-dev@xiph.org
Subject: RE: [Vorbis-dev] Checking accuracy of the output
Hi,
I agree with you that it would be better to run the compliance tests
similar to MP3. I also agree with you that given the lack of any
reference test vectors, we may have to depend on the Vorbis reference
Encoder and Vorbis reference Decoder (both floating point
implementations) as the reference for accuracy check of given decoder
using the above tests. But did anyone tried this test on the tremor code
?
Tremor code being a Fixed point code, might have the loss of some bits
compared to the original implementation in floating point arithmetic. I
did a random check (of course not with any reference sine wave or so,
but with the input file as some audio song) and the result was shocking
when I tried the above test.
The procedure was like this:
I converted the song to a vorbis file using the reference vorbis
encoder. I passed the same file to (a) reference vorbis decoder and (b)
tremor code and got the outputs. Then tried to find the RMS difference
as per the procedure above (the same as the MP3 test suits).
Surprising to me, I got only a SNR of about 3 bits !! Probably something
is wrong in getting the results ?? Another observation is that through
listening test I could not make out the difference in quality (at least
not much, though I'm not a serious listner). Can anybody check up this
point w.r.t. tremor code accuracy ?
Best regards,
Nags..
-----Original Message-----
From: vorbis-dev-bounces@xiph.org [mailto:vorbis-dev-bounces@xiph.org]
On Behalf Of John Ripley
Sent: Sunday, January 23, 2005 8:57 PM
To: Tal
Cc: vorbis-dev@xiph.org
Subject: Re: [Vorbis-dev] Checking accuracy of the output
Michael Smith wrote:
>>If you've made changes that produce (deliberately) different
output,>>then it's much, much more complex. You really need to do
double-blind
>>listening tests to see whether your improvements are, in fact,
>>improvements.
Tal wrote:> Hi,
> Thanks for the quick response,
> We did some optimizations that deliberately cut some corners. Is there
> a more accurate test to see if the the output is not hurt?
I'd run a set of tests similar to the MP3 compliance tests used for MAD:
http://www.underbit.com/resources/mpeg/audio/compliance/
Take the difference signal of your output and the "ideal" output, then
calculate RMS and peak. A subtle thing here is that the ISO MP3 test
streams are not created by an encoder - the coefficients are directly
generated. You'll have to compare the accuracy of your decoder against
the accuracy of the reference Vorbis encoder + decoder combination.
Personally, I think the "fully compliant" criteria aren't nasty
enough.
I would only accept an error due to truncation (+/-1 max). On the other
hand, I agree with the "limited accuracy" criteria: you can get away
with about 12 bit SNR without double blind tests picking up on the
difference. Just don't say that within earshot of a so called
"audiophile" :)
(It's also a nice sanity check for work-in-progress code. At one point
while tweaking the MDCT, I'd accidentally reduced the quality of my
decoder to 8 bit SNR and removed everything from 11kHz to 22kHz. It took
me 2 weeks to notice...)
Are you reducing quality in order to save CPU/battery, or memory usage?
- John Ripley.
_______________________________________________
Vorbis-dev mailing list
Vorbis-dev@xiph.org http://lists.xiph.org/mailman/listinfo/vorbis-dev
Confidentiality Notice
The information contained in this electronic message and any attachments
to this message are intended for the exclusive use of the addressee(s)
and may contain confidential or privileged information. If you are not
the intended recipient, please notify the sender at Wipro or
Mailadmin@wipro.com immediately and destroy all copies of this message
and any attachments. _______________________________________________
Vorbis-dev mailing list
Vorbis-dev@xiph.org http://lists.xiph.org/mailman/listinfo/vorbis-dev
Confidentiality Notice
The information contained in this electronic message and any attachments to this
message are intended
for the exclusive use of the addressee(s) and may contain confidential or
privileged information. If
you are not the intended recipient, please notify the sender at Wipro or
Mailadmin@wipro.com immediately
and destroy all copies of this message and any attachments.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test2_sine_sweep_48k.ogg
Type: application/octet-stream
Size: 66780 bytes
Desc: test2_sine_sweep_48k.ogg
Url :
http://lists.xiph.org/pipermail/vorbis-dev/attachments/20050201/d8283a61/test2_sine_sweep_48k-0001.obj