Martin Dufour
2009-Jan-26 16:34 UTC
[Vorbis-dev] float->int rounding problem in setup_tone_curves()
Hi, We have severe audio quality issues when we compile Vorbis with Microsoft VC 2005, while it works fine with VC 2003. I have isolated the problem to a rounding issue in setup_tone_curves() ; the "lo_curve" and "hi_curve" are different depending on the compiler. For example, here are the values for the last band (#17) at 32 kHz prior to { ceil(), floor() }: VC2003: lo_fcurve=15.997361 hi_fcurve=16.000000 VC2005: lo_fcurve:15.997360 hi_fcurve:15.999999 See how the floor() for hi_curve is problematic, as it will resolve to 15 or 16 depending on the compiler. This makes me think that perhaps a round-to-nearest would be more appropriate (and indeed, seems to make sense for other bands as well). Why is hi_curve floored? Thanks, Martin Dufour Audiokinetic