flac dev - Nov 2015 - [PATCH] stream_encoder.c: choose proper *fixed_compute_best_predictor* function

I found that 32-bit total_error_* variables in fixed_compute_best_predictor()
functions can overflow. For example it happens with the attached test.wav and
the following options: "-l 0 -b 4200 -m -r 5".

The attached patch should fix it.

(Now all three use_wide_by_* variables are unused, so I commented them out
with #ifdef 0).
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test.wav.zip
Type: application/zip
Size: 1214 bytes
Desc: not available
Url :
http://lists.xiph.org/pipermail/flac-dev/attachments/20151123/516d8a2b/attachment.zip
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fixed_wide.patch
Type: application/octet-stream
Size: 3407 bytes
Desc: not available
Url :
http://lists.xiph.org/pipermail/flac-dev/attachments/20151123/516d8a2b/attachment.obj

lvqcl wrote:
> I found that 32-bit total_error_* variables in
fixed_compute_best_predictor()
> functions can overflow. For example it happens with the attached test.wav
and
> the following options: "-l 0 -b 4200 -m -r 5".
> 
> The attached patch should fix it.
> 
> (Now all three use_wide_by_* variables are unused, so I commented them out
> with #ifdef 0).
Rather than '#ifdef 0'-ing them out, shouldn't we remove them
instead?

Erik
-- 
----------------------------------------------------------------------
Erik de Castro Lopo
http://www.mega-nerd.com/

Erik de Castro Lopo wrote:
> Rather than '#ifdef 0'-ing them out, shouldn't we remove them
instead?
Makes sense.


'use_slow' boolean variable was introduced in 30 Mar 2001:

<http://git.xiph.org/?p=flac.git;a=commitdiff;h=eef5670adc892b8fe687dbc0551e16ad06626660>

then it was refactored into use_wide_by_block / use_wide_by_partition /
use_wide_by_order:

<http://git.xiph.org/?p=flac.git;a=commitdiff;h=8395d02f73c8448e5a74a17b97c83136e7d85015>

and it seems that the last two variables were never used.

Erik de Castro Lopo wrote:
> Rather than '#ifdef 0'-ing them out, shouldn't we remove them
instead?
Patch ver.2 attached.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fixed_wide_v2.patch
Type: application/octet-stream
Size: 3303 bytes
Desc: not available
Url :
http://lists.xiph.org/pipermail/flac-dev/attachments/20151124/0f18050e/attachment.obj