flac dev - Jul 2013 - exhaustive-model-search issue results in multi-gigabyte FLAC file

On 16/07/13 8:10 PM, Erik de Castro Lopo wrote:
> Leigh Dyer wrote:
>
>> Certainly -- I've uploaded the analysis files for both the -6 and
-7
>> encodes, in case you wanted to compare:
>>
>> http://wootangent.net/~lsd/blah/6.ana
>> http://wootangent.net/~lsd/blah/7.ana
>>
>> The encode seems to proceed normally until 59% of the way through the
>> file, but then it takes a couple of minutes to proceed through to 61%
of
>> the way through -- it's during this period that the file inflates
up to
>> 9GB in size. The last 39% or so of the encode proceeds normally, too.
>
> What happens if you top and tail the file from say 57% to 64% of
> the file and try an encode that?
The first time I tried, the resulting file encoded just fine, but after 
trying a few more times, cutting at slightly different points, I was 
able to get a snippet of that section that exhibits the problem. This is 
short enough that I think I can link to it here:

http://wootangent.net/~lsd/blah/snippet6.wav

This results in a 4.1GB file when encoded with -7.

Thanks
Leigh

Leigh Dyer wrote:
> The first time I tried, the resulting file encoded just fine, but after 
> trying a few more times, cutting at slightly different points, I was 
> able to get a snippet of that section that exhibits the problem. This is 
> short enough that I think I can link to it here:
> 
> http://wootangent.net/~lsd/blah/snippet6.wav
Great, thanks! Confirmed the problem here. Will look at it ASAP.

Cheers,
Erik
-- 
----------------------------------------------------------------------
Erik de Castro Lopo
http://www.mega-nerd.com/

Erik de Castro Lopo wrote:
> > http://wootangent.net/~lsd/blah/snippet6.wav
> 
> Great, thanks! Confirmed the problem here. Will look at it ASAP.
Same problem with flac 1.2.1. Interesting!

Erik
-- 
----------------------------------------------------------------------
Erik de Castro Lopo
http://www.mega-nerd.com/

On 16-07-13 12:32, Leigh Dyer wrote:
> The first time I tried, the resulting file encoded just fine, but after
> trying a few more times, cutting at slightly different points, I was
> able to get a snippet of that section that exhibits the problem. This is
> short enough that I think I can link to it here:
>
> http://wootangent.net/~lsd/blah/snippet6.wav
>
> This results in a 4.1GB file when encoded with -7.
Have you notices that there are a few damaged areas in this file? For 
example at 4.854 seconds, 6.700s, 6.828s, 7.618s. It looks to me like 
damage accumulated during some process after mastering, somewhere in the 
process of transferring it as a WAVE-file. You might want to check this 
with the band that uploaded it to you.

In any case, it looks like these damaged areas triggered some unwanted 
FLAC behaviour. You've exposed at least two very serious FLAC bugs, 
namely a malfunctioning RICE2-partition encoder and a bug concerning 
choosing verbatim frames over fixed/lpc frames.

One last remark for anyone who didn't notice yet, this >4GB flac file 
consists mostly of zeros, zipping it reduces the file to 6.8MB. Looking 
at it with a hex-editor reveals this has to be an error with the RICE2 
partition. When this snippet is encoded with FLAC 1.3.0 with everything 
left default except setting the compression level to 7, the problem is 
in frame 52 and frame 73

Martijn van Beurden wrote:
> You've exposed at least two very serious FLAC bugs, 
> namely a malfunctioning RICE2-partition encoder and a bug concerning 
> choosing verbatim frames over fixed/lpc frames.
The second, not choosing verbatim frames over fixed/lpc frames is almost
certainly a direct result of the first problem.

The fix was changing one local variable from FLAC_uint32 to FLAC_uint64
in function precompute_partition_info_sums_().

   
https://git.xiph.org/?p=flac.git;a=commit;h=6f7ec60c7e7f05f5ab0b1cf6b7b0945e44afcd4b

Erik
-- 
----------------------------------------------------------------------
Erik de Castro Lopo
http://www.mega-nerd.com/