flac dev - Jul 2013 - exhaustive-model-search issue results in multi-gigabyte FLAC file

Hi,

On a particular input file, FLAC (testing with current git) greatly 
inflates its output if I encode at level 7, which enables 
--exhaustive-model-search. The source is a 24-bit WAV file of about 
60MB; flac -6 encodes this to a 43MB FLAC file, but flac -7 produces a 
9.1GB (!) file.

The enormous file does seem to be perfectly valid, FWIW -- it 
(eventually) decodes to a WAV that's identical to the source file.

The copyright on the offending file belongs to the band that uploaded it 
to our service, so I can't provide it as a public download, but I can 
supply it to a developer privately.

Command lines and encoder output is as follows:

lsd at cletus:~/projects/flac-bug$ time ./flac/src/flac/flac -f -7 -o 
/data/moredata/flac-bug/7.flac 2303997286.wav

flac 1.3.0, Copyright (C) 2000-2009, 2011-2013  Josh Coalson & Xiph.Org 
Foundation
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
welcome to redistribute it under certain conditions.  Type `flac' for 
details.

2303997286.wav: WARNING: legacy WAVE file has format type 1 but 
bits-per-sample=24
2303997286.wav: WARNING: skipping unknown chunk 'cue ' (use 
--keep-foreign-metadata to keep)
2303997286.wav: wrote 9708061456 bytes, ratio=164.494

real    2m18.059s
user    0m34.836s
sys     0m4.480s

lsd at cletus:~/projects/flac-bug$ time ./flac/src/flac/flac -f -6 -o 
/data/moredata/flac-bug/6.flac 2303997286.wav

flac 1.3.0, Copyright (C) 2000-2009, 2011-2013  Josh Coalson & Xiph.Org 
Foundation
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
welcome to redistribute it under certain conditions.  Type `flac' for 
details.

2303997286.wav: WARNING: legacy WAVE file has format type 1 but 
bits-per-sample=24
2303997286.wav: WARNING: skipping unknown chunk 'cue ' (use 
--keep-foreign-metadata to keep)
2303997286.wav: wrote 44402581 bytes, ratio=0.752

real    0m1.471s
user    0m1.368s
sys     0m0.065s

Thanks
Leigh

On 16-07-13 09:07, Leigh Dyer wrote:
> Hi,
>
> On a particular input file, FLAC (testing with current git) greatly
> inflates its output if I encode at level 7, which enables
> --exhaustive-model-search. The source is a 24-bit WAV file of about
> 60MB; flac -6 encodes this to a 43MB FLAC file, but flac -7 produces a
> 9.1GB (!) file.
That is a very serious bug, as the encoder should always take verbatim 
frames as its baseline. Can you supply the output of flac's analyse 
function (option -a on the command line) to give us some more 
information on what the resulting file looks like?

Leigh Dyer wrote:
> On a particular input file, FLAC (testing with current git) greatly 
> inflates its output if I encode at level 7, which enables 
> --exhaustive-model-search. The source is a 24-bit WAV file of about 
> 60MB; flac -6 encodes this to a 43MB FLAC file, but flac -7 produces a 
> 9.1GB (!) file.
Very interesting and definitely something we'd like to fix.

I'm going to have a try at recreating this problem on a smaller file.

Cheers,
Erik
-- 
----------------------------------------------------------------------
Erik de Castro Lopo
http://www.mega-nerd.com/

On 16/07/13 6:31 PM, Martijn van Beurden wrote:
> On 16-07-13 09:07, Leigh Dyer wrote:
>> Hi,
>>
>> On a particular input file, FLAC (testing with current git) greatly
>> inflates its output if I encode at level 7, which enables
>> --exhaustive-model-search. The source is a 24-bit WAV file of about
>> 60MB; flac -6 encodes this to a 43MB FLAC file, but flac -7 produces a
>> 9.1GB (!) file.
>
> That is a very serious bug, as the encoder should always take verbatim
> frames as its baseline. Can you supply the output of flac's analyse
> function (option -a on the command line) to give us some more
> information on what the resulting file looks like?
Certainly -- I've uploaded the analysis files for both the -6 and -7 
encodes, in case you wanted to compare:

http://wootangent.net/~lsd/blah/6.ana
http://wootangent.net/~lsd/blah/7.ana

The encode seems to proceed normally until 59% of the way through the 
file, but then it takes a couple of minutes to proceed through to 61% of 
the way through -- it's during this period that the file inflates up to 
9GB in size. The last 39% or so of the encode proceeds normally, too.

Thanks
Leigh