flac dev - Sep 2004 - ERROR: mismatch in decoded data, verify FAILED!

On Sun, Jun 24, 2001 at 02:30:56PM -0700, Josh Coalson wrote:
> > This error appeared in the other report as well.  It looks like a
memory
> > allocation failure is the cause of the problem. 
> 
> There have been reports of -9 using huge amounts of memory.  -9 is really
> theoretical, but people always seem to want to try the max setting. 
Anyway,
> that's not an excuse but figuring out why -9 is using so much memory is
lower
> on my list than other stuff.  -8 should get within 0.01% of -9 and is
pretty
> reliable.
> 
> If you're getting this error without -9, then there is a more serious
> problem.  If flac 0.10 works on openbsd then my hunch is that there is an
> arithmetic overflow in one of the assembly routines in the decoder, since I
> don't think the assembly stuff got compiled into the openbsd version in
0.10.
> If you are working with the source, you could try building two flacs on
> linux, one normally and one configured with --disable-asm-optimizations and
> see if the behavior is different.
I just encoded several CDs worth of data with the default flags, and was unable
to reproduce this error (Debian/i386).  I hadn't noticed that -9 was being
used.  I'll play around with -9 and see if I can find anything.

-- 
 - mdz

On Sun, Jun 24, 2001 at 02:30:56PM -0700, Josh Coalson wrote:
> There have been reports of -9 using huge amounts of memory.  -9 is really
> theoretical, but people always seem to want to try the max setting. 
Anyway,
> that's not an excuse but figuring out why -9 is using so much memory is
lower
> on my list than other stuff.  -8 should get within 0.01% of -9 and is
pretty
> reliable.
I am able to reliably reproduce the -9 memory problem with one of my WAV files.
I split it in two repeatedly, doing a binary search to find the smallest chunk
that would trigger the problem, and I got it down to 28224 samples (this is a
44.1/16bit/stereo file, so about 0.5 seconds).  I have put the file up at
<http://people.debian.org/~mdz/flac-test-case-1.wav>.  Here is a gdb
backtrace
at the point where it is allocating gobs of memory:

(gdb) bt
#0  0x400ba565 in memset () from /lib/libc.so.6
#1  0x805f782 in bitbuffer_resize_ (bb=0x4085c800, new_capacity=134479872)
    at bitbuffer.c:62
#2  0x805f8ec in bitbuffer_grow_ (bb=0x4085c800, min_bytes_to_add=33554433)
    at bitbuffer.c:86
#3  0x805f976 in bitbuffer_ensure_size_ (bb=0x4085c800, bits_to_add=268435453)
    at bitbuffer.c:94
#4  0x8060226 in FLAC__bitbuffer_write_zeroes (bb=0x4085c800, bits=268435453)
    at bitbuffer.c:254
#5  0x8061153 in FLAC__bitbuffer_write_rice_signed (bb=0x4085c800, 
    val=-2147483630, parameter=4) at bitbuffer.c:658
#6  0x8056603 in subframe_add_residual_partitioned_rice_ (bb=0x4085c800, 
    residual=0x8361200, residual_samples=4580, predictor_order=28, 
    rice_parameters=0x407dc520, raw_bits=0x407fc520, partition_order=0)
    at encoder_framing.c:366
#7  0x80563eb in FLAC__subframe_add_lpc (subframe=0x407dc518, 
    residual_samples=4580, subframe_bps=17, wasted_bits=0, bb=0x4085c800)
    at encoder_framing.c:310
#8  0x80544c8 in encoder_add_subframe_ (encoder=0x8314758, 
    frame_header=0xbfffe5ac, subframe_bps=17, subframe=0x407dc514, 
    frame=0x4085c800) at encoder.c:1030
#9  0x8053b48 in encoder_process_subframes_ (encoder=0x8314758, 
    is_last_frame=0) at encoder.c:863
#10 0x8052f50 in encoder_process_frame_ (encoder=0x8314758, is_last_frame=0)
    at encoder.c:632
#11 0x8052a46 in FLAC__encoder_process (encoder=0x8314758, buf=0x83080e0, 
    samples=2048) at encoder.c:536
#12 0x804bb31 in flac__encode_wav (infile=0x8314478, 
    infilename=0xbffffa09 "/tmp/test-9.wav", 
    outfilename=0xbfffe7d8 "/tmp/test-9.flac", verbose=1, skip=0,
verify=1,
    lax=0, do_mid_side=1, loose_mid_side=0, do_exhaustive_model_search=1, 
    do_qlp_coeff_prec_search=1, min_residual_partition_order=0, 
    max_residual_partition_order=16, rice_parameter_search_dist=0, 
    max_lpc_order=32, blocksize=4608, qlp_coeff_precision=0, padding=0, 
    requested_seek_points=0x8308120 "100x",
num_requested_seek_points=-1)
    at encode.c:292
#13 0x8050aa4 in encode_file (infilename=0xbffffa09 "/tmp/test-9.wav",
    forced_outfilename=0xbfffe7d8 "/tmp/test-9.flac") at main.c:581
#14 0x804fd34 in main (argc=4, argv=0xbffff884) at main.c:377
#15 0x4006038b in __libc_start_main () from /lib/libc.so.6

Also, Kai has been kind enough to send me a copy of his file which has a
problem only on -8, which I'll be looking into soon.

-- 
 - mdz

> > There have been reports of -9 using huge amounts of memory.  -9 is
> > really
> > theoretical, but people always seem to want to try the max setting.
> >  Anyway,
> > that's not an excuse but figuring out why -9 is using so much
> > memory is lower
> > on my list than other stuff.  -8 should get within 0.01% of -9 and
> > is pretty
> > reliable.
> 
> I am able to reliably reproduce the -9 memory problem with one of my
> WAV files.
> I split it in two repeatedly, doing a binary search to find the
> smallest chunk
> that would trigger the problem, and I got it down to 28224 samples
> (this is a
> 44.1/16bit/stereo file, so about 0.5 seconds).  I have put the file
> up at
> <http://people.debian.org/~mdz/flac-test-case-1.wav>.  Here is a gdb
> backtrace
> at the point where it is allocating gobs of memory:
I think I see the problem just from the trace but I will
grab the file and check it out.
> Also, Kai has been kind enough to send me a copy of his file which
> has a
> problem only on -8, which I'll be looking into soon.
Matt, if you whittle this file down can you make it
available also?

One more thing I forgot to mention about debugging.  The
first point of suspicion is the assembly in the decoder.
That you can try with 'configure --disable-asm-optimizations'
The second is arithmetic overflow in the linear predictor.
You can test that by 'configure --enable-debug' which does
overflow checking.

Josh


__________________________________________________
Do You Yahoo!?
Get personalized email addresses from Yahoo! Mail
http://personal.mail.yahoo.com/

> Also, Kai has been kind enough to send me a copy of his file which
> has a
> problem only on -8, which I'll be looking into soon.
After some intense debugging, I found the problem.  One
block in the file triggered a very rare bug in the LPC
coefficient quantizer caused by insufficient floating
point precision.  There is a snippet to compute the
log(base 2) of a number:

	floor(log(cmax) / M_LN2)

It turns out on at least some x86's compiling with gcc-2.91
(at least) the result of floor(log(8.0) / M_LN2) is 2.0 not
3.0.  Anyway, I made a fix and also added code to detect
and handle similar problems (related to FP accuracy).

Josh


__________________________________________________
Do You Yahoo!?
Get personalized email addresses from Yahoo! Mail
http://personal.mail.yahoo.com/

> > There have been reports of -9 using huge amounts of memory.  -9 is
> > really
> > theoretical, but people always seem to want to try the max setting.
> >  Anyway,
> > that's not an excuse but figuring out why -9 is using so much
> > memory is lower
> > on my list than other stuff.  -8 should get within 0.01% of -9 and
> > is pretty
> > reliable.
> 
> I am able to reliably reproduce the -9 memory problem with one of my
> WAV files.
> I split it in two repeatedly, doing a binary search to find the
> smallest chunk
> that would trigger the problem, and I got it down to 28224 samples
> (this is a
> 44.1/16bit/stereo file, so about 0.5 seconds).  I have put the file
> up at
> <http://people.debian.org/~mdz/flac-test-case-1.wav>.
OK, I have fixed this bug too and checked it in.  Again
it was in the LPC coeff quantizer in a wierd corner case.

Josh


__________________________________________________
Do You Yahoo!?
Get personalized email addresses from Yahoo! Mail
http://personal.mail.yahoo.com/

> > I also had this verify error encoding a wav I ripped from a CD. I
> didn't
> > report this as it happended on flac running on debian linux
> -current unstable.
> > This error happened only with one track of a CD I was ripping.
> > 
> > Another reason for me not reporting this was, that flac 1.0 running
> on OpenBSD
> > 2.9 encoded the wav with no error.. maybe this is linux-specific.
> > 
> > Now this is how it looked..
> > 
> > --snip--
> > 
> > [...]
> > ERROR: mismatch in decoded data, verify FAILED!
> > [...]
> > ERROR during encoding, state >
15:FLAC__ENCODER_MEMORY_ALLOCATION_ERROR
> 
> This error appeared in the other report as well.  It looks like a
> memory
> allocation failure is the cause of the problem. 
There have been reports of -9 using huge amounts of memory.
-9 is really theoretical, but people always seem to want to
try the max setting.  Anyway, that's not an excuse but
figuring out why -9 is using so much memory is lower on my
list than other stuff.  -8 should get within 0.01% of -9
and is pretty reliable.

If you're getting this error without -9, then there is
a more serious problem.  If flac 0.10 works on openbsd
then my hunch is that there is an arithmetic overflow
in one of the assembly routines in the decoder, since
I don't think the assembly stuff got compiled into the
openbsd version in 0.10.  If you are working with the
source, you could try building two flacs on linux, one
normally and one configured with
--disable-asm-optimizations and see if the behavior is
different.

But a sample for download of a track that fails would
really help.

Josh


__________________________________________________
Do You Yahoo!?
Get personalized email addresses from Yahoo! Mail
http://personal.mail.yahoo.com/