flac dev - Feb 2010 - Decoded files identical, encoded files different (linux vs win encoder)

Thanks for helping Erik,
I have uploaded a small file example for you to check.
I am curious about what is exactly producing this mismatch filesize!
I know it's ridiculous and probably too paranoid, but knowledge is always
welcome.


-rw-r--r-- 1 djzn djzn 341418 2010-02-11 20:34 bandstand-linux.flac
-rw-r--r-- 1 djzn djzn 556308 2010-02-11 20:34 bandstand.wav
-rw-r--r-- 1 djzn djzn 341429 2010-02-11 20:34 bandstand-win32.flac

HINT: if you RE-flac the win32 file with flac linux, it will match the original
linux created file. If you RE-flac the linux original file, it won't change
at all.


Here is the link:
http://www.easy-share.com/1909277789/bandstand.zip

> 
> Another possibility could be different line endings in some
> metadata file (ie '\r\n' on windows and '\n' on Linux).
> 
The produced files are tagless. So I don't know about this... but it is a
good guess
-- 
Djeizon Barros <djzn.br at gmail.com>

Djeizon Barros wrote:
> > Another possibility could be different line endings in some
> > metadata file (ie '\r\n' on windows and '\n' on
Linux).
> > 
> 
> The produced files are tagless. So I don't know about this... but
> it is a good guess
Its a good guess and its wrong :-).
> I have uploaded a small file example for you to check.
> I am curious about what is exactly producing this mismatch filesize!
> I know it's ridiculous and probably too paranoid, but knowledge is
always welcome.
> 
> 
> -rw-r--r-- 1 djzn djzn 341418 2010-02-11 20:34 bandstand-linux.flac
> -rw-r--r-- 1 djzn djzn 556308 2010-02-11 20:34 bandstand.wav
> -rw-r--r-- 1 djzn djzn 341429 2010-02-11 20:34 bandstand-win32.flac
Wow, this is weird.

I have a program thats part of libsndfile called sndfile-cmp. Run as:

    sndfile-cmp bandstand-linux.flac bandstand-win32.flac

it shows nothing which means that the decoded audio of the two files is
identical.

However, I also have a hexdiff program which shows the two files to contain
the identical byte data all the way  up to offset 0x2080. Then the files
diverge wildly:

00002070:  FF F8 C9 A8 00 8D 4E 00  00 00 00 FF FF FF FF FF   ......N. ........
           FF F8 C9 A8 00 8D 4E 00  00 00 00 FF FF FF FF FF   ......N. ........

00002080:  FF FF FE FF FE FF FE B4  A2 7D E8 9D 2E 79 91 47   ........ .}...y.G
           FF FF FE FF FE FF FE B4  A2 7D E8 9D 36 79 11 47   ........ .}..6y.G

00002090:  57 05 9F DD 0A 2A 54 52  A5 13 13 2D 4B 2C B2 94   W....*TR ...-K,..
           57 05 9F DD 0A 2A 54 52  A5 13 13 2D 4B 2C 59 4A   W....*TR ...-K,YJ

000020A0:  95 6A 2D 25 24 85 95 96  26 49 0A 13 53 A9 24 51   .j-%$...
&I..S.$Q
           4A B5 16 92 92 42 CA CB  13 24 85 09 A9 D4 92 28   J....B.. .$.....(

The first column is the file offset and the first line for a given offset
is one file and the second line is the second file.

Certainly flac should produce the same files cross platform.
Failing any other explanation, try bad ram, it bit me before and
flac was where I first noticed it. Now I test all new/flaky machines.
See: memtest.org

On Feb 12, 2010, at 18:06, Erik de Castro Lopo wrote:
> Wow, this is weird.
>
> I have a program thats part of libsndfile called sndfile-cmp. Run as:
>
>     sndfile-cmp bandstand-linux.flac bandstand-win32.flac
>
> it shows nothing which means that the decoded audio of the two  
> files is
> identical.
>
> However, I also have a hexdiff program which shows the two files to  
> contain
> the identical byte data all the way  up to offset 0x2080. Then the  
> files
> diverge wildly:
>
> 00002070:  FF F8 C9 A8 00 8D 4E 00  00 00 00 FF FF FF FF  
> FF   ......N. ........
>            FF F8 C9 A8 00 8D 4E 00  00 00 00 FF FF FF FF  
> FF   ......N. ........
>
> 00002080:  FF FF FE FF FE FF FE B4  A2 7D E8 9D 2E 79 91  
> 47   ........ .}...y.G
>            FF FF FE FF FE FF FE B4  A2 7D E8 9D 36 79 11  
> 47   ........ .}..6y.G
>
> 00002090:  57 05 9F DD 0A 2A 54 52  A5 13 13 2D 4B 2C B2 94    
> W....*TR ...-K,..
>            57 05 9F DD 0A 2A 54 52  A5 13 13 2D 4B 2C 59 4A    
> W....*TR ...-K,YJ
>
> 000020A0:  95 6A 2D 25 24 85 95 96  26 49 0A 13 53 A9 24 51   .j-% 
> $... &I..S.$Q
>            4A B5 16 92 92 42 CA CB  13 24 85 09 A9 D4 92 28    
> J....B.. .$.....(

If you're willing to write some code, you could probably write a  
simple C program that looks for FLAC block headers in two files, then  
compares block by block.  Once you get a mismatch in one pair of  
blocks, it's probably going to take comparison by FLAC block offset  
instead of file byte offset.  Unfortunately, FLAC is a bit stream  
(the block header is 14 bits, not 16 bits).  It seems entirely  
possible that after your initial mismatch above, the rest of the FLAC  
blocks could be identical between the files, but their offset has  
shifted by 1 to 7 bits (if it had shifted by 8 bytes, then you'd  
probably notice identical bytes offset by one index).

In other words, all it takes is for one FLAC file to have a block  
that is longer or shorter by a multiple of 1 to 7 bits.  After that,  
even if the rest of the blocks are identical, a byte comparison is  
going to be different.

Brian Willoughby
Sound Consulting