flac dev - Jan 2007 - Help upgrading to 1.1.3 (MD5 sum issues, album art corrupts files)

Hello all,

I recently upgraded the libFLAC used in my application Max (http:// 
sbooth.org/Max/) to 1.1.3 and added preliminary support for album  
art.  During the upgrade I evidently made some coding mistakes with  
interesting results.  I've combed everything over and can't quite see  
the problems.  I've become somewhat frustrated because my code didn't  
really change significantly from my usage of the 1.1.2 APIs yet  
things don't work like they did previously.

My first issue is this: files generated by my application are  
reported as having "ERROR, MD5 signature mismatch" when tested with  
flac -t.  I am at a loss as to why!  When I set  
FLAC__stream_encoder_set_verify to true no errors are reported during  
the encode.  Has anyone seen anything like this?  The source for my  
encoder may be viewed at http://macaudiox.svn.sourceforge.net/viewvc/ 
macaudiox/trunk/Encoders/FLACEncoder.m?view=markup if anyone is curious.

The second issue is likely more serious.  When I attempt to add album  
art to one of the FLAC files I've created (using the level 2 metadata  
interface), the files are becoming corrupt (in a sense).  Testing  
with flac-t indicates "ERROR, --until value is before --skip  
point".   Interestingly, the file seems to work with metaflac:

METADATA block #0
   type: 0 (STREAMINFO)
   is last: false
   length: 34
   minimum blocksize: 4608 samples
   maximum blocksize: 4608 samples
   minimum framesize: 893 bytes
   maximum framesize: 14662 bytes
   sample_rate: 44100 Hz
   channels: 2
   bits-per-sample: 16
   total samples: 9626736
   MD5 signature: a9b57ffa2020017504e551c6ded0e092
METADATA block #1
   type: 4 (VORBIS_COMMENT)
   is last: false
   length: 336
   vendor string: reference libFLAC 1.1.3 20061120
   comments: 7
     comment[0]: ALBUM=Hope Floats
     comment[1]: ARTIST=Lyle Lovett
     comment[2]: TITLE=Smile
     comment[3]: TRACKNUMBER=11
     comment[4]: TRACKTOTAL=13
     comment[5]: ENCODER=Max 0.7.1b
     comment[6]: ENCODING=FLAC settings: exhaustiveModelSearch:0  
midSideStereo:1 looseMidSideStereo:0 QLPCoeffPrecision:0,  
minResidualPartitionOrder:0, maxResidualPartitionOrder:4, maxLPCOrder:8
METADATA block #2
   type: 6 (PICTURE)
   is last: false
   length: 32
   type: 3 (Cover (front))
   MIME type: image/png
   description:
   width: 800
   height: 600
   depth: 24
   colors: 0 (unindexed)
   data length: 685873
   data:
     00000000: 89 50 4E 47 0D 0A 1A 0A 00 00 00 0D 49 48 44  
52 .PNG........IHDR
     ! <snip> !
     000A7730: 82 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 .
METADATA block #3
   type: 1 (PADDING)
   is last: true
   length: 7860

Obviously I am doing something wrong, but again I'm not sure what.   
The tagging/album art source code is at http:// 
macaudiox.svn.sourceforge.net/viewvc/macaudiox/trunk/Tasks/ 
EncoderTasks/FLACEncoderTask.m?view=markup for the curious.

Does anyone have any ideas?

Stephen
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2409 bytes
Desc: not available
Url :
http://lists.xiph.org/pipermail/flac-dev/attachments/20070116/b6580cd8/smime.bin

--- "Stephen F. Booth" <me@sbooth.org>
wrote:
> Hello all,
> 
> I recently upgraded the libFLAC used in my application Max (http:// 
> sbooth.org/Max/) to 1.1.3 and added preliminary support for album  
> art.  During the upgrade I evidently made some coding mistakes with  
> interesting results.  I've combed everything over and can't quite
see
> the problems.  I've become somewhat frustrated because my code
didn't
> really change significantly from my usage of the 1.1.2 APIs yet  
> things don't work like they did previously.
> 
> My first issue is this: files generated by my application are  
> reported as having "ERROR, MD5 signature mismatch" when tested
with
> flac -t.  I am at a loss as to why!  When I set  
> FLAC__stream_encoder_set_verify to true no errors are reported during
> the encode.  Has anyone seen anything like this?  The source for my  
> encoder may be viewed at http://macaudiox.svn.sourceforge.net/viewvc/
> macaudiox/trunk/Encoders/FLACEncoder.m?view=markup if anyone is
> curious.
I took a quick look and it seems fine, but I'm assuming some
things about the format of the incoming samples.  the way samples
are passed to the encoder has not changed in flac-1.1.3 so if the
same code worked before it should still work.

when you decode the flac file with the 'bad md5' back to PCM,
does it match the original?  the best way to check is to just
feed it a file of 1 sample, then increase until you trigger
the problem.

a couple other things that are probably not related:
1. you can replace a lot of the settings more reliably and get better
   compression using FLAC__stream_encoder_set_compression_level()  
   instead, see
http://flac.sourceforge.net/api/group__flac__stream__encoder.html#ga20
2. libFLAC does not support 32 bps yet
> The second issue is likely more serious.  When I attempt to add album
> art to one of the FLAC files I've created (using the level 2 metadata
> interface), the files are becoming corrupt (in a sense).  Testing  
> with flac-t indicates "ERROR, --until value is before --skip  
> point".   Interestingly, the file seems to work with metaflac:
> 
> METADATA block #0
>    type: 0 (STREAMINFO)
>    is last: false
>    length: 34
>    minimum blocksize: 4608 samples
>    maximum blocksize: 4608 samples
>    minimum framesize: 893 bytes
>    maximum framesize: 14662 bytes
>    sample_rate: 44100 Hz
>    channels: 2
>    bits-per-sample: 16
>    total samples: 9626736
>    MD5 signature: a9b57ffa2020017504e551c6ded0e092
> METADATA block #1
>    type: 4 (VORBIS_COMMENT)
>    is last: false
>    length: 336
>    vendor string: reference libFLAC 1.1.3 20061120
>    comments: 7
>      comment[0]: ALBUM=Hope Floats
>      comment[1]: ARTIST=Lyle Lovett
>      comment[2]: TITLE=Smile
>      comment[3]: TRACKNUMBER=11
>      comment[4]: TRACKTOTAL=13
>      comment[5]: ENCODER=Max 0.7.1b
>      comment[6]: ENCODING=FLAC settings: exhaustiveModelSearch:0  
> midSideStereo:1 looseMidSideStereo:0 QLPCoeffPrecision:0,  
> minResidualPartitionOrder:0, maxResidualPartitionOrder:4,
> maxLPCOrder:8
> METADATA block #2
>    type: 6 (PICTURE)
>    is last: false
>    length: 32
>    type: 3 (Cover (front))
>    MIME type: image/png
>    description:
>    width: 800
>    height: 600
>    depth: 24
>    colors: 0 (unindexed)
>    data length: 685873
>    data:
>      00000000: 89 50 4E 47 0D 0A 1A 0A 00 00 00 0D 49 48 44  
> 52 .PNG........IHDR
>      ! <snip> !
>      000A7730: 82 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 .
> METADATA block #3
>    type: 1 (PADDING)
>    is last: true
>    length: 7860
> 
> Obviously I am doing something wrong, but again I'm not sure what.   
> The tagging/album art source code is at http:// 
> macaudiox.svn.sourceforge.net/viewvc/macaudiox/trunk/Tasks/ 
> EncoderTasks/FLACEncoderTask.m?view=markup for the curious.
that looks ok too, assuming addVorbisComment works.  the metadata
interface also has not changed in flac-1.1.3

but you should use the FLAC__metadata_object_picture_set_* methods
to set the picture data instead of altering the block directly:
http://flac.sourceforge.net/api/group__flac__metadata__object.html#ga42
http://flac.sourceforge.net/api/group__flac__metadata__object.html#ga43
http://flac.sourceforge.net/api/group__flac__metadata__object.html#ga44
that could be a source of mem corruption or leakage.

that's a wierd error though.  can you send or host one of the bad
FLAC files for me to check out?

also, are you using a libFLAC you built yourself or a prebuilt one
from sourceforge or somewhere else?

Josh



 
____________________________________________________________________________________
Don't get soaked.  Take a quick peak at the forecast
with the Yahoo! Search weather shortcut.
http://tools.search.yahoo.com/shortcuts/#loc_weather

Hi Josh,

Thanks for the response!
> I took a quick look and it seems fine, but I'm assuming some
> things about the format of the incoming samples.  the way samples
> are passed to the encoder has not changed in flac-1.1.3 so if the
> same code worked before it should still work.
This is the main source of my consternation, because my previous code  
did not generate files with MD5 warnings.  The incoming samples are  
packed, big-endian signed integers.  FWIW, the output files play fine  
and contain the correct data (see below); it's the MD5 sum that isn't  
working the way it should.
> when you decode the flac file with the 'bad md5' back to PCM,
> does it match the original?  the best way to check is to just
> feed it a file of 1 sample, then increase until you trigger
> the problem.
Decoding the files back to PCM using flac -dF produces the same audio  
data as the original, so no data was lost.  I have hosted example  
files at http://sbooth.org/files/flac/ if you care to take a look at  
an offender.  I am able to cause the problem using a file consisting  
of only one sample!  I will go over my code again- there must be a  
mistake in it somewhere.
> a couple other things that are probably not related:
> 1. you can replace a lot of the settings more reliably and get better
>    compression using FLAC__stream_encoder_set_compression_level()
>    instead, see
> http://flac.sourceforge.net/api/group__flac__stream__encoder.html#ga20
This is mainly due to my preferences interface to the encoder;  
advanced users can specify most settings individually.  I originally  
copied the settings you had for each compression level manually; this  
function should simplify the common use cases.
> 2. libFLAC does not support 32 bps yet
Thanks for pointing this out!
> that looks ok too, assuming addVorbisComment works.  the metadata
> interface also has not changed in flac-1.1.3
addVorbisComment seems to work, although I am not using your  
FLAC__metadata_object_vorbiscomment_entry_from_name_value_pair function:

void
addVorbisComment(FLAC__StreamMetadata		*block,
				 NSString					*key,
				 NSString					*value)
{
	NSString									*string;
	FLAC__StreamMetadata_VorbisComment_Entry	entry;
	
	string			= [NSString stringWithFormat:@"%@=%@", key, value];
	entry.entry		= (unsigned char *)strdup([string UTF8String]);
	NSCAssert(NULL != entry.entry, NSLocalizedStringFromTable(@"Unable  
to allocate memory.", @"Exceptions", @""));
	
	entry.length	= strlen((const char *)entry.entry);
	if(NO == FLAC__metadata_object_vorbiscomment_append_comment(block,  
entry, NO)) {
		free(entry.entry);
		@throw [NSException exceptionWithName:@"FLACException"
									   reason:[NSString  
stringWithFormat:NSLocalizedStringFromTable(@"The call to %@  
failed.", @"Exceptions", @""),  
@"FLAC__metadata_object_vorbiscomment_append_comment"]
									 userInfo:nil];
	}	
}

Is there an advantage/reason to/for using the library's function to  
do this?
> but you should use the FLAC__metadata_object_picture_set_* methods
> to set the picture data instead of altering the block directly:
> http://flac.sourceforge.net/api/ 
> group__flac__metadata__object.html#ga42
> http://flac.sourceforge.net/api/ 
> group__flac__metadata__object.html#ga43
> http://flac.sourceforge.net/api/ 
> group__flac__metadata__object.html#ga44
> that could be a source of mem corruption or leakage.
Great!  I switched to using your functions and the files are no  
longer corrupted.  Maybe this alone is reason to use  
FLAC__metadata_object_vorbiscomment_entry_from_name_value_pair :)
> that's a wierd error though.  can you send or host one of the bad
> FLAC files for me to check out?
If you're still interested in looking at a problem file I will be  
happy to upload it to my server.
> also, are you using a libFLAC you built yourself or a prebuilt one
> from sourceforge or somewhere else?
I am using an OS X framework build that I created myself.

Stephen
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2409 bytes
Desc: not available
Url :
http://lists.xiph.org/pipermail/flac-dev/attachments/20070118/b55ab1ff/smime.bin

Hi again Josh,
> also, are you using a libFLAC you built yourself or a prebuilt one
> from sourceforge or somewhere else?
This comment got me thinking, and I recompiled the Framework with asm  
disabled and now everything works just great.  Sorry for wasting your  
time on this- but thanks for the help!

Stephen
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2409 bytes
Desc: not available
Url :
http://lists.xiph.org/pipermail/flac-dev/attachments/20070118/c3ab114c/smime.bin