On Nov 26 22:39:27, hans at stare.cz wrote:> ./test_replaygain.sh fails for me in tonegenerator(), saying: > > ./test_replaygain.sh[91]: mawk: not found > Testing FLAC replaygain 8000 (8000 x 1) ... -: ERROR: got partial sample > > Apparently, the tone-generating awk script does not work with > my system's awk, which is "awk version 20110810" as distributed > with current OpenBSD 5.6 GENERIC.MP#610 amd64. > With mawk-1.3.4.20140914 it works fine > (and ./test_replaygain.sh passes). > > Can anyone please shed some light on what's mawk-spacific > about that awk script? It seems to be just a sine-generator.> Ha! If the final > printf("%c", byte[bx]); > is changed to > printf("%d", byte[bx]); > the results produced by mawk and awk are identical. > So the diference must be just that mawk printf's "%c" differently.> the two sequences of samples are identical as sequences of integers > - they just differ in the way they are printed out with AWK's printf("%c") > as opposed to MAWK's printf("%c"). > > I'll try to find time and look into it; this shouldn't be hard, > we just want to generate a sine wave portably.Indeed, the difference seems to be in how my systems AWK, as opposed to MAWK, printf's zero bytes. With mawk, a printf("%c") of a zero is always a zero byte (^@) in the output. With my system awk, it's sometimes a zero byte, and sometimes nothing. There are no other differences in the two outputs. As AWK variables are not typed, I suspect the zero byte might *sometimes* be considered a delimiter of an empty string. But that's where my incoplete knowledge of AWK chimes in. Anyway, the whole point of that AWK script is to produce a raw sine wave. Why are we doing it with AWK in the first place? Surely the FLAC distrubution contains tools to produce test sine waves, e.g. src/test_streams/main.c contains some, used in test_streams.sh. Can we make test_replaygain.sh also use the produced sine waves, instead of generating its own with AWK? Anyway, below is a patch to test_replaygain.sh that replaces the current AWK script with a simpler one. Is there a specific point in making the stream big-endian, or shifting it to positive values to have it unsigned? While this is a simpler script, it produces a different wave, so this breaks the precious pre-computed *GAIN values which we test against, right? It also leaves the test waves unremoved; this will go in a later version when I figure it out. Jan --- test_replaygain.sh.orig Fri Nov 28 18:15:11 2014 +++ test_replaygain.sh Sun Nov 30 14:17:39 2014 @@ -82,11 +82,9 @@ fi check_flac -if mawk ; then +if `which mawk >/dev/null 2>&1` ; then AWK=mawk else - # Really hope awk is not gawk, because the following AWK script doesn't - # work correctly with gawk 4.0.1 but did with earlier versions. AWK=awk fi @@ -95,41 +93,28 @@ else tonegenerator () { - # When using GAWK, use --lint=posix to identify non-POSIX awk usages. - $AWK -- ' - BEGIN { - samplerate = '$1'; + $AWK ' + BEGIN { + tone = 1000; + duration = 1; + samplerate = '$1'; + pi = 4 * atan2(1,1); - tone = 1000; - duration = 1; - bitspersample = 24; + bitspersample = 24; + amplitude = 2^bitpersample - 1; - samplemidpoint = 1; - for (sps = 0 ; sps < bitspersample - 1 ; sps++) { - samplemidpoint *= 2; + for (s = 0; s < duration * samplerate; s++) { + sample = sin(2 * pi * tone * s / samplerate); + sample = int(amplitude * sample); + for (b = 0; b < bitspersample/8; b++) { + printf("%c", sample % 256); + sample /= 256; } + } - samplerange = samplemidpoint - 1; - - pi = 4 * atan2(1,1); - - for (ix = 0; ix < duration * samplerate; ++ix) { - sample = sin(2 * pi * tone * ix / samplerate); - sample *= samplerange; - sample += samplemidpoint; - sample = int(sample); - for (bx = 0; bx < bitspersample/8; ++bx) { - byte[bx] = sample % 256; - sample /= 256; - } - while (bx--) { - printf("%c", byte[bx]); - } - } - - }' /dev/null | - flac${EXE} --silent --no-error-on-compression-fail --force-raw-format \ - --endian=big --channels=1 --bps=24 --sample-rate=$1 --sign=unsigned - + }' \ + | flac${EXE} --silent --no-error-on-compression-fail --force-raw-format\ + --endian=little --channels=1 --bps=24 --sample-rate=$1 --sign=signed - } REPLAYGAIN_FREQ@@ -168,6 +153,7 @@ for ACTION in $REPLAYGAIN_FREQ ; do RATE=$(($MULTIPLE * FREQ)) [ $MULTIPLE -eq 1 -o -n "${REPLAYGAIN_FREQ##* $RATE/*}" ] || break echo -n "Testing FLAC replaygain $RATE ($FREQ x $MULTIPLE) ... " + flacfile=$AWK.$RATE.replaygain.flac tonegenerator $RATE > $flacfile run_metaflac --add-replay-gain $flacfile run_metaflac --list $flacfile | grep REPLAYGAIN.*GAIN= | @@ -175,9 +161,11 @@ for ACTION in $REPLAYGAIN_FREQ ; do MEASUREDGAIN="${REPLAYGAIN##*=}" MEASUREDGAIN="${MEASUREDGAIN%% *}" if [ x"$MEASUREDGAIN" != x"$GAIN" ] ; then - die "ERROR, Expected $GAIN db instead of $REPLAYGAIN" + #die "ERROR, Expected $GAIN db instead of $REPLAYGAIN" + echo "ERROR, Expected $GAIN db instead of $REPLAYGAIN" fi done + #rm -f $flacfile echo OK done done
On Nov 30 14:33:47, hans at stare.cz wrote:> On Nov 26 22:39:27, hans at stare.cz wrote: > > ./test_replaygain.sh fails for me in tonegenerator(), saying: > > > > ./test_replaygain.sh[91]: mawk: not found > > Testing FLAC replaygain 8000 (8000 x 1) ... -: ERROR: got partial sample > > > > Apparently, the tone-generating awk script does not work with > > my system's awk, which is "awk version 20110810" as distributed > > with current OpenBSD 5.6 GENERIC.MP#610 amd64. > > With mawk-1.3.4.20140914 it works fine > > (and ./test_replaygain.sh passes). > > > > Can anyone please shed some light on what's mawk-spacific > > about that awk script? It seems to be just a sine-generator. > > > Ha! If the final > > printf("%c", byte[bx]); > > is changed to > > printf("%d", byte[bx]); > > the results produced by mawk and awk are identical. > > So the diference must be just that mawk printf's "%c" differently. > > > the two sequences of samples are identical as sequences of integers > > - they just differ in the way they are printed out with AWK's printf("%c") > > as opposed to MAWK's printf("%c"). > > > > I'll try to find time and look into it; this shouldn't be hard, > > we just want to generate a sine wave portably. > > Indeed, the difference seems to be in how my systems AWK, > as opposed to MAWK, printf's zero bytes. > > With mawk, a printf("%c") of a zero is always a zero byte (^@) in the output. > With my system awk, it's sometimes a zero byte, and sometimes nothing.Turns out it's a 20 year old bug in OpenBSD's awk: http://marc.info/?l=openbsd-misc&m=141735117703872&w=2 http://marc.info/?l=openbsd-tech&m=141739334413710&w=2 Now ./test_replaygain.sh seems to work the same with awk, mawk and gawk on OpenBSD/amd64, OpenBSD/i386, and Debian/x86_64. The diff below removes the check for mawk, and the comments about different AWKs, and just calls 'awk' for the script. gawk 4.0.1 and 4.1.1 works for me too. Erik, do you please remember what exactly happened here? Does gawk >= still fail for you? https://git.xiph.org/?p=flac.git;a=commit;h=5797009fa2beb0426d74485e7624775e2e58e1d1 While there, remove the *=2 loop, exponentiation is in the language.> Anyway, the whole point of that AWK script is to produce a raw sine wave. > Why are we doing it with AWK in the first place? Surely the FLAC > distrubution contains tools to produce test sine waves, > e.g. src/test_streams/main.c contains some, used in test_streams.sh. > Can we make test_replaygain.sh also use the produced sine waves, > instead of generating its own with AWK?This I haven't looked at yet, but I believe we could remove the awk sine generator altogether and just ask test_streams to make the streams. Also, the current script stores the individual bytes of the 24bit sample in an array, just to print them back in reverse and feed them to flac --endian=big. Why is that? Why doesn't it just print the bytes from the lowest and feed them to --endian=little? Also, the current script shifts the whole wave to the positive and feeds it to --sign=unsigned; why can't we just leave the wave as it is and feed it to --sign=signed? Jan --- test_replaygain.sh.orig Wed Dec 3 12:11:46 2014 +++ test_replaygain.sh Wed Dec 3 12:23:24 2014 @@ -81,34 +81,19 @@ fi check_flac - -if mawk ; then - AWK=mawk -else - # Really hope awk is not gawk, because the following AWK script doesn't - # work correctly with gawk 4.0.1 but did with earlier versions. - AWK=awk - fi - # Replay gain tests - Test the rates which have specific filter table entries # and verify that harmonics can be processed correctly. tonegenerator () { - # When using GAWK, use --lint=posix to identify non-POSIX awk usages. - $AWK -- ' + awk -- ' BEGIN { samplerate = '$1'; tone = 1000; duration = 1; bitspersample = 24; - - samplemidpoint = 1; - for (sps = 0 ; sps < bitspersample - 1 ; sps++) { - samplemidpoint *= 2; - } - + samplemidpoint = 2^(bitspersample - 1); samplerange = samplemidpoint - 1; pi = 4 * atan2(1,1); @@ -127,7 +112,7 @@ tonegenerator () } } - }' /dev/null | + }' /dev/null | \ flac${EXE} --silent --no-error-on-compression-fail --force-raw-format \ --endian=big --channels=1 --bps=24 --sample-rate=$1 --sign=unsigned - }
On Dec 03 13:21:16, hans at stare.cz wrote:> gawk 4.0.1 and 4.1.1 works for me too. > Erik, do you please remember what exactly happened here? > Does gawk >= still fail for you?Meaning gawk >= 4.0.1 of course, sorry.> https://git.xiph.org/?p=flac.git;a=commit;h=5797009fa2beb0426d74485e7624775e2e58e1d1
(ping, if there is any interest). On Dec 03 13:21:16, hans at stare.cz wrote:> On Nov 30 14:33:47, hans at stare.cz wrote: > > On Nov 26 22:39:27, hans at stare.cz wrote: > > > ./test_replaygain.sh fails for me in tonegenerator(), saying: > > > > > > ./test_replaygain.sh[91]: mawk: not found > > > Testing FLAC replaygain 8000 (8000 x 1) ... -: ERROR: got partial sample > > > > > > Apparently, the tone-generating awk script does not work with > > > my system's awk, which is "awk version 20110810" as distributed > > > with current OpenBSD 5.6 GENERIC.MP#610 amd64. > > > With mawk-1.3.4.20140914 it works fine > > > (and ./test_replaygain.sh passes). > > > > > > Can anyone please shed some light on what's mawk-spacific > > > about that awk script? It seems to be just a sine-generator. > > > > > Ha! If the final > > > printf("%c", byte[bx]); > > > is changed to > > > printf("%d", byte[bx]); > > > the results produced by mawk and awk are identical. > > > So the diference must be just that mawk printf's "%c" differently. > > > > > the two sequences of samples are identical as sequences of integers > > > - they just differ in the way they are printed out with AWK's printf("%c") > > > as opposed to MAWK's printf("%c"). > > > > > > I'll try to find time and look into it; this shouldn't be hard, > > > we just want to generate a sine wave portably. > > > > Indeed, the difference seems to be in how my systems AWK, > > as opposed to MAWK, printf's zero bytes. > > > > With mawk, a printf("%c") of a zero is always a zero byte (^@) in the output. > > With my system awk, it's sometimes a zero byte, and sometimes nothing. > > Turns out it's a 20 year old bug in OpenBSD's awk: > http://marc.info/?l=openbsd-misc&m=141735117703872&w=2 > http://marc.info/?l=openbsd-tech&m=141739334413710&w=2 > > Now ./test_replaygain.sh seems to work the same with awk, mawk and gawk > on OpenBSD/amd64, OpenBSD/i386, and Debian/x86_64. > > The diff below removes the check for mawk, and the comments > about different AWKs, and just calls 'awk' for the script. > > gawk 4.0.1 and 4.1.1 works for me too. > Erik, do you please remember what exactly happened here? > Does gawk >= still fail for you? > https://git.xiph.org/?p=flac.git;a=commit;h=5797009fa2beb0426d74485e7624775e2e58e1d1 > > While there, remove the *=2 loop, exponentiation is in the language. > > > Anyway, the whole point of that AWK script is to produce a raw sine wave. > > Why are we doing it with AWK in the first place? Surely the FLAC > > distrubution contains tools to produce test sine waves, > > e.g. src/test_streams/main.c contains some, used in test_streams.sh. > > Can we make test_replaygain.sh also use the produced sine waves, > > instead of generating its own with AWK? > > This I haven't looked at yet, but I believe we could > remove the awk sine generator altogether and just ask > test_streams to make the streams. > > Also, the current script stores the individual bytes > of the 24bit sample in an array, just to print them > back in reverse and feed them to flac --endian=big. > Why is that? Why doesn't it just print the bytes > from the lowest and feed them to --endian=little? > > Also, the current script shifts the whole wave to the positive > and feeds it to --sign=unsigned; why can't we just leave the wave > as it is and feed it to --sign=signed? > > > Jan > > > --- test_replaygain.sh.orig Wed Dec 3 12:11:46 2014 > +++ test_replaygain.sh Wed Dec 3 12:23:24 2014 > @@ -81,34 +81,19 @@ fi > > check_flac > > - > -if mawk ; then > - AWK=mawk > -else > - # Really hope awk is not gawk, because the following AWK script doesn't > - # work correctly with gawk 4.0.1 but did with earlier versions. > - AWK=awk > - fi > - > # Replay gain tests - Test the rates which have specific filter table entries > # and verify that harmonics can be processed correctly. > > tonegenerator () > { > - # When using GAWK, use --lint=posix to identify non-POSIX awk usages. > - $AWK -- ' > + awk -- ' > BEGIN { > samplerate = '$1'; > > tone = 1000; > duration = 1; > bitspersample = 24; > - > - samplemidpoint = 1; > - for (sps = 0 ; sps < bitspersample - 1 ; sps++) { > - samplemidpoint *= 2; > - } > - > + samplemidpoint = 2^(bitspersample - 1); > samplerange = samplemidpoint - 1; > > pi = 4 * atan2(1,1); > @@ -127,7 +112,7 @@ tonegenerator () > } > } > > - }' /dev/null | > + }' /dev/null | \ > flac${EXE} --silent --no-error-on-compression-fail --force-raw-format \ > --endian=big --channels=1 --bps=24 --sample-rate=$1 --sign=unsigned - > } > _______________________________________________ > flac-dev mailing list > flac-dev at xiph.org > http://lists.xiph.org/mailman/listinfo/flac-dev
On Dec 10 23:04:44, mle+la at mega-nerd.com wrote:> Definitely interested in a cross platform solution to this.Diff below. Please test. On Dec 03 13:21:16, hans at stare.cz wrote:> On Nov 30 14:33:47, hans at stare.cz wrote: > > On Nov 26 22:39:27, hans at stare.cz wrote: > > > ./test_replaygain.sh fails for me in tonegenerator(), saying: > > > > > > ./test_replaygain.sh[91]: mawk: not found > > > Testing FLAC replaygain 8000 (8000 x 1) ... -: ERROR: got partial sample > > > > > > Apparently, the tone-generating awk script does not work with > > > my system's awk, which is "awk version 20110810" as distributed > > > with current OpenBSD 5.6 GENERIC.MP#610 amd64. > > > With mawk-1.3.4.20140914 it works fine > > > (and ./test_replaygain.sh passes). > > > > > > Can anyone please shed some light on what's mawk-spacific > > > about that awk script? It seems to be just a sine-generator. > > > > > Ha! If the final > > > printf("%c", byte[bx]); > > > is changed to > > > printf("%d", byte[bx]); > > > the results produced by mawk and awk are identical. > > > So the diference must be just that mawk printf's "%c" differently. > > > > > the two sequences of samples are identical as sequences of integers > > > - they just differ in the way they are printed out with AWK's printf("%c") > > > as opposed to MAWK's printf("%c"). > > > > > > I'll try to find time and look into it; this shouldn't be hard, > > > we just want to generate a sine wave portably. > > > > Indeed, the difference seems to be in how my systems AWK, > > as opposed to MAWK, printf's zero bytes. > > > > With mawk, a printf("%c") of a zero is always a zero byte (^@) in the output. > > With my system awk, it's sometimes a zero byte, and sometimes nothing. > > Turns out it's a 20 year old bug in OpenBSD's awk: > http://marc.info/?l=openbsd-misc&m=141735117703872&w=2 > http://marc.info/?l=openbsd-tech&m=141739334413710&w=2Now ./test_replaygain.sh seems to work the same with awk, mawk and gawk on OpenBSD/amd64, OpenBSD/i386, and Debian/x86_64. The diff below removes the check for mawk, and the comments about different AWKs, and just calls 'awk' for the script. gawk 4.0.1 and 4.1.1 works for me too. Erik, do you please remember what exactly happened here: https://git.xiph.org/?p=flac.git;a=commit;h=5797009fa2beb0426d74485e7624775e2e58e1d1 Does gawk >= 4.0.1 still fail for you? With either the current script or with this diff? While there, remove the *=2 loop, exponentiation is in the language.> > Anyway, the whole point of that AWK script is to produce a raw sine wave. > > Why are we doing it with AWK in the first place? Surely the FLAC > > distrubution contains tools to produce test sine waves, > > e.g. src/test_streams/main.c contains some, used in test_streams.sh. > > Can we make test_replaygain.sh also use the produced sine waves, > > instead of generating its own with AWK?I still believe we don't need the awk script to generate a sine wave in the first place - we can ask test_streams; but I haven't looked into that yet. Two points not addressed in the diff: The current script stores the individual bytes of the 24bit sample in an array, just to print them back in reverse and feed them to flac --endian=big. Why is that? Why can't we just print the bytes from the lowest as they come by /256 and feed them to --endian=little? The current script shifts the whole wave to the positive and feeds it to --sign=unsigned; why can't we just leave the wave as it is and feed it to --sign=signed? Jan --- ./test_replaygain.sh.orig Thu Dec 11 08:57:12 2014 +++ ./test_replaygain.sh Wed Dec 3 13:11:33 2014 @@ -81,34 +81,19 @@ fi check_flac - -if mawk ; then - AWK=mawk -else - # Really hope awk is not gawk, because the following AWK script doesn't - # work correctly with gawk 4.0.1 but did with earlier versions. - AWK=awk - fi - # Replay gain tests - Test the rates which have specific filter table entries # and verify that harmonics can be processed correctly. tonegenerator () { - # When using GAWK, use --lint=posix to identify non-POSIX awk usages. - $AWK -- ' + awk -- ' BEGIN { samplerate = '$1'; tone = 1000; duration = 1; bitspersample = 24; - - samplemidpoint = 1; - for (sps = 0 ; sps < bitspersample - 1 ; sps++) { - samplemidpoint *= 2; - } - + samplemidpoint = 2^(bitspersample - 1); samplerange = samplemidpoint - 1; pi = 4 * atan2(1,1); @@ -127,7 +112,7 @@ tonegenerator () } } - }' /dev/null | + }' /dev/null | \ flac${EXE} --silent --no-error-on-compression-fail --force-raw-format \ --endian=big --channels=1 --bps=24 --sample-rate=$1 --sign=unsigned - }