flac dev - Jan 2011 - Synchronizing a streaming client to the server Was: Idea to possibly improve flac?

First of all, I am not aware of any official source of FLAC files  
that provide MP3 sourced data.  I meticulously check the music I  
purchase, especially when it is 24/48 or 24/96 material, because this  
is new technology, and sometimes people get it wrong.

However, you should be aware that many modern producers use software  
to create their music, and when the software stores sound clips in  
MP3 format, what you end up with is music that sometimes looks like  
MP3.  I recently purchased a second release of an old download from  
an artist who has his material re-mastered.  Since he made such a big  
deal about the re-mastering, I took a close look at the quality.  For  
some reason, the second track looked like an MP3 source, but I'm sure  
it just has to do with the software that was used to create the music  
originally.

In other words, if you try to shut down the FLAC encoder based on an  
FFT, you might have a lot of false triggers!

I purchase a great deal of music, exclusively in FLAC format.  I  
purchase from LINN Records, Bleep.com, Warp Records, and also  
directly from artists like Nine Inch Nails who provide FLAC files.  I  
have never seen anyone provide MP3 quality.

For that matter, OggFLAC seems to be a format that has never been  
used.  Ever.  I have simply never come across a legitimate source of  
music for purchasing which used the OggFLAC format.  I have seen FLAC  
come and go and come back again.

Various online record labels started out with FLAC for bandwidth  
reasons.  Then they seemed to switch over to WAV as bandwidth became  
less of an issue, and I assume that their customers were confused by  
FLAC because of the lack of support in iTunes and other highly  
popular players.  Meanwhile, hardware such as the Sound Devices 700  
Series, the Squeezebox, and many other professional products has  
started with FLAC and stuck with it.  The sites who switched to WAV  
are now bringing back FLAC, but none of them have ever used OggFLAC.

Finally, I think that people who are not embedded firmware developers  
do not understand why the FLAC sources have stopped changing.  What  
we have here is a rare case of a professional set of sources which do  
not have bugs, and which represents a solid standard that does not  
need changing.  People are selling hardware devices in droves, and  
they cannot afford to change their firmware every time some random  
change happens in the FLAC source.  It's actually way better that  
FLAC is not changing.

Even when Apple came out with ALAC, their version of FLAC, I noticed  
that they could not consistently beat FLAC on coding speed and file  
size.  Some audio turns out smaller with ALAC, other audio turns out  
smaller with FLAC.  Overall, the average performance is identical.   
Apple hired some of the most amazing geniuses of physics to design  
ALAC, and if they can't beat the performance of FLAC in all  
situations, then what makes you think there is any reason to make a  
single change to the FLAC sources?

While I'm writing, I also want to respond to the question about how  
to change FLAC so that all of the third party tools pick up the  
change.  Well, I don't think that is possible.  Many tools run the  
command-line flac utility behind the scenes.  Others use the FLAC  
library directly.  The problem is that both of them often run with  
out of date versions of the FLAC code, so no matter which way they  
incorporate the official FLAC sources, you cannot make them update to  
your anti-MP3 version.

On that last note, I want to encourage you to experiment and have fun  
trying to create an MP3 detector that could warn users about quality  
issues.  However, I believe it is extremely unlikely that you would  
ever be successful in getting your code into the official FLAC  
sources.  This kind of change has nothing to do with the official  
FLAC format, and thus I doubt there would be any professional  
interest in changing things just for the sake of change or "newness."

Brian Willoughby
Sound Consultinf


On Jan 7, 2011, at 12:56, David Richards wrote:
> Its really sad to hear thats happening but even more sad is the fact
> that flac is becoming a very common format for music on the interweb
> whilst at the same time the development has ceased. I've found some
> severe issues with OggFLAC that essentially make it a useless format
> for streaming, no one cared.
>
> On Fri, Jan 7, 2011 at 3:42 PM, J?rgen Vigdal <jorgen at anion.no>
wrote:
>> Due to the fact that more and more users increasingly use MP3 <  
>> 320kbps as
>> their source for encoding music, and publish it as flac files, I  
>> suggest
>> that something is done in the flac encoder to possible avoid this.
>> My idea is kinda easy/stupid, but might work;
>> Implement a function that use a FFT to check if the input has  
>> frequencies >
>> 16kHz, and informs the user that the file would not be encoded  
>> unless a
>> -force parameter is specified (or at least ask the user if he or  
>> she want to
>> do this :) )
>> Hopefully, this will reduce the number of files released on the  
>> internet,
>> re-encoded from a lossy file format. Unfortunately, many users  
>> avoid using
>> flac, because they think the encoder is lossy due to the poor  
>> sound on some
>> files released.
>
>

I actually agree with pretty much everything Brian just said.

To add to that though, I'd say that mp3-to-FLAC transcodes are a very
real problem for, shall we say, illegitimate sources of material. (And
that is a totally legitimate thing, in and of itself... er, what?)

- BW



On Fri, Jan 7, 2011 at 5:22 PM, Brian Willoughby <brianw at sounds.wa.com>
wrote:
> First of all, I am not aware of any official source of FLAC files
> that provide MP3 sourced data. ?I meticulously check the music I
> purchase, especially when it is 24/48 or 24/96 material, because this
> is new technology, and sometimes people get it wrong.
>
> However, you should be aware that many modern producers use software
> to create their music, and when the software stores sound clips in
> MP3 format, what you end up with is music that sometimes looks like
> MP3. ?I recently purchased a second release of an old download from
> an artist who has his material re-mastered. ?Since he made such a big
> deal about the re-mastering, I took a close look at the quality. ?For
> some reason, the second track looked like an MP3 source, but I'm sure
> it just has to do with the software that was used to create the music
> originally.
>
> In other words, if you try to shut down the FLAC encoder based on an
> FFT, you might have a lot of false triggers!
>
> I purchase a great deal of music, exclusively in FLAC format. ?I
> purchase from LINN Records, Bleep.com, Warp Records, and also
> directly from artists like Nine Inch Nails who provide FLAC files. ?I
> have never seen anyone provide MP3 quality.
>
> For that matter, OggFLAC seems to be a format that has never been
> used. ?Ever. ?I have simply never come across a legitimate source of
> music for purchasing which used the OggFLAC format. ?I have seen FLAC
> come and go and come back again.
>
> Various online record labels started out with FLAC for bandwidth
> reasons. ?Then they seemed to switch over to WAV as bandwidth became
> less of an issue, and I assume that their customers were confused by
> FLAC because of the lack of support in iTunes and other highly
> popular players. ?Meanwhile, hardware such as the Sound Devices 700
> Series, the Squeezebox, and many other professional products has
> started with FLAC and stuck with it. ?The sites who switched to WAV
> are now bringing back FLAC, but none of them have ever used OggFLAC.
>
> Finally, I think that people who are not embedded firmware developers
> do not understand why the FLAC sources have stopped changing. ?What
> we have here is a rare case of a professional set of sources which do
> not have bugs, and which represents a solid standard that does not
> need changing. ?People are selling hardware devices in droves, and
> they cannot afford to change their firmware every time some random
> change happens in the FLAC source. ?It's actually way better that
> FLAC is not changing.
>
> Even when Apple came out with ALAC, their version of FLAC, I noticed
> that they could not consistently beat FLAC on coding speed and file
> size. ?Some audio turns out smaller with ALAC, other audio turns out
> smaller with FLAC. ?Overall, the average performance is identical.
> Apple hired some of the most amazing geniuses of physics to design
> ALAC, and if they can't beat the performance of FLAC in all
> situations, then what makes you think there is any reason to make a
> single change to the FLAC sources?
>
> While I'm writing, I also want to respond to the question about how
> to change FLAC so that all of the third party tools pick up the
> change. ?Well, I don't think that is possible. ?Many tools run the
> command-line flac utility behind the scenes. ?Others use the FLAC
> library directly. ?The problem is that both of them often run with
> out of date versions of the FLAC code, so no matter which way they
> incorporate the official FLAC sources, you cannot make them update to
> your anti-MP3 version.
>
> On that last note, I want to encourage you to experiment and have fun
> trying to create an MP3 detector that could warn users about quality
> issues. ?However, I believe it is extremely unlikely that you would
> ever be successful in getting your code into the official FLAC
> sources. ?This kind of change has nothing to do with the official
> FLAC format, and thus I doubt there would be any professional
> interest in changing things just for the sake of change or
"newness."
>
> Brian Willoughby
> Sound Consultinf
>
>
> On Jan 7, 2011, at 12:56, David Richards wrote:
>> Its really sad to hear thats happening but even more sad is the fact
>> that flac is becoming a very common format for music on the interweb
>> whilst at the same time the development has ceased. I've found some
>> severe issues with OggFLAC that essentially make it a useless format
>> for streaming, no one cared.
>>
>> On Fri, Jan 7, 2011 at 3:42 PM, J?rgen Vigdal <jorgen at
anion.no> wrote:
>>> Due to the fact that more and more users increasingly use MP3 <
>>> 320kbps as
>>> their source for encoding music, and publish it as flac files, I
>>> suggest
>>> that something is done in the flac encoder to possible avoid this.
>>> My idea is kinda easy/stupid, but might work;
>>> Implement a function that use a FFT to check if the input has
>>> frequencies >
>>> 16kHz, and informs the user that the file would not be encoded
>>> unless a
>>> -force parameter is specified (or at least ask the user if he or
>>> she want to
>>> do this :) )
>>> Hopefully, this will reduce the number of files released on the
>>> internet,
>>> re-encoded from a lossy file format. Unfortunately, many users
>>> avoid using
>>> flac, because they think the encoder is lossy due to the poor
>>> sound on some
>>> files released.
>>
>>
> _______________________________________________
> Flac-dev mailing list
> Flac-dev at xiph.org
> http://lists.xiph.org/mailman/listinfo/flac-dev
>

I for one am not worried about getting mp3 encoded stuff in my flac
files, but I want to respond about "legitimate" OggFLAC.

OggFLAC as a format for files, I agree, used by no one. However, I
don't know of any other open source way to stream lossless audio.
Maybe I did not look hard enough. Certainly nothing I can think of
that would hope to be compatible with standard media player softwares
out there.

I am interested in streaming lossless audio, FLAC is probably the best
option for that. Currently the OggFLAC way of doing it mostly works
with a few hacks in libflac and my version of edcast. It might be that
the Ogg packaging layer is ill suited for this purpose, and an
alternative model developed.  I've seen that its possible to stream
native flac with netcat, but thats not really the solution I'm looking
for.

I'm also interested in another concept of lossless streaming with
flac. Lets call it broadcast flac. A problem with streaming for long
periods of time is that the sending and receiving computers clocks go
out of sync, for example even if I stream myself on localhost, with
something like 1 second of buffertime, the client will eat into that
over about one hour and run out of buffer causing a skip. It could
also theoretically keep getting more and more lagged until its way
behind. I'd like to develop a standard system that could be sent
between the server and client side to keep there clients in sync to a
fairly exacting amount of lag time behind the server (say 1.5 seconds,
I think you end up with about a half second of delay with flac
encoding no matter what but my brain is blanking on this right now,
meaning it would not be suitable for 'real time" remote sound mixing).
Anyway what could happen is the client could do a little bit of
re-sampling here or there to ensure its in sync with the servers
clock.

If anyone is interested in such things, or knows of an existing system
that I don't know of, give me a hollar!

-David



On Fri, Jan 7, 2011 at 5:22 PM, Brian Willoughby <brianw at sounds.wa.com>
wrote:
> First of all, I am not aware of any official source of FLAC files that
> provide MP3 sourced data. ?I meticulously check the music I purchase,
> especially when it is 24/48 or 24/96 material, because this is new
> technology, and sometimes people get it wrong.
>
> However, you should be aware that many modern producers use software to
> create their music, and when the software stores sound clips in MP3 format,
> what you end up with is music that sometimes looks like MP3. ?I recently
> purchased a second release of an old download from an artist who has his
> material re-mastered. ?Since he made such a big deal about the
re-mastering,
> I took a close look at the quality. ?For some reason, the second track
> looked like an MP3 source, but I'm sure it just has to do with the
software
> that was used to create the music originally.
>
> In other words, if you try to shut down the FLAC encoder based on an FFT,
> you might have a lot of false triggers!
>
> I purchase a great deal of music, exclusively in FLAC format. ?I purchase
> from LINN Records, Bleep.com, Warp Records, and also directly from artists
> like Nine Inch Nails who provide FLAC files. ?I have never seen anyone
> provide MP3 quality.
>
> For that matter, OggFLAC seems to be a format that has never been used.
> ?Ever. ?I have simply never come across a legitimate source of music for
> purchasing which used the OggFLAC format. ?I have seen FLAC come and go and
> come back again.
>
> Various online record labels started out with FLAC for bandwidth reasons.
> ?Then they seemed to switch over to WAV as bandwidth became less of an
> issue, and I assume that their customers were confused by FLAC because of
> the lack of support in iTunes and other highly popular players. ?Meanwhile,
> hardware such as the Sound Devices 700 Series, the Squeezebox, and many
> other professional products has started with FLAC and stuck with it. ?The
> sites who switched to WAV are now bringing back FLAC, but none of them have
> ever used OggFLAC.
>
> Finally, I think that people who are not embedded firmware developers do
not
> understand why the FLAC sources have stopped changing. ?What we have here
is
> a rare case of a professional set of sources which do not have bugs, and
> which represents a solid standard that does not need changing. ?People are
> selling hardware devices in droves, and they cannot afford to change their
> firmware every time some random change happens in the FLAC source.
?It's
> actually way better that FLAC is not changing.
>
> Even when Apple came out with ALAC, their version of FLAC, I noticed that
> they could not consistently beat FLAC on coding speed and file size. ?Some
> audio turns out smaller with ALAC, other audio turns out smaller with FLAC.
> ?Overall, the average performance is identical. ?Apple hired some of the
> most amazing geniuses of physics to design ALAC, and if they can't beat
the
> performance of FLAC in all situations, then what makes you think there is
> any reason to make a single change to the FLAC sources?
>
> While I'm writing, I also want to respond to the question about how to
> change FLAC so that all of the third party tools pick up the change. ?Well,
> I don't think that is possible. ?Many tools run the command-line flac
> utility behind the scenes. ?Others use the FLAC library directly. ?The
> problem is that both of them often run with out of date versions of the
FLAC
> code, so no matter which way they incorporate the official FLAC sources,
you
> cannot make them update to your anti-MP3 version.
>
> On that last note, I want to encourage you to experiment and have fun
trying
> to create an MP3 detector that could warn users about quality issues.
> ?However, I believe it is extremely unlikely that you would ever be
> successful in getting your code into the official FLAC sources. ?This kind
> of change has nothing to do with the official FLAC format, and thus I doubt
> there would be any professional interest in changing things just for the
> sake of change or "newness."
>
> Brian Willoughby
> Sound Consultinf
>
>
> On Jan 7, 2011, at 12:56, David Richards wrote:
>>
>> Its really sad to hear thats happening but even more sad is the fact
>> that flac is becoming a very common format for music on the interweb
>> whilst at the same time the development has ceased. I've found some
>> severe issues with OggFLAC that essentially make it a useless format
>> for streaming, no one cared.
>>
>> On Fri, Jan 7, 2011 at 3:42 PM, J?rgen Vigdal <jorgen at
anion.no> wrote:
>>>
>>> Due to the fact that more and more users increasingly use MP3 <
320kbps
>>> as
>>> their source for encoding music, and publish it as flac files, I
suggest
>>> that something is done in the flac encoder to possible avoid this.
>>> My idea is kinda easy/stupid, but might work;
>>> Implement a function that use a FFT to check if the input has
frequencies
>>> >
>>> 16kHz, and informs the user that the file would not be encoded
unless a
>>> -force parameter is specified (or at least ask the user if he or
she want
>>> to
>>> do this :) )
>>> Hopefully, this will reduce the number of files released on the
internet,
>>> re-encoded from a lossy file format. Unfortunately, many users
avoid
>>> using
>>> flac, because they think the encoder is lossy due to the poor sound
on
>>> some
>>> files released.
>>
>>
>

Hi Brian.

I also agree with you on these points you mention. If you guys are familiar on
how the piracy groups work on the internet, you are aware that they have
"releases" with their names on it. In the piracy "scene",
some groups are competing on getting the first release out, and could only be
beaten by another group releasing another higher quality release. Some groups
(or even individuals) are releasing their stuff that is being "ripped
off" another release, and they transcode the original release (mp3 320kbps
for example) to a flac release (that really isn't a flac).

Some of these groups or individuals are young people, tinking that they know
everything. My idea was based on this. It would be fun stopping this, and also,
as you mention in your answer, having fun and experimenting with the flac code.

Thanks,

J.
 
On Jan 7, 2011, at 11:22 PM, Brian Willoughby wrote:
> First of all, I am not aware of any official source of FLAC files  
> that provide MP3 sourced data.  I meticulously check the music I  
> purchase, especially when it is 24/48 or 24/96 material, because this  
> is new technology, and sometimes people get it wrong.
> 
> However, you should be aware that many modern producers use software  
> to create their music, and when the software stores sound clips in  
> MP3 format, what you end up with is music that sometimes looks like  
> MP3.  I recently purchased a second release of an old download from  
> an artist who has his material re-mastered.  Since he made such a big  
> deal about the re-mastering, I took a close look at the quality.  For  
> some reason, the second track looked like an MP3 source, but I'm sure  
> it just has to do with the software that was used to create the music  
> originally.
> 
> In other words, if you try to shut down the FLAC encoder based on an  
> FFT, you might have a lot of false triggers!
> 
> I purchase a great deal of music, exclusively in FLAC format.  I  
> purchase from LINN Records, Bleep.com, Warp Records, and also  
> directly from artists like Nine Inch Nails who provide FLAC files.  I  
> have never seen anyone provide MP3 quality.
> 
> For that matter, OggFLAC seems to be a format that has never been  
> used.  Ever.  I have simply never come across a legitimate source of  
> music for purchasing which used the OggFLAC format.  I have seen FLAC  
> come and go and come back again.
> 
> Various online record labels started out with FLAC for bandwidth  
> reasons.  Then they seemed to switch over to WAV as bandwidth became  
> less of an issue, and I assume that their customers were confused by  
> FLAC because of the lack of support in iTunes and other highly  
> popular players.  Meanwhile, hardware such as the Sound Devices 700  
> Series, the Squeezebox, and many other professional products has  
> started with FLAC and stuck with it.  The sites who switched to WAV  
> are now bringing back FLAC, but none of them have ever used OggFLAC.
> 
> Finally, I think that people who are not embedded firmware developers  
> do not understand why the FLAC sources have stopped changing.  What  
> we have here is a rare case of a professional set of sources which do  
> not have bugs, and which represents a solid standard that does not  
> need changing.  People are selling hardware devices in droves, and  
> they cannot afford to change their firmware every time some random  
> change happens in the FLAC source.  It's actually way better that  
> FLAC is not changing.
> 
> Even when Apple came out with ALAC, their version of FLAC, I noticed  
> that they could not consistently beat FLAC on coding speed and file  
> size.  Some audio turns out smaller with ALAC, other audio turns out  
> smaller with FLAC.  Overall, the average performance is identical.   
> Apple hired some of the most amazing geniuses of physics to design  
> ALAC, and if they can't beat the performance of FLAC in all  
> situations, then what makes you think there is any reason to make a  
> single change to the FLAC sources?
> 
> While I'm writing, I also want to respond to the question about how  
> to change FLAC so that all of the third party tools pick up the  
> change.  Well, I don't think that is possible.  Many tools run the  
> command-line flac utility behind the scenes.  Others use the FLAC  
> library directly.  The problem is that both of them often run with  
> out of date versions of the FLAC code, so no matter which way they  
> incorporate the official FLAC sources, you cannot make them update to  
> your anti-MP3 version.
> 
> On that last note, I want to encourage you to experiment and have fun  
> trying to create an MP3 detector that could warn users about quality  
> issues.  However, I believe it is extremely unlikely that you would  
> ever be successful in getting your code into the official FLAC  
> sources.  This kind of change has nothing to do with the official  
> FLAC format, and thus I doubt there would be any professional  
> interest in changing things just for the sake of change or
"newness."
> 
> Brian Willoughby
> Sound Consultinf
>

Am 07.01.2011 23:38, schrieb David Richards:
> I'm also interested in another concept of lossless streaming with
> flac. Lets call it broadcast flac. A problem with streaming for long
> periods of time is that the sending and receiving computers clocks go
> out of sync, for example even if I stream myself on localhost, with
This is not a FLAC specific problem, but has to be handled in all 
situations where the streaming server is in control of the transmitting 
data rate. It's caused by a playback device, which actual sample rate is 
slightly higher than the sample rate actually requested or a streaming 
source, which system clock is running slowly. Since these parameters (at 
least an exact playback sample rate) is hard to achieve, this is a 
rather common problem. Or to shorten it: If the data has a sample rate 
of 44100 and your sound card consumes more than 44100 samples per 
"sender-time" second, your buffer will eventually exhaust. If it's
the
other way around, your buffer may overflow if the client does not handle 
these cases properly.
> Anyway what could happen is the client could do a little bit of
> re-sampling here or there to ensure its in sync with the servers
> clock.
That is how streaming clients usually solve this problem, although is 
not really improving sound quality.

Tor

On 1/7/2011 11:42 PM, J?rgen Vigdal wrote:
> Hi Brian.
>
> I also agree with you on these points you mention. If you guys are familiar
on how the piracy groups work on the internet, you are aware that they have
"releases" with their names on it. In the piracy "scene",
some groups are competing on getting the first release out, and could only be
beaten by another group releasing another higher quality release. Some groups
(or even individuals) are releasing their stuff that is being "ripped
off" another release, and they transcode the original release (mp3 320kbps
for example) to a flac release (that really isn't a flac).
>
> Some of these groups or individuals are young people, tinking that they
know everything. My idea was based on this. It would be fun stopping this, and
also, as you mention in your answer, having fun and experimenting with the flac
code.
>
... that really sucks. Pirates giving a genuinely great codec a bad name 
because of the way their ecosystem promotes treachery. Though I wonder 
if they wouldn't self-regulate by requiring EAC .logs or something like 
that?

I think an simple tool that is run on existing FLAC files and gives a 
clear good/bad answer (perhaps with a probability to remain fair) could 
spread like wildfire amongst audiophiles if publicized in the right 
channels.
> Thanks,
>
> J.
-Markus-

On Fri, Jan 07, 2011 at 02:22:51PM -0800, brianw at sounds.wa.com
wrote:
> 
> First of all, I am not aware of any official source of FLAC files  
> that provide MP3 sourced data.
Unofficial sources (such as Usenet and that torrent site with the old
fashioned sailing ship as its logo) are much more likely to have FLAC
files that were made from lossy audio.

And I vaguely remember reading about an illegal download site that
stored all audio in MP3 (at less than 320k) and transcoded on the fly
for all other bitrates and formats, including FLAC and 320k MP3. They
did it to save storage space.
> However, you should be aware that many modern producers use software  
> to create their music, and when the software stores sound clips in  
> MP3 format, what you end up with is music that sometimes looks like  
> MP3.
I recently bought the double-CD "Influence" remaster by The Art Of
Noise
and some rarer tracks were sourced from MP3 because that was all their
archivist could find. Most of the reissue was direct from analogue tapes
so this wasn't a quick "shovelware" reissue job.
> it just has to do with the software that was used to create the music  
> originally.
A friend of mine recorded his band's last album on DCC in the mid 1990s
and released it on CD. It sounds horrible; the lossy compression of DCC
is even worse than MiniDisc's ATRAC. I'm sure this CD would fail most
FFT quality tests, as literally everyone who heard it (not just people
with "golden ears" or good sound systems) complained about the
quality.
> In other words, if you try to shut down the FLAC encoder based on an  
> FFT, you might have a lot of false triggers!
I think it's a bad idea for a lot of reasons: checking the source audio
quality should be a job for another tool. Most FLAC users won't need to
check (most of my FLAC files are ripped from original CDs that I own),
and anyone who was trying to fool listeners (or fellow piracy groups)
would either work out how to bypass the check, or (more likely) use an
older version of FLAC.

And it's not in keeping with the philosophy behind FLAC: one thing that
I regularly say to people who aren't sure about using FLAC is that Josh
designed it with no copy protection support: if it was there, someone
would only crack it, so it is effectively useless. And that's probably
why Apple's ALAC is usually bigger than FLAC for the same uncompressed
audio (and why Apple still don't support FLAC in their products).

Stopping a pirate from encoding FLAC is similar to stopping a pirate
from ripping a copy-protected CD: it's a challenge to be overcome, and
it will probably take "them" less time to work it out than it took
"us"
to build it. And "they" only need to work it out once. Which is why
all
copy protection and DRM sucks, for everyone.

> Nine Inch Nails who provide FLAC files.
The initial free online release of NIN's The Slip had a problem with the
24-bit version: it wasn't the full 24/96. Turns out it was a genuine
oversight by the mastering lab (who used the 24/96 to master the vinyl),
and the true 24/96 files were reissued less than a week later.
So even with the artist fully behind 24-bit FLAC, and doing almost all
of the work in their own studio, things can still go wrong.
> People are selling hardware devices in droves, and  
> they cannot afford to change their firmware every time some random  
> change happens in the FLAC source.  It's actually way better that  
> FLAC is not changing.
The FLAC source can change without violating compatibility. Faster or
more efficient encoding can still produce compressed data that older
decoders can process.
> Some audio turns out smaller with ALAC, other audio turns out  
> smaller with FLAC.  Overall, the average performance is identical.   
I've been trying Flake (an alternative FLAC encoder; all it does is
encode) recently and while it goes way faster than the reference FLAC
encoder in terms of speed, the first encode I tried with it ended up
larger than with the reference encoder, at all compression levels.
The compression levels in Flake go up to 11, but past 9 or 10 there is
no guarantee of full compatibility for playback.
> Many tools run the  
> command-line flac utility behind the scenes.  Others use the FLAC  
> library directly.
About a month ago I was setting up FLAC support for a friend on a
Windows PC. Almost every GUI front end (based on the latest available
download) that I tried was using an out of date version of the FLAC
binary or library, and had the default compression set to -6 or lower.
There is no excuse for not cranking up the compression level on any
modern computer: surely you want to get the files as small as possible?
> I doubt there would be any professional  
> interest in changing things just for the sake of change or
"newness."
The only "new" thing I want in FLAC is a fix for a minor bug in the
way
the command line tool treats the end of line. If long filenames push the
"percent complete" past the screen width and you are encoding or
testing
a lot of files, you can end up with the longer file names having 20 or
more lines on the screen output, and scrolling the previous files off
the screen. My workaround is to use a longer scrollback in whatever
terminal I'm using, or use GNU Screen to get scrollback on a text
console.

-- 
-Dec.
---

Of Course!

It would be fun. I don't have too much time to spend, but I'll try my
best to contribute to the project :)

If anyone more interested, feel free to get in touch!

J.


On Jan 7, 2011, at 11:55 PM, Brian Waters wrote:
> The scene is full of scumbag hackers who would have no trouble getting
> around anything done on the encode side anyway. A better solution
> would be a decoder that warns the user when there's a potential
> transcode... of course, that's really not something that the FLAC
> decoder should do. Could be a fun "third party" project for users
who
> care about that sort of thing. Plus, I'm pretty sure you can benefit
> from doing an FFT on a GPU (even more fun!).
> 
> J?rgen, you interested? I think you're right, it's not a very big
> project at all.
> 
> - Brian
>

Cool, thanks for all the great comments. 

I think we agree now on that the "find mp3 before encoding" feature
would not be a good idea to implement in the flac core. As Brian pointed out, it
might be a better idea to create a program that automatically checks if a flac
might have been an mp3 source.

My first suggestion was to use FFT, because I know that 128kbps mp3 have a
low-pass filter at 16kHz (Fraunhofer IIS Encoder). The program should not decide
whether or not the file is a mp3 based on only that, but it could give an
indication on that. Perhaps with people not very familiar with advanced audio
tools know how to spot out a mp3, they just want to know if the flac they just
bought or downloaded is good or not?

Another test, if the FFT test is unclear, is to check the correlation between
left and right channel below 3kHz, which should be just about nothing if the
source is a mp3 (since mp3 encoders sum L and R below 3kHz at low bitrates). If
also doing further testing, and knowing how the mp3 encoders work, it should be
fairly easy to determine if a source file might have been an mp3. At least the
program would be able to tell if the input file is poor quality? :)

J. 
On Jan 8, 2011, at 12:28 AM, Declan Kelly wrote:
> On Fri, Jan 07, 2011 at 02:22:51PM -0800, brianw at sounds.wa.com wrote:
>> 
>> First of all, I am not aware of any official source of FLAC files  
>> that provide MP3 sourced data.
> 
> Unofficial sources (such as Usenet and that torrent site with the old
> fashioned sailing ship as its logo) are much more likely to have FLAC
> files that were made from lossy audio.
> 
> And I vaguely remember reading about an illegal download site that
> stored all audio in MP3 (at less than 320k) and transcoded on the fly
> for all other bitrates and formats, including FLAC and 320k MP3. They
> did it to save storage space.
> 
>> However, you should be aware that many modern producers use software  
>> to create their music, and when the software stores sound clips in  
>> MP3 format, what you end up with is music that sometimes looks like  
>> MP3.
> 
> I recently bought the double-CD "Influence" remaster by The Art
Of Noise
> and some rarer tracks were sourced from MP3 because that was all their
> archivist could find. Most of the reissue was direct from analogue tapes
> so this wasn't a quick "shovelware" reissue job.
> 
>> it just has to do with the software that was used to create the music  
>> originally.
> 
> A friend of mine recorded his band's last album on DCC in the mid 1990s
> and released it on CD. It sounds horrible; the lossy compression of DCC
> is even worse than MiniDisc's ATRAC. I'm sure this CD would fail
most
> FFT quality tests, as literally everyone who heard it (not just people
> with "golden ears" or good sound systems) complained about the
quality.
> 
>> In other words, if you try to shut down the FLAC encoder based on an  
>> FFT, you might have a lot of false triggers!
> 
> I think it's a bad idea for a lot of reasons: checking the source audio
> quality should be a job for another tool. Most FLAC users won't need to
> check (most of my FLAC files are ripped from original CDs that I own),
> and anyone who was trying to fool listeners (or fellow piracy groups)
> would either work out how to bypass the check, or (more likely) use an
> older version of FLAC.
> 
> And it's not in keeping with the philosophy behind FLAC: one thing that
> I regularly say to people who aren't sure about using FLAC is that Josh
> designed it with no copy protection support: if it was there, someone
> would only crack it, so it is effectively useless. And that's probably
> why Apple's ALAC is usually bigger than FLAC for the same uncompressed
> audio (and why Apple still don't support FLAC in their products).
> 
> Stopping a pirate from encoding FLAC is similar to stopping a pirate
> from ripping a copy-protected CD: it's a challenge to be overcome, and
> it will probably take "them" less time to work it out than it
took "us"
> to build it. And "they" only need to work it out once. Which is
why all
> copy protection and DRM sucks, for everyone.

Lots of comments throughout this one...

On Jan 7, 2011, at 15:28, Declan Kelly wrote:
> On Fri, Jan 07, 2011 at 02:22:51PM -0800, brianw at sounds.wa.com wrote:
>> However, you should be aware that many modern producers use software
>> to create their music, and when the software stores sound clips in
>> MP3 format, what you end up with is music that sometimes looks like
>> MP3.
>
> I recently bought the double-CD "Influence" remaster by The Art
Of
> Noise
> and some rarer tracks were sourced from MP3 because that was all their
> archivist could find. Most of the reissue was direct from analogue  
> tapes
> so this wasn't a quick "shovelware" reissue job.
Very interesting.  I've purchased many CD titles with reissued  
material where they clearly did not have access to anything but vinyl  
for some tracks.  It was weird to hear the needle drop, with rumble  
throughout, especially since I has the original CD versions of the  
same tracks in my collection.  Also, the early days of Warp Records  
involved many artists who sent in demoes on analog tape, and there is  
obvious wow and flutter on these tracks, especially at the beginning  
before the tape was properly tensioned.

A while back, when HD audio was quite rare, I purchased some titles  
that were clearly low quality.  Some 24/88.2 and 24/96 material had  
nothing but noise above 20 kHz, but the audio content was clearly  
from CD sources.  I suppose the thought was that there was less noise  
in the audible range below 20 kHz.  Certain "HD" titles were  
atrocious, with obvious indications that the source was 44.1 kHz and  
the upper frequencies were clearly nothing more than aliased  
frequencies!  What's worse is that the aliased titles were samples on  
an HD audio site.  They clearly didn't get my money.

>> it just has to do with the software that was used to create the music
>> originally.
>
> A friend of mine recorded his band's last album on DCC in the mid  
> 1990s
> and released it on CD. It sounds horrible; the lossy compression of  
> DCC
> is even worse than MiniDisc's ATRAC. I'm sure this CD would fail
most
> FFT quality tests, as literally everyone who heard it (not just people
> with "golden ears" or good sound systems) complained about the  
> quality.
I have also worked with mastering of a compilation CD including some  
material recorded and mixed on DCC.  What's interesting in this one  
case is that the "music" was in the "noise" genre, and the
DCC was
used as an effect.  It could not handle the content and therefore  
ending up changing it.  What I found most interesting was that I had  
hired a professional studio in Seattle, and the owner actually stuck  
his head in the room for this one track.  He'd heard a lot of  
interesting music in the local scene, but nothing that messed up.

>> Nine Inch Nails who provide FLAC files.
>
> The initial free online release of NIN's The Slip had a problem  
> with the
> 24-bit version: it wasn't the full 24/96. Turns out it was a genuine
> oversight by the mastering lab (who used the 24/96 to master the  
> vinyl),
> and the true 24/96 files were reissued less than a week later.
> So even with the artist fully behind 24-bit FLAC, and doing almost all
> of the work in their own studio, things can still go wrong.
Thanks!  That's interesting to note.  I think that I ended up with  
the true 24/96 files, but I am curious: How do you tell whether you  
have the full 24/96 or not?  I have written software which can detect  
16-bit audio samples stored in a 24-bit file format.  Frequency  
analysis makes it obvious whether the content extends above 20 kHz.   
But I just want to make sure that there isn't another technique that  
I'm missing.

>> Some audio turns out smaller with ALAC, other audio turns out
>> smaller with FLAC.  Overall, the average performance is identical.
>
> I've been trying Flake (an alternative FLAC encoder; all it does is
> encode) recently and while it goes way faster than the reference FLAC
> encoder in terms of speed, the first encode I tried with it ended up
> larger than with the reference encoder, at all compression levels.
> The compression levels in Flake go up to 11, but past 9 or 10 there is
> no guarantee of full compatibility for playback.
The whole picture is a bit inconsistent.  If Flake is only an  
encoder, and compression levels above 8 are not guaranteed to be  
compatible, then what's the purpose?  If Flake cannot decode, then  
what good is it to create a file that no other decoder can handle?

>> Many tools run the
>> command-line flac utility behind the scenes.  Others use the FLAC
>> library directly.
>
> About a month ago I was setting up FLAC support for a friend on a
> Windows PC. Almost every GUI front end (based on the latest available
> download) that I tried was using an out of date version of the FLAC
> binary or library, and had the default compression set to -6 or lower.
> There is no excuse for not cranking up the compression level on any
> modern computer: surely you want to get the files as small as  
> possible?
Based on similar experiences, I refuse to use anything but the  
official FLAC command-line for all of my work.  The GUI front ends  
sometimes look convenient, but I don't trust the quality.  Besides,  
many of the GUI front ends are actually quite horrible from a UI  
perspective, so they seem to have no redeeming qualities.

>> I doubt there would be any professional
>> interest in changing things just for the sake of change or
"newness."
>
> The only "new" thing I want in FLAC is a fix for a minor bug in
the
> way
> the command line tool treats the end of line. If long filenames  
> push the
> "percent complete" past the screen width and you are encoding or
> testing
> a lot of files, you can end up with the longer file names having 20 or
> more lines on the screen output, and scrolling the previous files off
> the screen. My workaround is to use a longer scrollback in whatever
> terminal I'm using, or use GNU Screen to get scrollback on a text
> console.
Ok, despite the fact that I said FLAC should not be changed, I  
actually agree with you that the command-line could use some  
improvements and bug fixes.  Every time I see the long filenames  
issue, I worry that there is a problem, until I remember that this is  
a known issue and just ignore it.

One of my goals would be to add support for the Apple CAF format.   
This would actually serve a very important need. My FLAC recorder can  
create files that are up to 4 GB compressed, which is literally too  
large when uncompressed to WAV or AIFF.  Usually, I'm dealing with  
long recordings of multiple performances, so I can just tell FLAC to  
grab the first hour, or the last hour, and that's small enough to fit  
within the 2 GB or 4 GB limit.  In contrast, CAF has no limit on file  
length, so I could theoretically uncompressed a 4 GB FLAC into a 9 GB  
CAF without any problem.

I can't decide whether FLAC-to-CAF (and CAF-to-FLAC) should be  
developed into a new program based on the FLAC library, perhaps with  
a GUI, or if it should be incorporated into the standard flac command- 
line tool.  At the very least, if I ever get this working standalone,  
I would eventually want to roll the feature into the flac command- 
line for others to use.

Anyway, the flac command-line should be improved, even though the  
file format and library are probably just fine being left alone.

Brian Willoughby
Sound Consulting