Hello all I am using libFLAC in a corner application, compressing *a lot* of small signals. First is a general question: in our application we have signals in range 5-10 MHz, potentially 40MHz! Is there any potential problem with that?? The mac sample rate is limited in flac, but it doesn't really seem to be a problem. The output is stored as blob in a sqlite database, it *never *needs to be a valid audio file outside our application. In my tests, the signals are compressed very well, much better than general compression libraries like zlib, zstd, etc. Now other small issues; I also made some tickets about them, but I thought asking here might be better. 1. I would like to avoid saving vorbis comment, by default ~40 bytes. Right now the only option is to modify stream_encoder.c, see "metadata_has_vorbis_comment". 2. Speed is very important, therefore I would like to reuse an encoder without re-initializing everything. Ideally I would like 2 (exported) functions: "flush" and "restart". "Flush" is self-explanatory, should properly end the encoding. I could split myself "flush" from "finish", it looks relatively simple. "Restart" should keep all current settings, generate a new stream header and clear everything for encoding a new signal. It' clear that current settings, re-creating windows, cpu-dependent functions, etc could be kept around. I was not quickly able to extract all the necessary initialization from "init_stream_internal_" into a new "FLAC__stream_encoder_restart" function. Regards, Gabriel Corneanu -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.xiph.org/pipermail/flac-dev/attachments/20180201/b4579a19/attachment.html>
Gabriel Corneanu wrote:> I am using libFLAC in a corner application, compressing *a lot* of small > signals.You are of course aware that this is *not* what FLAC was designed for,> First is a general question: in our application we have signals in range > 5-10 MHz, potentially 40MHz! Is there any potential problem with that??No idea whatsoever. It really depends on the signal. FLAC is *designed* and optimised for compressing audio signals, specifically music. For cases where the FLAC compressed signal is does not actually compress well (on example is white noise) the audio is included uncompressed.> The > mac sample rate is limited in flac, but it doesn't really seem to be a > problem.FLAC is the Free Lossless **AUDIO** codec.> The output is stored as blob in a sqlite database, it *never *needs to be a > valid audio file outside our application. > In my tests, the signals are compressed very well, much better than general > compression libraries like zlib, zstd, etc.So regardless of the higher sample rate your signals seem to be enough like audio for FLAC to compress them well.> Now other small issues; I also made some tickets about them, but I thought > asking here might be better. > > 1. I would like to avoid saving vorbis comment, by default ~40 bytes. Right > now the only option is to modify stream_encoder.c, see > "metadata_has_vorbis_comment".Are you using the FLAC library directly or the flac command line program? If you are using the flac command line program, please feel free to hack that to your hearts content (as long as you abide by the very libreral license).> 2. Speed is very important, therefore I would like to reuse an encoder > without re-initializing everything. > Ideally I would like 2 (exported) functions: "flush" and "restart". > "Flush" is self-explanatory, should properly end the encoding. I could > split myself "flush" from "finish", it looks relatively simple. > "Restart" should keep all current settings, generate a new stream header > and clear everything for encoding a new signal. > It' clear that current settings, re-creating windows, cpu-dependent > functions, etc could be kept around. > I was not quickly able to extract all the necessary initialization from > "init_stream_internal_" into a new "FLAC__stream_encoder_restart" function.I would theoretically accept a patch that allows direct access to the functionality you seek, as long as it doesn't change the existing API. Erik -- ---------------------------------------------------------------------- Erik de Castro Lopo http://www.mega-nerd.com/
Gabriel, metadata_has_vorbis_comment is a FLAC__bool which defaults to false. There should be no reason to modify stream_encoder.c, but just modify the caller. The following command: metaflac —remove —block-type=VORBIS_COMMENT —don’t-use-padding … will remove Vorbis comments from existing files, so is must be legal without modifying the library. metaflac can clearly create a new FLAC file without the Vorbis comment. I use the flac command-line, and I never get Vorbis comments in the files that I create. Perhaps you are using another tool which assumes Vorbis comments are needed. The FLAC algorithm is not dependent upon sample rate. AIFF has an 80-bit floating point type for sample rate, so it should be able to handle 40 MHz. I assume that any AIFF can be converted to FLAC losslessly, but I have not tested whether certain sample rates are rejected. FLAC itself only supports sample rates up to 655,350 Hz, so you may have a problem there unless you “lie” about the sample rate when creating the file. Maybe you could just establish a private convention to divide the sample rate by 100 to make yours fit. 40 MHz would map to 400 kHz, 10 MHz would map to 100 kHz, and 5 MHz would map to 50 kHz. You’re probably asking for trouble if you try to reuse an encoder. It seems like there would always be some risk that details from the previous file would bleed through into the next. Have you benchmarked allocation and initialization? Is it really that slow? In order to reuse an encoder, you’ll need to overwrite all state variables, and I don’t see how that could be much faster than simply allocating them anew. Perhaps you could allocate groups of encoders at once, if that would speed the process. On Feb 1, 2018, at 4:29 AM, Gabriel Corneanu <gabrielcorneanu at gmail.com> wrote:> Hello all > > I am using libFLAC in a corner application, compressing a lot of small signals. > First is a general question: in our application we have signals in range 5-10 MHz, potentially 40MHz! Is there any potential problem with that?? The mac sample rate is limited in flac, but it doesn't really seem to be a problem. > The output is stored as blob in a sqlite database, it never needs to be a valid audio file outside our application. > In my tests, the signals are compressed very well, much better than general compression libraries like zlib, zstd, etc. > > Now other small issues; I also made some tickets about them, but I thought asking here might be better. > > 1. I would like to avoid saving vorbis comment, by default ~40 bytes. Right now the only option is to modify stream_encoder.c, see "metadata_has_vorbis_comment". > > 2. Speed is very important, therefore I would like to reuse an encoder without re-initializing everything. > Ideally I would like 2 (exported) functions: "flush" and "restart". > "Flush" is self-explanatory, should properly end the encoding. I could split myself "flush" from "finish", it looks relatively simple. > "Restart" should keep all current settings, generate a new stream header and clear everything for encoding a new signal. > It' clear that current settings, re-creating windows, cpu-dependent functions, etc could be kept around. > I was not quickly able to extract all the necessary initialization from "init_stream_internal_" into a new "FLAC__stream_encoder_restart" function. > > Regards, > Gabriel Corneanu
Correction, the flac command-line does create a 40-byte Vorbis comment by default. I just never noticed it before. I’ve been using —no-padding all these years for minimal file sizes without realizing that I could save another 40 bytes. Anyway, since metaflac can remove the Vorbis comment using the standard library, then you should be able to create a solution without modifying libFLAC. Brian On Feb 4, 2018, at 12:43 AM, Brian Willoughby <brianw at audiobanshee.com> wrote:> Gabriel, > > metadata_has_vorbis_comment is a FLAC__bool which defaults to false. There should be no reason to modify stream_encoder.c, but just modify the caller. > > The following command: > > metaflac —remove —block-type=VORBIS_COMMENT —don’t-use-padding > > … will remove Vorbis comments from existing files, so is must be legal without modifying the library. metaflac can clearly create a new FLAC file without the Vorbis comment. > > I use the flac command-line, and I never get Vorbis comments in the files that I create. Perhaps you are using another tool which assumes Vorbis comments are needed. > > > The FLAC algorithm is not dependent upon sample rate. AIFF has an 80-bit floating point type for sample rate, so it should be able to handle 40 MHz. I assume that any AIFF can be converted to FLAC losslessly, but I have not tested whether certain sample rates are rejected. FLAC itself only supports sample rates up to 655,350 Hz, so you may have a problem there unless you “lie” about the sample rate when creating the file. Maybe you could just establish a private convention to divide the sample rate by 100 to make yours fit. 40 MHz would map to 400 kHz, 10 MHz would map to 100 kHz, and 5 MHz would map to 50 kHz. > > > You’re probably asking for trouble if you try to reuse an encoder. It seems like there would always be some risk that details from the previous file would bleed through into the next. Have you benchmarked allocation and initialization? Is it really that slow? In order to reuse an encoder, you’ll need to overwrite all state variables, and I don’t see how that could be much faster than simply allocating them anew. Perhaps you could allocate groups of encoders at once, if that would speed the process. > > > On Feb 1, 2018, at 4:29 AM, Gabriel Corneanu <gabrielcorneanu at gmail.com> wrote: >> Hello all >> >> I am using libFLAC in a corner application, compressing a lot of small signals. >> First is a general question: in our application we have signals in range 5-10 MHz, potentially 40MHz! Is there any potential problem with that?? The mac sample rate is limited in flac, but it doesn't really seem to be a problem. >> The output is stored as blob in a sqlite database, it never needs to be a valid audio file outside our application. >> In my tests, the signals are compressed very well, much better than general compression libraries like zlib, zstd, etc. >> >> Now other small issues; I also made some tickets about them, but I thought asking here might be better. >> >> 1. I would like to avoid saving vorbis comment, by default ~40 bytes. Right now the only option is to modify stream_encoder.c, see "metadata_has_vorbis_comment". >> >> 2. Speed is very important, therefore I would like to reuse an encoder without re-initializing everything. >> Ideally I would like 2 (exported) functions: "flush" and "restart". >> "Flush" is self-explanatory, should properly end the encoding. I could split myself "flush" from "finish", it looks relatively simple. >> "Restart" should keep all current settings, generate a new stream header and clear everything for encoding a new signal. >> It' clear that current settings, re-creating windows, cpu-dependent functions, etc could be kept around. >> I was not quickly able to extract all the necessary initialization from "init_stream_internal_" into a new "FLAC__stream_encoder_restart" function. >> >> Regards, >> Gabriel Corneanu