I am looking to losslessly archive a lot of music and share it over a
network. The following stipulations lead me to an interesting conclusion:
1. Almost no programs can read or write flac files directly.
2. Almost all programs can read and write wav files directly.
3. The cost of encoding flac files is fairly low
4. The cost of decoding them is even lower
5. These files would be read much more often than they would
be written.
So I was wondering if flac could be incorporated into the filesystem
driver so that any file ending in .wav would be transparently encoded
into flac. I don't know if this could be done at the vfs level so that it
could be done to all filesystems simultaneously, or if individual
filesystems would need to be modified.
Everyone has heard of or experienced Stacker or DoubleSpace woes, and may
wonder why I would actually encourage this kind of thing. There are a few
reasons why I find this sane nevertheless:
1. The overhead of decoding is so low that reduced disk I/O may even make
reads faster than not compressing [0].
2. It's entirely opt-in -- if you don't want transparent compression
to happen, just name your file sound.wave or sound.WAV or sound.notwav
However, I have limited expertise. I'd like to know your thoughts on this
idea -- especially if you're familiar with filesystem code. I don't know
enough to answer questions like "when would compression happen?" or
"what
about concurrent reads?" .
On an unrelated note, have you considered the possibility I mentioned
several months ago of making it possible to write the STREAMINFO block
without knowing anything about the format? I would link to the original
message in the archive to refresh your memory, but geocrawler is down for
nightly maintenance.
Joshua
[0] I made that claim up, but I hope it's true. In any case, the reduced
I/O should at least help compensate for the overhead of having to
decompress on-the-fly.
--
Joshua Haberman <joshua@haberman.com>
Hi Matt, thanks for the reply. * Matt Zimmerman (mdz@debian.org) wrote:> On Tue, Oct 30, 2001 at 11:39:11PM -0800, Joshua Haberman wrote: > > > So I was wondering if flac could be incorporated into the filesystem > > driver so that any file ending in .wav would be transparently encoded > > into flac. I don't know if this could be done at the vfs level so that > > it could be done to all filesystems simultaneously, or if individual > > filesystems would need to be modified. > > If you really want this, the right place to implement it is in > userspace, not in the kernel. Take a look at libbz2 and zlib for > examples of how this can be done. You could write a FLACopen routine > like gzopen, which would transparently encode/decode from/to WAV format. > This would probably go in the FLAC library, and make it trivial to add > FLAC support to many WAV-supporting programs (those which don't use > mmap).This doesn't give me the transparency I desire.> If you wanted transparency, you would write a shared library which would > override open(), and perhaps read(), etc. and do its magic there. > fakeroot, esddsp, and dmalloc are examples of how this is done. > However, in this case, you'd want to name all of your FLAC files with a > .wav extension so that the programs in question would open them. The > right thing to do is to add FLAC support to these programs, and the > above technique might make that easier.I hadn't thought of that. I like how this lifts me of the burden of having to muck around in kernel code. However I dislike how flacs are deceptively given .wav extensions, which will mess up programs that are linked statically or somehow run without LD_PRELOAD set correctly. I disagree that adding flac support to all programs is "the right thing to do," simply because it's not necessary or realistic. In my scenerio, people will be accessing the music using many different programs and OSs, and there's no reason why each one should be required to learn about a different format when it's conceptually identical to reading a straight wav file. As someone who's doing development on an application involving digital audio, I can tell you that it's a pain spending time supporting umpteen different formats when each of them has a command-line encoder that understands wav just fine (but alas, that's what users want...). I really appreciate the suggestion to LD_PRELOAD, thanks. Joshua -- Joshua Haberman <joshua@haberman.com>
On Tue, Oct 30, 2001 at 11:39:11PM -0800, Joshua Haberman wrote:> So I was wondering if flac could be incorporated into the filesystem > driver so that any file ending in .wav would be transparently encoded > into flac. I don't know if this could be done at the vfs level so that > it could be done to all filesystems simultaneously, or if individual > filesystems would need to be modified.If you really want this, the right place to implement it is in userspace, not in the kernel. Take a look at libbz2 and zlib for examples of how this can be done. You could write a FLACopen routine like gzopen, which would transparently encode/decode from/to WAV format. This would probably go in the FLAC library, and make it trivial to add FLAC support to many WAV-supporting programs (those which don't use mmap). If you wanted transparency, you would write a shared library which would override open(), and perhaps read(), etc. and do its magic there. fakeroot, esddsp, and dmalloc are examples of how this is done. However, in this case, you'd want to name all of your FLAC files with a .wav extension so that the programs in question would open them. The right thing to do is to add FLAC support to these programs, and the above technique might make that easier. -- - mdz