This came out of the OggPCM discussion, but I think it needs to be addressed on
a wider scale.
Let's start here, 5 years ago..
http://lists.xiph.org/pipermail/vorbis-dev/2000-July/009513.html
(I included this email, below)
I emailed David (author of that email) and asked him to join this list.
I'm thinking, as I look at the problem, that surround sound needs to be
defined
_outside_ the audio codec stream. Comments are inappropriate, since we need to
know if our apps can support a stream based on packet0.
As Silvia mentioned recently, we've been lacking a multi-audio codec
handling
system for a long time. Wether that be for surround sound, prehaps with
different samplerate/samplesize, or for alternative languages, or for recorded
tracks which need to be mixed.
Let's begin with the above email and continue from there. If this is to be
outside of OggPCM, then it should be seperate from the thread for OggPCM beside
not encoding the data within it's header.
==================================================================================
Hi everyone,
Over the last two weeks or so, I've been thinking about how to add surround
sound to Ogg -- and more than that, to do it in the best way possible. With
this in mind, I started considering using Ambisonic surround sound. The
advantages of this format are considerable:
a) It was developed in the early to mid '70s, so the patents should
be expired by now.
b) It's scalable -- it can handle everything from mono, to stereo,
full horizontal surround, or full 360-degree spherical surround
(periphonic).
c) It's based on a sound mathematical foundation (unlike Dolby's
5.1 system), allowing you to calculate how it should be able
to perform.
d) Depending on how many channels you are willing to use (and the
source of your material) you can make the positioning of sound
in the sound field as accurate as you want. (First-order
Ambisonics requires four channels for full periphonic surround,
while second-order Ambisonics provide a larger 'sweet spot'
with
eight channels.) Since an Ogg stream can have up to 255 channels,
the format itself will not be a limitation.
e) It's compatible with existing decoder technology -- if your
player
does not support surround decoding, two of the channels can be
decoded as M/S stereo. (Once patent issues regarding M/S stereo
in digital audio encoding are worked out, that is... Monty thinks
that M/S stereo may have been patented by Fraunhofer for audio
compression uses.)
Many articles further describing the format itself are available at
www.ambisonic.net, but I'll try to give a brief overview here for those of
you who are unfamiliar with it.
Ambisonics is based on the principle of encoding spherical harmonic components
of a sound field into one or more audio channels, which allow reproduction of
that sound field later on with an array of speakers in a certain configuration.
The spherical harmonics it reproduces are similar in shape to the various
electron orbitals in an atom, if any of you are familiar with that. The first
component, the 'W' signal, reproduces the pressure of the signal. (This
would
be 0th-order Ambisonics, and gives you mono. This takes the shape of the
'S'
orbitals in an atom.) The next three signals are the directional components
of the signal, which give you the 2- or 3-D spatial information. These are
the X, Y, and Z signals. (These make up 1st-order Ambisonics, and have the
shape of the 'P' orbitals in an atom. If you decode only the W and Y
signals
through an M/S decoder, you have stereo. If you decode WXY, you have
horizontal surround. If you decode all four, you have periphonic surround.
Anyway, you get the picture.) The next four signals describe the curvature
of the signal (I think -- I don't understand this realm terribly well yet),
and have the shape of the 'D' orbitals of an atom. These make up the
signals
of 2nd-order Ambisonics, which will, if used, increase the accuracy of the
reproduced sound field considerably, and widen the 'sweet spot' by a
fair
amount. Higher-order Ambisonics are possible, but little research has been
done to this date. All existing Ambisonic recordings of actual events are
in first-order Ambisonic formats.
Now that I've tried to describe what the Ambisonic system is (hopefully I
didn't confuse you too much -- read some of the articles on the site I gave
earlier for more in-depth information), I'll describe more specifically what
I have in mind for the Ogg project.
First of all, though there are several formats currently in use for
distributing Ambisonic material (including one -- UHJ -- which is stereo
compatible) I propose that we use Ambisonic B-format (using the harmonics
directly -- we would have W, X, Y, etc. channels in the Ogg stream), as it
is the easiest to decode (if we used another format like UHJ, the player
would have to decode it to B-format before it could do more with it), is the
oldest (it's been used since the early '70s, so any patents should have
expired
YEARS ago), and it's the most flexible. What I would propose would be a
field
for each track that is marked Ambisonic, of one byte. Of this byte, the first
three bits would indicate which order of Ambisonics were being used, and the
last five bits would indicate which signal in the hierarchy this stream
represented. For example: (These are big-endian)
Component Order Signal
W 000 00000
X 001 00000
Y 001 00001
Z 001 00010
R 010 00000
etc.
One additional format which may become important as time goes by is G-format,
which is basically an Ambisonic signal pre-decoded for a standard DVD-type
5.1 speaker array. Once this begins to be used, we may want to incorporate
a G-format-to-B-format converter into the encoder so we're again working
with
B-format signals.
Even though no one anywhere is talking about using 7th-order Ambisonic signals,
this is a good way of future-proofing the format, should any such system be
developed. (If it was easier to implement, each of these fields could be a
whole byte.)
Once we have an Ambisonically-encoded Ogg stream, how do we play it? There
will be two ways. The first way is the way Ambisonic material has
traditionally been played -- the B-format signals are outputted directly, and
fed to an outboard decoder module. This offers maximum flexibility, as this
decoder could be a 128-speaker auditorium model. The drawbacks, however,
would be showstoppers if this were the only playback method. (The biggest
drawback is the need for everyone to have a decoder -- they aren't very
common,
so they're fairly expensive.)
The second way is the way I propose that most of us will play this -- software
decoding. The B-format components will be rematrixed in software into a
certain number of speaker feeds, which will then be outputted to your amplifier
and speaker array. You would create a configuration file (either manually or
with some sort of GUI) which would tell the decoder about your speaker array
(location, possibly frequency range as well), and then the decoder would make
the appropriate adjustments to the signals to reproduce the sound field as
accurately as possible.
At first, I was thinking that this could be done using the four-channel output
of a Sound Blaster Live (or similar card), feeding the 5.1 channel analog input
of a home theater receiver. If sound cards are developed which integrate AC-3
encoding (allowing you a virtual 5.1 outputs which are encoded by a chip into
an AC-3 or DTS stream), this could also be used. (We probably wouldn't be
able
to integrate AC-3 encoding directly into the player until the patent on it
expires, which won't be for quite a while. I doubt Dolby would give us the
free license to it we would need to do so -- especially since we would be
competing with Dolby for surround mindshare!) Hopefully, we would also be
able to drive two SBLive cards in parallel eventually, giving us 8-channels
of output, but that remains to be seen. (The driver doesn't even support
4-channel output yet...)
For now, however, independant of how fast the software decoding side of this
develops, the format can at least be defined to create a framework for future
work. One thing I've been keeping in mind during this process is that we
don't necessarily have to be using the Vorbis codec for this -- once the
Squish
lossless codec is available, that could be used for any or all components of an
Ambisonic signal.
At this point, several things need to happen.
a) The format need to be defined, preferably in such a way that it is
as flexible as possible, and allows non-surround-capable decoders
to decode the W-Y components as M/S stereo.
b) I'm going to ask Richard Furse, who has already written a
software
Ambisonic encoder and decoder, if he will relicense these tools
to us under the LGPL. If he agrees, this will make many things
much easier and faster.
c) The patent situation of Ambisonics in general, and particularly
newer Ambisonic developments (G-format pre-decoded 5.1, 2nd- and
higher-order Ambisonics in general, etc.) needs to be examined to
ensure that we know of any patent pitfalls so they can be worked
around.
d) Work towards 4-channel output on the SBLive (and maybe Aureal
Vortex, if they ever open their specs) drivers needs to proceed.
e) Someone should look into how easy or hard it will be to output
four channels under Windows -- we'll want to be able to for a
future Winamp plugin.
f) Once b) happens, or we write our own equivalents, functions will
need to be either added to libvorbis, or possibly a standalone
library, to work with Ambisonic material. (On the encoding side,
UHJ-encoded material, such as all the CDs put out by Nimbus in
England, will be decoded into B-format before it is encoded. A
G-format-to-B-format decoder may also be useful, if DVD-Audio
discs begin to be released in G-format. On the decoding side,
we will probably want to put the software Ambisonic decoding in
its own library to reduce bloat in the main plugin. The main
plugin could output raw B-format without this extra plugin, but
would output stereo by default.)
I realize that this is a LOT to digest at once, but hopefully this will
stimulate discussion about the future of surround support in the Ogg format,
and get things off and running.
I'm really excited about the whole Ogg project, and Vorbis in particular,
and
I think we can all look forward to an exciting future ahead... Watch out
Dolby and DTS, here we come!
David
--
David Carter ** dcarter at sigfs.org ** dcarter at visi.com
PGP Key 581CBE61: E07EE199C767C752 8A8B1A9F015BF2EA
Key available by finger or www.keyserver.net
==================================================================================
--
The recognition of individual possibility,
to allow each to be what she and he can be,
rests inherently upon the availability of knowledge;
The perpetuation of ignorance is the beginning of slavery.
from "Die Gedanken Sind Frei": Free Software and the Struggle for Free
Thought
by Eben Moglen, General council of the Free Software Foundation
