opus - Apr 2016 - Channel Mapping Family for Ambisonics

On Mon, Apr 25, 2016 at 9:32 AM, Timothy B. Terriberry
<tterribe at xiph.org> wrote:
> Jean-Marc Valin wrote:
>>
>> Would it make sense to allow an arbitrary number of channels and just
>> "truncate" the list of channels. For example, two-channel
ambisonics
>> would be W plus X and three-channel would be W, X and Y. The idea is
>> that you would get these mappings for free -- if there's any use
for
>> them anyway.
>
>
> Well, more specifically, the channel count is stored in the main header,
not
> in the channel mapping table, so it's possible to put any value in
there, up
> to 255 (instead of, for example, simply storing l). We should say what a
> decoder is expected to do in the case it encounters a stream with such a
> value. "Treat the stream as invalid," is a perfectly acceptable
option, as
> is Jean-Marc's, "pretend the missing channels are silent,"
but we have to
> pick one.
>
It would still be possible to send WYZ by sending 4 channels and
setting index 4 (X) to the "pure silence" value 255 in the channel
mapping array. I would consider it a special case optimization for an
ambisonic renderer to decide to simplify computation because X is
silent. In fact, most cases would not involve truncation but rather
"holes" in the mapping, ie WY_X instead of WYZX for horizontal only.

Restricting channel counts to (l + 1)^2 would allow us to include
rules for "hemispherical" ambisonics. These are sets of ambisonic
components with different shapes more suitable for encoding certain
content. An example is described in the ambix paper [1]. I need to
talk with some of the VR people here to figure out whether these
hemispherical encodings are likely to be used by anyone (at Google or
elsewhere). I could not find a commonly used set of basis functions
like the spherical harmonics used in ordinary ambisonics, so we may
want to keep things simple and avoid including them.

If we were to include hemispherical ambisonics, streams with 3, 6, 10,
etc channels could signal their usage.
>>> 1. Should I be more explicit about what "ambisonics" is,
what
>>> normalization is, and what each channel really means?
>>
>>
>> For these kinds of things, you can just use references.
>
>
> Yes, hopefully this is all well-described somewhere (at least enough for
> someone to implement it), so we don't have to reinvent the wheel.
>
I will include a reference to the Ambix paper, which describes
ambisonics, SN3D normalization, and ACN channel order.
>>> 2. Do I need to list the specific meaning of each possible channel
>>> count? For example, should I write "4 channels: First order
>>> ambisonics" similar to what is done for surround sound?
>>
>>
>> I'm not sure I understand what your asking here.
>
>
> I think he's just asking if he needs to make a bulleted list as we did
in
> 5.1.1.1 and 5.1.1.2 of the Ogg Opus draft. To which I think the answer is
> "no". Those are in lists because it was the easiest way to
present the
> information. When you have a parametric mapping, a description in terms of
> the parameters makes more sense.
Yes, I meant a bulleted list. Great, I'll stick with the parametric
description.
>>> 3. Should we define whether downmixing should occur when Ogg Opus
>>> players do not support the channel mapping? Since channel 1 is a
mono
>>> channel, it is always possible to play something reasonable.
>>
>>
>> It's not absolutely required, but it would indeed be nice if you
could
>> define a simple way that players can downmix. I guess mono is just
"take
>> the W channel", but maybe a reasonable stereo would be nice too.
>
>
> Well, there will always be players that don't support the channel
mapping at
> all, just because there's 4 years' worth of deployed software that
knows
> nothing about it. I think the actual question is, "Should we give
advice on
> how players with common speaker configurations (instead of a custom
> ambisonics rig or a virtual environment or whatever) can render the
stream?"
> If you want to see support in common media players like VLC, I think we
> should.
>
Okay, I will include mixing matrices for stereo and common surround setups.


I will talk with the VR people here about hemispherical ambisonics and
the mixing matrices, then follow up with a revised version of the
mapping description.

[1]
http://iem.kug.ac.at/fileadmin/media/iem/projects/2011/ambisonics11_nachbar_zotter_sontacchi_deleflie.pdf

-- 

Thanks,
Michael Graczyk

I've discussed hemispherical ambisonics and mixing matrices with a few
people. The consensus is that there is no set of hemispherical basis
functions common enough to warrant inclusion yet. We should force
channel counts to be values (l + 1)^2 for simplicity and to keep the
possibility open of including hemispherical bases should one ever
become popular.

As for mixing matrices, we are not confident in any choices for setups
beyond stereo. Although there have been papers and studies on
HOA->surround decoding, there are no readily available options for
orders besides  4. (HOA is higher order ambisonics, orders > 1).
Since these are only recommendations, we should be cautious and
provide only a stereo downmixing matrix. It looks like this would go
in 5.1.1.5? The matrix should be

 L = 0.5(W + Y)
 R = 0.5(W - Y)

and can be used for all orders (all channel counts >= 4). W is channel
1, Y is channel 2.

Here's the revised content. What do you guys think?

-----
Channel Mapping Family 2

Allowed numbers of channels: (1 + l)^2 for l = 1...15.

Explicitly 1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196,
225. Ambisonics from first to fifteenth order.


Each channel is assigned to an ambisonic component in Ambisonic
Channel Number (ACN) order. The ambisonic component with degree n and
ambisonic index m corresponds to channel (n * (n + 1) + m).

Channels are normalized with Schmidt Semi-Normalization (SN3D). In
SN3D, the spherical harmonic of degree n and index m is normalized
according to sqrt((2 - delta(m)) * ((l - m)! / (l + m)!)), where
delta(0) = 1 and delta(m) = 0 otherwise.

The interpretation of the ambisonics signal as well as the channel
order and normalization are described in [ambix].
----

and in section 5.1.1.5

-----
Implementations MAY use the matrix in Figures ? to implement
downmixing from multichannel files using Channel Mapping Family 2
(Section 5.1.1.?), which are known to give acceptable results for
stereo.


     /   \   /                  \ /  W  \
     | L |   | 0.5  0.5 0.0 ... | |  Y  |
     | R | = | 0.5 -0.5 0.0 ... | | ... |
     \   /   \                  / \ ... /
                   Figure ?
-----

Michael Graczyk wrote:
> As for mixing matrices, we are not confident in any choices for setups
> beyond stereo. Although there have been papers and studies on
Okay, better to give no advice than bad advice.
> provide only a stereo downmixing matrix. It looks like this would go
> in 5.1.1.5? The matrix should be
As a general point, it's too late to add anything to the soon-to-be RFC 
7845. It's too far along in the publication process. This would have to 
go in a separate document (which, other than having to add an abstract, 
brief introduction, and some other boilerplate sections, shouldn't be a 
big deal).
> Allowed numbers of channels: (1 + l)^2 for l = 1...15.
>
> Explicitly 1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196,
> 225. Ambisonics from first to fifteenth order.
You say l = 1...15 but the explicit list actually shows (l + 1) ranging 
from 1 to 15, meaning l ranges from 0 to 14.
> Each channel is assigned to an ambisonic component in Ambisonic
> Channel Number (ACN) order. The ambisonic component with degree n and
> ambisonic index m corresponds to channel (n * (n + 1) + m).
"Channel" is ambiguous. There are "internal" or
"encoded channels"
(signaled in each Opus packet using the stereo flag), "decoded
channels"
(based on the configuration applied to each decoder, as specified by the 
stream count and coupled stream count), and "output channels", which
are
what you wind up with after applying the channel mapping table. I assume 
you mean "output channels" here, and should say so.

It's also probably a good idea to explicitly say that you use the same 
channel mapping table format as channel mapping families 1 and 255. At 
least, I'm assuming you do. I'm also assuming you don't plan to
support
Ambix's "extended format" with its adaptor matrix.