opus - Dec 2013 - Extension Packets?

I have a project I'm investigating.  The goal is to basically add lossless
extensions to Opus.  You have an Opus stream with standard packets, but
interwoven in there are extension packets that contain the residuals.
 Ideally, compliant decoders play the stream back and ignore the extension
packets.  This (hopefully) makes the "lossless" stream compatible with
existing players.  Specialized decoders decode the standard Opus (CELT)
packets, the residual extension packets, and then combines them to
reproduce a bit-exact stream to the original PCM.

Now those of you familiar with how Opus works will readily identify some
issues with why this might be completely impossible.  Those problems are to
be explored later.  First, I need to know that current players can even
handle streams with extension packets.  The problem is that, currently,
these packets are completely undefined.

My current idea is to have a packet with zero frames (illegal) and then put
the extension data in that (illegal) packet's padding area.  As I remember
it, this is against the Opus RFC for at least two reasons:

1. An Opus packet must have at least one frame.
2. A packet's padding may not contain anything but null bytes.

However, the spec also says packets that violate this should simply be
ignored, which is what I'm hoping players do.  Another question is how
streams with extension packets get muxed into containers like Matroska.
 Will mkvtools drop the extension packets?

So how should I go about structuring these things?  I assume somewhere in
the padding I need to place an identifier in there so that later decoders
don't confuse my extension packets with other types.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.xiph.org/pipermail/opus/attachments/20131212/bb6039f3/attachment.htm

Hi,

We have already pointed out multiple times in the HydrogenAudio thread (
http://www.hydrogenaudio.org/forums/index.php?showtopic=99653 ) why we
think this is a bad idea. These include (off the top of my head):

1) The Opus decoder does not have a bit-exact definition
2) It would require a bit-exact resampler too (the one from opus-tools
isn't bit-exact even in fixed-point)
3) Opus has features that improve the quality by making the residual
larger, so you can't have both efficient lossless and efficient
non-lossless.
4) It would result in something very complex for little gain over (e.g.)
simply packing an alternate FLAC stream next to the Opus stream in the
Ogg file.
5) Even if it was a good idea, such kind of extension would need to go
through the IETF.

	Jean-Marc

On 12/13/2013 12:34 AM, William Swartzendruber wrote:
> I have a project I'm investigating.  The goal is to basically add
> lossless extensions to Opus.  You have an Opus stream with standard
> packets, but interwoven in there are extension packets that contain the
> residuals.  Ideally, compliant decoders play the stream back and ignore
> the extension packets.  This (hopefully) makes the "lossless"
stream
> compatible with existing players.  Specialized decoders decode the
> standard Opus (CELT) packets, the residual extension packets, and then
> combines them to reproduce a bit-exact stream to the original PCM.
> 
> Now those of you familiar with how Opus works will readily identify some
> issues with why this might be completely impossible.  Those problems are
> to be explored later.  First, I need to know that current players can
> even handle streams with extension packets.  The problem is that,
> currently, these packets are completely undefined.
> 
> My current idea is to have a packet with zero frames (illegal) and then
> put the extension data in that (illegal) packet's padding area.  As I
> remember it, this is against the Opus RFC for at least two reasons:
> 
> 1. An Opus packet must have at least one frame.
> 2. A packet's padding may not contain anything but null bytes.
> 
> However, the spec also says packets that violate this should simply be
> ignored, which is what I'm hoping players do.  Another question is how
> streams with extension packets get muxed into containers like Matroska.
>  Will mkvtools drop the extension packets?
> 
> So how should I go about structuring these things?  I assume somewhere
> in the padding I need to place an identifier in there so that later
> decoders don't confuse my extension packets with other types.
> 
> 
> 
> 
> _______________________________________________
> opus mailing list
> opus at xiph.org
> http://lists.xiph.org/mailman/listinfo/opus
>

These are all good and valid points.  For #1 through #4, I intend to
address these by implementing my own CELT encoder and decoder.  Maybe that
will enable me to solve these issues, maybe it won't.  I don't know
because
I don't truly understand the nature of the problems (yet).  The most likely
outcome is that solving these issues would require such overhead in the
extension packets that it won't even be worth it.  But I won't know
until I
try.

But solving the first four issues is completely irrelevant if players can't
handle these packets.

In the HA thread, you expressed concern that I would be creating a rogue
extension.  This is the last thing I want to do.  To me, it would be neat
if extension packets as a concept on their own were standardized.  My
current idea is like this:

Have a Type-III packet that signals zero frames followed by the padding
length descriptor.  Then within that padding, start with some sort of
unique identifier that runs until the next null byte, then have the
arbitrary payload.

Current decoders should see a frameless Type-III packet and simply discard
it.  Later decoders would simply know to check for those packets, and then
check to see if the unique identifier is anything they're interested in.


On Thu, Dec 12, 2013 at 9:53 PM, Jean-Marc Valin <jmvalin at jmvalin.ca>
wrote:
> Hi,
>
> We have already pointed out multiple times in the HydrogenAudio thread (
> http://www.hydrogenaudio.org/forums/index.php?showtopic=99653 ) why we
> think this is a bad idea. These include (off the top of my head):
>
> 1) The Opus decoder does not have a bit-exact definition
> 2) It would require a bit-exact resampler too (the one from opus-tools
> isn't bit-exact even in fixed-point)
> 3) Opus has features that improve the quality by making the residual
> larger, so you can't have both efficient lossless and efficient
> non-lossless.
> 4) It would result in something very complex for little gain over (e.g.)
> simply packing an alternate FLAC stream next to the Opus stream in the
> Ogg file.
> 5) Even if it was a good idea, such kind of extension would need to go
> through the IETF.
>
>         Jean-Marc
>
> On 12/13/2013 12:34 AM, William Swartzendruber wrote:
> > I have a project I'm investigating.  The goal is to basically add
> > lossless extensions to Opus.  You have an Opus stream with standard
> > packets, but interwoven in there are extension packets that contain
the
> > residuals.  Ideally, compliant decoders play the stream back and
ignore
> > the extension packets.  This (hopefully) makes the
"lossless" stream
> > compatible with existing players.  Specialized decoders decode the
> > standard Opus (CELT) packets, the residual extension packets, and then
> > combines them to reproduce a bit-exact stream to the original PCM.
> >
> > Now those of you familiar with how Opus works will readily identify
some
> > issues with why this might be completely impossible.  Those problems
are
> > to be explored later.  First, I need to know that current players can
> > even handle streams with extension packets.  The problem is that,
> > currently, these packets are completely undefined.
> >
> > My current idea is to have a packet with zero frames (illegal) and
then
> > put the extension data in that (illegal) packet's padding area. 
As I
> > remember it, this is against the Opus RFC for at least two reasons:
> >
> > 1. An Opus packet must have at least one frame.
> > 2. A packet's padding may not contain anything but null bytes.
> >
> > However, the spec also says packets that violate this should simply be
> > ignored, which is what I'm hoping players do.  Another question is
how
> > streams with extension packets get muxed into containers like
Matroska.
> >  Will mkvtools drop the extension packets?
> >
> > So how should I go about structuring these things?  I assume somewhere
> > in the padding I need to place an identifier in there so that later
> > decoders don't confuse my extension packets with other types.
> >
> >
> >
> >
> > _______________________________________________
> > opus mailing list
> > opus at xiph.org
> > http://lists.xiph.org/mailman/listinfo/opus
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.xiph.org/pipermail/opus/attachments/20131212/54475763/attachment.htm