Boris Grozev wrote:> The ogg/opus draft describes[1] how to fill in gaps by generating
> zero-byte frames, but I do not understand how (and if) FEC can be used.
> Is this possible, and if so, what is the recommended way of doing it?
> Which component(s) would be responsible for handling FEC: the muxer
> (e.g. by extracting FEC data from a previous packet and inserting this
> instead of a zero-byte frame), the ogg player (by calling the opus
> decoder with decode_fec=1 whenever necessary), or the opus decoder itself?
>
> I failed to find any information about this specific topic. Please point
> me to relevant documentation if it exists.
I don't think we have any.
You could take a packet with FEC, extract the FEC data, and repackage it
as a normal SILK frame in the proper place, but this requires
re-encoding the indices with the entropy encoder (no actual
re-quantization or searches would need to be done, so no quality would
be lost). I am not aware of any code that exists to do this. Possibly
such code should live in libopus (so it could re-use the existing
encoder functions). While you're implementing that, you could also strip
out the FEC data from the original packets, meaning you would only store
FEC data in the saved file in places where there were actual losses. As
long as you are not planning to re-broadcast the stream over a lossy
channel later, that seems like a good idea. This approach also does not
require upgrading existing players. The downside is it requires
understanding the internals of an Opus packet.
Alternatively you could leave it up to the decoder. I did not add FEC
decoding support to libopusfile, but only because I wanted to get
something out the door, and it was not needed for the main use-case
(streaming radio over http). If someone wanted to add it there, it would
automatically mean a bunch of players would start handling this how you
want, but certainly not all of them (Firefox, VLC, anything libav/FFmpeg
based, etc., none of which, AFAIK, will look for FEC when they encounter
a PLC packet). The simple case (where both the PLC packet and the
following packet with FEC are on the same Ogg page) should be easy. The
case where there is a page break between them would require some
thinking to do properly, but should be doable.
I'm not sure where you draw the distinction between "decoder" and
"player". Certainly libopus does not have enough information to solve
this problem (it only sees one packet at a time, so it cannot look into
the future), and libopusfile does not export enough information for a
subsequent layer to fix up the loss after the fact, so I think
libopusfile (or the analog in other software stacks) is the right layer
to handle this if you want to do it from the decode side.