opus - Nov 2008 - adding celt support to netjack some questions.

hi.

i am currently adding celt support to netjack.
very nice to see a free low-latency codec :)

i currently dont require robustness against packet loss,
because the sync code of netjack does not handle packet loss very
gracefully. how much bandwidth is wasted for this feature ?

is it sensible, to have the data downsampled berfore encoding , in 
order to reduce bandwidth ? i suspect that just using celt to do this
is more sensible, and leaves me with higher quality of the signal.

the signal data is obtained from individual jack ports.
i would need one additional step to make the frames interleaved.
how much bandwidth would i save if i was using one encoder, for
all channels, instead of n encoders for n channels ?
considering the signals are not really correlated.

there might be transmission of several stereo or otherwise
correlated signals though. But i would be happier with an API
that supported individual buffers for each channel.

can i hook up the decoder to an already running stream ?
netjacks protocol is stateless. and i was using downsampling and
bit rate reduction to do compression only. which are stateless
things to do.

at first glance i did not see a function which tells me the
latency of the encoding/decoding (its the overlap i guess)
i need this to make the latency compensation code work.

how RT-safe is the code ?
does it use malloc during encoding/decoding ?


well... i will make the code use several encoders for now, and await
some responses.



-- 
torben Hohn

On Mon, Nov 24, 2008 at 3:33 PM,  <torbenh at gmx.de>
wrote:
> i am currently adding celt support to netjack.
> very nice to see a free low-latency codec :)
Greetings. I have written a yet-not-officially released tool which
interconnects multiple Jack servers using CELT.  It would be released
already but I got off on a tangent adding the ability to rewire the
remote jack system. (I.e. so you can say plug Jack_B mic_l to Jack_A
mixer_in_4 and it would handle all the routing on both sides to pass
through the interconnect.)

So, in addition to being a occasional CELT developer, I may have some
helpful thoughts for your application.
> i currently dont require robustness against packet loss,
> because the sync code of netjack does not handle packet loss very
> gracefully. how much bandwidth is wasted for this feature ?
It's inherent in the format and not deactiavtable. It costs only a
very small amount.  You want it anyways, more below.
> is it sensible, to have the data downsampled berfore encoding , in
> order to reduce bandwidth ? i suspect that just using celt to do this
> is more sensible, and leaves me with higher quality of the signal.
The CELT library officially supports integer sample rates between
32,000 Hz and 64,000 Hz.  If your jack is running at higher speeds you
will need to resample externally to CELT.

I do not expect CELT to ever truly support higher sampling rates, as
humans can not (safely) hear above ~22kHz, fruitcake audiophile
be-damned.   The correct way to support 88 and 96kHz is to resample to
44.1kHz or 48kHz.  The proper way to handle 44/48kHz is to simply run
CELT at the correct rate.
> the signal data is obtained from individual jack ports.
> i would need one additional step to make the frames interleaved.
> how much bandwidth would i save if i was using one encoder, for
> all channels, instead of n encoders for n channels ?
> considering the signals are not really correlated.
You should run N encoders. Pack their output into a single packet. You
will save considerable bandwidth from the packing (At typical CELT
latencies IP+UDP+etc overhead is significant).
> there might be transmission of several stereo or otherwise
> correlated signals though.
The CELT encoder currently supports stereo and mono modes. The stereo
mode is incomplete.  In my own app I have a separate add stereo add
mono mode. I think at this point it's overkill.
> But i would be happier with an API
> that supported individual buffers for each channel.
By buffers what exactly do you mean?  You'll suffer significant
overhead if you don't share packets. :)
> can i hook up the decoder to an already running stream ?
Yes. This is a property of the robusness to packet loss. It will take
'several' packets to synchronize and produce decent quality.  Note:
CELT never resyncronizes perfectly. So if you compare two decoders one
which started before the encoder and one which started after their
output will probably never be identical. The difference should not be
perceptible in any case.
> netjacks protocol is stateless. and i was using downsampling and
> bit rate reduction to do compression only. which are stateless
> things to do.
As you've seen, CELT is not stateless. But the robusness to loss means
that the state is soft? you can resume at any point.
> at first glance i did not see a function which tells me the
> latency of the encoding/decoding (its the overlap i guess)
> i need this to make the latency compensation code work.
CELT_GET_LOOKAHEAD should tell you the additional latency beyond a
single frame. but CELT delay is always 1.5x the frame_size today, I do
not expect this to change.

You can select any even frame size, but power-of-two are recommended
(sizes with large prime factors have reduced quality right now).
Since you should probably make the CELT frame size either be equal to
or an integer factor of the Jack frame size in order to reduce latency
sticking to power-of-two CELT frames should be acceptable for your
purposes.
> how RT-safe is the code ?
> does it use malloc during encoding/decoding ?
CELT never mallocs during encode and decode. It mallocs only during
mode setup or the creation of an encoder/decoder.

The CELT library is setup to make it fairly easy to replace malloc
with an RT-safer one at compile time for even these calls. I'd like to
make it easier to do at runtime, but I don't see an obvious way to do
it without making a mess of the API.

I have done extensive profiling of the worst case encode and decode
computational load. Nanosecond precision executing timing on core2 RT
patched linux is part of my regular test suite.  There should be
surprises waiting for you. Low complexity mode results in both lower
average CPU consumption (roughly 50% for the encoder) and something
like a ~20% reduction in worst/average ratio (from memory), but I
doubt that full complexity CELT will result in serious CPU constraints
in your usage.
> well... i will make the code use several encoders for now, and await
> some responses.
Sounds good.

On Mon, Nov 24, 2008 at 4:21 PM, Gregory Maxwell <greg at xiph.org>
wrote:
> Nanosecond precision executing timing on core2 RT
> patched linux is part of my regular test suite.  There should be
> surprises waiting for you.
Er. No surprises.

(And everywhere else where I said something foolish, feel free to
menially substitute what I meant to say)

On Mon, Nov 24, 2008 at 7:49 PM, Jean-Marc Valin
<jean-marc.valin at usherbrooke.ca> wrote:
[snip]
> Actually, support for > 64 kHz would be relatively easy to do, although
> there would be no point in doing it other than convenience since the
> encoding would still stop at 20 kHz.
I expect that (2:1) resampling would be computationally cheaper than
running CELT at a higher rate.  Perhaps we should consider including
an internal resampler?
>> The CELT encoder currently supports stereo and mono modes. The stereo
>> mode is incomplete.  In my own app I have a separate add stereo add
>> mono mode. I think at this point it's overkill.
>
> Could you translate that to English? :-)
Sorry.  I mangled that in editing.

I have [add stereo channel] and [add mono channel] buttons in the app
and I create jack ports with L/R appended names in the stereo case.
You could support both channel types but I think that it's currently
not worth the effort (and more over, user interface complexity) even
in case where your audio is correlated.
>> CELT_GET_LOOKAHEAD should tell you the additional latency beyond a
>> single frame. but CELT delay is always 1.5x the frame_size today, I do
>> not expect this to change.
>
> Not only it's likely to change, but it never was the case. The delay is
> often 1.5x the frame size, but not for all frame sizes. For instance, at
> 512-sample frames, the look-ahead is 128 samples, so the total delay is
> 1.25x frame size.
Sorry, JM is right. Worse, I was the last person to edit the code
which selects the overlap.  I was having a blonde moment during my
reply.
>> You can select any even frame size, but power-of-two are recommended
>> (sizes with large prime factors have reduced quality right now).
>> Since you should probably make the CELT frame size either be equal to
>> or an integer factor of the Jack frame size in order to reduce latency
>> sticking to power-of-two CELT frames should be acceptable for your
>> purposes.
>
> Actually, I think you're still OK if you have a factor of 3 or 5 in the
> frame size, but large prime factors are indeed bad -- both for quality
> and for performance.
Right. Doesn't everyone know that they should read what I mean and not
what I say? ? Thank you for your corrections.

On Mon, Nov 24, 2008 at 08:04:33PM -0500, Gregory Maxwell
wrote:
> On Mon, Nov 24, 2008 at 7:49 PM, Jean-Marc Valin
> <jean-marc.valin at usherbrooke.ca> wrote:
> [snip]
> > Actually, support for > 64 kHz would be relatively easy to do,
although
> > there would be no point in doing it other than convenience since the
> > encoding would still stop at 20 kHz.
> 
> I expect that (2:1) resampling would be computationally cheaper than
> running CELT at a higher rate.  Perhaps we should consider including
> an internal resampler?
ok. i overlooked that the minimum rate was 32kHz.
for big samplerates i will do the downsampling using libsamplerate then.

it already works like a charm.
nice work guys :)
> 
> >> The CELT encoder currently supports stereo and mono modes. The
stereo
> >> mode is incomplete.  In my own app I have a separate add stereo
add
> >> mono mode. I think at this point it's overkill.
> >
> > Could you translate that to English? :-)
> 
> Sorry.  I mangled that in editing.
> 
> I have [add stereo channel] and [add mono channel] buttons in the app
> and I create jack ports with L/R appended names in the stereo case.
> You could support both channel types but I think that it's currently
> not worth the effort (and more over, user interface complexity) even
> in case where your audio is correlated.
ok. so i stick with my N encoders code.
> 
> >> CELT_GET_LOOKAHEAD should tell you the additional latency beyond a
> >> single frame. but CELT delay is always 1.5x the frame_size today,
I do
> >> not expect this to change.
> >
> > Not only it's likely to change, but it never was the case. The
delay is
> > often 1.5x the frame size, but not for all frame sizes. For instance,
at
> > 512-sample frames, the look-ahead is 128 samples, so the total delay
is
> > 1.25x frame size.
> 
> Sorry, JM is right. Worse, I was the last person to edit the code
> which selects the overlap.  I was having a blonde moment during my
> reply.
ok... i guess i will implement this later.
is support for more then 512 sample blocks planned ?
i am putting one jack period of data into one packet.
so currently it only works with a period of <=512.
i could split that myself. but i dont see why the API should not handle
bigger blocks. and just hide the splitting to smaller blocks.

not sure if this would gain something quality wise.
the property netjack requires from the codec is that it has fixed
in:out ratio. so i guess celt is my only choice. and considering
how good this already works. will remain my choice ;)
> 
> >> You can select any even frame size, but power-of-two are
recommended
> >> (sizes with large prime factors have reduced quality right now).
> >> Since you should probably make the CELT frame size either be equal
to
> >> or an integer factor of the Jack frame size in order to reduce
latency
> >> sticking to power-of-two CELT frames should be acceptable for your
> >> purposes.
> >
> > Actually, I think you're still OK if you have a factor of 3 or 5
in the
> > frame size, but large prime factors are indeed bad -- both for quality
> > and for performance.
> 
> Right. Doesn't everyone know that they should read what I mean and not
> what I say? ??? Thank you for your corrections.
thanks for the prompt reply.
are you interested in the patch ? i can post it here, if you like.

needs some cleanup until it gets committed into jack-svn but i expect
that to happen soon.

i need to fix alsa_in and alsa_out now, because they dont handle the big
timing jitter of an internet connection gracefully.

-- 
torben Hohn

Hi all,
> ok... i guess i will implement this later.
> is support for more then 512 sample blocks planned ?
> i am putting one jack period of data into one packet.
> so currently it only works with a period of <=512.
> i could split that myself. but i dont see why the API should not handle
> bigger blocks. and just hide the splitting to smaller blocks.
+1 ---  I do something similar to netjack (currently called
"Soundjack") in order to play live music on a network (
https://www.itm.uni-luebeck.de/users/carot/TMT08.pdf ).

My settings vary from 1024 down to 128 sample-blocks (depending on the current
network situation). CELT works excellent in that context but it would be nice to
have 1024 samples supported as well.
  
> i need to fix alsa_in and alsa_out now, because they dont handle the big
> timing jitter of an internet connection gracefully.
I`d be curious to know what you are approaching to change in that context. What
amount of packets do you normally buffer on the network input in order to cope
with the jitter ?

Best

-- A l e x



-- 
Dipl.-Ing. Alexander Car?t
PhD Candidate
Email : Alexander at Carot.de
Tel.: +49 (0)177 5719797



Psssst! Schon vom neuen GMX MultiMessenger geh?rt? Der kann`s mit allen:
http://www.gmx.net/de/go/multimessenger

-- 
Dipl.-Ing. Alexander Car?t
PhD Candidate
Email : Alexander at Carot.de
Tel.: +49 (0)177 5719797



Psssst! Schon vom neuen GMX MultiMessenger geh?rt? Der kann`s mit allen:
http://www.gmx.net/de/go/multimessenger

On Tue, Nov 25, 2008 at 08:36:39AM +0100, "Alexander Car?t"
wrote:
> Hi all,
> 
> > ok... i guess i will implement this later.
> > is support for more then 512 sample blocks planned ?
> > i am putting one jack period of data into one packet.
> > so currently it only works with a period of <=512.
> > i could split that myself. but i dont see why the API should not
handle
> > bigger blocks. and just hide the splitting to smaller blocks.
> 
> +1 ---  I do something similar to netjack (currently called
"Soundjack") in order to play live music on a network (
https://www.itm.uni-luebeck.de/users/carot/TMT08.pdf ).
> 
> My settings vary from 1024 down to 128 sample-blocks (depending on the
current network situation). CELT works excellent in that context but it would be
nice to have 1024 samples supported as well.
>   
> > i need to fix alsa_in and alsa_out now, because they dont handle the
big
> > timing jitter of an internet connection gracefully.
> 
> I`d be curious to know what you are approaching to change in that context.
What amount of packets do you normally buffer on the network input in order to
cope with the jitter ?
i mainly only fixed the segfaults which occured in alsa_in and alsa_out.
and made them RT-safe finally.

the netjack approach is somewhat different. it operates synchronised to
only one soundcard. the slave machine has to use alsa_in and alsa_out to
connect to the soundcard.

these tools use the soundcard dma buffer as "jitter buffer"
they work very well under low jitter conditions, as seen, when using
netjack with ethernet. i normally use 1024 samples as buffer.

the internet is a new use case. i discovered celt yesterday.
and just started testing with internet.
alsa_out was already quite robust on a berlin <-> new york connection.
of course its hitting buffer boundaries sometimes.
but the other end is not seeing these problems. because of the
synchronous operation.

paul davis told me about your software. you demoed it on Tonmeister
Tagung, right ?


-- 
torben Hohn