Eventmachine talk - Aug 2008 - JRuby and not flushing the send queue

Hi,

I recently grabbed Raggi''s trunk version of EventMachine, built the
jar & gem and got my application up and running under JRuby 1.1.3 on
OS X 10.5, Java 1.5.

I''ve implemented a very "chatty" protocol which works great
under Ruby
1.8.6 & EM 0.12.0, but under JRuby with the Jar reactor small
quantities of data sent with send_data do not get flushed quickly
enough.  The TCP connection times out before the data is sent unless I
arbitrarily send more data.  This won''t work for the protocol
I''m
using, since it has a lot of small request/response  pairs.

I''ve done a lot of threaded blocking network I/O programming in Java,
but the channel-based approach in EM is unfamiliar to me. How can I
modify the java portion of EM to be more aggressive about flushing the
outgoing queue?

Thanks,
-Ian

On 21 Aug 2008, at 06:57, Ian Levesque wrote:
> Hi,
>
> I recently grabbed Raggi''s trunk version of EventMachine, built
the
> jar & gem and got my application up and running under JRuby 1.1.3 on
> OS X 10.5, Java 1.5.
>
> I''ve implemented a very "chatty" protocol which works
great under
> Ruby 1.8.6 & EM 0.12.0, but under JRuby with the Jar reactor small
> quantities of data sent with send_data do not get flushed quickly
> enough.  The TCP connection times out before the data is sent unless
> I arbitrarily send more data.  This won''t work for the protocol
I''m
> using, since it has a lot of small request/response  pairs.
>
> I''ve done a lot of threaded blocking network I/O programming in
> Java, but the channel-based approach in EM is unfamiliar to me. How
> can I modify the java portion of EM to be more aggressive about
> flushing the outgoing queue?
There is currently a test in the test_basic.rb test suite which fails
on this, iirc, it''s ticket #50 (or very nearby) and is currently open.
I''m hoping to have some time to do more work on the Java reactor soon,
but I have just returned from holiday.

In the meantime, if you are able to replicate other bugs, please
submit tickets to rubyeventmachine.com, once it''s back up, or just
send ticket details to the list or myself and we''ll get them detailed
at the appropriate time.

Thank you for picking it up and providing us with feedback :)
>
>
> Thanks,
> -Ian
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk

Perhaps the java sockets need to have nagle disabled?
> I''ve implemented a very "chatty" protocol which works
great under Ruby 1.8.6
> & EM 0.12.0, but under JRuby with the Jar reactor small quantities of
data
> sent with send_data do not get flushed quickly enough.  The TCP connection

Any advice for tuning EventMachine for minimal Jitter and Latency? I
am working on an audio project that sends open sound control udp
messages via event machine. I am aiming for below 10ms of total jitter
and latency (this is the threshold of audible rhythmic inconsistencies
for control data).

More specifically:
- Are there tunings for EventMachine or Ruby that would decrease
jitter and latency?
- Is there a way to queue udp messages and send them without calling
ruby again? I am currently using lambdas to send UDP messages via a
ruby OSC library.
- Is there example code that might be relevant to hard real time
systems using EventMachine.

Thanks!
Noah

Please try to start new threads, rather than replying to other threads
if changing the topic. There is an ID field in e-mail used by various
MUAs which will now group this to the thread you replied to.

On 21 Aug 2008, at 18:48, Noah Thorp wrote:
> Any advice for tuning EventMachine for minimal Jitter and Latency?
Under concurrent load, yes, you need to free up the reactor regularly
enough to split out any non-compute bound or out of KPI target time
between as many clients as possible. Under linear load or compute
bound scnearios, there''s little you can do to enhance
''speed'' other
than general application optimization.
> I am working on an audio project that sends open sound control udp
> messages via event machine. I am aiming for below 10ms of total
> jitter and latency (this is the threshold of audible rhythmic
> inconsistencies for control data).
Understood, completely.
> More specifically:
> - Are there tunings for EventMachine or Ruby that would decrease
> jitter and latency?
As above, the only main help can be applied on the latency (and by
implication in this case, jitter), by rapid scheduling of partial
tasks to appear concurrent. See the Deferrable pattern, applied to
many small runtime callbacks, rather than one long callback servicing
a whole ''operation'' or ''request'', or maybe
in this case ''render''.
> - Is there a way to queue udp messages and send them without calling
> ruby again? I am currently using lambdas to send UDP messages via a
> ruby OSC library.
Just send them... UDP delivery time and order is never guaranteed. The
control protocol surely has the means to deal with out of order and
late messages.
> - Is there example code that might be relevant to hard real time
> systems using EventMachine.
Very little at this time. I can provide commercial consultancy if that
is what you require.

It should be noted that the boundaries applied by the speed of ruby
and cross language nature of EventMachine may result in somewhat poor
performance for an application with such stern latency demands,
however, I have not tried to do such voip work using ruby, so I cannot
be sure of the exact constraints without doing at least some stab cases.
>
>
> Thanks!
> Noah
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk

Hmm, didn''t have to do that with the C++ reactor, but it''s
worth a shot.

-Ian

On Aug 21, 2008, at 10:09 AM, Roger Pack wrote:
> Perhaps the java sockets need to have nagle disabled?
>
>> I''ve implemented a very "chatty" protocol which
works great under
>> Ruby 1.8.6
>> & EM 0.12.0, but under JRuby with the Jar reactor small quantities
>> of data
>> sent with send_data do not get flushed quickly enough.  The TCP
>> connection
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk

Nope, disabling and explicitly enabling nagle using
Socket.setTcpNoDelay() had no effect.  Same failure as before.

Thanks though,
-Ian

On Aug 21, 2008, at 5:37 PM, Ian Levesque wrote:
> Hmm, didn''t have to do that with the C++ reactor, but
it''s worth a
> shot.
>
> -Ian
>
> On Aug 21, 2008, at 10:09 AM, Roger Pack wrote:
>
>> Perhaps the java sockets need to have nagle disabled?
>>
>>> I''ve implemented a very "chatty" protocol which
works great under
>>> Ruby 1.8.6
>>> & EM 0.12.0, but under JRuby with the Jar reactor small
quantities
>>> of data
>>> sent with send_data do not get flushed quickly enough.  The TCP
>>> connection
>> _______________________________________________
>> Eventmachine-talk mailing list
>> Eventmachine-talk at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>

> More specifically:
> - Are there tunings for EventMachine or Ruby that would decrease jitter and
> latency?
a fast CPU and make sure that you release back to the em loop frequently.

> - Is there a way to queue udp messages and send them without calling ruby
> again? I am currently using lambdas to send UDP messages via a ruby OSC
> library.
I believe UDP messages, as mentioned by James, are just sent ''all at
once'' so...currently you''d really have to queue them in Ruby. 
Or hack
the C code.

> - Is there example code that might be relevant to hard real time systems
> using EventMachine.
Jruby might be more consistent speed-wise--maybe.
Good luck!
-=R

On Fri, Aug 22, 2008 at 12:14 PM, Roger Pack
<roger.pack at leadmediapartners.com> wrote:
>> More specifically:
>> - Are there tunings for EventMachine or Ruby that would decrease jitter
and
>> latency?
>
> a fast CPU and make sure that you release back to the em loop frequently.
>
>
>> - Is there a way to queue udp messages and send them without calling
ruby
>> again? I am currently using lambdas to send UDP messages via a ruby OSC
>> library.
>
> I believe UDP messages, as mentioned by James, are just sent ''all
at
> once'' so...currently you''d really have to queue them in
Ruby.  Or hack
> the C code.
>
>
>> - Is there example code that might be relevant to hard real time
systems
>> using EventMachine.
>
> Jruby might be more consistent speed-wise--maybe.
> Good luck!
Umm well no, if it comes to raw socket performance EventMachine
matches java performace ( I have done benchmarks with raw java nio and
EM, and my conclusion is, EM is as fast as Java NIO), but Java
obviously has huge performance edge when it comes to executing
application level code, but you won''t see that advantage if you are
using JRuby. EM, IO performance is really good because its close to
the metal.

On 22 Aug 2008, at 08:25, hemant wrote:
> On Fri, Aug 22, 2008 at 12:14 PM, Roger Pack
> <roger.pack at leadmediapartners.com> wrote:
>>> More specifically:
>>> - Are there tunings for EventMachine or Ruby that would decrease
>>> jitter and
>>> latency?
>>
>> a fast CPU and make sure that you release back to the em loop
>> frequently.
>>
>>
>>> - Is there a way to queue udp messages and send them without
>>> calling ruby
>>> again? I am currently using lambdas to send UDP messages via a
>>> ruby OSC
>>> library.
>>
>> I believe UDP messages, as mentioned by James, are just sent
''all at
>> once'' so...currently you''d really have to queue them
in Ruby.  Or
>> hack
>> the C code.
>>
>>
>>> - Is there example code that might be relevant to hard real time
>>> systems
>>> using EventMachine.
>>
>> Jruby might be more consistent speed-wise--maybe.
>> Good luck!
>
> Umm well no, if it comes to raw socket performance EventMachine
> matches java performace ( I have done benchmarks with raw java nio and
> EM, and my conclusion is, EM is as fast as Java NIO), but Java
> obviously has huge performance edge when it comes to executing
> application level code, but you won''t see that advantage if you
are
> using JRuby. EM, IO performance is really good because its close to
> the metal.
FYI - Jitter and Latency issues are not IO bound issues. Typically a
VOIP application of this nature pushes far far less than 0.5mb/s.
>
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk

Thanks James, Roger, and Hemant for replies on this thread. I''ll
report back on my findings. I am setting up some performance stats and
will optimize against those. It may take me a bit of time to post
results as the timing issue has moved a little further down the queue.

Best,
Noah


On Aug 21, 2008, at 12:49 PM, James Tucker wrote:
> Please try to start new threads, rather than replying to other
> threads if changing the topic. There is an ID field in e-mail used
> by various MUAs which will now group this to the thread you replied
> to.
>
> On 21 Aug 2008, at 18:48, Noah Thorp wrote:
>
>> Any advice for tuning EventMachine for minimal Jitter and Latency?
>
> Under concurrent load, yes, you need to free up the reactor
> regularly enough to split out any non-compute bound or out of KPI
> target time between as many clients as possible. Under linear load
> or compute bound scnearios, there''s little you can do to enhance
> ''speed'' other than general application optimization.
>
>> I am working on an audio project that sends open sound control udp
>> messages via event machine. I am aiming for below 10ms of total
>> jitter and latency (this is the threshold of audible rhythmic
>> inconsistencies for control data).
>
> Understood, completely.
>
>> More specifically:
>> - Are there tunings for EventMachine or Ruby that would decrease
>> jitter and latency?
>
> As above, the only main help can be applied on the latency (and by
> implication in this case, jitter), by rapid scheduling of partial
> tasks to appear concurrent. See the Deferrable pattern, applied to
> many small runtime callbacks, rather than one long callback
> servicing a whole ''operation'' or
''request'', or maybe in this case
> ''render''.
>
>> - Is there a way to queue udp messages and send them without
>> calling ruby again? I am currently using lambdas to send UDP
>> messages via a ruby OSC library.
>
> Just send them... UDP delivery time and order is never guaranteed.
> The control protocol surely has the means to deal with out of order
> and late messages.
>
>> - Is there example code that might be relevant to hard real time
>> systems using EventMachine.
>
> Very little at this time. I can provide commercial consultancy if
> that is what you require.
>
> It should be noted that the boundaries applied by the speed of ruby
> and cross language nature of EventMachine may result in somewhat
> poor performance for an application with such stern latency demands,
> however, I have not tried to do such voip work using ruby, so I
> cannot be sure of the exact constraints without doing at least some
> stab cases.
>
>>
>>
>> Thanks!
>> Noah
>> _______________________________________________
>> Eventmachine-talk mailing list
>> Eventmachine-talk at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk

As far as I can tell, the present way EventMachine schedules Ruby threads
increases the Reactor latency as control is handed over to the Ruby
scheduler for 10ms at a time in MRI.

A possible optimization might be to have EventMachine block 10ms at a time
and schedule Ruby threads with rb_thread_schedule()

On Wed, Aug 27, 2008 at 10:57 AM, Noah Thorp <noahlist at
listenlabs.com>wrote:
> Thanks James, Roger, and Hemant for replies on this thread. I''ll
report
> back on my findings. I am setting up some performance stats and will
> optimize against those. It may take me a bit of time to post results as the
> timing issue has moved a little further down the queue.
>
> Best,
> Noah
>
>
>
> On Aug 21, 2008, at 12:49 PM, James Tucker wrote:
>
>  Please try to start new threads, rather than replying to other threads if
>> changing the topic. There is an ID field in e-mail used by various MUAs
>> which will now group this to the thread you replied to.
>>
>> On 21 Aug 2008, at 18:48, Noah Thorp wrote:
>>
>>  Any advice for tuning EventMachine for minimal Jitter and Latency?
>>>
>>
>> Under concurrent load, yes, you need to free up the reactor regularly
>> enough to split out any non-compute bound or out of KPI target time
between
>> as many clients as possible. Under linear load or compute bound
scnearios,
>> there''s little you can do to enhance ''speed''
other than general application
>> optimization.
>>
>>  I am working on an audio project that sends open sound control udp
>>> messages via event machine. I am aiming for below 10ms of total
jitter and
>>> latency (this is the threshold of audible rhythmic inconsistencies
for
>>> control data).
>>>
>>
>> Understood, completely.
>>
>>  More specifically:
>>> - Are there tunings for EventMachine or Ruby that would decrease
jitter
>>> and latency?
>>>
>>
>> As above, the only main help can be applied on the latency (and by
>> implication in this case, jitter), by rapid scheduling of partial tasks
to
>> appear concurrent. See the Deferrable pattern, applied to many small
runtime
>> callbacks, rather than one long callback servicing a whole
''operation'' or
>> ''request'', or maybe in this case
''render''.
>>
>>  - Is there a way to queue udp messages and send them without calling
ruby
>>> again? I am currently using lambdas to send UDP messages via a ruby
OSC
>>> library.
>>>
>>
>> Just send them... UDP delivery time and order is never guaranteed. The
>> control protocol surely has the means to deal with out of order and
late
>> messages.
>>
>>  - Is there example code that might be relevant to hard real time
systems
>>> using EventMachine.
>>>
>>
>> Very little at this time. I can provide commercial consultancy if that
is
>> what you require.
>>
>> It should be noted that the boundaries applied by the speed of ruby and
>> cross language nature of EventMachine may result in somewhat poor
>> performance for an application with such stern latency demands,
however, I
>> have not tried to do such voip work using ruby, so I cannot be sure of
the
>> exact constraints without doing at least some stab cases.
>>
>>
>>>
>>> Thanks!
>>> Noah
>>> _______________________________________________
>>> Eventmachine-talk mailing list
>>> Eventmachine-talk at rubyforge.org
>>> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>>>
>>
>> _______________________________________________
>> Eventmachine-talk mailing list
>> Eventmachine-talk at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>>
>
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>


--
Tony Arcieri
medioh.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://rubyforge.org/pipermail/eventmachine-talk/attachments/20080827/dc2113db/attachment-0001.html>

> As far as I can tell, the present way EventMachine schedules Ruby threads
> increases the Reactor latency as control is handed over to the Ruby
> scheduler for 10ms at a time in MRI.
Since EM is using rb_select for its select loop [oh -- except for
kqueue--not sure about epoll--they might block] it effectively returns
control back to Ruby threads.
Another possibility would be to use Ruby 1.9, since it releases the
global lock while in its select loop.

Also MRI is not known for its amazing realtime nature, due to the GC.
Give me a holler sometime and I could try to finish up a non-blocking
GC I''ve been dabbling with, but...still might not be all there.
-=R
> A possible optimization might be to have EventMachine block 10ms at a time
> and schedule Ruby threads with rb_thread_schedule()