While profiling one of my applicaitons, I created 1200 threaded clients to the server and server crashed with following error, This server connects to some external server in return and acts as an proxy, so the backtrace shows that error happened while connecting to the external server and I thought external server might have crashed, but that was not the case. While external server was still up, my application crashed. /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:598:in `connect_server'': no connection (RuntimeError) from /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:598:in `reconnect'' from /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:1076:in `reconnect'' from ./bin/../lib/data_handler.rb:82:in `reconnect_to_server'' from ./bin/../lib/protocol_handler.rb:55:in `dispatch_request'' from ./bin/../lib/protocol_handler.rb:17:in `receive_data'' from ./bin/../lib/protocol_handler.rb:14:in `each'' from ./bin/../lib/protocol_handler.rb:14:in `receive_data'' from /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:794:in `event_callback'' from /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:207:in `run_machine'' from /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:207:in `run'' from bin/activestock.rb:65:in `initialize'' from bin/activestock.rb:105:in `new'' from bin/activestock.rb:105 -- gnufied ----------- There was only one Road; that it was like a great river: its springs were at every doorstep, and every path was its tributary.
On 2/2/07, hemant <gethemant at gmail.com> wrote:> While profiling one of my applicaitons, I created 1200 threaded > clients to the server and server crashed with following error, > > This server connects to some external server in return and acts as an > proxy, so the backtrace shows that error happened while connecting to > the external server and I thought external server might have crashed, > but that was not the case. While external server was still up, my > application crashed. > > /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:598:in > `connect_server'': no connection (RuntimeError) > from /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:598:in `reconnect'' > from /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:1076:in `reconnect'' > from ./bin/../lib/data_handler.rb:82:in `reconnect_to_server'' > from ./bin/../lib/protocol_handler.rb:55:in `dispatch_request'' > from ./bin/../lib/protocol_handler.rb:17:in `receive_data'' > from ./bin/../lib/protocol_handler.rb:14:in `each'' > from ./bin/../lib/protocol_handler.rb:14:in `receive_data'' > from /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:794:in `event_callback'' > from /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:207:in `run_machine'' > from /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:207:in `run'' > from bin/activestock.rb:65:in `initialize'' > from bin/activestock.rb:105:in `new'' > from bin/activestock.rb:105 >Ok, I Also got a exception when clients attempt to connect firsttime: "Starting the the connection" /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:575:in `connect_server'': no connection (RuntimeError) from /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:575:in `connect'' from ./bin/../lib/data_handler.rb:41:in `request'' from bin/activestock.rb:68:in `initialize'' from /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:808:in `call'' from /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:808:in `event_callback'' from /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:207:in `run_machine'' from /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:207:in `run'' from bin/activestock.rb:64:in `initialize'' from bin/activestock.rb:104:in `new'' from bin/activestock.rb:104 -- gnufied ----------- There was only one Road; that it was like a great river: its springs were at every doorstep, and every path was its tributary.
On 2/2/07, hemant <gethemant at gmail.com> wrote:> > While profiling one of my applicaitons, I created 1200 threaded > clients to the server and server crashed with following error, > > This server connects to some external server in return and acts as an > proxy, so the backtrace shows that error happened while connecting to > the external server and I thought external server might have crashed, > but that was not the case. While external server was still up, my > application crashed. > > /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:598:in > `connect_server'': no connection (RuntimeError) > from /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:598:in > `reconnect'' > from /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:1076:in > `reconnect'' > from ./bin/../lib/data_handler.rb:82:in `reconnect_to_server'' > from ./bin/../lib/protocol_handler.rb:55:in `dispatch_request'' > from ./bin/../lib/protocol_handler.rb:17:in `receive_data'' > from ./bin/../lib/protocol_handler.rb:14:in `each'' > from ./bin/../lib/protocol_handler.rb:14:in `receive_data'' > from /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:794:in > `event_callback'' > from /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:207:in > `run_machine'' > from /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:207:in > `run'' > from bin/activestock.rb:65:in `initialize'' > from bin/activestock.rb:105:in `new'' > from bin/activestock.rb:105Hi Hemant. Do you get the same problem if you make only (let''s say) 900 connections? A Ruby process is often restricted to no more than 1024 file descriptors. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070202/a17430d5/attachment-0001.html
On 2/2/07, Francis Cianfrocca <garbagecat10 at gmail.com> wrote:> On 2/2/07, hemant <gethemant at gmail.com> wrote: > > While profiling one of my applicaitons, I created 1200 threaded > > clients to the server and server crashed with following error, > > > > This server connects to some external server in return and acts as an > > proxy, so the backtrace shows that error happened while connecting to > > the external server and I thought external server might have crashed, > > but that was not the case. While external server was still up, my > > application crashed. > > > > /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:598:in > > `connect_server'': no connection (RuntimeError) > > from > /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:598:in > `reconnect'' > > from > /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:1076:in > `reconnect'' > > from ./bin/../lib/data_handler.rb:82:in > `reconnect_to_server'' > > from ./bin/../lib/protocol_handler.rb:55:in > `dispatch_request'' > > from ./bin/../lib/protocol_handler.rb:17:in > `receive_data'' > > from ./bin/../lib/protocol_handler.rb:14:in > `each'' > > from ./bin/../lib/protocol_handler.rb:14:in > `receive_data'' > > from > /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:794:in > `event_callback'' > > from > /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:207:in > `run_machine'' > > from > /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:207:in > `run'' > > from bin/activestock.rb:65:in `initialize'' > > from bin/activestock.rb:105:in `new'' > > from bin/activestock.rb:105 > > > > > Hi Hemant. Do you get the same problem if you make only (let''s say) 900 > connections? A Ruby process is often restricted to no more than 1024 file > descriptors.-- gnufied ----------- There was only one Road; that it was like a great river: its springs were at every doorstep, and every path was its tributary.
On 2/2/07, Francis Cianfrocca <garbagecat10 at gmail.com> wrote:> On 2/2/07, hemant <gethemant at gmail.com> wrote: > > While profiling one of my applicaitons, I created 1200 threaded > > clients to the server and server crashed with following error, > > > > This server connects to some external server in return and acts as an > > proxy, so the backtrace shows that error happened while connecting to > > the external server and I thought external server might have crashed, > > but that was not the case. While external server was still up, my > > application crashed. > > > > /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:598:in > > `connect_server'': no connection (RuntimeError) > > from > /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:598:in > `reconnect'' > > from > /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:1076:in > `reconnect'' > > from ./bin/../lib/data_handler.rb:82:in > `reconnect_to_server'' > > from ./bin/../lib/protocol_handler.rb:55:in > `dispatch_request'' > > from ./bin/../lib/protocol_handler.rb:17:in > `receive_data'' > > from ./bin/../lib/protocol_handler.rb:14:in > `each'' > > from ./bin/../lib/protocol_handler.rb:14:in > `receive_data'' > > from > /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:794:in > `event_callback'' > > from > /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:207:in > `run_machine'' > > from > /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:207:in > `run'' > > from bin/activestock.rb:65:in `initialize'' > > from bin/activestock.rb:105:in `new'' > > from bin/activestock.rb:105 > > > > > Hi Hemant. Do you get the same problem if you make only (let''s say) 900 > connections? A Ruby process is often restricted to no more than 1024 file > descriptors.Yes Francis, I still get the crash. And i thought 1024 limit is set by the platform, not Ruby and we can change it using ulimit. Also, I can very well make 1200 connections to the backend data server, which is again in Ruby but written using threads. -- gnufied ----------- There was only one Road; that it was like a great river: its springs were at every doorstep, and every path was its tributary.
Does the epoll-based version have similar limitations? We were running into the same problem at around 1200 connections. Also, any idea when that''s going to wind up in svn? - Tony On 2/2/07, Francis Cianfrocca <garbagecat10 at gmail.com> wrote:> > On 2/2/07, hemant <gethemant at gmail.com> wrote: > > > > While profiling one of my applicaitons, I created 1200 threaded > > clients to the server and server crashed with following error, > > > > This server connects to some external server in return and acts as an > > proxy, so the backtrace shows that error happened while connecting to > > the external server and I thought external server might have crashed, > > but that was not the case. While external server was still up, my > > application crashed. > > > > /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:598:in > > `connect_server'': no connection (RuntimeError) > > from /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:598:in > > `reconnect'' > > from /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:1076:in > > `reconnect'' > > from ./bin/../lib/data_handler.rb:82:in `reconnect_to_server'' > > from ./bin/../lib/protocol_handler.rb:55:in `dispatch_request'' > > from ./bin/../lib/protocol_handler.rb:17:in `receive_data'' > > from ./bin/../lib/protocol_handler.rb:14:in `each'' > > from ./bin/../lib/protocol_handler.rb:14:in `receive_data'' > > from /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:794:in > > `event_callback'' > > from /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:207:in > > `run_machine'' > > from /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:207:in > > `run'' > > from bin/activestock.rb:65:in `initialize'' > > from bin/activestock.rb:105:in `new'' > > from bin/activestock.rb:105 > > > > > > Hi Hemant. Do you get the same problem if you make only (let''s say) 900 > connections? A Ruby process is often restricted to no more than 1024 file > descriptors. > > _______________________________________________ > Eventmachine-talk mailing list > Eventmachine-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/eventmachine-talk >-- Tony Arcieri ClickCaster, Inc. tony at clickcaster.com (970) 232-4208 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070202/3fdf8788/attachment.html
On 2/2/07, Tony Arcieri <tony at clickcaster.com> wrote:> > Does the epoll-based version have similar limitations? We were running > into the same problem at around 1200 connections. > > Also, any idea when that''s going to wind up in svn?Epoll doesn''t increase the number of connections available in a process unless you turn up the ulimit, as Hemant suggests. (Which usually requires root privs.) The epoll code should be in SVN now, Tony. Have a look. We need to add some config settings to make it available. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070202/5cb3b19f/attachment.html
On 2/2/07, hemant <gethemant at gmail.com> wrote:> > On 2/2/07, hemant <gethemant at gmail.com> wrote: > > While profiling one of my applicaitons, I created 1200 threaded > > clients to the server and server crashed with following error, > > > > This server connects to some external server in return and acts as an > > proxy, so the backtrace shows that error happened while connecting to > > the external server and I thought external server might have crashed, > > but that was not the case. While external server was still up, my > > application crashed. > > > > /usr/local/lib/ruby/site_ruby/1.8/eventmachine.rb:598:in > > `connect_server'': no connection (RuntimeError)The "no connection" error almost always means some problem at the TCP level. Can you connect to the server with telnet? Can you create a small EM program that is able to connect to that server? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070202/991443fd/attachment.html
On 2/2/07, Francis Cianfrocca <garbagecat10 at gmail.com> wrote:> On 2/2/07, Tony Arcieri <tony at clickcaster.com> wrote: > > Does the epoll-based version have similar limitations? We were running > into the same problem at around 1200 connections. > > > > Also, any idea when that''s going to wind up in svn? > > > Epoll doesn''t increase the number of connections available in a process > unless you turn up the ulimit, as Hemant suggests. (Which usually requires > root privs.) > > The epoll code should be in SVN now, Tony. Have a look. We need to add some > config settings to make it available. >Does epoll adds any benefits compared to select? Also, it might be a good idea to make use of non blocking IO introduced in 1.8.5. But Why, do i get crahes even with 900 connections? -- gnufied ----------- There was only one Road; that it was like a great river: its springs were at every doorstep, and every path was its tributary.
On 2/2/07, hemant <gethemant at gmail.com> wrote:> > > Does epoll adds any benefits compared to select? > > Also, it might be a good idea to make use of non blocking IO > introduced in 1.8.5. But Why, do i get crahes even with 900 > connections?The expectation is that epoll is faster than select because it''s much more efficient. (Of course the downside is it only works on Linux 2.6 kernels.) EM does use the nonblocking I/O introduced in Ruby last May. In fact, the final impetus for getting it added was to support EM :-) Hemant, does your application still crash with a much smaller number of connections? Say, 100? If so, then we''re obviously looking for the problem in the wrong place. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070202/4b9b60b1/attachment-0001.html
On Fri, 2007-02-02 at 14:43 -0500, Francis Cianfrocca wrote:> On 2/2/07, hemant <gethemant at gmail.com> wrote: > > Does epoll adds any benefits compared to select? > > Also, it might be a good idea to make use of non blocking IO > introduced in 1.8.5. But Why, do i get crahes even with 900 > connections? > > > The expectation is that epoll is faster than select because it''s much > more efficient. (Of course the downside is it only works on Linux 2.6 > kernels.) > > EM does use the nonblocking I/O introduced in Ruby last May. In fact, > the final impetus for getting it added was to support EM :-) > > Hemant, does your application still crash with a much smaller number > of connections? Say, 100? If so, then we''re obviously looking for the > problem in the wrong place. > > >Hmm, after further investigation I found that, the backend server seems to be locking at 900 concurrent connections and this may be the reason. I am investigating further and keep you updated.
Where is the epoll implementation? The last change I''m seeing to the Subversion repository was by me... - Tony On 2/2/07, Francis Cianfrocca <garbagecat10 at gmail.com> wrote:> > On 2/2/07, Tony Arcieri <tony at clickcaster.com> wrote: > > > > Does the epoll-based version have similar limitations? We were running > > into the same problem at around 1200 connections. > > > > Also, any idea when that''s going to wind up in svn? > > > > Epoll doesn''t increase the number of connections available in a process > unless you turn up the ulimit, as Hemant suggests. (Which usually requires > root privs.) > > The epoll code should be in SVN now, Tony. Have a look. We need to add > some config settings to make it available. > > > _______________________________________________ > Eventmachine-talk mailing list > Eventmachine-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/eventmachine-talk >-- Tony Arcieri ClickCaster, Inc. tony at clickcaster.com (970) 232-4208 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070204/804d7456/attachment.html
On 2/4/07, Tony Arcieri <tony at clickcaster.com> wrote:> > Where is the epoll implementation? The last change I''m seeing to the > Subversion repository was by me...Should be in em.cpp. It''s been there for a long time. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070205/ef63763f/attachment.html
These are the only references to epoll I''m seeing in em.cpp:
~/src/eventmachine/ext tarcieri$ grep epoll em.cpp
// This is based on a select loop. Alternately provide epoll
// epoll will be effective if we provide it as an alternative,
I''m at revision 302
- Tony
On 2/5/07, Francis Cianfrocca <garbagecat10 at gmail.com>
wrote:>
> On 2/4/07, Tony Arcieri <tony at clickcaster.com> wrote:
> >
> > Where is the epoll implementation? The last change I''m
seeing to the
> > Subversion repository was by me...
>
>
>
> Should be in em.cpp. It''s been there for a long time.
>
>
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>
--
Tony Arcieri
ClickCaster, Inc.
tony at clickcaster.com
(970) 232-4208
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070206/7b8557a2/attachment.html
On 2/6/07, Tony Arcieri <tony at clickcaster.com> wrote:> > These are the only references to epoll I''m seeing in em.cpp: > > ~/src/eventmachine/ext tarcieri$ grep epoll em.cpp > // This is based on a select loop. Alternately provide epoll > // epoll will be effective if we provide it as an alternative, > > I''m at revision 302Aargh. Sorry, Tony, I must not have checked it in. Will do so as soon as I get a free moment. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070206/29d43456/attachment.html
Get a free moment yet? :) - Tony On 2/6/07, Francis Cianfrocca <garbagecat10 at gmail.com> wrote:> > On 2/6/07, Tony Arcieri <tony at clickcaster.com> wrote: > > > > These are the only references to epoll I''m seeing in em.cpp: > > > > ~/src/eventmachine/ext tarcieri$ grep epoll em.cpp > > // This is based on a select loop. Alternately provide epoll > > // epoll will be effective if we provide it as an alternative, > > > > I''m at revision 302 > > > > Aargh. Sorry, Tony, I must not have checked it in. Will do so as soon as I > get a free moment. > > > _______________________________________________ > Eventmachine-talk mailing list > Eventmachine-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/eventmachine-talk >-- Tony Arcieri ClickCaster, Inc. tony at clickcaster.com (970) 232-4208 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070211/4d0c020e/attachment-0001.html