Mongrel users - Oct 2006 - Mongrel is crashing under heavy loads

On Fri, 6 Oct 2006 09:58:36 -0600
Donald Marino <donaldmarino at mac.com> wrote:
> Hi
<snip>
> 
> The line in particular that appears to  be breaking is:
>
>   $mongrel_sleeper_thread = Thread.new { loop { sleep 1 } }
> 
> Found at line 274 of configurator.rb
<snip>

Well, there''s not a whole lot Mongrel can do to keep Ruby from
segfaults.  I''d suggest just removing this line or commenting it out
for now.  What that does is keep a thread going so that Ruby will actually exit
when told to (since, for some dumbass reason select() in Ruby needs one thread
to actually exit without IO).

If you still segfault after this, then you''ll have to do the following:

1) grab the ruby source and recompile with debugging turned on (probably
requires:  CFLAGS="-g" make)
2) run your app until it explodes again and keep the core file.
3) open the core file in gdb and type:  backtrace
4) take whatever gdb says is the ruby backtrace and post it to ruby-lang asking
for help.  Someone there might know (if they don''t argue with your for
days on end about the merits of OS level VM malloc strategies and how
it''s not really a bug).

If you get ultra stuck, contact me off-list and i''ll try to help, but
there''s probably not much I can do since I''m not a ruby
internals expert (more of a death ninja hater really).

-- 
Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
http://www.zedshaw.com/
http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.

Hi

We''re running mongrel on CentOS 4.2 with Apache httpd 2.2  
mod_proxy_balancer, mongrel_cluster-0.2.0, Rails 1.1.6, Ruby 1.8.4,  
Pg 8.1.4 backend.

We are on the verge of going live with our site and I am stress  
testing the server.
What we are seeing is under heavy loads, our mongrels will seg fault  
and die one by one.

Following the advice in a previous thread (http://www.mail- 
archive.com/mongrel-users at rubyforge.org/msg01734.html),
  I installed mongrel 0.3.13.5, but the problem remains

The line in particular that appears to  be breaking is:

  $mongrel_sleeper_thread = Thread.new { loop { sleep 1 } }

Found at line 274 of configurator.rb

I do have a core file, but I''m not really proficient with gdb.
I can send it along if someone is interested.

I''m using JMeter to simulate loads on the server. The mongrels only  
seem to die under high loads.
Anyone have any ideas why this isn''t stable for us? We''d like
to be
able to go live with this setup if we can figure this out.

Thanks,
Donnie

Donald Marino
donaldmarino at mac dot com

On 10/6/06, Donald Marino <donaldmarino at mac.com> wrote:
> I''m using JMeter to simulate loads on the server. The mongrels
only
> seem to die under high loads.
> Anyone have any ideas why this isn''t stable for us? We''d
like to be
> able to go live with this setup if we can figure this out.
What sort of loads are you calling high loads?  High levels of
concurrency?  How high?


Kirk Haines

Zed,Kirk,

Thanks for your replies. I commented out line 274 of configurator.rb  
and was unable to replicate the crash after that.

As far as loads, I suppose it''s always relative, I''d say these
loads
aren''t really what most would call ''heavy''
But, in the end, those crashes were happening under any load, it was  
just apparent more quickly under load.

I was using up to 5 JMeter clients to simulate loads of  
4,8,16,32,64,128, and 256 concurrent users logging in, hitting 8 URLs  
and logging out, repeatedly.
I did this first with 8 mongrel servers on the balancer and then with  
16 mongrels.

So, for now, we''re going to call it good as I can''t replicate
the
''crash'' with that line removed. If I come across this again
I''ll
repost with better incident info.

Thanks,
Donnie

donaldmarino-at-mac-dot-com


On Oct 6, 2006, at 9:15 AM, Zed A. Shaw wrote:
> On Fri, 6 Oct 2006 09:58:36 -0600
> Donald Marino <donaldmarino at mac.com> wrote:
>
>> Hi
> <snip>
>>
>> The line in particular that appears to  be breaking is:
>>
>>   $mongrel_sleeper_thread = Thread.new { loop { sleep 1 } }
>>
>> Found at line 274 of configurator.rb
> <snip>
>
> Well, there''s not a whole lot Mongrel can do to keep Ruby from  
> segfaults.  I''d suggest just removing this line or commenting it  
> out for now.  What that does is keep a thread going so that Ruby  
> will actually exit when told to (since, for some dumbass reason  
> select() in Ruby needs one thread to actually exit without IO).
>
> If you still segfault after this, then you''ll have to do the  
> following:
>
> 1) grab the ruby source and recompile with debugging turned on  
> (probably requires:  CFLAGS="-g" make)
> 2) run your app until it explodes again and keep the core file.
> 3) open the core file in gdb and type:  backtrace
> 4) take whatever gdb says is the ruby backtrace and post it to ruby- 
> lang asking for help.  Someone there might know (if they don''t  
> argue with your for days on end about the merits of OS level VM  
> malloc strategies and how it''s not really a bug).
>
> If you get ultra stuck, contact me off-list and i''ll try to help,
> but there''s probably not much I can do since I''m not a
ruby
> internals expert (more of a death ninja hater really).
>
> -- 
> Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
> http://www.zedshaw.com/
> http://mongrel.rubyforge.org/
> http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users