Mongrel users - Dec 2007 - Segmentation fault in Mongrel when run with --debug

I am having a problem with mongrel just stopping inside a
periodically_call_remote function so I ran mongrel with debug enabled
Mongrel seg faults after a few minutes

The mongrel filelog shows 5 files open when this happens. Same log files
that are open all the time.

Wed Dec 19 15:10:21 -0700 2007 FILES OPEN BEFORE REQUEST
/calibration/ackAdjustDistance
---
log/mongrel_debug/objects.log: 2
/home/scott/NetBeansProjects/webmonitor/config/../log/development.log: 1
log/mongrel_debug/rails.log: 1
log/mongrel_debug/files.log: 1
log/mongrel_debug/threads.log: 1
log/mongrel_debug/access.log: 1


There are 2 threads running
Wed Dec 19 15:10:21 -0700 2007 REQUEST /calibration/ackAdjustDistance
0.0.0.0:3000 -- THREADS: 2 -----
  KEYS:
    -- #<Thread:0xb6e10738>: [:started_on]
    -- #<Thread:0xb75194bc>: [:started_on, :__inspect_key__]

Mongrel stops with this output to stdout.

127.0.0.1 - [Wed, 19 Dec 2007 22:10:17 GMT] "POST
/calibration/ackAdjustDistance HTTP/1.1"
127.0.0.1 - [Wed, 19 Dec 2007 22:10:19 GMT] "POST
/calibration/ackAdjustDistance HTTP/1.1"
127.0.0.1 - [Wed, 19 Dec 2007 22:10:21 GMT] "POST
/calibration/ackAdjustDistance HTTP/1.1"
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.2/bin/../lib/mongrel/debug.rb:128:
[BUG] Segmentation fault
ruby 1.8.6 (2007-09-24) [i686-linux]

the last part of the development log is

Processing CalibrationController#ackAdjustDistance (for 127.0.0.1 at
2007-12-19 15:10:23) [POST]
  Session ID: 5f0b23a7ae94ac5648268113cd3b4560
  Parameters: {"action"=>"ackAdjustDistance",
"controller"=>"calibration"}
  [4;36;1mCalibration Load (0.000802)   [0;1mSELECT * FROM
configurations WHERE (Name = ''CalibrationMessage'') LIMIT 1
DEPRECATION WARNING: @session is deprecated! Call session.respond_to?
instead of @session.respond_to?. Args: [:length]  See
http://www.rubyonrails.org/deprecation for details. (called from process
at
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.2/bin/../lib/mongrel/debug.rb:128)
Completed in 0.03658 (27 reqs/sec) | Rendering: 0.00144 (3%) | 200 OK
[http://localhost/calibration/ackAdjustDistance]
DEPRECATION WARNING: @session is deprecated! Call session.class instead
of @session.class. Args: []  See http://www.rubyonrails.org/deprecation
for details. (called from process at
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.2/bin/../lib/mongrel/debug.rb:136)
DEPRECATION WARNING: @response is deprecated! Call response.respond_to?
instead of @response.respond_to?. Args: [:length]  See
http://www.rubyonrails.org/deprecation for details. (called from process
at
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.2/bin/../lib/mongrel/debug.rb:128)

both outputs refer to debug.rb line 128 which is

# stolen from Robert Klemme
  class Objects < GemPlugin::Plugin "/handlers"
    include Mongrel::HttpHandlerPlugin

 121   def process(request,response)
 122    begin
 123       stats = Hash.new(0)
 124       lengths = {}
 125       begin
 126         ObjectSpace.each_object do |o|
 127          begin
 128             if o.respond_to? :length
 129               len = o.length
 130               lengths[o.class] ||= Mongrel::Stats.new(o.class)
 131               lengths[o.class].sample(len)
 132             end
 133           rescue Object
 134           end

anybody else seen this?

thanks,

Scott
-- 
Posted via http://www.ruby-forum.com/.

On Wed, 19 Dec 2007 23:51:55 +0100
Scott Derrick <lists at ruby-forum.com> wrote:
> I am having a problem with mongrel just stopping inside a
> periodically_call_remote function so I ran mongrel with debug enabled
> Mongrel seg faults after a few minutes
> 
> The mongrel filelog shows 5 files open when this happens. Same log files
> that are open all the time.
No idea what would cause it, since that''s a simple respond to, but
usually when you get segfaults in weird places it''s caused by some
other completely unknown library that doesn''t do ram properly.

What you should do is run your mongrel under gdb, when it blows up, gdb
will stop and you can then use the backtrace command to see what was
going on.

Apart from that, not sure what else.

-- 
Zed A. Shaw
- Hate: http://savingtheinternetwithhate.com/
- Good: http://www.zedshaw.com/
- Evil: http://yearofevil.com/

I replaced mongrel with webrick and the problem went away.

The offending page has been running for 45 minutes now, using mongrel I 
always died between 5 and 10 minutes.

Guess I''ll use lighthttp or something...

Scott
-- 
Posted via http://www.ruby-forum.com/.

On Thu, 20 Dec 2007 18:42:47 +0100
Scott Derrick <lists at ruby-forum.com> wrote:
> I replaced mongrel with webrick and the problem went away.
> 
> The offending page has been running for 45 minutes now, using mongrel I 
> always died between 5 and 10 minutes.
> 
> Guess I''ll use lighthttp or something...
Well, webrick might be hiding a problem that''s caused by speed like a
race condition or similar.  When your app starts to get more traffic
you''ll probably hit it again.

Take a look at the extensions you''re using.  I''m betting one
of them
has a threading problem or isn''t written correctly.

And FYI, nginx, apache, and lighttpd will all work with webrick the
same way they work with Mongrel.

-- 
Zed A. Shaw
- Hate: http://savingtheinternetwithhate.com/
- Good: http://www.zedshaw.com/
- Evil: http://yearofevil.com/

Zed,

I think your right because webrick finally stopped after about 90 
minutes.

It was locked up and when I killed it it complained of a deadlock.

I''m now thinking it may bein a call to the sqlite3_ruby gem.

I tried to run mongrel_rails under gdb but gdb complains its not the 
correct file type.

Scott

Zed A. Shaw wrote:
> On Thu, 20 Dec 2007 18:42:47 +0100
> Scott Derrick <lists at ruby-forum.com> wrote:
> 
>> I replaced mongrel with webrick and the problem went away.
>> 
>> The offending page has been running for 45 minutes now, using mongrel I
>> always died between 5 and 10 minutes.
>> 
>> Guess I''ll use lighthttp or something...
> 
> Well, webrick might be hiding a problem that''s caused by speed
like a
> race condition or similar.  When your app starts to get more traffic
> you''ll probably hit it again.
> 
> Take a look at the extensions you''re using.  I''m betting
one of them
> has a threading problem or isn''t written correctly.
> 
> And FYI, nginx, apache, and lighttpd will all work with webrick the
> same way they work with Mongrel.
> 
> --
> Zed A. Shaw
> - Hate: http://savingtheinternetwithhate.com/
> - Good: http://www.zedshaw.com/
> - Evil: http://yearofevil.com/
-- 
Posted via http://www.ruby-forum.com/.

On Dec 20, 2007 3:51 PM, Scott Derrick <lists at ruby-forum.com>
wrote:
> Zed,
>
> I think your right because webrick finally stopped after about 90
> minutes.
>
> It was locked up and when I killed it it complained of a deadlock.
>
> I''m now thinking it may bein a call to the sqlite3_ruby gem.
>
Oh, sqlite3 gem!

I got a few segfaults with it, like unrolling transactions and other
weird stuff.

Still waiting for some tickets get fixed, but without luck.
> I tried to run mongrel_rails under gdb but gdb complains its not the
> correct file type.
you need to run gdb on "ruby", not mongrel_rails (since mongrel_rails
is a ruby script).

gdb /path/to/ruby/ruby.exe /path/to/mongrel_rails param1 param2 param3

-- 
Luis Lavena
Multimedia systems
-
A common mistake that people make when trying to design
something completely foolproof is to underestimate
the ingenuity of complete fools.
Douglas Adams

I thinking rails just can''t handle running periodically_call_remote
with
an interval shorter than 1 second. eventually two calls are synced up 
and it deadlocks.  the faster I run periodically..  the sooner the 
deadlock.

If I run lighttpd as a load balancer with cluster would that alleviate 
the problem?  If that is the problem?

Scott

-- 
Posted via http://www.ruby-forum.com/.

On Dec 21, 2007, at 9:38 AM, Scott Derrick wrote:
> I thinking rails just can''t handle running
periodically_call_remote
> with
> an interval shorter than 1 second. eventually two calls are synced up
> and it deadlocks.  the faster I run periodically..  the sooner the
> deadlock.
Rails''s code is probably OK in this case -- periodically_call_remote  
isn''t inherently dangerous -- it just writes some javascript that  
periodically calls (via Prototype''s Ajax.Request) a controller /  
action. If you use Firebug and make the same request in the console  
(without using p_c_r), you''ll see the exact same problem after  
several (hundred?) requests.

The problem is probably (almost certainly) in either code you''ve  
written or in a library you''re using. gdb will help you out here.

Really, though, this is offtopic for Mongrel''s list -- either the  
Rails list or the Ruby list is a better match for these problems. The  
Rails IRC channel might also be useful. Or contact me off-list, and  
I''ll help if I can...
> If I run lighttpd as a load balancer with cluster would that alleviate
> the problem?  If that is the problem?
Nope; it''ll just put off the inevitable. You need to find and fix the  
deadlocking code. This can be challenging; race conditions are one of  
the least fun parts of programming.

Cheers,
-Nate

Nathan Vack wrote:
> Rails''s code is probably OK in this case --
periodically_call_remote
> isn''t inherently dangerous -- it just writes some javascript that
> periodically calls (via Prototype''s Ajax.Request) a controller /
> action. If you use Firebug and make the same request in the console
> (without using p_c_r), you''ll see the exact same problem after
> several (hundred?) requests.
> 
> The problem is probably (almost certainly) in either code you''ve
> written or in a library you''re using. gdb will help you out here.
I am making a  find(.. call which uses the sqlite3 gem
> Really, though, this is offtopic for Mongrel''s list -- either the
> Rails list or the Ruby list is a better match for these problems. The
> Rails IRC channel might also be useful. Or contact me off-list, and
> I''ll help if I can...
thats true if its not a mongrel problem. I am pursuing a solution on 
other forums.

thanks,

Scott
-- 
Posted via http://www.ruby-forum.com/.