Hello List, I have a production machine with very low ram (Xen Virtual Server) that runs mysql and lighty->pound->mongrel (2 instances). About once a day, one of the mongrels just locks up, leaving that in its log files: *** glibc detected *** /usr/bin/ruby: free(): invalid next size (fast): 0x0a744780 *** That process (mongrel) will stay locked, not answer any connections anymore and just generally behave bad; for instance kill -s SIGKILL PID will give me a ''Permission denied'', even though I am impersonating root. The only real solution I have had is to reboot the server, which is obviously very bad. Has this behavior been observed before? If not, what can I do to help tracking down this bug? Is it even a mongrel bug? I mean it could be that mongrel gets back at me because I feed it not enough RAM. But even so, It should remain sigkillable, should it not? Thank you, kaspar ruby 1.8.4 (2005-12-24) [i386-linux] mongrel (0.3.13.3) Rails 1.1.6
On Tue, 2006-08-22 at 22:00 +0200, Kaspar Schiess wrote:> Hello List, > > I have a production machine with very low ram (Xen Virtual Server) that > runs mysql and lighty->pound->mongrel (2 instances). About once a day, > one of the mongrels just locks up, leaving that in its log files: > > *** glibc detected *** /usr/bin/ruby: free(): invalid next size (fast): > 0x0a744780 ***That''s super deep in the ruby interpreter. My thinking is that you''ve just run out of ram and this is what happens to ruby when a malloc fails. Try out the latest pre-release to see if you still have the lock-ups: sudo gem install mongrel --source=http://mongrel.rubyforge.org/releases/ But the ram problem is only solved by adding more ram or swap. http://www.netadmintools.com/art1.html -- Zed A. Shaw http://www.zedshaw.com/ http://mongrel.rubyforge.org/ http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
Hi Zed, Thanks for the quick answer.> That''s super deep in the ruby interpreter. My thinking is that you''ve > just run out of ram and this is what happens to ruby when a malloc > fails. > > Try out the latest pre-release to see if you still have the lock-ups: > > sudo gem install mongrel > --source=http://mongrel.rubyforge.org/releases/I am now running the prerelease. I will keep you posted on what I find or not. Since this is a pretty hard to trace bug/problem (except that we both suspect its because of low ram), I can''t promise any results.> But the ram problem is only solved by adding more ram or swap. > > http://www.netadmintools.com/art1.htmlI have enough swap, but it seems that I can''t ever use all of it, since the ram is very scarce (128mb). kaspar neotrivium.com - the swiss ruby shop
> Try out the latest pre-release to see if you still have the lock-ups: > > sudo gem install mongrel > --source=http://mongrel.rubyforge.org/releases/Now I get /usr/lib/ruby/1.8/erb.rb:781: [BUG] Segmentation fault ruby 1.8.4 (2005-12-24) [i386-linux] That might mean that I''m onto something that has some connection to mongrel ;) kaspar
On Thu, 2006-08-24 at 09:19 +0200, Kaspar Schiess wrote:> > Try out the latest pre-release to see if you still have the lock-ups: > > > > sudo gem install mongrel > > --source=http://mongrel.rubyforge.org/releases/ > Now I get > > /usr/lib/ruby/1.8/erb.rb:781: [BUG] Segmentation fault > ruby 1.8.4 (2005-12-24) [i386-linux] > > That might mean that I''m onto something that has some connection to > mongrel ;)Hopefully you''re joking since that''s in erb.rb (not a mongrel file). Also, you''re the only person hitting these so you should look at what software you have that others might not have. Run you program under gdb and do a backtrace when it crashes to see what might cause it. -- Zed A. Shaw http://www.zedshaw.com/ http://mongrel.rubyforge.org/ http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
> Hopefully you''re joking since that''s in erb.rb (not a mongrel file). > Also, you''re the only person hitting these so you should look at what > software you have that others might not have. Run you program under gdb > and do a backtrace when it crashes to see what might cause it.I was joking - halfway. But since the error changes when I change mongrel version, it can''t be wholly unrelated neither. I don''t think I will find that error, even though I would like. Since the application it happens in is rather stable otherwise and it only seems to surface when memory runs out, it is rather hard to get to the bottom of this. gdb won''t help me there. Anyway, thanks for your help. Since nobody has hit on this before, I will continue to tweak variables until I''m out of danger zone. (Not config variables, meaning it more in the sense of a physical experiment) best regards, kaspar
I''d post that on the Xen mailing list or your hosting provider and see what they say. Linux isn''t exactly known for handling out of memory conditions very well, so who knows what might happen with Xen added to the picture in a case like this. I''ll bet it''s come up before though and someone familiar with Xen would have a clue what''s going on.