Hi, I have been using mongrel + Pound for the past 6 months on a single server deployment. Recently mongrel has started to behave very weirdly after we?moved app to a bunch of servers from a single server deployment. - It crashes on every exception, throws 500 and throws 500 for ALL instances of the cluster? after one exception has occurred - Connectivity to SQL seems to take a rather long time (the db is a different server on the same network, servers are connected through a gigaport switch) - Images deliveries have slowed down since we migrated from single server to multi server deployment (still using pound, not yet moved to apache / nginx) - The application architecture uses lot of background processes?- built using?DRB which again are deployed physically on different dedicated machines to distribute load - ?Mongrel doesn''t always log the error 500 within its logs (mongrel may not be getting these requests). -? Other mongrel instances in the same cluster do not respond even if one mongrel has crashed or has encountered an exception. Here is some basic info of the deployment 3 web servers - Dualcore Dell servers with Xeon 2.0 ghz processor running Cent OS 5 4 GB Ram Each 1 seperate components server - for threaded Drbs, same config as above 1 seperate database server (not yet clustered, same config as above) I would appreciate if anyone can analyze the issues and provide me a solution as it is becoming frustrating to have too much down time in the application regards Mohammed ________________________________________________________________________ You are invited to Get a Free AOL Email ID. - http://webmail.aol.in -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20080226/0a14f146/attachment-0001.html
What version of ruby are you running ruby -v On 26/02/2008, at 7:25 PM, mohammed.a at aol.in wrote:> 3 web servers - Dualcore Dell servers with Xeon 2.0 ghz processor > running Cent OS 5
Dave, Here is the version of Ruby along with other Gems installed ruby 1.8.6 (2007-03-13 patchlevel 0) [x86_64-linux] *** LOCAL GEMS *** abstract (1.0.0) ??? a library which enable you to define abstract method in Ruby actionmailer (1.3.6, 1.3.5) ??? Service layer for easy email delivery and testing. actionpack (1.13.6, 1.13.5) ??? Web-flow and rendering framework putting the VC in MVC. actionwebservice (1.2.6, 1.2.5) ??? Web service support for Action Pack. activerecord (1.15.6) ??? Implements the ActiveRecord pattern for ORM. activesupport (2.0.2, 1.4.4) ??? Support and utility classes used by the Rails framework. ap4r (0.3.5) ??? Asynchr onous Processing for Ruby. archive-tar-minitar (0.5.1) ??? Provides POSIX tarchive management from Ruby programs. builder (2.1.2) ??? Builders for MarkUp. cached_model (1.3.1) ??? An ActiveRecord abstract model that caches records in memcached capistrano (2.1.0) ??? Capistrano is a utility and framework for executing commands in ??? parallel on multiple remote machines, via SSH. cgi_multipart_eof_fix (2.5.0, 2.3) ??? Fix an exploitable bug in CGI multipart parsing. color (1.4.0) ??? Colour management with Ruby color-tools (1.3.0) ??? color-tools provides colour space definition and manpiulation as ??? well as commonly named RGB colours. daemo ns (1.0.9) ??? A toolkit to create and control daemons in different ways documatic (0.1.0) ??? Documatic is an OpenDocument extension for Ruby Reports (Ruport). It ??? is a template-driven formatter that can be used to produce ??? attractive printable documents such as database reports, invoices, ??? letters, faxes and more. erubis (2.4.1) ??? a fast and extensible eRuby implementation which supports ??? multi-language ezcrypto (0.7) ??? Simplified encryption library. fastercsv (1.2.3, 1.2.1) ??? FasterCSV is CSV, but faster, smaller, and cleaner. fastthread (1.0.1) ??? Optimized replacement for thread.rb primitives gem_plugin (0.2 .3) ??? A plugin system based on rubygems that uses dependencies only gruff (0.2.8) ??? Beautiful graphs for one or multiple datasets. highline (1.4.0) ??? HighLine is a high-level command-line IO library. hoe (1.5.0, 1.3.0) ??? Hoe is a simple rake/rubygems helper for project Rakefiles hpricot (0.6) ??? a swift, liberal HTML parser with a fantastic library localization_generator (1.0.8) ??? [Rails] Localization generator. log4r (1.0.5) ??? Log4r is a comprehensive and flexible logging library for Ruby. mailfactory (1.2.3) ??? MailFactory is a pure-ruby MIME mail generator memcache-client (1.5.0) ??? A Ruby memcached client merb (0 .4.1) ??? Merb == Mongrel + Erb. Pocket rocket web framework. mime-types (1.15) ??? Manages a MIME Content-Type that will return the Content-Type for a ??? given filename. mongrel (1.1.1) ??? A small fast HTTP library and server that runs Rails, Camping, Nitro ??? and Iowa apps. mongrel_cluster (1.0.5) ??? Mongrel plugin that provides commands and Capistrano tasks for ??? managing multiple Mongrel processes. needle (1.3.0) ??? Needle is a Dependency Injection/Inversion of Control container for ??? Ruby. It supports both type-2 (setter) and type-3 (constructor) ??? injection. It takes advantage of the dynamic nature of Ruby 20to ??? provide a rich and flexible approach to injecting dependencies. net-sftp (1.1.0) ??? Net::SFTP is a pure-Ruby implementation of the SFTP client protocol. net-ssh (1.1.2) ??? Net::SSH is a pure-Ruby implementation of the SSH2 client protocol. pdf-writer (1.1.7) ??? A pure Ruby PDF document creation library. rails (1.2.6) ??? Web-application framework with template engine, control-flow layer, ??? and ORM. rails_analyzer_tools (1.4.0) ??? Tools for analyzing the performance of web sites. railsmachine (1.0.0) ??? The Rails Machine task library rake (0.8.1, 0.7.3) ??? Ruby based make-like utility. RedCloth (3.0.4) ??? RedCl oth is a module for using Textile and Markdown in Ruby. Textile ??? and Markdown are text formats.? A very simple text format. Another ??? stab at making readable text that can be converted to HTML. reliable-msg (1.1.0) ??? Reliable messaging and persistent queues for building asynchronous ??? applications in Ruby rmagick (2.2.2) ??? Ruby binding to ImageMagick rubigen (1.2.3, 1.1.1) ??? A framework to allow Ruby applications to generate file/folder stubs ??? (like the rails command does for Ruby on Rails, and the ??? ?script/generate? command within a Rails application during ??? development). ruby-debug (0.10.0) ?? C2 Command line interface (CLI) for ruby-debug-base ruby-debug-base (0.10.0, 0.9.3) ??? Fast Ruby debugger - core component ruby-debug-ide (0.1.9) ??? IDE interface for ruby-debug. rubyforge (0.4.4) ??? A script which automates a limited set of rubyforge operations. rubyzip (0.9.1) ??? rubyzip is a ruby module for reading and writing zip files ruport (1.4.0) ??? A generalized Ruby report generation and templating engine. ruport-util (0.8.0) ??? A set of tools and helper libs for Ruby Reports salted_login_generator (1.1.1) ??? [Rails] Login generator with salted passwords. scruffy (0.2.2) ??? A powerful, clean graphing library for Ruby. sources (0.0.1) ??? This package provides download sources for remote gem installation sql_dep_graph (1.0.0) ??? Graphs table dependencies based on usage from SQL logs SyslogLogger (1.4.0) ??? SyslogLogger is a Logger replacement that logs to syslog.? It is ??? almost drop-in with a few caveats. tlsmail (0.0.1) ??? This library enables pop or smtp via ssl/tls by dynamically ??? replacing these classes to these in ruby 1.9. transaction-simple (1.4.0) ??? Simple object transaction support for Ruby. uuid (1.0.4) ??? UUID generator ZenTest (3.9.1, 3.6.1) ??? ZenTest provides 4 different tools and 1 library: zentest, ??? unit_diff , autotest, multiruby, and Test::Rails regards Mohammed -----Original Message----- From: Dave Cheney <dave at cheney.net> To: mongrel-users at rubyforge.org Sent: Tue, 26 Feb 2008 2:35 pm Subject: Re: [Mongrel] problems with Mongrel + Pound What version of ruby are you running ruby -v On 26/02/2008, at 7:25 PM, mohammed.a at aol.in wrote:> 3 web servers - Dualcore Dell servers with Xeon 2.0 ghz processorrunning Cent OS 5 _______________________________________________ ongrel-users mailing list ongrel-users at rubyforge.org ttp://rubyforge.org/mailman/listinfo/mongrel-users ________________________________________________________________________ You are invited to Get a Free AOL Email ID. - http://webmail.aol.in -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20080226/ffd1e69f/attachment.html
Hi~ On Feb 26, 2008, at 2:44 AM, mohammed.a at aol.in wrote:> Dave, > > Here is the version of Ruby along with other Gems installed > > ruby 1.8.6 (2007-03-13 patchlevel 0) [x86_64-linux]This version of ruby has serious bugs! You need to either down grade to ruby 1.8.5 or upgrade to ruby 1.8.6 p 111. You need at least patch level 110 or else do not use 1.8.6 Cheers- - Ezra Zygmuntowicz -- Founder & Software Architect -- ezra at engineyard.com -- EngineYard.com
On 26 Feb 2008, at 19:53, Ezra Zygmuntowicz wrote:>> ruby 1.8.6 (2007-03-13 patchlevel 0) [x86_64-linux] > > This version of ruby has serious bugs! You need to either down grade > to ruby 1.8.5 or upgrade to ruby 1.8.6 p 111. You need at least patch > level 110 or else do not use 1.8.6Thanks for that tip -- I checked my Ruby installation and found it was that very same buggy version. I hadn''t realised it was flawed.... Regards, Andy Stewart ------- http://airbladesoftware.com
Zed, Emobius, Thank you for the replies. Here''s what I found - mysql gem was not installed and that maybe a core source of problems as suggested by Zed. We''ve had? too many problems when we tried to install this, and found that none of the solutions that are mentioned really work. We will try to install this gem again. An application that we built using DRB designed to be asynchronous failed to process asynchronous requests, had each instance of ruby piling up request queues and subsequently throwing errors (as all ports were getting requests continuously). I will be keen to get inputs from anyone who has used DRB instead of AP4R for asynchronous requests Ruby version is 1.8.6 (we''re upgrading this today night) Ngnix - we''re trying this out today on our staging machines and after it is found to be stable, we will move to production. Temporarily, we''ve moved from DRB to AP4R for asynchronous processing. Personally, I am still comfortable with DRB than AP4R but that is another discussion altogether. ---- Another question: Are there benchmarked transaction statistics on standard hardware for ROR / mongrel platform (similar to whats available on TPC.org?). We''re looking at benchmarking our system to initially support 10m requests per day (consider each one to be a simple find_by_sql query against a database of 1000 requests + 1 asynchronous process for billing. regards Mohammed. A -----Original Message----- From: Andrew Stewart <boss at airbladesoftware.com> To: mongrel-users at rubyforge.org Sent: Wed, 27 Feb 2008 1:40 am Subject: Re: [Mongrel] problems with Mongrel + Pound On 26 Feb 2008, at 19:53, Ezra Zygmuntowicz wrote:>> ruby 1.8.6 (2007-03-13 patchlevel 0) [x86_64-linux] > > This version of ruby has serious bugs! You need to either down grade > to ruby 1.8.5 or upgrade to ruby 1.8.6 p 111. You need at least patch > level 110 or else do not use 1.8.6Thanks for that tip -- I checked my Ruby installation and found it was that very same buggy version. I hadn''t realised it was flawed.... Regards, Andy Stewart ------- http://airbladesoftware.com _______________________________________________ Mongrel-users mailing list Mongrel-users at rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-users ________________________________________________________________________ You are invited to Get a Free AOL Email ID. - http://webmail.aol.in -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20080227/f4122624/attachment.html
I hacked our vendor tree to explode on startup if the mysql gem could not be found. IMHO this should be the default behavior, the built in mysql adapter is crap and its silent use is a ticking time bomb Cheers Dave On 27/02/2008, at 6:42 PM, mohammed.a at aol.in wrote:> Thank you for the replies. > > Here''s what I found - mysql gem was not installed and that maybe a > core source of problems as suggested by Zed. We''ve had too many > problems when we tried to install this, and found that none of the > solutions that are mentioned really work. We will try to install > this gem again.-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20080227/fd367a4d/attachment.html
To install mysql gem you need to specify the mysql_config path(if it is not installed then into mysql dev or devel package) To install sudo gem install mysql -- --with-mysql-config=<path-to-mysql_config like on my ubuntu it is /usr/bin/mysql_config> And your problems are similar(not same) I used to encounter. I had these problems with my mongrel handler when it used to write to database in asynchronous queries by the users. Log file used to scream connection lost to mysql. After spending a few days trying to boost up my mysql server I did the gem install and since then it has handled 100K + queries (the mongrel handler part of the app)each day without any problem whatsoever. Also switch to apache/ngnix if you can. Pound could also be the trouble maker though I am not very sure about this comment. On Wed, Feb 27, 2008 at 1:29 PM, Dave Cheney <dave at cheney.net> wrote:> I hacked our vendor tree to explode on startup if the mysql gem could not > be found. IMHO this should be the default behavior, the built in mysql > adapter is crap and its silent use is a ticking time bomb > Cheers > > Dave > > On 27/02/2008, at 6:42 PM, mohammed.a at aol.in wrote: > > Thank you for the replies. > > Here''s what I found - mysql gem was not installed and that maybe a core > source of problems as suggested by Zed. We''ve had too many problems when we > tried to install this, and found that none of the solutions that are > mentioned really work. We will try to install this gem again. > > > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20080227/d2950aa6/attachment.html
On Tue, 26 Feb 2008 03:25:46 -0500 mohammed.a at aol.in wrote:> Hi, > > I have been using mongrel + Pound for the past 6 months on a single server deployment. > > Recently mongrel has started to behave very weirdly after we?moved app to a bunch of servers from a single server deployment.These answers might help:> - It crashes on every exception, throws 500 and throws 500 for ALL instances of the cluster? after one exception has occurredFor the crashes and database problems, make sure you have the mysql or postgresql gem installed and that it''s using the actual binary library not the Rails built-in stuff. The built-in driver for those databases is bad and causes exceptions, lost connections, etc.> - Connectivity to SQL seems to take a rather long time (the db is a different server on the same network, servers are connected through a gigaport switch)Rule out your network config before you assume it''s a *Rails* issue (not a mongrel issue). Try running a few simple scripts and test the perf, then start from the most basic network config like half-duplex and see what you can find.> - Images deliveries have slowed down since we migrated from single server to multi server deployment (still using pound, not yet moved to apache / nginx)Yeah, time to upgrade to nginx. Nginx is actually easier to configure than pound, makes more sense in the config, and is a hell of a lot faster and stable. Especially for having it serve static files. Just drop pound.> - The application architecture uses lot of background processes?- built using?DRB which again are deployed physically on different dedicated machines to distribute loadYeah, this really makes me wonder what you''re doing. I''d go and double and triple check every line of code and watch network traffic to make sure you aren''t doing something dumb like sending huge files over DRb. I''m guessing you''ve got something dumb that''s overloading the Ruby interpreter.> - ?Mongrel doesn''t always log the error 500 within its logs (mongrel may not be getting these requests).Mongrel doesn''t log most stuff, rails does. Check in the production.log file and see what''s there.> -? Other mongrel instances in the same cluster do not respond even if one mongrel has crashed or has encountered an exception.That sounds like Pound more than anything. Isolate the difference between a connection to pound and one to mongrel.> Here is some basic info of the deploymentWith this much hardware you should be able to crank tons of requests through the system. If you have this many problems, it''s entirely in your design. Remember, tons and tons of other people do way more with much less. -- Zed A. Shaw - Hate: http://savingtheinternetwithhate.com/ - Good: http://www.zedshaw.com/ - Evil: http://yearofevil.com/
Switching to NGINX indeed fixed much of the problems. But mysql gem continues to evade no matter what kind of config I use. But SQL has been tuned up, major bugs have been tuned. Much better now. Will post later with detailed analysis of the problem and solutions regards Mohammed. A -----Original Message----- From: Piyush Ranjan <piyush.pr at gmail.com> To: mongrel-users at rubyforge.org Sent: Wed, 27 Feb 2008 1:50 pm Subject: Re: [Mongrel] problems with Mongrel + Pound To install mysql gem you need to specify the mysql_config path(if it is not installed then into mysql dev or devel package) To install sudo gem install mysql -- --with-mysql-config=<path-to-mysql_config like on my ubuntu it is /usr/bin/mysql_config> And your problems are similar(not same) I used to encounter. I had these problems with my mongrel handler when it used to? write to database in asynchronous queries by the users. Log file used to scream connection lost to mysql. After spending a few days trying to boost up my mysql server I did the gem install and since then it has handled 100K + queries (the mongrel handler part of the app)each day without any problem whatsoever. Also switch to apache/ngnix if you can. Pound could also be the trouble maker though I am not very sure about this comment. On Wed, Feb 27, 2008 at 1:29 PM, Dave Cheney <dave at cheney.net> wrote: I hacked our vendor tree to explode on startup if the mysql gem could not be found. IMHO this should be the default?behavior,?the built in mysql adapter is crap and its silent use is a ticking time bomb Cheers Dave On 27/02/2008, at 6:42 PM, mohammed.a at aol.in wrote: Thank you for the replies. Here''s what I found - mysql gem was not installed and that maybe a core source of problems as suggested by Zed. We''ve had? too many problems when we tried to install this, and found that none of the solutions that are mentioned really work. We will try to install this gem again.? _______________________________________________ Mongrel-users mailing list Mongrel-users at rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-users _______________________________________________ Mongrel-users mailing list Mongrel-users at rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-users ________________________________________________________________________ You are invited to Get a Free AOL Email ID. - http://webmail.aol.in -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20080305/b37322c5/attachment-0001.html