Hello all,
I''m new to Mongrel and am very impressed with what I see so far. This
thing beats FastCGI hands down, and is even simpler to use than SCGI.
Great job Zed! Thanks for all the hard work.
I am a bit surprised at the memory usage patterns though. I''m not
sure if I''m doing something wrong (most likely I am), or if this is a
problem with the app I''m running (Typo trunk), but after the initial
mongrel_rails startup, each server uses about 22MB of memory.
However, after serving pages for a day or so, that memory usage will
have increased from 22MB to ~40MB. If I leave the server running, it
just keeps climbing. My memory usage with FastCGI was pretty stable
at about 22-24MB per dispatcher, regardless of how long they''d been
running. I''m far from a high-traffic site, so this is a bit
unexpected.
I''ve tried this both in a single-server setup and using
mongrel_cluster, and the results are similar. I''m running on Debian
testing behind LiteSpeed web server. I keep up to date with Typo
trunk, and am using the latest gems for mongrel, mongrel_cluster, and
sendfile. I tried removing sendfile, and that slowed things down, but
didn''t improve the memory consumption, so I put it back in.
I''ve used both httperf and ab to test the app server, and they both
show the memory consumption problem. I''ve done the testing both
through LiteSpeed as the proxy and by connecting directly to the
port(s) mongrel is running on. When starting mongrel_rails, I use -e
production, and the cluster setup also specifies production for the
environment. Just to be doubly sure, I also set RAILS_ENV to
production on the command line and force it to production in
config/environment.rb as well. None of these things seems to help.
Some system information:
burks at cracked:~$ uname -a
Linux www.jasonburks.com 2.6.16.13-xenU #1 SMP Sat May 13 14:18:58 EDT
2006 i686 GNU/Linux
burks at cracked:~$ ruby -v
ruby 1.8.4 (2005-12-24) [i486-linux]
burks at cracked:~$ gem list
*** LOCAL GEMS ***
actionmailer (1.2.1, 1.2.0, 1.1.5, 1.1.3)
Service layer for easy email delivery and testing.
actionpack (1.12.1, 1.12.0, 1.11.2, 1.11.0)
Web-flow and rendering framework putting the VC in MVC.
actionwebservice (1.1.2, 1.1.0, 1.0.0, 0.9.3)
Web service support for Action Pack.
activerecord (1.14.2, 1.14.0, 1.13.2, 1.13.0)
Implements the ActiveRecord pattern for ORM.
activesupport (1.3.1, 1.3.0, 1.2.5, 1.2.3)
Support and utility classes used by the Rails framework.
cmdparse (2.0.2, 2.0.0)
Advanced command line parser supporting commands
daemons (0.4.4)
A toolkit to create and control daemons in different ways
fcgi (0.8.6.1)
FastCGI ruby binding.
gem_plugin (0.2.1)
A plugin system based only on rubygems that uses dependencies only
highline (1.2.1, 1.2.0, 1.0.1)
HighLine is a high-level command-line IO library.
mongrel (0.3.13.2, 0.3.13.1, 0.3.13)
A small fast HTTP library and server that runs Rails, Camping, and
Nitro apps.
mongrel_cluster (0.2.0)
Mongrel plugin that provides commands and Capistrano tasks for
managing multiple Mongrel processes.
rails (1.1.2, 1.1.0, 1.0.0, 0.14.3)
Web-application framework with template engine, control-flow layer,
and ORM.
rake (0.7.1, 0.6.2)
Ruby based make-like utility.
sendfile (0.9.2)
Ruby interface to sendfile(2) system call
sources (0.0.1)
This package provides download sources for remote gem installation
Now, if I''m not mistaken, Typo trunk uses it''s own version of
Rails as
an external resource. I checked, and I''m at svn 4511 of Rails in
vendor/rails in my Typo install. My cluster config looks like this:
burks at cracked:/var/www/typo/htdocs$ cat config/mongrel_cluster.yml
---
user: nobody
port: "8000"
environment: production
group: www-edit
address: 127.0.0.1
pid_file: log/mongrel.pid
servers: 2
I''m really not sure what else I should be looking at to try to solve
the memory issues. I can certainly set up a cron job to restart the
service every day, but I shouldn''t think this is necessary. The
following message (with different action names) shows up a few times
in log/production.log:
ActionController::UnknownAction (No action responded to 2005):
/vendor/rails/actionpack/lib/action_controller/filters.rb:368:in
`perform_action_without_benchmark''
/vendor/rails/actionpack/lib/action_controller/benchmarking.rb:69:in
`perform_action_without_rescue''
/usr/lib/ruby/1.8/benchmark.rb:293:in `measure''
/vendor/rails/actionpack/lib/action_controller/benchmarking.rb:69:in
`perform_action_without_rescue''
/vendor/rails/actionpack/lib/action_controller/rescue.rb:82:in
`perform_action''
/vendor/rails/actionpack/lib/action_controller/base.rb:381:in
`process_without_filters''
/vendor/rails/actionpack/lib/action_controller/filters.rb:377:in
`process_without_session_management_support''
/vendor/rails/actionpack/lib/action_controller/session_management.rb:117:in
`process''
/vendor/rails/railties/lib/dispatcher.rb:38:in `dispatch''
/usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.2/lib/mongrel/rails.rb:85:in
`process''
/usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.2/lib/mongrel.rb:563:in
`process_client''
/usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.2/lib/mongrel.rb:562:in
`process_client''
/usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.2/lib/mongrel.rb:648:in
`run''
/usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.2/lib/mongrel.rb:648:in
`run''
/usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.2/lib/mongrel.rb:637:in
`run''
/usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.2/lib/mongrel.rb:969:in
`run''
/usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.2/lib/mongrel.rb:968:in
`run''
/usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.2/bin/mongrel_rails:119:in
`run''
/usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.2/lib/mongrel/command.rb:211:in
`run''
/usr/lib/ruby/gems/1.8/gems/mongrel-0.3.13.2/bin/mongrel_rails:227
/usr/bin/mongrel_rails:18
log/mongrel.log shows the following a whole bunch (94,000+ times in
the last week):
Thu Jun 29 02:53:07 EDT 2006: BAD CLIENT (127.0.0.1): Invalid HTTP
format, parsing fails.
I haven''t turned on debug mode yet because I wasn''t sure what
sort of
info might be helpful for troubleshooting this. Google didn''t seem to
yield much in the way of similar problems, and I''m far from a Ruby or
Rails expert, so I''m not really sure how to proceed. Any guidance or
suggestions or requests for additional debugging information to assist
me in stabilizing my memory usage would be greatly appreciated.
Again, Zed and others, great work! I''m really impressed with Mongrel
so far. Please help me to keep my doggie lean and happy...
Regards,
Jason Burks
On Thu, 2006-06-29 at 08:14 -0400, Jason Burks wrote:> Hello all, > > I''m new to Mongrel and am very impressed with what I see so far. This > thing beats FastCGI hands down, and is even simpler to use than SCGI. > Great job Zed! Thanks for all the hard work. >Thanks Jason, don''t forget that Luis did a lot of work too and other folks like _why, Rick Olsen, and many others contributed code or documentation. Just wanted to throw that in since many times people say I rock, but really it''s all the people helping out that make Mongrel rock.> I am a bit surprised at the memory usage patterns though. I''m not > sure if I''m doing something wrong (most likely I am), or if this is a > problem with the app I''m running (Typo trunk), but after the initial > mongrel_rails startup, each server uses about 22MB of memory. > However, after serving pages for a day or so, that memory usage will > have increased from 22MB to ~40MB. If I leave the server running, it > just keeps climbing. My memory usage with FastCGI was pretty stable > at about 22-24MB per dispatcher, regardless of how long they''d been > running. I''m far from a high-traffic site, so this is a bit > unexpected.So, with memory usage you''ve got a few things potentially going on: 1) Ruby''s GC just isn''t all that great. With FastCGI they actually had this lame "run the GC after X requests" thing, but Mongrel''s just Ruby (mostly) so having that just seems wrong. 2) Typo actually does have a leak. This is pretty unlikely since you say it ran under fastcgi. I''m still betting on #1. 3) Mongrel has a leak. Probably not, there''s not a lot in Mongrel to leak, and many people use it without any problems. So, what I''d recommend is actually installing Monit and configuring it to watch your Mongrel processes for memory and CPU usage. Monit will actually e-mail you when they reach certain limits and can even restart them. Let this run for a while and keep the e-mails that monit sends so you can get a sense of how long before your application start to use too much ram. I''m thinking your issue is #1 (GC) so the ram will increase and then just go down, which is normal. Finally, try just throwing in a call to force the GC to run after certain requests and that might help you prove/disprove the above. -- Zed A. Shaw http://www.zedshaw.com/ http://mongrel.rubyforge.org/ http://www.railsmachine.com/ -- Need Mongrel support?
Zed, Thanks for the feedback, I''ll give it a try and report results. I''m on vacation all of next week and may not be able to send anything until I return. Also, thanks for the correction on Mongrel contributors, and thanks to all of your for a great little app server. -Jason On 6/29/06, Zed Shaw <zedshaw at zedshaw.com> wrote:> On Thu, 2006-06-29 at 08:14 -0400, Jason Burks wrote: > > Hello all, > > > > I''m new to Mongrel and am very impressed with what I see so far. This > > thing beats FastCGI hands down, and is even simpler to use than SCGI. > > Great job Zed! Thanks for all the hard work. > > > Thanks Jason, don''t forget that Luis did a lot of work too and other > folks like _why, Rick Olsen, and many others contributed code or > documentation. Just wanted to throw that in since many times people say > I rock, but really it''s all the people helping out that make Mongrel > rock. > > > I am a bit surprised at the memory usage patterns though. I''m not > > sure if I''m doing something wrong (most likely I am), or if this is a > > problem with the app I''m running (Typo trunk), but after the initial > > mongrel_rails startup, each server uses about 22MB of memory. > > However, after serving pages for a day or so, that memory usage will > > have increased from 22MB to ~40MB. If I leave the server running, it > > just keeps climbing. My memory usage with FastCGI was pretty stable > > at about 22-24MB per dispatcher, regardless of how long they''d been > > running. I''m far from a high-traffic site, so this is a bit > > unexpected. > > So, with memory usage you''ve got a few things potentially going on: > > 1) Ruby''s GC just isn''t all that great. With FastCGI they actually had > this lame "run the GC after X requests" thing, but Mongrel''s just Ruby > (mostly) so having that just seems wrong. > 2) Typo actually does have a leak. This is pretty unlikely since you > say it ran under fastcgi. I''m still betting on #1. > 3) Mongrel has a leak. Probably not, there''s not a lot in Mongrel to > leak, and many people use it without any problems. > > So, what I''d recommend is actually installing Monit and configuring it > to watch your Mongrel processes for memory and CPU usage. Monit will > actually e-mail you when they reach certain limits and can even restart > them. Let this run for a while and keep the e-mails that monit sends so > you can get a sense of how long before your application start to use too > much ram. > > I''m thinking your issue is #1 (GC) so the ram will increase and then > just go down, which is normal. > > Finally, try just throwing in a call to force the GC to run after > certain requests and that might help you prove/disprove the above. > > > -- > Zed A. Shaw > http://www.zedshaw.com/ > http://mongrel.rubyforge.org/ > http://www.railsmachine.com/ -- Need Mongrel support? > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >
An update on this situation. Typo has seen a great deal of development in the last two weeks. Running the latest svn as of today seems to yield *much* more stable memory usage patterns. Even after hitting my site with a few passes of ab and httperf, the memory usage per server instance is only 35-37MB. Much better than the 4X that amount that I had been seeing. It appears that Typo fixed something (or else the newer version of Rails they''re including in svn did). Either way, it looks like I''m much better off now. Thanks for the suggestions and guidance, and keep up the great work. -Jason On 6/29/06, Jason Burks <jburks725 at gmail.com> wrote:> Zed, > > Thanks for the feedback, I''ll give it a try and report results. I''m > on vacation all of next week and may not be able to send anything > until I return. > > Also, thanks for the correction on Mongrel contributors, and thanks to > all of your for a great little app server. > > -Jason > > On 6/29/06, Zed Shaw <zedshaw at zedshaw.com> wrote: > > On Thu, 2006-06-29 at 08:14 -0400, Jason Burks wrote: > > > Hello all, > > > > > > I''m new to Mongrel and am very impressed with what I see so far. This > > > thing beats FastCGI hands down, and is even simpler to use than SCGI. > > > Great job Zed! Thanks for all the hard work. > > > > > Thanks Jason, don''t forget that Luis did a lot of work too and other > > folks like _why, Rick Olsen, and many others contributed code or > > documentation. Just wanted to throw that in since many times people say > > I rock, but really it''s all the people helping out that make Mongrel > > rock. > > > > > I am a bit surprised at the memory usage patterns though. I''m not > > > sure if I''m doing something wrong (most likely I am), or if this is a > > > problem with the app I''m running (Typo trunk), but after the initial > > > mongrel_rails startup, each server uses about 22MB of memory. > > > However, after serving pages for a day or so, that memory usage will > > > have increased from 22MB to ~40MB. If I leave the server running, it > > > just keeps climbing. My memory usage with FastCGI was pretty stable > > > at about 22-24MB per dispatcher, regardless of how long they''d been > > > running. I''m far from a high-traffic site, so this is a bit > > > unexpected. > > > > So, with memory usage you''ve got a few things potentially going on: > > > > 1) Ruby''s GC just isn''t all that great. With FastCGI they actually had > > this lame "run the GC after X requests" thing, but Mongrel''s just Ruby > > (mostly) so having that just seems wrong. > > 2) Typo actually does have a leak. This is pretty unlikely since you > > say it ran under fastcgi. I''m still betting on #1. > > 3) Mongrel has a leak. Probably not, there''s not a lot in Mongrel to > > leak, and many people use it without any problems. > > > > So, what I''d recommend is actually installing Monit and configuring it > > to watch your Mongrel processes for memory and CPU usage. Monit will > > actually e-mail you when they reach certain limits and can even restart > > them. Let this run for a while and keep the e-mails that monit sends so > > you can get a sense of how long before your application start to use too > > much ram. > > > > I''m thinking your issue is #1 (GC) so the ram will increase and then > > just go down, which is normal. > > > > Finally, try just throwing in a call to force the GC to run after > > certain requests and that might help you prove/disprove the above. > > > > > > -- > > Zed A. Shaw > > http://www.zedshaw.com/ > > http://mongrel.rubyforge.org/ > > http://www.railsmachine.com/ -- Need Mongrel support? > > > > _______________________________________________ > > Mongrel-users mailing list > > Mongrel-users at rubyforge.org > > http://rubyforge.org/mailman/listinfo/mongrel-users > > >
> It appears that Typo fixed something (or else the newer version of > Rails they''re including in svn did). Either way, it looks like I''m > much better off now.Scott fixed the "Great Typo Memory Leak of 2006": http://scottstuff.net/blog/articles/2006/07/08/ding-dong-the-memory- leak-in-typo-is-dead I upgraded my personal blog a few days ago and my doggies are at 27mb each. Regards, Bradley Taylor ------ Rails Optimized Hosting ~ VPS and Dedicated Servers Simplified Deployment ~ Services and Software http://railsmachine.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20060711/80748265/attachment-0001.html
On 11/07/06, Bradley Taylor <bradley at railsmachine.com> wrote:> I upgraded my personal blog a few days ago and my doggies are at 27mb each.Mine are behaving much better now as well. Cheers, -- Dave Murphy (Schwuk) http://schwuk.com