We have a site that is running Mongrel and is down. Below is the Mongrel log file information. The server is a Linux server with 2gb of RAM. EV1 (hosting site) says that since this is a Linux server that the RAM is cached and that there is plenty of free memory available. Can anyone make sense of this error and have some suggestion of options that I may take to correct this issue and get our site up and running? Below is the Mongrel.log file: ** Daemonized, any open files are closed. Look at log/mongrel.3010.pid and log/mongrel.log for info. ** Starting Mongrel listening at 127.0.0.1:3010 ** Starting Rails with production environment... ** Rails loaded. ** Loading any Rails specific GemPlugins ** Signals ready. TERM => stop. USR2 => restart. INT => stop (no restart). ** Rails signals registered. HUP => reload (without restart). It might not work well. ** Mongrel available at 127.0.0.1:3010 [FATAL] failed to allocate memory Here is the server information: General CPU GenuineIntel, Intel(R) Pentium(R) 4 CPU 3.06GHz Version psa v7.5.4_build75050824.12 os_RedHat el3 OS Linux 2.4.20-021stab028.3.777-smp Key number PLSK.00317196.0002 System Uptime: 02:40 CPU usage Last 1 minute 0.24 Last 5 minutes 0.38 Last 15 minutes 0.38 Memory Usage Total Used Free Shared Buffer Cached Usage 1.97 GB 1.85 GB 123.08 MB 0 B 249.13 MB 726.05 MB 93.9% Swap usage Total Used Free Usage 1.95 GB 736.53 MB 1.23 GB 36.82% Hard Disk Usage Filesystem Total Used Available Capacity /dev/vzfs 15 360.00 MB 2 677.79 MB 12 682.21 MB 17.43% Directories /usr/local/psa /var/lib/mysql /var/www/vhosts /var/qmail/mailnames /var/named/run-root proc 0.00 MB 0.00 MB 0.00 MB 0% devpts 0.00 MB 0.00 MB 0.00 MB 0% Domains Active 3 Problem 0 Passive 0 Thanks!! Roy -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20070221/cfb3660b/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 104 bytes Desc: image001.gif Url : http://rubyforge.org/pipermail/mongrel-users/attachments/20070221/cfb3660b/attachment-0005.gif -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 70 bytes Desc: image002.gif Url : http://rubyforge.org/pipermail/mongrel-users/attachments/20070221/cfb3660b/attachment-0006.gif -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 54 bytes Desc: image003.gif Url : http://rubyforge.org/pipermail/mongrel-users/attachments/20070221/cfb3660b/attachment-0007.gif -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 100 bytes Desc: image004.gif Url : http://rubyforge.org/pipermail/mongrel-users/attachments/20070221/cfb3660b/attachment-0008.gif -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 76 bytes Desc: image005.gif Url : http://rubyforge.org/pipermail/mongrel-users/attachments/20070221/cfb3660b/attachment-0009.gif
Did you look at these files from your log output? You should include those, and the rails log if you have anything there. Look at log/mongrel.3010.pid and log/mongrel.log for info. - Rob On 2/21/07, Roy Satterfield <roys at okabashi.com> wrote:> > We have a site that is running Mongrel and is down. > > Below is the Mongrel log file information. > > > > The server is a Linux server with 2gb of RAM. > > EV1 (hosting site) says that since this is a Linux server that the RAM is > cached and that there is plenty of free memory available. > > > > Can anyone make sense of this error and have some suggestion of options > that I may take to correct this issue and get our site up and running? > > > > Below is the Mongrel.log file: > > ** Daemonized, any open files are closed. Look at log/mongrel.3010.pid > and log/mongrel.log for info. > > ** Starting Mongrel listening at 127.0.0.1:3010 > > ** Starting Rails with production environment... > > ** Rails loaded. > > ** Loading any Rails specific GemPlugins > > ** Signals ready. TERM => stop. USR2 => restart. INT => stop (no > restart). > > ** Rails signals registered. HUP => reload (without restart). It might > not work well. > > ** Mongrel available at 127.0.0.1:3010 > > [FATAL] failed to allocate memory > > > > Here is the server information: > > > > General > > CPU > > GenuineIntel, Intel(R) Pentium(R) 4 CPU 3.06GHz > > Version > > psa v7.5.4_build75050824.12 os_RedHat el3 > > OS > > Linux 2.4.20-021stab028.3.777-smp > > Key number > > PLSK.00317196.0002 > > System Uptime: > > 02:40 > > CPU usage > > Last 1 minute > > 0.24 > > Last 5 minutes > > 0.38 > > Last 15 minutes > > 0.38 > > Memory Usage > > *Total* > > *Used* > > *Free* > > *Shared* > > *Buffer* > > *Cached* > > *Usage* > > 1.97 GB > > 1.85 GB > > 123.08 MB > > 0 B > > 249.13 MB > > 726.05 MB > > 93.9% > > [image: Bar Begin] > > [image: 93.9% used] > > [image: 6.1% available] > > [image: Bar End] > > Swap usage > > *Total* > > *Used* > > *Free* > > *Usage* > > 1.95 GB > > 736.53 MB > > 1.23 GB > > 36.82% > > [image: Bar Begin] > > [image: 36.82% used] > > [image: 63.18% available] > > [image: Bar End] > > Hard Disk Usage > > *Filesystem* > > *Total* > > *Used* > > *Available* > > *Capacity* > > /dev/vzfs > > 15 360.00 MB > > 2 677.79 MB > > 12 682.21 MB > > 17.43% > > [image: Bar Begin] > > [image: 17.43% used] > > [image: 82.57% available] > > [image: Bar End] > > Directories > > /usr/local/psa > /var/lib/mysql > /var/www/vhosts > /var/qmail/mailnames > /var/named/run-root > > proc > > 0.00 MB > > 0.00 MB > > 0.00 MB > > 0% > > [image: Bar Begin] > > [image: 100% available] > > [image: Bar End] > > devpts > > 0.00 MB > > 0.00 MB > > 0.00 MB > > 0% > > [image: Bar Begin] > > [image: 100% available] > > [image: Bar End] > > Domains > > Active > > 3 > > Problem > > 0 > > Passive > > 0 > > > > > > > > Thanks!! > > Roy > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20070221/872292f0/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 104 bytes Desc: not available Url : http://rubyforge.org/pipermail/mongrel-users/attachments/20070221/872292f0/attachment-0005.gif -------------- next part -------------- A non-text attachment was scrubbed... Name: image005.gif Type: image/gif Size: 76 bytes Desc: not available Url : http://rubyforge.org/pipermail/mongrel-users/attachments/20070221/872292f0/attachment-0006.gif -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.gif Type: image/gif Size: 54 bytes Desc: not available Url : http://rubyforge.org/pipermail/mongrel-users/attachments/20070221/872292f0/attachment-0007.gif -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.gif Type: image/gif Size: 100 bytes Desc: not available Url : http://rubyforge.org/pipermail/mongrel-users/attachments/20070221/872292f0/attachment-0008.gif -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.gif Type: image/gif Size: 70 bytes Desc: not available Url : http://rubyforge.org/pipermail/mongrel-users/attachments/20070221/872292f0/attachment-0009.gif
Zed, Are you saying that this issue is caused by there not being enough RAM on the server? There are 2 sites running on this server. EV1 restarted the VPS and I took my other site down.... Even with the other site down this site would not restart. But the weird part is the other site will restart without any problems. So if it is a memory problem then why will one site start properly? Thanks, Roy -----Original Message----- From: mongrel-users-bounces at rubyforge.org [mailto:mongrel-users-bounces at rubyforge.org] On Behalf Of Zed A. Shaw Sent: Wednesday, February 21, 2007 6:33 PM To: mongrel-users at rubyforge.org Subject: Re: [Mongrel] Critical Issue - Site down... On Wed, 21 Feb 2007 14:47:41 -0500 "Roy Satterfield" <roys at okabashi.com> wrote:> [FATAL] failed to allocate memoryYou ran out of memory. -- Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu http://www.zedshaw.com/ http://www.awprofessional.com/title/0321483502 -- The Mongrel Book http://mongrel.rubyforge.org/ http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help. _______________________________________________ Mongrel-users mailing list Mongrel-users at rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-users
The graphic that you sent has 123.08 MB of RAM free (not too much), and you''re exhausting that with the single mongrel process. Regards, Jason On Feb 21, 2007, at 12:15 PM, Rob Sanheim wrote:> Did you look at these files from your log output? You should > include those, and the rails log if you have anything there. > > > Look at log/mongrel.3010.pid and log/mongrel.log for info. > > - Rob > > On 2/21/07, Roy Satterfield <roys at okabashi.com > wrote: > We have a site that is running Mongrel and is down. > > Below is the Mongrel log file information. > > > The server is a Linux server with 2gb of RAM. > > EV1 (hosting site) says that since this is a Linux server that the > RAM is cached and that there is plenty of free memory available. > > > Can anyone make sense of this error and have some suggestion of > options that I may take to correct this issue and get our site up > and running? > > > Below is the Mongrel.log file: > > ** Daemonized, any open files are closed. Look at log/mongrel. > 3010.pid and log/mongrel.log for info. > > ** Starting Mongrel listening at 127.0.0.1:3010 > > ** Starting Rails with production environment... > > ** Rails loaded. > > ** Loading any Rails specific GemPlugins > > ** Signals ready. TERM => stop. USR2 => restart. INT => stop (no > restart). > > ** Rails signals registered. HUP => reload (without restart). It > might not work well. > > ** Mongrel available at 127.0.0.1:3010 > > [FATAL] failed to allocate memory > > > Here is the server information: > > > General > > CPU > > GenuineIntel, Intel(R) Pentium(R) 4 CPU 3.06GHz > > Version > > psa v7.5.4_build75050824.12 os_RedHat el3 > > OS > > Linux 2.4.20-021stab028.3.777-smp > > Key number > > PLSK.00317196.0002 > > System Uptime: > > 02:40 > > CPU usage > > Last 1 minute > > 0.24 > > Last 5 minutes > > 0.38 > > Last 15 minutes > > 0.38 > > Memory Usage > > Total > > Used > > Free > > Shared > > Buffer > > Cached > > Usage > > 1.97 GB > > 1.85 GB > > 123.08 MB > > 0 B > > 249.13 MB > > 726.05 MB > > 93.9% > > <image001.gif> > > <image002.gif> > > <image003.gif> > > <image004.gif> > > Swap usage > > Total > > Used > > Free > > Usage > > 1.95 GB > > 736.53 MB > > 1.23 GB > > 36.82% > > <image001.gif> > > <image002.gif> > > <image003.gif> > > <image004.gif> > > Hard Disk Usage > > Filesystem > > Total > > Used > > Available > > Capacity > > /dev/vzfs > > 15 360.00 MB > > 2 677.79 MB > > 12 682.21 MB > > 17.43% > > <image001.gif> > > <image002.gif> > > <image003.gif> > > <image004.gif> > > Directories > > /usr/local/psa > /var/lib/mysql > /var/www/vhosts > /var/qmail/mailnames > /var/named/run-root > > proc > > 0.00 MB > > 0.00 MB > > 0.00 MB > > 0% > > <image005.gif> > > <image003.gif> > > <image004.gif> > > devpts > > 0.00 MB > > 0.00 MB > > 0.00 MB > > 0% > > <image005.gif> > > <image003.gif> > > <image004.gif> > > Domains > > Active > > 3 > > Problem > > 0 > > Passive > > 0 > > > > > Thanks!! > > Roy > > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users > > <image001.gif> > <image005.gif> > <image003.gif> > <image004.gif> > <image002.gif> > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users
What happens when you fire up WEBrick (script/server) instead of Mongrel? Op 21-feb-2007, om 21:54 heeft Roy Satterfield het volgende geschreven:> Zed, > Are you saying that this issue is caused by there not being enough RAM > on the server? > There are 2 sites running on this server. EV1 restarted the VPS and I > took my other site down.... > Even with the other site down this site would not restart. > > But the weird part is the other site will restart without any > problems. > So if it is a memory problem then why will one site start properly? > > Thanks, > Roy > > -----Original Message----- > From: mongrel-users-bounces at rubyforge.org > [mailto:mongrel-users-bounces at rubyforge.org] On Behalf Of Zed A. Shaw > Sent: Wednesday, February 21, 2007 6:33 PM > To: mongrel-users at rubyforge.org > Subject: Re: [Mongrel] Critical Issue - Site down... > > On Wed, 21 Feb 2007 14:47:41 -0500 > "Roy Satterfield" <roys at okabashi.com> wrote: > >> [FATAL] failed to allocate memory > > You ran out of memory. > > -- > Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu > http://www.zedshaw.com/ > http://www.awprofessional.com/title/0321483502 -- The Mongrel Book > http://mongrel.rubyforge.org/ > http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help. > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users-- www.gomagazine.nl +31643904460 pobox 51059 nl-1007eb amsterdam -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2423 bytes Desc: not available Url : http://rubyforge.org/pipermail/mongrel-users/attachments/20070221/3a21921b/attachment.bin
At Wed, 21 Feb 2007 15:54:11 -0500, "Roy Satterfield" <roys at okabashi.com> wrote:> > Zed, > Are you saying that this issue is caused by there not being enough RAM > on the server? > There are 2 sites running on this server. EV1 restarted the VPS and I > took my other site down.... > Even with the other site down this site would not restart. > > But the weird part is the other site will restart without any problems. > So if it is a memory problem then why will one site start properly?On a VPS the numbers which may be reported by utilities like top, etc. don?t correspond to the amount of memory which actually available for your server instance. I?m not all that familiar with the way these things work but it?s happened to me. You are running out of memory. It has nothing to do with mongrel except mongrel is using that last bit of memory. Kill unnecessary processes, or talk to your ISP and have them increase your memory allocation. best, Erik Hetzner -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://rubyforge.org/pipermail/mongrel-users/attachments/20070221/957348ae/attachment.bin
I''m also looking at this same box and what''s happening is that I''m starting a single mongrel server which starts, but the first proxied request to it causes the out of memory error. It''s only using about 24M of memory at the time it starts and it dies immediately. I''m not sure exactly what the VPS is doing or how it handles its memory. Nothing happens in the /var/log/messages log when it dies so I''m not really sure if it''s a mongrel issue or a Xen issue. Any other ideas? Josh On Feb 21, 2007, at 4:00 PM, Jason A. Hoffman wrote:> The graphic that you sent has 123.08 MB of RAM free (not too much), > and you''re exhausting that with the single mongrel process. > > Regards, Jason > > > On Feb 21, 2007, at 12:15 PM, Rob Sanheim wrote: > >> Did you look at these files from your log output? You should >> include those, and the rails log if you have anything there. >> >> >> Look at log/mongrel.3010.pid and log/mongrel.log for info. >> >> - Rob >> >> On 2/21/07, Roy Satterfield <roys at okabashi.com > wrote: >> We have a site that is running Mongrel and is down. >> >> Below is the Mongrel log file information. >> >> >> The server is a Linux server with 2gb of RAM. >> >> EV1 (hosting site) says that since this is a Linux server that the >> RAM is cached and that there is plenty of free memory available. >> >> >> Can anyone make sense of this error and have some suggestion of >> options that I may take to correct this issue and get our site up >> and running? >> >> >> Below is the Mongrel.log file: >> >> ** Daemonized, any open files are closed. Look at log/mongrel. >> 3010.pid and log/mongrel.log for info. >> >> ** Starting Mongrel listening at 127.0.0.1:3010 >> >> ** Starting Rails with production environment... >> >> ** Rails loaded. >> >> ** Loading any Rails specific GemPlugins >> >> ** Signals ready. TERM => stop. USR2 => restart. INT => stop (no >> restart). >> >> ** Rails signals registered. HUP => reload (without restart). It >> might not work well. >> >> ** Mongrel available at 127.0.0.1:3010 >> >> [FATAL] failed to allocate memory >> >> >> Here is the server information: >> >> >> General >> >> CPU >> >> GenuineIntel, Intel(R) Pentium(R) 4 CPU 3.06GHz >> >> Version >> >> psa v7.5.4_build75050824.12 os_RedHat el3 >> >> OS >> >> Linux 2.4.20-021stab028.3.777-smp >> >> Key number >> >> PLSK.00317196.0002 >> >> System Uptime: >> >> 02:40 >> >> CPU usage >> >> Last 1 minute >> >> 0.24 >> >> Last 5 minutes >> >> 0.38 >> >> Last 15 minutes >> >> 0.38 >> >> Memory Usage >> >> Total >> >> Used >> >> Free >> >> Shared >> >> Buffer >> >> Cached >> >> Usage >> >> 1.97 GB >> >> 1.85 GB >> >> 123.08 MB >> >> 0 B >> >> 249.13 MB >> >> 726.05 MB >> >> 93.9% >> >> <image001.gif> >> >> <image002.gif> >> >> <image003.gif> >> >> <image004.gif> >> >> Swap usage >> >> Total >> >> Used >> >> Free >> >> Usage >> >> 1.95 GB >> >> 736.53 MB >> >> 1.23 GB >> >> 36.82% >> >> <image001.gif> >> >> <image002.gif> >> >> <image003.gif> >> >> <image004.gif> >> >> Hard Disk Usage >> >> Filesystem >> >> Total >> >> Used >> >> Available >> >> Capacity >> >> /dev/vzfs >> >> 15 360.00 MB >> >> 2 677.79 MB >> >> 12 682.21 MB >> >> 17.43% >> >> <image001.gif> >> >> <image002.gif> >> >> <image003.gif> >> >> <image004.gif> >> >> Directories >> >> /usr/local/psa >> /var/lib/mysql >> /var/www/vhosts >> /var/qmail/mailnames >> /var/named/run-root >> >> proc >> >> 0.00 MB >> >> 0.00 MB >> >> 0.00 MB >> >> 0% >> >> <image005.gif> >> >> <image003.gif> >> >> <image004.gif> >> >> devpts >> >> 0.00 MB >> >> 0.00 MB >> >> 0.00 MB >> >> 0% >> >> <image005.gif> >> >> <image003.gif> >> >> <image004.gif> >> >> Domains >> >> Active >> >> 3 >> >> Problem >> >> 0 >> >> Passive >> >> 0 >> >> >> >> >> Thanks!! >> >> Roy >> >> >> _______________________________________________ >> Mongrel-users mailing list >> Mongrel-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/mongrel-users >> >> <image001.gif> >> <image005.gif> >> <image003.gif> >> <image004.gif> >> <image002.gif> >> _______________________________________________ >> Mongrel-users mailing list >> Mongrel-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/mongrel-users > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users
You really shouldn''t be running out of memory with 2GB unless you have a huge amount of traffic or other crap going on that server. To me it looks like it is caused by one of the following: 1: horribly put together ruby app with bad memory leaking gem/plugins like rmagick 2: Lots of other crap running on you server that shouldn''t be that is causing high memory leaking. 3: a linux kernel or VPS issue. There are too many possibilities for us to pin point exactly what is going on unless you explain what you are doing with your server. On 2/21/07, joost baaij <joost at spacebabies.nl> wrote:> > What happens when you fire up WEBrick (script/server) instead of > Mongrel? > > Op 21-feb-2007, om 21:54 heeft Roy Satterfield het volgende geschreven: > > > Zed, > > Are you saying that this issue is caused by there not being enough RAM > > on the server? > > There are 2 sites running on this server. EV1 restarted the VPS and I > > took my other site down.... > > Even with the other site down this site would not restart. > > > > But the weird part is the other site will restart without any > > problems. > > So if it is a memory problem then why will one site start properly? > > > > Thanks, > > Roy > > > > -----Original Message----- > > From: mongrel-users-bounces at rubyforge.org > > [mailto:mongrel-users-bounces at rubyforge.org] On Behalf Of Zed A. Shaw > > Sent: Wednesday, February 21, 2007 6:33 PM > > To: mongrel-users at rubyforge.org > > Subject: Re: [Mongrel] Critical Issue - Site down... > > > > On Wed, 21 Feb 2007 14:47:41 -0500 > > "Roy Satterfield" <roys at okabashi.com> wrote: > > > >> [FATAL] failed to allocate memory > > > > You ran out of memory. > > > > -- > > Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu > > http://www.zedshaw.com/ > > http://www.awprofessional.com/title/0321483502 -- The Mongrel Book > > http://mongrel.rubyforge.org/ > > http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help. > > _______________________________________________ > > Mongrel-users mailing list > > Mongrel-users at rubyforge.org > > http://rubyforge.org/mailman/listinfo/mongrel-users > > _______________________________________________ > > Mongrel-users mailing list > > Mongrel-users at rubyforge.org > > http://rubyforge.org/mailman/listinfo/mongrel-users > > -- > www.gomagazine.nl +31643904460 pobox 51059 nl-1007eb amsterdam > > > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users > >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20070221/f6464869/attachment-0001.html
I started webrick and get the same issue, I think it''s a virtual server problem. Here is what''s happening: 1) Start webrick on port 3010 in production mode (or mongrel) 2) make a request to the site 3) Production log shows a request to the front page 4) Fatal message occurs in webrick/mongrel log 5) Web browser recieves 500 Application Error response If you refresh the page you get a proxy error response similar to this from apache: Proxy Error The proxy server received an invalid response from an upstream server. The proxy server could not handle the request GET /. Reason: Error reading from remote server If you refresh it again you get the normal Bad Gateway response, and then forever more it is that. Here''s webrick running -bash-2.05b$ script/server -p 3010 -e production => Booting WEBrick... => Rails application started on http://0.0.0.0:3010 => Ctrl-C to shutdown server; call with --help for options [2007-02-21 15:21:33] INFO WEBrick 1.3.1 [2007-02-21 15:21:33] INFO ruby 1.8.4 (2005-12-24) [i686-linux] [2007-02-21 15:21:33] INFO WEBrick::HTTPServer#start: pid=26139 port=3010 127.0.0.1 - - [21/Feb/2007:15:21:35 CST] "GET / HTTP/1.1" 200 60 - -> / [FATAL] failed to allocate memory This is the site running the Radiant CMS and it has been running stable for the past 6 months or so until now. Josh On Feb 21, 2007, at 4:06 PM, joost baaij wrote:> What happens when you fire up WEBrick (script/server) instead of > Mongrel? > > Op 21-feb-2007, om 21:54 heeft Roy Satterfield het volgende > geschreven: > >> Zed, >> Are you saying that this issue is caused by there not being enough >> RAM >> on the server? >> There are 2 sites running on this server. EV1 restarted the VPS and I >> took my other site down.... >> Even with the other site down this site would not restart. >> >> But the weird part is the other site will restart without any >> problems. >> So if it is a memory problem then why will one site start properly? >> >> Thanks, >> Roy >> >> -----Original Message----- >> From: mongrel-users-bounces at rubyforge.org >> [mailto:mongrel-users-bounces at rubyforge.org] On Behalf Of Zed A. Shaw >> Sent: Wednesday, February 21, 2007 6:33 PM >> To: mongrel-users at rubyforge.org >> Subject: Re: [Mongrel] Critical Issue - Site down... >> >> On Wed, 21 Feb 2007 14:47:41 -0500 >> "Roy Satterfield" <roys at okabashi.com> wrote: >> >>> [FATAL] failed to allocate memory >> >> You ran out of memory. >> >> -- >> Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu >> http://www.zedshaw.com/ >> http://www.awprofessional.com/title/0321483502 -- The Mongrel Book >> http://mongrel.rubyforge.org/ >> http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help. >> _______________________________________________ >> Mongrel-users mailing list >> Mongrel-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/mongrel-users >> _______________________________________________ >> Mongrel-users mailing list >> Mongrel-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/mongrel-users > > -- > www.gomagazine.nl +31643904460 pobox 51059 nl-1007eb amsterdam > > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users
use ''free -m'' and look at the "-/+ buffers/cache'' line. Bradley Erik Hetzner wrote:> At Wed, 21 Feb 2007 15:54:11 -0500, > "Roy Satterfield" <roys at okabashi.com> wrote: >> Zed, >> Are you saying that this issue is caused by there not being enough RAM >> on the server? >> There are 2 sites running on this server. EV1 restarted the VPS and I >> took my other site down.... >> Even with the other site down this site would not restart. >> >> But the weird part is the other site will restart without any problems. >> So if it is a memory problem then why will one site start properly? > > On a VPS the numbers which may be reported by utilities like top, etc. > don?t correspond to the amount of memory which actually available for > your server instance. I?m not all that familiar with the way these > things work but it?s happened to me. You are running out of memory. It > has nothing to do with mongrel except mongrel is using that last bit > of memory. Kill unnecessary processes, or talk to your ISP and have > them increase your memory allocation. > > best, > Erik Hetzner > > > ------------------------------------------------------------------------ > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users
Have you tried backing out any recent changes (if any?) that were made to the code or the configuration? Especially gems or plugins used? - Rob On 2/21/07, Josh Ferguson <josh at besquared.net> wrote:> I started webrick and get the same issue, I think it''s a virtual > server problem. Here is what''s happening: > > 1) Start webrick on port 3010 in production mode (or mongrel) > 2) make a request to the site > 3) Production log shows a request to the front page > 4) Fatal message occurs in webrick/mongrel log > 5) Web browser recieves 500 Application Error response > > If you refresh the page you get a proxy error response similar to > this from apache: > > Proxy Error > > The proxy server received an invalid response from an upstream server. > The proxy server could not handle the request GET /. > > Reason: Error reading from remote server > > If you refresh it again you get the normal Bad Gateway response, and > then forever more it is that. > > Here''s webrick running > > -bash-2.05b$ script/server -p 3010 -e production > => Booting WEBrick... > => Rails application started on http://0.0.0.0:3010 > => Ctrl-C to shutdown server; call with --help for options > [2007-02-21 15:21:33] INFO WEBrick 1.3.1 > [2007-02-21 15:21:33] INFO ruby 1.8.4 (2005-12-24) [i686-linux] > [2007-02-21 15:21:33] INFO WEBrick::HTTPServer#start: pid=26139 > port=3010 > 127.0.0.1 - - [21/Feb/2007:15:21:35 CST] "GET / HTTP/1.1" 200 60 > - -> / > [FATAL] failed to allocate memory > > This is the site running the Radiant CMS and it has been running > stable for the past 6 months or so until now. > > Josh > > On Feb 21, 2007, at 4:06 PM, joost baaij wrote: > > > What happens when you fire up WEBrick (script/server) instead of > > Mongrel? > > > > Op 21-feb-2007, om 21:54 heeft Roy Satterfield het volgende > > geschreven: > > > >> Zed, > >> Are you saying that this issue is caused by there not being enough > >> RAM > >> on the server? > >> There are 2 sites running on this server. EV1 restarted the VPS and I > >> took my other site down.... > >> Even with the other site down this site would not restart. > >> > >> But the weird part is the other site will restart without any > >> problems. > >> So if it is a memory problem then why will one site start properly? > >> > >> Thanks, > >> Roy > >> > >> -----Original Message----- > >> From: mongrel-users-bounces at rubyforge.org > >> [mailto:mongrel-users-bounces at rubyforge.org] On Behalf Of Zed A. Shaw > >> Sent: Wednesday, February 21, 2007 6:33 PM > >> To: mongrel-users at rubyforge.org > >> Subject: Re: [Mongrel] Critical Issue - Site down... > >> > >> On Wed, 21 Feb 2007 14:47:41 -0500 > >> "Roy Satterfield" <roys at okabashi.com> wrote: > >> > >>> [FATAL] failed to allocate memory > >> > >> You ran out of memory. > >> > >> -- > >> Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu > >> http://www.zedshaw.com/ > >> http://www.awprofessional.com/title/0321483502 -- The Mongrel Book > >> http://mongrel.rubyforge.org/ > >> http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help. > >> _______________________________________________ > >> Mongrel-users mailing list > >> Mongrel-users at rubyforge.org > >> http://rubyforge.org/mailman/listinfo/mongrel-users > >> _______________________________________________ > >> Mongrel-users mailing list > >> Mongrel-users at rubyforge.org > >> http://rubyforge.org/mailman/listinfo/mongrel-users > > > > -- > > www.gomagazine.nl +31643904460 pobox 51059 nl-1007eb amsterdam > > > > > > _______________________________________________ > > Mongrel-users mailing list > > Mongrel-users at rubyforge.org > > http://rubyforge.org/mailman/listinfo/mongrel-users > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >
Also, this might help in profiling possible memory leaks. http://scottstuff.net/blog/articles/2006/08/17/memory-leak-profiling- with-rails Op 21-feb-2007, om 23:08 heeft Bradley Taylor het volgende geschreven:> use ''free -m'' and look at the "-/+ buffers/cache'' line. > > Bradley > > Erik Hetzner wrote: >> At Wed, 21 Feb 2007 15:54:11 -0500, >> "Roy Satterfield" <roys at okabashi.com> wrote: >>> Zed, >>> Are you saying that this issue is caused by there not being >>> enough RAM >>> on the server? >>> There are 2 sites running on this server. EV1 restarted the VPS >>> and I >>> took my other site down.... >>> Even with the other site down this site would not restart. >>> >>> But the weird part is the other site will restart without any >>> problems. >>> So if it is a memory problem then why will one site start properly? >> >> On a VPS the numbers which may be reported by utilities like top, >> etc. >> don?t correspond to the amount of memory which actually available for >> your server instance. I?m not all that familiar with the way these >> things work but it?s happened to me. You are running out of >> memory. It >> has nothing to do with mongrel except mongrel is using that last bit >> of memory. Kill unnecessary processes, or talk to your ISP and have >> them increase your memory allocation. >> >> best, >> Erik Hetzner >> >> >> --------------------------------------------------------------------- >> --- >> >> _______________________________________________ >> Mongrel-users mailing list >> Mongrel-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/mongrel-users > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users-- www.gomagazine.nl +31643904460 pobox 51059 nl-1007eb amsterdam -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2423 bytes Desc: not available Url : http://rubyforge.org/pipermail/mongrel-users/attachments/20070222/49dbac2a/attachment.bin
On Wed, 21 Feb 2007 14:47:41 -0500 "Roy Satterfield" <roys at okabashi.com> wrote:> [FATAL] failed to allocate memoryYou ran out of memory. -- Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu http://www.zedshaw.com/ http://www.awprofessional.com/title/0321483502 -- The Mongrel Book http://mongrel.rubyforge.org/ http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
On Wed, 21 Feb 2007 15:54:11 -0500 "Roy Satterfield" <roys at okabashi.com> wrote:> Zed, > Are you saying that this issue is caused by there not being enough RAM > on the server? > There are 2 sites running on this server. EV1 restarted the VPS and I > took my other site down.... > Even with the other site down this site would not restart. > > But the weird part is the other site will restart without any problems. > So if it is a memory problem then why will one site start properly?I don''t know, but what I see is a fatal out of memory error. No matter what the other application does, this one seems to be trying to use too much RAM and then dying. You can get this because of process restrictions, not enough RAM, etc. -- Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu http://www.zedshaw.com/ http://www.awprofessional.com/title/0321483502 -- The Mongrel Book http://mongrel.rubyforge.org/ http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
Hi, On 21-Feb-07, at 4:10 PM, Josh Ferguson wrote:> I''m also looking at this same box and what''s happening is that I''m > starting a single mongrel server which starts, but the first proxied > request to it causes the out of memory error. It''s only using about > 24M of memory at the time it starts and it dies immediately. I''m not > sure exactly what the VPS is doing or how it handles its memory. > Nothing happens in the /var/log/messages log when it dies so I''m not > really sure if it''s a mongrel issue or a Xen issue. Any other ideas? > JoshI had something similar happen on a new VPS. It was a user quota problem. I whined to the ISP and it was fixed in a couple of minutes. I''d suggest looking into user and process quotas that might be exceeded or changed recently (doing this isn''t necessarily all that easy and you might need the sys admin''s help/co-operation). Cheers, Bob ---- Bob Hutchison -- blogs at <http://www.recursive.ca/ hutch/> Recursive Design Inc. -- <http://www.recursive.ca/> Raconteur -- <http://www.raconteur.info/> xampl for Ruby -- <http://rubyforge.org/projects/xampl/>
Here is the current view of system resources on our site configuration... Resources based on VPS template Name VPS-184MB-15GB-150GB-754 Description 184 MB RAM, 15 GB Hard Drive Space, 150 GB Bandwidth Tools ExtendedExtended CPU Usage Resource Capacity CPU 2.0% Load Average 0.3, 0.48, 0.45 System Usage Resource Capacity System 67.68% Disk Usage Resource Capacity Total Used Available Disk Space 17.48% 15 Gb 3 Gb 12 Gb Disk Inodes 20.31% 634,507 128,867 505,640 -----Original Message----- From: mongrel-users-bounces at rubyforge.org [mailto:mongrel-users-bounces at rubyforge.org] On Behalf Of Bob Hutchison Sent: Thursday, February 22, 2007 8:42 AM To: mongrel-users at rubyforge.org Subject: Re: [Mongrel] Critical Issue - Site down... Hi, On 21-Feb-07, at 4:10 PM, Josh Ferguson wrote:> I''m also looking at this same box and what''s happening is that I''m> starting a single mongrel server which starts, but the first proxied> request to it causes the out of memory error. It''s only using about> 24M of memory at the time it starts and it dies immediately. I''m not> sure exactly what the VPS is doing or how it handles its memory.> Nothing happens in the /var/log/messages log when it dies so I''m not> really sure if it''s a mongrel issue or a Xen issue. Any other ideas?> JoshI had something similar happen on a new VPS. It was a user quota problem. I whined to the ISP and it was fixed in a couple of minutes. I''d suggest looking into user and process quotas that might be exceeded or changed recently (doing this isn''t necessarily all that easy and you might need the sys admin''s help/co-operation). Cheers, Bob ---- Bob Hutchison -- blogs at <http://www.recursive.ca/ hutch/> Recursive Design Inc. -- <http://www.recursive.ca/> Raconteur -- <http://www.raconteur.info/> xampl for Ruby -- <http://rubyforge.org/projects/xampl/> _______________________________________________ Mongrel-users mailing list Mongrel-users at rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-users -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20070222/b87fe813/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 559 bytes Desc: image001.gif Url : http://rubyforge.org/pipermail/mongrel-users/attachments/20070222/b87fe813/attachment-0005.gif -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 104 bytes Desc: image002.gif Url : http://rubyforge.org/pipermail/mongrel-users/attachments/20070222/b87fe813/attachment-0006.gif -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 70 bytes Desc: image003.gif Url : http://rubyforge.org/pipermail/mongrel-users/attachments/20070222/b87fe813/attachment-0007.gif -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 54 bytes Desc: image004.gif Url : http://rubyforge.org/pipermail/mongrel-users/attachments/20070222/b87fe813/attachment-0008.gif -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 100 bytes Desc: image005.gif Url : http://rubyforge.org/pipermail/mongrel-users/attachments/20070222/b87fe813/attachment-0009.gif
On Thu, 22 Feb 2007 11:41:41 -0500 "Roy Satterfield" <roys at okabashi.com> wrote:> 184 MB RAM, 15 GB Hard Drive Space, 150 GB BandwidthRoy, 184M of ram is *not* enough to run most production sites. Ever. This is why you get an out of memory error. Sorry to be short, but you seriously need to expand this or get someone to help you. -- Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu http://www.zedshaw.com/ http://www.awprofessional.com/title/0321483502 -- The Mongrel Book http://mongrel.rubyforge.org/ http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
Zed,
I haven''t followed this discussion, but I was running Apache 2.2 + 3
mongrels on a Pent 2 with 64Mb of memory running Linux and it worked.
It was dog slow because it was always swapping, but it ran. Does his
system not swap?
BTW, just an excuse to say thanks for Mongrel. I was ready to abandon
the whole rails thing after many months of fighting with various
mod_ruby, fastcgi, etc solutions only to have them work sometimes and
sometimes not. Mongrel saved the day. Thanks!
- Jim
----- Original Message -----
From: "Zed A. Shaw" <zedshaw at zedshaw.com>
To: <mongrel-users at rubyforge.org>
Sent: Thursday, February 22, 2007 4:55 PM
Subject: Re: [Mongrel] Critical Issue - Site down...
> On Thu, 22 Feb 2007 11:41:41 -0500
> "Roy Satterfield" <roys at okabashi.com> wrote:
>
>> 184 MB RAM, 15 GB Hard Drive Space, 150 GB Bandwidth
>
> Roy, 184M of ram is *not* enough to run most production sites.
> Ever.
>
> This is why you get an out of memory error.
>
> Sorry to be short, but you seriously need to expand this or get
> someone
> to help you.
>
> --
> Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
> http://www.zedshaw.com/
> http://www.awprofessional.com/title/0321483502 -- The Mongrel Book
> http://mongrel.rubyforge.org/
> http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>
I have set up a few VPS''s with low ram (one had 128MB with no swap) a couple mongrels behind nginx and it is rather fast and can handle it well. The Rails app on that site is fairly small but uses file_upload and rmagick and has never stalled in production (not saying i like rmagick or anything). The traffic on it is fairly low but you can hit it with httperf at 50 requests per second (10,000 request batch) and it doesn''t even slow down in response time too much. This site isn''t anything to brag about but I just want to make the point on what you can do as long as you set things up correctly. if you are just running single mongrel instance you shouldn''t need very much ram, if you are having problems that tells me that something else is amiss. Kyle Kochis On 2/22/07, Jim Hogue <jjhogue at sbcglobal.net> wrote:> Zed, > > I haven''t followed this discussion, but I was running Apache 2.2 + 3 > mongrels on a Pent 2 with 64Mb of memory running Linux and it worked. > It was dog slow because it was always swapping, but it ran. Does his > system not swap? > > BTW, just an excuse to say thanks for Mongrel. I was ready to abandon > the whole rails thing after many months of fighting with various > mod_ruby, fastcgi, etc solutions only to have them work sometimes and > sometimes not. Mongrel saved the day. Thanks! > > - Jim > > ----- Original Message ----- > From: "Zed A. Shaw" <zedshaw at zedshaw.com> > To: <mongrel-users at rubyforge.org> > Sent: Thursday, February 22, 2007 4:55 PM > Subject: Re: [Mongrel] Critical Issue - Site down... > > > > On Thu, 22 Feb 2007 11:41:41 -0500 > > "Roy Satterfield" <roys at okabashi.com> wrote: > > > >> 184 MB RAM, 15 GB Hard Drive Space, 150 GB Bandwidth > > > > Roy, 184M of ram is *not* enough to run most production sites. > > Ever. > > > > This is why you get an out of memory error. > > > > Sorry to be short, but you seriously need to expand this or get > > someone > > to help you. > > > > -- > > Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu > > http://www.zedshaw.com/ > > http://www.awprofessional.com/title/0321483502 -- The Mongrel Book > > http://mongrel.rubyforge.org/ > > http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help. > > _______________________________________________ > > Mongrel-users mailing list > > Mongrel-users at rubyforge.org > > http://rubyforge.org/mailman/listinfo/mongrel-users > > > > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >
On 2/22/07, Roy Satterfield <roys at okabashi.com> wrote:> [FATAL] failed to allocate memoryLogin via SSH as the user your mongrels are running as. What does the command ''ulimit -a'' report? There may be resource limits in place. If that''s not it, you probably have very little (or no) swap, and really are running out of RAM. As Zed said, 184mb really isn''t enough, especially if you also have a database server on there. - Ben