First off, a big thank you to Zed and the development team for mongrel and mongrel cluster. I''ve deployed it for two apps (through an apache 2.2 proxy) and it''s amazing how much snappier everything is. Previously we were using lighttpd and lighttpd-fastcgi proxying through apache 2.0, which was turning into a hassle for lots of reasons, most notably the ruby fcgi bindings. I was shocked at how easy everything was to get everything up and running with mongrel, and found the documentation to be just what I needed - not more or less. My only question is this: when I run mongrel_rails cluster::restart or mongrel_rails cluster::stop/start the old processes do not get killed off and I have to do it manually. I''ve noticed this on 3 boxes, all running FC5 with user/group mongrel/mongrel. Any ideas? (Just as an aside, is there a gateway somewhere where I can search this list?)
cleaner416 wrote:> My only question is this: when I run mongrel_rails cluster::restart > or mongrel_rails cluster::stop/start the old processes do not get > killed off and I have to do it manually. I''ve noticed this on 3 > boxes, all running FC5 with user/group mongrel/mongrel. Any ideas?Do you get any error messages? Are you in the right directory? usually /path/to/app/current? Do you run the commands as root? Jeroen
On Dec 7, 2006, at 2:52 PM, Jeroen Houben wrote:> cleaner416 wrote: >> My only question is this: when I run mongrel_rails cluster::restart >> or mongrel_rails cluster::stop/start the old processes do not get >> killed off and I have to do it manually. I''ve noticed this on 3 >> boxes, all running FC5 with user/group mongrel/mongrel. Any ideas? > > Do you get any error messages? Are you in the right directory? usually > /path/to/app/current? Do you run the commands as root? >Yeah, ran as root, no error messages, in the root directory of the app. Smells like a permissions issue? Root is doing the restarting, the app files are all owned by nobody/nobody (expect for config/mongrel.log) and now I''m seeing that no log/mongrel.pid is getting created. Here''s the config/mongrel.yml: --- user: mongrel cwd: /data/wwwroot/myapp port: "8000" environment: production group: mongrel address: 127.0.0.1 pid_file: log/mongrel.pid servers: 3> Jeroen > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users
On Thu, 7 Dec 2006 12:43:02 -0500 cleaner416 <cleaner416 at gmail.com> wrote:> First off, a big thank you to Zed and the development team for > mongrel and mongrel cluster. I''ve deployed it for two apps (through > an apache 2.2 proxy) and it''s amazing how much snappier everything > is. Previously we were using lighttpd and lighttpd-fastcgi proxying > through apache 2.0, which was turning into a hassle for lots of > reasons, most notably the ruby fcgi bindings. I was shocked at how > easy everything was to get everything up and running with mongrel, > and found the documentation to be just what I needed - not more or less.Awesome. It''s nice to hear people move to a simpler tech AND have it seem snappier for them.> My only question is this: when I run mongrel_rails cluster::restart > or mongrel_rails cluster::stop/start the old processes do not get > killed off and I have to do it manually. I''ve noticed this on 3 > boxes, all running FC5 with user/group mongrel/mongrel. Any ideas?There is kind of a dumb race condition I''ve gotta fix this week where if you stop a mongrel process, but mongrel needs to wait on some threads to finish, and then start a new one, all hell breaks loose. Check if this is the case by doing this: 1) Tail the mongrel.log in a window off to the side. 2) Start some requests or something that''s common usage. 3) Stop mongrel and look for log messages saying that process is waiting on threads to finish. 4) Right away start a new mongrel to replace this one. The .pid file gets wiped out by this, and since the previous process is waiting, the new process can''t bind. 5) Once the original process dies you''ll be in the situation you''ve got. See if that''s the problem. If it is then hang on as I''m beefing up the start/stop logic to be a bit smarter about this.> (Just as an aside, is there a gateway somewhere where I can search > this list?)Here''s the mailman page, but otherwise you''ll need to search it with google: http://rubyforge.org/mailman/listinfo/mongrel-users -- Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu http://www.zedshaw.com/ http://www.awprofessional.com/title/0321483502 -- The Mongrel Book http://mongrel.rubyforge.org/ http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
cleaner416 wrote:> On Dec 7, 2006, at 2:52 PM, Jeroen Houben wrote: > >> cleaner416 wrote: >>> My only question is this: when I run mongrel_rails cluster::restart >>> or mongrel_rails cluster::stop/start the old processes do not get >>> killed off and I have to do it manually. I''ve noticed this on 3 >>> boxes, all running FC5 with user/group mongrel/mongrel. Any ideas? >> Do you get any error messages? Are you in the right directory? usually >> /path/to/app/current? Do you run the commands as root? >> > > Yeah, ran as root, no error messages, in the root directory of the > app. Smells like a permissions issue? Root is doing the > restarting, the app files are all owned by nobody/nobody (expect for > config/mongrel.log) and now I''m seeing that no log/mongrel.pid is > getting created. > > > Here''s the config/mongrel.yml: > > --- > user: mongrel > cwd: /data/wwwroot/myapp > port: "8000" > environment: production > group: mongrel > address: 127.0.0.1 > pid_file: log/mongrel.pid > servers: 3Can you do a mongrel_rails cluster::restart -v That should tell you a little more. Also, are you using capistrano? HTH Jeroen
Ezra Zygmuntowicz
2006-Dec-07 21:23 UTC
[Mongrel] Cluster restart leaving orphaned processes?
On Dec 7, 2006, at 12:44 PM, Zed A. Shaw wrote:> On Thu, 7 Dec 2006 12:43:02 -0500 > cleaner416 <cleaner416 at gmail.com> wrote: > <snip> > >> My only question is this: when I run mongrel_rails cluster::restart >> or mongrel_rails cluster::stop/start the old processes do not get >> killed off and I have to do it manually. I''ve noticed this on 3 >> boxes, all running FC5 with user/group mongrel/mongrel. Any ideas? > > There is kind of a dumb race condition I''ve gotta fix this week > where if you stop a mongrel process, but mongrel needs to wait on > some threads to finish, and then start a new one, all hell breaks > loose. > > Check if this is the case by doing this: > > 1) Tail the mongrel.log in a window off to the side. > 2) Start some requests or something that''s common usage. > 3) Stop mongrel and look for log messages saying that process is > waiting on threads to finish. > 4) Right away start a new mongrel to replace this one. The .pid > file gets wiped out by this, and since the previous process is > waiting, the new process can''t bind. > 5) Once the original process dies you''ll be in the situation you''ve > got. > > See if that''s the problem. If it is then hang on as I''m beefing up > the start/stop logic to be a bit smarter about this. >Hey Zed- I have been seeing this a lot when deploying apps from capistrano. It would be awesome to get it fixed in mongrel. What I have been doing for now is using gentoo''s init.d script options so it will invoke the stop command and wait for 15 seconds, if mongrel doesn''t go down by then it issues a kill -9 and then starts things up again. Here is an init.d script I use if anyone is interested. Its gentoo specifc afaik. #!/sbin/runscript # Copyright 1999-2004 Gentoo Foundation # Distributed under the terms of the GNU General Public License v2 # $Header: /var/cvsroot/gentoo-x86/www-servers/nginx/files/nginx-r1,v 1.1 2006/07/04 16:58:38 voxus Exp $ # Modified by tmornini at engineyard.com for mongrel_cluster startup EY_USER=whatever EY_CONF=/data/$EY_USER/current/config/mongrel_cluster.yml depend() { need net use dns logger after gfs } start() { ebegin "Starting mongrel_cluster" start-stop-daemon --start \ --name mongrel_rails \ --chuid $EY_USER \ --exec /usr/bin/mongrel_rails -- cluster::start -C $EY_CONF eend $? "Failed to start mongrel_cluster" } stop() { ebegin "Stopping mongrel_cluster" start-stop-daemon --stop --retry 15 --oknodo --name mongrel_rails eend $? "Failed to stop mongrel_cluster" } It''s the --retry 15 --oknodo part that does the right thing. Before I set this we had random mongrel restart failures during deploys because of this issue. I''d love to see mongrel stop be a bit more robust though for sure. Cheers- -- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273)
On 12/7/06, Ezra Zygmuntowicz <ezmobius at gmail.com> wrote:> > On Dec 7, 2006, at 12:44 PM, Zed A. Shaw wrote: > > > On Thu, 7 Dec 2006 12:43:02 -0500 > > cleaner416 <cleaner416 at gmail.com> wrote: > > <snip> > > > >> My only question is this: when I run mongrel_rails cluster::restart > >> or mongrel_rails cluster::stop/start the old processes do not get > >> killed off and I have to do it manually. I''ve noticed this on 3 > >> boxes, all running FC5 with user/group mongrel/mongrel. Any ideas? > > > > There is kind of a dumb race condition I''ve gotta fix this week > > where if you stop a mongrel process, but mongrel needs to wait on > > some threads to finish, and then start a new one, all hell breaks > > loose. > > > > Check if this is the case by doing this: > > > > 1) Tail the mongrel.log in a window off to the side. > > 2) Start some requests or something that''s common usage. > > 3) Stop mongrel and look for log messages saying that process is > > waiting on threads to finish. > > 4) Right away start a new mongrel to replace this one. The .pid > > file gets wiped out by this, and since the previous process is > > waiting, the new process can''t bind. > > 5) Once the original process dies you''ll be in the situation you''ve > > got. > > > > See if that''s the problem. If it is then hang on as I''m beefing up > > the start/stop logic to be a bit smarter about this. > > > > Hey Zed- > > I have been seeing this a lot when deploying apps from capistrano. > It would be awesome to get it fixed in mongrel. What I have been > doing for now is using gentoo''s init.d script options so it will > invoke the stop command and wait for 15 seconds, if mongrel doesn''t > go down by then it issues a kill -9 and then starts things up again. >On similar topic, but different platform (win32). I have been doing the same, and changed mongrel_service to perform in the same way. [snip]> ... I''d love to see mongrel stop be a bit > more robust though for sure. >How do you think we should fix this? how will be the step by step needed by mongrel to perform when restarting (mongrel and cluster). Later, -- Luis Lavena Multimedia systems - Leaders are made, they are not born. They are made by hard effort, which is the price which all of us must pay to achieve any goal that is worthwhile. Vince Lombardi
On Dec 7, 2006, at 3:44 PM, Zed A. Shaw wrote:> > There is kind of a dumb race condition I''ve gotta fix this week > where if you stop a mongrel process, but mongrel needs to wait on > some threads to finish, and then start a new one, all hell breaks > loose. > > Check if this is the case by doing this: > > 1) Tail the mongrel.log in a window off to the side. > 2) Start some requests or something that''s common usage. > 3) Stop mongrel and look for log messages saying that process is > waiting on threads to finish. > 4) Right away start a new mongrel to replace this one. The .pid > file gets wiped out by this, and since the previous process is > waiting, the new process can''t bind. > 5) Once the original process dies you''ll be in the situation you''ve > got. > > See if that''s the problem. If it is then hang on as I''m beefing up > the start/stop logic to be a bit smarter about this.Yep, that''s it. Thanks for the reply - I''ll just hang on for now. Cheers