Mongrel users - Aug 2007 - what is the correct way to stop/start a mongrel instance using monit with mongrel cluster

Hi --

I have been reading documentation and googling around to find the
correct way to do this but I have found many ways that seem to not
work, or the documentation makes no reference to.

I am using mongrel cluster with 10 mongrels for each server. Recently
I installed monit but which lead me to find the correct way to
start/stop mongrel instances one pid at a time. I am assuming one pid
at a time is the correct way...

Currently, when I do a cap deploy we just kick mongrels with
/etc/init.d/mongrel_rails.. I am assuming I should be using monit now
when I deploy to do this. Correct? Can you give me an example of this?
I know this is not the monit list, but I was thinking you guys know
what I am talking about.

So in my monit.conf/monitrc file here is what I have:

check process mongrel_9001 with pidfile /var/run/mongrel/mongrel.9001.pid
  start program = "/usr/local/bin/mongrel_rails start -d -e production
-p 9009 -a 127.0.0.1 -l log/mongrel.log -P
/var/run/mongrel/mongrel.9009.pid -c /home/app/current"
  stop program = "/usr/local/bin/mongrel_rails  stop -P
/var/run/mongrel/mongrel.9009.pid"

  if failed host 127.0.0.1 port 9001 protocol http
    with timeout 10 seconds
    then restart

  if totalmem > 128 Mb then restart
  if cpu is greater than 60% for 2 cycles then alert
  if cpu > 90% for 5 cycles then restart
  if loadavg(5min) greater than 10 for 8 cycles then restart
  if 3 restarts within 5 cycles then timeout


Part of my confusion is if I should use mongrel_rails cluster::start
or mongrel_rails start, Is there a difference?

Any suggestions is really appreciated.

-- 
Michael Steinfeld
Linux Admin/Developer
AIM: mikesteinfeld

On 8/9/07, Michael Steinfeld <mikeisgreat at gmail.com>
wrote:
> I installed monit but which lead me to find the correct way to
> start/stop mongrel instances one pid at a time.
Monit''s handling of pid files and daemonizing in general is pretty
weak. If you googled for "mogrel pid" (or had a long hard look at the
code) you already know that.

In RubyWorks Production Stack we solve this problem with runit. And it
was 37 Signals guys who told me about it. runit doesn''t require
service processes to detach themselves, so Mongrel''s shortcomings in
this area become a non-issue.

... shameless self-promotion
Actually, if you install RubyWorks stack, you get the whole
runit/monit/mongrel combo configured and ready to go out of the box...
:)

--
Alex Verkhovsky

On Aug 9, 2007, at 6:25 PM, Michael Steinfeld wrote:
> Hi --
>
> I have been reading documentation and googling around to find the
> correct way to do this but I have found many ways that seem to not
> work, or the documentation makes no reference to.
>
> I am using mongrel cluster with 10 mongrels for each server. Recently
> I installed monit but which lead me to find the correct way to
> start/stop mongrel instances one pid at a time. I am assuming one pid
> at a time is the correct way...
>
> Currently, when I do a cap deploy we just kick mongrels with
> /etc/init.d/mongrel_rails.. I am assuming I should be using monit now
> when I deploy to do this. Correct? Can you give me an example of this?
> I know this is not the monit list, but I was thinking you guys know
> what I am talking about.

	So the better way to go about it is to use mongrel_cluster with the  
--clean and --only options. Here is a monit file that works very well:

check process mongrel_warehouse_5000
   with pidfile /data/warehouse/shared/log/mongrel.5000.pid
   start program = "/usr/bin/mongrel_rails cluster::start -C /data/ 
warehouse/current/config/mongrel_cluster.yml --clean --only 5000"
   stop program = "/usr/bin/mongrel_rails cluster::stop -C /data/ 
warehouse/current/config/mongrel_cluster.yml --clean --only 5000"
   if totalmem is greater than 110.0 MB for 3 cycles then  
restart       # eating up memory?
   if loadavg(5min) greater than 10 for 8 cycles then  
restart          # bad, bad, bad
   if 20 restarts within 20 cycles then  
timeout                         # something is wrong, call the sys-admin
   group mongrel

	You must have the latest version of mongrel_cluster installed for  
the --clean option.  Then to restart in your deploy.rb set  
the :monit_group variable and use these tasks for restarting:


set : monit_group, ''mongrel''

desc <<-DESC
Restart the Mongrel processes on the app server by calling  
restart_mongrel_cluster.
DESC
task :restart, :roles => :app do
   restart_mongrel_cluster
end

desc <<-DESC
Start the Mongrel processes on the app server by calling  
start_mongrel_cluster.
DESC
task :spinner, :roles => :app do
   start_mongrel_cluster
end
desc <<-DESC
Start Mongrel processes on the app server.  This uses the :use_sudo  
variable to determine whether to use sudo or not. By  
default, :use_sudo is
set to true.
DESC
task :start_mongrel_cluster , :roles => :app do
   sudo "/usr/bin/monit start all -g #{monit_group}"
end

desc <<-DESC
Restart the Mongrel processes on the app server by starting and  
stopping the cluster. This uses the :use_sudo
variable to determine whether to use sudo or not. By  
default, :use_sudo is set to true.
DESC
task :restart_mongrel_cluster , :roles => :app do
   sudo "/usr/bin/monit restart all -g #{monit_group}"
end

desc <<-DESC
Stop the Mongrel processes on the app server.  This uses the :use_sudo
variable to determine whether to use sudo or not. By  
default, :use_sudo is
set to true.
DESC
task :stop_mongrel_cluster , :roles => :app do
   sudo "/usr/bin/monit stop all -g #{monit_group}"
end


	
Cheers-

-- Ezra Zygmuntowicz 
-- Founder & Ruby Hacker
-- ez at engineyard.com
-- Engine Yard, Serious Rails Hosting
-- (866) 518-YARD (9273)

Do what Ezra says; he''s the master of mongrel pids. Helped me in the
same way a while back and it''s worked great.

Evan

On 8/10/07, Ezra Zygmuntowicz <ezmobius at gmail.com>
wrote:
>
> On Aug 9, 2007, at 6:25 PM, Michael Steinfeld wrote:
>
> > Hi --
> >
> > I have been reading documentation and googling around to find the
> > correct way to do this but I have found many ways that seem to not
> > work, or the documentation makes no reference to.
> >
> > I am using mongrel cluster with 10 mongrels for each server. Recently
> > I installed monit but which lead me to find the correct way to
> > start/stop mongrel instances one pid at a time. I am assuming one pid
> > at a time is the correct way...
> >
> > Currently, when I do a cap deploy we just kick mongrels with
> > /etc/init.d/mongrel_rails.. I am assuming I should be using monit now
> > when I deploy to do this. Correct? Can you give me an example of this?
> > I know this is not the monit list, but I was thinking you guys know
> > what I am talking about.
>
>
>         So the better way to go about it is to use mongrel_cluster with the
> --clean and --only options. Here is a monit file that works very well:
>
> check process mongrel_warehouse_5000
>    with pidfile /data/warehouse/shared/log/mongrel.5000.pid
>    start program = "/usr/bin/mongrel_rails cluster::start -C /data/
> warehouse/current/config/mongrel_cluster.yml --clean --only 5000"
>    stop program = "/usr/bin/mongrel_rails cluster::stop -C /data/
> warehouse/current/config/mongrel_cluster.yml --clean --only 5000"
>    if totalmem is greater than 110.0 MB for 3 cycles then
> restart       # eating up memory?
>    if loadavg(5min) greater than 10 for 8 cycles then
> restart          # bad, bad, bad
>    if 20 restarts within 20 cycles then
> timeout                         # something is wrong, call the sys-admin
>    group mongrel
>
>         You must have the latest version of mongrel_cluster installed for
> the --clean option.  Then to restart in your deploy.rb set
> the :monit_group variable and use these tasks for restarting:
>
>
> set : monit_group, ''mongrel''
>
> desc <<-DESC
> Restart the Mongrel processes on the app server by calling
> restart_mongrel_cluster.
> DESC
> task :restart, :roles => :app do
>    restart_mongrel_cluster
> end
>
> desc <<-DESC
> Start the Mongrel processes on the app server by calling
> start_mongrel_cluster.
> DESC
> task :spinner, :roles => :app do
>    start_mongrel_cluster
> end
> desc <<-DESC
> Start Mongrel processes on the app server.  This uses the :use_sudo
> variable to determine whether to use sudo or not. By
> default, :use_sudo is
> set to true.
> DESC
> task :start_mongrel_cluster , :roles => :app do
>    sudo "/usr/bin/monit start all -g #{monit_group}"
> end
>
> desc <<-DESC
> Restart the Mongrel processes on the app server by starting and
> stopping the cluster. This uses the :use_sudo
> variable to determine whether to use sudo or not. By
> default, :use_sudo is set to true.
> DESC
> task :restart_mongrel_cluster , :roles => :app do
>    sudo "/usr/bin/monit restart all -g #{monit_group}"
> end
>
> desc <<-DESC
> Stop the Mongrel processes on the app server.  This uses the :use_sudo
> variable to determine whether to use sudo or not. By
> default, :use_sudo is
> set to true.
> DESC
> task :stop_mongrel_cluster , :roles => :app do
>    sudo "/usr/bin/monit stop all -g #{monit_group}"
> end
>
>
>
> Cheers-
>
> -- Ezra Zygmuntowicz
> -- Founder & Ruby Hacker
> -- ez at engineyard.com
> -- Engine Yard, Serious Rails Hosting
> -- (866) 518-YARD (9273)
>
>
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>

-- 
Evan Weaver
Cloudburst, LLC

On 8/9/07, Ezra Zygmuntowicz <ezmobius at gmail.com>
wrote:
>         So the better way to go about it is to use mongrel_cluster with the
> --clean and --only options.
Better than runit or telinit, or some other way to run it in
non-detached mode? Why?

--
Alex

On Aug 9, 2007, at 10:37 PM, Alexey Verkhovsky wrote:
> On 8/9/07, Ezra Zygmuntowicz <ezmobius at gmail.com> wrote:
>>         So the better way to go about it is to use mongrel_cluster  
>> with the
>> --clean and --only options.
>
> Better than runit or telinit, or some other way to run it in
> non-detached mode? Why?
	As far as I know runit does not support restarting when memory or  
cpu limits are hit and does not support making http health checks to  
the backends. I like runit myself for some things but monit works  
very well once you have a good config for it and offers more options  
for controlling your mongrels.


Cheers-
-- Ezra Zygmuntowicz 
-- Founder & Ruby Hacker
-- ez at engineyard.com
-- Engine Yard, Serious Rails Hosting
-- (866) 518-YARD (9273)

At Thu, 9 Aug 2007 21:25:23 -0400,
"Michael Steinfeld" <mikeisgreat at gmail.com>
wrote:
>
> Hi --
>
> I have been reading documentation and googling around to find the
> correct way to do this but I have found many ways that seem to not
> work, or the documentation makes no reference to.
I gave up on using mongrel_cluster. It doesn?t offer much benefit as
far as I can tell, and monit?s process oriented system didn?t play
that well with it. Some simple capistrano recipes can start multiple
mongrel easily, and, additionally, you can use the same settings
(number of mongrels, ports, etc.) with capistrano to generate monit
config files for monit with the help of ERB.

best,
Erik Hetzner
;; Erik Hetzner, California Digital Library
;; gnupg key id: 1024D/01DB07E3
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url :
http://rubyforge.org/pipermail/mongrel-users/attachments/20070809/7b11a468/attachment.bin

On Thu, 9 Aug 2007 20:13:27 -0600
"Alexey Verkhovsky" <alexey.verkhovsky at gmail.com> wrote:
 
> On 8/9/07, Michael Steinfeld <mikeisgreat at gmail.com> wrote:
> > I installed monit but which lead me to find the correct way to
> > start/stop mongrel instances one pid at a time.
> In RubyWorks Production Stack we solve this problem with runit. And it
> was 37 Signals guys who told me about it. runit doesn''t require
> service processes to detach themselves, so Mongrel''s shortcomings
in
> this area become a non-issue.
So let me see if I have this right.  You are saying the PID handling in Mongrel
isn''t right, but every single patch you''ve submitted to me in
an attempt to fix it has been an ugly shitfuck patch that isn''t cross
platform and myopically only solves your problem.

Now, I could probably handle that, but then you try to say that the solution is
to use runit.  You seen the code in runit?  Let me give you a taste of the
if-forrest extravaganza:

    if (verbose)
      strerr_warn5(INFO, "processing: ", ld->name, "/",
ld->fnsave, 0);
    if ((fd =open_read(ld->fnsave)) == -1)
      fatal2("unable to open input for processor", ld->name);
    if (fd_move(0, fd) == -1)
      fatal2("unable to move filedescriptor for processor",
ld->name);
    ld->fnsave[26] =''t'';
    if ((fd =open_trunc(ld->fnsave)) == -1)
      fatal2("unable to open output for processor", ld->name);
    if (fd_move(1, fd) == -1)
      fatal2("unable to move filedescriptor for processor",
ld->name);
    if ((fd =open_read("state")) == -1) {
      if (errno == error_noent) {
        if ((fd =open_trunc("state")) == -1)
          fatal2("unable to create empty state for processor",
ld->name);
        close(fd);
        if ((fd =open_read("state")) == -1)
          fatal2("unable to open state for processor", ld->name);

Oh yeah, that is sooooo fucking great.  This kind of bullshit DJB style code is
all over runit.  I wouldn''t touch it if it was wrapped in cellophane
and I was behind a cement wall accessible only via a BL4 airlock.  Runit is
nasty and it amazes me that people use the damn thing at all.

The fact of the matter is, *RUBY* has shitty PID management.  That ultimately,
*YOU* are supposed to use Mongrel as a goddamned *LIBRARY* if you want to embed
it in your own app server and manage the processes better.  The existing scripts
are just to handle the 90% cases of developers running stuff locally or the
majority of hosting situations.  It''s not like Apache, you are not only
allowed to modify it and put it in your own stuff but you are violently
encouraged to do it.  Hell there''s several books that teach you how and
a ton of examples and SIX fucking entire web frameworks that all use Mongrel
this way.

Get off your ass and read some code for a change.

Now normally, I wouldn''t get bent out of shape, but you represent a
company that''s using my software to make money (or try to make money,
since you guys probably couldn''t sell sweet tea to a dying man in the
Sahara).  What pisses me off is if you don''t like it, then why the hell
do you have "Mongrel is stable, reasonably fast, very easy to configure,
and can be used in both production and development environments." on your
damn web site you leeches?

Oh, because putting that there makes potential customers think you like Mongrel
and since they all like mongrel (love even) that gives you some free street
cred.

Well Alex and Thoughtworks, I''m going to be giving full control of
Mongrel to a group of people who just volunteered.  You''re free to use
all your powers of Neural Linguistic Programming to convince them to include
every dumbass thing you want, and they''re free to include it if they
want.  There won''t be anyone standing in your way and you''re
free to completely ruin my baby.

And without any more excuses, I feel that it''s time you start paying a
bit of this free love and work back to the community in the form of some damn
code.

Oh, and quit calling it a damn "stack".  It''s not a
application stack in any way because it doesn''t have a complete
administrative system that ties it all together.  It''s a bunch of RPMs
that you either stole or repackaged and you provide nothing more than a basic
hosting location for people.  Add some value for once and maybe people might
actually pay you for the damn product.

-- 
Zed A. Shaw
- Hate: http://savingtheinternetwithhate.com/
- Good: http://www.zedshaw.com/
- Evil: http://yearofevil.com/

On 8/9/07, Ezra Zygmuntowicz <ezmobius at gmail.com>
wrote:
> As far as I know runit does not support restarting when memory or
> cpu limits are hit and does not support making http health checks to
> the backends. I like runit myself for some things but monit works
> very well once you have a good config for it and offers more options
> for controlling your mongrels.
That''s right. The only thing we use runit for is process control
(i.e., starting and stopping things). And then monit to do the health
checking.

runit maintains pid files and current state for processes it controls,
which makes it easy to monitor.

--
Alex

On 8/10/07, Zed A. Shaw <zedshaw at zedshaw.com>
wrote:
> So let me see if I have this right.
No.

--
Alex

Really good stuff. Thanks everyone for all the suggestions and info.
It was very helpful.

On 8/10/07, Alexey Verkhovsky <alexey.verkhovsky at gmail.com>
wrote:
> On 8/9/07, Ezra Zygmuntowicz <ezmobius at gmail.com> wrote:
> > As far as I know runit does not support restarting when memory or
> > cpu limits are hit and does not support making http health checks to
> > the backends. I like runit myself for some things but monit works
> > very well once you have a good config for it and offers more options
> > for controlling your mongrels.
>
> That''s right. The only thing we use runit for is process control
> (i.e., starting and stopping things). And then monit to do the health
> checking.
>
> runit maintains pid files and current state for processes it controls,
> which makes it easy to monitor.
>
> --
> Alex
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>

-- 
Michael Steinfeld
Linux Admin/Developer
AIM: mikesteinfeld
GTALK: mikeisgreat at gmail.com