Hi - I tried to get Daemontools and Unicorn working together a while back - there are issues on USR2 restart because of the pid change - I''m hoping someone in the community will have some understanding of this issue I documented my experience and eventual defeat here : http://log.robotarmyma.de/post/2053448029/daemontools-ubuntu-rvm-bundler-unicorn-install Any help would be received graciously. -- make haste slowly \ festina lente ?\ - mobile ?+1_415_632_6001 curtis.schofield at gmail.com http://robotarmyma.de
Curtis j Schofield <curtis.schofield at gmail.com> wrote:> I tried to get Daemontools and Unicorn working together a while back - > there are issues on USR2 restart because of the pid > change - I''m hoping someone in the community will have some > understanding of this issueI have no experience with daemontools directly, but I expect any solutions people found to make daemontools work with nginx USR2 would also work for unicorn since they have the same process/signal model. Maybe a hacky way would be to have daemontools run this wrapper script instead of Unicorn: -------------------------------- 8< ------------------------- #!/bin/sh set -e # this script is totally untested and written by someone who''s never # used daemontools # start Unicorn in the background: bundle exec unicorn -D ... # forward signals received in this wrapper to the Unicorn process: for sig in HUP USR1 USR2 QUIT TERM QUIT do trap ''kill -''$sig'' $(cat $UNICORN_PID_FILE)'' $sig done # loop forever while Unicorn has its pid file # a smarter, Linux-only version of this would use inotify instead while test -f $UNICORN_PID_FILE do sleep 1 done -- Eric Wong
On Mon, Apr 25, 2011 at 11:34 AM, Eric Wong <normalperson at yhbt.net> wrote:> Curtis j Schofield <curtis.schofield at gmail.com> wrote: >> I tried to get Daemontools and Unicorn working together a while back - >> there are issues on USR2 restart because of the pid >> change - I''m hoping someone ?in the community will have some >> understanding of this issue > > I have no experience with daemontools directly, but I expect any > solutions people found to make daemontools work with nginx USR2 > would also work for unicorn since they have the same process/signal > model. > > Maybe a hacky way would be to have daemontools run this wrapper > script instead of Unicorn: > > -------------------------------- 8< ------------------------- > #!/bin/sh > set -e > > # this script is totally untested and written by someone who''s never > # used daemontools > > # start Unicorn in the background: > bundle exec unicorn -D ... > > # forward signals received in this wrapper to the Unicorn process: > for sig in HUP USR1 USR2 QUIT TERM QUIT > do > ? ? ? ?trap ''kill -''$sig'' $(cat $UNICORN_PID_FILE)'' $sig > done > > # loop forever while Unicorn has its pid file > # a smarter, Linux-only version of this would use inotify instead > while test -f $UNICORN_PID_FILE > do > ? ? ? ?sleep 1 > done >Eric! This is fascinating! If i run a wrapper with unicorn as a child process - i can detect the absence of the wrapper - and i can detect the absence of the pid file - but i cannot detect a hard fail in the unicorn process unless i have some kind of pipe open to the child and register that failure. Is my understanding correct? Daemontools is design to take a process that is running in the foreground and monitor it - keep it alive - the breakdown in unicorn is when a USR2 arrives at unicorn - and the pid switch occurs - daemon tools believes it has lost the process. I see this as a design flaw in all pid based monitoring solutions - in my experiences.> -- > Eric Wong > _______________________________________________ > Unicorn mailing list - mongrel-unicorn at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-unicorn > Do not quote signatures (like this one) or top post when replying >-- make haste slowly \ festina lente ?\ - mobile ?+1_415_632_6001 curtis.schofield at gmail.com http://robotarmyma.de
Curtis j Schofield <curtis.schofield at gmail.com> wrote:> On Mon, Apr 25, 2011 at 11:34 AM, Eric Wong <normalperson at yhbt.net> wrote:<snip>> > while test -f $UNICORN_PID_FILE > > do > > ? ? ? ?sleep 1 > > done > > If i run a wrapper with unicorn as a child process - i can detect the > absence of the wrapper - and > i can detect the absence of the pid file - but i cannot detect a hard > fail in the unicorn process unless i have some kind of pipe > open to the child and register that failure. > > Is my understanding correct?Yes, but the absence of the pid file usually means the unicorn process failed hard. Only unicorn error logs can tell you that definitively, though. I would probably make the wrapper script "exit 1" after the while loop in my script...> Daemontools is design to take a process that is running in the > foreground and monitor it - keep it alive - the breakdown in unicorn > is when a USR2 arrives at unicorn - and the pid switch occurs - daemon > tools believes it has lost the process. > > I see this as a design flaw in all pid based monitoring solutions - in > my experiences.Yes, it''s a problem with the monitoring solution trying to do the job of the unicorn master process. Unicorn is designed to use the master process and there''s no other way to do what Unicorn does without it. The unicorn master process itself should be very robust and never fail during normal operation (upgrades may break it if things go really wrong, but you already pay attention to what you''re upgrading, right?). -- Eric Wong
> > Yes, it''s a problem with the monitoring solution trying to do the job of > the unicorn master process. ?Unicorn is designed to use the master > process and there''s no other way to do what Unicorn does without it. > > The unicorn master process itself should be very robust and never fail > during normal operation (upgrades may break it if things go really > wrong, but you already pay attention to what you''re upgrading, right?). >I''m not concerned about the unicorn process failing - more i use daemon tools on boot and to make sure the process comes alive properly at boot and because daemon tools has a simple and intelligent interface that doesn''t have me writing blue-pill conf files or monit conf files. That said - it may be enough to monitor that wrapper script in this case. -- make haste slowly \ festina lente ?\ - mobile ?+1_415_632_6001 curtis.schofield at gmail.com http://robotarmyma.de
There are USR2 patches for daemontools, we used them for unicorn on a large production app worked just fine. Can''t remember the url but google should show it. Chris Ochs On Mon, Apr 25, 2011 at 9:37 AM, Curtis j Schofield <curtis.schofield at gmail.com> wrote:> Hi ?- > > I tried to get Daemontools and Unicorn working together a while back - > there are issues on USR2 restart because of the pid > change - I''m hoping someone ?in the community will have some > understanding of this issue > > I documented my experience and eventual defeat here : > http://log.robotarmyma.de/post/2053448029/daemontools-ubuntu-rvm-bundler-unicorn-install > > > Any help would be received graciously. > > -- > make haste slowly \ > festina lente ?\ > - > mobile ?+1_415_632_6001 > curtis.schofield at gmail.com > http://robotarmyma.de > _______________________________________________ > Unicorn mailing list - mongrel-unicorn at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-unicorn > Do not quote signatures (like this one) or top post when replying >
I read one possible patch - it doesn''t do pid handling - just lets svc send different signals My understanding of how unicorn responds to USR2 is that it changes it''s pid and respawns - this would mean that Daemontools is now monitoring the wrong pid and will think the process went down. On Mon, Apr 25, 2011 at 3:07 PM, snacktime <snacktime at gmail.com> wrote:> There are USR2 patches for daemontools, we used them for unicorn on a > large production app worked just fine. ?Can''t remember the url but > google should show it. > > Chris Ochs > > On Mon, Apr 25, 2011 at 9:37 AM, Curtis j Schofield > <curtis.schofield at gmail.com> wrote: >> Hi ?- >> >> I tried to get Daemontools and Unicorn working together a while back - >> there are issues on USR2 restart because of the pid >> change - I''m hoping someone ?in the community will have some >> understanding of this issue >> >> I documented my experience and eventual defeat here : >> http://log.robotarmyma.de/post/2053448029/daemontools-ubuntu-rvm-bundler-unicorn-install >> >> >> Any help would be received graciously. >> >> -- >> make haste slowly \ >> festina lente ?\ >> - >> mobile ?+1_415_632_6001 >> curtis.schofield at gmail.com >> http://robotarmyma.de >> _______________________________________________ >> Unicorn mailing list - mongrel-unicorn at rubyforge.org >> http://rubyforge.org/mailman/listinfo/mongrel-unicorn >> Do not quote signatures (like this one) or top post when replying >> > _______________________________________________ > Unicorn mailing list - mongrel-unicorn at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-unicorn > Do not quote signatures (like this one) or top post when replying >-- make haste slowly \ festina lente ?\ - mobile ?+1_415_632_6001 curtis.schofield at gmail.com http://robotarmyma.de