Hi,
I''m deploying Unicorn on a Rails application with RUnit. Technically
I''m using Chef''s deployment tools, if any of you are familiar
with it
(https://github.com/opscode-cookbooks/application_ruby) but to be clear
they aren''t doing anything magical, so this is purely an issue with
RUnit and Unicorn.
The TL;DR of the following post, which has lots of pastes (linking to
pastebins for easier reading:) when I run /etc/init.d/myzippykid
restart, the old unicorn master fails to die, causing the new master to
be stuck in a loop waiting to own port 8080. The new master never takes
over, as the old master never dies. When I manually kill the master,
the new one takes over just fine.
What am I missing in my restart process to make the old master exit?
--
More details:
When I deploy a new version of software, the unicorn master fails to
restart. Here is an example of my ps auxfg output after I run
`/etc/init.d/myzippykid restart` (which is symlinked to /usr/bin/sv):
https://gist.github.com/grahamc/35715c1a2c9717e461e3
When I run `killall unicorn`, the ruby script under runsvdir gets
replaced with the unicorn worker. This is because the ruby script is
attempting to bind to port 8080, which is already taken up by the
unicorn master process. As soon as the old master is out of the way,
its capable of binding.
Here is my unicorn configuration at /etc/unicorn/myzippykid.rb:
https://gist.github.com/grahamc/08aaed97c9b39cf68f01
Here is my /etc/sv/myzippykid/run script which is how runit runs my
unicorn daemon: https://gist.github.com/grahamc/7ca7e3942d19f60d339a
my bundle_wrapper.sh simply inits the environment with the proper ruby
and path: https://gist.github.com/grahamc/9760001e0e20c915d9e9
--
The contents of all my pastes for those of you without internet at time
of reading:
https://gist.github.com/grahamc/35715c1a2c9717e461e3:
runsvdir -P /etc/service log:
.............................................................................................................................................
\_ runsv myzippykid
\_ svlogd -tt ./main
\_ /bin/bash /opt/myzippykid/bundle_wrapper.sh exec unicorn -E
staging -c /etc/unicorn/myzippykid.rb
\_ ruby
/opt/myzippykid/deployment/releases/6328251212178494a6c82b11ec518dc4d1962f9c/vendor/bundle/ruby/2.0.0/bin/unicorn
-E staging -c /etc/unicorn/myzippykid.rb
unicorn master -E staging -c /etc/unicorn/myzippykid.rb
\_ unicorn worker[0] -E staging -c /etc/unicorn/myzippykid.rb
https://gist.github.com/grahamc/08aaed97c9b39cf68f01:
# What ports/sockets to listen on, and what options for them.
listen "8080", :tcp_nodelay => true, :backlog => 100
working_directory ''/opt/myzippykid/deployment/current''
# What the timeout for killing busy workers is, in seconds
timeout 60
# Whether the app should be pre-loaded
preload_app false
# How many worker processes
worker_processes 1
# What to do before we fork a worker
before_fork do |server, worker|
sleep 1
end
https://gist.github.com/grahamc/7ca7e3942d19f60d339a:
#!/bin/bash
cd /opt/myzippykid/deployment/current
exec 2>&1
exec /usr/bin/chpst \
-u myzippykid:myzippykid \
/opt/myzippykid/bundle_wrapper.sh exec \
unicorn \
-E staging \
-c /etc/unicorn/myzippykid.rb
https://gist.github.com/grahamc/9760001e0e20c915d9e9:
#!/bin/bash
source /etc/profile.d/rbenv.sh
bundle $@
Thank you for your help,
-- Graham Christensen
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
Graham Christensen <graham@grahamc.com> wrote:> I''m deploying Unicorn on a Rails application with RUnit. Technically > I''m using Chef''s deployment tools, if any of you are familiar with > it (https://github.com/opscode-cookbooks/application_ruby) but to be > clear they aren''t doing anything magical, so this is purely an issue > with RUnit and Unicorn. > > The TL;DR of the following post, which has lots of pastes (linking > to pastebins for easier reading:) when I run /etc/init.d/myzippykid > restart, the old unicorn master fails to die, causing the new master > to be stuck in a loop waiting to own port 8080. The new master never > takes over, as the old master never dies. When I manually kill the > master, the new one takes over just fine. > > What am I missing in my restart process to make the old master exit?It seems you''re missing /etc/init.d/myzippykid in your pastes. Are you sending SIGQUIT (or SIGTERM) to the old unicorn master? If you have old workers servicing requests, the old master will wait until all the old workers are done. Since your timeout is 60s, everything (workers and master) should exit around that time period if you have stalled/long-running requests, though. _______________________________________________ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying
On 2013-05-20 17:28:51 +0000, Eric Wong said:> Graham Christensen <graham@grahamc.com> wrote: >> I''m deploying Unicorn on a Rails application with RUnit. Technically >> I''m using Chef''s deployment tools, if any of you are familiar with >> it (https://github.com/opscode-cookbooks/application_ruby) but to be >> clear they aren''t doing anything magical, so this is purely an issue >> with RUnit and Unicorn. >> >> The TL;DR of the following post, which has lots of pastes (linking >> to pastebins for easier reading:) when I run /etc/init.d/myzippykid >> restart, the old unicorn master fails to die, causing the new master >> to be stuck in a loop waiting to own port 8080. The new master never >> takes over, as the old master never dies. When I manually kill the >> master, the new one takes over just fine. >> >> What am I missing in my restart process to make the old master exit? > > It seems you''re missing /etc/init.d/myzippykid in your pastes.The /etc/init.d/myzippykid is symlinked to /usr/bin/sv which is a binary provided by runit.> > Are you sending SIGQUIT (or SIGTERM) to the old unicorn master? > If you have old workers servicing requests, the old master will wait > until all the old workers are done.This is an interesting question. runit appear to be capturing the PID of my bundle_wrapper.sh instead of the spawned unicorn process. Since this isn''t interactive, it probably isn''t forwarding the signals to Unicorn. After replacing the `bundle $@` with `exec bundle $@`, and therefore replacing the process with the unicorn command, it all works as intended. I suppose all it took was that question to jog my memory. I appreciate your assistance! Here is the complete bundle_wrapper.sh wthich implements everything as intended: #!/bin/bash source /etc/profile.d/rbenv.sh exec bundle $@> > Since your timeout is 60s, everything (workers and master) should exit > around that time period if you have stalled/long-running requests, > though.Since Unicorn doen''t seem to be receiving the signals, this would explain why the timeout never was hit, and nothing ever exited.> _______________________________________________ > Unicorn mailing list - mongrel-unicorn@rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-unicorn > Do not quote signatures (like this one) or top post when replyingThank you a lot! Graham Christensen _______________________________________________ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying