Jason Su
2011-May-19 05:56 UTC
unicorn doesn''t restart properly after cap deploy (not using Bundler)
Hey guys, I''m sending USR2 to unicorn after cap deploy, and the old master is getting replaced by a new master that doesn''t work. Here are some similar threads I found ... http://rubyforge.org/pipermail/mongrel-unicorn/2010-October/000733.html http://rubyforge.org/pipermail/mongrel-unicorn/2010-October/000717.html I''m not using Bundler, and I have working_directory set in my unicorn config: Unicorn::HttpServer::START_CTX[0] "/opt/ruby-enterprise-1.8.7-2010.02/bin/unicorn_rails" pid "/var/www/lookbook/shared/pids/unicorn.pid" stderr_path "/var/www/lookbook/shared/log/unicorn.stderr.log" stdout_path "/var/www/lookbook/shared/log/unicorn.stdout.log" preload_app true if GC.respond_to?(:copy_on_write_friendly=) ? GC.copy_on_write_friendly = true end before_fork do |server, worker| ? STDERR.puts "BEFORE FORK:" ? STDERR.puts ENV.inspect ? defined?(ActiveRecord::Base) and ? ? ActiveRecord::Base.connection.disconnect! ? if old_pid != server.pid ? ? begin ? ? ? sig = (worker.nr?+ 1) >= server.worker_processes ? :QUIT : :TTOU ? ? ? Process.kill(sig, File.read(old_pid).to_i) ? ? rescue Errno::ENOENT, Errno::ESRCH ? ? end ? end end after_fork do |server, worker| ? STDERR.puts "AFTER FORK:" ? STDERR.puts ENV.inspect ? defined?(ActiveRecord::Base) and ? ? ActiveRecord::Base.establish_connection ? begin ? ? uid, gid = Process.euid, Process.egid ? ? user, group = ''apache'', ''apache'' ? ? target_uid = Etc.getpwnam(user).uid ? ? target_gid = Etc.getgrnam(group).gid ? ? worker.tmp.chown(target_uid, target_gid) ? ? if uid != target_uid || gid != target_gid ? ? ? Process.initgroups(user, target_gid) ? ? ? Process::GID.change_privilege(target_gid) ? ? ? Process::UID.change_privilege(target_uid) ? ? end ? rescue => e ? ? if RAILS_ENV == ''development'' ? ? ? STDERR.puts "couldn''t change user, oh well" ? ? else ? ? ? raise e ? ? end ? end end before_exec do |server| ? STDERR.puts "BEFORE EXEC:" ? ENV["PATH"] "/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local:/opt/ruby-enterprise-1.8.7-2010.02/bin" ? STDERR.puts ENV.inspect end Below is the output from unicorn.stderr.log.. I''m not sure if UNICORN_FD has something to do with it? 1) starting unicorn BEFORE FORK: {"SHELL"=>"/bin/bash", "TERM"=>"xterm", "SSH_CLIENT"=>"24.23.182.32 38170 22", "OLDPWD"=>"/root", "SSH_TTY"=>"/dev/pts/5", "USER"=>"root", "SSH_AUTH_SOCK"=>"/tmp/ssh-eeoDcvH461/agent.461", "PATH"=>"/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin", "MAIL"=>"/var/mail/root", "PWD"=>"/var/www/lookbook/current", "HOME"=>"/root", "SHLVL"=>"2", "LOGNAME"=>"root", "SSH_CONNECTION"=>"24.23.182.32 38170 98.158.22.100 22", "_"=>"/opt/ruby-enterprise-1.8.7-2010.02/bin/unicorn_rails", "RAILS_ENV"=>"production"} AFTER FORK: {"SHELL"=>"/bin/bash", "TERM"=>"xterm", "SSH_CLIENT"=>"24.23.182.32 38170 22", "OLDPWD"=>"/root", "SSH_TTY"=>"/dev/pts/5", "USER"=>"root", "SSH_AUTH_SOCK"=>"/tmp/ssh-eeoDcvH461/agent.461", "PATH"=>"/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin", "MAIL"=>"/var/mail/root", "PWD"=>"/var/www/lookbook/current", "HOME"=>"/root", "SHLVL"=>"2", "LOGNAME"=>"root", "SSH_CONNECTION"=>"24.23.182.32 38170 98.158.22.100 22", "_"=>"/opt/ruby-enterprise-1.8.7-2010.02/bin/unicorn_rails", "RAILS_ENV"=>"production"} 2) restarting unicorn (kill -USR2) BEFORE EXEC: {"SHELL"=>"/bin/bash", "TERM"=>"xterm", "SSH_CLIENT"=>"24.23.182.32 38170 22", "OLDPWD"=>"/root", "SSH_TTY"=>"/dev/pts/5", "USER"=>"root", "SSH_AUTH_SOCK"=>"/tmp/ssh-eeoDcvH461/agent.461", "PATH"=>"/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local:/opt/ruby-enterprise-1.8.7-2010.02/bin", "MAIL"=>"/var/mail/root", "PWD"=>"/var/www/lookbook/current", "HOME"=>"/root", "SHLVL"=>"2", "LOGNAME"=>"root", "SSH_CONNECTION"=>"24.23.182.32 38170 98.158.22.100 22", "_"=>"/opt/ruby-enterprise-1.8.7-2010.02/bin/unicorn_rails", "RAILS_ENV"=>"production", "UNICORN_FD"=>"3"} I, [2011-05-18T18:13:56.318531 #510] ?INFO -- : inherited addr=/var/www/lookbook/current/tmp/sockets/unicorn.sock fd=3 I, [2011-05-18T18:13:56.318819 #510] ?INFO -- : Refreshing Gem list BEFORE FORK: {"SHELL"=>"/bin/bash", "TERM"=>"xterm", "SSH_CLIENT"=>"24.23.182.32 38170 22", "OLDPWD"=>"/root", "SSH_TTY"=>"/dev/pts/5", "USER"=>"root", "SSH_AUTH_SOCK"=>"/tmp/ssh-eeoDcvH461/agent.461", "PATH"=>"/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local:/opt/ruby-enterprise-1.8.7-2010.02/bin", "MAIL"=>"/var/mail/root", "PWD"=>"/var/www/lookbook/current", "HOME"=>"/root", "SHLVL"=>"2", "LOGNAME"=>"root", "SSH_CONNECTION"=>"24.23.182.32 38170 98.158.22.100 22", "_"=>"/opt/ruby-enterprise-1.8.7-2010.02/bin/unicorn_rails", "RAILS_ENV"=>"production", "UNICORN_FD"=>"3"} AFTER FORK: {"SHELL"=>"/bin/bash", "TERM"=>"xterm", "SSH_CLIENT"=>"24.23.182.32 38170 22", "OLDPWD"=>"/root", "SSH_TTY"=>"/dev/pts/5", "USER"=>"root", "SSH_AUTH_SOCK"=>"/tmp/ssh-eeoDcvH461/agent.461", "PATH"=>"/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local:/opt/ruby-enterprise-1.8.7-2010.02/bin", "MAIL"=>"/var/mail/root", "PWD"=>"/var/www/lookbook/current", "HOME"=>"/root", "SHLVL"=>"2", "LOGNAME"=>"root", "SSH_CONNECTION"=>"24.23.182.32 38170 98.158.22.100 22", "_"=>"/opt/ruby-enterprise-1.8.7-2010.02/bin/unicorn_rails", "RAILS_ENV"=>"production", "UNICORN_FD"=>"3"} 3) cap deploy with symlinks + restart unicorn (kill -USR2) BEFORE EXEC: {"SHELL"=>"/bin/bash", "TERM"=>"xterm", "SSH_CLIENT"=>"24.23.182.32 38170 22", "OLDPWD"=>"/root", "SSH_TTY"=>"/dev/pts/5", "USER"=>"root", "SSH_AUTH_SOCK"=>"/tmp/ssh-eeoDcvH461/agent.461", "PATH"=>"/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local:/opt/ruby-enterprise-1.8.7-2010.02/bin", "MAIL"=>"/var/mail/root", "PWD"=>"/var/www/lookbook/current", "HOME"=>"/root", "SHLVL"=>"2", "LOGNAME"=>"root", "SSH_CONNECTION"=>"24.23.182.32 38170 98.158.22.100 22", "_"=>"/opt/ruby-enterprise-1.8.7-2010.02/bin/unicorn_rails", "RAILS_ENV"=>"production", "UNICORN_FD"=>"3"} I, [2011-05-18T18:17:13.732600 #1589] ?INFO -- : inherited addr=/var/www/lookbook/current/tmp/sockets/unicorn.sock fd=3 I, [2011-05-18T18:17:13.732942 #1589] ?INFO -- : Refreshing Gem list BEFORE FORK: {"SHELL"=>"/bin/bash", "TERM"=>"xterm", "SSH_CLIENT"=>"24.23.182.32 38170 22", "OLDPWD"=>"/root", "SSH_TTY"=>"/dev/pts/5", "USER"=>"root", "SSH_AUTH_SOCK"=>"/tmp/ssh-eeoDcvH461/agent.461", "PATH"=>"/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local:/opt/ruby-enterprise-1.8.7-2010.02/bin", "MAIL"=>"/var/mail/root", "PWD"=>"/var/www/lookbook/current", "HOME"=>"/root", "SHLVL"=>"2", "LOGNAME"=>"root", "SSH_CONNECTION"=>"24.23.182.32 38170 98.158.22.100 22", "_"=>"/opt/ruby-enterprise-1.8.7-2010.02/bin/unicorn_rails", "RAILS_ENV"=>"production", "UNICORN_FD"=>"3"} AFTER FORK: {"SHELL"=>"/bin/bash", "TERM"=>"xterm", "SSH_CLIENT"=>"24.23.182.32 38170 22", "OLDPWD"=>"/root", "SSH_TTY"=>"/dev/pts/5", "USER"=>"root", "SSH_AUTH_SOCK"=>"/tmp/ssh-eeoDcvH461/agent.461", "PATH"=>"/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local:/opt/ruby-enterprise-1.8.7-2010.02/bin", "MAIL"=>"/var/mail/root", "PWD"=>"/var/www/lookbook/current", "HOME"=>"/root", "SHLVL"=>"2", "LOGNAME"=>"root", "SSH_CONNECTION"=>"24.23.182.32 38170 98.158.22.100 22", "_"=>"/opt/ruby-enterprise-1.8.7-2010.02/bin/unicorn_rails", "RAILS_ENV"=>"production", "UNICORN_FD"=>"3"} worker=1 ready any help appreciated, thanks :D Jason
Eric Wong
2011-May-19 09:27 UTC
unicorn doesn''t restart properly after cap deploy (not using Bundler)
Jason Su <jason at lookbook.nu> wrote:> Hey guys, > > I''m sending USR2 to unicorn after cap deploy, and the old master is > getting replaced by a new master that doesn''t work.Can you expand on "doesn''t work"?> Here are some similar threads I found ... > http://rubyforge.org/pipermail/mongrel-unicorn/2010-October/000733.html > http://rubyforge.org/pipermail/mongrel-unicorn/2010-October/000717.html > > I''m not using Bundler, and I have working_directory set in my unicorn config:I don''t see where you have working_directory set below> Unicorn::HttpServer::START_CTX[0] > "/opt/ruby-enterprise-1.8.7-2010.02/bin/unicorn_rails"You shouldn''t need to touch START_CTX unless you''re switching Ruby installations. It''s rope to hang yourself with. I would simplify the config file as much as possible if you''re debugging a problem.> pid "/var/www/lookbook/shared/pids/unicorn.pid" > stderr_path "/var/www/lookbook/shared/log/unicorn.stderr.log" > stdout_path "/var/www/lookbook/shared/log/unicorn.stdout.log" > preload_app true > if GC.respond_to?(:copy_on_write_friendly=) > ? GC.copy_on_write_friendly = true > end > before_fork do |server, worker| > ? STDERR.puts "BEFORE FORK:" > ? STDERR.puts ENV.inspect > ? defined?(ActiveRecord::Base) and > ? ? ActiveRecord::Base.connection.disconnect!The auto-killing old workers bit is fragile and you shouldn''t need it unless you''re very low on memory.> ? if old_pid != server.pid > ? ? begin > ? ? ? sig = (worker.nr?+ 1) >= server.worker_processes ? :QUIT : :TTOU > ? ? ? Process.kill(sig, File.read(old_pid).to_i) > ? ? rescue Errno::ENOENT, Errno::ESRCH > ? ? end > ? end > > end > after_fork do |server, worker| > ? STDERR.puts "AFTER FORK:" > ? STDERR.puts ENV.inspect > > ? defined?(ActiveRecord::Base) and > ? ? ActiveRecord::Base.establish_connection > ? beginIf you want to use user switching, the "user" directive which appeared a while back is much simpler and less error prone.> ? ? uid, gid = Process.euid, Process.egid > ? ? user, group = ''apache'', ''apache'' > ? ? target_uid = Etc.getpwnam(user).uid > ? ? target_gid = Etc.getgrnam(group).gid > ? ? worker.tmp.chown(target_uid, target_gid) > ? ? if uid != target_uid || gid != target_gid > ? ? ? Process.initgroups(user, target_gid) > ? ? ? Process::GID.change_privilege(target_gid) > ? ? ? Process::UID.change_privilege(target_uid) > ? ? end<snip>> Below is the output from unicorn.stderr.log.. I''m not sure if > UNICORN_FD has something to do with it?UNICORN_FD is absolutely for the old master tells the new master about the listener sockets to inherit. Don''t touch it. <snip>> 3) cap deploy with symlinks + restart unicorn (kill -USR2)<snip>> addr=/var/www/lookbook/current/tmp/sockets/unicorn.sock fd=3You should make sure your socket is in a shared path, not current. Whatever''s in current gets replaced by Capistrano, right? I''m not sure if tmp is a symlink that''s relinked on a new deploy, but there could be a race condition where the path to the socket is unreachable and nginx can''t see it. -- Eric Wong
Jason Su
2011-May-19 20:26 UTC
unicorn doesn''t restart properly after cap deploy (not using Bundler)
You were right, the socket was not in a shared path... D''oh! Thanks so much~ On Thu, May 19, 2011 at 2:27 AM, Eric Wong <normalperson at yhbt.net> wrote:> Jason Su <jason at lookbook.nu> wrote: >> Hey guys, >> >> I''m sending USR2 to unicorn after cap deploy, and the old master is >> getting replaced by a new master that doesn''t work. > > Can you expand on "doesn''t work"? > >> Here are some similar threads I found ... >> http://rubyforge.org/pipermail/mongrel-unicorn/2010-October/000733.html >> http://rubyforge.org/pipermail/mongrel-unicorn/2010-October/000717.html >> >> I''m not using Bundler, and I have working_directory set in my unicorn config: > > I don''t see where you have working_directory set below > >> Unicorn::HttpServer::START_CTX[0] >> "/opt/ruby-enterprise-1.8.7-2010.02/bin/unicorn_rails" > > You shouldn''t need to touch START_CTX unless you''re switching Ruby > installations. ?It''s rope to hang yourself with. > > I would simplify the config file as much as possible if you''re > debugging a problem. > >> pid "/var/www/lookbook/shared/pids/unicorn.pid" >> stderr_path "/var/www/lookbook/shared/log/unicorn.stderr.log" >> stdout_path "/var/www/lookbook/shared/log/unicorn.stdout.log" >> preload_app true >> if GC.respond_to?(:copy_on_write_friendly=) >> ? GC.copy_on_write_friendly = true >> end >> before_fork do |server, worker| >> ? STDERR.puts "BEFORE FORK:" >> ? STDERR.puts ENV.inspect >> ? defined?(ActiveRecord::Base) and >> ? ? ActiveRecord::Base.connection.disconnect! > > The auto-killing old workers bit is fragile and you shouldn''t need > it unless you''re very low on memory. > >> ? if old_pid != server.pid >> ? ? begin >> ? ? ? sig = (worker.nr?+ 1) >= server.worker_processes ? :QUIT : :TTOU >> ? ? ? Process.kill(sig, File.read(old_pid).to_i) >> ? ? rescue Errno::ENOENT, Errno::ESRCH >> ? ? end >> ? end >> >> end >> after_fork do |server, worker| >> ? STDERR.puts "AFTER FORK:" >> ? STDERR.puts ENV.inspect >> >> ? defined?(ActiveRecord::Base) and >> ? ? ActiveRecord::Base.establish_connection >> ? begin > > If you want to use user switching, the "user" directive which > appeared a while back is much simpler and less error prone. > >> ? ? uid, gid = Process.euid, Process.egid >> ? ? user, group = ''apache'', ''apache'' >> ? ? target_uid = Etc.getpwnam(user).uid >> ? ? target_gid = Etc.getgrnam(group).gid >> ? ? worker.tmp.chown(target_uid, target_gid) >> ? ? if uid != target_uid || gid != target_gid >> ? ? ? Process.initgroups(user, target_gid) >> ? ? ? Process::GID.change_privilege(target_gid) >> ? ? ? Process::UID.change_privilege(target_uid) >> ? ? end > > <snip> > >> Below is the output from unicorn.stderr.log.. I''m not sure if >> UNICORN_FD has something to do with it? > > UNICORN_FD is absolutely for the old master tells the new master about > the listener sockets to inherit. ?Don''t touch it. > > <snip> > >> 3) cap deploy with symlinks + restart unicorn (kill -USR2) > > <snip> > >> addr=/var/www/lookbook/current/tmp/sockets/unicorn.sock fd=3 > > You should make sure your socket is in a shared path, not current. > > Whatever''s in current gets replaced by Capistrano, right? ?I''m not sure > if tmp is a symlink that''s relinked on a new deploy, but there could be > a race condition where the path to the socket is unreachable and > nginx can''t see it. > > -- > Eric Wong > _______________________________________________ > Unicorn mailing list - mongrel-unicorn at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-unicorn > Do not quote signatures (like this one) or top post when replying >