thr3ads.net - mongrel unicorn - unicorn doesn''t restart properly after cap deploy (not using Bundler) [May 2011]

If this information is useful, please help other people find it:
Share via:

Jason Su

2011-May-19 05:56 UTC

unicorn doesn''t restart properly after cap deploy (not using Bundler)

Hey guys,

I''m sending USR2 to unicorn after cap deploy, and the old master is
getting replaced by a new master that doesn''t work.

Here are some similar threads I found ...
http://rubyforge.org/pipermail/mongrel-unicorn/2010-October/000733.html
http://rubyforge.org/pipermail/mongrel-unicorn/2010-October/000717.html

I''m not using Bundler, and I have working_directory set in my unicorn
config:

Unicorn::HttpServer::START_CTX[0]
"/opt/ruby-enterprise-1.8.7-2010.02/bin/unicorn_rails"
pid "/var/www/lookbook/shared/pids/unicorn.pid"
stderr_path "/var/www/lookbook/shared/log/unicorn.stderr.log"
stdout_path "/var/www/lookbook/shared/log/unicorn.stdout.log"
preload_app true
if GC.respond_to?(:copy_on_write_friendly=)
? GC.copy_on_write_friendly = true
end
before_fork do |server, worker|
? STDERR.puts "BEFORE FORK:"
? STDERR.puts ENV.inspect
? defined?(ActiveRecord::Base) and
? ? ActiveRecord::Base.connection.disconnect!
? if old_pid != server.pid
? ? begin
? ? ? sig = (worker.nr?+ 1) >= server.worker_processes ? :QUIT : :TTOU
? ? ? Process.kill(sig, File.read(old_pid).to_i)
? ? rescue Errno::ENOENT, Errno::ESRCH
? ? end
? end

end
after_fork do |server, worker|
? STDERR.puts "AFTER FORK:"
? STDERR.puts ENV.inspect

? defined?(ActiveRecord::Base) and
? ? ActiveRecord::Base.establish_connection
? begin
? ? uid, gid = Process.euid, Process.egid
? ? user, group = ''apache'', ''apache''
? ? target_uid = Etc.getpwnam(user).uid
? ? target_gid = Etc.getgrnam(group).gid
? ? worker.tmp.chown(target_uid, target_gid)
? ? if uid != target_uid || gid != target_gid
? ? ? Process.initgroups(user, target_gid)
? ? ? Process::GID.change_privilege(target_gid)
? ? ? Process::UID.change_privilege(target_uid)
? ? end
? rescue => e
? ? if RAILS_ENV == ''development''
? ? ? STDERR.puts "couldn''t change user, oh well"
? ? else
? ? ? raise e
? ? end
? end
end
before_exec do |server|
? STDERR.puts "BEFORE EXEC:"
? ENV["PATH"]
"/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local:/opt/ruby-enterprise-1.8.7-2010.02/bin"
? STDERR.puts ENV.inspect
end



Below is the output from unicorn.stderr.log.. I''m not sure if
UNICORN_FD has something to do with it?
1) starting unicorn
BEFORE FORK:
{"SHELL"=>"/bin/bash",
"TERM"=>"xterm",
"SSH_CLIENT"=>"24.23.182.32
38170 22", "OLDPWD"=>"/root",
"SSH_TTY"=>"/dev/pts/5",
"USER"=>"root",
"SSH_AUTH_SOCK"=>"/tmp/ssh-eeoDcvH461/agent.461",
"PATH"=>"/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin",
"MAIL"=>"/var/mail/root",
"PWD"=>"/var/www/lookbook/current",
"HOME"=>"/root", "SHLVL"=>"2",
"LOGNAME"=>"root",
"SSH_CONNECTION"=>"24.23.182.32 38170 98.158.22.100 22",
"_"=>"/opt/ruby-enterprise-1.8.7-2010.02/bin/unicorn_rails",
"RAILS_ENV"=>"production"}
AFTER FORK:
{"SHELL"=>"/bin/bash",
"TERM"=>"xterm",
"SSH_CLIENT"=>"24.23.182.32
38170 22", "OLDPWD"=>"/root",
"SSH_TTY"=>"/dev/pts/5",
"USER"=>"root",
"SSH_AUTH_SOCK"=>"/tmp/ssh-eeoDcvH461/agent.461",
"PATH"=>"/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin",
"MAIL"=>"/var/mail/root",
"PWD"=>"/var/www/lookbook/current",
"HOME"=>"/root", "SHLVL"=>"2",
"LOGNAME"=>"root",
"SSH_CONNECTION"=>"24.23.182.32 38170 98.158.22.100 22",
"_"=>"/opt/ruby-enterprise-1.8.7-2010.02/bin/unicorn_rails",
"RAILS_ENV"=>"production"}


2) restarting unicorn (kill -USR2)

BEFORE EXEC:
{"SHELL"=>"/bin/bash",
"TERM"=>"xterm",
"SSH_CLIENT"=>"24.23.182.32
38170 22", "OLDPWD"=>"/root",
"SSH_TTY"=>"/dev/pts/5",
"USER"=>"root",
"SSH_AUTH_SOCK"=>"/tmp/ssh-eeoDcvH461/agent.461",
"PATH"=>"/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local:/opt/ruby-enterprise-1.8.7-2010.02/bin",
"MAIL"=>"/var/mail/root",
"PWD"=>"/var/www/lookbook/current",
"HOME"=>"/root", "SHLVL"=>"2",
"LOGNAME"=>"root",
"SSH_CONNECTION"=>"24.23.182.32 38170 98.158.22.100 22",
"_"=>"/opt/ruby-enterprise-1.8.7-2010.02/bin/unicorn_rails",
"RAILS_ENV"=>"production",
"UNICORN_FD"=>"3"}
I, [2011-05-18T18:13:56.318531 #510] ?INFO -- : inherited
addr=/var/www/lookbook/current/tmp/sockets/unicorn.sock fd=3
I, [2011-05-18T18:13:56.318819 #510] ?INFO -- : Refreshing Gem list
BEFORE FORK:
{"SHELL"=>"/bin/bash",
"TERM"=>"xterm",
"SSH_CLIENT"=>"24.23.182.32
38170 22", "OLDPWD"=>"/root",
"SSH_TTY"=>"/dev/pts/5",
"USER"=>"root",
"SSH_AUTH_SOCK"=>"/tmp/ssh-eeoDcvH461/agent.461",
"PATH"=>"/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local:/opt/ruby-enterprise-1.8.7-2010.02/bin",
"MAIL"=>"/var/mail/root",
"PWD"=>"/var/www/lookbook/current",
"HOME"=>"/root", "SHLVL"=>"2",
"LOGNAME"=>"root",
"SSH_CONNECTION"=>"24.23.182.32 38170 98.158.22.100 22",
"_"=>"/opt/ruby-enterprise-1.8.7-2010.02/bin/unicorn_rails",
"RAILS_ENV"=>"production",
"UNICORN_FD"=>"3"}
AFTER FORK:
{"SHELL"=>"/bin/bash",
"TERM"=>"xterm",
"SSH_CLIENT"=>"24.23.182.32
38170 22", "OLDPWD"=>"/root",
"SSH_TTY"=>"/dev/pts/5",
"USER"=>"root",
"SSH_AUTH_SOCK"=>"/tmp/ssh-eeoDcvH461/agent.461",
"PATH"=>"/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local:/opt/ruby-enterprise-1.8.7-2010.02/bin",
"MAIL"=>"/var/mail/root",
"PWD"=>"/var/www/lookbook/current",
"HOME"=>"/root", "SHLVL"=>"2",
"LOGNAME"=>"root",
"SSH_CONNECTION"=>"24.23.182.32 38170 98.158.22.100 22",
"_"=>"/opt/ruby-enterprise-1.8.7-2010.02/bin/unicorn_rails",
"RAILS_ENV"=>"production",
"UNICORN_FD"=>"3"}

3) cap deploy with symlinks + restart unicorn (kill -USR2)

BEFORE EXEC:
{"SHELL"=>"/bin/bash",
"TERM"=>"xterm",
"SSH_CLIENT"=>"24.23.182.32
38170 22", "OLDPWD"=>"/root",
"SSH_TTY"=>"/dev/pts/5",
"USER"=>"root",
"SSH_AUTH_SOCK"=>"/tmp/ssh-eeoDcvH461/agent.461",
"PATH"=>"/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local:/opt/ruby-enterprise-1.8.7-2010.02/bin",
"MAIL"=>"/var/mail/root",
"PWD"=>"/var/www/lookbook/current",
"HOME"=>"/root", "SHLVL"=>"2",
"LOGNAME"=>"root",
"SSH_CONNECTION"=>"24.23.182.32 38170 98.158.22.100 22",
"_"=>"/opt/ruby-enterprise-1.8.7-2010.02/bin/unicorn_rails",
"RAILS_ENV"=>"production",
"UNICORN_FD"=>"3"}
I, [2011-05-18T18:17:13.732600 #1589] ?INFO -- : inherited
addr=/var/www/lookbook/current/tmp/sockets/unicorn.sock fd=3
I, [2011-05-18T18:17:13.732942 #1589] ?INFO -- : Refreshing Gem list
BEFORE FORK:
{"SHELL"=>"/bin/bash",
"TERM"=>"xterm",
"SSH_CLIENT"=>"24.23.182.32
38170 22", "OLDPWD"=>"/root",
"SSH_TTY"=>"/dev/pts/5",
"USER"=>"root",
"SSH_AUTH_SOCK"=>"/tmp/ssh-eeoDcvH461/agent.461",
"PATH"=>"/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local:/opt/ruby-enterprise-1.8.7-2010.02/bin",
"MAIL"=>"/var/mail/root",
"PWD"=>"/var/www/lookbook/current",
"HOME"=>"/root", "SHLVL"=>"2",
"LOGNAME"=>"root",
"SSH_CONNECTION"=>"24.23.182.32 38170 98.158.22.100 22",
"_"=>"/opt/ruby-enterprise-1.8.7-2010.02/bin/unicorn_rails",
"RAILS_ENV"=>"production",
"UNICORN_FD"=>"3"}
AFTER FORK:
{"SHELL"=>"/bin/bash",
"TERM"=>"xterm",
"SSH_CLIENT"=>"24.23.182.32
38170 22", "OLDPWD"=>"/root",
"SSH_TTY"=>"/dev/pts/5",
"USER"=>"root",
"SSH_AUTH_SOCK"=>"/tmp/ssh-eeoDcvH461/agent.461",
"PATH"=>"/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local:/opt/ruby-enterprise-1.8.7-2010.02/bin",
"MAIL"=>"/var/mail/root",
"PWD"=>"/var/www/lookbook/current",
"HOME"=>"/root", "SHLVL"=>"2",
"LOGNAME"=>"root",
"SSH_CONNECTION"=>"24.23.182.32 38170 98.158.22.100 22",
"_"=>"/opt/ruby-enterprise-1.8.7-2010.02/bin/unicorn_rails",
"RAILS_ENV"=>"production",
"UNICORN_FD"=>"3"}
worker=1 ready


any help appreciated, thanks :D
Jason

Eric Wong

2011-May-19 09:27 UTC

head link

unicorn doesn''t restart properly after cap deploy (not using Bundler)

Jason Su <jason at lookbook.nu> wrote:> Hey guys,
> 
> I''m sending USR2 to unicorn after cap deploy, and the old master
is
> getting replaced by a new master that doesn''t work.
Can you expand on "doesn''t work"?
> Here are some similar threads I found ...
> http://rubyforge.org/pipermail/mongrel-unicorn/2010-October/000733.html
> http://rubyforge.org/pipermail/mongrel-unicorn/2010-October/000717.html
> 
> I''m not using Bundler, and I have working_directory set in my
unicorn config:
I don''t see where you have working_directory set below
> Unicorn::HttpServer::START_CTX[0] >
"/opt/ruby-enterprise-1.8.7-2010.02/bin/unicorn_rails"
You shouldn''t need to touch START_CTX unless you''re switching
Ruby
installations.  It''s rope to hang yourself with.

I would simplify the config file as much as possible if you''re
debugging a problem.
> pid "/var/www/lookbook/shared/pids/unicorn.pid"
> stderr_path "/var/www/lookbook/shared/log/unicorn.stderr.log"
> stdout_path "/var/www/lookbook/shared/log/unicorn.stdout.log"
> preload_app true
> if GC.respond_to?(:copy_on_write_friendly=)
> ? GC.copy_on_write_friendly = true
> end
> before_fork do |server, worker|
> ? STDERR.puts "BEFORE FORK:"
> ? STDERR.puts ENV.inspect
> ? defined?(ActiveRecord::Base) and
> ? ? ActiveRecord::Base.connection.disconnect!
The auto-killing old workers bit is fragile and you shouldn''t need
it unless you''re very low on memory.
> ? if old_pid != server.pid
> ? ? begin
> ? ? ? sig = (worker.nr?+ 1) >= server.worker_processes ? :QUIT : :TTOU
> ? ? ? Process.kill(sig, File.read(old_pid).to_i)
> ? ? rescue Errno::ENOENT, Errno::ESRCH
> ? ? end
> ? end
> 
> end
> after_fork do |server, worker|
> ? STDERR.puts "AFTER FORK:"
> ? STDERR.puts ENV.inspect
> 
> ? defined?(ActiveRecord::Base) and
> ? ? ActiveRecord::Base.establish_connection
> ? begin
If you want to use user switching, the "user" directive which
appeared a while back is much simpler and less error prone.
> ? ? uid, gid = Process.euid, Process.egid
> ? ? user, group = ''apache'', ''apache''
> ? ? target_uid = Etc.getpwnam(user).uid
> ? ? target_gid = Etc.getgrnam(group).gid
> ? ? worker.tmp.chown(target_uid, target_gid)
> ? ? if uid != target_uid || gid != target_gid
> ? ? ? Process.initgroups(user, target_gid)
> ? ? ? Process::GID.change_privilege(target_gid)
> ? ? ? Process::UID.change_privilege(target_uid)
> ? ? end
<snip>
> Below is the output from unicorn.stderr.log.. I''m not sure if
> UNICORN_FD has something to do with it?
UNICORN_FD is absolutely for the old master tells the new master about
the listener sockets to inherit.  Don''t touch it.

<snip>
> 3) cap deploy with symlinks + restart unicorn (kill -USR2)
<snip>
> addr=/var/www/lookbook/current/tmp/sockets/unicorn.sock fd=3
You should make sure your socket is in a shared path, not current.

Whatever''s in current gets replaced by Capistrano, right?  I''m
not sure
if tmp is a symlink that''s relinked on a new deploy, but there could be
a race condition where the path to the socket is unreachable and
nginx can''t see it.

-- 
Eric Wong

Jason Su

2011-May-19 20:26 UTC

head link

unicorn doesn''t restart properly after cap deploy (not using Bundler)

You were right, the socket was not in a shared path... D''oh!

Thanks so much~

On Thu, May 19, 2011 at 2:27 AM, Eric Wong <normalperson at yhbt.net>
wrote:> Jason Su <jason at lookbook.nu> wrote:
>> Hey guys,
>>
>> I''m sending USR2 to unicorn after cap deploy, and the old
master is
>> getting replaced by a new master that doesn''t work.
>
> Can you expand on "doesn''t work"?
>
>> Here are some similar threads I found ...
>> http://rubyforge.org/pipermail/mongrel-unicorn/2010-October/000733.html
>> http://rubyforge.org/pipermail/mongrel-unicorn/2010-October/000717.html
>>
>> I''m not using Bundler, and I have working_directory set in my
unicorn config:
>
> I don''t see where you have working_directory set below
>
>> Unicorn::HttpServer::START_CTX[0] >>
"/opt/ruby-enterprise-1.8.7-2010.02/bin/unicorn_rails"
>
> You shouldn''t need to touch START_CTX unless you''re
switching Ruby
> installations. ?It''s rope to hang yourself with.
>
> I would simplify the config file as much as possible if you''re
> debugging a problem.
>
>> pid "/var/www/lookbook/shared/pids/unicorn.pid"
>> stderr_path "/var/www/lookbook/shared/log/unicorn.stderr.log"
>> stdout_path "/var/www/lookbook/shared/log/unicorn.stdout.log"
>> preload_app true
>> if GC.respond_to?(:copy_on_write_friendly=)
>> ? GC.copy_on_write_friendly = true
>> end
>> before_fork do |server, worker|
>> ? STDERR.puts "BEFORE FORK:"
>> ? STDERR.puts ENV.inspect
>> ? defined?(ActiveRecord::Base) and
>> ? ? ActiveRecord::Base.connection.disconnect!
>
> The auto-killing old workers bit is fragile and you shouldn''t need
> it unless you''re very low on memory.
>
>> ? if old_pid != server.pid
>> ? ? begin
>> ? ? ? sig = (worker.nr?+ 1) >= server.worker_processes ? :QUIT :
:TTOU
>> ? ? ? Process.kill(sig, File.read(old_pid).to_i)
>> ? ? rescue Errno::ENOENT, Errno::ESRCH
>> ? ? end
>> ? end
>>
>> end
>> after_fork do |server, worker|
>> ? STDERR.puts "AFTER FORK:"
>> ? STDERR.puts ENV.inspect
>>
>> ? defined?(ActiveRecord::Base) and
>> ? ? ActiveRecord::Base.establish_connection
>> ? begin
>
> If you want to use user switching, the "user" directive which
> appeared a while back is much simpler and less error prone.
>
>> ? ? uid, gid = Process.euid, Process.egid
>> ? ? user, group = ''apache'',
''apache''
>> ? ? target_uid = Etc.getpwnam(user).uid
>> ? ? target_gid = Etc.getgrnam(group).gid
>> ? ? worker.tmp.chown(target_uid, target_gid)
>> ? ? if uid != target_uid || gid != target_gid
>> ? ? ? Process.initgroups(user, target_gid)
>> ? ? ? Process::GID.change_privilege(target_gid)
>> ? ? ? Process::UID.change_privilege(target_uid)
>> ? ? end
>
> <snip>
>
>> Below is the output from unicorn.stderr.log.. I''m not sure if
>> UNICORN_FD has something to do with it?
>
> UNICORN_FD is absolutely for the old master tells the new master about
> the listener sockets to inherit. ?Don''t touch it.
>
> <snip>
>
>> 3) cap deploy with symlinks + restart unicorn (kill -USR2)
>
> <snip>
>
>> addr=/var/www/lookbook/current/tmp/sockets/unicorn.sock fd=3
>
> You should make sure your socket is in a shared path, not current.
>
> Whatever''s in current gets replaced by Capistrano, right?
?I''m not sure
> if tmp is a symlink that''s relinked on a new deploy, but there
could be
> a race condition where the path to the socket is unreachable and
> nginx can''t see it.
>
> --
> Eric Wong
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying
>

Maybe Matching Threads

Search for more seemingly similar threads

mongrel unicorn - May 2011 - unicorn doesn't restart properly after cap deploy (not using Bundler)

unicorn doesn''t restart properly after cap deploy (not using Bundler)

unicorn doesn''t restart properly after cap deploy (not using Bundler)

unicorn doesn''t restart properly after cap deploy (not using Bundler)

Maybe Matching Threads