I upgraded to Unicorn 0.97.0 from 0.96.1. Unicorn starts fine, but sending a USR2 to the 0.97.0 master launches a new master and all the workers, but never kills the old master. The new workers retry binding their admin TCP ports forever. Killing all the Unicorns and downgrading to returns peace to the forest. Here are the before_fork and after_fork from my config/unicorn.rb: http://codepad.org/5LyVtyq7 Did something change with the Unicorn master in 0.97.0 that would require a change in my config, or is this a bug? -Greg
ghazel at gmail.com wrote:> I upgraded to Unicorn 0.97.0 from 0.96.1. Unicorn starts fine, but > sending a USR2 to the 0.97.0 master launches a new master and all the > workers, but never kills the old master. The new workers retry binding > their admin TCP ports forever. Killing all the Unicorns and > downgrading to returns peace to the forest. > > Here are the before_fork and after_fork from my config/unicorn.rb: > http://codepad.org/5LyVtyq7 > > Did something change with the Unicorn master in 0.97.0 that would > require a change in my config, or is this a bug? > > -GregHi Greg, Odd, are you passing the pid file path via the command-line? I don''t see the pid directive in the config file you posted there, so there was nothing to check with File.exists?. Are you using `unicorn'' or `unicorn_rails''? There were some startup changes in 0.97.0, but I couldn''t reproduce what you''re seeing with ''pid''. That said, I''m not sure if that method of killing old masters in the after_fork is very common. -- Eric Wong
On Thu, Mar 18, 2010 at 1:40 AM, Eric Wong <normalperson at yhbt.net> wrote:> ghazel at gmail.com wrote: >> I upgraded to Unicorn 0.97.0 from 0.96.1. Unicorn starts fine, but >> sending a USR2 to the 0.97.0 master launches a new master and all the >> workers, but never kills the old master. The new workers retry binding >> their admin TCP ports forever. Killing all the Unicorns and >> downgrading to returns peace to the forest. >> >> Here are the before_fork and after_fork from my config/unicorn.rb: >> http://codepad.org/5LyVtyq7 >> >> Did something change with the Unicorn master in 0.97.0 that would >> require a change in my config, or is this a bug? >> >> -Greg > > Hi Greg, > > Odd, are you passing the pid file path via the command-line? > > I don''t see the pid directive in the config file you posted there, so > there was nothing to check with File.exists?. ?Are you using `unicorn'' > or `unicorn_rails''? > > There were some startup changes in 0.97.0, but I couldn''t reproduce what > you''re seeing with ''pid''. ?That said, I''m not sure if that method of > killing old masters in the after_fork is very common. > > -- > Eric Wong >Eric, I am using "unicorn_rails -D -E $ENV -c config/unicorn.rb". Here is the entire config file: http://codepad.org/v6lUsuzD What is the preferred way to do an nginx-style restart? My system does not really have the memory to start twice the number of workers. -Greg
ghazel at gmail.com wrote:> On Thu, Mar 18, 2010 at 1:40 AM, Eric Wong <normalperson at yhbt.net> wrote: > > ghazel at gmail.com wrote: > >> I upgraded to Unicorn 0.97.0 from 0.96.1. Unicorn starts fine, but > >> sending a USR2 to the 0.97.0 master launches a new master and all the > >> workers, but never kills the old master. The new workers retry binding > >> their admin TCP ports forever. Killing all the Unicorns and > >> downgrading to returns peace to the forest. > >> > >> Here are the before_fork and after_fork from my config/unicorn.rb: > >> http://codepad.org/5LyVtyq7 > >> > >> Did something change with the Unicorn master in 0.97.0 that would > >> require a change in my config, or is this a bug? > >> > >> -Greg > > > > Hi Greg, > > > > Odd, are you passing the pid file path via the command-line? > > > > I don''t see the pid directive in the config file you posted there, so > > there was nothing to check with File.exists?. ?Are you using `unicorn'' > > or `unicorn_rails''? > > > > There were some startup changes in 0.97.0, but I couldn''t reproduce what > > you''re seeing with ''pid''. ?That said, I''m not sure if that method of > > killing old masters in the after_fork is very common. > > > > -- > > Eric Wong > > > > Eric, > > I am using "unicorn_rails -D -E $ENV -c config/unicorn.rb". Here is > the entire config file: http://codepad.org/v6lUsuzDUgh. I can''t even get to that file right now. I actually prefer to have config files inline so: a) I can comment easily on them b) I don''t have to be online when reading them (I sync my mail to a local machine and lose connectivity often) On a side note, maybe an strace/truss can help figure out what''s going on. James seems to be having a similar problem in the other thread...> What is the preferred way to do an nginx-style restart? My system does > not really have the memory to start twice the number of workers.If you''re doing it during low traffic periods, I would SIGTTOU away most (maybe all but one) of your workers and then SIGHUP the master -- Eric Wong
>> Here is the entire config file: http://codepad.org/v6lUsuzD > > Ugh. ?I can''t even get to that file right now. ?I actually prefer > to have config files inline so: > > a) I can comment easily on them > b) I don''t have to be online when reading them > ? (I sync my mail to a local machine and lose connectivity often)We''re still working on that internet that works when you''re not online. In the meantime: # Sample configuration file for Unicorn (not Rack) # # See http://unicorn.bogomips.org/Unicorn/Configurator.html for complete # documentation. require ''/data/myapp/current/config/unicorn_constants'' # Use at least one worker per core if you''re on a dedicated server, # more will usually help for _short_ waits on databases/caches. worker_processes UnicornConstants::NUM_WORKERS # Help ensure your application will always spawn in the symlinked # "current" directory that Capistrano sets up. working_directory "/data/myapp/current" # available in 0.94.0+ # listen on both a Unix domain socket and a TCP port, # we use a shorter backlog for quicker failover when busy listen "/tmp/unicorn.sock", :backlog => 64 listen 8080, :tcp_nopush => true, :tries => -1 # nuke workers after X seconds instead of (60 seconds default) timeout 300 # feel free to point this anywhere accessible on the filesystem pid "/data/myapp/shared/pids/unicorn.pid" # some applications/frameworks log to stderr or stdout, so prevent # them from going to /dev/null when daemonized here: stderr_path "/data/myapp/shared/log/unicorn.stderr.log" stdout_path "/data/myapp/shared/log/unicorn.stdout.log" # combine REE with "preload_app true" for memory savings # http://rubyenterpriseedition.com/faq.html#adapt_apps_for_cow preload_app true GC.respond_to?(:copy_on_write_friendly=) and GC.copy_on_write_friendly = true before_fork do |server, worker| # the following is highly recomended for Rails + "preload_app true" # as there''s no need for the master process to hold a connection defined?(ActiveRecord::Base) and ActiveRecord::Base.connection.disconnect! # The following is only recommended for memory/DB-constrained # installations. It is not needed if your system can house # twice as many worker_processes as you have configured. # # This allows a new master process to incrementally # phase out the old master process with SIGTTOU to avoid a # thundering herd (especially in the "preload_app false" case) # when doing a transparent upgrade. The last worker spawned # will then kill off the old master process with a SIGQUIT. old_pid = "#{server.config[:pid]}.oldbin" if File.exists?(old_pid) && server.pid != old_pid begin Process.kill("QUIT", File.read(old_pid).to_i) rescue Errno::ENOENT, Errno::ESRCH # someone else did our job for us end end end after_fork do |server, worker| # per-process listener ports for debugging/admin/migrations addr = "127.0.0.1:#{9000 + worker.nr}" server.listen(addr, :tries => -1, :delay => 5, :tcp_nopush => true) # the following is *required* for Rails + "preload_app true", defined?(ActiveRecord::Base) and ActiveRecord::Base.establish_connection # if preload_app is true, then you may also want to check and # restart any other shared sockets/descriptors such as Memcached, # and Redis. TokyoCabinet file handles are safe to reuse # between any number of forked children (assuming your kernel # correctly implements pread()/pwrite() system calls) end -Greg