Changes: Theses release fix a long-standing bug where the original PID file is not restored when rolling back from a USR2 upgrade. Presumably most upgrades aren''t rolled back, so it took over a year to notice this issue. Thanks to Lawrence Pit for discovering and reporting this issue. About Unicorn: Unicorn is an HTTP server for Rack applications designed to only serve fast clients on low-latency, high-bandwidth connections and take advantage of features in Unix/Unix-like kernels. Slow clients should only be served by placing a reverse proxy capable of fully buffering both the the request and response in between Unicorn and slow clients. * http://unicorn.bogomips.org/ * mongrel-unicorn at rubyforge.org * git://git.bogomips.org/unicorn.git -- Eric Wong
Hi Eric, Thanks mate. Unfortunately this still doesn''t seem to work for me. I went through the process manually, you can see a transcript at http://pastie.org/1043347.txt The indented text is what I see appearing in the unicorn stderr.log. As you can see it reports two errors during the process, saying unicorn.pid is stale. I wonder whether this is because in my unicorn config file I mention: pid "/var/www/staging/current/tmp/pids/unicorn.pid" whereas in your back_out_of_upgrade test you don''t have such a line? Cheers, Lawrence> Changes: > > Theses release fix a long-standing bug where the original PID > file is not restored when rolling back from a USR2 upgrade. > Presumably most upgrades aren''t rolled back, so it took over a > year to notice this issue. Thanks to Lawrence Pit for > discovering and reporting this issue. > > About Unicorn: > > Unicorn is an HTTP server for Rack applications designed to only serve > fast clients on low-latency, high-bandwidth connections and take > advantage of features in Unix/Unix-like kernels. Slow clients should > only be served by placing a reverse proxy capable of fully buffering > both the the request and response in between Unicorn and slow clients. > > * http://unicorn.bogomips.org/ > * mongrel-unicorn at rubyforge.org > * git://git.bogomips.org/unicorn.git > >
Lawrence Pit <lawrence.pit at gmail.com> wrote:> Hi Eric, > > Thanks mate. Unfortunately this still doesn''t seem to work for me. > > I went through the process manually, you can see a transcript at > http://pastie.org/1043347.txt > The indented text is what I see appearing in the unicorn stderr.log.Thanks Lawrence, the following line looks suspicious to me: reaped #<Process::Status: pid=4382,exited(0)> exec()-ed Which makes it seem like http://git.bogomips.org/cgit/unicorn.git/commit/?id=3f0f9d6 didn''t get applied (and I just checked the 1.1.2 gem and it looks good to me) Can you dump out the ENV hash in your before_exec hook of your unicorn.conf.rb? I''m wondering if GEM_HOME or GEM_PATH is set and doing anything funky, causing you to load an old version of Unicorn: before_exec do |server| ENV.each do |key,value| server.logger.info "#{key}=#{value}" end end> As you can see it reports two errors during the process, saying > unicorn.pid is stale. > > I wonder whether this is because in my unicorn config file I mention: > > pid "/var/www/staging/current/tmp/pids/unicorn.pid" > > whereas in your back_out_of_upgrade test you don''t have such a line?It does, I create a similar line in the unicorn_setup() function of test-lib.sh -- Eric Wong
Hi Eric, At the top of my pastie http://pastie.org/1043347.txt you can see that when I request of list of gems it mentions only unicorn v1.1.2 I also added the suggested logging to the before_fork block and I also logged the unicorn version. Interesting output. First it runs the master with Unicorn v1.1.2. But when I send a USR2 it exec''ed with a Unicorn v1.1.1 for the new master. So yes, I still had a version lying around, namely in the $APP_ROOT/vendor/bundler_gems dir. See also: http://pastie.org/1043487.txt It does do this though after a USR2: executing ["/usr/local/rvm/gems/ree-1.8.7-2010.02/bin/unicorn_rails", "-D", "-E", "staging", "-c", "/var/www/staging/current/config/unicorn/staging.rb"] I.e., it tries to exec exactly how I start unicorn. That unicorn_rails binary in that path is definitely v1.1.2. So I don''t quite understand why the reexeced one is picking the version from the vendor/bundler_gems dir. I need to play a bit more with this.. I''ll probably have to change my /etc/init.d/unicorn script so that it starts with the vendor/bundler_gems version instead of a system-wide version :o I''ll get back to you.. Cheers, Lawrence> Lawrence Pit <lawrence.pit at gmail.com> wrote: > >> Hi Eric, >> >> Thanks mate. Unfortunately this still doesn''t seem to work for me. >> >> I went through the process manually, you can see a transcript at >> http://pastie.org/1043347.txt >> The indented text is what I see appearing in the unicorn stderr.log. >> > > Thanks Lawrence, the following line looks suspicious to me: > > reaped #<Process::Status: pid=4382,exited(0)> exec()-ed > > Which makes it seem like > http://git.bogomips.org/cgit/unicorn.git/commit/?id=3f0f9d6 didn''t get > applied (and I just checked the 1.1.2 gem and it looks good to me) > > Can you dump out the ENV hash in your before_exec hook of your > unicorn.conf.rb? I''m wondering if GEM_HOME or GEM_PATH is set > and doing anything funky, causing you to load an old version > of Unicorn: > > before_exec do |server| > ENV.each do |key,value| > server.logger.info "#{key}=#{value}" > end > end > > >> As you can see it reports two errors during the process, saying >> unicorn.pid is stale. >> >> I wonder whether this is because in my unicorn config file I mention: >> >> pid "/var/www/staging/current/tmp/pids/unicorn.pid" >> >> whereas in your back_out_of_upgrade test you don''t have such a line? >> > > It does, I create a similar line in the unicorn_setup() function of > test-lib.sh > >
Lawrence Pit <lawrence.pit at gmail.com> wrote:> Hi Eric, > > At the top of my pastie http://pastie.org/1043347.txt you can see that > when I request of list of gems it mentions only unicorn v1.1.2 > > I also added the suggested logging to the before_fork block and I also > logged the unicorn version. Interesting output. First it runs the master > with Unicorn v1.1.2. But when I send a USR2 it exec''ed with a Unicorn > v1.1.1 for the new master. So yes, I still had a version lying around, > namely in the $APP_ROOT/vendor/bundler_gems dir. > > See also: http://pastie.org/1043487.txt > > It does do this though after a USR2: > > executing ["/usr/local/rvm/gems/ree-1.8.7-2010.02/bin/unicorn_rails", > "-D", "-E", "staging", "-c", > "/var/www/staging/current/config/unicorn/staging.rb"] > > I.e., it tries to exec exactly how I start unicorn. That unicorn_rails > binary in that path is definitely v1.1.2. So I don''t quite understand > why the reexeced one is picking the version from the vendor/bundler_gems > dir.It''s probably loading 1.1.1 because of GEM_HOME/GEM_PATH being set. The "unicorn_rails"/"unicorn" wrappers just activates whatever gems it sees, and GEM_HOME/GEM_PATH can influence the wrapper. Managing RubyGems installations is still tricky :<> I need to play a bit more with this.. I''ll probably have to change my > /etc/init.d/unicorn script so that it starts with the > vendor/bundler_gems version instead of a system-wide version :o I''ll > get back to you..Take a look at http://unicorn.bogomips.org/Sandbox.html According to Jamie, using "bundle exec unicorn_rails" is the correct way to go. -- Eric Wong
>> I need to play a bit more with this.. I''ll probably have to change my >> /etc/init.d/unicorn script so that it starts with the >> vendor/bundler_gems version instead of a system-wide version :o I''ll >> get back to you.. >> > > Take a look at http://unicorn.bogomips.org/Sandbox.html According to > Jamie, using "bundle exec unicorn_rails" is the correct way to go. >Thanks. Yes indeed, bundle exec is the way to go. I''ve got it rolling back nicely now. It appears unicorn rolls back by itself when it can''t start the new master. Nice! Jamie also mentions to use the shared vendor/bundler_gems path. Though I do use that in all my projects I''m not so sure it''s wise to do when you can''t afford any downtime. I''ll have to wait for unicorn v1.1.3 before I can test whether rolling back will indeed continue with unicorn v1.1.2 instead of v1.1.3 ;o Cheers, Lawrence
On Jul 14, 2010, at 12:00 AM, Lawrence Pit wrote:> It appears unicorn rolls back by itself when it can''t start the new master. Nice! > > Jamie also mentions to use the shared vendor/bundler_gems path. Though I do use that in all my projects I''m not so sure it''s wise to do when you can''t afford any downtime. I''ll have to wait for unicorn v1.1.3 before I can test whether rolling back will indeed continue with unicorn v1.1.2 instead of v1.1.3 ;oHi Lawrence, why does the shared vendor/bundler_gems cause you downtime? From not re-bundling during rollback? FWIW I made that recommendation because I ran into issues with unicorns not restarting correctly after running `cap deploy:cleanup`, since the `bundle exec` launches a binary with a path like /app/releases/201007125236/bin/unicorn, which goes missing after N deploys. I haven''t tested if this applies to more recent versions Shared bundles are also significantly faster -- `bundle check || bundle install` is ~1s for me vs. several minutes to totally rebundle -jamie http://jamiedubs.com http://fffff.at
>> It appears unicorn rolls back by itself when it can''t start the new master. Nice! >> >> Jamie also mentions to use the shared vendor/bundler_gems path. Though I do use that in all my projects I''m not so sure it''s wise to do when you can''t afford any downtime. I''ll have to wait for unicorn v1.1.3 before I can test whether rolling back will indeed continue with unicorn v1.1.2 instead of v1.1.3 ;o >> > > Hi Lawrence, why does the shared vendor/bundler_gems cause you downtime? From not re-bundling during rollback? > > FWIW I made that recommendation because I ran into issues with unicorns not restarting correctly after running `cap deploy:cleanup`, since the `bundle exec` launches a binary with a path like /app/releases/201007125236/bin/unicorn, which goes missing after N deploys. I haven''t tested if this applies to more recent versions > > Shared bundles are also significantly faster -- `bundle check || bundle install` is ~1s for me vs. several minutes to totally rebundle >Hi Jamie, I see what you mean. I haven''t tested this yet, so I don''t know for sure if it could cause downtime. I also always use bundle check || bundle install to get that time benefit. My worry was indeed that it might pick a newer unicorn to restart workers for the old master. I guess that can/should be prevented by having an on_rollback in my capistrano bundler task. What is the exact command you use to start unicorn? What I see is this, I start with: $ su -l deployer -c "cd /app/current ; bundle exec unicorn_rails -D -c /app/current/config/unicorn.rb" My after_fork and before_exec methods output the ENV vars to the unicorn log. I see for example: AFTER_FORK: VERSION 1 PATH=/app/releases/20100714054705/vendor/bundler_gems/bin:.... GEM_HOME=/app/releases/20100714054705/vendor/bundler_gems GEM_PATH=/app/releases/20100714054705/vendor/bundler_gems:/app/releases/20100714054705/vendor/bundler_gems/bundler/gems BUNDLE_GEMFILE=/app/releases/20100714054705/Gemfile Note that /app/releases/20100714054705/vendor/bundler_gems is a symlink to /app/releases/shared/vendor/bundler_gems When I deploy a new version, symlink /app/current to the new release directory, and send USR2 : executing ["/app/releases/20100714054705/vendor/bundler_gems/bin/unicorn_rails", "-D", "-E", "staging", "-c", "/app/current/config/unicorn.rb"] (in /app/releases/20100714055624) BEFORE_EXEC: VERSION 1 PATH=/app/releases/20100714054705/vendor/bundler_gems/bin:.... GEM_HOME=/app/releases/20100714054705/vendor/bundler_gems GEM_PATH=/app/releases/20100714054705/vendor/bundler_gems:/app/releases/20100714054705/vendor/bundler_gems/bundler/gems BUNDLE_GEMFILE=/app/releases/20100714054705/Gemfile I, [2010-07-15T23:31:47.765995 #23084] INFO -- : inherited addr=/tmp/.unicorn_sock fd=3 I, [2010-07-15T23:31:47.766646 #23084] INFO -- : Refreshing Gem list /app/releases/20100714055624/.bundle/environment.rb:175: warning: already initialized constant ENV_LOADED /app/releases/20100714055624/.bundle/environment.rb:176: warning: already initialized constant LOCKED_BY /app/releases/20100714055624/.bundle/environment.rb:177: warning: already initialized constant FINGERPRINT /app/releases/20100714055624/.bundle/environment.rb:178: warning: already initialized constant HOME /app/releases/20100714055624/.bundle/environment.rb:179: warning: already initialized constant AUTOREQUIRES /app/releases/20100714055624/.bundle/environment.rb:181: warning: already initialized constant SPECS AFTER_FORK: VERSION 2 PATH=/app/releases/20100714054705/vendor/bundler_gems/bin:.... GEM_HOME=/app/releases/20100714055624/vendor/bundler_gems GEM_PATH=/app/releases/20100714055624/vendor/bundler_gems:/app/releases/20100714055624/vendor/bundler_gems/bundler/gems BUNDLE_GEMFILE=/app/releases/20100714054705/Gemfile What you see here is that the new worker does have correct GEM_HOME and GEM_PATH values, but the PATH and BUNDLE_GEMFILE values are pointing to the old dir. Are those bundler warnings something to worry about? The BUNDLE_GEMFILE value is worrying I think. I haven''t tested this, but I''m pretty sure when Bundler.setup is called within your app it will actually setup using the old Gemfile then. So that would mean you need to reset it somehow? I don''t see how at the moment... should unicorn provide a method similar to +working_directory+ to help ensure the application always uses the "current" Gemfile (eg "/app/current/Gemfile") ? Sound kind of strange unicorn should provide such a method, is there another way? What about that PATH value? What happens if 10 deployments later the dir /app/releases/20100714054705 is pruned? I tried. After removing that dir I could still send a HUP, but when I send a USR2 I get this: /app/releases/20100714054705/vendor/bundler_gems/gems/unicorn-1.1.2/lib/unicorn.rb:573:in `exec'': No such file or directory - app/releases/20100714054705/vendor/bundler_gems/bin/unicorn_rails (Errno::ENOENT) from /app/releases/20100714054705/vendor/bundler_gems/gems/unicorn-1.1.2/lib/unicorn.rb:573:in `reexec'' > since the `bundle exec` launches a binary with a path like /app/releases/201007125236/bin/unicorn, > which goes missing after N deploys That''s what I''m seeing. How did using shared vendor/bundler_gems help you? Because I''m starting with bundle exec using shared vendor/bundler_gems as well. Cheers, Lawrence
> What about that PATH value? What happens if 10 deployments later the > dir /app/releases/20100714054705 is pruned? I tried. After removing > that dir I could still send a HUP, but when I send a USR2 I get this:Eric had already pointed me to http://unicorn.bogomips.org/Sandbox.html before, I forgot to add that.. this does the trick: Unicorn::HttpServer::START_CTX[0] = "/app/shared/bundler_gems/bin/unicorn_rails" Cheers, Lawrence