Hi, When deploying new code we go through the USR2 QUIT sequence. This works very nicely and gives zero downtime. My question is whether there are instances when this sequence is known to not work, and instead you should really send QUIT first and then start all over? I didn''t expect that to be the case, but in the past year occasionally I''ve experienced I had to resort to QUIT and start all over in order to get all components loaded correctly. Specifically: yesterday I upgraded several projects to rails 3.0.11 and added a new i18n .yml file in the config dir. After the USR2 QUIT sequence all new code seemed to work fine. Except the new .yml file wasn''t loaded. Another run of USR2 and QUIT didn''t resolve it. Only after QUIT and start of unicorn was the new .yml file loaded. This was not just on one machine, which might have been a fluke. This was on all machines for all projects, consistently. Any ideas? Cheers, Lawrence
Laurens Pit <laurens.pit at mirror42.com> wrote:> I didn''t expect that to be the case, but in the past year occasionally > I''ve experienced I had to resort to QUIT and start all over in order > to get all components loaded correctly. > > Specifically: yesterday I upgraded several projects to rails 3.0.11 > and added a new i18n .yml file in the config dir. After the USR2 QUIT > sequence all new code seemed to work fine. Except the new .yml file > wasn''t loaded. Another run of USR2 and QUIT didn''t resolve it. Only > after QUIT and start of unicorn was the new .yml file loaded. > > This was not just on one machine, which might have been a fluke. This > was on all machines for all projects, consistently. > > Any ideas?Anything from stderr log files? USR2 will fail if there''s * compile/load error when loading the app (if preload_app=true) * unicorn/unicorn_rails executable script is missing (maybe Bundler is moving it?) * Ruby installation got moved/shifted/changed * Working directory got _moved_ (cap may cycle those out) You can set "working_directory" in your unicorn config to work around it. (and maybe other reasons I can''t think of right now) Come to think of it, the missing working directory case could be the most common... But stderr log files (stderr_path) should always tell you what''s going on. == Linux(-only?) tip: If the log files got deleted somehow, you may still be able to read it via: "cat /proc/$PID/fd/2" on either the PID of the master or _any_ worker process. To read some other log files, you can just replace the "2" with whatever file descriptor. Reading the symlinks ("ls -l /proc/$PID/fd/") will tell you where each descriptor is pointed to, even if it is a deleted file.
> Anything from stderr log files? USR2 will fail if there''sRight:> ruby: no such file to load -- bundler/setup (LoadError)> * unicorn/unicorn_rails executable script is missing > (maybe Bundler is moving it?)Bundler has been updated several times this year, I guess every time that happens and a new version gets installed a complete restart of unicorn is required? Am I the only one seeing this? Perhaps I''m managing bundler wrong? Bundler and rake are the only two gems I install in the global gem space. With an update of bundler I do "gem uninstall bundler ; gem bundler install". Should I manage this differently? Cheers, Lawrence
Lawrence Pit <lawrence.pit at gmail.com> wrote:> Bundler has been updated several times this year, I guess every time > that happens and a new version gets installed a complete restart of > unicorn is required? Am I the only one seeing this?Managing bunder has always been tricky, unfortunately.> Perhaps I''m managing bundler wrong? Bundler and rake are the only two > gems I install in the global gem space. With an update of bundler I do > "gem uninstall bundler ; gem bundler install". Should I manage this > differently?I brought up a "bring-your-own-executable" deployment a few months ago but haven''t heard any feedback. Here it is for Bundler: http://mid.gmane.org/20110819022316.GA2951 at dcvr.yhbt.net I am also using something very similar with Isolate: ------------- /full/path/to/app_root/script/unicorn-byoe --------- #!/home/ew/ruby-1.9.3/bin/ruby # Unicorn: bring-your-own executable require "isolate/now" $LOAD_PATH << "./lib" load Gem.bin_path("unicorn", "unicorn") ------------------------------- 8< ------------------------------- I run the path using the full pathname, so there''s less ambiguity as to what gets captured as $0 and executed on USR2. I don''t know of anybody else doing/trying this, though.
> I brought up a "bring-your-own-executable" deployment a few months ago > but haven''t heard any feedback. Here it is for Bundler: > > http://mid.gmane.org/20110819022316.GA2951 at dcvr.yhbt.netI tried this out by going back to a previous version of bundler, then deployed, unfortunately doesn''t seem to work for me: executing ["/srv/ec/current/script/unicorn_byoe", "-D", "-E", "staging", "-c", "/srv/ec/current/config/unicorn/staging.rb"] (in /srv/ec/releases/20111123052840) /opt/ree/bin/ruby: no such file to load -- bundler/setup (LoadError) Sending QUIT and then start anew works fine. Upgraded back to latest, tried again, and again no luck. QUIT + start is the only thing that works then. Tried a slightly modified version of unicorn_byoe, I''m using rails: #!/opt/ree/bin/ruby require File.expand_path("../../config/boot.rb", __FILE__) # Gem.bin_path(gem_name, executable_name, *version_requirements) load Gem.bin_path("unicorn", "unicorn_rails") Cheers, Lawrence