mongrel unicorn - Sep 2010 - Unicorn fails to restart gracefully on capistrano deploy

First off thanks very much for all the hard work on unicorn. Alas,
we''ve encountered an issue where unicorn fails to spawn new workers
that have loaded the incoming revision on a capistrano deploy. I''m not
entirely sure the issue is due to unicorn as it appears that bundler was
responsible for a similar issue in the past:
http://github.com/carlhuda/bundler/issues/issue/259/#comment_180830

But presumably bundler has corrected that issue and any help in sorting out the
issue would be greatly appreciated.

We are following EngineYard''s instructions for loading unicorn via
bundle exec:
http://docs.engineyard.com/appcloud/howtos/cutomization/customize-unicorn

So far we''ve tried deploying with the unicorn config set to both
"preload_app true" and "preload_app false".
(unicorn.rb being used: https://gist.github.com/9a208e3d1d1e44161018)

When "preload_app true" is used the "uninitialized constant
Prefix::OurCustomApi (NameError)" log entry/stacktrace appears once. And
the workers never load the incoming code. So while the incoming code is in place
and the symlinks are all updated we''re still running the previous
revision until a unicorn stop/start or reload is sent.

When "preload_app false" is used the "uninitialized constant
Prefix::OurCustomApi (NameError)" log entry/stacktrace is written to the
unicorn.stderr.log continuously as it keeps trying to respawn workers in a loop.
This goes on until a unicorn stop command is sent. Then a unicorn start can be
used to load the incoming revision successfully.

The common deployment stack and versions follow:

Capistrano 2.5.19
Bundler 1.0.0rc6
Rails 2.3.8
Unicorn 1.0.1 (tested under unicorn 1.1.3 with same results)

The deployments differ as follows:

The deployed versioned directory is 20100830230613
The incoming versioned directory is 20100902192111

Code in version 20100830230613 uses our_custom_api.gem version 1.0 via the
Gemfile
Code in version 20100830230613 uses the constant OurCustomApi in
config/initializers/our_custom_api.rb

Code in version 20100902192111 uses prefix_our_custom_api.gem version 2.0 via
the Gemfile
Code in version 20100902192111 uses the constant Prefix::OurCustomApi in
config/initializers/our_custom_api.rb

The relevant relevant unicorn.stderr.log can be found below and appear to show
some sort of issue where both versioned directories are being loaded into the
same process?

Unicorn 1.0.1:
https://gist.github.com/ddc852b23617fc8584e0

Unicorn 1.1.3:
https://gist.github.com/6314ea0e14ff78b4e8dc

Thanks in advance,

Eirik Dentz Sinclair

On Sep 9, 2010, at 4:28 PM, Eirik Dentz Sinclair wrote:
> First off thanks very much for all the hard work on unicorn. Alas,
we''ve encountered an issue where unicorn fails to spawn new workers
that have loaded the incoming revision on a capistrano deploy. I''m not
entirely sure the issue is due to unicorn as it appears that bundler was
responsible for a similar issue in the past:
http://github.com/carlhuda/bundler/issues/issue/259/#comment_180830
> 
> But presumably bundler has corrected that issue and any help in sorting out
the issue would be greatly appreciated.
> 
> [snip]
> 
> The deployed versioned directory is 20100830230613
> The incoming versioned directory is 20100902192111
I think I may have run into a potentially related bundler+unicorn+capistrano
issue just today!

With bundler+unicorn the shell environment does not really get cleared between
USR2 restarts/forkings, which was the cause of the aforementioned GitHub
ticket... PATH/RUBYOPT would just get infinitely appended to by bundler, and
unicorn is happy to let you do whatever with the environment

The bug I found today cropped up while upgrading from rails 2.3.8 to rails 2.3.9
-- on deploy the new unicorn master would die with "hey you''ve
already loaded rails 2.3.8!". Weird.

Some quick debugging showed my BUNDLE_GEMFILE environment variable was
referencing a Gemfile from a (very) old capistrano release.... I''m
actively debugging it now. I''m also using preload_app=true which might
have contributed.

Eirik, can you confirm any similar "old release" weirdnesses by
dumping the ENV in your unicorn.rb''s before_exec / before_fork?

	 STDERR.puts ENV.inspect

See the before_exec in http://gist.github.com/534668


Cheers,
Lawrence
> On Sep 9, 2010, at 4:28 PM, Eirik Dentz Sinclair wrote:
>
>> First off thanks very much for all the hard work on unicorn. Alas,
we''ve encountered an issue where unicorn fails to spawn new workers
that have loaded the incoming revision on a capistrano deploy. I''m not
entirely sure the issue is due to unicorn as it appears that bundler was
responsible for a similar issue in the past:
http://github.com/carlhuda/bundler/issues/issue/259/#comment_180830
>>
>> But presumably bundler has corrected that issue and any help in sorting
out the issue would be greatly appreciated.
>>
>> [snip]
>>
>> The deployed versioned directory is 20100830230613
>> The incoming versioned directory is 20100902192111
> I think I may have run into a potentially related
bundler+unicorn+capistrano issue just today!
>
> With bundler+unicorn the shell environment does not really get cleared
between USR2 restarts/forkings, which was the cause of the aforementioned GitHub
ticket... PATH/RUBYOPT would just get infinitely appended to by bundler, and
unicorn is happy to let you do whatever with the environment
>
> The bug I found today cropped up while upgrading from rails 2.3.8 to rails
2.3.9 -- on deploy the new unicorn master would die with "hey
you''ve already loaded rails 2.3.8!". Weird.
>
> Some quick debugging showed my BUNDLE_GEMFILE environment variable was
referencing a Gemfile from a (very) old capistrano release.... I''m
actively debugging it now. I''m also using preload_app=true which might
have contributed.
>
> Eirik, can you confirm any similar "old release" weirdnesses by
dumping the ENV in your unicorn.rb''s before_exec / before_fork?
>
> 	 STDERR.puts ENV.inspect
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying
>

Lawrence, Jamie

Thanks very much for your responses.

The before_exec block in Lawrence''s gist was just what was needed. Not
sure how many others are using Capistrano and bundler, but seems like that
before_exec block would be a good addition to this page:
http://unicorn.bogomips.org/Sandbox.html

Thanks again for the help.

-Eirik

Eirik Dentz Sinclair <eirik at efficiency20.com>
wrote:
> Lawrence, Jamie
> 
> Thanks very much for your responses.
> 
> The before_exec block in Lawrence''s gist was just what was needed.
Not sure how many others are using Capistrano and bundler, but seems like that
before_exec block would be a good addition to this page:
http://unicorn.bogomips.org/Sandbox.html
> 
> Thanks again for the help.
Thanks for confirming the fix.  I''ve updated the Sandbox document
with the following patch.

Patches/pull-requests to the mailing list for documentation are very
much appreciated, especially for Bundler and maybe other things I don''t
use myself.  Lets try to treat the in-source documentation much like a
wiki, except anybody who grabs the source also has the latest up-to-date
docs ready for offline reading (and readable without a web browser :)
>From 1a75966a5d1a1f6307ed3386e2f91a28bbb72ca0 Mon Sep 17 00:00:00 2001
From: Eric Wong <normalperson at yhbt.net>
Date: Wed, 15 Sep 2010 14:42:54 -0700
Subject: [PATCH] doc: update Sandbox document for Bundler

Thanks to Lawrence Pit, Jamie Wilkinson, and Eirik Dentz Sinclair.

ref: mid.gmane.org/4C8986DA.7090603 at gmail.com
ref: mid.gmane.org/5F1A02DB-CBDA-4302-9E26-8050C2D72433 at efficiency20.com
---
 Sandbox |    9 +++++++++
 1 files changed, 9 insertions(+), 0 deletions(-)

diff --git a/Sandbox b/Sandbox
index d2f7590..d101106 100644
--- a/Sandbox
+++ b/Sandbox
@@ -24,6 +24,9 @@ this:
 Then use HUP to reload, and then continue with the USR2+QUIT upgrade
 sequence.
 
+Environment variable pollution when exec-ing a new process (with USR2)
+is the primary issue with sandboxing tools such as Bundler and Isolate.
+
 == Bundler
 
 === Running
@@ -42,6 +45,12 @@ This is no longer be an issue as of bundler 0.9.17
 
 ref: http://mid.gmane.org/8FC34B23-5994-41CC-B5AF-7198EF06909E at tramchase.com
 
+=== Other ENV pollution issues
+
+You may need to set or reset BUNDLE_GEMFILE, GEM_HOME, GEM_PATH and PATH
+environment variables in the before_exec hook as illustrated by
+http://gist.github.com/534668
+
 == Isolate
 
 === Running
-- 
Eric Wong