mongrel unicorn - Feb 2012 - Bug in unicorn_rails when Encoding.default_internal is set

Hi,

We recently made the switch to 1.9 for our large rails 2.3.14 project
and ran into some strange behavior from the unicorns where encoding
problem would cause blank 500 pages to be "randomly" served to users
(we run ~10 x 5 unicorn workers behind our load balancer). We found
that unicorn workers were dying when someone requested a url with
utf-8 in it, and died in such a way that subsequent requests to that
worker would get the same blank 500 page even for plain ascii urls.

I''ve been able to recreate this behavior one a fresh rails project by
setting Encoding.default_internal to utf-8 and very little else. I''ve
created an example repo at:

    https://github.com/lparry/how_to_kill_a_unicorn

All you need to do is hit http://localhost:8080/users/? and the
unicorn worker will die with an error:

"Error during failsafe response: "\xE2" from ASCII-8BIT to
UTF-8"
"app error: "You have a nil object when you didn''t expect
it!\nYou
might have expected an instance of Array.\nThe error occurred while
evaluating nil.each" (NoMethodError)"

All subsequent requests to that worker will fail with the same exception.

Is this a known issue? Is there a recommended work around?

Cheers,

Lucas Parry
Envato

On Fri, Feb 17, 2012 at 1:29 PM, Lucas Parry <lucas at envato.com>
wrote:
> We recently made the switch to 1.9 for our large rails 2.3.14 project
I guess I should be more precise; This problem is occurring on MRI1.9.2-p290

Lucas Parry <lucas at envato.com> wrote:
> On Fri, Feb 17, 2012 at 1:29 PM, Lucas Parry <lucas at envato.com>
wrote:
> > We recently made the switch to 1.9 for our large rails 2.3.14 project
(odd, neither I nor gmane got your original message:
 http://rubyforge.org/pipermail/mongrel-unicorn/2012-February/001310.html
 Perhaps Mailman misdelivers UTF-8...
)
> I guess I should be more precise; This problem is occurring on
MRI1.9.2-p290
Setting Encoding.default_* is probably a bad idea in your app.
If you run Ruby with warnings (-w switch), you''ll see setting
Encoding.default_* will generate warnings to discourage their use.

If you want UTF-8, set LANG=en_US.UTF-8 in the environment _before_
starting Ruby, which will only set Encoding.default_external.

I''ve always just left Encoding.default_internal alone.  I''ll
let others
on the list with more Rails/encoding experience help you with that.
> All subsequent requests to that worker will fail with the same
> exception.
Seems to be Rails bug.  Rails 3.2.1 seems to handle the situation
without difficulty.


Fwiw, neither Rack nor HTTP specify an encoding for HTTP headers/paths.
The only place where encoding is specified for Rack is "rack.input",
which must be binary.  There was talk on the Rack mailing list a long
while back about enforcing encodings for certain headers, but nothing
came of it.