Hi There, Im running an amazon instance with nginx proxying to a unicorn sock. For some reason, even though i specify the production environment, when being visited by nginx, the site shows errors in development form. Interestingly, when running on a port rather than a sock, if i visit that port, the errors are rendered as normal with a 500 page, the same port, throught nginx, shows errors like you do in development. The app is rails latest (not 3), i run it with unicorn_rails -E production -c /root/pbr/current/config/unicorn.rb -D and unicorn.rb looks like.. worker_processes (20) preload_app(true) Im thinking it could be a permissions problem, the rails directory is in under root, nginx runs as user nginx, but i have given chown permission the the directory? Cheers, Alex Please CC alexbarlowis at me.com
Alex Barlow <alexbarlowis at me.com> wrote:> Hi There, > > Im running an amazon instance with nginx proxying to a unicorn sock. > > For some reason, even though i specify the production environment, > when being visited by nginx, the site shows errors in development > form. > > Interestingly, when running on a port rather than a sock, if i visit > that port, the errors are rendered as normal with a 500 page, the same > port, throught nginx, shows errors like you do in development.Hi Alex, Sorry, I''m having a little trouble following you, so you''re saying: client <- TCP -> Unicorn => quiet errors (good) client <- TCP -> nginx <- TCP -> Unicorn => verbose errors (bad) client <- TCP -> nginx <- Unix socket-> Unicorn => verbose errors (bad) Are you sure you have nginx configured correctly and pointing to the right instance of your app? Mind sharing your nginx config file?> The app is rails latest (not 3), i run it with unicorn_rails -E > production -c /root/pbr/current/config/unicorn.rb -D > > and unicorn.rb looks like.. > > worker_processes (20) > preload_app(true) > > Im thinking it could be a permissions problem, the rails directory is > in under root, nginx runs as user nginx, but i have given chown > permission the the directory?nginx error logs should tell you if you have permissions problems. I usually specify domain sockets as a dot file in /tmp since I know the directory is world read/writeable on a properly configured system and path resolution is slightly faster :) -- Eric Wong
Hi
It goes
client <- TCP -> Unicorn => quiet errors (good)
client <- TCP -> nginx <- TCP -> Unicorn => quiet errors
(good)
client <- TCP -> nginx <- Unix socket-> Unicorn => verbose
errors (bad)
Strange i know!
Ill try the socket in /tmp/ see what that does.
The Nginx error logs show nothing error wise really. My Nginx config is...
user nginx;
worker_processes 1;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
gzip on;
upstream unicorn_sock {
server unix:/root/pbr/unicorn.sock;
}
server {
listen 80;
server_name localhost;
proxy_set_header Host $host;
location / {
proxy_pass http://unicorn_sock;
}
}
}
This is currently throwing out verbose errors, in production (must be, its using
asset host and the production database)
Put the sock in /tmp, no difference
On 15 Feb 2010, at 07:30, Eric Wong wrote:
> Alex Barlow <alexbarlowis at me.com> wrote:
>> Hi There,
>>
>> Im running an amazon instance with nginx proxying to a unicorn sock.
>>
>> For some reason, even though i specify the production environment,
>> when being visited by nginx, the site shows errors in development
>> form.
>>
>> Interestingly, when running on a port rather than a sock, if i visit
>> that port, the errors are rendered as normal with a 500 page, the same
>> port, throught nginx, shows errors like you do in development.
>
> Hi Alex,
>
> Sorry, I''m having a little trouble following you, so
you''re saying:
>
> client <- TCP -> Unicorn => quiet errors
(good)
> client <- TCP -> nginx <- TCP -> Unicorn => verbose
errors (bad)
> client <- TCP -> nginx <- Unix socket-> Unicorn => verbose
errors (bad)
>
> Are you sure you have nginx configured correctly and pointing to the
> right instance of your app? Mind sharing your nginx config file?
>
>> The app is rails latest (not 3), i run it with unicorn_rails -E
>> production -c /root/pbr/current/config/unicorn.rb -D
>>
>> and unicorn.rb looks like..
>>
>> worker_processes (20)
>> preload_app(true)
>>
>> Im thinking it could be a permissions problem, the rails directory is
>> in under root, nginx runs as user nginx, but i have given chown
>> permission the the directory?
>
> nginx error logs should tell you if you have permissions problems.
>
> I usually specify domain sockets as a dot file in /tmp since I know
> the directory is world read/writeable on a properly configured system
> and path resolution is slightly faster :)
>
> --
> Eric Wong
Alex Barlow <alexbarlowis at me.com> wrote:> Hi > > It goes > > client <- TCP -> Unicorn => quiet errors (good) > client <- TCP -> nginx <- TCP -> Unicorn => quiet errors (good) > client <- TCP -> nginx <- Unix socket-> Unicorn => verbose errors (bad) > > Strange i know!Hi Alex, My gut feeling is that somehow nginx is hitting a different instance of your app when using the Unix socket. Other than that, I''m confused. More info/questions below.> Ill try the socket in /tmp/ see what that does. > > The Nginx error logs show nothing error wise really. My Nginx config is... > > user nginx; > worker_processes 1; > > events { > worker_connections 1024; > } > > http { > include mime.types; > default_type application/octet-stream; > > sendfile on; > keepalive_timeout 65; > gzip on; > > upstream unicorn_sock { > server unix:/root/pbr/unicorn.sock; > } > > server { > listen 80; > server_name localhost; > proxy_set_header Host $host; > > location / { > proxy_pass http://unicorn_sock; > } > } > }Nothing strange there, what''s the verbosity of the nginx error_log? Also, anything enlightening in the Rails production.log or Unicorn stderr? Which OS are you running? Maybe there''s a platform-specific bug somewhere, too...> This is currently throwing out verbose errors, in production (must be, > its using asset host and the production database) > > Put the sock in /tmp, no differenceIs there another Unicorn instance on the same box that it might be somehow hitting? What happens when you try have Unicorn listening on both TCP and a Unix socket? Just put both "listen" directives in your config file and point nginx to the Unix socket. Then try hitting the Unicorn TCP port directly, and then also the Unix socket via nginx. You can also try hitting the Unix socket directly by crafting your own HTTP request using socat from th shell: req=''GET / HTTP/1.1\r\nHost: example.com\r\n\r\n'' printf "$req" | socat - UNIX:/root/pbr/unicorn.sock P.S.: I might not have a chance to respond again for the next day or so due to personal matters. Maybe somebody else on this list can help.... -- Eric Wong
Fixed it!
Rails was considering all requests to be local. As the request was coming from
the local ip (from nginx i assume)
i put these headers in nginx
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect false
Works fine now
Alex
On 15 Feb 2010, at 16:19, Eric Wong wrote:
> Alex Barlow <alexbarlowis at me.com> wrote:
>> Hi
>>
>> It goes
>>
>> client <- TCP -> Unicorn => quiet
errors (good)
>> client <- TCP -> nginx <- TCP -> Unicorn =>
quiet errors (good)
>> client <- TCP -> nginx <- Unix socket-> Unicorn =>
verbose errors (bad)
>>
>> Strange i know!
>
> Hi Alex,
>
> My gut feeling is that somehow nginx is hitting a different instance of
> your app when using the Unix socket. Other than that, I''m
confused.
> More info/questions below.
>
>> Ill try the socket in /tmp/ see what that does.
>>
>> The Nginx error logs show nothing error wise really. My Nginx config
is...
>>
>> user nginx;
>> worker_processes 1;
>>
>> events {
>> worker_connections 1024;
>> }
>>
>> http {
>> include mime.types;
>> default_type application/octet-stream;
>>
>> sendfile on;
>> keepalive_timeout 65;
>> gzip on;
>>
>> upstream unicorn_sock {
>> server unix:/root/pbr/unicorn.sock;
>> }
>>
>> server {
>> listen 80;
>> server_name localhost;
>> proxy_set_header Host $host;
>>
>> location / {
>> proxy_pass http://unicorn_sock;
>> }
>> }
>> }
>
> Nothing strange there, what''s the verbosity of the nginx
error_log?
>
> Also, anything enlightening in the Rails production.log or Unicorn
> stderr?
>
> Which OS are you running? Maybe there''s a platform-specific bug
> somewhere, too...
>
>> This is currently throwing out verbose errors, in production (must be,
>> its using asset host and the production database)
>>
>> Put the sock in /tmp, no difference
>
> Is there another Unicorn instance on the same box that it might be
> somehow hitting?
>
> What happens when you try have Unicorn listening on both TCP and a Unix
> socket? Just put both "listen" directives in your config file
and point
> nginx to the Unix socket. Then try hitting the Unicorn TCP port
> directly, and then also the Unix socket via nginx.
>
> You can also try hitting the Unix socket directly by crafting your
> own HTTP request using socat from th shell:
>
> req=''GET / HTTP/1.1\r\nHost: example.com\r\n\r\n''
> printf "$req" | socat - UNIX:/root/pbr/unicorn.sock
>
> P.S.: I might not have a chance to respond again for the next day
> or so due to personal matters. Maybe somebody else on this
> list can help....
>
> --
> Eric Wong
Alex Barlow <alexbarlowis at me.com> wrote:> Fixed it! > > Rails was considering all requests to be local. As the request was > coming from the local ip (from nginx i assume)Awesome. Good to know, I''ve been setting these headers for so long that it''s become second nature and I''ve forgotten about side effects with Rails error messages :x> i put these headers in nginx > > proxy_set_header X-Real-IP $remote_addr;You probably don''t need X-Real-IP, actually, it''s quite nginx-specific and X-Forwarded-For covers you.> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; > > proxy_set_header Host $http_host; > > proxy_redirect false
Cool, thanxs for your support and great software. I get about 40 requests per second on a 1.7 gb instance with the db on the same host. Thats awesome! Well, i think so.. Alex On 17 Feb 2010, at 23:35, Eric Wong wrote:> Alex Barlow <alexbarlowis at me.com> wrote: >> Fixed it! >> >> Rails was considering all requests to be local. As the request was >> coming from the local ip (from nginx i assume) > > Awesome. > > Good to know, I''ve been setting these headers for so long that it''s > become second nature and I''ve forgotten about side effects with Rails > error messages :x > >> i put these headers in nginx >> >> proxy_set_header X-Real-IP $remote_addr; > > You probably don''t need X-Real-IP, actually, it''s quite nginx-specific > and X-Forwarded-For covers you. > >> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; >> >> proxy_set_header Host $http_host; >> >> proxy_redirect false