mongrel unicorn - Jun 2012 - Sinatra app fails to perform at scale

Hello,

We''re attempting to move our set of Rails and Sinatra apps from
Passenger/REE to Unicorn/1.9.3. We''ve started with one of the Sinatra
apps, which typically sees about 50 req/s. It happens to talk to both
Redis (Resque) and MongoDB GridFS.

The basic setup works fine when tested under ab, but we''re having
trouble getting the deploy into production. It performs fine for a
bit, then the nginx write queue fills up and begins returning 502s. A
colleague has posted a more detailed description of the issue and our
setup on ServerFault:
http://serverfault.com/questions/398972/need-to-increase-nginx-throughput-to-an-upstream-unix-socket-linux-kernel-tun

Additional information on the environment
curbed at app1:~$ uname -a
Linux app1 3.2.0-24-generic #39-Ubuntu SMP Mon May 21 16:52:17 UTC
2012 x86_64 x86_64 x86_64 GNU/Linux
curbed at app1:~$ ruby -v
ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-linux]
curbed at app1:~$ unicorn -v
unicorn v4.3.1
curbed at app1:~$ nginx -V
nginx version: nginx/1.2.1
built by gcc 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5)
TLS SNI support enabled
configure arguments: --prefix=/usr/local/nginx --with-http_ssl_module
--with-cc-opt=-Wno-error --with-http_gzip_static_module
--with-http_stub_status_module
--add-module=/home/curbed/src/nginx-modules/nginx-gridfs
--add-module=/home/curbed/src/nginx-modules/ngx_http_redis-0.3.6
--add-module=/home/curbed/src/nginx-modules/headers-more-nginx-module

Kernel tweaks:
net.core.rmem_default = 65536
net.core.wmem_default = 65536
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_mem = 16777216 16777216 16777216
net.ipv4.tcp_window_scaling = 1
net.ipv4.route.flush = 1
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_moderate_rcvbuf = 1
net.core.somaxconn = 8192
net.netfilter.nf_conntrack_max = 131072

Any suggestions on configuration, kernel tuning, etc. would be
welcomed (here or on SF). Please CC me if you answer through the list.
Thanks for your time.

Eliot

-- 
Eliot Shepard
Head of Tech, Curbed Network

Eliot Shepard <eshepard at slower.net> wrote:
> Hello,
> 
> We''re attempting to move our set of Rails and Sinatra apps from
> Passenger/REE to Unicorn/1.9.3. We''ve started with one of the
Sinatra
> apps, which typically sees about 50 req/s. It happens to talk to both
> Redis (Resque) and MongoDB GridFS.
Are you having problems at 50 req/s or higher than that with ab?
> The basic setup works fine when tested under ab, but we''re having
> trouble getting the deploy into production. It performs fine for a
> bit, then the nginx write queue fills up and begins returning 502s.
Is ab hitting nginx or unicorn?  You''ll probably get more
accurate/consistent results using ab with keepalive (-k) and hitting
nginx with ab.
> A colleague has posted a more detailed description of the issue and our
> setup on ServerFault:
>
http://serverfault.com/questions/398972/need-to-increase-nginx-throughput-to-an-upstream-unix-socket-linux-kernel-tun
Fwiw, I don''t (and don''t intend to) monitor external websites
for
questions, especially when they require signups.  Feel free to post
my response in part or full (or link back to this ML archive).

I''ll quote the relevant parts

<snip> nginx config looks fine.

The Unicorn config would be helpful here, too.

Can you try setting the listen:backlog directive to a higher number?
Something like:

  listen "/tmp/app.sock", :backlog => 8192

You''ll want to stay within your net.core.somaxconn = 8192 value or
raise
that sysctl to match unicorn.  However, if you have multiple machines
and want to load balance, I recommend trying a lower backlog.
>    The issue is, it just seems that past a certain amount of load, nginx
>    can''t get requests through the socket at a fast enough rate. It
doesn''t
>    matter how many app server processes I set up, it doesn''t even
matter
>    what the app is (tried it with a dummy app with just a single endpoint
>    that returned an empty page with status 404). The bottleneck seems to
>    be the socket, not the app.
>    I''m getting a flood of these messages in the nginx error log:
>    connect() to unix:/tmp/app.sock failed (11: Resource temporarily
>    unavailable) wh ile connecting to upstream
>    Many requests result in status code 502, and those that don''t
take a
>    long time to complete. The nginx write queue stat hovers around 1000.
Can you also verify unicorn is correctly closing connections that nginx
opens to it?  unicorn 4.3.1 should be fine in this regard...
> Additional information on the environment
> curbed at app1:~$ uname -a
> Linux app1 3.2.0-24-generic #39-Ubuntu SMP Mon May 21 16:52:17 UTC
> 2012 x86_64 x86_64 x86_64 GNU/Linux
> curbed at app1:~$ ruby -v
> ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-linux]
> curbed at app1:~$ unicorn -v
> unicorn v4.3.1
> curbed at app1:~$ nginx -V
> nginx version: nginx/1.2.1
I haven''t tested with nginx 1.2.x, yet, but I expect it to be fine.
I know 1.2.x also supports keepalive to backend connections, but
I don''t expect it to benefit on a fast LAN or local Unix socket...
> Any suggestions on configuration, kernel tuning, etc. would be
> welcomed (here or on SF). Please CC me if you answer through the list.
> Thanks for your time.
Thanks for reminding us to Cc: :)

I think Tom had a similar question a few years ago
http://mid.gmane.org/20090918064831.GA5285 at dcvr.yhbt.net

However, hitting issues with the default backlog (1024) with 50 req/s
wouldn''t be expected...