Mongrel users - Apr 2007 - FastCGI performing better than Mongrel - what am I doing wrong?

I tried benchmarking the same site behind an NGINX proxy with both
fastcgi and mongrel, and for some reason mongrel is performing pretty
poorly in comparison.

Any idea what I might be doing wrong?

Here''s my benchmarks for 1 fcgi:

Server Software:        nginx/0.4.0
Server Hostname:        eship.com.br
Server Port:            80

Document Path:          /
Document Length:        95 bytes

Concurrency Level:      100
Time taken for tests:   10.437 seconds
Complete requests:      1000
Failed requests:        0
Broken pipe errors:     0
Non-2xx responses:      1000
Total transferred:      366000 bytes
HTML transferred:       95000 bytes
Requests per second:    95.81 [#/sec] (mean)
Time per request:       1043.70 [ms] (mean)
Time per request:       10.44 [ms] (mean, across all concurrent requests)
Transfer rate:          35.07 [Kbytes/sec] received

Connnection Times (ms)
              min  mean[+/-sd] median   max
Connect:      182   435  294.5    430  3428
Processing:   371   569  296.5    505  2674
Waiting:      189   569  296.5    505  2674
Total:        371  1004  418.8    938  3963




And for 2 mongrels:

Server Software:        nginx/0.4.0
Server Hostname:        eship.com.br
Server Port:            80

Document Path:          /
Document Length:        95 bytes

Concurrency Level:      100
Time taken for tests:   13.041 seconds
Complete requests:      1000
Failed requests:        0
Broken pipe errors:     0
Non-2xx responses:      1000
Total transferred:      417000 bytes
HTML transferred:       95000 bytes
Requests per second:    76.68 [#/sec] (mean)
Time per request:       1304.10 [ms] (mean)
Time per request:       13.04 [ms] (mean, across all concurrent requests)
Transfer rate:          31.98 [Kbytes/sec] received

Connnection Times (ms)
              min  mean[+/-sd] median   max
Connect:      175   234  292.9    187  3099
Processing:   204   897  806.4    611  5619
Waiting:      187   897  806.5    611  5619
Total:        365  1132  840.6    842  5804

Looking even at your standard deviations, I don''t see much of a  
difference between these. What''s your SD on the req/sec?

Regards, Jason

On Apr 3, 2007, at 1:33 PM, Benjamin Jackson wrote:
> I tried benchmarking the same site behind an NGINX proxy with both
> fastcgi and mongrel, and for some reason mongrel is performing pretty
> poorly in comparison.
>
> Any idea what I might be doing wrong?
>
> Here''s my benchmarks for 1 fcgi:
>
> Server Software:        nginx/0.4.0
> Server Hostname:        eship.com.br
> Server Port:            80
>
> Document Path:          /
> Document Length:        95 bytes
>
> Concurrency Level:      100
> Time taken for tests:   10.437 seconds
> Complete requests:      1000
> Failed requests:        0
> Broken pipe errors:     0
> Non-2xx responses:      1000
> Total transferred:      366000 bytes
> HTML transferred:       95000 bytes
> Requests per second:    95.81 [#/sec] (mean)
> Time per request:       1043.70 [ms] (mean)
> Time per request:       10.44 [ms] (mean, across all concurrent  
> requests)
> Transfer rate:          35.07 [Kbytes/sec] received
>
> Connnection Times (ms)
>               min  mean[+/-sd] median   max
> Connect:      182   435  294.5    430  3428
> Processing:   371   569  296.5    505  2674
> Waiting:      189   569  296.5    505  2674
> Total:        371  1004  418.8    938  3963
>
>
>
>
> And for 2 mongrels:
>
> Server Software:        nginx/0.4.0
> Server Hostname:        eship.com.br
> Server Port:            80
>
> Document Path:          /
> Document Length:        95 bytes
>
> Concurrency Level:      100
> Time taken for tests:   13.041 seconds
> Complete requests:      1000
> Failed requests:        0
> Broken pipe errors:     0
> Non-2xx responses:      1000
> Total transferred:      417000 bytes
> HTML transferred:       95000 bytes
> Requests per second:    76.68 [#/sec] (mean)
> Time per request:       1304.10 [ms] (mean)
> Time per request:       13.04 [ms] (mean, across all concurrent  
> requests)
> Transfer rate:          31.98 [Kbytes/sec] received
>
> Connnection Times (ms)
>               min  mean[+/-sd] median   max
> Connect:      175   234  292.9    187  3099
> Processing:   204   897  806.4    611  5619
> Waiting:      187   897  806.5    611  5619
> Total:        365  1132  840.6    842  5804
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 223 bytes
Desc: not available
Url :
http://rubyforge.org/pipermail/mongrel-users/attachments/20070403/885eef57/attachment.bin

Are you running these tests cold? You probably want to throw out the  
first bunch of requests (say, 1000) to better simulate real-world  
running conditions.

Also, what''s up with the non-2xx responses? Are you benchmarking an  
error page or something?

-Nate

On Apr 3, 2007, at 3:33 PM, Benjamin Jackson wrote:
> I tried benchmarking the same site behind an NGINX proxy with both
> fastcgi and mongrel, and for some reason mongrel is performing pretty
> poorly in comparison.
>
> Any idea what I might be doing wrong?
>
> Here''s my benchmarks for 1 fcgi:
>
> Server Software:        nginx/0.4.0
> Server Hostname:        eship.com.br
> Server Port:            80
>
> Document Path:          /
> Document Length:        95 bytes
>
> Concurrency Level:      100
> Time taken for tests:   10.437 seconds
> Complete requests:      1000
> Failed requests:        0
> Broken pipe errors:     0
> Non-2xx responses:      1000
> Total transferred:      366000 bytes
> HTML transferred:       95000 bytes
> Requests per second:    95.81 [#/sec] (mean)
> Time per request:       1043.70 [ms] (mean)
> Time per request:       10.44 [ms] (mean, across all concurrent  
> requests)
> Transfer rate:          35.07 [Kbytes/sec] received
>
> Connnection Times (ms)
>               min  mean[+/-sd] median   max
> Connect:      182   435  294.5    430  3428
> Processing:   371   569  296.5    505  2674
> Waiting:      189   569  296.5    505  2674
> Total:        371  1004  418.8    938  3963
>
>
>
>
> And for 2 mongrels:
>
> Server Software:        nginx/0.4.0
> Server Hostname:        eship.com.br
> Server Port:            80
>
> Document Path:          /
> Document Length:        95 bytes
>
> Concurrency Level:      100
> Time taken for tests:   13.041 seconds
> Complete requests:      1000
> Failed requests:        0
> Broken pipe errors:     0
> Non-2xx responses:      1000
> Total transferred:      417000 bytes
> HTML transferred:       95000 bytes
> Requests per second:    76.68 [#/sec] (mean)
> Time per request:       1304.10 [ms] (mean)
> Time per request:       13.04 [ms] (mean, across all concurrent  
> requests)
> Transfer rate:          31.98 [Kbytes/sec] received
>
> Connnection Times (ms)
>               min  mean[+/-sd] median   max
> Connect:      175   234  292.9    187  3099
> Processing:   204   897  806.4    611  5619
> Waiting:      187   897  806.5    611  5619
> Total:        365  1132  840.6    842  5804
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>

You might try a more current version of Nginx and see if the proxying
performance has improved compared to the FastCGI support. You''re using
a fairly old version.

http://nginx.net/CHANGES


On 4/3/07, Nathan Vack <njvack at wisc.edu> wrote:
> Are you running these tests cold? You probably want to throw out the
> first bunch of requests (say, 1000) to better simulate real-world
> running conditions.
>
> Also, what''s up with the non-2xx responses? Are you benchmarking
an
> error page or something?
>
> -Nate
>
> On Apr 3, 2007, at 3:33 PM, Benjamin Jackson wrote:
>
> > I tried benchmarking the same site behind an NGINX proxy with both
> > fastcgi and mongrel, and for some reason mongrel is performing pretty
> > poorly in comparison.
> >
> > Any idea what I might be doing wrong?
> >
> > Here''s my benchmarks for 1 fcgi:
> >
> > Server Software:        nginx/0.4.0
> > Server Hostname:        eship.com.br
> > Server Port:            80
> >
> > Document Path:          /
> > Document Length:        95 bytes
> >
> > Concurrency Level:      100
> > Time taken for tests:   10.437 seconds
> > Complete requests:      1000
> > Failed requests:        0
> > Broken pipe errors:     0
> > Non-2xx responses:      1000
> > Total transferred:      366000 bytes
> > HTML transferred:       95000 bytes
> > Requests per second:    95.81 [#/sec] (mean)
> > Time per request:       1043.70 [ms] (mean)
> > Time per request:       10.44 [ms] (mean, across all concurrent
> > requests)
> > Transfer rate:          35.07 [Kbytes/sec] received
> >
> > Connnection Times (ms)
> >               min  mean[+/-sd] median   max
> > Connect:      182   435  294.5    430  3428
> > Processing:   371   569  296.5    505  2674
> > Waiting:      189   569  296.5    505  2674
> > Total:        371  1004  418.8    938  3963
> >
> >
> >
> >
> > And for 2 mongrels:
> >
> > Server Software:        nginx/0.4.0
> > Server Hostname:        eship.com.br
> > Server Port:            80
> >
> > Document Path:          /
> > Document Length:        95 bytes
> >
> > Concurrency Level:      100
> > Time taken for tests:   13.041 seconds
> > Complete requests:      1000
> > Failed requests:        0
> > Broken pipe errors:     0
> > Non-2xx responses:      1000
> > Total transferred:      417000 bytes
> > HTML transferred:       95000 bytes
> > Requests per second:    76.68 [#/sec] (mean)
> > Time per request:       1304.10 [ms] (mean)
> > Time per request:       13.04 [ms] (mean, across all concurrent
> > requests)
> > Transfer rate:          31.98 [Kbytes/sec] received
> >
> > Connnection Times (ms)
> >               min  mean[+/-sd] median   max
> > Connect:      175   234  292.9    187  3099
> > Processing:   204   897  806.4    611  5619
> > Waiting:      187   897  806.5    611  5619
> > Total:        365  1132  840.6    842  5804
> > _______________________________________________
> > Mongrel-users mailing list
> > Mongrel-users at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/mongrel-users
> >
>
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>

-- 
Cheers,

Kevin Williams
http://www.almostserio.us/

"Any sufficiently advanced technology is indistinguishable from
Magic." - Arthur C. Clarke

> Looking even at your standard deviations, I don''t see much of a
> difference between these. What''s your SD on the req/sec?
Thanks for the heads-up Jason.

ab doesn''t have a SD on the req/sec AFAIK, tried doing (I think)
equivalent benchmarks with httperf, this time from the server, and I
got the following results. It seems like FastCGI is marginally faster
than Mongrel for my site when both are run with 2 dispatchers.

Does this sound like an accurate test, or did I miss something
essential in the benchmarking process? Thanks,

Ben

---------------------------------------------------

FastCGI (1 proc):

$ httperf --server=eship.com.br --rate=100 --num-conns=1000
httperf --client=0/1 --server=eship.com.br --port=80 --uri=/
--rate=100 --send-buffer=4096 --recv-buffer=16384 --num-conns=1000
--num-calls=1
Maximum connect burst length: 7

Total: connections 1000 requests 1000 replies 1000 test-duration 16.141 s

Connection rate: 62.0 conn/s (16.1 ms/conn, <=257 concurrent connections)
Connection time [ms]: min 10.2 avg 1741.3 max 9040.8 median 1614.5 stddev 1737.0
Connection time [ms]: connect 0.0
Connection length [replies/conn]: 1.000

Request rate: 62.0 req/s (16.1 ms/req)
Request size [B]: 63.0

Reply rate [replies/s]: min 48.2 avg 65.7 max 75.0 stddev 15.1 (3 samples)
Reply time [ms]: response 1741.3 transfer 0.0
Reply size [B]: header 304.0 content 95.0 footer 2.0 (total 401.0)
Reply status: 1xx=0 2xx=0 3xx=1000 4xx=0 5xx=0

CPU time [s]: user 1.31 system 13.72 (user 8.1% system 85.0% total 93.1%)
Net I/O: 28.0 KB/s (0.2*10^6 bps)

Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

---------------------------------------------------

FastCGI (2 procs):

$ httperf --server=eship.com.br --rate=100 --num-conns=1000
httperf --client=0/1 --server=eship.com.br --port=80 --uri=/
--rate=100 --send-buffer=4096 --recv-buffer=16384 --num-conns=1000
--num-calls=1
Maximum connect burst length: 8

Total: connections 1000 requests 1000 replies 1000 test-duration 11.993 s

Connection rate: 83.4 conn/s (12.0 ms/conn, <=169 concurrent connections)
Connection time [ms]: min 10.7 avg 837.5 max 2047.6 median 831.5 stddev 662.5
Connection time [ms]: connect 0.3
Connection length [replies/conn]: 1.000

Request rate: 83.4 req/s (12.0 ms/req)
Request size [B]: 63.0

Reply rate [replies/s]: min 82.4 avg 83.8 max 85.2 stddev 2.0 (2 samples)
Reply time [ms]: response 837.2 transfer 0.0
Reply size [B]: header 304.0 content 95.0 footer 2.0 (total 401.0)
Reply status: 1xx=0 2xx=0 3xx=1000 4xx=0 5xx=0

CPU time [s]: user 1.23 system 8.75 (user 10.3% system 73.0% total 83.2%)
Net I/O: 37.6 KB/s (0.3*10^6 bps)

Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

---------------------------------------------------

Mongrel:


$ httperf --server=eship.com.br --rate=100 --num-conns=1000
httperf --client=0/1 --server=eship.com.br --port=80 --uri=/
--rate=100 --send-buffer=4096 --recv-buffer=16384 --num-conns=1000
--num-calls=1
Maximum connect burst length: 4

Total: connections 1000 requests 1000 replies 1000 test-duration 13.000 s

Connection rate: 76.9 conn/s (13.0 ms/conn, <=212 concurrent connections)
Connection time [ms]: min 19.5 avg 1117.1 max 4115.2 median 1036.5 stddev 936.0
Connection time [ms]: connect 0.0
Connection length [replies/conn]: 1.000

Request rate: 76.9 req/s (13.0 ms/req)
Request size [B]: 63.0

Reply rate [replies/s]: min 74.4 avg 78.9 max 83.4 stddev 6.4 (2 samples)
Reply time [ms]: response 1117.1 transfer 0.0
Reply size [B]: header 304.0 content 95.0 footer 2.0 (total 401.0)
Reply status: 1xx=0 2xx=0 3xx=1000 4xx=0 5xx=0

CPU time [s]: user 1.02 system 11.22 (user 7.8% system 86.3% total 94.2%)
Net I/O: 34.7 KB/s (0.3*10^6 bps)

Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

> Are you running these tests cold? You probably want to throw out the
> first bunch of requests (say, 1000) to better simulate real-world
> running conditions.
You mean disregard them in the stats? Would you mind explaining this a
little more?
> Also, what''s up with the non-2xx responses? Are you benchmarking
an
> error page or something?
No, maybe because I''m hitting eship.com.br and not www.eship.com.br
it''s throwing a 301? Will have to double-check my setup.

> You might try a more current version of Nginx and see if the proxying
> performance has improved compared to the FastCGI support. You''re
using
> a fairly old version.
>
> http://nginx.net/CHANGES
Will check this out and report the results. Thanks :)

On Apr 6, 2007, at 12:00 AM, Benjamin Jackson wrote:
>> Are you running these tests cold? You probably want to throw out the
>> first bunch of requests (say, 1000) to better simulate real-world
>> running conditions.
>
> You mean disregard them in the stats? Would you mind explaining this a
> little more?
Right after you start a lot of processes, there''s a bunch of  
optimization that takes place -- frequently used files are read into  
cache, database connections are set up, necessary parts of the  
operating system are paged into memory, perhaps your VM is setting  
itself up, maybe Rails is generating caches...

All of this stuff takes time, and can make requests really slow.  
Fortunately, you almost never run Rails immediately after startup  
(unless you''re using rmagick or something ;-) so throwing out the  
data with that startup hit will give you a better idea of how your  
site will perform in the real world.

In your case, it looked to me like Mongrel''s median times were faster  
-- your totals were longer because of a very large max time.

-Nate

> Right after you start a lot of processes, there''s a bunch of
> optimization that takes place -- frequently used files are read into
> cache, database connections are set up, necessary parts of the
> operating system are paged into memory, perhaps your VM is setting
> itself up, maybe Rails is generating caches...
>
> All of this stuff takes time, and can make requests really slow.
> Fortunately, you almost never run Rails immediately after startup
> (unless you''re using rmagick or something ;-) so throwing out the
> data with that startup hit will give you a better idea of how your
> site will perform in the real world.
>
> In your case, it looked to me like Mongrel''s median times were
faster
> -- your totals were longer because of a very large max time.
I understand the startup overhead, but was under the impression that
it only affected the first request, not the first thousand. I assume
you''re under the impression that I''m running these immediately
after
starting up the server without hitting the page once. While I''m pretty
sure I hit all the pages before testing, I''ll run some more tests,
this time making sure that the page requested has been hit once in the
browser beforehand.

Looking back at the results, yeah, the median connection time for two
mongrels is indeed faster than for one FCGI, but two FCGIs still comes
in considerably faster than two mongrels. Is there any RAM advantage
gained by using two mongrels instead of two FCGI procs, or any other
compelling reason for me to use mongrel instead if my connection times
remain significantly faster for FCGI when the server is warmed up and
ready?

Ben