Mongrel users - Jun 2007 - mongrel tuning with httperf - suspicious results

Hello all,

I''m attempting to test/tune a mongrel cluster according to the tuning
instructions on the mongrel site (using httperf). Anecdotally, the site
itself ''feels'' snappy, but testing it with httperf reveals
what appears to
be terrible throughput. I''m kind of at a loss to describe the results,
and
was hoping someone could verify that I''m testing incorrectly
(enormously
likely), or truly have a problem that needs to be ferreted out. When testing
a page that is served solely by apache, I get about 3000 req/s, while pages
served by mongrel are getting around 1.6 req/s! Let me add that I am doing
this test from a different machine, however, I am not able to actually
remove network latency as a confounding factor (as per the tuning notes). If
that renders this entire test moot, please let me know.

The virtual hosts section of my apache config is included at the end of this
post.

The httperf command:
httperf --server xx.xxx.xx.xx --port 80 --uri /something/like/this/page
--num-conns 20 --hog -v

httperf results:

httperf --verbose --hog --client=0/1 --server=xx.xxx.xx.xx --port=80
--uri=/something/like/this/page --send-buffer=4096 --recv-buffer=16384
--num-conns=20 --num-calls=1
httperf: maximum number of open descriptors = 1024
reply-rate = 1.4
reply-rate = 1.6
Maximum connect burst length: 1

Total: connections 20 requests 20 replies 20 test-duration 13.002 s

Connection rate: 1.5 conn/s (650.1 ms/conn, <=1 concurrent connections)
Connection time [ms]: min 592.5 avg 650.1 max 714.0 median 659.5 stddev 39.7
Connection time [ms]: connect 1.9
Connection length [replies/conn]: 1.000

Request rate: 1.5 req/s (650.1 ms/req)
Request size [B]: 96.0

Reply rate [replies/s]: min 1.4 avg 1.5 max 1.6 stddev 0.1 (2 samples)
Reply time [ms]: response 565.8 transfer 82.4
Reply size [B]: header 291.0 content 96746.0 footer 0.0 (total 97037.0)
Reply status: 1xx=0 2xx=20 3xx=0 4xx=0 5xx=0

CPU time [s]: user 3.44 system 9.43 (user 26.5% system 72.5% total 99.0%)
Net I/O: 145.9 KB/s (1.2*10^6 bps)

Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0



Apache Config, Virtual Hosts section:

NameVirtualHost *:80


# Configure mongrel_cluster
<Proxy balancer://the-cluster>
  BalancerMember http://127.0.0.1:8000
  BalancerMember http://127.0.0.1:8001
  BalancerMember http://127.0.0.1:8002
  BalancerMember http://127.0.0.1:8003
  BalancerMember http://127.0.0.1:8004
  BalancerMember http://127.0.0.1:8005
  BalancerMember http://127.0.0.1:8006
  BalancerMember http://127.0.0.1:8007
  #[srb] Uncomment when reverse proxying across other machines
  #BalancerMember http://xx.xxx.xx.xx loadfactor=2
</Proxy>

#[srb] This configuration pulled from here
# http://www.rss-spider.com/article.php?WID=305127
# and here:
# http://www.howtoforge.com/load_balancing_apache_mod_proxy_balancer
# and here:
# http://reductivelabs.com/trac/puppet/wiki/UsingMongrel
#
<VirtualHost *:80>
  #[srb] next line added
  ServerAdmin info at blaho.com
  ServerName blah.com
  DocumentRoot /var/www/apps/blah/public

  #[srb] Turn off ProxyRequests if using ProxyPassReverse
  # http://httpd.apache.org/docs/2.2/mod/mod_proxy.html#proxypass
  ProxyRequests Off

  <Directory "/var/www/apps/blah/public">
    Options FollowSymLinks
    AllowOverride None
    Order allow,deny
    Allow from all
  </Directory>

  RewriteEngine On
  #[srb] TODO this next line was mentioned as a way to propogate
  # an identifier to stickily preserve login information.  At this
  # time it seems unnecessary
  #RewriteRule .* - [CO=BALANCEID:balancer.http1:.blah.com]

  # Uncomment for rewrite debugging
  RewriteLog logs/the_cluster_rewrite_log
  RewriteLogLevel 9

  # Check for maintenance file and redirect all requests
  RewriteCond %{DOCUMENT_ROOT}/system/maintenance.html -f
  RewriteCond %{SCRIPT_FILENAME} !maintenance.html
  RewriteRule ^.*$ /system/maintenance.html [L]

  # Rewrite index to check for static
  RewriteRule ^/$ /index.html [QSA]

  # Rewrite to check for Rails cached page
  RewriteRule ^([^.]+)$ $1.html [QSA]

  # Redirect all non-static requests to cluster
  RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
  RewriteRule ^/(.*)$ balancer://the-cluster%{REQUEST_URI} [P,QSA,L]

  # Configure reverse (must match the BalancerMember values)
  #[srb] Note: these next lines appear to be redundant with the
  # rewrite rules above
  #ProxyPass / balancer://the-cluster
  #ProxyPassReverse / balancer://the-cluster
  ProxyPreserveHost on


  # Deflate
  # http://rubyforge.org/pipermail/mongrel-users/2006-May/000247.html
  AddOutputFilterByType DEFLATE text/html text/plain text/xml
  BrowserMatch ^Mozilla/4 gzip-only-text/html
  BrowserMatch ^Mozilla/4\.0[678] no-gzip
  BrowserMatch \bMSIE !no-gzip !gzip-only-text/html

  # this not only blocks access to .svn directories, but makes it appear
  # as though they aren''t even there, not just that they are forbidden
  <DirectoryMatch "^/.*/\.svn/">
    ErrorDocument 403 /404.html
    Order allow,deny
    Deny from all
    Satisfy All
  </DirectoryMatch>


  ErrorLog logs/the-cluster-error_log
  CustomLog logs/the_cluster_access_log combined
</VirtualHost>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20070629/2c221b91/attachment-0001.html

On 6/29/07, Mike Roeder <mcroeder at gmail.com>
wrote:
> Anecdotally, the site ''feels'' snappy, but testing it with
httperf reveals what appears to
> be terrible throughput.
What''s surprising? 700 msec response time would feel snappy,
it''s
sub-second, after all. Of course, it''s too long if you are looking
for
> 10 req/sec throughput.
As an aside, you should use bigger num-calls value, opening 20
simultaneous connections and issuing one call on each doesn''t usually
tell you much.

--
Alex

Thanks Alex -

I must have missed the num-calls value on my initial reading of the man page
for httperf. I suppose my real question is more along the lines of the fact
that I suspected the Request Rate to be significantly higher than
1.7requests per second, especially in light of request rate for a page
served
only by apache (closer to 3000 requests per second). Per the tuning
document, if my mongrel-served requests per second are HORRIBLY slower than
the apache numbers, there''s tuning to be done. If anyone has a thought
on
where this tuning might occur, I''d love to hear it. Thanks for your
help.
> Anecdotally, the site ''feels'' snappy, but testing it with
httperf reveals
what appears to
> be terrible throughput.
What''s surprising? 700 msec response time would feel snappy,
it''s
sub-second, after all. Of course, it''s too long if you are looking
for
> 10 req/sec throughput.
As an aside, you should use bigger num-calls value, opening 20
simultaneous connections and issuing one call on each doesn''t usually
tell you much.

--
Alex
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20070629/899bf884/attachment.html

At Fri, 29 Jun 2007 12:37:03 -0500,
"Mike Roeder" <mcroeder at gmail.com>
wrote:
> Thanks Alex -
>
> I must have missed the num-calls value on my initial reading of the
> man page for httperf. I suppose my real question is more along the
> lines of the fact that I suspected the Request Rate to be
> significantly higher than 1.7requests per second, especially in
> light of request rate for a page served only by apache (closer to
> 3000 requests per second). Per the tuning document, if my
> mongrel-served requests per second are HORRIBLY slower than the
> apache numbers, there''s tuning to be done. If anyone has a thought
> on where this tuning might occur, I''d love to hear it. Thanks for
> your help.
Apache serving static files is very fast (two orders of magnitude
faster than any given rails+mongrel app).

** See what kind of numbers you can get out of a single mongrel
   instance. **

Also, in my experience, sending requests at too high a rate can cause
a server?s response rate to go down; try 10/s on a single server & see
if that works. Then bump it up.

Search the mongrel-users archives for many pointers to using httperf
with mongrel.

And if you still are getting horrible numbers, try ruby-prof to
profile you application. It is often the case that you can isolate a
few terrible methods which are taking up most of your server?s time.

best,
Erik Hetzner
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url :
http://rubyforge.org/pipermail/mongrel-users/attachments/20070629/7688ad85/attachment.bin