Mike Roeder
2007-Jun-29 15:46 UTC
[Mongrel] mongrel tuning with httperf - suspicious results
Hello all, I''m attempting to test/tune a mongrel cluster according to the tuning instructions on the mongrel site (using httperf). Anecdotally, the site itself ''feels'' snappy, but testing it with httperf reveals what appears to be terrible throughput. I''m kind of at a loss to describe the results, and was hoping someone could verify that I''m testing incorrectly (enormously likely), or truly have a problem that needs to be ferreted out. When testing a page that is served solely by apache, I get about 3000 req/s, while pages served by mongrel are getting around 1.6 req/s! Let me add that I am doing this test from a different machine, however, I am not able to actually remove network latency as a confounding factor (as per the tuning notes). If that renders this entire test moot, please let me know. The virtual hosts section of my apache config is included at the end of this post. The httperf command: httperf --server xx.xxx.xx.xx --port 80 --uri /something/like/this/page --num-conns 20 --hog -v httperf results: httperf --verbose --hog --client=0/1 --server=xx.xxx.xx.xx --port=80 --uri=/something/like/this/page --send-buffer=4096 --recv-buffer=16384 --num-conns=20 --num-calls=1 httperf: maximum number of open descriptors = 1024 reply-rate = 1.4 reply-rate = 1.6 Maximum connect burst length: 1 Total: connections 20 requests 20 replies 20 test-duration 13.002 s Connection rate: 1.5 conn/s (650.1 ms/conn, <=1 concurrent connections) Connection time [ms]: min 592.5 avg 650.1 max 714.0 median 659.5 stddev 39.7 Connection time [ms]: connect 1.9 Connection length [replies/conn]: 1.000 Request rate: 1.5 req/s (650.1 ms/req) Request size [B]: 96.0 Reply rate [replies/s]: min 1.4 avg 1.5 max 1.6 stddev 0.1 (2 samples) Reply time [ms]: response 565.8 transfer 82.4 Reply size [B]: header 291.0 content 96746.0 footer 0.0 (total 97037.0) Reply status: 1xx=0 2xx=20 3xx=0 4xx=0 5xx=0 CPU time [s]: user 3.44 system 9.43 (user 26.5% system 72.5% total 99.0%) Net I/O: 145.9 KB/s (1.2*10^6 bps) Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0 Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0 Apache Config, Virtual Hosts section: NameVirtualHost *:80 # Configure mongrel_cluster <Proxy balancer://the-cluster> BalancerMember http://127.0.0.1:8000 BalancerMember http://127.0.0.1:8001 BalancerMember http://127.0.0.1:8002 BalancerMember http://127.0.0.1:8003 BalancerMember http://127.0.0.1:8004 BalancerMember http://127.0.0.1:8005 BalancerMember http://127.0.0.1:8006 BalancerMember http://127.0.0.1:8007 #[srb] Uncomment when reverse proxying across other machines #BalancerMember http://xx.xxx.xx.xx loadfactor=2 </Proxy> #[srb] This configuration pulled from here # http://www.rss-spider.com/article.php?WID=305127 # and here: # http://www.howtoforge.com/load_balancing_apache_mod_proxy_balancer # and here: # http://reductivelabs.com/trac/puppet/wiki/UsingMongrel # <VirtualHost *:80> #[srb] next line added ServerAdmin info at blaho.com ServerName blah.com DocumentRoot /var/www/apps/blah/public #[srb] Turn off ProxyRequests if using ProxyPassReverse # http://httpd.apache.org/docs/2.2/mod/mod_proxy.html#proxypass ProxyRequests Off <Directory "/var/www/apps/blah/public"> Options FollowSymLinks AllowOverride None Order allow,deny Allow from all </Directory> RewriteEngine On #[srb] TODO this next line was mentioned as a way to propogate # an identifier to stickily preserve login information. At this # time it seems unnecessary #RewriteRule .* - [CO=BALANCEID:balancer.http1:.blah.com] # Uncomment for rewrite debugging RewriteLog logs/the_cluster_rewrite_log RewriteLogLevel 9 # Check for maintenance file and redirect all requests RewriteCond %{DOCUMENT_ROOT}/system/maintenance.html -f RewriteCond %{SCRIPT_FILENAME} !maintenance.html RewriteRule ^.*$ /system/maintenance.html [L] # Rewrite index to check for static RewriteRule ^/$ /index.html [QSA] # Rewrite to check for Rails cached page RewriteRule ^([^.]+)$ $1.html [QSA] # Redirect all non-static requests to cluster RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f RewriteRule ^/(.*)$ balancer://the-cluster%{REQUEST_URI} [P,QSA,L] # Configure reverse (must match the BalancerMember values) #[srb] Note: these next lines appear to be redundant with the # rewrite rules above #ProxyPass / balancer://the-cluster #ProxyPassReverse / balancer://the-cluster ProxyPreserveHost on # Deflate # http://rubyforge.org/pipermail/mongrel-users/2006-May/000247.html AddOutputFilterByType DEFLATE text/html text/plain text/xml BrowserMatch ^Mozilla/4 gzip-only-text/html BrowserMatch ^Mozilla/4\.0[678] no-gzip BrowserMatch \bMSIE !no-gzip !gzip-only-text/html # this not only blocks access to .svn directories, but makes it appear # as though they aren''t even there, not just that they are forbidden <DirectoryMatch "^/.*/\.svn/"> ErrorDocument 403 /404.html Order allow,deny Deny from all Satisfy All </DirectoryMatch> ErrorLog logs/the-cluster-error_log CustomLog logs/the_cluster_access_log combined </VirtualHost> -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20070629/2c221b91/attachment-0001.html
Alexey Verkhovsky
2007-Jun-29 16:03 UTC
[Mongrel] mongrel tuning with httperf - suspicious results
On 6/29/07, Mike Roeder <mcroeder at gmail.com> wrote:> Anecdotally, the site ''feels'' snappy, but testing it with httperf reveals what appears to > be terrible throughput.What''s surprising? 700 msec response time would feel snappy, it''s sub-second, after all. Of course, it''s too long if you are looking for> 10 req/sec throughput.As an aside, you should use bigger num-calls value, opening 20 simultaneous connections and issuing one call on each doesn''t usually tell you much. -- Alex
Mike Roeder
2007-Jun-29 17:37 UTC
[Mongrel] mongrel tuning with httperf - suspicious results
Thanks Alex - I must have missed the num-calls value on my initial reading of the man page for httperf. I suppose my real question is more along the lines of the fact that I suspected the Request Rate to be significantly higher than 1.7requests per second, especially in light of request rate for a page served only by apache (closer to 3000 requests per second). Per the tuning document, if my mongrel-served requests per second are HORRIBLY slower than the apache numbers, there''s tuning to be done. If anyone has a thought on where this tuning might occur, I''d love to hear it. Thanks for your help.> Anecdotally, the site ''feels'' snappy, but testing it with httperf revealswhat appears to> be terrible throughput.What''s surprising? 700 msec response time would feel snappy, it''s sub-second, after all. Of course, it''s too long if you are looking for> 10 req/sec throughput.As an aside, you should use bigger num-calls value, opening 20 simultaneous connections and issuing one call on each doesn''t usually tell you much. -- Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20070629/899bf884/attachment.html
Erik Hetzner
2007-Jun-29 17:56 UTC
[Mongrel] mongrel tuning with httperf - suspicious results
At Fri, 29 Jun 2007 12:37:03 -0500, "Mike Roeder" <mcroeder at gmail.com> wrote:> Thanks Alex - > > I must have missed the num-calls value on my initial reading of the > man page for httperf. I suppose my real question is more along the > lines of the fact that I suspected the Request Rate to be > significantly higher than 1.7requests per second, especially in > light of request rate for a page served only by apache (closer to > 3000 requests per second). Per the tuning document, if my > mongrel-served requests per second are HORRIBLY slower than the > apache numbers, there''s tuning to be done. If anyone has a thought > on where this tuning might occur, I''d love to hear it. Thanks for > your help.Apache serving static files is very fast (two orders of magnitude faster than any given rails+mongrel app). ** See what kind of numbers you can get out of a single mongrel instance. ** Also, in my experience, sending requests at too high a rate can cause a server?s response rate to go down; try 10/s on a single server & see if that works. Then bump it up. Search the mongrel-users archives for many pointers to using httperf with mongrel. And if you still are getting horrible numbers, try ruby-prof to profile you application. It is often the case that you can isolate a few terrible methods which are taking up most of your server?s time. best, Erik Hetzner -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://rubyforge.org/pipermail/mongrel-users/attachments/20070629/7688ad85/attachment.bin