Mongrel users - Dec 2006 - production-izing a popular site

Lets say you have a site that is serving, oh, around 100k unique
visitors a day (plain ole'' browser requests) - plus probably ~ 40k
uniques to feeds.  Assume this site is 90% read like most of the web,
so the traffic looks like your typical news or portal site.  There are
two web boxes behind a hardware load balancer, each doing apache 2.2.3
-> mongrel_proxy_balancer -> mongrel clusters.  One db machine, but
that shouldn''t be stressed at all due to heavy caching of the content.

Where to being with performance testing something like this?  How many
mongrels per web box to start with?  Any comparable sites in
production to compare it to (Ezra?)

thanks,
- Rob

On Dec 28, 2006, at 5:45 PM, Rob Sanheim wrote:
> Lets say you have a site that is serving, oh, around 100k unique
> visitors a day (plain ole'' browser requests) - plus probably ~ 40k
> uniques to feeds.  Assume this site is 90% read like most of the web,
> so the traffic looks like your typical news or portal site.  There are
> two web boxes behind a hardware load balancer, each doing apache 2.2.3
> -> mongrel_proxy_balancer -> mongrel clusters.  One db machine, but
> that shouldn''t be stressed at all due to heavy caching of the
content.
>
> Where to being with performance testing something like this?  How many
> mongrels per web box to start with?  Any comparable sites in
> production to compare it to (Ezra?)
>
> thanks,
> - Rob

Hey Rob-

	I am hosting a site that does about 110k unique visitors per day and  
about 250k page views/day. I would say it is a typical rails app with  
about 85% read and then some write and user contributed uploads and  
such. This app is pretty well optimized and doesn''t have any requests  
except for file uploads that take longer then about 1.5 seconds to  
complete with most requests going out at about 35-45req/sec. Pretty  
typical rails app.

	It is running on two of our slices which are Xen VM''s where there is  
a hardware load balancer up front, then nginx proxying to mongrel  
cluster on each node. It runs 3 mongrels on each VM for a total of 6.  
The database is on another super beefy machine but is not dedicated  
entirely to this app and is shared by others. I think this will be a  
good place for you to start load testing, 3 - 5 mongrels on each of  
your app servers. First get a baseline for your hardware by using ab  
or httperf on a fresh rails app that just does render :text =>  
''hello'' to get the best case scenario for a rails request on
your
hardware. This is the fastest you can expect rails to serve a   
dynamic request on your boxes. It will probably be around 100-140req/ 
sec with sessions turned on. Then do a few more of this same  
benchmark against one mongrel and a few of your rails actions that  
represent short and long requests for your app. Write these results  
down or save them as you go. This is important because I discovered  
that 3 mongrels performs *better* then 5 mongrels on our  
architecture. This will vary widely though over different os and  
hardware configurations.

	Now its time to add more mongrels until you hit a point of  
diminishing returns or you are confident in your ability to handle  
the traffic you expect plus some. ounds like you have two nice boxes  
for your app servers so start with 3 mongrels on one of the servers.  
Get this benchmark for the render :text action as well as the other  
actions you benched in your real app. At this point you should get  
more req/sec when you measure with concurrent users at 20-50. Then  
add one mongrel at a time and measure until adding one more mongrel  
does not buy you any more performance. This is your sweet spot where  
you are maxing out your proxy_balancers efficiency.

	Now that you hit the sweet spot for one app server setup the second  
one the same way and do some more benches at high concurrency. If you  
have good hardware load balancers you should get a good improvement  
with the two app servers over just one of them. In reality software  
load balancers don''t multiply the perf of a single mongrel by the  
number of mongrels. Its more like adding another mongrel may get you  
20-50% more until you hit the sweet spot and can''t get anymore. But  
the good hardware load balancers are more efficient and get closer to  
50-80% improvement when adding more nodes. Sometimes it''s nice to go  
straight from the hardware load balancers right to mongrel and use an  
asset_host for static content. I don''t think you will need this for  
the app you are talking about.

	I suspect you will end up with 4-6 mongrels per app server and this  
will leave you some overhead for what you need to serve. Another  
point of reference is the http://yakimaherald.com gets about 75-90k  
page views/day on 3 mongrels behind apache22 on an xserve.

	

-- Ezra Zygmuntowicz 
-- Lead Rails Evangelist
-- ez at engineyard.com
-- Engine Yard, Serious Rails Hosting
-- (866) 518-YARD (9273)

Great email Ezra, if other people would like a link to what he just said you
can check out what Zed has up on the mongrel site:
http://mongrel.rubyforge.org/docs/how_many_mongrels.html.  I have done this
myself and found that 4 mongrels behind nginx was my sweet spot.  Some notes
however, if you decide to test with sessions turned on, make sure that you
clear out the sessions inbetween runs, also make sure you are benchmarking
the same setup you have in production, MySql store etc.  Lastly, make sure
you test over and over again with the same setup at least 5 times per
cluster of mongrels to ensure that you get consistent results.  You want to
hone in on consistency rather than 1 spike in performance.

.: Michael :.

On 12/28/06, Ezra Zygmuntowicz <ezmobius at gmail.com>
wrote:
>
>
> On Dec 28, 2006, at 5:45 PM, Rob Sanheim wrote:
>
> > Lets say you have a site that is serving, oh, around 100k unique
> > visitors a day (plain ole'' browser requests) - plus probably
~ 40k
> > uniques to feeds.  Assume this site is 90% read like most of the web,
> > so the traffic looks like your typical news or portal site.  There are
> > two web boxes behind a hardware load balancer, each doing apache 2.2.3
> > -> mongrel_proxy_balancer -> mongrel clusters.  One db machine,
but
> > that shouldn''t be stressed at all due to heavy caching of the
content.
> >
> > Where to being with performance testing something like this?  How many
> > mongrels per web box to start with?  Any comparable sites in
> > production to compare it to (Ezra?)
> >
> > thanks,
> > - Rob
>
>
> Hey Rob-
>
>         I am hosting a site that does about 110k unique visitors per day
> and
> about 250k page views/day. I would say it is a typical rails app with
> about 85% read and then some write and user contributed uploads and
> such. This app is pretty well optimized and doesn''t have any
requests
> except for file uploads that take longer then about 1.5 seconds to
> complete with most requests going out at about 35-45req/sec. Pretty
> typical rails app.
>
>         It is running on two of our slices which are Xen VM''s
where there
> is
> a hardware load balancer up front, then nginx proxying to mongrel
> cluster on each node. It runs 3 mongrels on each VM for a total of 6.
> The database is on another super beefy machine but is not dedicated
> entirely to this app and is shared by others. I think this will be a
> good place for you to start load testing, 3 - 5 mongrels on each of
> your app servers. First get a baseline for your hardware by using ab
> or httperf on a fresh rails app that just does render :text =>
> ''hello'' to get the best case scenario for a rails request
on your
> hardware. This is the fastest you can expect rails to serve a
> dynamic request on your boxes. It will probably be around 100-140req/
> sec with sessions turned on. Then do a few more of this same
> benchmark against one mongrel and a few of your rails actions that
> represent short and long requests for your app. Write these results
> down or save them as you go. This is important because I discovered
> that 3 mongrels performs *better* then 5 mongrels on our
> architecture. This will vary widely though over different os and
> hardware configurations.
>
>         Now its time to add more mongrels until you hit a point of
> diminishing returns or you are confident in your ability to handle
> the traffic you expect plus some. ounds like you have two nice boxes
> for your app servers so start with 3 mongrels on one of the servers.
> Get this benchmark for the render :text action as well as the other
> actions you benched in your real app. At this point you should get
> more req/sec when you measure with concurrent users at 20-50. Then
> add one mongrel at a time and measure until adding one more mongrel
> does not buy you any more performance. This is your sweet spot where
> you are maxing out your proxy_balancers efficiency.
>
>         Now that you hit the sweet spot for one app server setup the
> second
> one the same way and do some more benches at high concurrency. If you
> have good hardware load balancers you should get a good improvement
> with the two app servers over just one of them. In reality software
> load balancers don''t multiply the perf of a single mongrel by the
> number of mongrels. Its more like adding another mongrel may get you
> 20-50% more until you hit the sweet spot and can''t get anymore.
But
> the good hardware load balancers are more efficient and get closer to
> 50-80% improvement when adding more nodes. Sometimes it''s nice to
go
> straight from the hardware load balancers right to mongrel and use an
> asset_host for static content. I don''t think you will need this
for
> the app you are talking about.
>
>         I suspect you will end up with 4-6 mongrels per app server and
> this
> will leave you some overhead for what you need to serve. Another
> point of reference is the http://yakimaherald.com gets about 75-90k
> page views/day on 3 mongrels behind apache22 on an xserve.
>
>
>
> -- Ezra Zygmuntowicz
> -- Lead Rails Evangelist
> -- ez at engineyard.com
> -- Engine Yard, Serious Rails Hosting
> -- (866) 518-YARD (9273)
>
>
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20061228/8716973e/attachment-0001.html

On 12/28/06, Ezra Zygmuntowicz <ezmobius at gmail.com>
wrote:
>
> On Dec 28, 2006, at 5:45 PM, Rob Sanheim wrote:
>
> > Lets say you have a site that is serving, oh, around 100k unique
> > visitors a day (plain ole'' browser requests) - plus probably
~ 40k
> > uniques to feeds.  Assume this site is 90% read like most of the web,
> > so the traffic looks like your typical news or portal site.  There are
> > two web boxes behind a hardware load balancer, each doing apache 2.2.3
> > -> mongrel_proxy_balancer -> mongrel clusters.  One db machine,
but
> > that shouldn''t be stressed at all due to heavy caching of the
content.
> >
> > Where to being with performance testing something like this?  How many
> > mongrels per web box to start with?  Any comparable sites in
> > production to compare it to (Ezra?)
> >
> > thanks,
> > - Rob
>
>
> Hey Rob-
>
>         I am hosting a site that does about 110k unique visitors per day
and
> about 250k page views/day. I would say it is a typical rails app with
> about 85% read and then some write and user contributed uploads and
> such. This app is pretty well optimized and doesn''t have any
requests
> except for file uploads that take longer then about 1.5 seconds to
> complete with most requests going out at about 35-45req/sec. Pretty
> typical rails app.
>
>         It is running on two of our slices which are Xen VM''s
where there is
> a hardware load balancer up front, then nginx proxying to mongrel
> cluster on each node. It runs 3 mongrels on each VM for a total of 6.
> The database is on another super beefy machine but is not dedicated
> entirely to this app and is shared by others. I think this will be a
> good place for you to start load testing, 3 - 5 mongrels on each of
> your app servers. First get a baseline for your hardware by using ab
> or httperf on a fresh rails app that just does render :text =>
> ''hello'' to get the best case scenario for a rails request
on your
> hardware. This is the fastest you can expect rails to serve a
> dynamic request on your boxes. It will probably be around 100-140req/
> sec with sessions turned on. Then do a few more of this same
> benchmark against one mongrel and a few of your rails actions that
> represent short and long requests for your app. Write these results
> down or save them as you go. This is important because I discovered
> that 3 mongrels performs *better* then 5 mongrels on our
> architecture. This will vary widely though over different os and
> hardware configurations.
>
>         Now its time to add more mongrels until you hit a point of
> diminishing returns or you are confident in your ability to handle
> the traffic you expect plus some. ounds like you have two nice boxes
> for your app servers so start with 3 mongrels on one of the servers.
> Get this benchmark for the render :text action as well as the other
> actions you benched in your real app. At this point you should get
> more req/sec when you measure with concurrent users at 20-50. Then
> add one mongrel at a time and measure until adding one more mongrel
> does not buy you any more performance. This is your sweet spot where
> you are maxing out your proxy_balancers efficiency.
>
>         Now that you hit the sweet spot for one app server setup the second
> one the same way and do some more benches at high concurrency. If you
> have good hardware load balancers you should get a good improvement
> with the two app servers over just one of them. In reality software
> load balancers don''t multiply the perf of a single mongrel by the
> number of mongrels. Its more like adding another mongrel may get you
> 20-50% more until you hit the sweet spot and can''t get anymore.
But
> the good hardware load balancers are more efficient and get closer to
> 50-80% improvement when adding more nodes. Sometimes it''s nice to
go
> straight from the hardware load balancers right to mongrel and use an
> asset_host for static content. I don''t think you will need this
for
> the app you are talking about.
>
>         I suspect you will end up with 4-6 mongrels per app server and this
> will leave you some overhead for what you need to serve. Another
> point of reference is the http://yakimaherald.com gets about 75-90k
> page views/day on 3 mongrels behind apache22 on an xserve.
>
>
>
> -- Ezra Zygmuntowicz
> -- Lead Rails Evangelist
> -- ez at engineyard.com
> -- Engine Yard, Serious Rails Hosting
> -- (866) 518-YARD (9273)
Ezra, you the man.

I''m gonna read this over a few more times when its not 3 am, and
I''m
sure it will be very helpful with my current project.

thanks a lot,
Rob