Mongrel users - Sep 2007 - Apache mod_proxy_balancer hang on high traffic hour

more info

just 1 server

webserver + appserver + dbserver

On 9/14/07, Yan Meng <dreamwords at gmail.com>
wrote:
>
> Hi, Guys
>
> Our team is running a web2.0 finance site in China
>
> http://www.caibangzi.com/
>
> We use APACHE + mod_proxy_balancer + mongrel to deploy our application, it
> workd super in the past few monthes.
>
> However, just  from last week, at high traffic hour (about 40,000 request
> per hour), our site is very very slow, it usually took more than 10 seconds
> to get a page or simply return timeout error.
>
> I telnet to the server and found that when I visit directly to one mongrel
> instance, it can return shortly, however when I visit from apache
interface,
> it hang.
>
> I also cannot visit (timeout) the mod_proxy_balancer''s frontend to
see the
> balance-manager.
>
> However, the server load is just normal :(
>
> We guess maybe the reason is slow action block mongrel, is it the reason?
> Have any of you experienced of this?
>
> Any comments are appreciated.
>
> Our Server:
>
> Dual-Core AMD Opteron(tm) Processor 2214 HE * 2
> 4G RAM
> 20 mongrel instances
> mysql
>
> --
> Best Regards,
>
> -Yan Meng
> http://www.caibangzi.com/



-- 
Best Regards,

-Yan Meng
http://www.caibangzi.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20070914/3b7e6372/attachment.html

I am a neophyte at this, but if you read the documentation about 
mongrel and load balancing they strongly suggest going to more boxes 
and only running 4 or so Mongrel instances per box. If you only have 
a dual core server, it does not make much sense to jack up the RAM so 
you can stuff more processes in. Performance is all about eliminating 
latency, if you have two lowly cpus handling 20k requests per hour 
yes, I bet you''re slow.

Other factors
1. Database should be clustered and running on its own box.
2. Are you properly managing logs? They would be growing at a rapid rate.
3. Any chance of going to a quad or octal CPU configuration?

Best regards,
Bill

At 11:50 AM 9/13/2007, you wrote:
>more info
>
>just 1 server
>
>webserver + appserver + dbserver
>
>On 9/14/07, Yan Meng 
><<mailto:dreamwords at gmail.com>dreamwords at gmail.com >
wrote:
>Hi, Guys
>
>Our team is running a web2.0 finance site in China
>
><http://www.caibangzi.com/>http://www.caibangzi.com/
>
>We use APACHE + mod_proxy_balancer + mongrel to deploy our 
>application, it workd super in the past few monthes.
>
>However, just  from last week, at high traffic hour (about 40,000 
>request per hour), our site is very very slow, it usually took more 
>than 10 seconds to get a page or simply return timeout error.
>
>I telnet to the server and found that when I visit directly to one 
>mongrel instance, it can return shortly, however when I visit from 
>apache interface, it hang.
>
>I also cannot visit (timeout) the mod_proxy_balancer''s frontend to 
>see the balance-manager.
>
>However, the server load is just normal :(
>
>We guess maybe the reason is slow action block mongrel, is it the 
>reason? Have any of you experienced of this?
>
>Any comments are appreciated.
>
>Our Server:
>
>Dual-Core AMD Opteron(tm) Processor 2214 HE * 2
>4G RAM
>20 mongrel instances
>mysql
>
>--
>Best Regards,
>
>-Yan Meng
><http://www.caibangzi.com/>http://www.caibangzi.com/
>
>
>
>
>--
>Best Regards,
>
>-Yan Meng
><http://www.caibangzi.com/>http://www.caibangzi.com/
>_______________________________________________
>Mongrel-users mailing list
>Mongrel-users at rubyforge.org
>http://rubyforge.org/mailman/listinfo/mongrel-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20070913/5e2744e0/attachment.html

Have you run system profilers to see where the bottlenecks are? You 
can do this when the system is moderately loaded, it will give you an 
idea what problem will hit when the load goes up.

At 11:50 AM 9/13/2007, you wrote:
>more info
>
>just 1 server
>
>webserver + appserver + dbserver
>
>On 9/14/07, Yan Meng 
><<mailto:dreamwords at gmail.com>dreamwords at gmail.com >
wrote:
>Hi, Guys
>
>Our team is running a web2.0 finance site in China
>
><http://www.caibangzi.com/>http://www.caibangzi.com/
>
>We use APACHE + mod_proxy_balancer + mongrel to deploy our 
>application, it workd super in the past few monthes.
>
>However, just  from last week, at high traffic hour (about 40,000 
>request per hour), our site is very very slow, it usually took more 
>than 10 seconds to get a page or simply return timeout error.
>
>I telnet to the server and found that when I visit directly to one 
>mongrel instance, it can return shortly, however when I visit from 
>apache interface, it hang.
>
>I also cannot visit (timeout) the mod_proxy_balancer''s frontend to 
>see the balance-manager.
>
>However, the server load is just normal :(
>
>We guess maybe the reason is slow action block mongrel, is it the 
>reason? Have any of you experienced of this?
>
>Any comments are appreciated.
>
>Our Server:
>
>Dual-Core AMD Opteron(tm) Processor 2214 HE * 2
>4G RAM
>20 mongrel instances
>mysql
>
>--
>Best Regards,
>
>-Yan Meng
><http://www.caibangzi.com/>http://www.caibangzi.com/
>
>
>
>
>--
>Best Regards,
>
>-Yan Meng
><http://www.caibangzi.com/>http://www.caibangzi.com/
>_______________________________________________
>Mongrel-users mailing list
>Mongrel-users at rubyforge.org
>http://rubyforge.org/mailman/listinfo/mongrel-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20070913/99d10b92/attachment.html

We are having a similar problem.  For the last 6 months everything has been
fine.  The last couple of weeks as our pageviews continue to rise we are now
facing an unresponsive Mongrel intermittently.

We are running on BSD 6.2 with 4 Mongrels in the cluster.

Comments are appreciated as well!

Thank you,

Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/mongrel-users/attachments/20070915/26a77799/attachment.html

Apache''s mod_proxy_balancer module is a fully blocking module and with
the default httpd.conf you''re going to max out in the 120-160 requests/
second range on a decent box. You can tune up its proxying to about a  
1000 req/sec

So yes the net result is that you can really only put a couple of  
mongrels behind apache''s proxy engine (about 2   "hello
world" rails
mongrels).

If you can hit mongrels directly and all is fine, but it''s clogged up  
via apache ( and you''re doing in the hundreds of request/sec), than  
this is the botteneck that you''ve hit. This at least explains the  
original poster''s problem.

-Jason

On Sep 15, 2007, at 6:05 AM, "Ben Kruger" <benkruger at
gmail.com> wrote:
> We are having a similar problem.  For the last 6 months everything  
> has been fine.  The last couple of weeks as our pageviews continue  
> to rise we are now facing an unresponsive Mongrel intermittently.
>
> We are running on BSD 6.2 with 4 Mongrels in the cluster.
>
> Comments are appreciated as well!
>
> Thank you,
>
> Ben
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users

On 9/13/07, Yan Meng <dreamwords at gmail.com>
wrote:
> more info
>
> just 1 server
>
> webserver + appserver + dbserver
>
> On 9/14/07, Yan Meng <dreamwords at gmail.com > wrote:
> > We use APACHE + mod_proxy_balancer + mongrel to deploy our
application, it
> workd super in the past few monthes.
> >
> > However, just  from last week, at high traffic hour (about 40,000
request
> per hour), our site is very very slow, it usually took more than 10 seconds
> to get a page or simply return timeout error.
> >
> > I telnet to the server and found that when I visit directly to one
mongrel
> instance, it can return shortly, however when I visit from apache
interface,
> it hang.
> >
> > I also cannot visit (timeout) the mod_proxy_balancer''s
frontend to see the
> balance-manager.
I''ve been letting this one bounce around in my head for a while.

I think that sentence there, right at the end of what I quoted, is an
important clue.

If you can''t get to the mod_proxy_balancer''s front end, then
Apache is
failing to field the request in the first place.  Start there.  Look
at your Apache config.  Look at your Apache.  Are you using an Apache
that has things built into it, or loaded, which are not needed?  If
so, rebuild without those items, or comment them out of your
configuration.  Look at your allowances for keep alive, and for
MaxClients and related load handling config items.

After you look at that, then come back and look at mongrel and your
application.  Do you know how fast your app actions are?  Do you have
some actions that are very slow?  Are you facing a situation where
some bottleneck is preventing your one server from handling requests
as fast as they are coming in?

40k requests/hour on your single server may or may not be a problem.
It all depends on the application.  That''s only about 11
requests/second, which isn''t really a lot.

That''s about all that can be said without more information and
analysis.


Kirk Haines