Mongrel users - Jan 2008 - Again: Workaround found for request queuing vs. num_processors, accept/close

Hello all.
I too found out that I sometimes have some action that can take up to 10 
seconds in my rails application. I''ve read all arguments Zed made about
polling/and inbox strategies, and I think I just can''t work around my
feeling
that a "wrong" request that takes up too much time should be able to
lock
subsequent requests in mongrels queue. That''s what I have a balancer
for,
right? :)
So I tried Robert Melas hint[1] and used
BalancerMember http://foo max=1 acquire=100
to see if that helps. But I may have hit a bug[2], and this does not work for 
me.
Does this work for someone? With what version of Apache? I am running 2.2.4. 
Or do all of you have only very fast requests that do not take more than a 
few seconds max to complete? Or do you run a different load balancer? 
HAProxy?
I also think that the overall speed of the application would incerase even 
with relatively fast requests, because the balancer would forward the 
requests always to a free mongrel that has nothing to do, and can handle the 
request immediately.

BTW, it looks like the acquire parameter is not in seconds, but in 
milliseconds, so I tried acquire=100 as well, but no difference.
From[2]
        /* Acquire timeout in milliseconds.
         * If set this will be the maximum time to
         * wait for a free connection.
         */

Maybe this belongs to the Apache ml and not to mongrel, but I think maybe some 
of you hit this problem and solved it somehow.


Another issue: on my system, 
mongrel_rails --version
Mongrel Web Server 1.1.3 (installed as a gem)
the --num-procs parameter does not seem to work. To debug the problem I added
STDERR.puts "TEST: #{worker_list.length}  #{@num_processors}"
on line 279 of /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb, and 
to my surprise it looks like the parameter is ignored. If I run it

$ mongrel_rails start -a 127.0.0.1 -p 8999 --num-procs 1 &
[1] 18617
** Starting Mongrel listening at 127.0.0.1:8999
** Starting Rails with development environment...
config/boot.rb:28:Warning: require_gem is obsolete.  Use gem instead.
** Rails loaded.
** Loading any Rails specific GemPlugins
** Signals ready.  TERM => stop.  USR2 => restart.  INT => stop (no
restart).
** Rails signals registered.  HUP => reload (without restart).  It might not 
work well.
** Mongrel 1.1.3 available at 127.0.0.1:8999
** Use CTRL-C to stop.

and then:

$ w3m -dump http://localhost:8999/test >/dev/null
TEST: 0  950
$

So it looks like @num_processors is still at the default value of 950, and not 
1 as I told it on the command line. Does the deprecation warning maybe have 
something to do with this (I don''t think so, but you never know...)?


Thank you!
Simon

[1]http://rubyforge.org/pipermail/mongrel-users/2007-October/004145.html
[2]http://svn.apache.org/viewvc/httpd/httpd/trunk/modules/proxy/mod_proxy.c?revision=608189&view=markup

On Jan 23, 2008 12:18 PM, Simon Santoro <simon.santoro at gmail.com>
wrote:
> Or do you run a different load balancer [for limiting the number of
outstanding requests].
Many people do it with HAProxy. EngineYard built an Nginx plugin that
does the same (don''t think they released it yet).

-- 
Alexey Verkhovsky
CruiseControl.rb [http://cruisecontrolrb.thoughtworks.com]
RubyWorks [http://rubyworks.thoughtworks.com]

On Wed, 23 Jan 2008, Simon Santoro wrote:
> ...
> So it looks like @num_processors is still at the default value of 950, and
not
> 1 as I told it on the command line. Does the deprecation warning maybe have
> something to do with this (I don''t think so, but you never
know...)?

hum.. this seems like a bug. Could you please open a ticket for it?

Cheers,

filipe {
  @              icewall.org
  GPG         1024D/A6BA423E
  http://filipe.icewall.org/
}

On Wednesday 23 January 2008 21:36:39 Alexey Verkhovsky
wrote:
> Many people do it with HAProxy. EngineYard built an Nginx plugin that
> does the same (don''t think they released it yet).
Thanks for your replay. At least now I know that someone understood my poor 
English :)

On Jan 23, 2008, at 12:36 PM, Alexey Verkhovsky wrote:
> On Jan 23, 2008 12:18 PM, Simon Santoro <simon.santoro at gmail.com>
> wrote:
>> Or do you run a different load balancer [for limiting the number of  
>> outstanding requests].
>
> Many people do it with HAProxy. EngineYard built an Nginx plugin that
> does the same (don''t think they released it yet).

	Actually the nginx fair balancer has been released for a while. You  
can read up on it and get it here:

http://wiki.codemongers.com/NginxHttpUpstreamFairModule

Cheers-
- Ezra Zygmuntowicz
-- Founder & Software Architect
-- ezra at engineyard.com
-- EngineYard.com

On Wednesday 23 January 2008 22:07:45 Filipe wrote:
> hum.. this seems like a bug. Could you please open a ticket for it?
sure.
http://rubyforge.org/tracker/index.php?func=detail&aid=17428&group_id=1306&atid=5145

On Wednesday 23 January 2008 23:03:21 Ezra Zygmuntowicz
wrote:
> Actually the nginx fair balancer has been released for a while. You ?
> can read up on it and get it here:
>
> http://wiki.codemongers.com/NginxHttpUpstreamFairModule
Hmmm. I can''t use a no-name web server with a load balancer that is a
set of
patches for that server on our intranet production server. At the first 
problem I would lose my job.

On Jan 23, 2008, at 2:36 PM, Simon Santoro wrote:
> On Wednesday 23 January 2008 23:03:21 Ezra Zygmuntowicz wrote:
>> Actually the nginx fair balancer has been released for a while. You
>> can read up on it and get it here:
>>
>> http://wiki.codemongers.com/NginxHttpUpstreamFairModule
>
> Hmmm. I can''t use a no-name web server with a load balancer that
is
> a set of
> patches for that server on our intranet production server. At the  
> first
> problem I would lose my job.

	That''s too bad then, you will have to suffer on with apache i guess.  
Nginx is a lot more solid then apache is in stability and performance.  
I''m personally running over 1000 nginx daemons with absolutely no  
problems to speak of with it.

Cheers-
- Ezra Zygmuntowicz
-- Founder & Software Architect
-- ezra at engineyard.com
-- EngineYard.com

On Wednesday 23 January 2008 23:45:43 Ezra Zygmuntowicz
wrote:
> That''s too bad then, you will have to suffer on with apache i
guess.
Probably.
> Nginx is a lot more solid then apache is in stability and performance. ?
> I''m personally running over 1000 nginx daemons with absolutely no
?
> problems to speak of with it.
Do you have your architecture documented somewhere online? Can I read it?
I need Apache for kerberos for SSO, and I don''t think nginx supports
that
(could not find it). I could put Apache in front of nginx balancing the 
mongrels, but at that point it''s maybe better to use haproxy like the
the
rubyworks guys suggest[1] (no need for patches).
I''m open for suggestions.

[1]http://rubyworks.rubyforge.org/manual/haproxy.html

On Jan 23, 2008, at 2:57 PM, Simon Santoro wrote:
> On Wednesday 23 January 2008 23:45:43 Ezra Zygmuntowicz wrote:
>> That''s too bad then, you will have to suffer on with apache i
guess.
>
> Probably.
>
>> Nginx is a lot more solid then apache is in stability and  
>> performance.
>> I''m personally running over 1000 nginx daemons with absolutely
no
>> problems to speak of with it.
>
> Do you have your architecture documented somewhere online? Can I  
> read it?
> I need Apache for kerberos for SSO, and I don''t think nginx
supports
> that
> (could not find it). I could put Apache in front of nginx balancing  
> the
> mongrels, but at that point it''s maybe better to use haproxy like
> the the
> rubyworks guys suggest[1] (no need for patches).
> I''m open for suggestions.
>
> [1]http://rubyworks.rubyforge.org/manual/haproxy.html

Hey Simon-

	I don''t have any whitepapers about our architecture currently  
available. But the 50,000 ft view is like this:

Http request from client ->  LVS redundant load balancers -> Nginx ->  
mongrel_cluster.

	The nginx and mongrel clusters run on the same Xen VM. with multiple  
VM''s behind the LVS load balancers.

	You are correct that nginx does not support kerberos so you will need  
to keep apache. Haproxy is probably a good solution for you in this  
case.

Cheers-

- Ezra Zygmuntowicz
-- Founder & Software Architect
-- ezra at engineyard.com
-- EngineYard.com

On Jan 23, 2008 4:42 PM, Ezra Zygmuntowicz <ezmobius at gmail.com>
wrote:
> You are correct that nginx does not support kerberos so you will need
> to keep apache. Haproxy is probably a good solution for you in this
> case.
+1

About Nginx: besides EngineYard and a lot of other Rails deployments,
Nginx serves something like one-fourth of Russian internet traffic,
IIRC. It''s not exactly no-name, just hasn''t been discovered by
the
rest of the world until recently.

-- 
Alexey Verkhovsky
CruiseControl.rb [http://cruisecontrolrb.thoughtworks.com]
RubyWorks [http://rubyworks.thoughtworks.com]

On Wed, 23 Jan 2008 23:36:34 +0100
Simon Santoro <simon.santoro at gmail.com> wrote:
> On Wednesday 23 January 2008 23:03:21 Ezra Zygmuntowicz wrote:
> > Actually the nginx fair balancer has been released for a while. You ?
> > can read up on it and get it here:
> >
> > http://wiki.codemongers.com/NginxHttpUpstreamFairModule
> 
> Hmmm. I can''t use a no-name web server with a load balancer that
is a set of
> patches for that server on our intranet production server. At the first 
> problem I would lose my job.
Uh, but, Apache is quite literally "a patchy" version of NCSA httpd.
:-)

-- 
Zed A. Shaw
- Hate: http://savingtheinternetwithhate.com/
- Good: http://www.zedshaw.com/
- Evil: http://yearofevil.com/

On Wed, 23 Jan 2008 14:45:43 -0800
Ezra Zygmuntowicz <ezmobius at gmail.com> wrote:
> 	That''s too bad then, you will have to suffer on with apache i
guess.
> Nginx is a lot more solid then apache is in stability and performance.  
> I''m personally running over 1000 nginx daemons with absolutely no
> problems to speak of with it.
Yep, a *single* nginx server on a 256M virtual machine with just 2
workers handled a simultaneous Redditing, TechCrunch, TechMeme, and
Slashdotting without so much as a hitch in performance.  That''s 170k
uniq visitors in 3 days.

To put this into perspective, I had people ask me to take links off my
pages because the trail-off onto their pages was killing their server.

Of course, I did do some special tuning, but that''s another story.

-- 
Zed A. Shaw
- Hate: http://savingtheinternetwithhate.com/
- Good: http://www.zedshaw.com/
- Evil: http://yearofevil.com/

Ezra Zygmuntowicz wrote:
> Hey Simon-
> 
> 	I don''t have any whitepapers about our architecture currently  
> available. But the 50,000 ft view is like this:
> 
> Http request from client ->  LVS redundant load balancers -> Nginx
->
> mongrel_cluster.
> 
> 	The nginx and mongrel clusters run on the same Xen VM. with multiple  
> VM''s behind the LVS load balancers.
Cool, good to know. This setup would be a bit overkill to me, but good 
to know how you set things up and how they work.
> 	You are correct that nginx does not support kerberos so you will need  
> to keep apache. Haproxy is probably a good solution for you in this  
> case.
All right. I''ve set up apache as reverse-/proxy serving haproxy 
balancing the mongrels.
Just to help someone who would like to go down the same road, this is my 
apache site config:

<Virtualhost *:80>
   ServerName whatever
   DocumentRoot /home/myhome/MyHome/public/


   ProxyPass /images !
   ProxyPass /stylesheets !
   ProxyPass /javascripts !

   Alias /images /home/myhome/MyHome/public/images
   Alias /stylesheets /home/myhome/MyHome/public/stylesheets
   Alias /javascripts /home/myhome/MyHome/public/javascripts

   ProxyPass / http://127.0.0.1:8000/
   ProxyPassReverse / http://127.0.0.1:8000
   ProxyPreserveHost on

   <Directory /home/myhome/MyHome/public>
      Options ExecCGI FollowSymLinks
      AllowOverride all
      Order allow,deny
      Allow from all
   </Directory>
</Virtualhost>

and my haproxy.cfg file (took it from rubyworks):

global
web server
   maxconn 500
socket, therefore
   log   127.0.0.1       local0
   user nobody
   group nogroup
defaults
   log global
   mode http
   balance roundrobin
   maxconn 500
   option httplog
   option abortonclose
   retries 3
   redispatch
   clitimeout 120000
   contimeout 120000
   srvtimeout 120000
listen rails :8000
   server rails-1 localhost:8001 maxconn 1 check
   server rails-2 localhost:8002 maxconn 1 check
   server rails-3 localhost:8003 maxconn 1 check


to test I set the clitimeout, contimeout and srvtimeout to high numbers, 
so I can see how long actions/requests get handled.
The "maxconn 1" parameter does exactly what it says (one request per 
mongrel).
I have to test this setup a bit more, and see what haproxy does if a 
mongrel goes down or is disabled.

Thank you all for your feedback.

Simon

On 24 Jan 2008, at 01:46, Zed A. Shaw wrote:
> On Wed, 23 Jan 2008 14:45:43 -0800
> Ezra Zygmuntowicz <ezmobius at gmail.com> wrote:
>
>> 	That''s too bad then, you will have to suffer on with apache i
guess.
>> Nginx is a lot more solid then apache is in stability and  
>> performance.
>> I''m personally running over 1000 nginx daemons with absolutely
no
>> problems to speak of with it.
>
> Yep, a *single* nginx server on a 256M virtual machine with just 2
> workers handled a simultaneous Redditing, TechCrunch, TechMeme, and
> Slashdotting without so much as a hitch in performance.  That''s
170k
> uniq visitors in 3 days.
>
> To put this into perspective, I had people ask me to take links off my
> pages because the trail-off onto their pages was killing their server.
Just a "me too" post - nginx is solid. We ran a site with a ~1Mb  
homepage weight (once all the graphics and video clips all over it had  
loaded - yeah, I know ...) on nginx with 2 workers, and it took 570k  
PIs across 65k unique visitors in one day, and that mostly in a 5-hour  
period. OK, it was a dedicated dual Xeon with 4GB RAM and not much on  
it apart from CentOS base + nginx, but I kept a close eye and I didn''t
see it go above 500Mb, and the process load was low enough to make a  
brave man weep.
> Of course, I did do some special tuning, but that''s another story.
A story you might be happy to share, perhaps?

i

--
Igor Clark // POKE // 10 Redchurch Street // E2 7DD // +44 (0)20 7749  
5355 // www.pokelondon.com