Mongrel users - Nov 2006 - Determining ideal number of Mongrels for an app?

What''s a rule of thumb for guesstimating how many
Mongrels to use in a cluster for an app? I have an app
that gets about 5000 unique visitors per day. I
figured I''d give it plenty of Mongrels -- twenty to be
specific. After running out of memory and hitting the
swap periodically, I scaled it back to five and it
still seems to serve up visitors fine. So, is there
some super-secret math fomula I can use to determine
the ideal number of Mongrels I should start in a
cluster for an app?

Joe


 
____________________________________________________________________________________
Do you Yahoo!?
Everyone is raving about the all-new Yahoo! Mail beta.
http://new.mail.yahoo.com

On 11/28/06, Joe Ruby <joeat303 at yahoo.com>
wrote:
> What''s a rule of thumb for guesstimating how many
> Mongrels to use in a cluster for an app? I have an app
> that gets about 5000 unique visitors per day. I
> figured I''d give it plenty of Mongrels -- twenty to be
> specific. After running out of memory and hitting the
> swap periodically, I scaled it back to five and it
> still seems to serve up visitors fine. So, is there
> some super-secret math fomula I can use to determine
> the ideal number of Mongrels I should start in a
> cluster for an app?
>
More information in the docs:
http://mongrel.rubyforge.org/docs/how_many_mongrels.html

Also check about deployment:
http://mongrel.rubyforge.org/docs/choosing_deployment.html

> Joe
>
Hope That Helps,

-- 
Luis Lavena
Multimedia systems
-
Leaders are made, they are not born. They are made by hard effort,
which is the price which all of us must pay to achieve any goal that
is worthwhile.
Vince Lombardi

Hey Joe,

here''s the link I have bookmarked for this:
http://rubyforge.org/pipermail/mongrel-users/2006-May/000200.html

As far as I know, that''s still the way to go.

Cheers,
greg


----- Joe Ruby <joeat303 at yahoo.com> wrote:
> What''s a rule of thumb for guesstimating how many
> Mongrels to use in a cluster for an app? I have an app
> that gets about 5000 unique visitors per day. I
> figured I''d give it plenty of Mongrels -- twenty to be
> specific. After running out of memory and hitting the
> swap periodically, I scaled it back to five and it
> still seems to serve up visitors fine. So, is there
> some super-secret math fomula I can use to determine
> the ideal number of Mongrels I should start in a
> cluster for an app?
> 
> Joe
> 
> 
>  
>
____________________________________________________________________________________
> Do you Yahoo!?
> Everyone is raving about the all-new Yahoo! Mail beta.
> http://new.mail.yahoo.com
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users

> What''s a rule of thumb for guesstimating how many
> Mongrels to use in a cluster for an app? I have an app
> that gets about 5000 unique visitors per day. I
> figured I''d give it plenty of Mongrels -- twenty to be
> specific. After running out of memory and hitting the
> swap periodically, I scaled it back to five and it
> still seems to serve up visitors fine. So, is there
> some super-secret math fomula I can use to determine
> the ideal number of Mongrels I should start in a
> cluster for an app?
http://mongrel.rubyforge.org/docs/how_many_mongrels.html

Just for reference, we get about 500,000 pages a day and run 4 mongrel 
processes... we''ve got about 20 servers[1], but 4 gives us the best 
performance.

[1] We don''t need 20 except for about one month a year when we do about
8
million pages a day... then they come in handy, but even then they
don''t
sweat that much.

-philip

Hi,

On Die 28.11.2006 08:27, Joe Ruby wrote:
>
>What''s a rule of thumb for guesstimating how many Mongrels to use
in a
>cluster for an app? I have an app that gets about 5000 unique visitors
>per day. I figured I''d give it plenty of Mongrels -- twenty to be
>specific. After running out of memory and hitting the swap
>periodically, I scaled it back to five and it still seems to serve up
>visitors fine. So, is there some super-secret math fomula I can use to
>determine the ideal number of Mongrels I should start in a cluster for
>an app?
Super-Secret, I think maybe but a possible start for a *normal* formula
could be:

http://mongrel.rubyforge.org/docs/how_many_mongrels.html

Hth

Regards

Aleks

On 11/28/06, Joe Ruby <joeat303 at yahoo.com>
wrote:
> What''s a rule of thumb for guesstimating how many
> Mongrels to use in a cluster for an app? I have an app
> that gets about 5000 unique visitors per day. I
> figured I''d give it plenty of Mongrels -- twenty to be
> specific. After running out of memory and hitting the
> swap periodically, I scaled it back to five and it
> still seems to serve up visitors fine. So, is there
> some super-secret math fomula I can use to determine
> the ideal number of Mongrels I should start in a
> cluster for an app?
See this:
http://mongrel.rubyforge.org/docs/how_many_mongrels.html

On Tue, Nov 28, 2006 at 11:03:06AM -0600, Philip Hallstrom
wrote:
> > What''s a rule of thumb for guesstimating how many
> > Mongrels to use in a cluster for an app? I have an app
> > that gets about 5000 unique visitors per day. I
> > figured I''d give it plenty of Mongrels -- twenty to be
> > specific. After running out of memory and hitting the
> > swap periodically, I scaled it back to five and it
> > still seems to serve up visitors fine. So, is there
> > some super-secret math fomula I can use to determine
> > the ideal number of Mongrels I should start in a
> > cluster for an app?
> 
> http://mongrel.rubyforge.org/docs/how_many_mongrels.html
> 
> Just for reference, we get about 500,000 pages a day and run 4 mongrel 
> processes... we''ve got about 20 servers[1], but 4 gives us the
best
> performance.
I''m curious - do you have 4 * 20 Mongrels or just 4? How many
requests/sec does Rails report in your logs?

-- 
Thanks,
- Jacob Atzen

On Tue, 28 Nov 2006 08:27:38 -0800 (PST)
Joe Ruby <joeat303 at yahoo.com> wrote:
> What''s a rule of thumb for guesstimating how many
> Mongrels to use in a cluster for an app? I have an app
> that gets about 5000 unique visitors per day. 
Pathetic.  5000/day?  Sad.  Oh wait, is that 5000 visitors doing 100k
request/second during the whole day, or 5000 visitors doing .005 request/second
at noon?

(I''m jokingly saying you should find out what their usage pattern is
and translate that to req/sec processed and kb/second transfered first.)
> I figured I''d give it plenty of Mongrels -- twenty to be
> specific. After running out of memory and hitting the
> swap periodically, I scaled it back to five and it
> still seems to serve up visitors fine. So, is there
> some super-secret math fomula I can use to determine
> the ideal number of Mongrels I should start in a
> cluster for an app?
Many folks pointed you at the semi-official tuning documentation.  20 is totally
over kill and will hurt you more than you know.  I find it''s about
2-4/CPU depending on the amount of ram you have and the speed of your CPU.  BUT
YOU SHOULD TUNE EVERY APP EVERY TIME NO MATTER WHAT YOU CHANGED.  Don''t
believe anything as gospel, always test it yourself.

Now, lately I''ve been wanting to add a couple of qualifiers on the
scaling issue related to your resource usage.  In some situations people are
using up tons of ram or tons of files.  If you have 2-4 mongrels you quickly
outstrip the ram and files.  For these folks I tell them to do a different test
that tries to do a lot of concurrent but slow connections to eat up as much ram
and file handles as they can.  Then, they expand the mongrels out to handle the
number of connections they need.  Many times, people in these situations have to
buy more machines.

The main culprit of this is RMagick.  If you have RMagick you need to simulate a
ton of people concurrently slamming you with files to make sure you have the
resources.

This is also a reasonably good description of file limit defaults and HTTP
performance:

http://www.xenoclast.org/doc/benchmark/HTTP-benchmarking-HOWTO/node7.html

Play with these settings as well for each, maybe even just increasing the
defaults.

-- 
Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
http://www.zedshaw.com/
http://www.awprofessional.com/title/0321483502 -- The Mongrel Book
http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.

> On Tue, Nov 28, 2006 at 11:03:06AM -0600, Philip Hallstrom wrote:
>>> What''s a rule of thumb for guesstimating how many
>>> Mongrels to use in a cluster for an app? I have an app
>>> that gets about 5000 unique visitors per day. I
>>> figured I''d give it plenty of Mongrels -- twenty to be
>>> specific. After running out of memory and hitting the
>>> swap periodically, I scaled it back to five and it
>>> still seems to serve up visitors fine. So, is there
>>> some super-secret math fomula I can use to determine
>>> the ideal number of Mongrels I should start in a
>>> cluster for an app?
>>
>> http://mongrel.rubyforge.org/docs/how_many_mongrels.html
>>
>> Just for reference, we get about 500,000 pages a day and run 4 mongrel
>> processes... we''ve got about 20 servers[1], but 4 gives us the
best
>> performance.
>
> I''m curious - do you have 4 * 20 Mongrels or just 4? How many
> requests/sec does Rails report in your logs?
We have 4 mongrels per server.

Just did a quick grep on one of the servers log files and for about 6000 
requests we are averaging 38 reqs/sec with a standard deviation of 113.

If I throw out about 2700 requests that I know are slow (< 5 req/s mostly 
cause we can''t cache them) the average goes to 65 reqs/sec.

Like I said, we only need those for a little bit every year... during the 
world series of poker.  People really like there poker :)

http://www.alexaholic.com/cardplayer.com+penny-arcade.com+basecamphq.com

-philip

On Tue, 28 Nov 2006 15:23:14 -0600 (CST)
Philip Hallstrom <mongrel at philip.pjkh.com> wrote:
> > On Tue, Nov 28, 2006 at 11:03:06AM -0600, Philip Hallstrom wrote:
> >> Just for reference, we get about 500,000 pages a day and run 4
mongrel
> >> processes... we''ve got about 20 servers[1], but 4 gives
us the best
> >> performance.
> >
> > I''m curious - do you have 4 * 20 Mongrels or just 4? How many
> > requests/sec does Rails report in your logs?
> 
> Just did a quick grep on one of the servers log files and for about 6000 
> requests we are averaging 38 reqs/sec with a standard deviation of 113.
Standard Deviation!  YES!

-- 
Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
http://www.zedshaw.com/
http://www.awprofessional.com/title/0321483502 -- The Mongrel Book
http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.

> On Tue, 28 Nov 2006 08:27:38 -0800 (PST)
> Joe Ruby <joeat303 at yahoo.com> wrote:
> 
> > What''s a rule of thumb for guesstimating how many
> > Mongrels to use in a cluster for an app? I have an
app
> > that gets about 5000 unique visitors per day. 
> 
> Pathetic.  5000/day?  Sad.  Oh wait, is that 5000
visitors doing 100k request/second during the whole
day, or 5000 visitors doing .005 request/second at
noon?

It''s both! Yeah, it''s pathetic and sad, but you think
I trust my enterprisey apps to this script-kiddy RoR
stuff?!? :P
> (I''m jokingly saying you should find out what their
usage pattern is and translate that to req/sec
processed and kb/second transfered first.)
> 
> > I figured I''d give it plenty of Mongrels -- twenty
to be
> > specific. After running out of memory and hitting
the
> > swap periodically, I scaled it back to five and it
> > still seems to serve up visitors fine. So, is
there
> > some super-secret math fomula I can use to
determine
> > the ideal number of Mongrels I should start in a
> > cluster for an app?
> 
> Many folks pointed you at the semi-official tuning
documentation.  20 is totally over kill and will hurt
you more than you know.  I find it''s about 2-4/CPU
depending on the amount of ram you have and the speed
of your CPU.  BUT YOU SHOULD TUNE EVERY APP EVERY TIME
NO MATTER WHAT YOU CHANGED.  Don''t believe anything as
gospel, always test it yourself.
> 
> Now, lately I''ve been wanting to add a couple of
qualifiers on the scaling issue related to your
resource usage.  In some situations people are using
up tons of ram or tons of files.  If you have 2-4
mongrels you quickly outstrip the ram and files.  For
these folks I tell them to do a different test that
tries to do a lot of concurrent but slow connections
to eat up as much ram and file handles as they can. 
Then, they expand the mongrels out to handle the
number of connections they need.  Many times, people
in these situations have to buy more machines.

I know you like doing long lines (scroll, scroll,
scroll), but check it, yo:

http://rubyforge.org/pipermail/mongrel-users/2006-November/002258.html
> A joke page about Mongrel MUDCRAP certification that
hasn''t changed but is now accesible from the site
menu: http://mongrel.rubyforge.org/certified.html

A JOKE?!? W. T. F. ?!?
> 
> The main culprit of this is RMagick.  If you have
RMagick you need to simulate a ton of people
concurrently slamming you with files to make sure you
have the resources.
> 
> This is also a reasonably good description of file
limit defaults and HTTP performance:
> 
>
http://www.xenoclast.org/doc/benchmark/HTTP-benchmarking-HOWTO/node7.html
> 
> Play with these settings as well for each, maybe
even just increasing the defaults.

But seriously, thanks for the tips. :)

Joe

> 
> -- 
> Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
> http://www.zedshaw.com/
> http://www.awprofessional.com/title/0321483502 --
The Mongrel Book
> http://mongrel.rubyforge.org/
> http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.

 
____________________________________________________________________________________
Do you Yahoo!?
Everyone is raving about the all-new Yahoo! Mail beta.
http://new.mail.yahoo.com

>> A joke page about Mongrel MUDCRAP certification that
>>
>>
> hasn''t changed but is now accesible from the site
> menu: http://mongrel.rubyforge.org/certified.html
>
> A JOKE?!? W. T. F. ?!?
>
>
No kidding, I''m still waiting for Mongrel Business & Enterprise  
Edition for the Marketplace... Deluxe.

:^)

Bryan Thompson
Loy Inc

(Sorry if this double-posts... wonky email settings here.)

On Nov 28, 2006, at 15:49 , Zed A. Shaw wrote:
> Many folks pointed you at the semi-official tuning documentation.   
> 20 is totally over kill and will hurt you more than you know.  I  
> find it''s about 2-4/CPU depending on the amount of ram you have
and
> the speed of your CPU.  BUT YOU SHOULD TUNE EVERY APP EVERY TIME NO  
> MATTER WHAT YOU CHANGED.  Don''t believe anything as gospel, always
> test it yourself.
Zed,

Do dual-core chips count as two CPU''s for this calculation?

Also what is the recommended amount of ram per mongrel process or per  
CPU based on your experience?

Thanks,
	~Wayne

On Wed, 29 Nov 2006 10:08:38 -0500
"Wayne E. Seguin" <wayneeseguin at gmail.com> wrote:
> 
> On Nov 28, 2006, at 15:49 , Zed A. Shaw wrote:
> > Many folks pointed you at the semi-official tuning documentation.   
> > 20 is totally over kill and will hurt you more than you know.  I  
> > find it''s about 2-4/CPU depending on the amount of ram you
have and
> > the speed of your CPU.  BUT YOU SHOULD TUNE EVERY APP EVERY TIME NO  
> > MATTER WHAT YOU CHANGED.  Don''t believe anything as gospel,
always
> > test it yourself.
> 
> Zed,
> 
> Do dual-core chips count as two CPU''s for this calculation?
Do believe anything I said above.  Find out your own optimal configuration.  The
reason is if I say 2-4 people will assume it''s gospel and never test
their apps.  You might find your app can get better performance with more or
less, or you might find it is so slow that really you just need to spread it out
across as many as you can.

Test it for yourself, and test it every time a new deployment goes out.
 
> Also what is the recommended amount of ram per mongrel process or per  
> CPU based on your experience?
Again, test it.  You should slam a bunch of your biggest pages with large
requests and try to see what it hits as a peak, then set that for you resource
calculations.  This is especially true if you use rmagick.

Big thing is to go through your test and development logs and look for huge
queries.  Those will kill you every time.  Also, make sure your session clearing
stuff is good.  Someone was getting huge memory penalties recently because they
had 5.2 million stale sessions in their mysql.

Again, believe nothing, test everything.

-- 
Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
http://www.zedshaw.com/
http://www.awprofessional.com/title/0321483502 -- The Mongrel Book
http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.

On Thu, 30 Nov 2006 00:22:03 -0500
"Zed A. Shaw" <zedshaw at zedshaw.com> wrote:
> Do believe anything I said above. 
Correction:

"DON''T believe anything I said above..."

-- 
Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
http://www.zedshaw.com/
http://www.awprofessional.com/title/0321483502 -- The Mongrel Book
http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.

> On Thu, 30 Nov 2006 00:22:03 -0500
> "Zed A. Shaw" <zedshaw at zedshaw.com> wrote:
>
>> Do believe anything I said above.
>
> Correction:
>
> "DON''T believe anything I said above..."
Well, now I''m just completely confused!  :-)  And I''m pretty
sure the
above is going to cause a memory leak of some sort :)