Puppet users - Jun 2009 - Performance of Passenger vs. Mongrel

Howdy:

How does Passenger perform compared to Mongrel?  For us Passenger looks
worse.  We have two puppetmasters; one does file serving, the other does
everything else.  We just started running 0.25 beta2 on both.  The
file-server is running RubyEE + Passenger and the non-file-server is
running Mongrel.  When our heavy-hitter schedule runs the load average
on our file server spikes much higher than it did when it was running
0.25 beta1 with Mongrel.

We will try 0.25 beta2 + Mongrel on our file server to see if that makes
a difference but I''m curious what others have seen and what we should
expect.

Thanks!


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Mark Plaksin wrote:
> Howdy:
> 
> How does Passenger perform compared to Mongrel?  For us Passenger looks
> worse.  We have two puppetmasters; one does file serving, the other does
> everything else.  We just started running 0.25 beta2 on both.  The
> file-server is running RubyEE + Passenger and the non-file-server is
> running Mongrel.  When our heavy-hitter schedule runs the load average
> on our file server spikes much higher than it did when it was running
> 0.25 beta1 with Mongrel.
> 
Are you running the same # of mongrels as Passenger workers? What is 
your Passenger config? What are the specs of your app server?

-scott

-- 
scott@ohlol.net

http://github.com/ohlol

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Scott Smith <scott@ohlol.net> writes:
> Mark Plaksin wrote:
>
>> Howdy:
>> 
>> How does Passenger perform compared to Mongrel?  For us Passenger looks
>> worse.  We have two puppetmasters; one does file serving, the other
does
>> everything else.  We just started running 0.25 beta2 on both.  The
>> file-server is running RubyEE + Passenger and the non-file-server is
>> running Mongrel.  When our heavy-hitter schedule runs the load average
>> on our file server spikes much higher than it did when it was running
>> 0.25 beta1 with Mongrel.
>
> Are you running the same # of mongrels as Passenger workers? 
Sort of :) We run 12 masters in mongrel.  Passenger doesn''t seem have a
"run at least this many puppetmasters" setting.  We set the max to 12
to
match mongrel but we never saw more than 6 masters running.  We bumped
max to 24 late yesterday and I now see 11 masters running.  We''ll see
how it performs during today''s big schedule run.
> What is your Passenger config? 
We''re using what the docs suggest except MaxPoolSize is now 24:
http://github.com/reductivelabs/puppet/tree/607b01e82ea294068fdd554e59bc8e5fe3f9761a/ext/rack/files
> What are the specs of your app server?
All of our puppetmasters are running on the same size hardware.  The
machines have 8 cores and 12G of RAM.

Do you expect Passenger to perform better?


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Fri, Jun 26, 2009 at 6:05 AM, Mark Plaksin<happy@usg.edu>
wrote:
>
> Scott Smith <scott@ohlol.net> writes:
>
>> Mark Plaksin wrote:
>>
>>> Howdy:
>>>
>>> How does Passenger perform compared to Mongrel?  For us Passenger
looks
>>> worse.  We have two puppetmasters; one does file serving, the other
does
>>> everything else.  We just started running 0.25 beta2 on both.  The
>>> file-server is running RubyEE + Passenger and the non-file-server
is
>>> running Mongrel.  When our heavy-hitter schedule runs the load
average
>>> on our file server spikes much higher than it did when it was
running
>>> 0.25 beta1 with Mongrel.
>>
>> Are you running the same # of mongrels as Passenger workers?
>
> Sort of :) We run 12 masters in mongrel.  Passenger doesn''t seem
have a
> "run at least this many puppetmasters" setting.  We set the max
to 12 to
> match mongrel but we never saw more than 6 masters running.  We bumped
> max to 24 late yesterday and I now see 11 masters running.  We''ll
see
> how it performs during today''s big schedule run.
You won''t see more passenger instances than the number of simultaneous
connections.


>
>> What is your Passenger config?
>
> We''re using what the docs suggest except MaxPoolSize is now 24:
>
http://github.com/reductivelabs/puppet/tree/607b01e82ea294068fdd554e59bc8e5fe3f9761a/ext/rack/files
>
>> What are the specs of your app server?
>
> All of our puppetmasters are running on the same size hardware.  The
> machines have 8 cores and 12G of RAM.
>
> Do you expect Passenger to perform better?
>
>
> >
>


-- 
Nigel Kersten
nigelk@google.com
System Administrator
Google, Inc.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Mark Plaksin wrote:
> Sort of :) We run 12 masters in mongrel.  Passenger doesn''t seem
have a
> "run at least this many puppetmasters" setting.  We set the max
to 12 to
> match mongrel but we never saw more than 6 masters running.  We bumped
> max to 24 late yesterday and I now see 11 masters running.  We''ll
see
> how it performs during today''s big schedule run.
> 
Yeah, PassengerMaxPoolSize is a bit confusing - they say it''s for
tuning
memory usage, but then say it is the max # of worker Rails processes 
it''ll spawn. I don''t care if you''ve got 12TB of ram,
if you''re only
running a single core you don''t want to set it very high.
> All of our puppetmasters are running on the same size hardware.  The
> machines have 8 cores and 12G of RAM.
> 
> Do you expect Passenger to perform better?
> 
Not necessarily. But load average isn''t exactly a great performance 
metric. BTW, what''s the load average getting up to? You''ve got
8 cores,
so something between 10-16 is probably fine, no?

-scott

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Scott Smith <scott@ohlol.net> writes:
>> All of our puppetmasters are running on the same size hardware.  The
>> machines have 8 cores and 12G of RAM.
>> 
>> Do you expect Passenger to perform better?
>
> Not necessarily. But load average isn''t exactly a great
performance
> metric. BTW, what''s the load average getting up to?
You''ve got 8 cores,
> so something between 10-16 is probably fine, no?
Load average is peaking around 10.  That''s not so good compared to
running under mongrel where the load peaked at less than 5 (both with
0.25b1 and 0.25b2).


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Jun 26, 3:05 pm, Mark Plaksin <ha...@usg.edu>
wrote:
> Scott Smith <sc...@ohlol.net> writes:
> > Mark Plaksin wrote:
> > Are you running the same # of mongrels as Passenger workers?
>
> Sort of :) We run 12 masters in mongrel.  Passenger doesn''t seem
have a
> "run at least this many puppetmasters" setting.  We set the max
to 12 to
> match mongrel but we never saw more than 6 masters running.  We bumped
> max to 24 late yesterday and I now see 11 masters running.  We''ll
see
> how it performs during today''s big schedule run.
If you only feed it high traffic once a day or so, you might want to
set PassengerPoolIdleTime much larger, say 172800. You probably would
only see an effect on the next day, though.

Christian

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Christian Hofstaedtler <ch+git@zeha.at> writes:
> On Jun 26, 3:05 pm, Mark Plaksin <ha...@usg.edu> wrote:
>
>> Scott Smith <sc...@ohlol.net> writes:
>> > Mark Plaksin wrote:
>> > Are you running the same # of mongrels as Passenger workers?
>>
>> Sort of :) We run 12 masters in mongrel.  Passenger doesn''t
seem have a
>> "run at least this many puppetmasters" setting.  We set the
max to 12 to
>> match mongrel but we never saw more than 6 masters running.  We bumped
>> max to 24 late yesterday and I now see 11 masters running.
 We''ll see
>> how it performs during today''s big schedule run.
>
> If you only feed it high traffic once a day or so, you might want to
> set PassengerPoolIdleTime much larger, say 172800. You probably would
> only see an effect on the next day, though.
Our servers don''t seem to get idle enough to get killed.  Once they
start they hang around until the next day (or until
PassengerMaxRequests, I guess).


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Mark Plaksin wrote:
> Load average is peaking around 10.  That''s not so good compared to
> running under mongrel where the load peaked at less than 5 (both with
> 0.25b1 and 0.25b2).
> 
Were the puppetmasters verifiably slower? If so, how did you measure it?

-scott

-- 
scott@ohlol.net

http://github.com/ohlol

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Scott Smith <scott@ohlol.net> writes:
> Mark Plaksin wrote:
>
>> Load average is peaking around 10.  That''s not so good
compared to
>> running under mongrel where the load peaked at less than 5 (both with
>> 0.25b1 and 0.25b2).
>
> Were the puppetmasters verifiably slower? If so, how did you measure it?
I didn''t measure but load average has been a good indicator for us. 
Oh,
and our stats system has stats based on Puppet reports.  The "how long
did the file part of all manifests on all machines take to run" graph
says that our file server is slower with Passenger than it is with
Mongrel.  The difference is about a factor of 2--just like the
difference in the peak load averages.


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Jun 26, 9:56 pm, Mark Plaksin <ha...@usg.edu>
wrote:
> Scott Smith <sc...@ohlol.net> writes:
> > Mark Plaksin wrote:
>
> >> Load average is peaking around 10.  That''s not so good
compared to
> >> running under mongrel where the load peaked at less than 5 (both
with
> >> 0.25b1 and 0.25b2).
>
> > Were the puppetmasters verifiably slower? If so, how did you measure
it?
>
> I didn''t measure but load average has been a good indicator for
us.  Oh,
> and our stats system has stats based on Puppet reports.  The "how long
> did the file part of all manifests on all machines take to run" graph
> says that our file server is slower with Passenger than it is with
> Mongrel.  The difference is about a factor of 2--just like the
> difference in the peak load averages.
What worries me is, that you are saying it takes twice as long and
your load is twice as high. So you''re actually seeing a 4-time worse
performance, which is /very/ bad.
Which processes do you see running? puppetmasterds and Apaches or only
one kind of them?

How long does file serving take for you? Are there very large files?
If so, you might want to test with PassengerUseGlobalQueue on.

Which versions are your clients? 0.25beta too?

Christian

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Christian Hofstaedtler <ch+git@zeha.at> writes:
> On Jun 26, 9:56 pm, Mark Plaksin <ha...@usg.edu> wrote:
>> Scott Smith <sc...@ohlol.net> writes:
>> > Mark Plaksin wrote:
>>
>> >> Load average is peaking around 10.  That''s not so
good compared to
>> >> running under mongrel where the load peaked at less than 5
(both with
>> >> 0.25b1 and 0.25b2).
>>
>> > Were the puppetmasters verifiably slower? If so, how did you
measure it?
>>
>> I didn''t measure but load average has been a good indicator
for us.  Oh,
>> and our stats system has stats based on Puppet reports.  The "how
long
>> did the file part of all manifests on all machines take to run"
graph
>> says that our file server is slower with Passenger than it is with
>> Mongrel.  The difference is about a factor of 2--just like the
>> difference in the peak load averages.
>
> What worries me is, that you are saying it takes twice as long and
> your load is twice as high. So you''re actually seeing a 4-time
worse
> performance, which is /very/ bad.
Heh.  Good point.
> Which processes do you see running? puppetmasterds and Apaches or only
> one kind of them?
By "running" do you mean "busy"?  I''m not positive
but I think
puppetmasterds were at the top of top when we were running Passenger.
We''re running Mongrel now so I can''t double-check.
> How long does file serving take for you? Are there very large files?
The vast majority are under a meg.  The top 15 are between .9M and 7M
and only get distributed to one machine.
> If so, you might want to test with PassengerUseGlobalQueue on.
>
> Which versions are your clients? 0.25beta too?
The clients are nearly all 0.24.7.  Once the file ignore bug (#2358) is
fixed we''ll try 0.25 on more clients.


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Christian Hofstaedtler wrote:
> 
> What worries me is, that you are saying it takes twice as long and
> your load is twice as high. So you''re actually seeing a 4-time
worse
> performance, which is /very/ bad.
> Which processes do you see running? puppetmasterds and Apaches or only
> one kind of them?
> 
Even then, I don''t think a load average of 10 on his system is
something
to worry about. A load average of 5 on his 8 core system simply means 
that he is not CPU bound. The CPU(s) are underutilized. Even at 10 I 
would wager that he still is not.

We need more data. What is the iowait? etc.

-scott

-- 
scott@ohlol.net

http://github.com/ohlol

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Scott Smith <scott@ohlol.net> writes:
> Christian Hofstaedtler wrote:
>
>> What worries me is, that you are saying it takes twice as long and
>> your load is twice as high. So you''re actually seeing a 4-time
worse
>> performance, which is /very/ bad.
>> Which processes do you see running? puppetmasterds and Apaches or only
>> one kind of them?
>
> Even then, I don''t think a load average of 10 on his system is
something
> to worry about. A load average of 5 on his 8 core system simply means 
> that he is not CPU bound. The CPU(s) are underutilized. Even at 10 I 
> would wager that he still is not.
>
> We need more data. What is the iowait? etc.
The machine almost never has iowait.  Spikes in user CPU match the load
average spikes (this is true with both Passenger and mongrel).  The
machine is doing nothing but Puppet file serving.  Its 12G of RAM is
never all used--it usually has about 8G completely free.  Buffer cache
is uses around 3G.



--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Jun 26, 11:19 pm, Scott Smith <sc...@ohlol.net>
wrote:
> Christian Hofstaedtler wrote:
>
> > What worries me is, that you are saying it takes twice as long and
> > your load is twice as high. So you''re actually seeing a
4-time worse
> > performance, which is /very/ bad.
> > Which processes do you see running? puppetmasterds and Apaches or only
> > one kind of them?
>
> Even then, I don''t think a load average of 10 on his system is
something
> to worry about. A load average of 5 on his 8 core system simply means
> that he is not CPU bound. The CPU(s) are underutilized. Even at 10 I
> would wager that he still is not.
It''s still a lot of worry if it''s 4-times worse. It will mean
less
clients per server when scaling up.

It''d be interesting to compare this setup with 0.24.8, as the
passenger-related integration code got completely rewritten for 0.25.

Christian

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

An update on this thread working with Mark at USG:

The example Apache config file with Puppet is not correct.  I have  
submitted a patch so the example configuration file is correct.

http://projects.reductivelabs.com/issues/2430

In communication with USG, once the options were properly set, the  
load is now much lower and similar with Mongrel but the Puppet compile  
times are much lower.

-L

--
Larry Ludwig
Reductive Labs


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---