Puppet users - Apr 2009 - tweaking Passenger/Rack for performance.

so here are my unorganized thoughts on Passenger settings.

* PassengerMaxPoolSize

This depends a lot on two primary variables. a) How much RAM you have.
b) How much resident memory is needed for your environments/modules.
Only way to determine this is with tweaking, and you''ll need to be
much more conservative than the suggested settings on the Passenger
docs as your app takes up a lot more memory than your average site.


* PassengerStatThrottleRate 600
I haven''t noticed this make a *huge* difference, but logically we
don''t need to stat for the relevant files at all really.

* RailsSpawnMethod
has made no difference to me, but I''m not using storeconfigs.

* PassengerUseGlobalQueue
This feels like it should make a difference under heavy load.... but
doesn''t appear to.

* PassengerHighPerformance
Turning this on appears to reduce performance for me.

* PassengerMaxRequests
I''m trying to work out at the moment whether we''re leaking
memory at
all with 0.24.8, but even if we''re not, I can imagine this making a
big difference for certain usage patterns. I''m thinking of say having
a lot of different environments that all puppet regularly, but then an
overnight period where only one or two environments are being used. If
I understand how this works correctly, this should reduce memory
usage.

* PassengerPoolIdleTime
The default is 300 seconds. I''m yet to look into this setting.


-- 
Nigel Kersten
nigelk@google.com
System Administrator
Google, Inc.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Nigel Kersten wrote:
> so here are my unorganized thoughts on Passenger settings.
> 
...
> 
> * PassengerPoolIdleTime
> The default is 300 seconds. I''m yet to look into this setting.
> 
> 
I''ve done fairly extensive testing over the past few weeks with a Rails
app on
Passenger, and I''ve found the following settings to be very useful in
improving
responsiveness:

    PassengerPoolIdleTime 0
    RailsFrameworkSpawnerIdleTime 0
    RailsAppSpawnerIdleTime 0

By setting these to zero, no user experiences a ''hang'' while
the application
starts up (for the application I was tuning, the startup time could be 5-30 
seconds, so it was very noticeable and painful).  The downside of these is that 
memory utilization is not tightly managed (i.e., Passenger never idles out).

Nigel - if you would pass a pointer to the configs you''re testing,
I''d be happy
to help take a look as well.

Steven

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Thu, Apr 30, 2009 at 11:00 AM, Steven Jenkins <steven@endpoint.com>
wrote:
>
> Nigel Kersten wrote:
>> so here are my unorganized thoughts on Passenger settings.
>>
> ...
>>
>> * PassengerPoolIdleTime
>> The default is 300 seconds. I''m yet to look into this setting.
>>
>>
>
> I''ve done fairly extensive testing over the past few weeks with a
Rails app on
> Passenger, and I''ve found the following settings to be very useful
in improving
> responsiveness:
>
>    PassengerPoolIdleTime 0
>    RailsFrameworkSpawnerIdleTime 0
>    RailsAppSpawnerIdleTime 0
>
> By setting these to zero, no user experiences a ''hang''
while the application
> starts up (for the application I was tuning, the startup time could be 5-30
> seconds, so it was very noticeable and painful).  The downside of these is
that
> memory utilization is not tightly managed (i.e., Passenger never idles
out).
So memory usage is one of our biggest problems with Puppet at the
moment, and application startup time doesn''t really affect us much.

With 2GB of ram on two VCPUs, I''m finding that a pool size of about 12
is optimal for our configuration.

That''s with 35 environments, each with about:

45 modules
350 files
200 .pp files (very close to 1:1 mapping of .pp -> class)
around 1000 resources defined, clients getting ~ 300 resource on average.


>
> Nigel - if you would pass a pointer to the configs you''re testing,
I''d be happy
> to help take a look as well.
>
> Steven
>
> >
>


-- 
Nigel Kersten
nigelk@google.com
System Administrator
Google, Inc.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Nigel Kersten wrote:
> On Thu, Apr 30, 2009 at 11:00 AM, Steven Jenkins
<steven@endpoint.com> wrote:
>> Nigel Kersten wrote:
>>> so here are my unorganized thoughts on Passenger settings.
>>>
>> ...
>>> * PassengerPoolIdleTime
>>> The default is 300 seconds. I''m yet to look into this
setting.
>>>
>>>
>> I''ve done fairly extensive testing over the past few weeks
with a Rails app on
>> Passenger, and I''ve found the following settings to be very
useful in improving
>> responsiveness:
>>
>>    PassengerPoolIdleTime 0
>>    RailsFrameworkSpawnerIdleTime 0
>>    RailsAppSpawnerIdleTime 0
>>
>> By setting these to zero, no user experiences a
''hang'' while the application
>> starts up (for the application I was tuning, the startup time could be
5-30
>> seconds, so it was very noticeable and painful).  The downside of these
is that
>> memory utilization is not tightly managed (i.e., Passenger never idles
out).
> 
> So memory usage is one of our biggest problems with Puppet at the
> moment, and application startup time doesn''t really affect us
much.
> 
> With 2GB of ram on two VCPUs, I''m finding that a pool size of
about 12
> is optimal for our configuration.
> 
> That''s with 35 environments, each with about:
> 
> 45 modules
> 350 files
> 200 .pp files (very close to 1:1 mapping of .pp -> class)
> around 1000 resources defined, clients getting ~ 300 resource on average.
> 
Given that, I agree.  You probably want to tune PassengerMaxRequests so that you
can keep memory usage better under control.  I''m thinking mostly of
StoreConfigs
here.

Steven

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Apr 30, 7:20 pm, Nigel Kersten <nig...@google.com>
wrote:
> so here are my unorganized thoughts on Passenger settings.
>
> * PassengerMaxPoolSize
>
> This depends a lot on two primary variables. a) How much RAM you have.
> b) How much resident memory is needed for your environments/modules.
> Only way to determine this is with tweaking, and you''ll need to be
> much more conservative than the suggested settings on the Passenger
> docs as your app takes up a lot more memory than your average site.
Sure; the puppetmaster processes are quite heavy.
Right now I''m running with a MaxPoolSize of 10, fitting nicely into
the 2.5GB of reserved memory.
> * PassengerStatThrottleRate 600
> I haven''t noticed this make a *huge* difference, but logically we
> don''t need to stat for the relevant files at all really.
I think it''s best to actually disable RackAutoDetect &
RailsAutoDetect, but I''m not sure if disabling them will avoid the
stats.
> * PassengerHighPerformance
> Turning this on appears to reduce performance for me.
Interesting. From the docs I''m not seeing how this could decrease
performance. Anyway, hosting puppetmaster doesn''t need any rewrites or
something, so turning it on/off shouldn''t hurt operations.
> * PassengerMaxRequests
> I''m trying to work out at the moment whether we''re
leaking memory at
> all with 0.24.8, but even if we''re not, I can imagine this making
a
> big difference for certain usage patterns. I''m thinking of say
having
> a lot of different environments that all puppet regularly, but then an
> overnight period where only one or two environments are being used. If
> I understand how this works correctly, this should reduce memory
> usage.
This probably depends heavily on your environment. On my setup the
puppetmaster processes stay between 200-300M VIRT, and don''t grow
larger, so I haven''t looked at this at all.
> * PassengerPoolIdleTime
> The default is 300 seconds. I''m yet to look into this setting.
I''m running with the default here, but still get long lived
puppetmaster processes. Probably they never get idle.
> * RailsSpawnMethod
> * PassengerUseGlobalQueue
Haven''t touched these at all.

Thanks for your feedback!

Christian

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Have you tried the 0.25 beta  with Passenger?

For the life of me I can''t get it to work on dapper or hardy.

[ pid=17005 file=ext/common/ApplicationPoolServerExecutable.cpp:307
time=2009-05-04 14:17:27.451 ]:
  Client 0x667060: SpawnException occured (with error page)


and am yet to obtain useful debugging anywhere....



On Mon, May 4, 2009 at 2:07 PM,  <ch+git@zeha.at>
wrote:
>
> On Apr 30, 7:20 pm, Nigel Kersten <nig...@google.com> wrote:
>> so here are my unorganized thoughts on Passenger settings.
>>
>> * PassengerMaxPoolSize
>>
>> This depends a lot on two primary variables. a) How much RAM you have.
>> b) How much resident memory is needed for your environments/modules.
>> Only way to determine this is with tweaking, and you''ll need
to be
>> much more conservative than the suggested settings on the Passenger
>> docs as your app takes up a lot more memory than your average site.
>
> Sure; the puppetmaster processes are quite heavy.
> Right now I''m running with a MaxPoolSize of 10, fitting nicely
into
> the 2.5GB of reserved memory.
>
>> * PassengerStatThrottleRate 600
>> I haven''t noticed this make a *huge* difference, but logically
we
>> don''t need to stat for the relevant files at all really.
>
> I think it''s best to actually disable RackAutoDetect &
> RailsAutoDetect, but I''m not sure if disabling them will avoid the
> stats.
>
>> * PassengerHighPerformance
>> Turning this on appears to reduce performance for me.
>
> Interesting. From the docs I''m not seeing how this could decrease
> performance. Anyway, hosting puppetmaster doesn''t need any
rewrites or
> something, so turning it on/off shouldn''t hurt operations.
>
>> * PassengerMaxRequests
>> I''m trying to work out at the moment whether we''re
leaking memory at
>> all with 0.24.8, but even if we''re not, I can imagine this
making a
>> big difference for certain usage patterns. I''m thinking of say
having
>> a lot of different environments that all puppet regularly, but then an
>> overnight period where only one or two environments are being used. If
>> I understand how this works correctly, this should reduce memory
>> usage.
>
> This probably depends heavily on your environment. On my setup the
> puppetmaster processes stay between 200-300M VIRT, and don''t grow
> larger, so I haven''t looked at this at all.
>
>> * PassengerPoolIdleTime
>> The default is 300 seconds. I''m yet to look into this setting.
>
> I''m running with the default here, but still get long lived
> puppetmaster processes. Probably they never get idle.
>
>> * RailsSpawnMethod
>> * PassengerUseGlobalQueue
>
> Haven''t touched these at all.
>
> Thanks for your feedback!
>
> Christian
>
> >
>


-- 
Nigel Kersten
nigelk@google.com
System Administrator
Google, Inc.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On May 4, 11:20 pm, Nigel Kersten <nig...@google.com>
wrote:
> Have you tried the 0.25 beta  with Passenger?
Passenger support has been completely redone for 0.25, check ext/rack
in the sources. If upgrading from 0.24.x, you at least need to update
the config.ru file.
> For the life of me I can''t get it to work on dapper or hardy.
>
> [ pid=17005 file=ext/common/ApplicationPoolServerExecutable.cpp:307
> time=2009-05-04 14:17:27.451 ]:
>   Client 0x667060: SpawnException occured (with error page)
>
> and am yet to obtain useful debugging anywhere....
You should be able to get some - maybe useful - output by pointing a
browser to https://puppetmaster:8140/
It probably gives a colorful Passenger error page in your case.

Christian

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Mon, May 4, 2009 at 2:48 PM,  <ch+git@zeha.at>
wrote:
>
>
>
> On May 4, 11:20 pm, Nigel Kersten <nig...@google.com> wrote:
>> Have you tried the 0.25 beta  with Passenger?
>
> Passenger support has been completely redone for 0.25, check ext/rack
> in the sources. If upgrading from 0.24.x, you at least need to update
> the config.ru file.
Yep. I really like the way it''s been distributed as a puppet manifest.
++
>
>> For the life of me I can''t get it to work on dapper or hardy.
>>
>> [ pid=17005 file=ext/common/ApplicationPoolServerExecutable.cpp:307
>> time=2009-05-04 14:17:27.451 ]:
>>   Client 0x667060: SpawnException occured (with error page)
>>
>> and am yet to obtain useful debugging anywhere....
>
> You should be able to get some - maybe useful - output by pointing a
> browser to https://puppetmaster:8140/
> It probably gives a colorful Passenger error page in your case.

ahah! now why couldn''t I get that to actually log somewhere useful...

ta.




-- 
Nigel Kersten
nigelk@google.com
System Administrator
Google, Inc.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Mon, May 4, 2009 at 2:07 PM,  <ch+git@zeha.at>
wrote:
>
> On Apr 30, 7:20 pm, Nigel Kersten <nig...@google.com> wrote:
>> so here are my unorganized thoughts on Passenger settings.
>>
>> * PassengerMaxPoolSize
>>
>> This depends a lot on two primary variables. a) How much RAM you have.
>> b) How much resident memory is needed for your environments/modules.
>> Only way to determine this is with tweaking, and you''ll need
to be
>> much more conservative than the suggested settings on the Passenger
>> docs as your app takes up a lot more memory than your average site.
>
> Sure; the puppetmaster processes are quite heavy.
> Right now I''m running with a MaxPoolSize of 10, fitting nicely
into
> the 2.5GB of reserved memory.
I''m finding a similar value works quite nicely in conjunction with
setting

PassengerMaxRequests 3000

Although I''m running with the defaults for idle time of 300 seconds
anyway, I find that on overnight puppet runs (where we have one
environment predominantly puppeting, but a few miscellaneous other
environments also checking in less frequently), this has a significant
impact upon memory usage.

Without the max requests setting (which for us is about the equivalent
of 20 puppet runs for that app instance), memory usage continues to
climb.

I''m not actually seeing memory leaks as such with 0.24.8. What I am
seeing is that there is a large memory hit for an app instance loading
a new environment, and that this doesn''t necessarily get reclaimed
quickly enough once that environment is no longer actively used by the
puppet server.

With that setting on, even if all 10 pool members are busy (man I love
having passenger-status and apachetop with Puppet now), after a member
has fulfilled about 20 client runs, it gets automatically restarted,
instantly reducing memory usage.

As my puppet servers are all virtualized, there''s a particularly
noticeable hit if I start to swap which has a snowball effect. The
pool members start to hit swap and load increases as they''re unable to
fulfill requests as quickly, which increases memory usage, etc etc.

Obviously picking a number for MaxRequests is going to depend upon
your server, the available memory, the number of members in the pool,
the requests per client run, etc etc etc.

the ''passenger-status'' tool is incredibly useful here, as it
lets you
peek nicely into the current state of the pool.
>
>> * PassengerStatThrottleRate 600
>> I haven''t noticed this make a *huge* difference, but logically
we
>> don''t need to stat for the relevant files at all really.
>
> I think it''s best to actually disable RackAutoDetect &
> RailsAutoDetect, but I''m not sure if disabling them will avoid the
> stats.
It doesn''t appear to, but the stat throttle rate only appears to make
a significant difference if you''re already swapped, which from my
experience means you''re in trouble already.
>
>> * PassengerHighPerformance
>> Turning this on appears to reduce performance for me.
>
> Interesting. From the docs I''m not seeing how this could decrease
> performance. Anyway, hosting puppetmaster doesn''t need any
rewrites or
> something, so turning it on/off shouldn''t hurt operations.
I''ve been really surprised by this, but using my own stress testing
scripts and the puppet-test executable confirms this absolutely.
Turning on high performance is about 2-5% slower.
>
>> * PassengerMaxRequests
>> I''m trying to work out at the moment whether we''re
leaking memory at
>> all with 0.24.8, but even if we''re not, I can imagine this
making a
>> big difference for certain usage patterns. I''m thinking of say
having
>> a lot of different environments that all puppet regularly, but then an
>> overnight period where only one or two environments are being used. If
>> I understand how this works correctly, this should reduce memory
>> usage.
>
> This probably depends heavily on your environment. On my setup the
> puppetmaster processes stay between 200-300M VIRT, and don''t grow
> larger, so I haven''t looked at this at all.
>
>> * PassengerPoolIdleTime
>> The default is 300 seconds. I''m yet to look into this setting.
>
> I''m running with the default here, but still get long lived
> puppetmaster processes. Probably they never get idle.
>
>> * RailsSpawnMethod
>> * PassengerUseGlobalQueue
>
> Haven''t touched these at all.
>
> Thanks for your feedback!
>
> Christian
>
> >
>


-- 
Nigel Kersten
nigelk@google.com
System Administrator
Google, Inc.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Nigel Kersten wrote:
> On Mon, May 4, 2009 at 2:07 PM,  <ch+git@zeha.at> wrote:
>> On Apr 30, 7:20 pm, Nigel Kersten <nig...@google.com> wrote:
>>> so here are my unorganized thoughts on Passenger settings.
>>>
>>> * PassengerMaxPoolSize
>>>
>>> This depends a lot on two primary variables. a) How much RAM you
have.
>>> b) How much resident memory is needed for your
environments/modules.
>>> Only way to determine this is with tweaking, and you''ll
need to be
>>> much more conservative than the suggested settings on the Passenger
>>> docs as your app takes up a lot more memory than your average site.
>> Sure; the puppetmaster processes are quite heavy.
>> Right now I''m running with a MaxPoolSize of 10, fitting nicely
into
>> the 2.5GB of reserved memory.
> 
> I''m finding a similar value works quite nicely in conjunction with
setting
> 
> PassengerMaxRequests 3000
> 
> Although I''m running with the defaults for idle time of 300
seconds
> anyway, I find that on overnight puppet runs (where we have one
> environment predominantly puppeting, but a few miscellaneous other
> environments also checking in less frequently), this has a significant
> impact upon memory usage.
> 
> Without the max requests setting (which for us is about the equivalent
> of 20 puppet runs for that app instance), memory usage continues to
> climb.
> 
> I''m not actually seeing memory leaks as such with 0.24.8. What I
am
> seeing is that there is a large memory hit for an app instance loading
> a new environment, and that this doesn''t necessarily get reclaimed
> quickly enough once that environment is no longer actively used by the
> puppet server.
You can tune the PassengerPoolIdleTime to a smaller value than the default 300 
seconds to make it reclaim  memory faster.  Have you experimented any with that?

Steven

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Tue, May 5, 2009 at 7:20 AM, Steven Jenkins <steven@endpoint.com>
wrote:
>
> Nigel Kersten wrote:
>> On Mon, May 4, 2009 at 2:07 PM,  <ch+git@zeha.at> wrote:
>>> On Apr 30, 7:20 pm, Nigel Kersten <nig...@google.com> wrote:
>>>> so here are my unorganized thoughts on Passenger settings.
>>>>
>>>> * PassengerMaxPoolSize
>>>>
>>>> This depends a lot on two primary variables. a) How much RAM
you have.
>>>> b) How much resident memory is needed for your
environments/modules.
>>>> Only way to determine this is with tweaking, and
you''ll need to be
>>>> much more conservative than the suggested settings on the
Passenger
>>>> docs as your app takes up a lot more memory than your average
site.
>>> Sure; the puppetmaster processes are quite heavy.
>>> Right now I''m running with a MaxPoolSize of 10, fitting
nicely into
>>> the 2.5GB of reserved memory.
>>
>> I''m finding a similar value works quite nicely in conjunction
with setting
>>
>> PassengerMaxRequests 3000
>>
>> Although I''m running with the defaults for idle time of 300
seconds
>> anyway, I find that on overnight puppet runs (where we have one
>> environment predominantly puppeting, but a few miscellaneous other
>> environments also checking in less frequently), this has a significant
>> impact upon memory usage.
>>
>> Without the max requests setting (which for us is about the equivalent
>> of 20 puppet runs for that app instance), memory usage continues to
>> climb.
>>
>> I''m not actually seeing memory leaks as such with 0.24.8. What
I am
>> seeing is that there is a large memory hit for an app instance loading
>> a new environment, and that this doesn''t necessarily get
reclaimed
>> quickly enough once that environment is no longer actively used by the
>> puppet server.
>
> You can tune the PassengerPoolIdleTime to a smaller value than the default
300
> seconds to make it reclaim  memory faster.  Have you experimented any with
that?
Overnight my servers are never ever idle. :)

I brought it as low as 10 seconds during heavy periods with almost zero effect.

I could do with adding more resources to the servers, but I''m trying
to be parsimonious here, and capping max requests does a decent enough
job so far.


-- 
Nigel Kersten
nigelk@google.com
System Administrator
Google, Inc.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---