After restarting our puppetmaster host, we are seeing high CPU load and a ton of queued passenger requests. I would suspect that our puppetmaster is falling victim to the thundering herd of puppet agents, but we have deployed most of our agents using a randomized cron job. The others are using ''splay = true'' if they are running puppet agent. Some info: Puppetmaster is RHEL6 x86_64 (VM) 4 vCPU 8GB memory Apache 2.2 Passenger 3.0.7 The vhost for our puppetmaster includes these passenger options: PassengerHighPerformance on PassengerUseGlobalQueue on PassengerPoolIdleTime 1500 PassengerMaxPoolSize 24 PassengerMaxRequests 4000 PassengerStatThrottleRate 120 PassengerSpawnMethod smart RackAutoDetect Off RailsAutoDetect Off Look at all of these queued requests! Passenger-status: ----------- General information ----------- max = 24 count = 24 active = 22 inactive = 2 Waiting on global queue: 234 ----------- Application groups ----------- /etc/puppet/rack: App root: /etc/puppet/rack * PID: 24840 Sessions: 1 Processed: 528 Uptime: 9h 41m 19s * PID: 24703 Sessions: 1 Processed: 513 Uptime: 9h 42m 17s * PID: 24642 Sessions: 1 Processed: 567 Uptime: 9h 42m 29s * PID: 24827 Sessions: 1 Processed: 572 Uptime: 9h 41m 28s * PID: 24869 Sessions: 1 Processed: 530 Uptime: 9h 41m 4s * PID: 24586 Sessions: 1 Processed: 572 Uptime: 9h 42m 32s * PID: 24715 Sessions: 1 Processed: 539 Uptime: 9h 42m 17s * PID: 29046 Sessions: 1 Processed: 487 Uptime: 8h 52m 35s * PID: 24609 Sessions: 1 Processed: 575 Uptime: 9h 42m 30s * PID: 24605 Sessions: 1 Processed: 488 Uptime: 9h 42m 30s * PID: 24831 Sessions: 1 Processed: 555 Uptime: 9h 41m 27s * PID: 29036 Sessions: 1 Processed: 549 Uptime: 8h 52m 35s * PID: 24583 Sessions: 1 Processed: 565 Uptime: 9h 42m 32s * PID: 24768 Sessions: 1 Processed: 521 Uptime: 9h 41m 56s * PID: 24850 Sessions: 1 Processed: 579 Uptime: 9h 41m 16s * PID: 29050 Sessions: 1 Processed: 497 Uptime: 8h 52m 35s * PID: 24567 Sessions: 1 Processed: 536 Uptime: 9h 42m 37s * PID: 24761 Sessions: 1 Processed: 570 Uptime: 9h 42m 5s * PID: 24709 Sessions: 1 Processed: 575 Uptime: 9h 42m 17s * PID: 29042 Sessions: 1 Processed: 488 Uptime: 8h 52m 35s * PID: 24865 Sessions: 1 Processed: 559 Uptime: 9h 41m 5s * PID: 29039 Sessions: 1 Processed: 506 Uptime: 8h 52m 35s /usr/share/foreman: App root: /usr/share/foreman * PID: 24724 Sessions: 0 Processed: 5743 Uptime: 9h 42m 16s * PID: 24717 Sessions: 0 Processed: 5099 Uptime: 9h 42m 17s -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/p1uXXzKJhCcJ. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Luke Baker
2012-Jul-31 21:17 UTC
[Puppet Users] Re: High CPU load / queued passenger requests
I ended up restart the mysql database on the storeconfigs server which helped this problem dramatically. But, I''m still open for more tuning options if they''re out there : ) On Tuesday, July 31, 2012 8:34:45 AM UTC-5, Luke Baker wrote:> > After restarting our puppetmaster host, we are seeing high CPU load and a > ton of queued passenger requests. I would suspect that our puppetmaster is > falling victim to the thundering herd of puppet agents, but we have > deployed most of our agents using a randomized cron job. The others are > using ''splay = true'' if they are running puppet agent. > > Some info: > Puppetmaster is RHEL6 x86_64 (VM) > 4 vCPU 8GB memory > Apache 2.2 > Passenger 3.0.7 > > The vhost for our puppetmaster includes these passenger options: > PassengerHighPerformance on > PassengerUseGlobalQueue on > PassengerPoolIdleTime 1500 > PassengerMaxPoolSize 24 > PassengerMaxRequests 4000 > PassengerStatThrottleRate 120 > PassengerSpawnMethod smart > RackAutoDetect Off > RailsAutoDetect Off > > > Look at all of these queued requests! > > Passenger-status: > ----------- General information ----------- > max = 24 > count = 24 > active = 22 > inactive = 2 > Waiting on global queue: 234 > > ----------- Application groups ----------- > /etc/puppet/rack: > App root: /etc/puppet/rack > * PID: 24840 Sessions: 1 Processed: 528 Uptime: 9h 41m 19s > * PID: 24703 Sessions: 1 Processed: 513 Uptime: 9h 42m 17s > * PID: 24642 Sessions: 1 Processed: 567 Uptime: 9h 42m 29s > * PID: 24827 Sessions: 1 Processed: 572 Uptime: 9h 41m 28s > * PID: 24869 Sessions: 1 Processed: 530 Uptime: 9h 41m 4s > * PID: 24586 Sessions: 1 Processed: 572 Uptime: 9h 42m 32s > * PID: 24715 Sessions: 1 Processed: 539 Uptime: 9h 42m 17s > * PID: 29046 Sessions: 1 Processed: 487 Uptime: 8h 52m 35s > * PID: 24609 Sessions: 1 Processed: 575 Uptime: 9h 42m 30s > * PID: 24605 Sessions: 1 Processed: 488 Uptime: 9h 42m 30s > * PID: 24831 Sessions: 1 Processed: 555 Uptime: 9h 41m 27s > * PID: 29036 Sessions: 1 Processed: 549 Uptime: 8h 52m 35s > * PID: 24583 Sessions: 1 Processed: 565 Uptime: 9h 42m 32s > * PID: 24768 Sessions: 1 Processed: 521 Uptime: 9h 41m 56s > * PID: 24850 Sessions: 1 Processed: 579 Uptime: 9h 41m 16s > * PID: 29050 Sessions: 1 Processed: 497 Uptime: 8h 52m 35s > * PID: 24567 Sessions: 1 Processed: 536 Uptime: 9h 42m 37s > * PID: 24761 Sessions: 1 Processed: 570 Uptime: 9h 42m 5s > * PID: 24709 Sessions: 1 Processed: 575 Uptime: 9h 42m 17s > * PID: 29042 Sessions: 1 Processed: 488 Uptime: 8h 52m 35s > * PID: 24865 Sessions: 1 Processed: 559 Uptime: 9h 41m 5s > * PID: 29039 Sessions: 1 Processed: 506 Uptime: 8h 52m 35s > > /usr/share/foreman: > App root: /usr/share/foreman > * PID: 24724 Sessions: 0 Processed: 5743 Uptime: 9h 42m 16s > * PID: 24717 Sessions: 0 Processed: 5099 Uptime: 9h 42m 17s > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/hNtnLgeBLzYJ. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Ramin K
2012-Aug-01 00:45 UTC
Re: [Puppet Users] Re: High CPU load / queued passenger requests
On 7/31/2012 2:17 PM, Luke Baker wrote:> I ended up restart the mysql database on the storeconfigs server which > helped this problem dramatically. But, I''m still open for more tuning > options if they''re out there : ) > > Some info: > Puppetmaster is RHEL6 x86_64 (VM) > 4 vCPU 8GB memory > Apache 2.2 > Passenger 3.0.7 >PassengerHighPerformance on PassengerMaxPoolSize 8 PassengerMinInstances 1 PassengerMaxRequests 10000 PassengerStatThrottleRate 120 PassengerPreStart https://puppet.mydomain.com:8140/ In Passenger 3.0.x GlobalQueue is on by default so you can drop that. MinInstances is new in 3.0.x. Using it obviates the need for timeout settings. Setting MinInstances beyond 1 unless done at the vhost level for specific reasons is a bad idea because Passenger can balance application instances better than you. Keep in mind that MinInstances does not start applications when your http daemon starts for the first time. PassengerPreStart does start applications when the server restarts and takes a URL as an argument which it uses in the host headers when it connects to localhost. This should also be applied at the vhost level. MaxRequests can be reasonably high with a mature app like Puppet. MaxPoolSize is a little harder to size. A Puppet master is usually CPU bound and I default to (cores * 2), but it depends on your apps. I''d aim for 80% of CPU with all Rack processes in use and add or drop from the poolsize to achieve it. Keep in mind that a Puppet Rack process will weigh in around 150-200MB and you should have enough RAM to support the max pool size. Rails/Rack autodetect shouldn''t be needed with a normal vhost for Puppet. I would not touch the Passenger spawn method at all unless there is a specific reason to modify it. There is not a reason to modify it with Puppet in my opinion. http://www.modrails.com/documentation/Users%20guide%20Apache.html#spawning_methods_explained Ramin -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Jake - USPS
2012-Aug-02 13:34 UTC
[Puppet Users] Re: High CPU load / queued passenger requests
If that seemed to help maybe you need to tune your mysql server(s)? I''ve tuned mine quite a bit from default OS distro settings using http://mysqltuner.pl/mysqltuner.pl as I myself am not a mysql expert. I''ve also separated mysql to a different server then what my puppetmasters run on. Regards, Jake -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/zo7mWV3ehGAJ. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On Thursday, August 2, 2012 8:34:55 AM UTC-5, Jake - USPS wrote:> > If that seemed to help maybe you need to tune your mysql server(s)? I''ve > tuned mine quite a bit from default OS distro settings using > http://mysqltuner.pl/mysqltuner.pl as I myself am not a mysql expert.Check out Sheeri K. Cabral''s fork here: https://launchpad.net/mysqltuner Regards, Adam -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/SVmfRSktCOwJ. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.