I am finding that the puppetlabs-apache module is somehow adding 30-60
seconds onto a host''s catalog compile time when the puppetmaster has no
other hosts contacting or generating catalogs. The Puppetmaster is setup
to use Puppet-2.7.18 - Apache & Passenger. RIght now only 2 hosts are even
configured to use this new PM, the PM itself and a Foreman host. With
neither hosts contacting the PM, and disabling the "include apache"
and
"require apache::mod::dev" in puppetlabs-passenger, I get "Config
retrieval: 2.45".
Catalogs had hostnames replaced, puppetmaster.tld is the CNAME for Puppet
master and puppetmaster-host.tld is the actual hostname.
Catalog: http://pastebin.com/V29JmB8J
# puppet agent --{summarize,test,debug,evaltrace,noop} | perl -pe
''s/^/localtime().": "/e''
<snip>
Sun Dec 9 13:05:35 2012: notice: Finished catalog run in 2.43 seconds
Sun Dec 9 13:05:35 2012: Changes:
Sun Dec 9 13:05:35 2012: Events:
Sun Dec 9 13:05:35 2012: Total: 1
Sun Dec 9 13:05:35 2012: Noop: 1
Sun Dec 9 13:05:35 2012: Resources:
Sun Dec 9 13:05:35 2012: Out of sync: 1
Sun Dec 9 13:05:35 2012: Total: 22
Sun Dec 9 13:05:35 2012: Skipped: 6
Sun Dec 9 13:05:35 2012: Time:
Sun Dec 9 13:05:35 2012: Resources: 0.00
Sun Dec 9 13:05:35 2012: Filebucket: 0.00
Sun Dec 9 13:05:35 2012: Package: 0.00
Sun Dec 9 13:05:35 2012: File: 0.00
Sun Dec 9 13:05:35 2012: Firewall: 0.01
Sun Dec 9 13:05:35 2012: Exec: 0.14
Sun Dec 9 13:05:35 2012: Last run: 1355079935
Sun Dec 9 13:05:35 2012: Config retrieval: 2.45
Sun Dec 9 13:05:35 2012: Total: 2.61
Sun Dec 9 13:05:35 2012: Version:
Sun Dec 9 13:05:35 2012: Config: 1355077701
Sun Dec 9 13:05:35 2012: Puppet: 2.7.18
Uncommenting "include apache" and "require apache::mod::dev"
in
puppetlabs-passenger about 40 seconds are added to the compile time. I
went through disabled all modules and parameters, and found that enabling
the passenger module alone with Apache included caused the huge delay.
Catalog: http://pastebin.com/wwcKgX5b
# puppet agent --{summarize,test,debug,evaltrace,noop} | perl -pe
''s/^/localtime().": "/e''
<snip>
Sun Dec 9 13:13:08 2012: notice: Finished catalog run in 2.23 seconds
Sun Dec 9 13:13:08 2012: Changes:
Sun Dec 9 13:13:08 2012: Events:
Sun Dec 9 13:13:08 2012: Total: 4
Sun Dec 9 13:13:08 2012: Noop: 4
Sun Dec 9 13:13:08 2012: Resources:
Sun Dec 9 13:13:08 2012: Out of sync: 4
Sun Dec 9 13:13:08 2012: Skipped: 6
Sun Dec 9 13:13:08 2012: Total: 80
Sun Dec 9 13:13:08 2012: Time:
Sun Dec 9 13:13:08 2012: Filebucket: 0.00
Sun Dec 9 13:13:08 2012: Resources: 0.00
Sun Dec 9 13:13:08 2012: Package: 0.00
Sun Dec 9 13:13:08 2012: Firewall: 0.01
Sun Dec 9 13:13:08 2012: File: 0.01
Sun Dec 9 13:13:08 2012: A2mod: 0.03
Sun Dec 9 13:13:08 2012: Service: 0.07
Sun Dec 9 13:13:08 2012: Exec: 0.14
Sun Dec 9 13:13:08 2012: Last run: 1355080388
Sun Dec 9 13:13:08 2012: Config retrieval: 42.79
Sun Dec 9 13:13:08 2012: Total: 43.06
Sun Dec 9 13:13:08 2012: Version:
Sun Dec 9 13:13:08 2012: Config: 1355080261
Sun Dec 9 13:13:08 2012: Puppet: 2.7.18
The install is via RPM. I''m using the Fedora SRPMs rebuilt for CentOS
6.
The Puppet master has two environments, development and production with
this as the puppet.conf
# cat /etc/puppet/puppet.conf
---
[main]
server = puppetmaster.tld
logdir = /var/log/puppet
vardir = /var/lib/puppet
ssldir = /var/lib/puppet/ssl
rundir = /var/run/puppet
factpath = $vardir/lib/facter
pluginsync = true
templatedir = $confdir/templates
[agent]
report = true
environment = production
preferred_serialization_format = yaml
[master]
modulepath =
/etc/puppet/environments/$environment/modules:/etc/puppet/modules
manifest = /etc/puppet/manifests/site.pp
environment = production
autosign = false
ssl_client_header = SSL_CLIENT_S_DN
ssl_client_verify_header = SSL_CLIENT_VERIFY
certname = puppetmaster.tld
dns_alt_names = puppetmaster.tld,puppetmaster-host.tld
report = true
reports = store
#reports = store, foreman
[production]
manifest = /etc/puppet/manifests/site.pp
Has anyone run into this issue with puppetlabs-apache or know why my
compile times are so long? I have a Puppet 2.6.17 server (Also
Apache+Passenger) that has one host, for example, with 524 resources , and
only a 21.36 second catalog compile time.
Thanks
- Trey
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/sfkaMi0uij0J.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
treydock
2012-Dec-15 18:23 UTC
[Puppet Users] Re: Catalog compile times > 40s - Puppet 2.7.18
On Sunday, December 9, 2012 1:41:34 PM UTC-6, treydock wrote:> > I am finding that the puppetlabs-apache module is somehow adding 30-60 > seconds onto a host''s catalog compile time when the puppetmaster has no > other hosts contacting or generating catalogs. The Puppetmaster is setup > to use Puppet-2.7.18 - Apache & Passenger. RIght now only 2 hosts are even > configured to use this new PM, the PM itself and a Foreman host. With > neither hosts contacting the PM, and disabling the "include apache" and > "require apache::mod::dev" in puppetlabs-passenger, I get "Config > retrieval: 2.45". > > Catalogs had hostnames replaced, puppetmaster.tld is the CNAME for Puppet > master and puppetmaster-host.tld is the actual hostname. > > Catalog: http://pastebin.com/V29JmB8J > # puppet agent --{summarize,test,debug,evaltrace,noop} | perl -pe > ''s/^/localtime().": "/e'' > <snip> > Sun Dec 9 13:05:35 2012: notice: Finished catalog run in 2.43 seconds > Sun Dec 9 13:05:35 2012: Changes: > Sun Dec 9 13:05:35 2012: Events: > Sun Dec 9 13:05:35 2012: Total: 1 > Sun Dec 9 13:05:35 2012: Noop: 1 > Sun Dec 9 13:05:35 2012: Resources: > Sun Dec 9 13:05:35 2012: Out of sync: 1 > Sun Dec 9 13:05:35 2012: Total: 22 > Sun Dec 9 13:05:35 2012: Skipped: 6 > Sun Dec 9 13:05:35 2012: Time: > Sun Dec 9 13:05:35 2012: Resources: 0.00 > Sun Dec 9 13:05:35 2012: Filebucket: 0.00 > Sun Dec 9 13:05:35 2012: Package: 0.00 > Sun Dec 9 13:05:35 2012: File: 0.00 > Sun Dec 9 13:05:35 2012: Firewall: 0.01 > Sun Dec 9 13:05:35 2012: Exec: 0.14 > Sun Dec 9 13:05:35 2012: Last run: 1355079935 > Sun Dec 9 13:05:35 2012: Config retrieval: 2.45 > Sun Dec 9 13:05:35 2012: Total: 2.61 > Sun Dec 9 13:05:35 2012: Version: > Sun Dec 9 13:05:35 2012: Config: 1355077701 > Sun Dec 9 13:05:35 2012: Puppet: 2.7.18 > > Uncommenting "include apache" and "require apache::mod::dev" in > puppetlabs-passenger about 40 seconds are added to the compile time. I > went through disabled all modules and parameters, and found that enabling > the passenger module alone with Apache included caused the huge delay. > > Catalog: http://pastebin.com/wwcKgX5b > # puppet agent --{summarize,test,debug,evaltrace,noop} | perl -pe > ''s/^/localtime().": "/e'' > <snip> > Sun Dec 9 13:13:08 2012: notice: Finished catalog run in 2.23 seconds > Sun Dec 9 13:13:08 2012: Changes: > Sun Dec 9 13:13:08 2012: Events: > Sun Dec 9 13:13:08 2012: Total: 4 > Sun Dec 9 13:13:08 2012: Noop: 4 > Sun Dec 9 13:13:08 2012: Resources: > Sun Dec 9 13:13:08 2012: Out of sync: 4 > Sun Dec 9 13:13:08 2012: Skipped: 6 > Sun Dec 9 13:13:08 2012: Total: 80 > Sun Dec 9 13:13:08 2012: Time: > Sun Dec 9 13:13:08 2012: Filebucket: 0.00 > Sun Dec 9 13:13:08 2012: Resources: 0.00 > Sun Dec 9 13:13:08 2012: Package: 0.00 > Sun Dec 9 13:13:08 2012: Firewall: 0.01 > Sun Dec 9 13:13:08 2012: File: 0.01 > Sun Dec 9 13:13:08 2012: A2mod: 0.03 > Sun Dec 9 13:13:08 2012: Service: 0.07 > Sun Dec 9 13:13:08 2012: Exec: 0.14 > Sun Dec 9 13:13:08 2012: Last run: 1355080388 > Sun Dec 9 13:13:08 2012: Config retrieval: 42.79 > Sun Dec 9 13:13:08 2012: Total: 43.06 > Sun Dec 9 13:13:08 2012: Version: > Sun Dec 9 13:13:08 2012: Config: 1355080261 > Sun Dec 9 13:13:08 2012: Puppet: 2.7.18 > > > The install is via RPM. I''m using the Fedora SRPMs rebuilt for CentOS 6. > The Puppet master has two environments, development and production with > this as the puppet.conf > > # cat /etc/puppet/puppet.conf > --- > [main] > server = puppetmaster.tld > logdir = /var/log/puppet > vardir = /var/lib/puppet > ssldir = /var/lib/puppet/ssl > rundir = /var/run/puppet > factpath = $vardir/lib/facter > pluginsync = true > templatedir = $confdir/templates > > > [agent] > report = true > environment = production > preferred_serialization_format = yaml > > [master] > modulepath = > /etc/puppet/environments/$environment/modules:/etc/puppet/modules > manifest = /etc/puppet/manifests/site.pp > environment = production > autosign = false > ssl_client_header = SSL_CLIENT_S_DN > ssl_client_verify_header = SSL_CLIENT_VERIFY > certname = puppetmaster.tld > dns_alt_names = puppetmaster.tld,puppetmaster-host.tld > report = true > reports = store > #reports = store, foreman > > [production] > manifest = /etc/puppet/manifests/site.pp > > > Has anyone run into this issue with puppetlabs-apache or know why my > compile times are so long? I have a Puppet 2.6.17 server (Also > Apache+Passenger) that has one host, for example, with 524 resources , and > only a 21.36 second catalog compile time. > > Thanks > - Trey >Continuing to debug this issue it seems the apache::mod::default is the bulk of the compile time. I''m curious if this is a symptom of the module itself or something with 2.7.x. I moved this puppet master to the EPEL 2.6.17 RPMs and now catalogs that previously took 40-60 seconds are now down to 1-2 seconds. I noticed that during the long compile times this VM''s (2 vCPU , 2GB RAM) CPU usage is at 100% from a single Ruby process that links to the PM Rack process. My other PM running same exact setup with about 50 managed hosts never has had performance issues always running 2.6.17. I''d like to move to 2.7.x for functionality of more modern modules but this PM is for an HPC and I''m already struggling to convince my predecessors that Puppetizing the HPC is not going impact performance. Is there possibly a specific version of 2.7.x or maybe even 3.0.x that may address the performance issues I''m seeing ? Thanks - Trey -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/cjOWDUiSrq4J. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
shurik
2012-Dec-19 16:30 UTC
[Puppet Users] Re: Catalog compile times > 40s - Puppet 2.7.18
Hi We are tried to update puppet master from 2.6.6 to 2.7.18 and saw similar picture. Catalog compile time took ~60-120 seconds. Before this update compilation took about ~1-5 second. After upgrade average CPU usage increased from 10-20% up to 20-40%, during some activity (compilation?) ruby process puppetmasterd take 100% too. All, is there are some magic to solve or this is expected behaviour? Thanks! --- Alexander On Saturday, December 15, 2012 10:23:49 PM UTC+4, treydock wrote:> > > > On Sunday, December 9, 2012 1:41:34 PM UTC-6, treydock wrote: >> >> I am finding that the puppetlabs-apache module is somehow adding 30-60 >> seconds onto a host''s catalog compile time when the puppetmaster has no >> other hosts contacting or generating catalogs. The Puppetmaster is setup >> to use Puppet-2.7.18 - Apache & Passenger. RIght now only 2 hosts are even >> configured to use this new PM, the PM itself and a Foreman host. With >> neither hosts contacting the PM, and disabling the "include apache" and >> "require apache::mod::dev" in puppetlabs-passenger, I get "Config >> retrieval: 2.45". >> >> Catalogs had hostnames replaced, puppetmaster.tld is the CNAME for Puppet >> master and puppetmaster-host.tld is the actual hostname. >> >> Catalog: http://pastebin.com/V29JmB8J >> # puppet agent --{summarize,test,debug,evaltrace,noop} | perl -pe >> ''s/^/localtime().": "/e'' >> <snip> >> Sun Dec 9 13:05:35 2012: notice: Finished catalog run in 2.43 seconds >> Sun Dec 9 13:05:35 2012: Changes: >> Sun Dec 9 13:05:35 2012: Events: >> Sun Dec 9 13:05:35 2012: Total: 1 >> Sun Dec 9 13:05:35 2012: Noop: 1 >> Sun Dec 9 13:05:35 2012: Resources: >> Sun Dec 9 13:05:35 2012: Out of sync: 1 >> Sun Dec 9 13:05:35 2012: Total: 22 >> Sun Dec 9 13:05:35 2012: Skipped: 6 >> Sun Dec 9 13:05:35 2012: Time: >> Sun Dec 9 13:05:35 2012: Resources: 0.00 >> Sun Dec 9 13:05:35 2012: Filebucket: 0.00 >> Sun Dec 9 13:05:35 2012: Package: 0.00 >> Sun Dec 9 13:05:35 2012: File: 0.00 >> Sun Dec 9 13:05:35 2012: Firewall: 0.01 >> Sun Dec 9 13:05:35 2012: Exec: 0.14 >> Sun Dec 9 13:05:35 2012: Last run: 1355079935 >> Sun Dec 9 13:05:35 2012: Config retrieval: 2.45 >> Sun Dec 9 13:05:35 2012: Total: 2.61 >> Sun Dec 9 13:05:35 2012: Version: >> Sun Dec 9 13:05:35 2012: Config: 1355077701 >> Sun Dec 9 13:05:35 2012: Puppet: 2.7.18 >> >> Uncommenting "include apache" and "require apache::mod::dev" in >> puppetlabs-passenger about 40 seconds are added to the compile time. I >> went through disabled all modules and parameters, and found that enabling >> the passenger module alone with Apache included caused the huge delay. >> >> Catalog: http://pastebin.com/wwcKgX5b >> # puppet agent --{summarize,test,debug,evaltrace,noop} | perl -pe >> ''s/^/localtime().": "/e'' >> <snip> >> Sun Dec 9 13:13:08 2012: notice: Finished catalog run in 2.23 seconds >> Sun Dec 9 13:13:08 2012: Changes: >> Sun Dec 9 13:13:08 2012: Events: >> Sun Dec 9 13:13:08 2012: Total: 4 >> Sun Dec 9 13:13:08 2012: Noop: 4 >> Sun Dec 9 13:13:08 2012: Resources: >> Sun Dec 9 13:13:08 2012: Out of sync: 4 >> Sun Dec 9 13:13:08 2012: Skipped: 6 >> Sun Dec 9 13:13:08 2012: Total: 80 >> Sun Dec 9 13:13:08 2012: Time: >> Sun Dec 9 13:13:08 2012: Filebucket: 0.00 >> Sun Dec 9 13:13:08 2012: Resources: 0.00 >> Sun Dec 9 13:13:08 2012: Package: 0.00 >> Sun Dec 9 13:13:08 2012: Firewall: 0.01 >> Sun Dec 9 13:13:08 2012: File: 0.01 >> Sun Dec 9 13:13:08 2012: A2mod: 0.03 >> Sun Dec 9 13:13:08 2012: Service: 0.07 >> Sun Dec 9 13:13:08 2012: Exec: 0.14 >> Sun Dec 9 13:13:08 2012: Last run: 1355080388 >> Sun Dec 9 13:13:08 2012: Config retrieval: 42.79 >> Sun Dec 9 13:13:08 2012: Total: 43.06 >> Sun Dec 9 13:13:08 2012: Version: >> Sun Dec 9 13:13:08 2012: Config: 1355080261 >> Sun Dec 9 13:13:08 2012: Puppet: 2.7.18 >> >> >> The install is via RPM. I''m using the Fedora SRPMs rebuilt for CentOS 6. >> The Puppet master has two environments, development and production with >> this as the puppet.conf >> >> # cat /etc/puppet/puppet.conf >> --- >> [main] >> server = puppetmaster.tld >> logdir = /var/log/puppet >> vardir = /var/lib/puppet >> ssldir = /var/lib/puppet/ssl >> rundir = /var/run/puppet >> factpath = $vardir/lib/facter >> pluginsync = true >> templatedir = $confdir/templates >> >> >> [agent] >> report = true >> environment = production >> preferred_serialization_format = yaml >> >> [master] >> modulepath = >> /etc/puppet/environments/$environment/modules:/etc/puppet/modules >> manifest = /etc/puppet/manifests/site.pp >> environment = production >> autosign = false >> ssl_client_header = SSL_CLIENT_S_DN >> ssl_client_verify_header = SSL_CLIENT_VERIFY >> certname = puppetmaster.tld >> dns_alt_names = puppetmaster.tld,puppetmaster-host.tld >> report = true >> reports = store >> #reports = store, foreman >> >> [production] >> manifest = /etc/puppet/manifests/site.pp >> >> >> Has anyone run into this issue with puppetlabs-apache or know why my >> compile times are so long? I have a Puppet 2.6.17 server (Also >> Apache+Passenger) that has one host, for example, with 524 resources , and >> only a 21.36 second catalog compile time. >> >> Thanks >> - Trey >> > > Continuing to debug this issue it seems the apache::mod::default is the > bulk of the compile time. I''m curious if this is a symptom of the module > itself or something with 2.7.x. I moved this puppet master to the EPEL > 2.6.17 RPMs and now catalogs that previously took 40-60 seconds are now > down to 1-2 seconds. > > I noticed that during the long compile times this VM''s (2 vCPU , 2GB RAM) > CPU usage is at 100% from a single Ruby process that links to the PM Rack > process. My other PM running same exact setup with about 50 managed hosts > never has had performance issues always running 2.6.17. > > I''d like to move to 2.7.x for functionality of more modern modules but > this PM is for an HPC and I''m already struggling to convince > my predecessors that Puppetizing the HPC is not going impact performance. > Is there possibly a specific version of 2.7.x or maybe even 3.0.x that may > address the performance issues I''m seeing ? > > Thanks > - Trey >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/cWlBauzx5hwJ. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Michael Stahnke
2012-Dec-23 05:04 UTC
Re: [Puppet Users] Re: Catalog compile times > 40s - Puppet 2.7.18
Puppet 3 has lots of performance improvements. In many cases it''s faster than 2.6.x was. Some of those fixes were back-ported to 2.7.20, but not all of them could be. For best performance, try out something in the Puppet 3 series. On Wed, Dec 19, 2012 at 8:30 AM, shurik <a.grushin@gmail.com> wrote:> Hi > > We are tried to update puppet master from 2.6.6 to 2.7.18 and saw similar > picture. Catalog > compile time took ~60-120 seconds. Before this update compilation took about > ~1-5 second. > After upgrade average CPU usage increased from 10-20% up to 20-40%, during > some activity > (compilation?) ruby process puppetmasterd take 100% too. > > All, is there are some magic to solve or this is expected behaviour? > > Thanks! > > --- > Alexander > > On Saturday, December 15, 2012 10:23:49 PM UTC+4, treydock wrote: >> >> >> >> On Sunday, December 9, 2012 1:41:34 PM UTC-6, treydock wrote: >>> >>> I am finding that the puppetlabs-apache module is somehow adding 30-60 >>> seconds onto a host''s catalog compile time when the puppetmaster has no >>> other hosts contacting or generating catalogs. The Puppetmaster is setup to >>> use Puppet-2.7.18 - Apache & Passenger. RIght now only 2 hosts are even >>> configured to use this new PM, the PM itself and a Foreman host. With >>> neither hosts contacting the PM, and disabling the "include apache" and >>> "require apache::mod::dev" in puppetlabs-passenger, I get "Config retrieval: >>> 2.45". >>> >>> Catalogs had hostnames replaced, puppetmaster.tld is the CNAME for Puppet >>> master and puppetmaster-host.tld is the actual hostname. >>> >>> Catalog: http://pastebin.com/V29JmB8J >>> # puppet agent --{summarize,test,debug,evaltrace,noop} | perl -pe >>> ''s/^/localtime().": "/e'' >>> <snip> >>> Sun Dec 9 13:05:35 2012: notice: Finished catalog run in 2.43 seconds >>> Sun Dec 9 13:05:35 2012: Changes: >>> Sun Dec 9 13:05:35 2012: Events: >>> Sun Dec 9 13:05:35 2012: Total: 1 >>> Sun Dec 9 13:05:35 2012: Noop: 1 >>> Sun Dec 9 13:05:35 2012: Resources: >>> Sun Dec 9 13:05:35 2012: Out of sync: 1 >>> Sun Dec 9 13:05:35 2012: Total: 22 >>> Sun Dec 9 13:05:35 2012: Skipped: 6 >>> Sun Dec 9 13:05:35 2012: Time: >>> Sun Dec 9 13:05:35 2012: Resources: 0.00 >>> Sun Dec 9 13:05:35 2012: Filebucket: 0.00 >>> Sun Dec 9 13:05:35 2012: Package: 0.00 >>> Sun Dec 9 13:05:35 2012: File: 0.00 >>> Sun Dec 9 13:05:35 2012: Firewall: 0.01 >>> Sun Dec 9 13:05:35 2012: Exec: 0.14 >>> Sun Dec 9 13:05:35 2012: Last run: 1355079935 >>> Sun Dec 9 13:05:35 2012: Config retrieval: 2.45 >>> Sun Dec 9 13:05:35 2012: Total: 2.61 >>> Sun Dec 9 13:05:35 2012: Version: >>> Sun Dec 9 13:05:35 2012: Config: 1355077701 >>> Sun Dec 9 13:05:35 2012: Puppet: 2.7.18 >>> >>> Uncommenting "include apache" and "require apache::mod::dev" in >>> puppetlabs-passenger about 40 seconds are added to the compile time. I went >>> through disabled all modules and parameters, and found that enabling the >>> passenger module alone with Apache included caused the huge delay. >>> >>> Catalog: http://pastebin.com/wwcKgX5b >>> # puppet agent --{summarize,test,debug,evaltrace,noop} | perl -pe >>> ''s/^/localtime().": "/e'' >>> <snip> >>> Sun Dec 9 13:13:08 2012: notice: Finished catalog run in 2.23 seconds >>> Sun Dec 9 13:13:08 2012: Changes: >>> Sun Dec 9 13:13:08 2012: Events: >>> Sun Dec 9 13:13:08 2012: Total: 4 >>> Sun Dec 9 13:13:08 2012: Noop: 4 >>> Sun Dec 9 13:13:08 2012: Resources: >>> Sun Dec 9 13:13:08 2012: Out of sync: 4 >>> Sun Dec 9 13:13:08 2012: Skipped: 6 >>> Sun Dec 9 13:13:08 2012: Total: 80 >>> Sun Dec 9 13:13:08 2012: Time: >>> Sun Dec 9 13:13:08 2012: Filebucket: 0.00 >>> Sun Dec 9 13:13:08 2012: Resources: 0.00 >>> Sun Dec 9 13:13:08 2012: Package: 0.00 >>> Sun Dec 9 13:13:08 2012: Firewall: 0.01 >>> Sun Dec 9 13:13:08 2012: File: 0.01 >>> Sun Dec 9 13:13:08 2012: A2mod: 0.03 >>> Sun Dec 9 13:13:08 2012: Service: 0.07 >>> Sun Dec 9 13:13:08 2012: Exec: 0.14 >>> Sun Dec 9 13:13:08 2012: Last run: 1355080388 >>> Sun Dec 9 13:13:08 2012: Config retrieval: 42.79 >>> Sun Dec 9 13:13:08 2012: Total: 43.06 >>> Sun Dec 9 13:13:08 2012: Version: >>> Sun Dec 9 13:13:08 2012: Config: 1355080261 >>> Sun Dec 9 13:13:08 2012: Puppet: 2.7.18 >>> >>> >>> The install is via RPM. I''m using the Fedora SRPMs rebuilt for CentOS 6. >>> The Puppet master has two environments, development and production with this >>> as the puppet.conf >>> >>> # cat /etc/puppet/puppet.conf >>> --- >>> [main] >>> server = puppetmaster.tld >>> logdir = /var/log/puppet >>> vardir = /var/lib/puppet >>> ssldir = /var/lib/puppet/ssl >>> rundir = /var/run/puppet >>> factpath = $vardir/lib/facter >>> pluginsync = true >>> templatedir = $confdir/templates >>> >>> >>> [agent] >>> report = true >>> environment = production >>> preferred_serialization_format = yaml >>> >>> [master] >>> modulepath >>> /etc/puppet/environments/$environment/modules:/etc/puppet/modules >>> manifest = /etc/puppet/manifests/site.pp >>> environment = production >>> autosign = false >>> ssl_client_header = SSL_CLIENT_S_DN >>> ssl_client_verify_header = SSL_CLIENT_VERIFY >>> certname = puppetmaster.tld >>> dns_alt_names = puppetmaster.tld,puppetmaster-host.tld >>> report = true >>> reports = store >>> #reports = store, foreman >>> >>> [production] >>> manifest = /etc/puppet/manifests/site.pp >>> >>> >>> Has anyone run into this issue with puppetlabs-apache or know why my >>> compile times are so long? I have a Puppet 2.6.17 server (Also >>> Apache+Passenger) that has one host, for example, with 524 resources , and >>> only a 21.36 second catalog compile time. >>> >>> Thanks >>> - Trey >> >> >> Continuing to debug this issue it seems the apache::mod::default is the >> bulk of the compile time. I''m curious if this is a symptom of the module >> itself or something with 2.7.x. I moved this puppet master to the EPEL >> 2.6.17 RPMs and now catalogs that previously took 40-60 seconds are now down >> to 1-2 seconds. >> >> I noticed that during the long compile times this VM''s (2 vCPU , 2GB RAM) >> CPU usage is at 100% from a single Ruby process that links to the PM Rack >> process. My other PM running same exact setup with about 50 managed hosts >> never has had performance issues always running 2.6.17. >> >> I''d like to move to 2.7.x for functionality of more modern modules but >> this PM is for an HPC and I''m already struggling to convince my predecessors >> that Puppetizing the HPC is not going impact performance. Is there possibly >> a specific version of 2.7.x or maybe even 3.0.x that may address the >> performance issues I''m seeing ? >> >> Thanks >> - Trey > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/puppet-users/-/cWlBauzx5hwJ. > > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en.-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Trey Dockendorf
2012-Dec-24 08:30 UTC
Re: [Puppet Users] Re: Catalog compile times > 40s - Puppet 2.7.18
On Dec 22, 2012 11:04 PM, "Michael Stahnke" <stahnma@puppetlabs.com> wrote:> > Puppet 3 has lots of performance improvements. In many cases it''s > faster than 2.6.x was. Some of those fixes were back-ported to 2.7.20, > but not all of them could be. > > For best performance, try out something in the Puppet 3 series. > > > > On Wed, Dec 19, 2012 at 8:30 AM, shurik <a.grushin@gmail.com> wrote: > > Hi > > > > We are tried to update puppet master from 2.6.6 to 2.7.18 and sawsimilar> > picture. Catalog > > compile time took ~60-120 seconds. Before this update compilation tookabout> > ~1-5 second. > > After upgrade average CPU usage increased from 10-20% up to 20-40%,during> > some activity > > (compilation?) ruby process puppetmasterd take 100% too. > > > > All, is there are some magic to solve or this is expected behaviour? > > > > Thanks! > > > > --- > > Alexander > > > > On Saturday, December 15, 2012 10:23:49 PM UTC+4, treydock wrote: > >> > >> > >> > >> On Sunday, December 9, 2012 1:41:34 PM UTC-6, treydock wrote: > >>> > >>> I am finding that the puppetlabs-apache module is somehow adding 30-60 > >>> seconds onto a host''s catalog compile time when the puppetmaster hasno> >>> other hosts contacting or generating catalogs. The Puppetmaster issetup to> >>> use Puppet-2.7.18 - Apache & Passenger. RIght now only 2 hosts areeven> >>> configured to use this new PM, the PM itself and a Foreman host. With > >>> neither hosts contacting the PM, and disabling the "include apache"and> >>> "require apache::mod::dev" in puppetlabs-passenger, I get "Configretrieval:> >>> 2.45". > >>> > >>> Catalogs had hostnames replaced, puppetmaster.tld is the CNAME forPuppet> >>> master and puppetmaster-host.tld is the actual hostname. > >>> > >>> Catalog: http://pastebin.com/V29JmB8J > >>> # puppet agent --{summarize,test,debug,evaltrace,noop} | perl -pe > >>> ''s/^/localtime().": "/e'' > >>> <snip> > >>> Sun Dec 9 13:05:35 2012: notice: Finished catalog run in 2.43 seconds > >>> Sun Dec 9 13:05:35 2012: Changes: > >>> Sun Dec 9 13:05:35 2012: Events: > >>> Sun Dec 9 13:05:35 2012: Total: 1 > >>> Sun Dec 9 13:05:35 2012: Noop: 1 > >>> Sun Dec 9 13:05:35 2012: Resources: > >>> Sun Dec 9 13:05:35 2012: Out of sync: 1 > >>> Sun Dec 9 13:05:35 2012: Total: 22 > >>> Sun Dec 9 13:05:35 2012: Skipped: 6 > >>> Sun Dec 9 13:05:35 2012: Time: > >>> Sun Dec 9 13:05:35 2012: Resources: 0.00 > >>> Sun Dec 9 13:05:35 2012: Filebucket: 0.00 > >>> Sun Dec 9 13:05:35 2012: Package: 0.00 > >>> Sun Dec 9 13:05:35 2012: File: 0.00 > >>> Sun Dec 9 13:05:35 2012: Firewall: 0.01 > >>> Sun Dec 9 13:05:35 2012: Exec: 0.14 > >>> Sun Dec 9 13:05:35 2012: Last run: 1355079935 > >>> Sun Dec 9 13:05:35 2012: Config retrieval: 2.45 > >>> Sun Dec 9 13:05:35 2012: Total: 2.61 > >>> Sun Dec 9 13:05:35 2012: Version: > >>> Sun Dec 9 13:05:35 2012: Config: 1355077701 > >>> Sun Dec 9 13:05:35 2012: Puppet: 2.7.18 > >>> > >>> Uncommenting "include apache" and "require apache::mod::dev" in > >>> puppetlabs-passenger about 40 seconds are added to the compile time.I went> >>> through disabled all modules and parameters, and found that enablingthe> >>> passenger module alone with Apache included caused the huge delay. > >>> > >>> Catalog: http://pastebin.com/wwcKgX5b > >>> # puppet agent --{summarize,test,debug,evaltrace,noop} | perl -pe > >>> ''s/^/localtime().": "/e'' > >>> <snip> > >>> Sun Dec 9 13:13:08 2012: notice: Finished catalog run in 2.23 seconds > >>> Sun Dec 9 13:13:08 2012: Changes: > >>> Sun Dec 9 13:13:08 2012: Events: > >>> Sun Dec 9 13:13:08 2012: Total: 4 > >>> Sun Dec 9 13:13:08 2012: Noop: 4 > >>> Sun Dec 9 13:13:08 2012: Resources: > >>> Sun Dec 9 13:13:08 2012: Out of sync: 4 > >>> Sun Dec 9 13:13:08 2012: Skipped: 6 > >>> Sun Dec 9 13:13:08 2012: Total: 80 > >>> Sun Dec 9 13:13:08 2012: Time: > >>> Sun Dec 9 13:13:08 2012: Filebucket: 0.00 > >>> Sun Dec 9 13:13:08 2012: Resources: 0.00 > >>> Sun Dec 9 13:13:08 2012: Package: 0.00 > >>> Sun Dec 9 13:13:08 2012: Firewall: 0.01 > >>> Sun Dec 9 13:13:08 2012: File: 0.01 > >>> Sun Dec 9 13:13:08 2012: A2mod: 0.03 > >>> Sun Dec 9 13:13:08 2012: Service: 0.07 > >>> Sun Dec 9 13:13:08 2012: Exec: 0.14 > >>> Sun Dec 9 13:13:08 2012: Last run: 1355080388 > >>> Sun Dec 9 13:13:08 2012: Config retrieval: 42.79 > >>> Sun Dec 9 13:13:08 2012: Total: 43.06 > >>> Sun Dec 9 13:13:08 2012: Version: > >>> Sun Dec 9 13:13:08 2012: Config: 1355080261 > >>> Sun Dec 9 13:13:08 2012: Puppet: 2.7.18 > >>> > >>> > >>> The install is via RPM. I''m using the Fedora SRPMs rebuilt forCentOS 6.> >>> The Puppet master has two environments, development and productionwith this> >>> as the puppet.conf > >>> > >>> # cat /etc/puppet/puppet.conf > >>> --- > >>> [main] > >>> server = puppetmaster.tld > >>> logdir = /var/log/puppet > >>> vardir = /var/lib/puppet > >>> ssldir = /var/lib/puppet/ssl > >>> rundir = /var/run/puppet > >>> factpath = $vardir/lib/facter > >>> pluginsync = true > >>> templatedir = $confdir/templates > >>> > >>> > >>> [agent] > >>> report = true > >>> environment = production > >>> preferred_serialization_format = yaml > >>> > >>> [master] > >>> modulepath > >>> /etc/puppet/environments/$environment/modules:/etc/puppet/modules > >>> manifest = /etc/puppet/manifests/site.pp > >>> environment = production > >>> autosign = false > >>> ssl_client_header = SSL_CLIENT_S_DN > >>> ssl_client_verify_header = SSL_CLIENT_VERIFY > >>> certname = puppetmaster.tld > >>> dns_alt_names = puppetmaster.tld,puppetmaster-host.tld > >>> report = true > >>> reports = store > >>> #reports = store, foreman > >>> > >>> [production] > >>> manifest = /etc/puppet/manifests/site.pp > >>> > >>> > >>> Has anyone run into this issue with puppetlabs-apache or know why my > >>> compile times are so long? I have a Puppet 2.6.17 server (Also > >>> Apache+Passenger) that has one host, for example, with 524 resources, and> >>> only a 21.36 second catalog compile time. > >>> > >>> Thanks > >>> - Trey > >> > >> > >> Continuing to debug this issue it seems the apache::mod::default is the > >> bulk of the compile time. I''m curious if this is a symptom of themodule> >> itself or something with 2.7.x. I moved this puppet master to the EPEL > >> 2.6.17 RPMs and now catalogs that previously took 40-60 seconds arenow down> >> to 1-2 seconds. > >> > >> I noticed that during the long compile times this VM''s (2 vCPU , 2GBRAM)> >> CPU usage is at 100% from a single Ruby process that links to the PMRack> >> process. My other PM running same exact setup with about 50 managedhosts> >> never has had performance issues always running 2.6.17. > >> > >> I''d like to move to 2.7.x for functionality of more modern modules but > >> this PM is for an HPC and I''m already struggling to convince mypredecessors> >> that Puppetizing the HPC is not going impact performance. Is therepossibly> >> a specific version of 2.7.x or maybe even 3.0.x that may address the > >> performance issues I''m seeing ? > >> > >> Thanks > >> - Trey > > > > -- > > You received this message because you are subscribed to the GoogleGroups> > "Puppet Users" group. > > To view this discussion on the web visit > > https://groups.google.com/d/msg/puppet-users/-/cWlBauzx5hwJ. > > > > To post to this group, send email to puppet-users@googlegroups.com. > > To unsubscribe from this group, send email to > > puppet-users+unsubscribe@googlegroups.com. > > For more options, visit this group at > > http://groups.google.com/group/puppet-users?hl=en. > > -- > You received this message because you are subscribed to the Google Groups"Puppet Users" group.> To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email topuppet-users+unsubscribe@googlegroups.com.> For more options, visit this group athttp://groups.google.com/group/puppet-users?hl=en.>In addition to upgrading to Puppet 3, would any other changes help improve performance? I''ve seen mention of switching to ruby enterprise edition for example. We will likely have to disable Puppet automatic runs entirely on compute nodes and have it only run as a administrative queue job or have it run as part of every job''s pre or post execution, so minimal catalog compile and run times are critical. Thanks - Trey -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.