christian.le.flamm@gmail.com
2013-Jun-13 14:26 UTC
[Puppet Users] puppet: 3.1.1 -> 3.2.1 load increase
Hi, I recently updated from puppet 3.1.1 to 3.2.1 and noticed quite a bit of increased load on the puppetmaster machine. I''m using the Apache/passenger/rack way of puppetmastering. Main symptom is: higher load on puppetmaster machine (8 cores): - 3.1.1: around 4 - 3.2.1: around 9-10 Any idea why there''s more load on the machine with 3.2.1? -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
On 06/13/2013 04:26 PM, christian.le.flamm@gmail.com wrote:> Hi, > I recently updated from puppet 3.1.1 to 3.2.1 and noticed quite a bit of > increased load on the puppetmaster machine. I''m using > the Apache/passenger/rack way of puppetmastering. > Main symptom is: higher load on puppetmaster machine (8 cores): > > * 3.1.1: around 4 > * 3.2.1: around 9-10 > > Any idea why there''s more load on the machine with 3.2.1?You have to be a little bit more specific. Is load only CPU related, or I/O? Also, what do the log say? 3.2 pushes lots of notices about syntax obsolescence in the logs... -- Jakov Sosic www.srce.unizg.hr -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
christian.le.flamm@gmail.com
2013-Jun-25 13:15 UTC
Re: [Puppet Users] puppet: 3.1.1 -> 3.2.1 load increase
Forgot to enable email notification here. Decided to go back to 3.1.1 a week ago. Will upgrade again to provide more specific load information. Thx! You have to be a little bit more specific. Is load only CPU related, or> I/O? Also, what do the log say? 3.2 pushes lots of notices about syntax > obsolescence in the logs... >>-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
Christian Flamm
2013-Jun-27 10:51 UTC
Re: [Puppet Users] puppet: 3.1.1 -> 3.2.1 load increase
Two load situations, one pretty high, one relatively low. Usually it''s
somewhere in between these.
top - 12:43:32 up 65 days, 3:27, 3 users, load average: 12.89, 9.29, 8.00
Tasks: 249 total, 15 running, 234 sleeping, 0 stopped, 0 zombie
Cpu0 : 83.7%us, 11.7%sy, 0.0%ni, 0.0%id, 0.0%wa, 4.3%hi, 0.3%si,
0.0%st
Cpu1 : 87.7%us, 12.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Cpu2 : 93.0%us, 7.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Cpu3 : 89.4%us, 10.6%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Cpu4 : 92.4%us, 7.6%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Cpu5 : 87.1%us, 12.9%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Cpu6 : 91.4%us, 8.6%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Cpu7 : 89.4%us, 10.6%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Mem: 12191256k total, 10148368k used, 2042888k free, 156196k buffers
Swap: 2031608k total, 32704k used, 1998904k free, 4762472k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20989 puppet 20 0 296m 195m 2832 R 70.7 1.6 1:21.05 ruby
14894 puppet 20 0 301m 201m 2832 R 68.7 1.7 5:10.94 ruby
22920 puppet 20 0 201m 101m 2832 R 66.4 0.8 0:22.10 ruby
17551 puppet 20 0 284m 183m 2832 R 61.7 1.5 2:17.69 ruby
17555 puppet 20 0 306m 204m 2832 R 57.7 1.7 3:15.24 ruby
22930 puppet 20 0 202m 101m 2820 S 51.8 0.9 0:07.88 ruby
16171 puppet-d 20 0 173m 73m 1788 R 51.4 0.6 5749:13 ruby
15194 puppet-d 20 0 227m 128m 1788 R 44.8 1.1 5743:53 ruby
16159 puppet-d 20 0 227m 128m 1788 R 43.8 1.1 5697:15 ruby
16166 puppet-d 20 0 213m 114m 1788 R 38.8 1.0 5742:11 ruby
16199 puppet-d 20 0 220m 121m 1788 R 36.2 1.0 5766:36 ruby
16216 puppet-d 20 0 216m 117m 1788 R 33.9 1.0 5719:14 ruby
16158 puppet-d 20 0 216m 117m 1788 R 31.5 1.0 5766:53 ruby
23391 root 20 0 188m 84m 3296 S 30.2 0.7 0:03.85 puppet
22140 puppet 20 0 282m 181m 2832 R 29.2 1.5 0:48.71 ruby
23543 root 20 0 442m 31m 7520 R 14.9 0.3 0:00.45 python
15049 mysql 20 0 1170m 20m 4420 S 5.6 0.2 1551:47 mysqld
20997 puppet 20 0 286m 186m 2832 S 5.3 1.6 1:23.59 ruby
674 root 20 0 0 0 0 S 3.3 0.0 43:29.96 flush-253:0
6765 apache 20 0 202m 5264 2464 S 3.3 0.0 1:24.64 httpd
6850 apache 20 0 202m 5264 2464 S 3.3 0.0 1:26.29 httpd
21001 puppet 20 0 207m 107m 2832 S 3.3 0.9 0:48.17 ruby
6751 root 20 0 1461m 4476 2796 S 3.0 0.0 1:29.47
PassengerHelper
6766 apache 20 0 202m 5264 2464 S 3.0 0.0 1:21.27 httpd
6767 apache 20 0 202m 5272 2464 S 3.0 0.0 1:22.50 httpd
6769 apache 20 0 202m 5264 2464 S 2.7 0.0 1:24.32 httpd
6770 apache 20 0 202m 5264 2464 S 2.7 0.0 1:23.05 httpd
6771 apache 20 0 202m 5264 2464 S 2.7 0.0 1:22.95 httpd
7192 apache 20 0 202m 5272 2464 S 2.7 0.0 1:21.21 httpd
9109 apache 20 0 202m 5264 2464 S 2.7 0.0 1:19.35 httpd
22924 apache 20 0 202m 5264 2464 S 2.7 0.0 0:01.04 httpd
6768 apache 20 0 202m 5264 2464 S 2.3 0.0 1:22.45 httpd
6798 apache 20 0 202m 5272 2464 S 2.3 0.0 1:24.98 httpd
7175 apache 20 0 202m 5264 2464 S 1.7 0.0 1:22.83 httpd
7191 apache 20 0 202m 5264 2464 S 1.7 0.0 1:22.30 httpd
15501 puppet-d 20 0 225m 121m 3728 S 1.7 1.0 271:12.00 ruby
6851 apache 20 0 202m 5264 2464 S 1.3 0.0 1:22.33 httpd
16025 puppet 20 0 301m 199m 2832 S 0.7 1.7 4:10.28 ruby
102 root 39 19 0 0 0 S 0.3 0.0 26:32.33 khugepaged
1165 root 20 0 192m 1352 800 S 0.3 0.0 19:24.76 snmpd
6091 puppet-d 20 0 219m 120m 1788 S 0.3 1.0 2814:11 ruby
12485 puppet 20 0 301m 201m 2832 S 0.3 1.7 5:45.50 ruby
23503 root 20 0 15164 1460 1012 R 0.3 0.0 0:00.04 top
1 root 20 0 19228 952 820 S 0.0 0.0 0:07.93 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 2:59.01 migration/0
4 root 20 0 0 0 0 S 0.0 0.0 3487:07 ksoftirqd/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
6 root RT 0 0 0 0 S 0.0 0.0 0:14.73 watchdog/0
and
top - 11:51:21 up 65 days, 2:35, 3 users, load average: 6.86, 6.95, 5.70
Tasks: 246 total, 10 running, 236 sleeping, 0 stopped, 0 zombie
Cpu0 : 88.8%us, 8.6%sy, 0.0%ni, 1.3%id, 0.0%wa, 0.7%hi, 0.7%si,
0.0%st
Cpu1 : 85.4%us, 11.6%sy, 0.0%ni, 3.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Cpu2 : 84.1%us, 15.9%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Cpu3 : 89.0%us, 9.0%sy, 0.0%ni, 2.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Cpu4 : 88.4%us, 11.6%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Cpu5 : 83.8%us, 15.9%sy, 0.0%ni, 0.3%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Cpu6 : 85.1%us, 12.3%sy, 0.0%ni, 2.6%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Cpu7 : 86.1%us, 13.9%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Mem: 12191256k total, 10695256k used, 1496000k free, 159008k buffers
Swap: 2031608k total, 36528k used, 1995080k free, 4940364k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6963 puppet 20 0 289m 189m 2832 R 98.0 1.6 3:42.39 ruby
7200 puppet 20 0 296m 195m 2832 R 98.0 1.6 5:16.97 ruby
6806 puppet 20 0 287m 187m 2832 R 90.7 1.6 2:51.12 ruby
16199 puppet-d 20 0 220m 121m 1788 R 90.7 1.0 5756:26 ruby
6815 puppet 20 0 300m 200m 2832 R 90.1 1.7 5:23.84 ruby
9111 puppet 20 0 284m 183m 2832 R 89.1 1.5 2:10.29 ruby
6949 puppet 20 0 298m 197m 2832 S 61.9 1.7 4:16.82 ruby
16067 root 20 0 189m 85m 3296 S 44.0 0.7 0:03.89 puppet
6772 apache 20 0 202m 5264 2464 S 6.0 0.0 0:17.48 httpd
6765 apache 20 0 202m 5264 2464 R 5.6 0.0 0:17.48 httpd
6768 apache 20 0 202m 5264 2464 S 5.3 0.0 0:17.72 httpd
7192 apache 20 0 202m 5272 2464 R 5.3 0.0 0:15.75 httpd
6798 apache 20 0 202m 5272 2464 S 4.6 0.0 0:17.14 httpd
6751 root 20 0 1332m 4308 2796 S 4.3 0.0 0:17.61
PassengerHelper
6766 apache 20 0 202m 5264 2464 S 4.3 0.0 0:17.32 httpd
6771 apache 20 0 202m 5264 2464 S 4.3 0.0 0:16.38 httpd
6850 apache 20 0 202m 5264 2464 R 4.3 0.0 0:16.98 httpd
7175 apache 20 0 202m 5260 2464 S 4.3 0.0 0:15.63 httpd
7180 puppet 20 0 301m 200m 2832 S 4.3 1.7 4:01.09 ruby
10049 apache 20 0 202m 5268 2464 S 4.3 0.0 0:10.43 httpd
10050 apache 20 0 202m 5268 2464 S 4.3 0.0 0:10.90 httpd
6767 apache 20 0 202m 5268 2464 S 4.0 0.0 0:15.59 httpd
6851 apache 20 0 202m 5264 2464 S 3.6 0.0 0:15.80 httpd
7191 apache 20 0 202m 5264 2464 S 3.6 0.0 0:16.16 httpd
15049 mysql 20 0 1170m 20m 4424 S 3.3 0.2 1548:50 mysqld
7064 puppet 20 0 296m 194m 2832 S 2.3 1.6 3:22.13 ruby
9109 apache 20 0 202m 5264 2464 S 1.3 0.0 0:12.50 httpd
15501 puppet-d 20 0 225m 121m 3728 S 1.0 1.0 270:45.64 ruby
13527 root 20 0 15164 1464 1012 R 0.7 0.0 0:00.62 top
4 root 20 0 0 0 0 S 0.3 0.0 3485:56 ksoftirqd/0
10 root RT 0 0 0 0 S 0.3 0.0 0:22.64 watchdog/1
29 root 20 0 0 0 0 S 0.3 0.0 2726:41 ksoftirqd/6
6811 puppet 20 0 298m 199m 2832 S 0.3 1.7 4:01.83 ruby
6908 puppet 20 0 291m 191m 2832 S 0.3 1.6 2:41.72 ruby
7072 puppet 20 0 298m 198m 2832 S 0.3 1.7 3:58.77 ruby
15194 puppet-d 20 0 227m 128m 1788 S 0.3 1.1 5733:11 ruby
1 root 20 0 19228 952 820 S 0.0 0.0 0:07.93 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 2:58.92 migration/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
6 root RT 0 0 0 0 S 0.0 0.0 0:14.72 watchdog/0
7 root RT 0 0 0 0 S 0.0 0.0 2:13.03 migration/1
8 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/1
9 root 20 0 0 0 0 S 0.0 0.0 8879:22 ksoftirqd/1
11 root RT 0 0 0 0 S 0.0 0.0 2:14.30 migration/2
12 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/2
13 root 20 0 0 0 0 S 0.0 0.0 6996:32 ksoftirqd/2
14 root RT 0 0 0 0 S 0.0 0.0 0:19.37 watchdog/2
15 root RT 0 0 0 0 S 0.0 0.0 2:18.66 migration/3
Btw: have already upgraded to 3.2.2 - doesn''t change a thing.
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.
Christian Flamm
2013-Jun-27 11:03 UTC
Re: [Puppet Users] puppet: 3.1.1 -> 3.2.1 load increase
Forgot to say this explicitly: config_retrieval times (guess this includes catlog compilation) on the agents explode. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.