We have quite a bit of puppet clients handled by Mongrel server (4instance) and off late been noticing performance issues with the client mostly. It takes almost 2-3mins to complete puppet runs on clients and takes almost 100%cpus. I know its hard to know what the problem is unless you see what the manifests are. My question is more on what are the tools or options puppet has to look more into details on what its doing or spending most time. When i run with "--debug --test" i see it mostly doing multiples "fileserver.describe" and we are not doing any file bucketing on the server. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
if you use summarize = true on your client configs, you should get a block
at the end that looks like this:
#puppetd -vt
<snip>
Changes:
Total: 1
Resources:
Applied: 1
Out of sync: 1
Restarted: 1
Scheduled: 317
Total: 266
Time:
Config retrieval: 139.14
Cron: 0.16
Exec: 5.28
File: 19.54
Filebucket: 0.00
Mount: 0.27
Package: 2.87
Schedule: 0.00
Service: 3.08
User: 0.06
Yumrepo: 0.01
Total: 170.43
info: Sent transaction report in 0.28 seconds
notice: Finished catalog run in 35.99 seconds
>
> We have quite a bit of puppet clients handled by Mongrel server
> (4instance) and off late been noticing performance issues with the
> client mostly. It takes almost 2-3mins to complete puppet runs on
> clients and takes almost 100%cpus.
>
> I know its hard to know what the problem is unless you see what the
> manifests are. My question is more on what are the tools or options
> puppet has to look more into details on what its doing or spending
> most time. When i run with "--debug --test" i see it mostly doing
> multiples "fileserver.describe" and we are not doing any file
> bucketing on the server.
>
> >
>
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---
Thank for the information, yeah looks like the client spend lot of
time in the file operations. Do you know anyway i can troubleshoot
there whats its doing. turning on debug just show me
fileserver.describe and couple of freeze.
# puppetd -vt
info: Loading fact hosttype
info: Loading fact subdomain
info: Loading fact dcloc
notice: Ignoring --listen on onetime run
info: Retrieving facts
info: Loading fact hosttype
info: Loading fact subdomain
info: Loading fact dcloc
info: mount[localhost]: Mounted /
info: mount[modules]: Mounted
info: mount[plugins]: Mounted
info: Caching catalog at /var/lib/puppet/localconfig.yaml
notice: Starting catalog run
info: Loading fact hosttype
info: Loading fact subdomain
info: Loading fact dcloc
...
.....
Changes:
Total: 1
Resources:
Applied: 1
Out of sync: 1
Scheduled: 1724
Total: 199
Time:
Config retrieval: 6.44
Cron: 0.05
Exec: 0.01
File: 42.64
Filebucket: 0.00
Group: 0.45
Mount: 0.57
Package: 1.26
Schedule: 0.00
Service: 0.92
User: 3.32
Total: 55.65
notice: Finished catalog run in 103.27 seconds
noticed couple of things:
* it computes facter multiple times, is that normal?
* i did a tcpdump and i see that the puppetserver send multiple FIN
during a session and the client reconnects, is that normal?
On Jul 29, 6:19 pm, "Jason Rojas" <ja...@nothingbeatsaduck.com>
wrote:> if you use summarize = true on your client configs, you should get a block
> at the end that looks like this:
> #puppetd -vt
>
> <snip>
>
> Changes:
> Total: 1
> Resources:
> Applied: 1
> Out of sync: 1
> Restarted: 1
> Scheduled: 317
> Total: 266
> Time:
> Config retrieval: 139.14
> Cron: 0.16
> Exec: 5.28
> File: 19.54
> Filebucket: 0.00
> Mount: 0.27
> Package: 2.87
> Schedule: 0.00
> Service: 3.08
> User: 0.06
> Yumrepo: 0.01
> Total: 170.43
> info: Sent transaction report in 0.28 seconds
> notice: Finished catalog run in 35.99 seconds
>
>
>
> > We have quite a bit of puppet clients handled by Mongrel server
> > (4instance) and off late been noticing performance issues with the
> > client mostly. It takes almost 2-3mins to complete puppet runs on
> > clients and takes almost 100%cpus.
>
> > I know its hard to know what the problem is unless you see what the
> > manifests are. My question is more on what are the tools or options
> > puppet has to look more into details on what its doing or spending
> > most time. When i run with "--debug --test" i see it mostly
doing
> > multiples "fileserver.describe" and we are not doing any
file
> > bucketing on the server.
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---
Do you have a lot of static files that you''re copying over using the puppet:// protocol? If so, this is probably what is taking up most of your time. If not, are you recursively managing any folders with lots of subdirectories and/or files? If so, this is probably the other problem. Trevor On Thu, Jul 30, 2009 at 17:53, mr<mroshan@gmail.com> wrote:> > Thank for the information, yeah looks like the client spend lot of > time in the file operations. Do you know anyway i can troubleshoot > there whats its doing. turning on debug just show me > fileserver.describe and couple of freeze. > > > # puppetd -vt > info: Loading fact hosttype > info: Loading fact subdomain > info: Loading fact dcloc > notice: Ignoring --listen on onetime run > info: Retrieving facts > info: Loading fact hosttype > info: Loading fact subdomain > info: Loading fact dcloc > info: mount[localhost]: Mounted / > info: mount[modules]: Mounted > info: mount[plugins]: Mounted > info: Caching catalog at /var/lib/puppet/localconfig.yaml > notice: Starting catalog run > info: Loading fact hosttype > info: Loading fact subdomain > info: Loading fact dcloc > ... > ..... > Changes: > Total: 1 > Resources: > Applied: 1 > Out of sync: 1 > Scheduled: 1724 > Total: 199 > Time: > Config retrieval: 6.44 > Cron: 0.05 > Exec: 0.01 > File: 42.64 > Filebucket: 0.00 > Group: 0.45 > Mount: 0.57 > Package: 1.26 > Schedule: 0.00 > Service: 0.92 > User: 3.32 > Total: 55.65 > notice: Finished catalog run in 103.27 seconds > > > noticed couple of things: > * it computes facter multiple times, is that normal? > * i did a tcpdump and i see that the puppetserver send multiple FIN > during a session and the client reconnects, is that normal? > > > On Jul 29, 6:19 pm, "Jason Rojas" <ja...@nothingbeatsaduck.com> wrote: >> if you use summarize = true on your client configs, you should get a block >> at the end that looks like this: >> #puppetd -vt >> >> <snip> >> >> Changes: >> Total: 1 >> Resources: >> Applied: 1 >> Out of sync: 1 >> Restarted: 1 >> Scheduled: 317 >> Total: 266 >> Time: >> Config retrieval: 139.14 >> Cron: 0.16 >> Exec: 5.28 >> File: 19.54 >> Filebucket: 0.00 >> Mount: 0.27 >> Package: 2.87 >> Schedule: 0.00 >> Service: 3.08 >> User: 0.06 >> Yumrepo: 0.01 >> Total: 170.43 >> info: Sent transaction report in 0.28 seconds >> notice: Finished catalog run in 35.99 seconds >> >> >> >> > We have quite a bit of puppet clients handled by Mongrel server >> > (4instance) and off late been noticing performance issues with the >> > client mostly. It takes almost 2-3mins to complete puppet runs on >> > clients and takes almost 100%cpus. >> >> > I know its hard to know what the problem is unless you see what the >> > manifests are. My question is more on what are the tools or options >> > puppet has to look more into details on what its doing or spending >> > most time. When i run with "--debug --test" i see it mostly doing >> > multiples "fileserver.describe" and we are not doing any file >> > bucketing on the server. > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
yeah we do have a quite a few static files wit subdirectories, however they are mounted nfs on the client and been copied, but i guess it needs to do a checksum every run. whats the solution. and i can''t understand why i get connection timeouts. we have puppetserver with apache/mongrel setup. On Jul 30, 3:00 pm, Trevor Vaughan <peiriann...@gmail.com> wrote:> Do you have a lot of static files that you''re copying over using the > puppet:// protocol? > > If so, this is probably what is taking up most of your time. > > If not, are you recursively managing any folders with lots of > subdirectories and/or files? > > If so, this is probably the other problem. > > Trevor > > On Thu, Jul 30, 2009 at 17:53, mr<mros...@gmail.com> wrote: > > > Thank for the information, yeah looks like the client spend lot of > > time in the file operations. Do you know anyway i can troubleshoot > > there whats its doing. turning on debug just show me > > fileserver.describe and couple of freeze. > > > # puppetd -vt > > info: Loading fact hosttype > > info: Loading fact subdomain > > info: Loading fact dcloc > > notice: Ignoring --listen on onetime run > > info: Retrieving facts > > info: Loading fact hosttype > > info: Loading fact subdomain > > info: Loading fact dcloc > > info: mount[localhost]: Mounted / > > info: mount[modules]: Mounted > > info: mount[plugins]: Mounted > > info: Caching catalog at /var/lib/puppet/localconfig.yaml > > notice: Starting catalog run > > info: Loading fact hosttype > > info: Loading fact subdomain > > info: Loading fact dcloc > > ... > > ..... > > Changes: > > Total: 1 > > Resources: > > Applied: 1 > > Out of sync: 1 > > Scheduled: 1724 > > Total: 199 > > Time: > > Config retrieval: 6.44 > > Cron: 0.05 > > Exec: 0.01 > > File: 42.64 > > Filebucket: 0.00 > > Group: 0.45 > > Mount: 0.57 > > Package: 1.26 > > Schedule: 0.00 > > Service: 0.92 > > User: 3.32 > > Total: 55.65 > > notice: Finished catalog run in 103.27 seconds > > > noticed couple of things: > > * it computes facter multiple times, is that normal? > > * i did a tcpdump and i see that the puppetserver send multiple FIN > > during a session and the client reconnects, is that normal? > > > On Jul 29, 6:19 pm, "Jason Rojas" <ja...@nothingbeatsaduck.com> wrote: > >> if you use summarize = true on your client configs, you should get a block > >> at the end that looks like this: > >> #puppetd -vt > > >> <snip> > > >> Changes: > >> Total: 1 > >> Resources: > >> Applied: 1 > >> Out of sync: 1 > >> Restarted: 1 > >> Scheduled: 317 > >> Total: 266 > >> Time: > >> Config retrieval: 139.14 > >> Cron: 0.16 > >> Exec: 5.28 > >> File: 19.54 > >> Filebucket: 0.00 > >> Mount: 0.27 > >> Package: 2.87 > >> Schedule: 0.00 > >> Service: 3.08 > >> User: 0.06 > >> Yumrepo: 0.01 > >> Total: 170.43 > >> info: Sent transaction report in 0.28 seconds > >> notice: Finished catalog run in 35.99 seconds > > >> > We have quite a bit of puppet clients handled by Mongrel server > >> > (4instance) and off late been noticing performance issues with the > >> > client mostly. It takes almost 2-3mins to complete puppet runs on > >> > clients and takes almost 100%cpus. > > >> > I know its hard to know what the problem is unless you see what the > >> > manifests are. My question is more on what are the tools or options > >> > puppet has to look more into details on what its doing or spending > >> > most time. When i run with "--debug --test" i see it mostly doing > >> > multiples "fileserver.describe" and we are not doing any file > >> > bucketing on the server.--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
The connection timeouts probably come from the number of hosts hitting the server at the same time. You can up the timeout in both Apache and Puppet if you want to wait for longer. The general solution is to forgo the ''file'' type and use an ''exec'' instead in the case of deep directories or lots of files with ''recurse''. You can try setting checksum to ''none'' and that might make a difference but puppet will still have to record information about every file and directory in that tree. Trevor On Thu, Jul 30, 2009 at 18:43, mr<mroshan@gmail.com> wrote:> > yeah we do have a quite a few static files wit subdirectories, however > they are mounted nfs on the client and been copied, but i guess it > needs to do a checksum every run. whats the solution. and i can''t > understand why i get connection timeouts. we have puppetserver with > apache/mongrel setup. > > On Jul 30, 3:00 pm, Trevor Vaughan <peiriann...@gmail.com> wrote: >> Do you have a lot of static files that you''re copying over using the >> puppet:// protocol? >> >> If so, this is probably what is taking up most of your time. >> >> If not, are you recursively managing any folders with lots of >> subdirectories and/or files? >> >> If so, this is probably the other problem. >> >> Trevor >> >> On Thu, Jul 30, 2009 at 17:53, mr<mros...@gmail.com> wrote: >> >> > Thank for the information, yeah looks like the client spend lot of >> > time in the file operations. Do you know anyway i can troubleshoot >> > there whats its doing. turning on debug just show me >> > fileserver.describe and couple of freeze. >> >> > # puppetd -vt >> > info: Loading fact hosttype >> > info: Loading fact subdomain >> > info: Loading fact dcloc >> > notice: Ignoring --listen on onetime run >> > info: Retrieving facts >> > info: Loading fact hosttype >> > info: Loading fact subdomain >> > info: Loading fact dcloc >> > info: mount[localhost]: Mounted / >> > info: mount[modules]: Mounted >> > info: mount[plugins]: Mounted >> > info: Caching catalog at /var/lib/puppet/localconfig.yaml >> > notice: Starting catalog run >> > info: Loading fact hosttype >> > info: Loading fact subdomain >> > info: Loading fact dcloc >> > ... >> > ..... >> > Changes: >> > Total: 1 >> > Resources: >> > Applied: 1 >> > Out of sync: 1 >> > Scheduled: 1724 >> > Total: 199 >> > Time: >> > Config retrieval: 6.44 >> > Cron: 0.05 >> > Exec: 0.01 >> > File: 42.64 >> > Filebucket: 0.00 >> > Group: 0.45 >> > Mount: 0.57 >> > Package: 1.26 >> > Schedule: 0.00 >> > Service: 0.92 >> > User: 3.32 >> > Total: 55.65 >> > notice: Finished catalog run in 103.27 seconds >> >> > noticed couple of things: >> > * it computes facter multiple times, is that normal? >> > * i did a tcpdump and i see that the puppetserver send multiple FIN >> > during a session and the client reconnects, is that normal? >> >> > On Jul 29, 6:19 pm, "Jason Rojas" <ja...@nothingbeatsaduck.com> wrote: >> >> if you use summarize = true on your client configs, you should get a block >> >> at the end that looks like this: >> >> #puppetd -vt >> >> >> <snip> >> >> >> Changes: >> >> Total: 1 >> >> Resources: >> >> Applied: 1 >> >> Out of sync: 1 >> >> Restarted: 1 >> >> Scheduled: 317 >> >> Total: 266 >> >> Time: >> >> Config retrieval: 139.14 >> >> Cron: 0.16 >> >> Exec: 5.28 >> >> File: 19.54 >> >> Filebucket: 0.00 >> >> Mount: 0.27 >> >> Package: 2.87 >> >> Schedule: 0.00 >> >> Service: 3.08 >> >> User: 0.06 >> >> Yumrepo: 0.01 >> >> Total: 170.43 >> >> info: Sent transaction report in 0.28 seconds >> >> notice: Finished catalog run in 35.99 seconds >> >> >> > We have quite a bit of puppet clients handled by Mongrel server >> >> > (4instance) and off late been noticing performance issues with the >> >> > client mostly. It takes almost 2-3mins to complete puppet runs on >> >> > clients and takes almost 100%cpus. >> >> >> > I know its hard to know what the problem is unless you see what the >> >> > manifests are. My question is more on what are the tools or options >> >> > puppet has to look more into details on what its doing or spending >> >> > most time. When i run with "--debug --test" i see it mostly doing >> >> > multiples "fileserver.describe" and we are not doing any file >> >> > bucketing on the server. > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---