sanjiv singh
2011-Jul-04 12:12 UTC
[Puppet Users] puppet master and puppet client running very slow
hi all, i m using puppet 0.25.4 . since last few days, client and master are very slow puppet client taking more time in geeting configuration from puppet master. specially for those servers for which long manifest to be deployed . thanks, Sanjiv Singh Impetus Infotech -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
sanjiv.singh
2011-Jul-04 12:46 UTC
[Puppet Users] puppet master and puppet client running very slow
hi all, I have configured puppet 0.25.4 setup. since last few days , puppet master and client running very slow .It seems puppet client taking soo much time in taking configuration from master (compilation manifest ) ,specially for those node(puppet client ) for which large manifest class to be compiled and to be deployed . what can i do to over come this issue ? Thanks, Sanjiv Singh Impetus Infotech -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Daniel Maher
2011-Jul-04 13:12 UTC
Re: [Puppet Users] puppet master and puppet client running very slow
On 07/04/2011 02:46 PM, sanjiv.singh wrote:> hi all, > I have configured puppet 0.25.4 setup. > > since last few days , puppet master and client running very slow .It > seems puppet client taking soo much time in taking configuration from > master (compilation manifest ) ,specially for those node(puppet > client ) for which large manifest class to be compiled and to be > deployed . > > what can i do to over come this issue ?What sorts of steps have you already taken in order to investigate the problem? Does the Puppetmaster have enough RAM? It is otherwise overloaded from a system load perspective? What about those two factors on the client(s)? In other words, have you eliminated all of the standard system-based considerations? If so, I would recommend running the client manually with test and debug enabled, and taking a look at the output there. As well, if it''s possible, you may wish to do the same for the master daemon - that will give you a much better idea of where along the process any potential hangs are occurring. Good luck ! -- Daniel Maher « makin'' plans now to live on Mars ''cuz I got Earth on lock. » -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
sanjiv.singh
2011-Jul-04 13:35 UTC
[Puppet Users] Re: puppet master and puppet client running very slow
On Jul 4, 6:12 pm, Daniel Maher <dma...@milestonelab.com> wrote:> On 07/04/2011 02:46 PM, sanjiv.singh wrote: > > > hi all, > > I have configured puppet 0.25.4 setup. > > > since last few days , puppet master and client running very slow .It > > seems puppet client taking soo much time in taking configuration from > > master (compilation manifest ) ,specially for those node(puppet > > client ) for which large manifest class to be compiled and to be > > deployed . > > > what can i do to over come this issue ? > > What sorts of steps have you already taken in order to investigate the > problem? Does the Puppetmaster have enough RAM? It is otherwise > overloaded from a system load perspective? What about those two factors > on the client(s)? In other words, have you eliminated all of the > standard system-based considerations? > > If so, I would recommend running the client manually with test and debug > enabled, and taking a look at the output there. As well, if it''s > possible, you may wish to do the same for the master daemon - that will > give you a much better idea of where along the process any potential > hangs are occurring. > > Good luck ! > > -- > Daniel Maher > makin'' plans now to live on Mars ''cuz I got Earth on lock.Hi Daniel, I tried to run puppet client /usr/sbin/puppetd --test --debug there are more entries as following : . . . . debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw yaml; using pson debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw yaml; using pson debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw yaml; using pson its throwing same entries for long time (in minutes), prior it take few seconds to get catalog from server and get deployed. and even when i fired puppetrun on master. it takes long time to trigger puppet client (listening to master). what cuold be the issue? thanks, Sanjiv singh -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Ken Barber
2011-Jul-04 13:46 UTC
Re: [Puppet Users] Re: puppet master and puppet client running very slow
How big are these manifests (ie. number of files, total lines for all files that get imported)? For example - are you pulling in a large file of all your nodes or something? I recall there has been problems in the past with parsing large files - its fixed now though. The obvious suggestion is to upgrade :-). ken. On Mon, Jul 4, 2011 at 2:35 PM, sanjiv.singh <sanjiv.singh@impetus.co.in> wrote:> > > On Jul 4, 6:12 pm, Daniel Maher <dma...@milestonelab.com> wrote: >> On 07/04/2011 02:46 PM, sanjiv.singh wrote: >> >> > hi all, >> > I have configured puppet 0.25.4 setup. >> >> > since last few days , puppet master and client running very slow .It >> > seems puppet client taking soo much time in taking configuration from >> > master (compilation manifest ) ,specially for those node(puppet >> > client ) for which large manifest class to be compiled and to be >> > deployed . >> >> > what can i do to over come this issue ? >> >> What sorts of steps have you already taken in order to investigate the >> problem? Does the Puppetmaster have enough RAM? It is otherwise >> overloaded from a system load perspective? What about those two factors >> on the client(s)? In other words, have you eliminated all of the >> standard system-based considerations? >> >> If so, I would recommend running the client manually with test and debug >> enabled, and taking a look at the output there. As well, if it''s >> possible, you may wish to do the same for the master daemon - that will >> give you a much better idea of where along the process any potential >> hangs are occurring. >> >> Good luck ! >> >> -- >> Daniel Maher >> makin'' plans now to live on Mars ''cuz I got Earth on lock. > > > > Hi Daniel, > I tried to run puppet client > /usr/sbin/puppetd --test --debug > > > there are more entries as following : > . > . > . > . > debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw > yaml; using pson > debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw > yaml; using pson > debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw > yaml; using pson > > > its throwing same entries for long time (in minutes), prior it take > few seconds to get catalog from server and get deployed. > > and > > even when i fired puppetrun on master. > it takes long time to trigger puppet client (listening to master). > > what cuold be the issue? > > > thanks, > Sanjiv singh > > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
sanjiv.singh
2011-Jul-04 14:23 UTC
[Puppet Users] Re: puppet master and puppet client running very slow
Hi Ken, my manifest contains number of file (flat and templete ) and for pulling some jars on client. prior few days , same manifest took 2-3 minutes , now its taking 20 minutes on all puppet clients. it seems(not sure), puppet client is not bottleneck . puppet master is an issue , bcuz all puppet client facing same problem. and there is sufficient RAM and storage on puppet master. and puppet master hardly serving few (4-5) puppet client at a time in my environment. feel free to ask , if some in put require to solve issue thanks, Sanjiv Singh On Jul 4, 6:46 pm, Ken Barber <k...@puppetlabs.com> wrote:> How big are these manifests (ie. number of files, total lines for all > files that get imported)? For example - are you pulling in a large > file of all your nodes or something? I recall there has been problems > in the past with parsing large files - its fixed now though. > > The obvious suggestion is to upgrade :-). > > ken. > > > > > > > > On Mon, Jul 4, 2011 at 2:35 PM, sanjiv.singh <sanjiv.si...@impetus.co.in> wrote: > > > On Jul 4, 6:12 pm, Daniel Maher <dma...@milestonelab.com> wrote: > >> On 07/04/2011 02:46 PM, sanjiv.singh wrote: > > >> > hi all, > >> > I have configured puppet 0.25.4 setup. > > >> > since last few days , puppet master and client running very slow .It > >> > seems puppet client taking soo much time in taking configuration from > >> > master (compilation manifest ) ,specially for those node(puppet > >> > client ) for which large manifest class to be compiled and to be > >> > deployed . > > >> > what can i do to over come this issue ? > > >> What sorts of steps have you already taken in order to investigate the > >> problem? Does the Puppetmaster have enough RAM? It is otherwise > >> overloaded from a system load perspective? What about those two factors > >> on the client(s)? In other words, have you eliminated all of the > >> standard system-based considerations? > > >> If so, I would recommend running the client manually with test and debug > >> enabled, and taking a look at the output there. As well, if it''s > >> possible, you may wish to do the same for the master daemon - that will > >> give you a much better idea of where along the process any potential > >> hangs are occurring. > > >> Good luck ! > > >> -- > >> Daniel Maher > >> makin'' plans now to live on Mars ''cuz I got Earth on lock. > > > Hi Daniel, > > I tried to run puppet client > > /usr/sbin/puppetd --test --debug > > > there are more entries as following : > > . > > . > > . > > . > > debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw > > yaml; using pson > > debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw > > yaml; using pson > > debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw > > yaml; using pson > > > its throwing same entries for long time (in minutes), prior it take > > few seconds to get catalog from server and get deployed. > > > and > > > even when i fired puppetrun on master. > > it takes long time to trigger puppet client (listening to master). > > > what cuold be the issue? > > > thanks, > > Sanjiv singh > > > -- > > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > > To post to this group, send email to puppet-users@googlegroups.com. > > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > > For more options, visit this group athttp://groups.google.com/group/puppet-users?hl=en.-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Ken Barber
2011-Jul-04 14:36 UTC
Re: [Puppet Users] Re: puppet master and puppet client running very slow
> my manifest contains number of file (flat and templete ) and for > pulling some jars on client.I guess I was looking to find out the specifics of how big these manifests and templates are in total on the nodes you are having specific performance issues with. You say ''big manifests'' but this is a relative statement :-). If ''nothing has changed'' (please confirm this with your svn/git history) then it must be something else other then content or puppet itself .... Without seeing your content its a hard one to trace via a mailing list, there could be a network issue and some function you are trying to execute could be lagging or having trouble resolving dns etc. Try running your puppetmasterd in the foreground using something like: puppetmasterd --no-daemonize --debug --trace --logdest console And as you do a single run, see if it gives you any clues as to what its blocking on during those 20 minutes. ken. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
sanjiv.singh
2011-Jul-04 15:13 UTC
[Puppet Users] Re: puppet master and puppet client running very slow
On Jul 4, 7:36 pm, Ken Barber <k...@puppetlabs.com> wrote:> > my manifest contains number of file (flat and templete ) and for > > pulling some jars on client. > > I guess I was looking to find out the specifics of how big these > manifests and templates are in total on the nodes you are having > specific performance issues with. You say ''big manifests'' but this is > a relative statement :-). > > If ''nothing has changed'' (please confirm this with your svn/git > history) then it must be something else other then content or puppet > itself .... > > Without seeing your content its a hard one to trace via a mailing > list, there could be a network issue and some function you are trying > to execute could be lagging or having trouble resolving dns etc. Try > running your puppetmasterd in the foreground using something like: > > puppetmasterd --no-daemonize --debug --trace --logdest console > > And as you do a single run, see if it gives you any clues as to what > its blocking on during those 20 minutes. > > ken.Hi Ken, thanks for quick reply. as per yr suggestion , sudo /usr/sbin/puppetmasterd --no-daemonize --debug --trace --logdest console fired in puppet master........ there is noting seems wrong in debug trace...., only taking time to do stuff. these is some clue for you that may help you to find out root cause. i have configured puppet master to log in file /var/log/puppet/ masterhttp.log . . . . . [2011-07-04 10:57:18] DEBUG close: 10.122.130.1:45575 [2011-07-04 10:57:24] DEBUG Puppet::Network::HTTP::WEBrickREST is invoked. [2011-07-04 10:57:24] 209.251.184.24 - - [04/Jul/2011:10:57:24 EDT] "GET/production/file_metadata/production/tomcat-web.xml HTTP/1.1" 200 391 [2011-07-04 10:57:24] - -> /production/file_metadata/production/tomcat- web.xml [2011-07-04 10:57:24] DEBUG close: 209.251.184.24:20132 [2011-07-04 10:57:24] DEBUG accept: 10.122.130.1:8767 [2011-07-04 10:56:33] ERROR OpenSSL::SSL::SSLError: /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb: 44:in `accept'' /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb: 44:in `listen'' /usr/lib/ruby/1.8/webrick/server.rb:173:in `call'' /usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread'' /usr/lib/ruby/1.8/webrick/server.rb:162:in `start'' /usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread'' /usr/lib/ruby/1.8/webrick/server.rb:95:in `start'' /usr/lib/ruby/1.8/webrick/server.rb:92:in `each'' /usr/lib/ruby/1.8/webrick/server.rb:92:in `start'' /usr/lib/ruby/1.8/webrick/server.rb:23:in `start'' /usr/lib/ruby/1.8/webrick/server.rb:82:in `start'' /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb: 42:in `listen'' /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb: 41:in `initialize'' /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb: 41:in `new'' /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb: 41:in `listen'' /usr/lib/ruby/1.8/thread.rb:135:in `synchronize'' /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb: 38:in `listen'' /usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:131:in `listen'' /usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:146:in `start'' /usr/lib/ruby/site_ruby/1.8/puppet/daemon.rb:128:in `start'' /usr/lib/ruby/site_ruby/1.8/puppet/application/ puppetmasterd.rb:122:in `main'' /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:226:in `send'' /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:226:in `run_command'' /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:217:in `run'' /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:306:in `exit_on_fail'' /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:217:in `run'' /usr/sbin/puppetmasterd:66 there are in log repetted error trace for each file puppet master serving to client. thanks, Sanjiv singh -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Luke Bigum
2011-Jul-04 15:32 UTC
[Puppet Users] Re: puppet master and puppet client running very slow
Sanjiv, How long does your master think that catalog compilation is taking? Look for these lines: Jul 4 15:13:53 puppetmaster puppet-master[26138]: Compiled catalog for puppetclient in environment live in 0.12 seconds And then the time for the agents applying that catalog: Jul 4 15:13:55 puppetclient puppet-agent[24198]: Finished catalog run in 1.22 seconds -Luke On Jul 4, 4:13 pm, "sanjiv.singh" <sanjiv.si...@impetus.co.in> wrote:> On Jul 4, 7:36 pm, Ken Barber <k...@puppetlabs.com> wrote: > > > > > > my manifest contains number of file (flat and templete ) and for > > > pulling some jars on client. > > > I guess I was looking to find out the specifics of how big these > > manifests and templates are in total on the nodes you are having > > specific performance issues with. You say ''big manifests'' but this is > > a relative statement :-). > > > If ''nothing has changed'' (please confirm this with your svn/git > > history) then it must be something else other then content or puppet > > itself .... > > > Without seeing your content its a hard one to trace via a mailing > > list, there could be a network issue and some function you are trying > > to execute could be lagging or having trouble resolving dns etc. Try > > running your puppetmasterd in the foreground using something like: > > > puppetmasterd --no-daemonize --debug --trace --logdest console > > > And as you do a single run, see if it gives you any clues as to what > > its blocking on during those 20 minutes. > > > ken. > > Hi Ken, > thanks for quick reply. > > as per yr suggestion , > > sudo /usr/sbin/puppetmasterd --no-daemonize --debug --trace --logdest > console > > fired in puppet master........ > there is noting seems wrong in debug trace...., only taking time to > do stuff. > > these is some clue for you that may help you to find out root cause. > i have configured puppet master to log in file /var/log/puppet/ > masterhttp.log > > . > . > . > . > . > [2011-07-04 10:57:18] DEBUG close: 10.122.130.1:45575 > [2011-07-04 10:57:24] DEBUG Puppet::Network::HTTP::WEBrickREST is > invoked. > [2011-07-04 10:57:24] 209.251.184.24 - - [04/Jul/2011:10:57:24 EDT] > "GET/production/file_metadata/production/tomcat-web.xml HTTP/1.1" 200 > 391 > [2011-07-04 10:57:24] - -> /production/file_metadata/production/tomcat- > web.xml > [2011-07-04 10:57:24] DEBUG close: 209.251.184.24:20132 > [2011-07-04 10:57:24] DEBUG accept: 10.122.130.1:8767 > [2011-07-04 10:56:33] ERROR OpenSSL::SSL::SSLError: > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb: > 44:in `accept'' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb: > 44:in `listen'' > /usr/lib/ruby/1.8/webrick/server.rb:173:in `call'' > /usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread'' > /usr/lib/ruby/1.8/webrick/server.rb:162:in `start'' > /usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread'' > /usr/lib/ruby/1.8/webrick/server.rb:95:in `start'' > /usr/lib/ruby/1.8/webrick/server.rb:92:in `each'' > /usr/lib/ruby/1.8/webrick/server.rb:92:in `start'' > /usr/lib/ruby/1.8/webrick/server.rb:23:in `start'' > /usr/lib/ruby/1.8/webrick/server.rb:82:in `start'' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb: > 42:in `listen'' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb: > 41:in `initialize'' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb: > 41:in `new'' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb: > 41:in `listen'' > /usr/lib/ruby/1.8/thread.rb:135:in `synchronize'' > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb: > 38:in `listen'' > /usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:131:in > `listen'' > /usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:146:in > `start'' > /usr/lib/ruby/site_ruby/1.8/puppet/daemon.rb:128:in `start'' > /usr/lib/ruby/site_ruby/1.8/puppet/application/ > puppetmasterd.rb:122:in `main'' > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:226:in > `send'' > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:226:in > `run_command'' > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:217:in `run'' > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:306:in > `exit_on_fail'' > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:217:in `run'' > /usr/sbin/puppetmasterd:66 > > there are in log repetted error trace for each file puppet master > serving to client. > > thanks, > Sanjiv singh-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
sanjiv.singh
2011-Jul-04 16:12 UTC
[Puppet Users] Re: puppet master and puppet client running very slow
Hi luke, thanks for quick reply. as i m able to see that puppet master taking around 7-8 seconds to compiler catalog and puppet client taking around 1106 second to apply catalog . seems strange , because some manifest and configuration (on both master and client) working fine with dozens of servers concurrently. Since Sunday getting strange behavior........ feel free to ask, if some more input required to resolve d issue. thanks, Sanjiv Singh On Jul 4, 8:32 pm, Luke Bigum <Luke.Bi...@lmax.com> wrote:> Sanjiv, > > How long does your master think that catalog compilation is taking? > Look for these lines: > > Jul 4 15:13:53 puppetmaster puppet-master[26138]: Compiled catalog > for puppetclient in environment live in 0.12 seconds > > And then the time for the agents applying that catalog: > > Jul 4 15:13:55 puppetclient puppet-agent[24198]: Finished catalog run > in 1.22 seconds >> -Luke > > On Jul 4, 4:13 pm, "sanjiv.singh" <sanjiv.si...@impetus.co.in> wrote: > > > > > > > > > On Jul 4, 7:36 pm, Ken Barber <k...@puppetlabs.com> wrote: > > > > > my manifest contains number of file (flat and templete ) and for > > > > pulling some jars on client. > > > > I guess I was looking to find out the specifics of how big these > > > manifests and templates are in total on the nodes you are having > > > specific performance issues with. You say ''big manifests'' but this is > > > a relative statement :-). > > > > If ''nothing has changed'' (please confirm this with your svn/git > > > history) then it must be something else other then content or puppet > > > itself .... > > > > Without seeing your content its a hard one to trace via a mailing > > > list, there could be a network issue and some function you are trying > > > to execute could be lagging or having trouble resolving dns etc. Try > > > running your puppetmasterd in the foreground using something like: > > > > puppetmasterd --no-daemonize --debug --trace --logdest console > > > > And as you do a single run, see if it gives you any clues as to what > > > its blocking on during those 20 minutes. > > > > ken. > > > Hi Ken, > > thanks for quick reply. > > > as per yr suggestion , > > > sudo /usr/sbin/puppetmasterd --no-daemonize --debug --trace --logdest > > console > > > fired in puppet master........ > > there is noting seems wrong in debug trace...., only taking time to > > do stuff. > > > these is some clue for you that may help you to find out root cause. > > i have configured puppet master to log in file /var/log/puppet/ > > masterhttp.log > > > . > > . > > . > > . > > . > > [2011-07-04 10:57:18] DEBUG close: 10.122.130.1:45575 > > [2011-07-04 10:57:24] DEBUG Puppet::Network::HTTP::WEBrickREST is > > invoked. > > [2011-07-04 10:57:24] 209.251.184.24 - - [04/Jul/2011:10:57:24 EDT] > > "GET/production/file_metadata/production/tomcat-web.xml HTTP/1.1" 200 > > 391 > > [2011-07-04 10:57:24] - -> /production/file_metadata/production/tomcat- > > web.xml > > [2011-07-04 10:57:24] DEBUG close: 209.251.184.24:20132 > > [2011-07-04 10:57:24] DEBUG accept: 10.122.130.1:8767 > > [2011-07-04 10:56:33] ERROR OpenSSL::SSL::SSLError: > > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb: > > 44:in `accept'' > > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb: > > 44:in `listen'' > > /usr/lib/ruby/1.8/webrick/server.rb:173:in `call'' > > /usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread'' > > /usr/lib/ruby/1.8/webrick/server.rb:162:in `start'' > > /usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread'' > > /usr/lib/ruby/1.8/webrick/server.rb:95:in `start'' > > /usr/lib/ruby/1.8/webrick/server.rb:92:in `each'' > > /usr/lib/ruby/1.8/webrick/server.rb:92:in `start'' > > /usr/lib/ruby/1.8/webrick/server.rb:23:in `start'' > > /usr/lib/ruby/1.8/webrick/server.rb:82:in `start'' > > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb: > > 42:in `listen'' > > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb: > > 41:in `initialize'' > > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb: > > 41:in `new'' > > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb: > > 41:in `listen'' > > /usr/lib/ruby/1.8/thread.rb:135:in `synchronize'' > > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb: > > 38:in `listen'' > > /usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:131:in > > `listen'' > > /usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:146:in > > `start'' > > /usr/lib/ruby/site_ruby/1.8/puppet/daemon.rb:128:in `start'' > > /usr/lib/ruby/site_ruby/1.8/puppet/application/ > > puppetmasterd.rb:122:in `main'' > > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:226:in > > `send'' > > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:226:in > > `run_command'' > > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:217:in `run'' > > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:306:in > > `exit_on_fail'' > > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:217:in `run'' > > /usr/sbin/puppetmasterd:66 > > > there are in log repetted error trace for each file puppet master > > serving to client. > > > thanks, > > Sanjiv singh-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Peter Meier
2011-Jul-04 17:37 UTC
Re: [Puppet Users] Re: puppet master and puppet client running very slow
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1> as i m able to see that puppet master taking around 7-8 seconds to > compiler catalog > and puppet client taking around 1106 second to apply catalog .run the client with --summarize and see on which resource types it is spending most of the time, then you can probably figure out what is wrong. also analyzing the reports with something like the dashboard or the foreman can help to trace these issues. Certainly something within your environment or its dependencies has changed. Puppet is heavily calling out to tools like facter, package managers, service scripts etc. which all can easily affect your puppet runs. ~pete -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk4R+mcACgkQbwltcAfKi3/d/QCeODSqHvF8OxULMcbw6gfuEOT9 noMAn2b0DySrqrZ27kA6Xfvb1WaVWKdX =pHX/ -----END PGP SIGNATURE----- -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Ken Barber
2011-Jul-04 18:10 UTC
Re: [Puppet Users] Re: puppet master and puppet client running very slow
Can you try the puppet run with the setting --summarize and show us the result ... (not sure if that flag was round then). ken. On Mon, Jul 4, 2011 at 5:12 PM, sanjiv.singh <sanjiv.singh@impetus.co.in> wrote:> > > Hi luke, > > thanks for quick reply. > > as i m able to see that puppet master taking around 7-8 seconds to > compiler catalog > and puppet client taking around 1106 second to apply catalog . > > seems strange , because some manifest and configuration (on both > master and client) working fine with dozens of servers concurrently. > Since Sunday getting strange behavior........ > > feel free to ask, if some more input required to resolve d issue. > > thanks, > Sanjiv Singh > > > On Jul 4, 8:32 pm, Luke Bigum <Luke.Bi...@lmax.com> wrote: >> Sanjiv, >> >> How long does your master think that catalog compilation is taking? >> Look for these lines: >> >> Jul 4 15:13:53 puppetmaster puppet-master[26138]: Compiled catalog >> for puppetclient in environment live in 0.12 seconds >> >> And then the time for the agents applying that catalog: >> >> Jul 4 15:13:55 puppetclient puppet-agent[24198]: Finished catalog run >> in 1.22 seconds >> > >> -Luke >> >> On Jul 4, 4:13 pm, "sanjiv.singh" <sanjiv.si...@impetus.co.in> wrote: >> >> >> >> >> >> >> >> > On Jul 4, 7:36 pm, Ken Barber <k...@puppetlabs.com> wrote: >> >> > > > my manifest contains number of file (flat and templete ) and for >> > > > pulling some jars on client. >> >> > > I guess I was looking to find out the specifics of how big these >> > > manifests and templates are in total on the nodes you are having >> > > specific performance issues with. You say ''big manifests'' but this is >> > > a relative statement :-). >> >> > > If ''nothing has changed'' (please confirm this with your svn/git >> > > history) then it must be something else other then content or puppet >> > > itself .... >> >> > > Without seeing your content its a hard one to trace via a mailing >> > > list, there could be a network issue and some function you are trying >> > > to execute could be lagging or having trouble resolving dns etc. Try >> > > running your puppetmasterd in the foreground using something like: >> >> > > puppetmasterd --no-daemonize --debug --trace --logdest console >> >> > > And as you do a single run, see if it gives you any clues as to what >> > > its blocking on during those 20 minutes. >> >> > > ken. >> >> > Hi Ken, >> > thanks for quick reply. >> >> > as per yr suggestion , >> >> > sudo /usr/sbin/puppetmasterd --no-daemonize --debug --trace --logdest >> > console >> >> > fired in puppet master........ >> > there is noting seems wrong in debug trace...., only taking time to >> > do stuff. >> >> > these is some clue for you that may help you to find out root cause. >> > i have configured puppet master to log in file /var/log/puppet/ >> > masterhttp.log >> >> > . >> > . >> > . >> > . >> > . >> > [2011-07-04 10:57:18] DEBUG close: 10.122.130.1:45575 >> > [2011-07-04 10:57:24] DEBUG Puppet::Network::HTTP::WEBrickREST is >> > invoked. >> > [2011-07-04 10:57:24] 209.251.184.24 - - [04/Jul/2011:10:57:24 EDT] >> > "GET/production/file_metadata/production/tomcat-web.xml HTTP/1.1" 200 >> > 391 >> > [2011-07-04 10:57:24] - -> /production/file_metadata/production/tomcat- >> > web.xml >> > [2011-07-04 10:57:24] DEBUG close: 209.251.184.24:20132 >> > [2011-07-04 10:57:24] DEBUG accept: 10.122.130.1:8767 >> > [2011-07-04 10:56:33] ERROR OpenSSL::SSL::SSLError: >> > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb: >> > 44:in `accept'' >> > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb: >> > 44:in `listen'' >> > /usr/lib/ruby/1.8/webrick/server.rb:173:in `call'' >> > /usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread'' >> > /usr/lib/ruby/1.8/webrick/server.rb:162:in `start'' >> > /usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread'' >> > /usr/lib/ruby/1.8/webrick/server.rb:95:in `start'' >> > /usr/lib/ruby/1.8/webrick/server.rb:92:in `each'' >> > /usr/lib/ruby/1.8/webrick/server.rb:92:in `start'' >> > /usr/lib/ruby/1.8/webrick/server.rb:23:in `start'' >> > /usr/lib/ruby/1.8/webrick/server.rb:82:in `start'' >> > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb: >> > 42:in `listen'' >> > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb: >> > 41:in `initialize'' >> > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb: >> > 41:in `new'' >> > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb: >> > 41:in `listen'' >> > /usr/lib/ruby/1.8/thread.rb:135:in `synchronize'' >> > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb: >> > 38:in `listen'' >> > /usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:131:in >> > `listen'' >> > /usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:146:in >> > `start'' >> > /usr/lib/ruby/site_ruby/1.8/puppet/daemon.rb:128:in `start'' >> > /usr/lib/ruby/site_ruby/1.8/puppet/application/ >> > puppetmasterd.rb:122:in `main'' >> > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:226:in >> > `send'' >> > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:226:in >> > `run_command'' >> > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:217:in `run'' >> > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:306:in >> > `exit_on_fail'' >> > /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:217:in `run'' >> > /usr/sbin/puppetmasterd:66 >> >> > there are in log repetted error trace for each file puppet master >> > serving to client. >> >> > thanks, >> > Sanjiv singh > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Patrick
2011-Jul-04 19:16 UTC
Re: [Puppet Users] Re: puppet master and puppet client running very slow
On Jul 4, 2011, at 6:35 AM, sanjiv.singh wrote:> > > On Jul 4, 6:12 pm, Daniel Maher <dma...@milestonelab.com> wrote: >> On 07/04/2011 02:46 PM, sanjiv.singh wrote: >> >>> hi all, >>> I have configured puppet 0.25.4 setup. >> >>> since last few days , puppet master and client running very slow .It >>> seems puppet client taking soo much time in taking configuration from >>> master (compilation manifest ) ,specially for those node(puppet >>> client ) for which large manifest class to be compiled and to be >>> deployed . >> >>> what can i do to over come this issue ? >> >> What sorts of steps have you already taken in order to investigate the >> problem? Does the Puppetmaster have enough RAM? It is otherwise >> overloaded from a system load perspective? What about those two factors >> on the client(s)? In other words, have you eliminated all of the >> standard system-based considerations? >> >> If so, I would recommend running the client manually with test and debug >> enabled, and taking a look at the output there. As well, if it''s >> possible, you may wish to do the same for the master daemon - that will >> give you a much better idea of where along the process any potential >> hangs are occurring. >> >> Good luck ! >> >> -- >> Daniel Maher >> makin'' plans now to live on Mars ''cuz I got Earth on lock. > > > > Hi Daniel, > I tried to run puppet client > /usr/sbin/puppetd --test --debug > > > there are more entries as following : > . > . > . > . > debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw > yaml; using pson > debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw > yaml; using pson > debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw > yaml; using pson > > > its throwing same entries for long time (in minutes), prior it take > few seconds to get catalog from server and get deployed.I''m pretty sure those entries you are looking at are file resources. In general, file resources scale very poorly with webrick. Even without webrick, they still tend to perform poorly. My advice: Stop using webrick Take a look at recuse=>remote if you use recursive file copies a lot Make sure you''re not swapping and that you have enough ram Consider copying big files or directories with a lot of files with a package or tar+wget. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
sanjiv.singh
2011-Jul-05 03:35 UTC
[Puppet Users] Re: puppet master and puppet client running very slow
On Jul 4, 10:37 pm, Peter Meier <peter.me...@immerda.ch> wrote:> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > > as i m able to see that puppet master taking around 7-8 seconds to > > compiler catalog > > and puppet client taking around 1106 second to apply catalog . > > run the client with --summarize and see on which resource types it is > spending most of the time, then you can probably figure out what is wrong. > > also analyzing the reports with something like the dashboard or the > foreman can help to trace these issues. > > Certainly something within your environment or its dependencies has > changed. Puppet is heavily calling out to tools like facter, package > managers, service scripts etc. which all can easily affect your puppet runs. > > ~pete > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.11 (GNU/Linux) > Comment: Using GnuPG with Mozilla -http://enigmail.mozdev.org/ > > iEYEARECAAYFAk4R+mcACgkQbwltcAfKi3/d/QCeODSqHvF8OxULMcbw6gfuEOT9 > noMAn2b0DySrqrZ27kA6Xfvb1WaVWKdX > =pHX/ > -----END PGP SIGNATURE-----Hi Peter, thanks for quick reply . as per yr suggestion , i have tried --summarize option on both puppet master and client. here is all the details, (on puppet master)>> sudo /usr/sbin/puppetrun -c generic::pb11-8Maintenance --summarizegeneric::pb11-8Maintenance: pb11-8.puppet.com Triggering pb11-8.puppet.com pb11-8.puppet.com finished with exit code 0 Finished but , it took around 5-6 minutes only to trigger puppet client(listen for puppet master). (on puppet client ) >> sudo /usr/sbin/puppetd --test --debug --summarize . . . debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw yaml; using pson debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw yaml; using pson debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw yaml; using pson debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw yaml; using pson debug: //templatebasedapplicationconfigs/File[/var/lib/dmb/conf/ memcached.properties]/content: Executing ''diff -u /var/lib/dmb/conf/ memcached.properties /tmp/puppet-diffing.8146.0'' --- /var/lib/dmb/conf/memcached.properties 2011-07-04 12:55:13.000000000 -0400 +++ /tmp/puppet-diffing.8146.0 2011-07-04 23:13:08.000000000 -0400 @@ -1,4 +1,4 @@ -memcached.servers=10.122.215.42:11211,10.122.215.41:11211 +memcached.servers=pb11-9:10.122.215.42,pb11-8:10.122.215.41:11211 memcached.weights=2 memcached.init.connections=5 memcached.min.connections=5 debug: //templatebasedapplicationconfigs/File[/var/lib/dmb/conf/ memcached.properties]: Changing checksum,content debug: //templatebasedapplicationconfigs/File[/var/lib/dmb/conf/ memcached.properties]: 2 change(s) debug: //templatebasedapplicationconfigs/File[/var/lib/dmb/conf/ memcached.properties]/checksum: Replacing /var/lib/dmb/conf/ memcached.properties checksum {md5}c428ee9bfadf4893267d39585ecac0fd with {md5}496f78a690517e9d4a3a05c7248de0b2 notice: //templatebasedapplicationconfigs/File[/var/lib/dmb/conf/ memcached.properties]/checksum: checksum changed ''{md5} c428ee9bfadf4893267d39585ecac0fd'' to ''{md5} 496f78a690517e9d4a3a05c7248de0b2'' info: //templatebasedapplicationconfigs/File[/var/lib/dmb/conf/ memcached.properties]: Filebucketed /var/lib/dmb/conf/ memcached.properties to puppet with sum 496f78a690517e9d4a3a05c7248de0b2 debug: //templatebasedapplicationconfigs/File[/var/lib/dmb/conf/ memcached.properties]/checksum: Replacing /var/lib/dmb/conf/ memcached.properties checksum {md5}496f78a690517e9d4a3a05c7248de0b2 with {md5}c428ee9bfadf4893267d39585ecac0fd notice: //templatebasedapplicationconfigs/File[/var/lib/dmb/conf/ memcached.properties]/content: content changed ''{md5} 496f78a690517e9d4a3a05c7248de0b2'' to ''unknown checksum'' .. debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw yaml; using pson debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw yaml; using pson debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw yaml; using pson debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw yaml; using pson debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw yaml; using pson debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw yaml; using pson debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw yaml; using pson debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw yaml; using pson debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw yaml; using pson debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw yaml; using pson debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw yaml; using pson debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw yaml; using pson debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw yaml; using pson debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw yaml; using pson . . . . . debug: Time for triggering 1 events to edges: 0.000236988067626953 debug: Finishing transaction 23607949333420 with 16 changes debug: Storing state debug: Stored state in 0.08 seconds Changes: Total: 16 Resources: Applied: 16 Out of sync: 14 Scheduled: 128 Total: 211 Time: Config retrieval: 130.36 Exec: 2.39 File: 1539.03 Filebucket: 0.00 Schedule: 0.00 Total: 1671.78 notice: Finished catalog run in 1542.26 seconds it seems, line debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw yaml; using pson thrown repeatedly wasting most of the time in it. is it some thing that can help you out to trace problem? thanks, Sanjiv Singh -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
sanjiv.singh
2011-Jul-05 04:11 UTC
[Puppet Users] Re: puppet master and puppet client running very slow
some times puppet client fails with following....... debug: Loaded state in 0.01 seconds debug: Using cached certificate for ca debug: Using cached certificate for pb11-9.domain.local debug: Using cached certificate_revocation_list for ca debug: catalog supports formats: b64_zlib_yaml marshal pson raw yaml; using pson err: Could not retrieve catalog from remote server: Could not intern from pson: expected value in object at ''"File[/var/lib/''! warning: Not using cache on failed catalog err: Could not retrieve catalog; skipping run thanks, Sanjiv Singh On Jul 5, 8:35 am, "sanjiv.singh" <sanjiv.si...@impetus.co.in> wrote:> On Jul 4, 10:37 pm, Peter Meier <peter.me...@immerda.ch> wrote: > > > > > > > > > > > -----BEGIN PGP SIGNED MESSAGE----- > > Hash: SHA1 > > > > as i m able to see that puppet master taking around 7-8 seconds to > > > compiler catalog > > > and puppet client taking around 1106 second to apply catalog . > > > run the client with --summarize and see on which resource types it is > > spending most of the time, then you can probably figure out what is wrong. > > > also analyzing the reports with something like the dashboard or the > > foreman can help to trace these issues. > > > Certainly something within your environment or its dependencies has > > changed. Puppet is heavily calling out to tools like facter, package > > managers, service scripts etc. which all can easily affect your puppet runs. > > > ~pete > > -----BEGIN PGP SIGNATURE----- > > Version: GnuPG v1.4.11 (GNU/Linux) > > Comment: Using GnuPG with Mozilla -http://enigmail.mozdev.org/ > > > iEYEARECAAYFAk4R+mcACgkQbwltcAfKi3/d/QCeODSqHvF8OxULMcbw6gfuEOT9 > > noMAn2b0DySrqrZ27kA6Xfvb1WaVWKdX > > =pHX/ > > -----END PGP SIGNATURE----- > > Hi Peter, > > thanks for quick reply . > > as per yr suggestion , i have tried --summarize option on both puppet > master and client. > > here is all the details, > > (on puppet master) > > >> sudo /usr/sbin/puppetrun -c generic::pb11-8Maintenance --summarize > > generic::pb11-8Maintenance: pb11-8.puppet.com > Triggering pb11-8.puppet.com > pb11-8.puppet.com finished with exit code 0 > Finished > > but , it took around 5-6 minutes only to trigger puppet client(listen > for puppet master). > > (on puppet client ) > >> sudo /usr/sbin/puppetd --test --debug --summarize > . > . > . > debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw > yaml; using pson > debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw > yaml; using pson > debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw > yaml; using pson > debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw > yaml; using pson > debug: //templatebasedapplicationconfigs/File[/var/lib/dmb/conf/ > memcached.properties]/content: Executing ''diff -u /var/lib/dmb/conf/ > memcached.properties /tmp/puppet-diffing.8146.0'' > --- /var/lib/dmb/conf/memcached.properties 2011-07-04 > 12:55:13.000000000 -0400 > +++ /tmp/puppet-diffing.8146.0 2011-07-04 23:13:08.000000000 -0400 > @@ -1,4 +1,4 @@ > -memcached.servers=10.122.215.42:11211,10.122.215.41:11211 > +memcached.servers=pb11-9:10.122.215.42,pb11-8:10.122.215.41:11211 > memcached.weights=2 > memcached.init.connections=5 > memcached.min.connections=5 > debug: //templatebasedapplicationconfigs/File[/var/lib/dmb/conf/ > memcached.properties]: Changing checksum,content > debug: //templatebasedapplicationconfigs/File[/var/lib/dmb/conf/ > memcached.properties]: 2 change(s) > debug: //templatebasedapplicationconfigs/File[/var/lib/dmb/conf/ > memcached.properties]/checksum: Replacing /var/lib/dmb/conf/ > memcached.properties checksum {md5}c428ee9bfadf4893267d39585ecac0fd > with {md5}496f78a690517e9d4a3a05c7248de0b2 > notice: //templatebasedapplicationconfigs/File[/var/lib/dmb/conf/ > memcached.properties]/checksum: checksum changed ''{md5} > c428ee9bfadf4893267d39585ecac0fd'' to ''{md5} > 496f78a690517e9d4a3a05c7248de0b2'' > info: //templatebasedapplicationconfigs/File[/var/lib/dmb/conf/ > memcached.properties]: Filebucketed /var/lib/dmb/conf/ > memcached.properties to puppet with sum > 496f78a690517e9d4a3a05c7248de0b2 > debug: //templatebasedapplicationconfigs/File[/var/lib/dmb/conf/ > memcached.properties]/checksum: Replacing /var/lib/dmb/conf/ > memcached.properties checksum {md5}496f78a690517e9d4a3a05c7248de0b2 > with {md5}c428ee9bfadf4893267d39585ecac0fd > notice: //templatebasedapplicationconfigs/File[/var/lib/dmb/conf/ > memcached.properties]/content: content changed ''{md5} > 496f78a690517e9d4a3a05c7248de0b2'' to ''unknown checksum'' > > .. > debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw > yaml; using pson > debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw > yaml; using pson > debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw > yaml; using pson > debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw > yaml; using pson > debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw > yaml; using pson > debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw > yaml; using pson > debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw > yaml; using pson > debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw > yaml; using pson > debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw > yaml; using pson > debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw > yaml; using pson > debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw > yaml; using pson > debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw > yaml; using pson > debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw > yaml; using pson > debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw > yaml; using pson > . > . > . > . > . > debug: Time for triggering 1 events to edges: 0.000236988067626953 > debug: Finishing transaction 23607949333420 with 16 changes > debug: Storing state > debug: Stored state in 0.08 seconds > Changes: > Total: 16 > Resources: > Applied: 16 > Out of sync: 14 > Scheduled: 128 > Total: 211 > Time: > Config retrieval: 130.36 > Exec: 2.39 > File: 1539.03 > Filebucket: 0.00 > Schedule: 0.00 > Total: 1671.78 > notice: Finished catalog run in 1542.26 seconds > > it seems, > line > > debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw > yaml; using pson > > thrown repeatedly wasting most of the time in it. > > is it some thing that can help you out to trace problem? > > thanks, > Sanjiv Singh-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
vagn scott
2011-Jul-05 04:23 UTC
Re: [Puppet Users] Re: puppet master and puppet client running very slow
Is the client healthy? File system not full? No errors in log files, like a disk might be failing? Load average reasonable? Any network changes? Do you have a switch plugged into itself? what does tcpdump show? Normal activity? -- vagn On 07/05/2011 12:11 AM, sanjiv.singh wrote:> some times puppet client fails with following....... > > debug: Loaded state in 0.01 seconds > debug: Using cached certificate for ca > debug: Using cached certificate for pb11-9.domain.local > debug: Using cached certificate_revocation_list for ca > debug: catalog supports formats: b64_zlib_yaml marshal pson raw yaml; > using pson > err: Could not retrieve catalog from remote server: Could not intern > from pson: expected value in object at ''"File[/var/lib/''! > warning: Not using cache on failed catalog > err: Could not retrieve catalog; skipping run > > thanks, > Sanjiv Singh >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Peter Meier
2011-Jul-05 05:53 UTC
Re: [Puppet Users] Re: puppet master and puppet client running very slow
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1> File: 1539.03This indicates clearly a problem in serving files, so does the other error you posted. As others asked: Are you still running with Webrick (means default puppetmaster without setting up passenger/mongrel)? And how many clients are connecting to the server how frequently? ~pete -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk4SpsUACgkQbwltcAfKi3+AeQCfVKw/DBKcMIlCzMfGK7gF0xpK 0BYAn3ph1iWV8YjO5WxuEzm7bhucvAU+ =Aj9t -----END PGP SIGNATURE----- -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
sanjiv.singh
2011-Jul-05 07:50 UTC
[Puppet Users] Re: puppet master and puppet client running very slow
Hi Peter, thanks for quick reply. ya i am using Webrick default with puppetmaster. As per performance guild line,for servicing more nodes it is recommended to move on passenger/mongrel. surely i will do this. There are around 100+ servers in my environment.but at most puppet master serving to 10-12 nodes (varies) concurrently. Thanks, Sanjiv Singh Impetus Info-tech. On Jul 5, 10:53 am, Peter Meier <peter.me...@immerda.ch> wrote:> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > > File: 1539.03 > > This indicates clearly a problem in serving files, so does the other > error you posted. > > As others asked: Are you still running with Webrick (means default > puppetmaster without setting up passenger/mongrel)? And how many > clients are connecting to the server how frequently? > > ~pete > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.11 (GNU/Linux) > Comment: Using GnuPG with Mozilla -http://enigmail.mozdev.org/ > > iEYEARECAAYFAk4SpsUACgkQbwltcAfKi3+AeQCfVKw/DBKcMIlCzMfGK7gF0xpK > 0BYAn3ph1iWV8YjO5WxuEzm7bhucvAU+ > =Aj9t > -----END PGP SIGNATURE------- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
sanjiv.singh
2011-Jul-05 07:58 UTC
[Puppet Users] Re: puppet master and puppet client running very slow
hi all,
I am using environment option in /etc/puppet/puppet.conf for
supporting various releases.
like
.
.
[release1]
modulepath = /usr/local/applications/release1
[release2]
modulepath = /usr/local/applications/release2
[release3]
modulepath = /usr/local/applications/release3
.
.
As i added new release [YYYY] with addition of few files/templates and
jars,
it seems following issue coming with only new release [YYYY].
what can be various way with puppet , if i wanna to do alot file
serving for puppet client ?
thanks,
Sanjiv Singh
Impetus Infotech
On Jul 5, 12:50 pm, "sanjiv.singh" <sanjiv.si...@impetus.co.in>
wrote:> Hi Peter,
> thanks for quick reply.
> ya i am using Webrick default with puppetmaster.
>
> As per performance guild line,for servicing more nodes it is
> recommended to move on passenger/mongrel. surely i will do this.
>
> There are around 100+ servers in my environment.but at most puppet
> master serving to 10-12 nodes (varies) concurrently.
>
> Thanks,
> Sanjiv Singh
> Impetus Info-tech.
>
> On Jul 5, 10:53 am, Peter Meier <peter.me...@immerda.ch> wrote:
>
>
>
>
>
>
>
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
>
> > > File: 1539.03
>
> > This indicates clearly a problem in serving files, so does the other
> > error you posted.
>
> > As others asked: Are you still running with Webrick (means default
> > puppetmaster without setting up passenger/mongrel)? And how many
> > clients are connecting to the server how frequently?
>
> > ~pete
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG v1.4.11 (GNU/Linux)
> > Comment: Using GnuPG with Mozilla -http://enigmail.mozdev.org/
>
> > iEYEARECAAYFAk4SpsUACgkQbwltcAfKi3+AeQCfVKw/DBKcMIlCzMfGK7gF0xpK
> > 0BYAn3ph1iWV8YjO5WxuEzm7bhucvAU+
> > =Aj9t
> > -----END PGP SIGNATURE-----
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
Ken Barber
2011-Jul-05 08:32 UTC
Re: [Puppet Users] Re: puppet master and puppet client running very slow
FYI - for performance I would upgrade Puppet (there were performance
improvements in newer revisions), and use something like
Apache/Passenger as a delivery mechanism.
One question - does this happen each time you run it on the same box?
If there is no new changes - is the performance still terrible or is
it only for systems where there is a new change?
I''m just pondering also - did you introduce a new large file?
I''m just
wondering if this time is spent doing an md5 sum of a huge file for
file {} comparison. You see the file provider will do an md5
comparison to make sure a a file that already exists doesn''t get
installed again. If the file is large - then it takes a long time.
Does this slowness happen even when it appears no changes need to be
made? If you think a file is a culprit you can emulate the md5
behaviour by running ''time md5sum bigassfile.iso'' to see if it
takes a
long time.
Otherwise its the delivery of such a large file ... which might be the
case, and you definitely should switch out your stack.
I see you have 211 resources or so ... so its not the quantity of
files here that is the problem I would guess.
> There are around 100+ servers in my environment.but at most puppet
> master serving to 10-12 nodes (varies) concurrently.
Webrick at best can manage 1 node concurrently properly. So the minute
you fire up more then 1 it will slow down as it tries to serve each
client when available. Switch to Apache/Mongrel :-).
ken.
On Tue, Jul 5, 2011 at 8:58 AM, sanjiv.singh <sanjiv.singh@impetus.co.in>
wrote:> hi all,
> I am using environment option in /etc/puppet/puppet.conf for
> supporting various releases.
> like
> .
> .
> [release1]
> modulepath = /usr/local/applications/release1
> [release2]
> modulepath = /usr/local/applications/release2
> [release3]
> modulepath = /usr/local/applications/release3
> .
> .
> As i added new release [YYYY] with addition of few files/templates and
> jars,
> it seems following issue coming with only new release [YYYY].
>
> what can be various way with puppet , if i wanna to do alot file
> serving for puppet client ?
>
>
> thanks,
> Sanjiv Singh
> Impetus Infotech
>
>
> On Jul 5, 12:50 pm, "sanjiv.singh"
<sanjiv.si...@impetus.co.in> wrote:
>> Hi Peter,
>> thanks for quick reply.
>> ya i am using Webrick default with puppetmaster.
>>
>> As per performance guild line,for servicing more nodes it is
>> recommended to move on passenger/mongrel. surely i will do this.
>>
>> There are around 100+ servers in my environment.but at most puppet
>> master serving to 10-12 nodes (varies) concurrently.
>>
>> Thanks,
>> Sanjiv Singh
>> Impetus Info-tech.
>>
>> On Jul 5, 10:53 am, Peter Meier <peter.me...@immerda.ch> wrote:
>>
>>
>>
>>
>>
>>
>>
>> > -----BEGIN PGP SIGNED MESSAGE-----
>> > Hash: SHA1
>>
>> > > File: 1539.03
>>
>> > This indicates clearly a problem in serving files, so does the
other
>> > error you posted.
>>
>> > As others asked: Are you still running with Webrick (means default
>> > puppetmaster without setting up passenger/mongrel)? And how many
>> > clients are connecting to the server how frequently?
>>
>> > ~pete
>> > -----BEGIN PGP SIGNATURE-----
>> > Version: GnuPG v1.4.11 (GNU/Linux)
>> > Comment: Using GnuPG with Mozilla -http://enigmail.mozdev.org/
>>
>> > iEYEARECAAYFAk4SpsUACgkQbwltcAfKi3+AeQCfVKw/DBKcMIlCzMfGK7gF0xpK
>> > 0BYAn3ph1iWV8YjO5WxuEzm7bhucvAU+
>> > =Aj9t
>> > -----END PGP SIGNATURE-----
>
> --
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.