Hello, I have 680 hosts reporting into a central puppet server. Yesterday I made a change to site.pp and the nodes/foo.pp and nodes/ bar.pp files to push out some new files (a script to enable LDAP) and it seems that only a handful of stores have downloaded the file and run the script. (I don''t have reporting setup yet other than on a few clients which I did manually) If I ssh into a client and run ''puppetd --test --verbose --debug'' it will in fact download the script to enable LDAP and run it with no problem. My understanding was that puppetd "checks in" with the puppetmaster server every 30 minutes (based on the runinterval variable in puppet.conf, which I''ve left at default) and if there were new files "waiting" for it, it would then do whatever, download them, etc. I restarted puppetmasterd on the puppetmaster box, and as I said I can manually run puppetd on a box and it will do the right thing and download the script and run it, but clearly logging into each box and manually doing that isn''t an option, otherwise I don''t need puppet ;) Am I missing something obvious here? The clients are all running puppet-0.24.4 with facter-1.3.8 and ruby 1.8.6. The server is running puppet-0.24.5, facter-1.5.2, and ruby 1.8.6. Thanks as always, Josh --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
I should have mentioned, I''m running four mongrel instances fronted by Apache/2.2.8 on the puppetmaster server for serving the clients. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
I would do something like an ssh for loop that looked for the log entries from the clients to see what they''re doing: for i in $NODES ssh $i grep puppetd /var/log/messages etc On Sep 17, 9:48 am, josh <dor...@gmail.com> wrote:> I should have mentioned, I''m running four mongrel instances fronted by > Apache/2.2.8 on the puppetmaster server for serving the clients.--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
With almost 700 nodes this isn''t an option. The whole point of installing puppet in the first place was to not have to do this. And like I said, if I run puppetd manually on each box it gets the files, runs the scripts, etc. etc. Also due to inconsistencies in the syslog config on these boxes, not all of them have puppet logging to syslog the right way. Maybe I will just configure puppet to log to a file locally on the box instead of via syslog and see if I can get some more insight that way. Josh On Sep 17, 12:23 pm, chalex <alex.chekho...@gmail.com> wrote:> I would do something like an ssh for loop that looked for the log > entries from the clients to see what they''re doing: > > for i in $NODES > ssh $i grep puppetd /var/log/messages > etc > > On Sep 17, 9:48 am, josh <dor...@gmail.com> wrote: > > > I should have mentioned, I''m running four mongrel instances fronted by > > Apache/2.2.8 on the puppetmaster server for serving the clients.--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
You''ll definitely be wanting to setup reporting, even just the log type to get your clients puppet logs back on the master. On 18/09/2008, at 7:31 AM, josh <dorqus@gmail.com> wrote:> > With almost 700 nodes this isn''t an option. The whole point of > installing puppet in the first place was to not have to do this. > > And like I said, if I run puppetd manually on each box it gets the > files, runs the scripts, etc. etc. > > Also due to inconsistencies in the syslog config on these boxes, not > all of them have puppet logging to syslog the right way. Maybe I > will just configure puppet to log to a file locally on the box instead > of via syslog and see if I can get some more insight that way. > > Josh > > > On Sep 17, 12:23 pm, chalex <alex.chekho...@gmail.com> wrote: >> I would do something like an ssh for loop that looked for the log >> entries from the clients to see what they''re doing: >> >> for i in $NODES >> ssh $i grep puppetd /var/log/messages >> etc >> >> On Sep 17, 9:48 am, josh <dor...@gmail.com> wrote: >> >>> I should have mentioned, I''m running four mongrel instances >>> fronted by >>> Apache/2.2.8 on the puppetmaster server for serving the clients. > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
I have that setup now, I have a class for puppet.conf and a command to restart it, but it isn''t getting pushed out automatically. I also just discovered a misconfiguration in a critical system config file, and since puppet isn''t doing it''s job, I may have to ssh into each box manually and fix it. Here''s another question, does the puppetd binary need to be in /usr/ bin or some "normal" place? Due to OpenSolaris being OpenSolaris, my copy of puppetd is in /usr/ruby/1.8/bin. A ps shows this: /usr/ruby/1.8/bin/ruby /usr/ruby/1.8/bin/puppetd But do I need to put /usr/ruby/1.8/bin in the "path =" statment in puppet.conf? (Currently it''s commented out by default or unset) On Sep 17, 4:16 pm, Aj <a...@junglist.gen.nz> wrote:> You''ll definitely be wanting to setup reporting, even just the log > type to get your clients puppet logs back on the master. > > On 18/09/2008, at 7:31 AM, josh <dor...@gmail.com> wrote: > > > > > With almost 700 nodes this isn''t an option. The whole point of > > installing puppet in the first place was to not have to do this. > > > And like I said, if I run puppetd manually on each box it gets the > > files, runs the scripts, etc. etc. > > > Also due to inconsistencies in the syslog config on these boxes, not > > all of them have puppet logging to syslog the right way. Maybe I > > will just configure puppet to log to a file locally on the box instead > > of via syslog and see if I can get some more insight that way. > > > Josh > > > On Sep 17, 12:23 pm, chalex <alex.chekho...@gmail.com> wrote: > >> I would do something like an ssh for loop that looked for the log > >> entries from the clients to see what they''re doing: > > >> for i in $NODES > >> ssh $i grep puppetd /var/log/messages > >> etc > > >> On Sep 17, 9:48 am, josh <dor...@gmail.com> wrote: > > >>> I should have mentioned, I''m running four mongrel instances > >>> fronted by > >>> Apache/2.2.8 on the puppetmaster server for serving the clients.--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Andrew Shafer
2008-Sep-17 21:41 UTC
[Puppet Users] Re: How long for changes to be pushed out?
Does it appear that puppet is doing other things? Because I suspect it isn''t doing anything. The mystery is to figure out why your puppet clients aren''t running/retrieving their catalogs. If you don''t have logging or reports, fix that so you can start to get insight into what is actually happening. On Wed, Sep 17, 2008 at 2:28 PM, josh <dorqus@gmail.com> wrote:> > I have that setup now, I have a class for puppet.conf and a command to > restart it, but it isn''t getting pushed out automatically. I also > just discovered a misconfiguration in a critical system config file, > and since puppet isn''t doing it''s job, I may have to ssh into each box > manually and fix it. > > Here''s another question, does the puppetd binary need to be in /usr/ > bin or some "normal" place? Due to OpenSolaris being OpenSolaris, my > copy of puppetd is in /usr/ruby/1.8/bin. > > A ps shows this: > > /usr/ruby/1.8/bin/ruby /usr/ruby/1.8/bin/puppetd > > But do I need to put /usr/ruby/1.8/bin in the "path =" statment in > puppet.conf? > (Currently it''s commented out by default or unset) > > On Sep 17, 4:16 pm, Aj <a...@junglist.gen.nz> wrote: > > You''ll definitely be wanting to setup reporting, even just the log > > type to get your clients puppet logs back on the master. > > > > On 18/09/2008, at 7:31 AM, josh <dor...@gmail.com> wrote: > > > > > > > > > With almost 700 nodes this isn''t an option. The whole point of > > > installing puppet in the first place was to not have to do this. > > > > > And like I said, if I run puppetd manually on each box it gets the > > > files, runs the scripts, etc. etc. > > > > > Also due to inconsistencies in the syslog config on these boxes, not > > > all of them have puppet logging to syslog the right way. Maybe I > > > will just configure puppet to log to a file locally on the box instead > > > of via syslog and see if I can get some more insight that way. > > > > > Josh > > > > > On Sep 17, 12:23 pm, chalex <alex.chekho...@gmail.com> wrote: > > >> I would do something like an ssh for loop that looked for the log > > >> entries from the clients to see what they''re doing: > > > > >> for i in $NODES > > >> ssh $i grep puppetd /var/log/messages > > >> etc > > > > >> On Sep 17, 9:48 am, josh <dor...@gmail.com> wrote: > > > > >>> I should have mentioned, I''m running four mongrel instances > > >>> fronted by > > >>> Apache/2.2.8 on the puppetmaster server for serving the clients. > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Larry Ludwig
2008-Sep-17 23:40 UTC
[Puppet Users] Re: How long for changes to be pushed out?
With 700 hosts you should be centralizing your syslog. From that you can send the puppetd data to a central server and see your output. -L -- Larry Ludwig Empowering Media 1-866-792-0489 x600 Managed Xen based VPSes http://www.hostcube.com/ --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Sep 17, 7:40 pm, Larry Ludwig <larry...@gmail.com> wrote:> With 700 hosts you should be centralizing your syslog. From that you > can send the puppetd data to a central server and see your output.And for making sense of all those log entries I can''t recommend Splunk[1] high enough. I have a simple cronjob (setup through Puppet) that rsyncs my /var/log directory to a single location. I then point Splunk to that location and it digs through everything and alerts me when something goes wrong. Cool! They have a free version that will process 500MB/day if you just want to give it a try. Later... Richard [1] http://www.splunk.com/ --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Splunk is in the works, puppet was the first step in the formula. We do have a centralized log server, it''s just that not all the client boxes have the proper syslog-ng config in place - another think that puppet was supposed to "fix" but since it''s not doing anything, I''m SOL. I have enabled reporting for some stores manually, and it looks like things are working, but out of ~680 I''m only seeing 40 reporting back. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Brian Mathis
2008-Sep-18 14:51 UTC
[Puppet Users] Re: How long for changes to be pushed out?
I understand your frustration with this problem, but you''ve now mentioned in almost every email that puppet is "not doing anything", "puppet isn''t doing it''s job", and "it''s not doing anything". Since there are many happy users of puppet, it''s a good indicator that puppet does indeed work, and it''s most likely an issue specific to your configuration. The comment from "chalex" was meant to indicate an option on how you can check what''s wrong with puppet, not as a way to manually perform the tasks instead of using puppet. Before this goes too off-topic about logging, it would be a good idea to reread the comment from Andrew Shafer where he suggests:> > "Does it appear that puppet is doing other things? Because I suspect it isn''t doing anything. > The mystery is to figure out why your puppet clients aren''t running/retrieving their catalogs. >That''s the right answer. Figure out why puppet isn''t working. On Thu, Sep 18, 2008 at 4:39 PM, josh <dorqus@gmail.com> wrote:> > Splunk is in the works, puppet was the first step in the formula. > We do have a centralized log server, it''s just that not all the client > boxes have the proper syslog-ng config in place - another think that > puppet was supposed to "fix" but since it''s not doing anything, I''m > SOL. > > I have enabled reporting for some stores manually, and it looks like > things are working, but out of ~680 I''m only seeing 40 reporting back. >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
OK so I''m looking at one client machine, and it looks like the information in /var/puppet/state/ is out of date by about a week. Shouldn''t those files get recreated anytime that puppetd runs? Also I''m runing puppetd --verbose, but not seeing anything in /var/ puppet/log/ I have the following in puppet.conf: puppetdlog = /var/puppet/log/puppetd.log The directory is there, owned by user puppet group puppet, but there''s no puppetd.log file being created. Am I just missing something obvious here? --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Brian, Sorry for getting a little bent out of shape, it''s just that my boss is pushing on me, etc. etc. Anyways, I think I see an issue. I saw this in the log file on a client, and I bet I''ll see it on many others if I looked. Sep 18 10:48:55 puppetd[17896]: [ID 702911 daemon.error] Could not retrieve catalog: undefined method `-'' for #<XMLRPC::DateTime: 0x8e6cf14> Looks like that started at around the same time that I went from using Webrick for the web server to Apache2 in front of mongrel. I basically used the httpd.conf that is on the Puppet web site. http://www.reductivelabs.com/trac/puppet/wiki/UsingMongrel Well at least now I see there''s a problem that can be resolved. Thank you for helping me see the forest for the trees (or whatever that expression is!) Now I need to troubleshoot my apache2/mongrel configuration (as I''m guessing that''s where the issue lies). On Sep 18, 10:51 am, "Brian Mathis" <brian.mat...@gmail.com> wrote:> I understand your frustration with this problem, but you''ve now > mentioned in almost every email that puppet is "not doing anything", > "puppet isn''t doing it''s job", and "it''s not doing anything". Since > there are many happy users of puppet, it''s a good indicator that > puppet does indeed work, and it''s most likely an issue specific to > your configuration. > > The comment from "chalex" was meant to indicate an option on how you > can check what''s wrong with puppet, not as a way to manually perform > the tasks instead of using puppet. > > Before this goes too off-topic about logging, it would be a good idea > to reread the comment from Andrew Shafer where he suggests: > > > > > "Does it appear that puppet is doing other things? Because I suspect it isn''t doing anything. > > The mystery is to figure out why your puppet clients aren''t running/retrieving their catalogs. > > That''s the right answer. Figure out why puppet isn''t working. > > On Thu, Sep 18, 2008 at 4:39 PM, josh <dor...@gmail.com> wrote: > > > Splunk is in the works, puppet was the first step in the formula. > > We do have a centralized log server, it''s just that not all the client > > boxes have the proper syslog-ng config in place - another think that > > puppet was supposed to "fix" but since it''s not doing anything, I''m > > SOL. > > > I have enabled reporting for some stores manually, and it looks like > > things are working, but out of ~680 I''m only seeing 40 reporting back.--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Nigel Kersten
2008-Sep-18 15:29 UTC
[Puppet Users] Re: How long for changes to be pushed out?
Josh, that''s this bug: http://projects.reductivelabs.com/issues/show/1551 and a simple server side patch will fix it, you can find the patch here: http://github.com/nigelkersten/puppetmaster/commit/65ce150b04e46cfb572d9d2682bf5d9a3b37da68 I''m not sure why more people haven''t seen this. Does everyone simply upgrade all their puppet clients at exactly the same time they upgrade their servers? On Thu, Sep 18, 2008 at 8:19 AM, josh <dorqus@gmail.com> wrote:> > Brian, > Sorry for getting a little bent out of shape, it''s just that my boss > is pushing on me, etc. etc. > > Anyways, I think I see an issue. > I saw this in the log file on a client, and I bet I''ll see it on many > others if I looked. > > Sep 18 10:48:55 puppetd[17896]: [ID 702911 daemon.error] Could not > retrieve catalog: undefined method `-'' for #<XMLRPC::DateTime: > 0x8e6cf14> > > Looks like that started at around the same time that I went from using > Webrick for the web server to Apache2 in front of mongrel. > > I basically used the httpd.conf that is on the Puppet web site. > http://www.reductivelabs.com/trac/puppet/wiki/UsingMongrel > > Well at least now I see there''s a problem that can be resolved. > Thank you for helping me see the forest for the trees (or whatever > that expression is!) > > Now I need to troubleshoot my apache2/mongrel configuration (as I''m > guessing that''s where the issue lies). > > > On Sep 18, 10:51 am, "Brian Mathis" <brian.mat...@gmail.com> wrote: > > I understand your frustration with this problem, but you''ve now > > mentioned in almost every email that puppet is "not doing anything", > > "puppet isn''t doing it''s job", and "it''s not doing anything". Since > > there are many happy users of puppet, it''s a good indicator that > > puppet does indeed work, and it''s most likely an issue specific to > > your configuration. > > > > The comment from "chalex" was meant to indicate an option on how you > > can check what''s wrong with puppet, not as a way to manually perform > > the tasks instead of using puppet. > > > > Before this goes too off-topic about logging, it would be a good idea > > to reread the comment from Andrew Shafer where he suggests: > > > > > > > > > "Does it appear that puppet is doing other things? Because I suspect > it isn''t doing anything. > > > The mystery is to figure out why your puppet clients aren''t > running/retrieving their catalogs. > > > > That''s the right answer. Figure out why puppet isn''t working. > > > > On Thu, Sep 18, 2008 at 4:39 PM, josh <dor...@gmail.com> wrote: > > > > > Splunk is in the works, puppet was the first step in the formula. > > > We do have a centralized log server, it''s just that not all the client > > > boxes have the proper syslog-ng config in place - another think that > > > puppet was supposed to "fix" but since it''s not doing anything, I''m > > > SOL. > > > > > I have enabled reporting for some stores manually, and it looks like > > > things are working, but out of ~680 I''m only seeing 40 reporting back. > > >-- Nigel Kersten Systems Administrator Tech Lead - MacOps --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Nigel, Thanks *phew* that was an easy one line fix! I had upgraded the server from 0.24.4 to 0.24.5 trying to troubleshoot another issue, and introduced this one. Oh well, live and learn. I''ll keep my eyes posted, see if the catalogs get run now on the client machines. JOsh On Sep 18, 11:29 am, "Nigel Kersten" <nig...@google.com> wrote:> Josh, that''s this bug: > > http://projects.reductivelabs.com/issues/show/1551 > > and a simple server side patch will fix it, you can find the patch here: > > http://github.com/nigelkersten/puppetmaster/commit/65ce150b04e46cfb57... > > I''m not sure why more people haven''t seen this. Does everyone simply upgrade > all their puppet clients at exactly the same time they upgrade their > servers? > > > > On Thu, Sep 18, 2008 at 8:19 AM, josh <dor...@gmail.com> wrote: > > > Brian, > > Sorry for getting a little bent out of shape, it''s just that my boss > > is pushing on me, etc. etc. > > > Anyways, I think I see an issue. > > I saw this in the log file on a client, and I bet I''ll see it on many > > others if I looked. > > > Sep 18 10:48:55 puppetd[17896]: [ID 702911 daemon.error] Could not > > retrieve catalog: undefined method `-'' for #<XMLRPC::DateTime: > > 0x8e6cf14> > > > Looks like that started at around the same time that I went from using > > Webrick for the web server to Apache2 in front of mongrel. > > > I basically used the httpd.conf that is on the Puppet web site. > >http://www.reductivelabs.com/trac/puppet/wiki/UsingMongrel > > > Well at least now I see there''s a problem that can be resolved. > > Thank you for helping me see the forest for the trees (or whatever > > that expression is!) > > > Now I need to troubleshoot my apache2/mongrel configuration (as I''m > > guessing that''s where the issue lies). > > > On Sep 18, 10:51 am, "Brian Mathis" <brian.mat...@gmail.com> wrote: > > > I understand your frustration with this problem, but you''ve now > > > mentioned in almost every email that puppet is "not doing anything", > > > "puppet isn''t doing it''s job", and "it''s not doing anything". Since > > > there are many happy users of puppet, it''s a good indicator that > > > puppet does indeed work, and it''s most likely an issue specific to > > > your configuration. > > > > The comment from "chalex" was meant to indicate an option on how you > > > can check what''s wrong with puppet, not as a way to manually perform > > > the tasks instead of using puppet. > > > > Before this goes too off-topic about logging, it would be a good idea > > > to reread the comment from Andrew Shafer where he suggests: > > > > > "Does it appear that puppet is doing other things? Because I suspect > > it isn''t doing anything. > > > > The mystery is to figure out why your puppet clients aren''t > > running/retrieving their catalogs. > > > > That''s the right answer. Figure out why puppet isn''t working. > > > > On Thu, Sep 18, 2008 at 4:39 PM, josh <dor...@gmail.com> wrote: > > > > > Splunk is in the works, puppet was the first step in the formula. > > > > We do have a centralized log server, it''s just that not all the client > > > > boxes have the proper syslog-ng config in place - another think that > > > > puppet was supposed to "fix" but since it''s not doing anything, I''m > > > > SOL. > > > > > I have enabled reporting for some stores manually, and it looks like > > > > things are working, but out of ~680 I''m only seeing 40 reporting back. > > -- > Nigel Kersten > Systems Administrator > Tech Lead - MacOps--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Nigel Kersten
2008-Sep-18 15:44 UTC
[Puppet Users] Re: How long for changes to be pushed out?
heh. This bug taught us something valuable though. When testing new versions, running with "--test" isn''t a good test :) as it ignores the local cache, and thus will never hit this bug.... On Thu, Sep 18, 2008 at 8:41 AM, josh <dorqus@gmail.com> wrote:> > Nigel, > > Thanks *phew* that was an easy one line fix! > > I had upgraded the server from 0.24.4 to 0.24.5 trying to troubleshoot > another issue, and introduced this one. > > Oh well, live and learn. > > I''ll keep my eyes posted, see if the catalogs get run now on the > client machines. > > JOsh > > > On Sep 18, 11:29 am, "Nigel Kersten" <nig...@google.com> wrote: > > Josh, that''s this bug: > > > > http://projects.reductivelabs.com/issues/show/1551 > > > > and a simple server side patch will fix it, you can find the patch here: > > > > http://github.com/nigelkersten/puppetmaster/commit/65ce150b04e46cfb57... > > > > I''m not sure why more people haven''t seen this. Does everyone simply > upgrade > > all their puppet clients at exactly the same time they upgrade their > > servers? > > > > > > > > On Thu, Sep 18, 2008 at 8:19 AM, josh <dor...@gmail.com> wrote: > > > > > Brian, > > > Sorry for getting a little bent out of shape, it''s just that my boss > > > is pushing on me, etc. etc. > > > > > Anyways, I think I see an issue. > > > I saw this in the log file on a client, and I bet I''ll see it on many > > > others if I looked. > > > > > Sep 18 10:48:55 puppetd[17896]: [ID 702911 daemon.error] Could not > > > retrieve catalog: undefined method `-'' for #<XMLRPC::DateTime: > > > 0x8e6cf14> > > > > > Looks like that started at around the same time that I went from using > > > Webrick for the web server to Apache2 in front of mongrel. > > > > > I basically used the httpd.conf that is on the Puppet web site. > > >http://www.reductivelabs.com/trac/puppet/wiki/UsingMongrel > > > > > Well at least now I see there''s a problem that can be resolved. > > > Thank you for helping me see the forest for the trees (or whatever > > > that expression is!) > > > > > Now I need to troubleshoot my apache2/mongrel configuration (as I''m > > > guessing that''s where the issue lies). > > > > > On Sep 18, 10:51 am, "Brian Mathis" <brian.mat...@gmail.com> wrote: > > > > I understand your frustration with this problem, but you''ve now > > > > mentioned in almost every email that puppet is "not doing anything", > > > > "puppet isn''t doing it''s job", and "it''s not doing anything". Since > > > > there are many happy users of puppet, it''s a good indicator that > > > > puppet does indeed work, and it''s most likely an issue specific to > > > > your configuration. > > > > > > The comment from "chalex" was meant to indicate an option on how you > > > > can check what''s wrong with puppet, not as a way to manually perform > > > > the tasks instead of using puppet. > > > > > > Before this goes too off-topic about logging, it would be a good idea > > > > to reread the comment from Andrew Shafer where he suggests: > > > > > > > "Does it appear that puppet is doing other things? Because I > suspect > > > it isn''t doing anything. > > > > > The mystery is to figure out why your puppet clients aren''t > > > running/retrieving their catalogs. > > > > > > That''s the right answer. Figure out why puppet isn''t working. > > > > > > On Thu, Sep 18, 2008 at 4:39 PM, josh <dor...@gmail.com> wrote: > > > > > > > Splunk is in the works, puppet was the first step in the formula. > > > > > We do have a centralized log server, it''s just that not all the > client > > > > > boxes have the proper syslog-ng config in place - another think > that > > > > > puppet was supposed to "fix" but since it''s not doing anything, I''m > > > > > SOL. > > > > > > > I have enabled reporting for some stores manually, and it looks > like > > > > > things are working, but out of ~680 I''m only seeing 40 reporting > back. > > > > -- > > Nigel Kersten > > Systems Administrator > > Tech Lead - MacOps > > >-- Nigel Kersten Systems Administrator Tech Lead - MacOps --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
So I made the change on the server roughly 30 minutes ago, hopefully within a few hours most of the clients will have checked back in to the server and downloaded the catalogs fresh (I shouldn''t need to restart puppetd or remove /var/puppet/state on the clients should I?) Josh On Sep 18, 11:44 am, "Nigel Kersten" <nig...@google.com> wrote:> heh. This bug taught us something valuable though. > > When testing new versions, running with "--test" isn''t a good test :) as it > ignores the local cache, and thus will never hit this bug....--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---