Earlier this week, I applied RHEL patches to a couple of dev server with puppet 0.25.5 and now I can no longer run puppetd commands without constantly getting the message: [root@dev2 ~]# puppetd --test --verbose --noop notice: Run of Puppet configuration client already in progress; skipping Killing the process and then clearing out the lock file every time is not really an option. Also, I am finding that puppetd --enable is not having any effect on my problem I am guessing that some puppet dependency got updated by the update from RHEL 5.5 to 5.6. Any suggestions on how to troubleshoot this? Thanks! -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
You should upgrade from 0.25.5. It is quite old and no longer supported. On Friday, May 10, 2013 2:17:00 PM UTC-6, dsdtas wrote:> > Earlier this week, I applied RHEL patches to a couple of dev server with > puppet 0.25.5 and now I can no longer run puppetd commands without > constantly getting the message: > > [root@dev2 ~]# puppetd --test --verbose --noop > notice: Run of Puppet configuration client already in progress; skipping > > Killing the process and then clearing out the lock file every time is not > really an option. Also, I am finding that puppetd --enable is not having > any effect on my problem > > I am guessing that some puppet dependency got updated by the update from > RHEL 5.5 to 5.6. Any suggestions on how to troubleshoot this? > > Thanks! > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
On Friday, May 10, 2013 3:17:00 PM UTC-5, dsdtas wrote:> > Earlier this week, I applied RHEL patches to a couple of dev server with > puppet 0.25.5 and now I can no longer run puppetd commands without > constantly getting the message: > > [root@dev2 ~]# puppetd --test --verbose --noop > notice: Run of Puppet configuration client already in progress; skipping > > Killing the process and then clearing out the lock file every time is not > really an option. >How about stopping the agent before performing OS upgrades (as opposed to just applying updates released for the current OS version)?> Also, I am finding that puppetd --enable is not having any effect on my > problem > >Just to be sure: you''re running that with privilege, right?> I am guessing that some puppet dependency got updated by the update from > RHEL 5.5 to 5.6. Any suggestions on how to troubleshoot this? > >If by "troubleshoot this" you mean get Puppet working correctly again, then there is probably no alternative to forcing Puppet stopped and then removing the lock file. That might involve manually killing a stalled puppetd. If you prefer, that could take the form of restarting the whole server, after which it would be safe to remove the lock file without shutting down the agent. You really should not remove the lock file, neither manually nor via "puppetd --enable", while the puppetd process that created it is alive, however. If by "troubleshoot this" you mean determine what went wrong and why, then you need to gather information, including: - The actual state of the system. Is the agent in fact running? Is the lock file in fact present? - The logs of the most recent Puppet activity not related to your failed / skipped runs and diagnostic efforts. What did puppet last do -- or what was it in the process of doing -- when it entered the state it is now in? - The updates that were actually applied to get from your (possibly updated) RHEL 5.5 to the current (possibly updated) 5.6. Then you need to conduct an analysis, on which I cannot advise you in any detail. I think it more likely that the update clobbered application of some resource, causing the agent to stall in a manner tied to the resource type and its chosen provider, than that the update clobbered the core puppet engine. But I can''t be sure. Ultimately, even if you are able to form a good hypothesis about what happened, and even if you are able to test that hypothesis to prove its plausibility, I don''t know any way to be *certain* that what you come up with is the correct explanation for what actually happened. You''ll have to use your best judgement. John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
On Monday, May 13, 2013 9:20:54 AM UTC-4, jcbollinger wrote:> > > > On Friday, May 10, 2013 3:17:00 PM UTC-5, dsdtas wrote: >> >> Earlier this week, I applied RHEL patches to a couple of dev server with >> puppet 0.25.5 and now I can no longer run puppetd commands without >> constantly getting the message: >> >> [root@dev2 ~]# puppetd --test --verbose --noop >> notice: Run of Puppet configuration client already in progress; skipping >> >> Killing the process and then clearing out the lock file every time is not >> really an option. >> > > > How about stopping the agent before performing OS upgrades (as opposed to > just applying updates released for the current OS version)? >This did not occur to me. Sounds Windows-y... LOL> > >> Also, I am finding that puppetd --enable is not having any effect on my >> problem >> >> > > Just to be sure: you''re running that with privilege, right? >Yes, running as root.> > > >> I am guessing that some puppet dependency got updated by the update from >> RHEL 5.5 to 5.6. Any suggestions on how to troubleshoot this? >> >> > If by "troubleshoot this" you mean get Puppet working correctly again, > then there is probably no alternative to forcing Puppet stopped and then > removing the lock file. That might involve manually killing a stalled > puppetd. If you prefer, that could take the form of restarting the whole > server, after which it would be safe to remove the lock file without > shutting down the agent. You really should not remove the lock file, > neither manually nor via "puppetd --enable", while the puppetd process that > created it is alive, however. > > If by "troubleshoot this" you mean determine what went wrong and why, then > you need to gather information, including: > > - The actual state of the system. Is the agent in fact running? Is > the lock file in fact present? > - The logs of the most recent Puppet activity not related to your > failed / skipped runs and diagnostic efforts. What did puppet last do -- > or what was it in the process of doing -- when it entered the state it is > now in? > - The updates that were actually applied to get from your (possibly > updated) RHEL 5.5 to the current (possibly updated) 5.6. > > Then you need to conduct an analysis, on which I cannot advise you in any > detail. I think it more likely that the update clobbered application of > some resource, causing the agent to stall in a manner tied to the resource > type and its chosen provider, than that the update clobbered the core > puppet engine. But I can''t be sure. > > Ultimately, even if you are able to form a good hypothesis about what > happened, and even if you are able to test that hypothesis to prove its > plausibility, I don''t know any way to be *certain* that what you come up > with is the correct explanation for what actually happened. You''ll have to > use your best judgement. > > > John > >Thanks for going into detail about this. I too believe that some puppet resource was indeed stepped on by the RHEL update from 5.5 to 5.6. I am in the process of reverting the ruby packages on the affected servers from 1.8.5.-19 to 1.8.5.-5. Also related, I discovered that the puppet listener wasn''t properly restarting itself due to the control script not specifying the correct location of the PID file. That issue has been fixed so I no longer have to manually kill the process. Will update the thread later this week with progress Thanks DS -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
On Tuesday, May 14, 2013 4:37:21 PM UTC-5, dsdtas wrote:> > > > On Monday, May 13, 2013 9:20:54 AM UTC-4, jcbollinger wrote: >> >> >> >> On Friday, May 10, 2013 3:17:00 PM UTC-5, dsdtas wrote: >>> >>> Earlier this week, I applied RHEL patches to a couple of dev server with >>> puppet 0.25.5 and now I can no longer run puppetd commands without >>> constantly getting the message: >>> >>> [root@dev2 ~]# puppetd --test --verbose --noop >>> notice: Run of Puppet configuration client already in progress; skipping >>> >>> Killing the process and then clearing out the lock file every time is >>> not really an option. >>> >> >> >> How about stopping the agent before performing OS upgrades (as opposed to >> just applying updates released for the current OS version)? >> > > This did not occur to me. Sounds Windows-y... LOL >A bit, maybe, but if it were really Windows-y then you''d need to reboot the system after the upgrade. :-) Conceptually, it doesn''t seem unreasonable to me to avoid attempting to perform conflicting actions concurrently. An OS upgrade certainly is at least a potential conflict with Puppet-controlled configuration management. It might be possible to harmonize these (for example, by making Puppet perform the upgrade), but it''s hard to be sure without knowing the nature of the problem that occurred. [...]> Thanks for going into detail about this. I too believe that some puppet > resource was indeed stepped on by the RHEL update from 5.5 to 5.6. I am in > the process of reverting the ruby packages on the affected servers from > 1.8.5.-19 to 1.8.5.-5. Also related, I discovered that the puppet listener > wasn''t properly restarting itself due to the control script not specifying > the correct location of the PID file. That issue has been fixed so I no > longer have to manually kill the process. Will update the thread later > this week with progress > >I would be very surprised if ruby-1.8.5-19 were the problem. Indeed, I have multiple CentOS 5.x systems under Puppet management, and I have never had a problem with any of the official CentOS Ruby packages. If you contemplate moving up to Puppet 2.x or higher in the future, however, then at that time you will need at least Ruby 1.8.7. That would need to be provided by a third-party package, given CentOS / RHEL policy on software versions. Indeed do please update us about what you find. I''m curious. John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Finally made progress today, the answer for me was to back out of the newer kernel version as mentioned here: https://groups.google.com/d/msg/puppet-users/ho5Thsm5q1E/FPe4N9KvhD4J After downgrading from 2.6.18-238.49.1.el5 back to 2.6.18-194.32.1.el5 my puppetd is functioning properly again. Something about puppet kick was causing the puppet client to get stuck indefinitely until manually killed or restarted via init script. So, locally puppetd worked fine *until* a puppetrun from the master happened, causing the client to enter a stuck state. Now I just have to upgrade the puppetmaster in a test environment and hope my code still works!! I am probably going to try baby steps and go to the latest 2.6 instead of straight to 3.x Thanks for the help! Dave On Friday, May 10, 2013 4:17:00 PM UTC-4, dsdtas wrote:> > Earlier this week, I applied RHEL patches to a couple of dev server with > puppet 0.25.5 and now I can no longer run puppetd commands without > constantly getting the message: > > [root@dev2 ~]# puppetd --test --verbose --noop > notice: Run of Puppet configuration client already in progress; skipping > > Killing the process and then clearing out the lock file every time is not > really an option. Also, I am finding that puppetd --enable is not having > any effect on my problem > > I am guessing that some puppet dependency got updated by the update from > RHEL 5.5 to 5.6. Any suggestions on how to troubleshoot this? > > Thanks! > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.