Romeo Theriault
2012-Mar-13 22:16 UTC
[Puppet Users] puppet eating solaris 10 crontab for lunch
Ugh, this isn''t a nice bug to find out about. Just found out that on a few of our Solaris 10 global zones, puppet is destroying the crontab entry of the root user. It seems to be related to a hang in facter. I''m not 100% sure, but it seems the issue is occurring when facter runs ''prtdiag'' on the hosts and prtdiag hangs midway (prtdiag is hanging due to the picld daemon being in a funky state and not returning the sensor data). It seems that this in turn puts puppet in a funky state, not sure how or why though. Here are the logs the solaris 10 box returns after it''s crontab gets destroyed: ERR Puppet Could not prefetch cron provider ''crontab'': Could not read crontab for root: No child processes NOTICE /Stage[main]/Puppet/Cron[puppet]/ensure created NOTICE Puppet Finished catalog run in 2.52 seconds After this the only thing that exists in the crontab is the entry we have puppet adding. I found this bug: http://projects.puppetlabs.com/issues/1672 which says there was a fix and it was merged but we''re still seeing this issue... puppet agent v. 2.7.9 facter v. 1.6.5 Any suggestions or work-arounds short of not using the cron provider or completely managing the hosts crontab''s centrally? Neither of which are ideal for us at the moment. Puppet should be returning the original crontab file should there be any failure. This is not comforting. -- Romeo -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
John Warburton
2012-Mar-13 22:26 UTC
Re: [Puppet Users] puppet eating solaris 10 crontab for lunch
On 14 March 2012 09:16, Romeo Theriault <romeo.theriault@maine.edu> wrote:> Here are the logs the solaris 10 box returns after it''s crontab gets > destroyed: > > ERR Puppet Could not prefetch cron provider ''crontab'': Could not read > crontab for root: No child processes > NOTICE /Stage[main]/Puppet/Cron[puppet]/ensure created > NOTICE Puppet Finished catalog run in 2.52 seconds > > After this the only thing that exists in the crontab is the entry we > have puppet adding. > > I found this bug: > > http://projects.puppetlabs.com/issues/1672 > > which says there was a fix and it was merged but we''re still seeing > this issue... > > puppet agent v. 2.7.9 > facter v. 1.6.5 > >It could be this bug - https://projects.puppetlabs.com/issues/5752 That and https://projects.puppetlabs.com/issues/9854 are keeping me from pushing migrating to 2.7 up my priority list Indeed, there are 5 issues marked Urgent in the 2.7.x bucket John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Have seen similar. Quite often when prtdiag fails to complete, I''ve found that restarting svc:/system/picl:default returns everything back to normal... Hopefully all your root cron jobs are in Puppet and will be rebuilt on the next run... Greg On Mar 14, 9:26 am, John Warburton <jwarbur...@gmail.com> wrote:> On 14 March 2012 09:16, Romeo Theriault <romeo.theria...@maine.edu> wrote: > > > > > > > > > > > Here are the logs the solaris 10 box returns after it''s crontab gets > > destroyed: > > > ERR Puppet Could not prefetch cron provider ''crontab'': Could not read > > crontab for root: No child processes > > NOTICE /Stage[main]/Puppet/Cron[puppet]/ensure created > > NOTICE Puppet Finished catalog run in 2.52 seconds > > > After this the only thing that exists in the crontab is the entry we > > have puppet adding. > > > I found this bug: > > >http://projects.puppetlabs.com/issues/1672 > > > which says there was a fix and it was merged but we''re still seeing > > this issue... > > > puppet agent v. 2.7.9 > > facter v. 1.6.5 > > It could be this bug -https://projects.puppetlabs.com/issues/5752 > > That andhttps://projects.puppetlabs.com/issues/9854are keeping me from > pushing migrating to 2.7 up my priority list > > Indeed, there are 5 issues marked Urgent in the 2.7.x bucket > > John-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
I don''t think it''s issue #5752, I opened that issue and provided a patch to resolve it. When I build new versions of Puppet for my Solaris hosts I apply that patch each time via my build script and I''m still having some hosts where the crontab gets "eaten" and it always seems to correspond with ''prtdiag'' issues or timeouts. Facter has a timeout built in for prtdiag and then does a kill routine if it needs to clean up. I wonder if it''s being overly agressive and accidentally killing some or all of the Puppet run in turn causing this issue? -Kent On Mar 13, 5:31 pm, Greg <greg.b...@gmail.com> wrote:> Have seen similar. Quite often when prtdiag fails to complete, I''ve > found that restarting svc:/system/picl:default returns everything back > to normal... > > Hopefully all your root cron jobs are in Puppet and will be rebuilt on > the next run... > > Greg > > On Mar 14, 9:26 am, John Warburton <jwarbur...@gmail.com> wrote: > > > > > > > > > On 14 March 2012 09:16, Romeo Theriault <romeo.theria...@maine.edu> wrote: > > > > Here are the logs the solaris 10 box returns after it''s crontab gets > > > destroyed: > > > > ERR Puppet Could not prefetch cron provider ''crontab'': Could not read > > > crontab for root: No child processes > > > NOTICE /Stage[main]/Puppet/Cron[puppet]/ensure created > > > NOTICE Puppet Finished catalog run in 2.52 seconds > > > > After this the only thing that exists in the crontab is the entry we > > > have puppet adding. > > > > I found this bug: > > > >http://projects.puppetlabs.com/issues/1672 > > > > which says there was a fix and it was merged but we''re still seeing > > > this issue... > > > > puppet agent v. 2.7.9 > > > facter v. 1.6.5 > > > It could be this bug -https://projects.puppetlabs.com/issues/5752 > > > That andhttps://projects.puppetlabs.com/issues/9854arekeeping me from > > pushing migrating to 2.7 up my priority list > > > Indeed, there are 5 issues marked Urgent in the 2.7.x bucket > > > John-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Romeo Theriault
2012-May-01 19:45 UTC
Re: [Puppet Users] Re: puppet eating solaris 10 crontab for lunch
On Tue, May 1, 2012 at 03:14, Kent <drizit@gmail.com> wrote:> I don''t think it''s issue #5752, I opened that issue and provided a > patch to resolve it. > When I build new versions of Puppet for my Solaris hosts I apply that > patch each time via my build script and I''m still having some hosts > where the crontab gets "eaten" and it always seems to correspond with > ''prtdiag'' issues or timeouts. > > Facter has a timeout built in for prtdiag and then does a kill routine > if it needs to clean up. > I wonder if it''s being overly agressive and accidentally killing some > or all of the Puppet run in turn causing this issue?I agree, that I think this is probably what is happening. I''ve since stopped using puppet to manage our solaris crontab files since we can''t currently fully puppetize our crontabs. Unfortunately, solaris doesn''t have a cron.d directory where we can drop crontab files either. Romeo> On Mar 13, 5:31 pm, Greg <greg.b...@gmail.com> wrote: >> Have seen similar. Quite often when prtdiag fails to complete, I''ve >> found that restarting svc:/system/picl:default returns everything back >> to normal... >> >> Hopefully all your root cron jobs are in Puppet and will be rebuilt on >> the next run... >> >> Greg >> >> On Mar 14, 9:26 am, John Warburton <jwarbur...@gmail.com> wrote: >> >> >> >> >> >> >> >> > On 14 March 2012 09:16, Romeo Theriault <romeo.theria...@maine.edu> wrote: >> >> > > Here are the logs the solaris 10 box returns after it''s crontab gets >> > > destroyed: >> >> > > ERR Puppet Could not prefetch cron provider ''crontab'': Could not read >> > > crontab for root: No child processes >> > > NOTICE /Stage[main]/Puppet/Cron[puppet]/ensure created >> > > NOTICE Puppet Finished catalog run in 2.52 seconds >> >> > > After this the only thing that exists in the crontab is the entry we >> > > have puppet adding. >> >> > > I found this bug: >> >> > >http://projects.puppetlabs.com/issues/1672 >> >> > > which says there was a fix and it was merged but we''re still seeing >> > > this issue... >> >> > > puppet agent v. 2.7.9 >> > > facter v. 1.6.5 >> >> > It could be this bug -https://projects.puppetlabs.com/issues/5752 >> >> > That andhttps://projects.puppetlabs.com/issues/9854arekeeping me from >> > pushing migrating to 2.7 up my priority list >> >> > Indeed, there are 5 issues marked Urgent in the 2.7.x bucket >> >> > John > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. >-- Romeo -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Russell Van Tassell
2012-May-01 20:35 UTC
Re: [Puppet Users] Re: puppet eating solaris 10 crontab for lunch
On Tue, May 1, 2012 at 12:45 PM, Romeo Theriault <romeo.theriault@maine.edu>wrote:> Unfortunately, solaris > doesn''t have a cron.d directory where we can drop crontab files > either. >Are you talking about /var/spool/cron/crontab on Solaris? (think that''s the right path) It won''t reload them without being kicked. But, you can play tricks with it by dropping the file there, then reload it by invoking "crontab" and feeding it the new file. You might have to massage it to get things to work properly, but it should be possible (ie. I''ve done it this way, manually, in a previous life). -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Romeo Theriault
2012-May-01 23:04 UTC
Re: [Puppet Users] Re: puppet eating solaris 10 crontab for lunch
On Tue, May 1, 2012 at 10:35 AM, Russell Van Tassell <russellvt@gmail.com> wrote:> On Tue, May 1, 2012 at 12:45 PM, Romeo Theriault <romeo.theriault@maine.edu> > wrote: >> >> Unfortunately, solaris >> doesn''t have a cron.d directory where we can drop crontab files >> either. > > > Are you talking about /var/spool/cron/crontab on Solaris? (think that''s the > right path) > > It won''t reload them without being kicked. But, you can play tricks with it > by dropping the file there, then reload it by invoking "crontab" and feeding > it the new file. You might have to massage it to get things to work > properly, but it should be possible (ie. I''ve done it this way, manually, in > a previous life).I was referring to something like the /etc/cron.d/ directory on linux where you can drop in new files that have specific entries. Are you suggesting that you can drop in new "files" into the /var/spool/cron/crontabs dir on solaris and it will pick them up after being reloaded? How would it know which user to run this crontab as if it''s not the actual users crontab file itself? See, I don''t want to manage the whole file. I''d like to just be able to manage individual entries by placing new files into the dir. -- Romeo -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.