Chris Phillips
2011-Jul-05 08:53 UTC
[Puppet Users] puppet terminating - continually forcing itself to stop
Hi, I''ve a puppet module running which ensures the puppet service is not running and not starting on boot. A few times a few different nodes have become stuck, apparently always thinking that the service is running when it is not. The following below excerpt just goes round and round run after run until I commented out the ensure and enable for the puppet service. It then completes and the node works as expected after this when the #''s are removed again. info: Applying configuration version ''1309852902'' debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw yaml; using pson debug: Service[puppet](provider=redhat): Executing ''/sbin/service puppet status'' debug: Puppet::Type::Service::ProviderRedhat: Executing ''/sbin/chkconfig puppet'' debug: Service[puppet](provider=redhat): Executing ''/sbin/service puppet stop'' notice: Caught TERM; calling stop [root@gibsvlin-erp2dst ~]# /etc/init.d/puppet status puppetd is stopped [root@gibsvlin-erp2dst ~]# /sbin/chkconfig puppet [root@gibsvlin-erp2dst ~]# echo $? 1 [root@gibsvlin-erp2dst ~]# ps -ef | grep puppet root 5548 4886 0 09:24 pts/0 00:00:00 grep puppet Everything is returning the right exit codes and such, so why after being very clearly told that puppet is NOT running does it stop it anyway, and terminate? 2.6.7 btw, eagerly awaiting 2.6.9 rpms in the yum repo. Thanks Chris -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Ken Barber
2011-Jul-05 09:42 UTC
Re: [Puppet Users] puppet terminating - continually forcing itself to stop
What does your service {} block look like? Are you using hasstatus =>
true? For example:
service { "puppet":
ensure => running,
enable => true,
hasstatus => true,
}
ken.
On Tue, Jul 5, 2011 at 9:53 AM, Chris Phillips <chris@untrepid.com>
wrote:> Hi,
> I''ve a puppet module running which ensures the puppet service is
not running
> and not starting on boot. A few times a few different nodes have become
> stuck, apparently always thinking that the service is running when it is
> not. The following below excerpt just goes round and round run after run
> until I commented out the ensure and enable for the puppet service. It then
> completes and the node works as expected after this when the #''s
are removed
> again.
> info: Applying configuration version ''1309852902''
> debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw yaml;
> using pson
> debug: Service[puppet](provider=redhat): Executing ''/sbin/service
puppet
> status''
> debug: Puppet::Type::Service::ProviderRedhat: Executing
''/sbin/chkconfig
> puppet''
> debug: Service[puppet](provider=redhat): Executing ''/sbin/service
puppet
> stop''
> notice: Caught TERM; calling stop
> [root@gibsvlin-erp2dst ~]# /etc/init.d/puppet status
> puppetd is stopped
> [root@gibsvlin-erp2dst ~]# /sbin/chkconfig puppet
> [root@gibsvlin-erp2dst ~]# echo $?
> 1
> [root@gibsvlin-erp2dst ~]# ps -ef | grep puppet
> root 5548 4886 0 09:24 pts/0 00:00:00 grep puppet
> Everything is returning the right exit codes and such, so why after being
> very clearly told that puppet is NOT running does it stop it anyway, and
> terminate?
> 2.6.7 btw, eagerly awaiting 2.6.9 rpms in the yum repo.
> Thanks
> Chris
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
Chris Phillips
2011-Jul-05 11:15 UTC
Re: [Puppet Users] puppet terminating - continually forcing itself to stop
Currently it''s just like this:
service { "puppet":
# ensure => running,
# enable => true,
ensure => stopped,
enable => false,
subscribe => File["/etc/puppet/puppet.conf"],
hasrestart => true,
hasstatus => true,
}
And 99% of the time all 66 of my clients are totally fine with it. I''m
calling puppet via a nagios check, so I went from ensuring it was running to
ensuring it was stopped. I don''t personally have any doubt that the
config
is absolutely fine, it''s just getting implemented wrong sometimes.
Thanks
Chris
On 5 July 2011 10:42, Ken Barber <ken@puppetlabs.com> wrote:
> What does your service {} block look like? Are you using hasstatus =>
> true? For example:
>
> service { "puppet":
> ensure => running,
> enable => true,
> hasstatus => true,
> }
>
> ken.
>
> On Tue, Jul 5, 2011 at 9:53 AM, Chris Phillips <chris@untrepid.com>
wrote:
> > Hi,
> > I''ve a puppet module running which ensures the puppet service
is not
> running
> > and not starting on boot. A few times a few different nodes have
become
> > stuck, apparently always thinking that the service is running when it
is
> > not. The following below excerpt just goes round and round run after
run
> > until I commented out the ensure and enable for the puppet service. It
> then
> > completes and the node works as expected after this when the
#''s are
> removed
> > again.
> > info: Applying configuration version ''1309852902''
> > debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw
> yaml;
> > using pson
> > debug: Service[puppet](provider=redhat): Executing
''/sbin/service puppet
> > status''
> > debug: Puppet::Type::Service::ProviderRedhat: Executing
''/sbin/chkconfig
> > puppet''
> > debug: Service[puppet](provider=redhat): Executing
''/sbin/service puppet
> > stop''
> > notice: Caught TERM; calling stop
> > [root@gibsvlin-erp2dst ~]# /etc/init.d/puppet status
> > puppetd is stopped
> > [root@gibsvlin-erp2dst ~]# /sbin/chkconfig puppet
> > [root@gibsvlin-erp2dst ~]# echo $?
> > 1
> > [root@gibsvlin-erp2dst ~]# ps -ef | grep puppet
> > root 5548 4886 0 09:24 pts/0 00:00:00 grep puppet
> > Everything is returning the right exit codes and such, so why after
being
> > very clearly told that puppet is NOT running does it stop it anyway,
and
> > terminate?
> > 2.6.7 btw, eagerly awaiting 2.6.9 rpms in the yum repo.
> > Thanks
> > Chris
> >
> > --
> > You received this message because you are subscribed to the Google
Groups
> > "Puppet Users" group.
> > To post to this group, send email to puppet-users@googlegroups.com.
> > To unsubscribe from this group, send email to
> > puppet-users+unsubscribe@googlegroups.com.
> > For more options, visit this group at
> > http://groups.google.com/group/puppet-users?hl=en.
> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
vagn scott
2011-Jul-05 11:49 UTC
Re: [Puppet Users] puppet terminating - continually forcing itself to stop
On 07/05/2011 07:15 AM, Chris Phillips wrote:> > > debug: Service[puppet](provider=redhat): Executing ''/sbin/service puppet > > stop'' > > notice: Caught TERM; calling stop > > [root@gibsvlin-erp2dst ~]# /etc/init.d/puppet status > > puppetd is stoppedThat''s just weird looking. So puppet is running, and trying to kill puppetd. Instead it catches the TERM signal itself. Is that what I''m seeing? That looks racy as hell. Maybe you should rethink that and instead have something like monit watch puppetd to make sure it stays off. Or have nagios start a script that makes sure puppet is not running, then runs it, then makes sure it is stopped. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Brian Gallew
2011-Jul-05 13:19 UTC
Re: [Puppet Users] puppet terminating - continually forcing itself to stop
I was seeing the behavior on my Solaris boxes when running Puppet under SMF. The issue, in my case, was that I was trying to work around an SMF bug. My "workaround" was to "svcadm disable puppetd;svccfg import /var/svc/manifest/network/puppetd.xml;svcadm enable puppet". The astute viewer will notice that "svcadm disable puppetd" will cause SMF to send SIGTERM to puppet, at which point it will stop (as instructed) the current run, so the new manifest will not be imported and the service will not be re-started. It was all very amusing, except for the bit where I had to fix a bunch of systems that weren''t running Puppet anymore. Mea culpa. On Tue, Jul 5, 2011 at 4:49 AM, vagn scott <vagnscott@gmail.com> wrote:> On 07/05/2011 07:15 AM, Chris Phillips wrote: > >> >> > debug: Service[puppet](provider=**redhat): Executing ''/sbin/service >> puppet >> > stop'' >> > notice: Caught TERM; calling stop >> > [root@gibsvlin-erp2dst ~]# /etc/init.d/puppet status >> > puppetd is stopped >> > That''s just weird looking. > So puppet is running, and trying to kill puppetd. > Instead it catches the TERM signal itself. > Is that what I''m seeing? > > That looks racy as hell. > > Maybe you should rethink that and instead have > something like monit watch puppetd to make > sure it stays off. > > Or have nagios start a script that makes sure > puppet is not running, then runs it, then makes > sure it is stopped. > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@** > googlegroups.com <puppet-users%2Bunsubscribe@googlegroups.com>. > For more options, visit this group at http://groups.google.com/** > group/puppet-users?hl=en<http://groups.google.com/group/puppet-users?hl=en> > . > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Martin Alfke
2011-Jul-05 13:24 UTC
Re: [Puppet Users] puppet terminating - continually forcing itself to stop
On 07/05/2011 03:19 PM, Brian Gallew wrote:> I was seeing the behavior on my Solaris boxes when running Puppet under > SMF. The issue, in my case, was that I was trying to work around an SMF > bug. My "workaround" was to "svcadm disable puppetd;svccfg import > /var/svc/manifest/network/puppetd.xml;svcadm enable puppet". The astute > viewer will notice that "svcadm disable puppetd" will cause SMF to send > SIGTERM to puppet, at which point it will stop (as instructed) the > current run, so the new manifest will not be imported and the service > will not be re-started. It was all very amusing, except for the bit > where I had to fix a bunch of systems that weren''t running Puppet > anymore. Mea culpa.I have seen a similar behavior on RHEL5. /sbin/service puppet status gets information on actual running puppet run. We fixed the puppet init script to verify whether the running puppet process has parent pid 1 (init).> > On Tue, Jul 5, 2011 at 4:49 AM, vagn scott <vagnscott@gmail.com > <mailto:vagnscott@gmail.com>> wrote: > > On 07/05/2011 07:15 AM, Chris Phillips wrote: > > > > debug: Service[puppet](provider=__redhat): Executing > ''/sbin/service puppet > > stop'' > > notice: Caught TERM; calling stop > > [root@gibsvlin-erp2dst ~]# /etc/init.d/puppet status > > puppetd is stopped > > That''s just weird looking. > So puppet is running, and trying to kill puppetd. > Instead it catches the TERM signal itself. > Is that what I''m seeing? > > That looks racy as hell. > > Maybe you should rethink that and instead have > something like monit watch puppetd to make > sure it stays off. > > Or have nagios start a script that makes sure > puppet is not running, then runs it, then makes > sure it is stopped. > > -- > You received this message because you are subscribed to the Google > Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com > <mailto:puppet-users@googlegroups.com>. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@__googlegroups.com > <mailto:puppet-users%2Bunsubscribe@googlegroups.com>. > For more options, visit this group at > http://groups.google.com/__group/puppet-users?hl=en > <http://groups.google.com/group/puppet-users?hl=en>. > > > -- > You received this message because you are subscribed to the Google > Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en.-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
vagn scott
2011-Jul-05 15:05 UTC
Re: [Puppet Users] puppet terminating - continually forcing itself to stop
On 07/05/2011 09:24 AM, Martin Alfke wrote:> On 07/05/2011 03:19 PM, Brian Gallew wrote: > >> > I was seeing the behavior on my Solaris boxes when running Puppet under >> > SMF. The issue, in my case, was that I was trying to work around an SMF >> > bug. My "workaround" was to "svcadm disable puppetd;svccfg import >> > /var/svc/manifest/network/puppetd.xml;svcadm enable puppet". The astute >> > viewer will notice that "svcadm disable puppetd" will cause SMF to send >> > SIGTERM to puppet, at which point it will stop (as instructed) the >> > current run, so the new manifest will not be imported and the service >> > will not be re-started. It was all very amusing, except for the bit >> > where I had to fix a bunch of systems that weren''t running Puppet >> > anymore. Mea culpa. >> > I have seen a similar behavior on RHEL5. > /sbin/service puppet status gets information on actual running puppet run. > We fixed the puppet init script to verify whether the running puppet > process has parent pid 1 (init). > >The key to that sort of thing is to make sure the process that you launch has a life of its own. So, instead of essentially system (" svcadm disable puppetd; \ svccfg import /var/svc/manifest/network/puppetd.xml; \ svcadm enable puppet \ " ) add an extra set of parenthesis and put the script in the background: system (" ( sleep 1; svcadm disable puppetd; \ svccfg import /var/svc/manifest/network/puppetd.xml; \ svcadm enable puppet ) & \ " ) The sleep is there because, on linux at least, the child will run first otherwise. Well, that''s the basic idea. It''s been a while since I had to do anything like that. -- vagn -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Chris Phillips
2011-Jul-05 15:26 UTC
Re: [Puppet Users] puppet terminating - continually forcing itself to stop
On 5 July 2011 14:24, Martin Alfke <tuxmea@gmail.com> wrote:> On 07/05/2011 03:19 PM, Brian Gallew wrote: > > I was seeing the behavior on my Solaris boxes when running Puppet under > > SMF. The issue, in my case, was that I was trying to work around an SMF > > bug. My "workaround" was to "svcadm disable puppetd;svccfg import > > /var/svc/manifest/network/puppetd.xml;svcadm enable puppet". The astute > > viewer will notice that "svcadm disable puppetd" will cause SMF to send > > SIGTERM to puppet, at which point it will stop (as instructed) the > > current run, so the new manifest will not be imported and the service > > will not be re-started. It was all very amusing, except for the bit > > where I had to fix a bunch of systems that weren''t running Puppet > > anymore. Mea culpa. > > I have seen a similar behavior on RHEL5. > /sbin/service puppet status gets information on actual running puppet run. > We fixed the puppet init script to verify whether the running puppet > process has parent pid 1 (init). >Yeah, so the stop will try a bunch of ways to find the process, and if the pid file doesn''t exist then it just uses pidof. Hmm, not great. So I can see how this happens IF the stop is executed, however there still seems to be no reason to get there in the first place. That said if I try to run a stop during a run it doesn''t work. I''ve cleared the issue as described, and I wouldn''t expect that what I did (commenting out the ensure / enable) would actually do anything like clearing any old pid files or anything... And it''s by no means prevalent on any of the rhel5 systems we have. Thanks Chris -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Chris Phillips
2011-Jul-11 08:18 UTC
Re: [Puppet Users] puppet terminating - continually forcing itself to stop
On 5 July 2011 16:26, Chris Phillips <chris@untrepid.com> wrote:> > > On 5 July 2011 14:24, Martin Alfke <tuxmea@gmail.com> wrote: > >> On 07/05/2011 03:19 PM, Brian Gallew wrote: >> > I was seeing the behavior on my Solaris boxes when running Puppet under >> > SMF. The issue, in my case, was that I was trying to work around an SMF >> > bug. My "workaround" was to "svcadm disable puppetd;svccfg import >> > /var/svc/manifest/network/puppetd.xml;svcadm enable puppet". The astute >> > viewer will notice that "svcadm disable puppetd" will cause SMF to send >> > SIGTERM to puppet, at which point it will stop (as instructed) the >> > current run, so the new manifest will not be imported and the service >> > will not be re-started. It was all very amusing, except for the bit >> > where I had to fix a bunch of systems that weren''t running Puppet >> > anymore. Mea culpa. >> >> I have seen a similar behavior on RHEL5. >> /sbin/service puppet status gets information on actual running puppet run. >> We fixed the puppet init script to verify whether the running puppet >> process has parent pid 1 (init). >> > > Yeah, so the stop will try a bunch of ways to find the process, and if the > pid file doesn''t exist then it just uses pidof. Hmm, not great. So I can see > how this happens IF the stop is executed, however there still seems to be no > reason to get there in the first place. That said if I try to run a stop > during a run it doesn''t work. I''ve cleared the issue as described, and I > wouldn''t expect that what I did (commenting out the ensure / enable) would > actually do anything like clearing any old pid files or anything... And it''s > by no means prevalent on any of the rhel5 systems we have. > > Thanks >Hi, Was there any view about this being a bug? There is talk in this thread about solaris but I''m only using rhel boxes. Coming into work this morning I''ve seen a bunch of func processes have stopped and puppet is doing nothing at all about it, despite a manifest advising something else: Manifest says - service { "funcd": ensure => running, enable => true, subscribe => File["/etc/func/minion.conf"], require => Package["func"], hasrestart => true, hasstatus => true } Puppet run is not interested, thinks it''s still running? - debug: Service[funcd](provider=redhat): Executing ''/sbin/service funcd status'' debug: Puppet::Type::Service::ProviderRedhat: Executing ''/sbin/chkconfig funcd'' However as per the puppet service being turned off when it already is, this is being left off when it should be being turned on - i.e. in both situations it is thinking it is running when it is not - and the init script CONFIRMS that it is NOT running, just like it does with puppet. Thanks Chris -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Dominic Cleal
2011-Jul-11 13:54 UTC
Re: [Puppet Users] puppet terminating - continually forcing itself to stop
On 11/07/11 09:18, Chris Phillips wrote:> Coming into work this morning I''ve seen a bunch of func processes have > stopped and puppet is doing nothing at all about it, despite a manifest > advising something else: > > Manifest says - > > service { "funcd": > ensure => running, > enable => true, > subscribe => File["/etc/func/minion.conf"], > require => Package["func"], > hasrestart => true, > hasstatus => true > } > > > Puppet run is not interested, thinks it''s still running? - > > debug: Service[funcd](provider=redhat): Executing ''/sbin/service funcd > status'' > debug: Puppet::Type::Service::ProviderRedhat: Executing ''/sbin/chkconfig > funcd'' > > However as per the puppet service being turned off when it already is, > this is being left off when it should be being turned on - i.e. in both > situations it is thinking it is running when it is not - and the init > script CONFIRMS that it is NOT running, just like it does with puppet.Is the init.d ''status'' script returning a non-zero exit code? I''ve seen bugs before with scripts that report on stdout that the process isn''t running, but failed to return the correct exit code (e.g. 1 or 3 for LSB). -- Dominic Cleal Red Hat Consulting m: +44 (0)7818 512168 -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Chris Phillips
2011-Jul-11 15:24 UTC
Re: [Puppet Users] puppet terminating - continually forcing itself to stop
On 11 July 2011 14:54, Dominic Cleal <dcleal@redhat.com> wrote:> On 11/07/11 09:18, Chris Phillips wrote: > > Coming into work this morning I''ve seen a bunch of func processes have > > stopped and puppet is doing nothing at all about it, despite a manifest > > advising something else: > > > > Manifest says - > > > > service { "funcd": > > ensure => running, > > enable => true, > > subscribe => File["/etc/func/minion.conf"], > > require => Package["func"], > > hasrestart => true, > > hasstatus => true > > } > > > > > > Puppet run is not interested, thinks it''s still running? - > > > > debug: Service[funcd](provider=redhat): Executing ''/sbin/service funcd > > status'' > > debug: Puppet::Type::Service::ProviderRedhat: Executing ''/sbin/chkconfig > > funcd'' > > > > However as per the puppet service being turned off when it already is, > > this is being left off when it should be being turned on - i.e. in both > > situations it is thinking it is running when it is not - and the init > > script CONFIRMS that it is NOT running, just like it does with puppet. > > Is the init.d ''status'' script returning a non-zero exit code? I''ve seen > bugs before with scripts that report on stdout that the process isn''t > running, but failed to return the correct exit code (e.g. 1 or 3 for LSB). > >Yes in both cases the status is returned as a 3, which is correct for a stopped service Cheers Chris -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.