Puppet users - Jun 2012 - Puppet can't successfully restart puppetd

Hi,

I have a puppet module which is relatively simple; it just defines a
couple of packages (puppet & facter) and a couple of files
(/etc/puppet/puppet.conf, /etc/puppet/namespaceauth.conf and
/etc/puppet/auth.conf) and a service (puppet), which is subscribed to
the puppet package and the 3 config files.

I changed auth.conf and then witnessed all of the puppet agents pick
the apply the change, then got this in the logs:

puppet-agent[6139]: Caught TERM; calling stop

At this point, the puppet agent has obviously died, and I have to ssh
on to the boxes to kick it back off again.  Other services seem to
restart fine on a config file change; is there something special about
the puppet service that it issuing the equivalent of ''service puppet
restart'' doesn''t actually work?  I''m using
puppet-2.7.17 in case it
matters.

Thanks,

Matt.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 06/27/2012 10:51 AM, Matthew Burgess wrote:
> is there something special about
> the puppet service that it issuing the equivalent of ''service
puppet
> restart'' doesn''t actually work?
Oh, it works, but think of what you''re having puppet do: By calling the
initscript and telling it to restart, it first has to "stop puppet",
i.e., send TERM to the agent process. So the agent is committing suicide
mid-run, taking the invocation of the initscript with it. Now nobody''s
there to start puppet up again.

I''ve worked around this using exec { "echo service puppet restart
| at
now+5min" } in the past, but this is quite the hack. Personally I have
found the cron model much more convenient that the long-running agent.

HTH,
Felix

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Wed, Jun 27, 2012 at 10:01 AM, Felix Frank
<felix.frank@alumni.tu-berlin.de> wrote:
> On 06/27/2012 10:51 AM, Matthew Burgess wrote:
>> is there something special about
>> the puppet service that it issuing the equivalent of ''service
puppet
>> restart'' doesn''t actually work?
>
> Oh, it works, but think of what you''re having puppet do: By
calling the
> initscript and telling it to restart, it first has to "stop
puppet",
> i.e., send TERM to the agent process. So the agent is committing suicide
> mid-run, taking the invocation of the initscript with it. Now
nobody''s
> there to start puppet up again.
Yeah, I did think of that, but I assumed that the ''service puppet
restart'' command had been invoked already, i.e. before the puppet
agent had been killed.  Obviously, if the restart of the service is
being handled by two separate commands, e.g. ''service puppet
stop''
followed by ''service puppet start'' then yes, I completely
understand
how I''m in the situation I''m now in.
> I''ve worked around this using exec { "echo service puppet
restart | at
> now+5min" } in the past, but this is quite the hack. Personally I have
> found the cron model much more convenient that the long-running agent.
I''m using the long-running agent so that things like ''puppet
kick''
work from the master, but that too could be implemented by using
passwordless ssh keys and just sshing from the master to the managed
node and kicking off the agent that way, I guess.  I''ll give the
''at''
variation a try first though.  Thanks for the hint!

Regards,

Matt.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 27.06.2012, at 11:01, Felix Frank wrote:
> On 06/27/2012 10:51 AM, Matthew Burgess wrote:
>> is there something special about
>> the puppet service that it issuing the equivalent of ''service
puppet
>> restart'' doesn''t actually work?
> 
> Oh, it works, but think of what you''re having puppet do: By
calling the
> initscript and telling it to restart, it first has to "stop
puppet",
> i.e., send TERM to the agent process. So the agent is committing suicide
> mid-run, taking the invocation of the initscript with it. Now
nobody''s
> there to start puppet up again.
> 
> I''ve worked around this using exec { "echo service puppet
restart | at
> now+5min" } in the past, but this is quite the hack. Personally I have
> found the cron model much more convenient that the long-running agent.
We redefined the restart command:

restart => ''/usr/bin/nohup /etc/init.d/puppet restart
&''

- Martin

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Wed, Jun 27, 2012 at 10:13 AM, Martin Alfke <tuxmea@gmail.com> wrote:
> We redefined the restart command:
>
> restart => ''/usr/bin/nohup /etc/init.d/puppet restart
&''
Nice!  Thanks very much, Martin, that works really nicely.

Regards,

Matt.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 06/27/2012 11:19 AM, Matthew Burgess wrote:
>> We redefined the restart command:
>> >
>> > restart => ''/usr/bin/nohup /etc/init.d/puppet restart
&''
> Nice!  Thanks very much, Martin, that works really nicely.
Yeah, this looks really slick. Thanks for sharing.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Wednesday, 27 June 2012 11:31:17 UTC+2, Felix.Frank
wrote:
>
> On 06/27/2012 11:19 AM, Matthew Burgess wrote: 
> >> We redefined the restart command: 
> >> > 
> >> > restart => ''/usr/bin/nohup /etc/init.d/puppet
restart &''
> > Nice!  Thanks very much, Martin, that works really nicely. 
>
>
Before I blindly try this how does the above work... 

I call this midway through a puppet run and it does no harm to the current 
run but 
loads new configuration for next run. Is that what happens? 

Steve. 
 
> Yeah, this looks really slick. Thanks for sharing. 
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/K60PncD0EqMJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 03.07.2012, at 21:31, Steve Traylen wrote:
> On Wednesday, 27 June 2012 11:31:17 UTC+2, Felix.Frank wrote:
> On 06/27/2012 11:19 AM, Matthew Burgess wrote: 
> >> We redefined the restart command: 
> >> > 
> >> > restart => ''/usr/bin/nohup /etc/init.d/puppet
restart &''
> > Nice!  Thanks very much, Martin, that works really nicely. 
> 
> 
> Before I blindly try this how does the above work... 
> 
> I call this midway through a puppet run and it does no harm to the current
run but
> loads new configuration for next run. Is that what happens? 
No.
It will stop the actual puppet run.
Then it will again start puppet daemon and initiate a new run.
(restart is short hand for stop and start).

We do puppet configuration in an early stage.
So we are sure that we use actual puppet configuration when entering the main
stage.

- Martin

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Wed, Jun 27, 2012 at 7:13 PM, Martin Alfke <tuxmea@gmail.com>
wrote:
>
> On 27.06.2012, at 11:01, Felix Frank wrote:
>
>> On 06/27/2012 10:51 AM, Matthew Burgess wrote:
>>> is there something special about
>>> the puppet service that it issuing the equivalent of
''service puppet
>>> restart'' doesn''t actually work?
>>
>> Oh, it works, but think of what you''re having puppet do: By
calling the
>> initscript and telling it to restart, it first has to "stop
puppet",
>> i.e., send TERM to the agent process. So the agent is committing
suicide
>> mid-run, taking the invocation of the initscript with it. Now
nobody''s
>> there to start puppet up again.
>>
>> I''ve worked around this using exec { "echo service puppet
restart | at
>> now+5min" } in the past, but this is quite the hack. Personally I
have
>> found the cron model much more convenient that the long-running agent.
>
> We redefined the restart command:
>
> restart => ''/usr/bin/nohup /etc/init.d/puppet restart
&''
>
> - Martin
>
Late reply, however if you''re using Ubuntu 10.04 or greater you can
avoid the nohup:

hasrestart      => true;

Edward

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

thx,

the master is Centos 5.8
the client   is  RedHat 5.4



2012/7/4 Edward Savage <epssyis@gmail.com>
> On Wed, Jun 27, 2012 at 7:13 PM, Martin Alfke <tuxmea@gmail.com>
wrote:
> >
> > On 27.06.2012, at 11:01, Felix Frank wrote:
> >
> >> On 06/27/2012 10:51 AM, Matthew Burgess wrote:
> >>> is there something special about
> >>> the puppet service that it issuing the equivalent of
''service puppet
> >>> restart'' doesn''t actually work?
> >>
> >> Oh, it works, but think of what you''re having puppet do:
By calling the
> >> initscript and telling it to restart, it first has to "stop
puppet",
> >> i.e., send TERM to the agent process. So the agent is committing
suicide
> >> mid-run, taking the invocation of the initscript with it. Now
nobody''s
> >> there to start puppet up again.
> >>
> >> I''ve worked around this using exec { "echo service
puppet restart | at
> >> now+5min" } in the past, but this is quite the hack.
Personally I have
> >> found the cron model much more convenient that the long-running
agent.
> >
> > We redefined the restart command:
> >
> > restart => ''/usr/bin/nohup /etc/init.d/puppet restart
&''
> >
> > - Martin
> >
>
> Late reply, however if you''re using Ubuntu 10.04 or greater you
can
> avoid the nohup:
>
> hasrestart      => true;
>
> Edward
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>

-- 
 如有未尽事宜，请及时联系。分机-8752


-----
朱文  PeterZhu

集团技术部  OPS
----------------
办公电话：021-61821155-8753
移动手机：15000654460
办公传真：021-61821150


上海市浦东新区陆家嘴环路166号未来资产大厦10楼 安居客集团

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Here is how we resolved this problem (although we are now running puppet
through cron).  By issuing the HUP puppet will restart, but not until the
run has completed.  This is largely for bootstrapping or when the machine
environment changes.

If your running puppetd change ''puppet agent'' to
''puppetd''.  At least if
for the releases you listed, I would move away from running daemonized
puppet unless theres a special requirement/reason, and stick it in a cron
instead.

== Begin CopyPasta =exec { "restart-puppet":
command => "ps -ef | grep ''puppet agent'' | grep -v
grep | awk {''print \$2''}
| xargs -iXX echo kill -HUP XX | sh",
path => "/sbin:/bin:/usr/sbin:/usr/bin",
refreshonly=>true
}

file { "/etc/puppet/puppet.conf":
notify => Exec["restart-puppet"]
}
== End CopyPasta =
On Wed, Jul 4, 2012 at 4:03 AM, 朱文 <peterzhu@anjuke.com> wrote:
> thx,
>
> the master is Centos 5.8
> the client   is  RedHat 5.4
>
>
>
> 2012/7/4 Edward Savage <epssyis@gmail.com>
>
>> On Wed, Jun 27, 2012 at 7:13 PM, Martin Alfke <tuxmea@gmail.com>
wrote:
>> >
>> > On 27.06.2012, at 11:01, Felix Frank wrote:
>> >
>> >> On 06/27/2012 10:51 AM, Matthew Burgess wrote:
>> >>> is there something special about
>> >>> the puppet service that it issuing the equivalent of
''service puppet
>> >>> restart'' doesn''t actually work?
>> >>
>> >> Oh, it works, but think of what you''re having puppet
do: By calling the
>> >> initscript and telling it to restart, it first has to
"stop puppet",
>> >> i.e., send TERM to the agent process. So the agent is
committing
>> suicide
>> >> mid-run, taking the invocation of the initscript with it. Now
nobody''s
>> >> there to start puppet up again.
>> >>
>> >> I''ve worked around this using exec { "echo
service puppet restart | at
>> >> now+5min" } in the past, but this is quite the hack.
Personally I have
>> >> found the cron model much more convenient that the
long-running agent.
>> >
>> > We redefined the restart command:
>> >
>> > restart => ''/usr/bin/nohup /etc/init.d/puppet restart
&''
>> >
>> > - Martin
>> >
>>
>> Late reply, however if you''re using Ubuntu 10.04 or greater
you can
>> avoid the nohup:
>>
>> hasrestart      => true;
>>
>> Edward
>>
>> --
>> You received this message because you are subscribed to the Google
Groups
>> "Puppet Users" group.
>> To post to this group, send email to puppet-users@googlegroups.com.
>> To unsubscribe from this group, send email to
>> puppet-users+unsubscribe@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/puppet-users?hl=en.
>>
>>
>
>
> --
>  如有未尽事宜，请及时联系。分机-8752
>
>
> -----
> 朱文  PeterZhu
>
> 集团技术部  OPS
> ----------------
> 办公电话：021-61821155-8753
> 移动手机：15000654460
> 办公传真：021-61821150
>
>
> 上海市浦东新区陆家嘴环路166号未来资产大厦10楼 安居客集团
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 07/04/2012 03:58 PM, Throwe, Jesse wrote:
> If your running puppetd change ''puppet agent'' to
''puppetd''.  At least if
> for the releases you listed, I would move away from running daemonized
> puppet unless theres a special requirement/reason, and stick it in a
> cron instead.
+1
> command => "ps -ef | grep ''puppet agent'' | grep
-v grep | awk {''print
> \$2''} | xargs -iXX echo kill -HUP XX | sh",
*cringe* I advise the use of pgrep to retrieve pids.

Also, why not pipe to "xargs kill -HUP" directly?

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Non homogeneous environment.  If I recall we had an issue with a
simpler command like you suggested with old versions of Solaris.

On Wed, Jul 4, 2012 at 10:11 AM, Felix Frank
<felix.frank@alumni.tu-berlin.de> wrote:
> On 07/04/2012 03:58 PM, Throwe, Jesse wrote:
>> If your running puppetd change ''puppet agent'' to
''puppetd''.  At least if
>> for the releases you listed, I would move away from running daemonized
>> puppet unless theres a special requirement/reason, and stick it in a
>> cron instead.
>
> +1
>
>> command => "ps -ef | grep ''puppet agent'' |
grep -v grep | awk {''print
>> \$2''} | xargs -iXX echo kill -HUP XX | sh",
>
> *cringe* I advise the use of pgrep to retrieve pids.
>
> Also, why not pipe to "xargs kill -HUP" directly?
>
> --
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 07/04/2012 04:18 PM, Throwe, Jesse wrote:
> Non homogeneous environment.  If I recall we had an issue with a
> simpler command like you suggested with old versions of Solaris.
Oh, the S-word. Well, that *does* indeed answer to both remarks.
Condolences. ;-)

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.