Puppet users - Feb 2009 - Lock file /var/lib/puppet/state/puppetdlock

I''m just starting a roll out of Puppet and I''m seeing a
problem on maybe
25% of client nodes. The symptoms are that the clients stop updating. In
the Puppetmaster log, I''m seeing things like:

Feb  9 20:10:23 vs4 puppetmasterd[17942]: Compiled catalog for xxxx in
0.05 seconds 
Feb  9 20:40:41 vs4 puppetmasterd[17942]: Compiled catalog for xxxx in
0.05 seconds 
Feb  9 21:11:16 vs4 puppetmasterd[17942]: Compiled catalog for xxxx in
1.83 seconds 
Feb  9 21:41:37 vs4 puppetmasterd[17942]: Compiled catalog for xxxx in
0.91 seconds

These are all for the same client; everything appears normal until 21:41,
then no more checks from the client (it''s now 10:17 on Feb 10).

On the client, I tried running puppetd manually:

# puppetd -t
notice: Lock file /var/lib/puppet/state/puppetdlock exists; skipping
catalog run

A look at the lock file:

# ls -l /var/lib/puppet/state/puppetdlock
-rw-r--r-- 1 root root 5 2009-02-09 22:11 /var/lib/puppet/state/puppetdlock

...shows that it was probably created at the next run after the last one
logged on the Puppetmaster (above).

Looking at the lock file:

# echo $(cat /var/lib/puppet/state/puppetdlock)
32400
# ps -fp 32400
UID        PID  PPID  C STIME TTY          TIME CMD
root     32400     1  0 Feb06 ?        00:01:41 ruby /usr/sbin/puppetd -w 5

...shows that the puppetd is still running.

Why would the lock file be created and not subsequently deleted?

If it helps, it is likely that the Puppetmaster was very busy at that
time, but even so I would expect the client to deal with that graciously.

Maybe related, maybe not: I can''t stop puppetd in the usual way:

# /etc/init.d/puppet stop
Stopping puppet configuration management tool.
# ps -fp 32400
UID        PID  PPID  C STIME TTY          TIME CMD
root     32400     1  0 Feb06 ?        00:01:41 ruby /usr/sbin/puppetd -w 5

If I ''kill -9'' the puppetd process, remove the
/var/run/puppetd.pid file
and remove the lock file, I can restart puppetd and it runs OK for a
while, but eventually the puppetdlock file causes this problem again.

Versions: 0.24.5-3, the Debian Lenny package compiled for Debian Etch.

Grateful for any suggestions / pointers / etc.

Thanks,
Keith

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

maybe not related but similar problem... Does puppet changes your time
(ntp)? If the time changes while puppet is running it could lead for a
very long time that puppet is waiting....

On 2/10/09, Keith Edmunds <kae@midnighthax.com>
wrote:
>
> I''m just starting a roll out of Puppet and I''m seeing a
problem on maybe
> 25% of client nodes. The symptoms are that the clients stop updating. In
> the Puppetmaster log, I''m seeing things like:
>
> Feb  9 20:10:23 vs4 puppetmasterd[17942]: Compiled catalog for xxxx in
> 0.05 seconds
> Feb  9 20:40:41 vs4 puppetmasterd[17942]: Compiled catalog for xxxx in
> 0.05 seconds
> Feb  9 21:11:16 vs4 puppetmasterd[17942]: Compiled catalog for xxxx in
> 1.83 seconds
> Feb  9 21:41:37 vs4 puppetmasterd[17942]: Compiled catalog for xxxx in
> 0.91 seconds
>
> These are all for the same client; everything appears normal until 21:41,
> then no more checks from the client (it''s now 10:17 on Feb 10).
>
> On the client, I tried running puppetd manually:
>
> # puppetd -t
> notice: Lock file /var/lib/puppet/state/puppetdlock exists; skipping
> catalog run
>
> A look at the lock file:
>
> # ls -l /var/lib/puppet/state/puppetdlock
> -rw-r--r-- 1 root root 5 2009-02-09 22:11 /var/lib/puppet/state/puppetdlock
>
> ...shows that it was probably created at the next run after the last one
> logged on the Puppetmaster (above).
>
> Looking at the lock file:
>
> # echo $(cat /var/lib/puppet/state/puppetdlock)
> 32400
> # ps -fp 32400
> UID        PID  PPID  C STIME TTY          TIME CMD
> root     32400     1  0 Feb06 ?        00:01:41 ruby /usr/sbin/puppetd -w 5
>
> ...shows that the puppetd is still running.
>
> Why would the lock file be created and not subsequently deleted?
>
> If it helps, it is likely that the Puppetmaster was very busy at that
> time, but even so I would expect the client to deal with that graciously.
>
> Maybe related, maybe not: I can''t stop puppetd in the usual way:
>
> # /etc/init.d/puppet stop
> Stopping puppet configuration management tool.
> # ps -fp 32400
> UID        PID  PPID  C STIME TTY          TIME CMD
> root     32400     1  0 Feb06 ?        00:01:41 ruby /usr/sbin/puppetd -w 5
>
> If I ''kill -9'' the puppetd process, remove the
/var/run/puppetd.pid file
> and remove the lock file, I can restart puppetd and it runs OK for a
> while, but eventually the puppetdlock file causes this problem again.
>
> Versions: 0.24.5-3, the Debian Lenny package compiled for Debian Etch.
>
> Grateful for any suggestions / pointers / etc.
>
> Thanks,
> Keith
>
> >
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

> Does puppet changes your time
> (ntp)? If the time changes while puppet is running it could lead for a
> very long time that puppet is waiting....
No, although all servers are running NTP. I don''t think that''s
the
problem: the problem is the stale puppetdlock.

Thanks for the suggestion -
Keith

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Feb 10, 2009, at 4:25 AM, Keith Edmunds wrote:
>
> I''m just starting a roll out of Puppet and I''m seeing a
problem on
> maybe
> 25% of client nodes. The symptoms are that the clients stop  
> updating. In
> the Puppetmaster log, I''m seeing things like:
>
> Feb  9 20:10:23 vs4 puppetmasterd[17942]: Compiled catalog for xxxx in
> 0.05 seconds
> Feb  9 20:40:41 vs4 puppetmasterd[17942]: Compiled catalog for xxxx in
> 0.05 seconds
> Feb  9 21:11:16 vs4 puppetmasterd[17942]: Compiled catalog for xxxx in
> 1.83 seconds
> Feb  9 21:41:37 vs4 puppetmasterd[17942]: Compiled catalog for xxxx in
> 0.91 seconds
>
> These are all for the same client; everything appears normal until  
> 21:41,
> then no more checks from the client (it''s now 10:17 on Feb 10).
>
> On the client, I tried running puppetd manually:
>
> # puppetd -t
> notice: Lock file /var/lib/puppet/state/puppetdlock exists; skipping
> catalog run
>
> A look at the lock file:
>
> # ls -l /var/lib/puppet/state/puppetdlock
> -rw-r--r-- 1 root root 5 2009-02-09 22:11 /var/lib/puppet/state/ 
> puppetdlock
>
> ...shows that it was probably created at the next run after the last  
> one
> logged on the Puppetmaster (above).
>
> Looking at the lock file:
>
> # echo $(cat /var/lib/puppet/state/puppetdlock)
> 32400
> # ps -fp 32400
> UID        PID  PPID  C STIME TTY          TIME CMD
> root     32400     1  0 Feb06 ?        00:01:41 ruby /usr/sbin/ 
> puppetd -w 5
>
> ...shows that the puppetd is still running.
>
> Why would the lock file be created and not subsequently deleted?
>
> If it helps, it is likely that the Puppetmaster was very busy at that
> time, but even so I would expect the client to deal with that  
> graciously.
>
> Maybe related, maybe not: I can''t stop puppetd in the usual way:
>
> # /etc/init.d/puppet stop
> Stopping puppet configuration management tool.
> # ps -fp 32400
> UID        PID  PPID  C STIME TTY          TIME CMD
> root     32400     1  0 Feb06 ?        00:01:41 ruby /usr/sbin/ 
> puppetd -w 5
>
> If I ''kill -9'' the puppetd process, remove the
/var/run/puppetd.pid
> file
> and remove the lock file, I can restart puppetd and it runs OK for a
> while, but eventually the puppetdlock file causes this problem again.
>
> Versions: 0.24.5-3, the Debian Lenny package compiled for Debian Etch.
>
> Grateful for any suggestions / pointers / etc.
I haven''t seen this for absolute ages, so whatever used to cause it is
unlikely to be the problem now.

Is there maybe a signal being sent to puppetd?  If not, any chance you  
can get a stack trace from the clients?    You''d have to leave a  
client running in the foreground with stdout redirected.

Otherwise....  I''ve no idea, because you''re the first to run
into it
that I know of.

Note that you should be able to run ''puppetd --enable'' to
remove that
stale lock file.

-- 
One of the keys to happiness is a bad memory. -- Rita Mae Brown
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Mon, Feb 16, 2009 at 7:39 PM, Luke Kanies <luke@madstop.com>
wrote:
> Otherwise....  I''ve no idea, because you''re the first to
run into it
> that I know of.
>
> Note that you should be able to run ''puppetd --enable'' to
remove that
> stale lock file.
I periodically run into this issue as well; it only happens on our Xen
VMs. Not sure if that helps.

I just have a cron job that runs puppetd --enable on these systems.

--Paul

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

We have had this happening occasionally (approx. 1 out of ~120 VMs, 
every other week or so) on our OpenVZ VMs as well.

We''re running Debian Etch, and over the past week upgraded to 0.24.5-3,
so I''m keeping an eye on things to see if that resolved it for us.

- Jeff

Paul Lathrop wrote:
> On Mon, Feb 16, 2009 at 7:39 PM, Luke Kanies <luke@madstop.com>
wrote:
>> Otherwise....  I''ve no idea, because you''re the first
to run into it
>> that I know of.
>>
>> Note that you should be able to run ''puppetd
--enable'' to remove that
>> stale lock file.
> 
> I periodically run into this issue as well; it only happens on our Xen
> VMs. Not sure if that helps.
> 
> I just have a cron job that runs puppetd --enable on these systems.
> 
> --Paul
> 
> > 
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Mon, 16 Feb 2009 21:39:55 -0600, luke@madstop.com said:
> Note that you should be able to run ''puppetd --enable'' to
remove that
> stale lock file.
Old thread revisited. Some clients occasionally don''t clear the
lockfile
in the subject. A couple of people have suggested, as above, that "puppetd
--enable" should clear it. It doesn''t:

# ls -l /var/lib/puppet/state/puppetdlock
-rw-r--r-- 1 root root 5 2009-03-20 14:50 /var/lib/puppet/state/puppetdlock
# puppetd --enable
# ls -l /var/lib/puppet/state/puppetdlock
-rw-r--r-- 1 root root 5 2009-03-20 14:50 /var/lib/puppet/state/puppetdlock
#

My only recourse now would seem to be a cron job that checks the age of
the puppetdlock and restarts the puppet client if the lockfile is older
than a given age.

This is with:

# puppetd --version
0.24.5
# 

Any ideas as to why this is happening?

Thanks,
Keith

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

We are also having this similar issue with v2.7.10.1.  Short term fix is 
deleting the lockfile that hasn''t updated in 2x{puppet-run-cycle} and 
restarting puppet. 

On Tuesday, February 10, 2009 2:25:53 AM UTC-8, Keith Edmunds
wrote:
>
> I''m just starting a roll out of Puppet and I''m seeing a
problem on maybe
> 25% of client nodes. The symptoms are that the clients stop updating. In
> the Puppetmaster log, I''m seeing things like:
>
> Feb  9 20:10:23 vs4 puppetmasterd[17942]: Compiled catalog for xxxx in
> 0.05 seconds 
> Feb  9 20:40:41 vs4 puppetmasterd[17942]: Compiled catalog for xxxx in
> 0.05 seconds 
> Feb  9 21:11:16 vs4 puppetmasterd[17942]: Compiled catalog for xxxx in
> 1.83 seconds 
> Feb  9 21:41:37 vs4 puppetmasterd[17942]: Compiled catalog for xxxx in
> 0.91 seconds
>
> These are all for the same client; everything appears normal until 21:41,
> then no more checks from the client (it''s now 10:17 on Feb 10).
>
> On the client, I tried running puppetd manually:
>
> # puppetd -t
> notice: Lock file /var/lib/puppet/state/puppetdlock exists; skipping
> catalog run
>
> A look at the lock file:
>
> # ls -l /var/lib/puppet/state/puppetdlock
> -rw-r--r-- 1 root root 5 2009-02-09 22:11 /var/lib/puppet/state/puppetdlock
>
> ...shows that it was probably created at the next run after the last one
> logged on the Puppetmaster (above).
>
> Looking at the lock file:
>
> # echo $(cat /var/lib/puppet/state/puppetdlock)
> 32400
> # ps -fp 32400
> UID        PID  PPID  C STIME TTY          TIME CMD
> root     32400     1  0 Feb06 ?        00:01:41 ruby /usr/sbin/puppetd -w 5
>
> ...shows that the puppetd is still running.
>
> Why would the lock file be created and not subsequently deleted?
>
> If it helps, it is likely that the Puppetmaster was very busy at that
> time, but even so I would expect the client to deal with that graciously.
>
> Maybe related, maybe not: I can''t stop puppetd in the usual way:
>
> # /etc/init.d/puppet stop
> Stopping puppet configuration management tool.
> # ps -fp 32400
> UID        PID  PPID  C STIME TTY          TIME CMD
> root     32400     1  0 Feb06 ?        00:01:41 ruby /usr/sbin/puppetd -w 5
>
> If I ''kill -9'' the puppetd process, remove the
/var/run/puppetd.pid file
> and remove the lock file, I can restart puppetd and it runs OK for a
> while, but eventually the puppetdlock file causes this problem again.
>
> Versions: 0.24.5-3, the Debian Lenny package compiled for Debian Etch.
>
> Grateful for any suggestions / pointers / etc.
>
> Thanks,
> Keith
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.