[Puppet Users] Lock file /var/lib/puppet/state/puppetdlock

Keith Edmunds
Tue, 10 Feb 2009 10:25:53 -0000

I''m just starting a roll out of Puppet and I''m seeing a
problem on maybe
25% of client nodes. The symptoms are that the clients stop updating. In
the Puppetmaster log, I''m seeing things like:

Feb  9 20:10:23 vs4 puppetmasterd[17942]: Compiled catalog for xxxx in
0.05 seconds
Feb  9 20:40:41 vs4 puppetmasterd[17942]: Compiled catalog for xxxx in
0.05 seconds
Feb  9 21:11:16 vs4 puppetmasterd[17942]: Compiled catalog for xxxx in
1.83 seconds
Feb  9 21:41:37 vs4 puppetmasterd[17942]: Compiled catalog for xxxx in
0.91 seconds

These are all for the same client; everything appears normal until 21:41,
then no more checks from the client (it''s now 10:17 on Feb 10).

On the client, I tried running puppetd manually:

# puppetd -t
notice: Lock file /var/lib/puppet/state/puppetdlock exists; skipping
catalog run

A look at the lock file:

# ls -l /var/lib/puppet/state/puppetdlock
-rw-r--r-- 1 root root 5 2009-02-09 22:11 /var/lib/puppet/state/puppetdlock

...shows that it was probably created at the next run after the last one
logged on the Puppetmaster (above).

Looking at the lock file:

# echo $(cat /var/lib/puppet/state/puppetdlock)
32400
# ps -fp 32400
UID        PID  PPID  C STIME TTY          TIME CMD
root     32400     1  0 Feb06 ?        00:01:41 ruby /usr/sbin/puppetd -w 5

...shows that the puppetd is still running.

Why would the lock file be created and not subsequently deleted?

If it helps, it is likely that the Puppetmaster was very busy at that
time, but even so I would expect the client to deal with that graciously.

Maybe related, maybe not: I can''t stop puppetd in the usual way:

# /etc/init.d/puppet stop
Stopping puppet configuration management tool.
# ps -fp 32400
UID        PID  PPID  C STIME TTY          TIME CMD
root     32400     1  0 Feb06 ?        00:01:41 ruby /usr/sbin/puppetd -w 5

If I ''kill -9'' the puppetd process, remove the
/var/run/puppetd.pid file
and remove the lock file, I can restart puppetd and it runs OK for a
while, but eventually the puppetdlock file causes this problem again.

Versions: 0.24.5-3, the Debian Lenny package compiled for Debian Etch.

Grateful for any suggestions / pointers / etc.

Thanks,
Keith