[Puppet Users] Re: Lock file /var/lib/puppet/state/puppetdlock

Luke Kanies
Tue, 17 Feb 2009 03:39:55 -0000

On Feb 10, 2009, at 4:25 AM, Keith Edmunds wrote:

>
> I''m just starting a roll out of Puppet and I''m seeing a
problem on
> maybe
> 25% of client nodes. The symptoms are that the clients stop
> updating. In
> the Puppetmaster log, I''m seeing things like:
>
> Feb  9 20:10:23 vs4 puppetmasterd[17942]: Compiled catalog for xxxx in
> 0.05 seconds
> Feb  9 20:40:41 vs4 puppetmasterd[17942]: Compiled catalog for xxxx in
> 0.05 seconds
> Feb  9 21:11:16 vs4 puppetmasterd[17942]: Compiled catalog for xxxx in
> 1.83 seconds
> Feb  9 21:41:37 vs4 puppetmasterd[17942]: Compiled catalog for xxxx in
> 0.91 seconds
>
> These are all for the same client; everything appears normal until
> 21:41,
> then no more checks from the client (it''s now 10:17 on Feb 10).
>
> On the client, I tried running puppetd manually:
>
> # puppetd -t
> notice: Lock file /var/lib/puppet/state/puppetdlock exists; skipping
> catalog run
>
> A look at the lock file:
>
> # ls -l /var/lib/puppet/state/puppetdlock
> -rw-r--r-- 1 root root 5 2009-02-09 22:11 /var/lib/puppet/state/
> puppetdlock
>
> ...shows that it was probably created at the next run after the last
> one
> logged on the Puppetmaster (above).
>
> Looking at the lock file:
>
> # echo $(cat /var/lib/puppet/state/puppetdlock)
> 32400
> # ps -fp 32400
> UID        PID  PPID  C STIME TTY          TIME CMD
> root     32400     1  0 Feb06 ?        00:01:41 ruby /usr/sbin/
> puppetd -w 5
>
> ...shows that the puppetd is still running.
>
> Why would the lock file be created and not subsequently deleted?
>
> If it helps, it is likely that the Puppetmaster was very busy at that
> time, but even so I would expect the client to deal with that
> graciously.
>
> Maybe related, maybe not: I can''t stop puppetd in the usual way:
>
> # /etc/init.d/puppet stop
> Stopping puppet configuration management tool.
> # ps -fp 32400
> UID        PID  PPID  C STIME TTY          TIME CMD
> root     32400     1  0 Feb06 ?        00:01:41 ruby /usr/sbin/
> puppetd -w 5
>
> If I ''kill -9'' the puppetd process, remove the
/var/run/puppetd.pid
> file
> and remove the lock file, I can restart puppetd and it runs OK for a
> while, but eventually the puppetdlock file causes this problem again.
>
> Versions: 0.24.5-3, the Debian Lenny package compiled for Debian Etch.
>
> Grateful for any suggestions / pointers / etc.

I haven''t seen this for absolute ages, so whatever used to cause it is
unlikely to be the problem now.

Is there maybe a signal being sent to puppetd?  If not, any chance you
can get a stack trace from the clients?    You''d have to leave a
client running in the foreground with stdout redirected.

Otherwise....  I''ve no idea, because you''re the first to run
into it
that I know of.

Note that you should be able to run ''puppetd --enable'' to
remove that
stale lock file.

--
One of the keys to happiness is a bad memory. -- Rita Mae Brown
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com