Dave Mankoff
2012-Dec-12 01:55 UTC
[Puppet Users] All Hosts Shown as Unresponsive. Worker Restart Fails. Clients Seem Happy?
[On my dev server] For the past 3 days, my puppet master has shown all of
the hosts as unresponsive. However, when I log into one of the hosts and
run "puppet agent --test" everything seems to run without complaint.
Meanwhile the master shows an ever increasing number of pending tasks.
When I try to restart the dashboard workers, I get the following:
# /etc/init.d/pe-puppet-dashboard-workers restart
* Restarting PE Puppet Dashboard Workers
ERROR: there is already one or more instance(s) of the program running
ERROR: there is already one or more instance(s) of the program running
* PE Puppet Dashboard Worker (pid ) is not running
[fail]
Rebooting the computer does not seem to help. For that matter, I think this
problem started after a recent system update and reboot. It is running
Ubuntu 12.04. I can not seem to find anything meaningful in the logs,
though perhaps I am looking in the wrong places. Any advice? Things worked
fine just a few days ago, and I have not touched Puppet.
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/KcbSltHMeroJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
Mike Beauchamp
2013-Feb-07 18:44 UTC
[Puppet Users] Re: All Hosts Shown as Unresponsive. Worker Restart Fails. Clients Seem Happy?
I was able to fix this by going into the
/etc/init.d/pe-puppet-dashboard-workers and editing the CPU line in the
start section.
Previously it had {CPU:-2}, and I changed that to 4. Instantly cleared
the queue, and nodes showed up.
On Tuesday, December 11, 2012 6:55:56 PM UTC-7, Dave Mankoff
wrote:>
> [On my dev server] For the past 3 days, my puppet master has shown all of
> the hosts as unresponsive. However, when I log into one of the hosts and
> run "puppet agent --test" everything seems to run without
complaint.
> Meanwhile the master shows an ever increasing number of pending tasks.
>
> When I try to restart the dashboard workers, I get the following:
>
> # /etc/init.d/pe-puppet-dashboard-workers restart
> * Restarting PE Puppet Dashboard Workers
> ERROR: there is already one or more instance(s) of the program running
> ERROR: there is already one or more instance(s) of the program running
> * PE Puppet Dashboard Worker (pid ) is not running
>
> [fail]
>
> Rebooting the computer does not seem to help. For that matter, I think
> this problem started after a recent system update and reboot. It is running
> Ubuntu 12.04. I can not seem to find anything meaningful in the logs,
> though perhaps I am looking in the wrong places. Any advice? Things worked
> fine just a few days ago, and I have not touched Puppet.
>
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
Gary Larizza
2013-Feb-13 19:40 UTC
Re: [Puppet Users] Re: All Hosts Shown as Unresponsive. Worker Restart Fails. Clients Seem Happy?
I think there might be a bug buried here. I hit this before a demo this
morning - the problem was that the pid files at
/opt/puppet/share/puppet-dashboard/tmp/pids/delayed_job.{0,1}.pid were
present but there was no PID in them (and the background workers
weren''t
running). Removing the pidfiles allowed the service to start, but we need
to discover why these files are being created empty. My master was on
Ubuntu 12.04 and PE 2.7.0
On Thu, Feb 7, 2013 at 10:44 AM, Mike Beauchamp
<beauchompers@gmail.com>wrote:
> I was able to fix this by going into the
> /etc/init.d/pe-puppet-dashboard-workers and editing the CPU line in the
> start section.
>
> Previously it had {CPU:-2}, and I changed that to 4. Instantly cleared
> the queue, and nodes showed up.
>
>
> On Tuesday, December 11, 2012 6:55:56 PM UTC-7, Dave Mankoff wrote:
>>
>> [On my dev server] For the past 3 days, my puppet master has shown all
of
>> the hosts as unresponsive. However, when I log into one of the hosts
and
>> run "puppet agent --test" everything seems to run without
complaint.
>> Meanwhile the master shows an ever increasing number of pending tasks.
>>
>> When I try to restart the dashboard workers, I get the following:
>>
>> # /etc/init.d/pe-puppet-**dashboard-workers restart
>> * Restarting PE Puppet Dashboard Workers
>> ERROR: there is already one or more instance(s) of the program running
>> ERROR: there is already one or more instance(s) of the program running
>> * PE Puppet Dashboard Worker (pid ) is not running
>>
>> [fail]
>>
>> Rebooting the computer does not seem to help. For that matter, I think
>> this problem started after a recent system update and reboot. It is
running
>> Ubuntu 12.04. I can not seem to find anything meaningful in the logs,
>> though perhaps I am looking in the wrong places. Any advice? Things
worked
>> fine just a few days ago, and I have not touched Puppet.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscribe@googlegroups.com.
>
> To post to this group, send email to puppet-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/puppet-users?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>
--
Gary Larizza
Professional Services Engineer
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
Erik Jones
2013-Apr-01 14:57 UTC
[Puppet Users] Re: All Hosts Shown as Unresponsive. Worker Restart Fails. Clients Seem Happy?
There is a pid file in /var/run/puppet-dashboard.pid, but this file isn''t what''s stopping the startup. Puppet Dashboard also stores PID information in /usr/share/puppet-dashboard/tmp/pids. You must remove /var/run/puppet-dashboard.pid and /usr/share/puppet-dashboard/tmp/pids/*. Then you''ll be able to start it just fine. Modifying the CPU settings only enables additional workers, but doesn''t actually unblock the first two that won''t start. The bug is that it actually doesn''t check to make sure there are still processes running on those pids listed in the /usr/share/puppet-dashboard/tmp/pids/*.pid files. Erik On Tuesday, December 11, 2012 5:55:56 PM UTC-8, Dave Mankoff wrote:> > [On my dev server] For the past 3 days, my puppet master has shown all of > the hosts as unresponsive. However, when I log into one of the hosts and > run "puppet agent --test" everything seems to run without complaint. > Meanwhile the master shows an ever increasing number of pending tasks. > > When I try to restart the dashboard workers, I get the following: > > # /etc/init.d/pe-puppet-dashboard-workers restart > * Restarting PE Puppet Dashboard Workers > ERROR: there is already one or more instance(s) of the program running > ERROR: there is already one or more instance(s) of the program running > * PE Puppet Dashboard Worker (pid ) is not running > > [fail] > > Rebooting the computer does not seem to help. For that matter, I think > this problem started after a recent system update and reboot. It is running > Ubuntu 12.04. I can not seem to find anything meaningful in the logs, > though perhaps I am looking in the wrong places. Any advice? Things worked > fine just a few days ago, and I have not touched Puppet. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.