Hi Guys, I am a new puppet user and wanted some type of monitoring for puppet so deployed puppet-dashboard. It has been working very well for a few days not, but all of a sudden I start getting pending tasks and they never finish even after restarting all processes. They keep accumulating and never seem to finish even though the clients are running fine. I have the puppet-dashboard running on a different server than my puppetmaster. Is there any way to troubleshoot what seems to be stuck and how to I get it going without having to destroy the dashboard mysql db and recreating it again? That is the only way I have got it working again but that is not the correct way of doing this :) Any help would be appreciated. Thanks! -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Make sure that your dashboard workers are running. If you are using CentOS then "service puppet-dashboard-workers status" will tell you. On Feb 8, 2013, at 1:53 PM, Jagga Soorma wrote:> I am a new puppet user and wanted some type of monitoring for puppet so deployed puppet-dashboard. It has been working very well for a few days not, but all of a sudden I start getting pending tasks and they never finish even after restarting all processes. They keep accumulating and never seem to finish even though the clients are running fine. I have the puppet-dashboard running on a different server than my puppetmaster. Is there any way to troubleshoot what seems to be stuck and how to I get it going without having to destroy the dashboard mysql db and recreating it again? That is the only way I have got it working again but that is not the correct way of doing this :) Any help would be appreciated. > > Thanks! > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > >-- Jo Rhett Net Consonance : net philanthropy to improve open source and internet projects. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Yes, the workers are definitely running and I have tried restarting puppet-dashboard-workers a few times as well as rebooting the vm completely without any luck. Something seems to be stuck somewhere but I can''t figure out what. This has happened a few times now and I have just been recreating from scratch to get it functional again. Are there any logs for the workers that might tell us why these pending jobs are stuck or not finishing? Thanks. On Fri, Feb 8, 2013 at 2:27 PM, Jo Rhett <jrhett@netconsonance.com> wrote:> Make sure that your dashboard workers are running. If you are using CentOS > then "service puppet-dashboard-workers status" will tell you. > > On Feb 8, 2013, at 1:53 PM, Jagga Soorma wrote: > > I am a new puppet user and wanted some type of monitoring for puppet so > deployed puppet-dashboard. It has been working very well for a few days > not, but all of a sudden I start getting pending tasks and they never > finish even after restarting all processes. They keep accumulating and > never seem to finish even though the clients are running fine. I have the > puppet-dashboard running on a different server than my puppetmaster. Is > there any way to troubleshoot what seems to be stuck and how to I get it > going without having to destroy the dashboard mysql db and recreating it > again? That is the only way I have got it working again but that is not > the correct way of doing this :) Any help would be appreciated. > > Thanks! > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > > > > > -- > Jo Rhett > Net Consonance : net philanthropy to improve open source and internet > projects. > > > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Anyone out there run into this issue before or have any idea about how to troubleshoot such a problem? Any help would be appreciated. I can''t keep rebuilding puppet dashboard every time this happens. Thanks On Friday, February 8, 2013 2:37:55 PM UTC-8, Jagga Soorma wrote:> > Yes, the workers are definitely running and I have tried restarting > puppet-dashboard-workers a few times as well as rebooting the vm completely > without any luck. Something seems to be stuck somewhere but I can''t figure > out what. This has happened a few times now and I have just been > recreating from scratch to get it functional again. Are there any logs for > the workers that might tell us why these pending jobs are stuck or not > finishing? > > Thanks. > > On Fri, Feb 8, 2013 at 2:27 PM, Jo Rhett <jrhett@netconsonance.com> wrote: > >> Make sure that your dashboard workers are running. If you are using >> CentOS then "service puppet-dashboard-workers status" will tell you. >> >> On Feb 8, 2013, at 1:53 PM, Jagga Soorma wrote: >> >> I am a new puppet user and wanted some type of monitoring for puppet so >> deployed puppet-dashboard. It has been working very well for a few days >> not, but all of a sudden I start getting pending tasks and they never >> finish even after restarting all processes. They keep accumulating and >> never seem to finish even though the clients are running fine. I have the >> puppet-dashboard running on a different server than my puppetmaster. Is >> there any way to troubleshoot what seems to be stuck and how to I get it >> going without having to destroy the dashboard mysql db and recreating it >> again? That is the only way I have got it working again but that is not >> the correct way of doing this :) Any help would be appreciated. >> >> Thanks! >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to puppet-users+unsubscribe@googlegroups.com. >> To post to this group, send email to puppet-users@googlegroups.com. >> Visit this group at http://groups.google.com/group/puppet-users?hl=en. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> >> >> -- >> Jo Rhett >> Net Consonance : net philanthropy to improve open source and internet >> projects. >> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to puppet-users+unsubscribe@googlegroups.com. >> To post to this group, send email to puppet-users@googlegroups.com. >> Visit this group at http://groups.google.com/group/puppet-users?hl=en. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Try looking in /usr/share/puppet-dashboard/log/. Did you update the dashboard or puppet master recently? I have ran into similar problems after updates, since I run the dashboard on a different port than the external_nodes script uses. The DASHBOARD_URL in /usr/share/puppet-dashboard/bin/external_node got changed back into localhost:3000 and as a result, all my nodes were seen as unresponsive. On 02/10/2013 04:11 AM, Jagga Soorma wrote:> Anyone out there run into this issue before or have any idea about how > to troubleshoot such a problem? Any help would be appreciated. I > can''t keep rebuilding puppet dashboard every time this happens. > > Thanks > > On Friday, February 8, 2013 2:37:55 PM UTC-8, Jagga Soorma wrote: > > Yes, the workers are definitely running and I have tried > restarting puppet-dashboard-workers a few times as well as > rebooting the vm completely without any luck. Something seems to > be stuck somewhere but I can''t figure out what. This has happened > a few times now and I have just been recreating from scratch to > get it functional again. Are there any logs for the workers that > might tell us why these pending jobs are stuck or not finishing? > > Thanks. > > On Fri, Feb 8, 2013 at 2:27 PM, Jo Rhett <jrhett@netconsonance.com > <mailto:jrhett@netconsonance.com>> wrote: > > Make sure that your dashboard workers are running. If you are > using CentOS then "service puppet-dashboard-workers status" > will tell you. > > On Feb 8, 2013, at 1:53 PM, Jagga Soorma wrote: >> I am a new puppet user and wanted some type of monitoring for >> puppet so deployed puppet-dashboard. It has been working >> very well for a few days not, but all of a sudden I start >> getting pending tasks and they never finish even after >> restarting all processes. They keep accumulating and never >> seem to finish even though the clients are running fine. I >> have the puppet-dashboard running on a different server than >> my puppetmaster. Is there any way to troubleshoot what seems >> to be stuck and how to I get it going without having to >> destroy the dashboard mysql db and recreating it again? That >> is the only way I have got it working again but that is not >> the correct way of doing this :) Any help would be appreciated. >> >> Thanks! >> >> -- >> You received this message because you are subscribed to the >> Google Groups "Puppet Users" group. >> To unsubscribe from this group and stop receiving emails from >> it, send an email to >> puppet-users+unsubscribe@googlegroups.com >> <mailto:puppet-users+unsubscribe@googlegroups.com>. >> To post to this group, send email to >> puppet-users@googlegroups.com >> <mailto:puppet-users@googlegroups.com>. >> Visit this group at >> http://groups.google.com/group/puppet-users?hl=en >> <http://groups.google.com/group/puppet-users?hl=en>. >> For more options, visit >> https://groups.google.com/groups/opt_out >> <https://groups.google.com/groups/opt_out>. >> >> > > -- > Jo Rhett > Net Consonance : net philanthropy to improve open source and > internet projects. > > > > -- > You received this message because you are subscribed to the > Google Groups "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from > it, send an email to puppet-users+unsubscribe@googlegroups.com > <mailto:puppet-users%2Bunsubscribe@googlegroups.com>. > To post to this group, send email to > puppet-users@googlegroups.com > <mailto:puppet-users@googlegroups.com>. > Visit this group at > http://groups.google.com/group/puppet-users?hl=en > <http://groups.google.com/group/puppet-users?hl=en>. > For more options, visit > https://groups.google.com/groups/opt_out > <https://groups.google.com/groups/opt_out>. > > > > -- > You received this message because you are subscribed to the Google > Groups "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Are you having the problem after running ''puppet agent --test'' a few times or it just happens without any manual run ? I was having the same problem some days ago, in my case because I was running ''puppet agent --test'', more specifically the --show_diff that is used in --test was automatically generating a lot of output and causing the puppet dashboard workers to fail and increase the processing queue. Look at /usr/share/puppet-dashboard/log/delayed_job.log (or whatever your dashboard location is) and check what error it is giving. You can clean the queue manually without reinstalling everything: service puppet-dashboard-workers stop cd /usr/share/puppet-dashboard rm -f spool/* rake jobs:clear RAILS_ENV=production service puppet-dashboard-workers start Regards, Felipe On Saturday, February 9, 2013 7:11:04 PM UTC-8, Jagga Soorma wrote:> > Anyone out there run into this issue before or have any idea about how to > troubleshoot such a problem? Any help would be appreciated. I can''t keep > rebuilding puppet dashboard every time this happens. > > Thanks > > On Friday, February 8, 2013 2:37:55 PM UTC-8, Jagga Soorma wrote: >> >> Yes, the workers are definitely running and I have tried restarting >> puppet-dashboard-workers a few times as well as rebooting the vm completely >> without any luck. Something seems to be stuck somewhere but I can''t figure >> out what. This has happened a few times now and I have just been >> recreating from scratch to get it functional again. Are there any logs for >> the workers that might tell us why these pending jobs are stuck or not >> finishing? >> >> Thanks. >> >> On Fri, Feb 8, 2013 at 2:27 PM, Jo Rhett <jrh...@netconsonance.com<javascript:> >> > wrote: >> >>> Make sure that your dashboard workers are running. If you are using >>> CentOS then "service puppet-dashboard-workers status" will tell you. >>> >>> On Feb 8, 2013, at 1:53 PM, Jagga Soorma wrote: >>> >>> I am a new puppet user and wanted some type of monitoring for puppet so >>> deployed puppet-dashboard. It has been working very well for a few days >>> not, but all of a sudden I start getting pending tasks and they never >>> finish even after restarting all processes. They keep accumulating and >>> never seem to finish even though the clients are running fine. I have the >>> puppet-dashboard running on a different server than my puppetmaster. Is >>> there any way to troubleshoot what seems to be stuck and how to I get it >>> going without having to destroy the dashboard mysql db and recreating it >>> again? That is the only way I have got it working again but that is not >>> the correct way of doing this :) Any help would be appreciated. >>> >>> Thanks! >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Puppet Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to puppet-users...@googlegroups.com <javascript:>. >>> To post to this group, send email to puppet...@googlegroups.com<javascript:> >>> . >>> Visit this group at http://groups.google.com/group/puppet-users?hl=en. >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >>> >>> >>> >>> -- >>> Jo Rhett >>> Net Consonance : net philanthropy to improve open source and internet >>> projects. >>> >>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Puppet Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to puppet-users...@googlegroups.com <javascript:>. >>> To post to this group, send email to puppet...@googlegroups.com<javascript:> >>> . >>> Visit this group at http://groups.google.com/group/puppet-users?hl=en. >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >>> >>> >> >>-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Thanks for pointing me to the log location. Looks like this might be the
cause of the problem:
--
..snip
2013-02-10T10:40:35-0800: [Worker(delayed_job.3 host:ssfpcond01 pid:25048)]
Report.create_from_yaml_file failed with ActiveRecord::StatementInvalid:
*Mysql::Error:
Data too long for column ''details'' at row 1: INSERT INTO
`delayed_job_failures` *(`summary`, `read`, `updated_at`, `backtrace`,
`details`, `created_at`) VALUES(''Importing report report-18121-1179.
yaml'', 0, ''2013-02-10 18:40:35'', ''--- \n-
/usr/share/puppet-dashboard/vendor/rails/activerecord/lib/active_record/connection_adapters/a
..snip..
--
I am managing my passwd & group file via puppet which at times have huge
amount of changes especially in the group file. This could possibly be
causing this error. Is there a way to increase the size of the details
column to enable it to accept larger data?
Has anyone seen this issue before?
Thanks again for your help with this.
On Friday, February 8, 2013 1:53:02 PM UTC-8, Jagga Soorma
wrote:>
> Hi Guys,
>
> I am a new puppet user and wanted some type of monitoring for puppet so
> deployed puppet-dashboard. It has been working very well for a few days
> not, but all of a sudden I start getting pending tasks and they never
> finish even after restarting all processes. They keep accumulating and
> never seem to finish even though the clients are running fine. I have the
> puppet-dashboard running on a different server than my puppetmaster. Is
> there any way to troubleshoot what seems to be stuck and how to I get it
> going without having to destroy the dashboard mysql db and recreating it
> again? That is the only way I have got it working again but that is not
> the correct way of doing this :) Any help would be appreciated.
>
> Thanks!
>
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
Looks like that was actually the problem. Searched google and came across the fix. Had to change the type of the messages column to mediumtext and the pending jobs started finishing. Did this by: alter table report_logs change column message message VARCHAR(65536); Just in case someone runs into this issue. Thanks. On Sunday, February 10, 2013 10:56:14 AM UTC-8, Jagga Soorma wrote:> > Thanks for pointing me to the log location. Looks like this might be the > cause of the problem: > > -- > ..snip > 2013-02-10T10:40:35-0800: [Worker(delayed_job.3 host:ssfpcond01 > pid:25048)] Report.create_from_yaml_file failed with > ActiveRecord::StatementInvalid: *Mysql::Error: Data too long for column > ''details'' at row 1: INSERT INTO `delayed_job_failures` *(`summary`, > `read`, `updated_at`, `backtrace`, `details`, `created_at`) > VALUES(''Importing report report-18121-1179. > yaml'', 0, ''2013-02-10 18:40:35'', ''--- \n- > /usr/share/puppet-dashboard/vendor/rails/activerecord/lib/active_record/connection_adapters/a > ..snip.. > -- > > I am managing my passwd & group file via puppet which at times have huge > amount of changes especially in the group file. This could possibly be > causing this error. Is there a way to increase the size of the details > column to enable it to accept larger data? > > Has anyone seen this issue before? > > Thanks again for your help with this. > > On Friday, February 8, 2013 1:53:02 PM UTC-8, Jagga Soorma wrote: >> >> Hi Guys, >> >> I am a new puppet user and wanted some type of monitoring for puppet so >> deployed puppet-dashboard. It has been working very well for a few days >> not, but all of a sudden I start getting pending tasks and they never >> finish even after restarting all processes. They keep accumulating and >> never seem to finish even though the clients are running fine. I have the >> puppet-dashboard running on a different server than my puppetmaster. Is >> there any way to troubleshoot what seems to be stuck and how to I get it >> going without having to destroy the dashboard mysql db and recreating it >> again? That is the only way I have got it working again but that is not >> the correct way of doing this :) Any help would be appreciated. >> >> Thanks! >> >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Jesse Martinich
2013-Jun-05 19:36 UTC
Re: [Puppet Users] Re: Puppet dashboard stuck pending jobs
Your ''alter table'' fix worked for me too. Thanks! -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Sean Penticoff
2013-Sep-06 20:21 UTC
Re: [Puppet Users] Re: Puppet dashboard stuck pending jobs
I ran into this as a result of the output of failed deps on a yum install and had a couple jobs that just wouldn''t go away. After fixing the condition creating the error I used /usr/bin/rake jobs:clear to clear out the delayed_job queue and everything was back to normal. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
That fixed the problem for me. Thanks. On Friday, September 6, 2013 10:21:12 PM UTC+2, Psyber wrote:> > I ran into this as a result of the output of failed deps on a yum > install and had a couple jobs that just wouldn''t go away. After fixing > the condition creating the error I used /usr/bin/rake jobs:clear to > clear out the delayed_job queue and everything was back to normal. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
Julien Nephtali
2013-Oct-21 17:12 UTC
Re: [Puppet Users] Re: Puppet dashboard stuck pending jobs
I found the solution to my problem here: http://projects.puppetlabs.com/issues/18411 ALTER TABLE delayed_job_failures MODIFY details BLOB; that resolved the error. On Friday, October 11, 2013 7:07:02 AM UTC-4, cko wrote:> > That fixed the problem for me. Thanks. > > On Friday, September 6, 2013 10:21:12 PM UTC+2, Psyber wrote: >> >> I ran into this as a result of the output of failed deps on a yum >> install and had a couple jobs that just wouldn''t go away. After fixing >> the condition creating the error I used /usr/bin/rake jobs:clear to >> clear out the delayed_job queue and everything was back to normal. >> >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.