Arnau Bria
2011-Mar-24 15:57 UTC
[Puppet Users] pupet dashboard taking more than 10 minutes to destroy a node
Hi all,
I''ve noticed that puppet-dashboard takes a lot of time to refresh some
webs, but today, I''ve started to remove some old nodes and it takes
more than 10 minutes to remove one.
The host hasn''t a really high load:
[root@puppet01 ~]# uptime
16:54:31 up 14 days, 5:17, 1 user, load average: 1.67, 1.48, 1.08
[root@puppet01 ~]# free -m
total used free shared buffers cached
Mem: 2026 1890 136 0 4 266
# grep -c "^proc" /proc/cpuinfo
2
and, obviously, ruby & mysql are eating all the cpu:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
31770 mysql 22 0 138m 28m 4616 S 117.2 1.4 18:42.69 mysqld
5111 root 16 0 861m 839m 2492 S 1.0 41.4 931:05.87 ruby
Anyone notices this behaviour? any advice for solving this?
puppet-dashboard-1.0.4.tgz
TIA,
Arnau
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
Stefan Goethals
2011-Mar-24 16:06 UTC
Re: [Puppet Users] pupet dashboard taking more than 10 minutes to destroy a node
On 24 Mar 2011, at 16:57, Arnau Bria wrote:> Hi all, > > I''ve noticed that puppet-dashboard takes a lot of time to refresh some > webs, but today, I''ve started to remove some old nodes and it takes > more than 10 minutes to remove one. > > The host hasn''t a really high load: > > [root@puppet01 ~]# uptime > 16:54:31 up 14 days, 5:17, 1 user, load average: 1.67, 1.48, 1.08 > [root@puppet01 ~]# free -m > total used free shared buffers cached > Mem: 2026 1890 136 0 4 266 > > # grep -c "^proc" /proc/cpuinfo > 2 > > and, obviously, ruby & mysql are eating all the cpu: > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 31770 mysql 22 0 138m 28m 4616 S 117.2 1.4 18:42.69 mysqld > 5111 root 16 0 861m 839m 2492 S 1.0 41.4 931:05.87 ruby > > Anyone notices this behaviour? any advice for solving this?Dashboard removes the reports for nodes one by one... This can take a long time with many reports. Solutions : - Delete reports in the db with a delete statement. - run the cleanup rake task for reports as explained on the github page. Regards, Stefan - Zipkid - Goethals> > puppet-dashboard-1.0.4.tgz > > > TIA, > Arnau > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Arnau Bria
2011-Mar-24 16:31 UTC
Re: [Puppet Users] pupet dashboard taking more than 10 minutes to destroy a node
On Thu, 24 Mar 2011 17:06:33 +0100 Stefan Goethals wrote:> Dashboard removes the reports for nodes one by one... > This can take a long time with many reports. > > Solutions : > - Delete reports in the db with a delete statement. > - run the cleanup rake task for reports as explained on the github > page.thanks. ''ll take a look.> > Regards, > > Stefan - Zipkid - GoethalsCheers, Arnau -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Randall Hansen
2011-Mar-24 16:36 UTC
Re: [Puppet Users] pupet dashboard taking more than 10 minutes to destroy a node
On Mar 24, 2011, at 9:06 AM, Stefan Goethals wrote:> Dashboard removes the reports for nodes one by one... > This can take a long time with many reports.If this is true (and I have no reason to doubt it), it''s Very Bad. I''ve filed it as a bug: http://projects.puppetlabs.com/issues/6828 If this affects you, please watch the bug. r -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Martin Willemsma
2011-Mar-24 18:59 UTC
Re: [Puppet Users] pupet dashboard taking more than 10 minutes to destroy a node
Yes, I''m experiencing the exact same problem when hitting the destroy button. Mysql and ruby1.8 are eating all resources. In my test env I had the agent runinterval set to 600 seconds to get lots of reports and stress the server a little. When I tried to take the node out of service by hitting ''DESTROY'' it took ages and I eventually killed the process myself. I keep a close eye on the bug. -- Kind regards Martin Willemsma 2011/3/24 Randall Hansen <randall@puppetlabs.com>> On Mar 24, 2011, at 9:06 AM, Stefan Goethals wrote: > > > Dashboard removes the reports for nodes one by one... > > This can take a long time with many reports. > > If this is true (and I have no reason to doubt it), it''s Very Bad. I''ve > filed it as a bug: > > http://projects.puppetlabs.com/issues/6828 > > If this affects you, please watch the bug. > > r > > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Arnau Bria
2011-Mar-24 22:32 UTC
Re: [Puppet Users] pupet dashboard taking more than 10 minutes to destroy a node
Hi again, I''m cleaning my DB (which was 15GB) following: https://github.com/puppetlabs/puppet-dashboard I did: rake RAILS_ENV=production reports:prune upto=1 unit=wk rake RAILS_ENV=development reports:prune upto=1 unit=wk (It has been running for about 3 months) and after: rake RAILS_ENV=production db:raw:optimize rake RAILS_ENV=development db:raw:optimize It has removed lots of entries, but it''s still 15GB. Am I doing something wrong? why is it not decreasing its size? TIA, Arnau -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Cody Robertson
2011-Mar-24 22:49 UTC
Re: [Puppet Users] pupet dashboard taking more than 10 minutes to destroy a node
On 03/24/2011 06:32 PM, Arnau Bria wrote:> Hi again, > > > I''m cleaning my DB (which was 15GB) following: > https://github.com/puppetlabs/puppet-dashboard > > I did: > > rake RAILS_ENV=production reports:prune upto=1 unit=wk > rake RAILS_ENV=development reports:prune upto=1 unit=wk > > (It has been running for about 3 months) > > and after: > > rake RAILS_ENV=production db:raw:optimize > rake RAILS_ENV=development db:raw:optimize > > It has removed lots of entries, but it''s still 15GB. > > Am I doing something wrong? why is it not decreasing its size? > > > TIA, > Arnau >This is because the dashboard uses InnoDB tables. You''ll probably have a huge `ibdata` file in /var/lib/mysql/ somewhere. It''s a known bug but the `ibdata` file grows but doesn''t shrink. http://bugs.mysql.com/bug.php?id=1341 I''d suggest making MySQL use a different file per *.ibd file as it makes it easier to see which table is the culprit. From there you should be able to figure out a way to have the files re-created once you''ve pruned them (albeit no "good" way that I know of). http://dev.mysql.com/doc/refman/5.0/en/innodb-multiple-tablespaces.html -- Cody Robertson 1-800-859-8803 ext. 5 -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Cody Robertson
2011-Mar-24 23:26 UTC
Re: [Puppet Users] pupet dashboard taking more than 10 minutes to destroy a node
On 03/24/2011 06:49 PM, Cody Robertson wrote:> On 03/24/2011 06:32 PM, Arnau Bria wrote: >> Hi again, >> >> >> I''m cleaning my DB (which was 15GB) following: >> https://github.com/puppetlabs/puppet-dashboard >> >> I did: >> >> rake RAILS_ENV=production reports:prune upto=1 unit=wk >> rake RAILS_ENV=development reports:prune upto=1 unit=wk >> >> (It has been running for about 3 months) >> >> and after: >> >> rake RAILS_ENV=production db:raw:optimize >> rake RAILS_ENV=development db:raw:optimize >> >> It has removed lots of entries, but it''s still 15GB. >> >> Am I doing something wrong? why is it not decreasing its size? >> >> >> TIA, >> Arnau >> > This is because the dashboard uses InnoDB tables. You''ll probably have > a huge `ibdata` file in /var/lib/mysql/ somewhere. It''s a known bug > but the `ibdata` file grows but doesn''t shrink. > > http://bugs.mysql.com/bug.php?id=1341 > > I''d suggest making MySQL use a different file per *.ibd file as it > makes it easier to see which table is the culprit. From there you > should be able to figure out a way to have the files re-created once > you''ve pruned them (albeit no "good" way that I know of). > > http://dev.mysql.com/doc/refman/5.0/en/innodb-multiple-tablespaces.html >Just to clarify - InnoDB should re-use that space in the ibdata file until it needs more at which point it''ll increase the size. This is why it''s important to run the cleanup jobs fairly frequently. You can also set the max size of the files I believe. Here''s some information (it looks like they mention a procedure to shrink existing ibdata files that aren''t fully used as well): http://dev.mysql.com/doc/refman/5.0/en/innodb-data-log-reconfiguration.html -- Cody Robertson 1-800-859-8803 ext. 5 -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Arnau Bria
2011-Mar-24 23:33 UTC
Re: [Puppet Users] pupet dashboard taking more than 10 minutes to destroy a node
Hi Cody, Thanks for both replies! tomorrow morning I''ll take a look to links. Cheers, Arnau -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Arnau Bria
2011-Mar-25 09:27 UTC
Re: [Puppet Users] pupet dashboard taking more than 10 minutes to destroy a node
Hi, After some prune /all excpet last week)/optimize the destroy action works great.. Thanks for the tips, now I have to take a lokk at mysql links. Cheers, Arnau -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.