Felipe Salum
2013-Jul-24 18:42 UTC
[Puppet Users] Problem running rake node:del for Puppet Dashboard databse hosted on RDS
Hi guys, The command below was working when my puppet dashboard mysql database was running in the same machine as the puppet master. rake RAILS_ENV=production -f /usr/share/puppet-dashboard/Rakefile node:del name=my-app-server However now I moved the mysql database to RDS and it doesn''t work anymore, my database.yml reflects the change and the Puppet Dashboard web interface works fine accessing the RDS database, but the rake task to delete is broken. Interesting is that node:list works and list all my nodes. production: database: dashboard_production username: username password: password host: something.at.rds.amazonaws.com encoding: utf8 adapter: mysql Did I miss something during the local mysql to RDS migration ? Thanks, Felipe -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
Felipe Salum
2013-Jul-24 20:49 UTC
[Puppet Users] Re: Problem running rake node:del for Puppet Dashboard databse hosted on RDS
I had a friend helping me to debug and looks like it is taking forever to delete the entries from resource status table (a lot of entries). Is that something we can improve ? On Wednesday, July 24, 2013 11:42:26 AM UTC-7, Felipe Salum wrote:> > Hi guys, > > The command below was working when my puppet dashboard mysql database was > running in the same machine as the puppet master. > > rake RAILS_ENV=production -f /usr/share/puppet-dashboard/Rakefile node:del > name=my-app-server > > However now I moved the mysql database to RDS and it doesn''t work anymore, > my database.yml reflects the change and the Puppet Dashboard web interface > works fine accessing the RDS database, but the rake task to delete is > broken. Interesting is that node:list works and list all my nodes. > > production: > database: dashboard_production > username: username > password: password > host: something.at.rds.amazonaws.com > encoding: utf8 > adapter: mysql > > Did I miss something during the local mysql to RDS migration ? > > Thanks, > Felipe >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
Ramin K
2013-Jul-24 21:03 UTC
Re: [Puppet Users] Re: Problem running rake node:del for Puppet Dashboard databse hosted on RDS
On 7/24/2013 1:49 PM, Felipe Salum wrote:> I had a friend helping me to debug and looks like it is taking forever > to delete the entries from resource status table (a lot of entries). > > Is that something we can improve ?env RAILS_ENV=production rake reports:prune upto=5 unit=day env RAILS_ENV=production rake reports:prune:orphaned You can change the RAILS_ENV and number of day, weeks, months, etc to match your system and its needs. Sounds like you''ve never pruned so it''s going to be slow going. Try walking it forward like so. env RAILS_ENV=production rake reports:prune upto=3 unit=mon env RAILS_ENV=production rake reports:prune upto=1 unit=mon env RAILS_ENV=production rake reports:prune upto=2 unit=wk env RAILS_ENV=production rake reports:prune upto=5 unit=day Don''t forget to add this to your crontab with the retentions settings that best fit your system. Ramin -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
Felipe Salum
2013-Jul-24 21:08 UTC
Re: [Puppet Users] Re: Problem running rake node:del for Puppet Dashboard databse hosted on RDS
I actually do for 2 weeks. rake RAILS_ENV=production reports:prune upto=2 unit=wk On Wed, Jul 24, 2013 at 2:03 PM, Ramin K <ramin-list@badapple.net> wrote:> On 7/24/2013 1:49 PM, Felipe Salum wrote: > >> I had a friend helping me to debug and looks like it is taking forever >> to delete the entries from resource status table (a lot of entries). >> >> Is that something we can improve ? >> > > env RAILS_ENV=production rake reports:prune upto=5 unit=day > env RAILS_ENV=production rake reports:prune:orphaned > > You can change the RAILS_ENV and number of day, weeks, months, etc to > match your system and its needs. > > Sounds like you''ve never pruned so it''s going to be slow going. Try > walking it forward like so. > > env RAILS_ENV=production rake reports:prune upto=3 unit=mon > env RAILS_ENV=production rake reports:prune upto=1 unit=mon > env RAILS_ENV=production rake reports:prune upto=2 unit=wk > env RAILS_ENV=production rake reports:prune upto=5 unit=day > > Don''t forget to add this to your crontab with the retentions settings that > best fit your system. > > Ramin > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Puppet Users" group. > To unsubscribe from this topic, visit https://groups.google.com/d/** > topic/puppet-users/7-**sjp1Ef1p0/unsubscribe<https://groups.google.com/d/topic/puppet-users/7-sjp1Ef1p0/unsubscribe> > . > To unsubscribe from this group and all its topics, send an email to > puppet-users+unsubscribe@**googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com> > . > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/**group/puppet-users<http://groups.google.com/group/puppet-users> > . > For more options, visit https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out> > . > > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
Ramin K
2013-Jul-24 22:21 UTC
Re: [Puppet Users] Re: Problem running rake node:del for Puppet Dashboard databse hosted on RDS
Doesn''t matter, it''s the size of the data and the amount you''re deleting. Let''s say that your reports:prune upto=2 unit=wk is ultimately going to delete 100k records because you''ve never pruned the database before. reports:prune upto=2 unit=wk, 100k, time: 3600s By breaking it up, it''ll run faster because large deletes can cause poor performance in Mysql. reports:prune upto=9 unit=wk, deletes: 9k, time: 280s reports:prune upto=8 unit=wk, deletes: 11k, time: 250s reports:prune upto=7 unit=wk, deletes: 10k, time: 251s reports:prune upto=6 unit=wk, deletes: 12k, time: 239s reports:prune upto=5 unit=wk, deletes: 11k, time: 205s reports:prune upto=4 unit=wk, deletes: 11k, time: 180s reports:prune upto=2 unit=wk, deletes: 22k, time: 200s Running it nightly with a fully prune data set. reports:prune upto=2 unit=wk, deletes: 1.5k, time: 25s ** I made up all these numbers up, but the performance ratio should be roughly correct based on past experience. ymmv. ** Also the Rake task will load the id of the records it plans to delete into memory. This can be quite large and is another reason to break it into smaller chunks. Ramin On 7/24/2013 2:08 PM, Felipe Salum wrote:> I actually do for 2 weeks. > > rake RAILS_ENV=production reports:prune upto=2 unit=wk > > > > On Wed, Jul 24, 2013 at 2:03 PM, Ramin K <ramin-list@badapple.net > <mailto:ramin-list@badapple.net>> wrote: > > On 7/24/2013 1:49 PM, Felipe Salum wrote: > > I had a friend helping me to debug and looks like it is taking > forever > to delete the entries from resource status table (a lot of entries). > > Is that something we can improve ? > > > env RAILS_ENV=production rake reports:prune upto=5 unit=day > env RAILS_ENV=production rake reports:prune:orphaned > > You can change the RAILS_ENV and number of day, weeks, months, etc > to match your system and its needs. > > Sounds like you''ve never pruned so it''s going to be slow going. Try > walking it forward like so. > > env RAILS_ENV=production rake reports:prune upto=3 unit=mon > env RAILS_ENV=production rake reports:prune upto=1 unit=mon > env RAILS_ENV=production rake reports:prune upto=2 unit=wk > env RAILS_ENV=production rake reports:prune upto=5 unit=day > > Don''t forget to add this to your crontab with the retentions > settings that best fit your system. > > Ramin > > -- > You received this message because you are subscribed to a topic in > the Google Groups "Puppet Users" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/__topic/puppet-users/7-__sjp1Ef1p0/unsubscribe > <https://groups.google.com/d/topic/puppet-users/7-sjp1Ef1p0/unsubscribe>. > To unsubscribe from this group and all its topics, send an email to > puppet-users+unsubscribe@__googlegroups.com > <mailto:puppet-users%2Bunsubscribe@googlegroups.com>. > To post to this group, send email to puppet-users@googlegroups.com > <mailto:puppet-users@googlegroups.com>. > Visit this group at http://groups.google.com/__group/puppet-users > <http://groups.google.com/group/puppet-users>. > For more options, visit https://groups.google.com/__groups/opt_out > <https://groups.google.com/groups/opt_out>. > > > > -- > You received this message because you are subscribed to the Google > Groups "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users. > For more options, visit https://groups.google.com/groups/opt_out. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
Felipe Salum
2013-Jul-24 22:28 UTC
Re: [Puppet Users] Re: Problem running rake node:del for Puppet Dashboard databse hosted on RDS
I understood, what I wanted to say is that I run a cron job every day for the 2 weeks prune.. and I have less than 2 weeks data in the database right now since I moved it recently to a new server. I''m running a optimize task, the resource_statuses is really big. On Wed, Jul 24, 2013 at 3:21 PM, Ramin K <ramin-list@badapple.net> wrote:> Doesn''t matter, it''s the size of the data and the amount you''re > deleting. > > Let''s say that your reports:prune upto=2 unit=wk is ultimately going to > delete 100k records because you''ve never pruned the database before. > > reports:prune upto=2 unit=wk, 100k, time: 3600s > > By breaking it up, it''ll run faster because large deletes can cause poor > performance in Mysql. > > reports:prune upto=9 unit=wk, deletes: 9k, time: 280s > reports:prune upto=8 unit=wk, deletes: 11k, time: 250s > reports:prune upto=7 unit=wk, deletes: 10k, time: 251s > reports:prune upto=6 unit=wk, deletes: 12k, time: 239s > reports:prune upto=5 unit=wk, deletes: 11k, time: 205s > reports:prune upto=4 unit=wk, deletes: 11k, time: 180s > reports:prune upto=2 unit=wk, deletes: 22k, time: 200s > > Running it nightly with a fully prune data set. > reports:prune upto=2 unit=wk, deletes: 1.5k, time: 25s > > ** I made up all these numbers up, but the performance ratio should be > roughly correct based on past experience. ymmv. ** > > Also the Rake task will load the id of the records it plans to delete into > memory. This can be quite large and is another reason to break it into > smaller chunks. > > Ramin > > > On 7/24/2013 2:08 PM, Felipe Salum wrote: > >> I actually do for 2 weeks. >> >> rake RAILS_ENV=production reports:prune upto=2 unit=wk >> >> >> >> On Wed, Jul 24, 2013 at 2:03 PM, Ramin K <ramin-list@badapple.net >> <mailto:ramin-list@badapple.**net <ramin-list@badapple.net>>> wrote: >> >> On 7/24/2013 1:49 PM, Felipe Salum wrote: >> >> I had a friend helping me to debug and looks like it is taking >> forever >> to delete the entries from resource status table (a lot of >> entries). >> >> Is that something we can improve ? >> >> >> env RAILS_ENV=production rake reports:prune upto=5 unit=day >> env RAILS_ENV=production rake reports:prune:orphaned >> >> You can change the RAILS_ENV and number of day, weeks, months, etc >> to match your system and its needs. >> >> Sounds like you''ve never pruned so it''s going to be slow going. Try >> walking it forward like so. >> >> env RAILS_ENV=production rake reports:prune upto=3 unit=mon >> env RAILS_ENV=production rake reports:prune upto=1 unit=mon >> env RAILS_ENV=production rake reports:prune upto=2 unit=wk >> env RAILS_ENV=production rake reports:prune upto=5 unit=day >> >> Don''t forget to add this to your crontab with the retentions >> settings that best fit your system. >> >> Ramin >> >> -- >> You received this message because you are subscribed to a topic in >> the Google Groups "Puppet Users" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/__**topic/puppet-users/7-__** >> sjp1Ef1p0/unsubscribe<https://groups.google.com/d/__topic/puppet-users/7-__sjp1Ef1p0/unsubscribe> >> <https://groups.google.com/d/**topic/puppet-users/7-** >> sjp1Ef1p0/unsubscribe<https://groups.google.com/d/topic/puppet-users/7-sjp1Ef1p0/unsubscribe> >> >. >> >> To unsubscribe from this group and all its topics, send an email to >> puppet-users+unsubscribe@__goo**glegroups.com<http://googlegroups.com> >> <mailto:puppet-users%**2Bunsubscribe@googlegroups.com<puppet-users%252Bunsubscribe@googlegroups.com> >> **>. >> >> To post to this group, send email to puppet-users@googlegroups.com >> <mailto:puppet-users@**googlegroups.com<puppet-users@googlegroups.com> >> >. >> Visit this group at http://groups.google.com/__**group/puppet-users<http://groups.google.com/__group/puppet-users> >> <http://groups.google.com/**group/puppet-users<http://groups.google.com/group/puppet-users> >> >. >> For more options, visit https://groups.google.com/__**groups/opt_out<https://groups.google.com/__groups/opt_out> >> <https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out> >> >. >> >> >> >> -- >> You received this message because you are subscribed to the Google >> Groups "Puppet Users" group. >> To unsubscribe from this group and stop receiving emails from it, send >> >> an email to puppet-users+unsubscribe@**googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com> >> . >> To post to this group, send email to puppet-users@googlegroups.com. >> Visit this group at http://groups.google.com/**group/puppet-users<http://groups.google.com/group/puppet-users> >> . >> For more options, visit https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out> >> . >> >> >> > -- > You received this message because you are subscribed to a topic in the > Google Groups "Puppet Users" group. > To unsubscribe from this topic, visit https://groups.google.com/d/** > topic/puppet-users/7-**sjp1Ef1p0/unsubscribe<https://groups.google.com/d/topic/puppet-users/7-sjp1Ef1p0/unsubscribe> > . > To unsubscribe from this group and all its topics, send an email to > puppet-users+unsubscribe@**googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com> > . > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/**group/puppet-users<http://groups.google.com/group/puppet-users> > . > For more options, visit https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out> > . > > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.