Russell Van Tassell
2012-Jan-03 22:52 UTC
[Puppet Users] Puppet Dashboard: db:migrate failure (1.2.2 to 1.2.4)
An error I managed to hit today, trying to migrate puppet dashboard from 1.2.2 to 1.2.4 (after upgrading the master to puppet 2.7.9 seemed to prevent the dashboard from importing new reports). I''m trying to re-run it now, but as you might guess, the DB is pretty huge and the process takes a while to execute. Executed from the puppet-dashboard directory: sudo rake db:migrate RAILS_ENV=production == AddReportForeignKeyConstraints: migrating ================================Going to delete orphaned records from metrics, report_logs, resource_statuses, resource_events Preparing to delete from metrics 2012-01-03 14:16:07: Deleting 0 orphaned records from metrics Deleting 100% |##########################################| Time: 00:00:00 Preparing to delete from report_logs 2012-01-03 14:16:41: Deleting 0 orphaned records from report_logs Deleting 100% |##########################################| Time: 00:00:00 Preparing to delete from resource_statuses 2012-01-03 14:16:51: Deleting 0 orphaned records from resource_statuses Deleting 100% |##########################################| Time: 00:00:00 Preparing to delete from resource_events 2012-01-03 14:19:13: Deleting 0 orphaned records from resource_events Deleting 100% |##########################################| Time: 00:00:00 -- execute("ALTER TABLE reports ADD CONSTRAINT fk_reports_node_id FOREIGN KEY (node_id) REFERENCES nodes(id) ON DELETE CASCADE;") -> 96.6684s -- execute("ALTER TABLE resource_events ADD CONSTRAINT fk_resource_events_resource_status_id FOREIGN KEY (resource_status_id) REFERENCES resource_statuses(id) ON DELETE CASCADE;") rake aborted! An error has occurred, all later migrations canceled: Mysql::Error: The total number of locks exceeds the lock table size: ALTER TABLE resource_events ADD CONSTRAINT fk_resource_events_resource_status_id FOREIGN KEY (resource_status_id) REFERENCES resource_statuses(id) ON DELETE CASCADE; Tasks: TOP => db:migrate (See full trace by running task with --trace) On a side note, this process is rather noisy, firing off a bunch of warnings from the start: NOTE: Gem.source_index is deprecated, use Specification. It will be removed on or after 2011-11-01. Gem.source_index called from /opt/puppet-dashboard-1.2.4/config/../vendor/rails/railties/lib/rails/gem_dependency.rb:21. NOTE: Gem::SourceIndex#initialize is deprecated with no replacement. It will be removed on or after 2011-11-01. Gem::SourceIndex#initialize called from /opt/puppet-dashboard-1.2.4/config/../vendor/rails/railties/lib/rails/vendor_gem_source_index.rb:100. NOTE: Gem::SourceIndex#add_spec is deprecated, use Specification.add_spec. It will be removed on or after 2011-11-01. Gem::SourceIndex#add_spec called from /usr/lib/ruby/site_ruby/1.8/rubygems/source_index.rb:91. NOTE: Gem::SourceIndex#add_spec is deprecated, use Specification.add_spec. It will be removed on or after 2011-11-01. Gem::SourceIndex#add_spec called from /usr/lib/ruby/site_ruby/1.8/rubygems/source_index.rb:91. NOTE: Gem::SourceIndex#add_spec is deprecated, use Specification.add_spec. It will be removed on or after 2011-11-01. Gem::SourceIndex#add_spec called from /usr/lib/ruby/site_ruby/1.8/rubygems/source_index.rb:91. NOTE: Gem::SourceIndex#add_spec is deprecated, use Specification.add_spec. It will be removed on or after 2011-11-01. Gem::SourceIndex#add_spec called from /usr/lib/ruby/site_ruby/1.8/rubygems/source_index.rb:91. NOTE: Gem::SourceIndex#add_spec is deprecated, use Specification.add_spec. It will be removed on or after 2011-11-01. Gem::SourceIndex#add_spec called from /usr/lib/ruby/site_ruby/1.8/rubygems/source_index.rb:91. NOTE: Gem::SourceIndex#add_spec is deprecated, use Specification.add_spec. It will be removed on or after 2011-11-01. Gem::SourceIndex#add_spec called from /usr/lib/ruby/site_ruby/1.8/rubygems/source_index.rb:91. NOTE: Gem::SourceIndex#add_spec is deprecated, use Specification.add_spec. It will be removed on or after 2011-11-01. Gem::SourceIndex#add_spec called from /usr/lib/ruby/site_ruby/1.8/rubygems/source_index.rb:91. NOTE: Gem::SourceIndex#add_spec is deprecated, use Specification.add_spec. It will be removed on or after 2011-11-01. Gem::SourceIndex#add_spec called from /usr/lib/ruby/site_ruby/1.8/rubygems/source_index.rb:91. NOTE: Gem::SourceIndex#add_spec is deprecated, use Specification.add_spec. It will be removed on or after 2011-11-01. Gem::SourceIndex#add_spec called from /usr/lib/ruby/site_ruby/1.8/rubygems/source_index.rb:91. NOTE: Gem::SourceIndex#add_spec is deprecated, use Specification.add_spec. It will be removed on or after 2011-11-01. Gem::SourceIndex#add_spec called from /usr/lib/ruby/site_ruby/1.8/rubygems/source_index.rb:91. NOTE: Gem::SourceIndex#add_spec is deprecated, use Specification.add_spec. It will be removed on or after 2011-11-01. Gem::SourceIndex#add_spec called from /usr/lib/ruby/site_ruby/1.8/rubygems/source_index.rb:91. NOTE: Gem::SourceIndex#add_spec is deprecated, use Specification.add_spec. It will be removed on or after 2011-11-01. Gem::SourceIndex#add_spec called from /usr/lib/ruby/site_ruby/1.8/rubygems/source_index.rb:91. NOTE: Gem::SourceIndex#add_spec is deprecated, use Specification.add_spec. It will be removed on or after 2011-11-01. Gem::SourceIndex#add_spec called from /usr/lib/ruby/site_ruby/1.8/rubygems/source_index.rb:91. NOTE: Gem::SourceIndex#add_spec is deprecated, use Specification.add_spec. It will be removed on or after 2011-11-01. Gem::SourceIndex#add_spec called from /usr/lib/ruby/site_ruby/1.8/rubygems/source_index.rb:91. NOTE: Gem::SourceIndex#add_spec is deprecated, use Specification.add_spec. It will be removed on or after 2011-11-01. Gem::SourceIndex#add_spec called from /usr/lib/ruby/site_ruby/1.8/rubygems/source_index.rb:91. NOTE: Gem::SourceIndex#add_spec is deprecated, use Specification.add_spec. It will be removed on or after 2011-11-01. Gem::SourceIndex#add_spec called from /usr/lib/ruby/site_ruby/1.8/rubygems/source_index.rb:91. NOTE: Gem::SourceIndex#add_spec is deprecated, use Specification.add_spec. It will be removed on or after 2011-11-01. Gem::SourceIndex#add_spec called from /usr/lib/ruby/site_ruby/1.8/rubygems/source_index.rb:91. NOTE: Gem::SourceIndex#add_spec is deprecated, use Specification.add_spec. It will be removed on or after 2011-11-01. Gem::SourceIndex#add_spec called from /usr/lib/ruby/site_ruby/1.8/rubygems/source_index.rb:91. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Daniel Pittman
2012-Jan-03 22:58 UTC
Re: [Puppet Users] Puppet Dashboard: db:migrate failure (1.2.2 to 1.2.4)
On Tue, Jan 3, 2012 at 14:52, Russell Van Tassell <russellvt@gmail.com> wrote:> An error I managed to hit today, trying to migrate puppet dashboard from > 1.2.2 to 1.2.4 (after upgrading the master to puppet 2.7.9 seemed to prevent > the dashboard from importing new reports). I''m trying to re-run it now, but > as you might guess, the DB is pretty huge and the process takes a while to > execute.> Mysql::Error: The total number of locks exceeds the lock table size: ALTER > TABLE resource_events ADD CONSTRAINT fk_resource_events_resource_status_id > FOREIGN KEY (resource_status_id) REFERENCES resource_statuses(id) ON DELETE > CASCADE;Wow. I don''t think I have seen a "total number of locks too big" error before. You need to tune MySQL to have a bigger lock table size for this to work. On the plus side, once you do that, run the migration thing again and it should pick up from where it left off. Daniel -- ⎋ Puppet Labs Developer – http://puppetlabs.com ♲ Made with 100 percent post-consumer electrons -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Russell Van Tassell
2012-Jan-03 23:02 UTC
Re: [Puppet Users] Puppet Dashboard: db:migrate failure (1.2.2 to 1.2.4)
On Tue, Jan 3, 2012 at 2:58 PM, Daniel Pittman <daniel@puppetlabs.com>wrote:> On Tue, Jan 3, 2012 at 14:52, Russell Van Tassell <russellvt@gmail.com> > wrote: > > > Mysql::Error: The total number of locks exceeds the lock table size: > ALTER > > TABLE resource_events ADD CONSTRAINT > fk_resource_events_resource_status_id > > FOREIGN KEY (resource_status_id) REFERENCES resource_statuses(id) ON > DELETE > > CASCADE; > > Wow. I don''t think I have seen a "total number of locks too big" > error before. You need to tune MySQL to have a bigger lock table size > for this to work. On the plus side, once you do that, run the > migration thing again and it should pick up from where it left off. > >Yeah, I''ve tried re-running it already, and now it''s breaking on something else... (BTW, it doesn''t actually pick up where it left off ... it still re-runs the deletes - though they run a little bit faster ,now) -- begin -- execute("ALTER TABLE reports ADD CONSTRAINT fk_reports_node_id FOREIGN KEY (node_id) REFERENCES nodes(id) ON DELETE CASCADE;") rake aborted! An error has occurred, all later migrations canceled: Mysql::Error: Can''t create table ''puppet_dashboard.#sql-78e_f3d'' (errno: 121): ALTER TABLE reports ADD CONSTRAINT fk_reports_node_id FOREIGN KEY (node_id) REFERENCES nodes(id) ON DELETE CASCADE; Tasks: TOP => db:migrate (See full trace by running task with --trace) -- end -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Russell Van Tassell
2012-Jan-04 00:01 UTC
Re: [Puppet Users] Puppet Dashboard: db:migrate failure (1.2.2 to 1.2.4)
For completeness sake, the subsequent error (with trace) is thus... I think at this point, I''m likely stuck -- at least short of wiping the DB and starting over, fresh (which I''d prefer to not do). -- begin -- execute("ALTER TABLE reports ADD CONSTRAINT fk_reports_node_id FOREIGN KEY (node_id) REFERENCES nodes(id) ON DELETE CASCADE;") rake aborted! An error has occurred, all later migrations canceled: Mysql::Error: Can''t create table ''puppet_dashboard.#sql-5675_5'' (errno: 121): ALTER TABLE reports ADD CONSTRAINT fk_reports_node_id FOREIGN KEY (node_id) REFERENCES nodes(id) ON DELETE CASCADE; /opt/puppet-dashboard-1.2.4/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract_adapter.rb:227:in `log'' /opt/puppet-dashboard-1.2.4/vendor/rails/activerecord/lib/active_record/connection_adapters/mysql_adapter.rb:324:in `execute'' /opt/puppet-dashboard-1.2.4/vendor/rails/activerecord/lib/active_record/migration.rb:352:in `send'' /opt/puppet-dashboard-1.2.4/vendor/rails/activerecord/lib/active_record/migration.rb:352:in `method_missing'' /opt/puppet-dashboard-1.2.4/vendor/rails/activerecord/lib/active_record/migration.rb:328:in `say_with_time'' /usr/lib/ruby/1.8/benchmark.rb:293:in `measure'' /opt/puppet-dashboard-1.2.4/vendor/rails/activerecord/lib/active_record/migration.rb:328:in `say_with_time'' /opt/puppet-dashboard-1.2.4/vendor/rails/activerecord/lib/active_record/migration.rb:348:in `method_missing'' ./db/migrate//20111118204121_add_report_foreign_key_constraints.rb:10:in `up_without_benchmarks'' ./db/migrate//20111118204121_add_report_foreign_key_constraints.rb:13:in `call'' ./db/migrate//20111118204121_add_report_foreign_key_constraints.rb:13:in `up_without_benchmarks'' /opt/puppet-dashboard-1.2.4/vendor/rails/activerecord/lib/active_record/migration.rb:282:in `send'' /opt/puppet-dashboard-1.2.4/vendor/rails/activerecord/lib/active_record/migration.rb:282:in `migrate'' /usr/lib/ruby/1.8/benchmark.rb:293:in `measure'' /opt/puppet-dashboard-1.2.4/vendor/rails/activerecord/lib/active_record/migration.rb:282:in `migrate'' /opt/puppet-dashboard-1.2.4/vendor/rails/activerecord/lib/active_record/migration.rb:365:in `__send__'' /opt/puppet-dashboard-1.2.4/vendor/rails/activerecord/lib/active_record/migration.rb:365:in `migrate'' /opt/puppet-dashboard-1.2.4/vendor/rails/activerecord/lib/active_record/migration.rb:491:in `migrate'' /opt/puppet-dashboard-1.2.4/vendor/rails/activerecord/lib/active_record/migration.rb:567:in `call'' /opt/puppet-dashboard-1.2.4/vendor/rails/activerecord/lib/active_record/migration.rb:567:in `ddl_transaction'' /opt/puppet-dashboard-1.2.4/vendor/rails/activerecord/lib/active_record/migration.rb:490:in `migrate'' /opt/puppet-dashboard-1.2.4/vendor/rails/activerecord/lib/active_record/migration.rb:477:in `each'' /opt/puppet-dashboard-1.2.4/vendor/rails/activerecord/lib/active_record/migration.rb:477:in `migrate'' /opt/puppet-dashboard-1.2.4/vendor/rails/activerecord/lib/active_record/migration.rb:401:in `up'' /opt/puppet-dashboard-1.2.4/vendor/rails/activerecord/lib/active_record/migration.rb:383:in `migrate'' /opt/puppet-dashboard-1.2.4/vendor/rails/railties/lib/tasks/databases.rake:112 /usr/lib/ruby/gems/1.8/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `call'' /usr/lib/ruby/gems/1.8/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `execute'' /usr/lib/ruby/gems/1.8/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `each'' /usr/lib/ruby/gems/1.8/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `execute'' /usr/lib/ruby/gems/1.8/gems/rake-0.9.2.2/lib/rake/task.rb:158:in `invoke_with_call_chain'' /usr/lib/ruby/1.8/monitor.rb:242:in `synchronize'' /usr/lib/ruby/gems/1.8/gems/rake-0.9.2.2/lib/rake/task.rb:151:in `invoke_with_call_chain'' /usr/lib/ruby/gems/1.8/gems/rake-0.9.2.2/lib/rake/task.rb:144:in `invoke'' /usr/lib/ruby/gems/1.8/gems/rake-0.9.2.2/lib/rake/application.rb:116:in `invoke_task'' /usr/lib/ruby/gems/1.8/gems/rake-0.9.2.2/lib/rake/application.rb:94:in `top_level'' /usr/lib/ruby/gems/1.8/gems/rake-0.9.2.2/lib/rake/application.rb:94:in `each'' /usr/lib/ruby/gems/1.8/gems/rake-0.9.2.2/lib/rake/application.rb:94:in `top_level'' /usr/lib/ruby/gems/1.8/gems/rake-0.9.2.2/lib/rake/application.rb:133:in `standard_exception_handling'' /usr/lib/ruby/gems/1.8/gems/rake-0.9.2.2/lib/rake/application.rb:88:in `top_level'' /usr/lib/ruby/gems/1.8/gems/rake-0.9.2.2/lib/rake/application.rb:66:in `run'' /usr/lib/ruby/gems/1.8/gems/rake-0.9.2.2/lib/rake/application.rb:133:in `standard_exception_handling'' /usr/lib/ruby/gems/1.8/gems/rake-0.9.2.2/lib/rake/application.rb:63:in `run'' /usr/lib/ruby/gems/1.8/gems/rake-0.9.2.2/bin/rake:33 /usr/bin/rake:19:in `load'' /usr/bin/rake:19 Tasks: TOP => db:migrate -- end -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Daniel Pittman
2012-Jan-04 00:30 UTC
Re: [Puppet Users] Puppet Dashboard: db:migrate failure (1.2.2 to 1.2.4)
On Tue, Jan 3, 2012 at 16:01, Russell Van Tassell <russellvt@gmail.com> wrote:> For completeness sake, the subsequent error (with trace) is thus... I think > at this point, I''m likely stuck -- at least short of wiping the DB and > starting over, fresh (which I''d prefer to not do).Both of those look like MySQL level errors - not specifically about this database; the first should be resolvable by rebuilding the table to reduce the number of locks required, according to the MySQL people. For the second, you might need to drop the foreign key constraints by hand, because table changes are not transactional in MySQL. :( I keep forgetting that, being used to a competent ^W fully transactional DBMS. You should be able to hand-write the appropriate delete constraint SQL, though, if you want. It isn''t that hard. Daniel -- ⎋ Puppet Labs Developer – http://puppetlabs.com ♲ Made with 100 percent post-consumer electrons -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Russell Van Tassell
2012-Jan-04 02:31 UTC
Re: [Puppet Users] Puppet Dashboard: db:migrate failure (1.2.2 to 1.2.4)
On Tue, Jan 3, 2012 at 4:30 PM, Daniel Pittman <daniel@puppetlabs.com>wrote:> On Tue, Jan 3, 2012 at 16:01, Russell Van Tassell <russellvt@gmail.com> > wrote: > > > For completeness sake, the subsequent error (with trace) is thus... I > think > > at this point, I''m likely stuck -- at least short of wiping the DB and > > starting over, fresh (which I''d prefer to not do). > > Both of those look like MySQL level errors - not specifically about > this database; the first should be resolvable by rebuilding the table > to reduce the number of locks required, according to the MySQL people. >...I just bumped the innodb_buffer_pool_size as recommended by an old bug. That seemed to make that go away... 8M is a little silly, anyway... -- begin my.cnf excerpt # Default of 8MB for innodb buffer pool is a little small - 2011/01/03 RVT # - Ref: http://bugs.mysql.com/bug.php?id=15667 innodb_buffer_pool_size = 83886080 -- end For the second, you might need to drop the foreign key constraints by> hand, because table changes are not transactional in MySQL. :( I > keep forgetting that, being used to a competent ^W fully transactional > DBMS. > > You should be able to hand-write the appropriate delete constraint > SQL, though, if you want. It isn''t that hard. >Okay... looks like I''m in an iterative process, now... this has allowed me to get past the reports table.. -- begin mysql> show create table reports; | reports | CREATE TABLE `reports` ( `id` int(11) NOT NULL AUTO_INCREMENT, `node_id` int(11) DEFAULT NULL, `host` varchar(255) DEFAULT NULL, `time` datetime DEFAULT NULL, `status` varchar(255) DEFAULT NULL, `kind` varchar(255) DEFAULT NULL, `puppet_version` varchar(255) DEFAULT NULL, `configuration_version` varchar(255) DEFAULT NULL, PRIMARY KEY (`id`), KEY `index_reports_on_node_id` (`node_id`), KEY `index_reports_on_time_and_node_id_and_status` (`time`,`node_id`,`status`), CONSTRAINT `fk_reports_node_id` FOREIGN KEY (`node_id`) REFERENCES `nodes` (`id`) ON DELETE CASCADE ) ENGINE=InnoDB AUTO_INCREMENT=594242 DEFAULT CHARSET=utf8 | 1 row in set (0.00 sec) mysql> SET FOREIGN_KEY_CHECKS=0; mysql> alter table reports drop foreign key `fk_reports_node_id`; mysql> SET FOREIGN_KEY_CHECKS=1; -- end ...and now I''m on to resource_statuses for a similar problem. Looks like it''s time to dump this in a screen session and let it run until I hit the next big boom (ie. and then try to fix it from home, rinse, repeat, etc). -- begin -- execute("ALTER TABLE reports ADD CONSTRAINT fk_reports_node_id FOREIGN KEY (node_id) REFERENCES nodes(id) ON DELETE CASCADE;") -> 27.5650s -- execute("ALTER TABLE resource_events ADD CONSTRAINT fk_resource_events_resource_status_id FOREIGN KEY (resource_status_id) REFERENCES resource_statuses(id) ON DELETE CASCADE;") -> 903.3903s -- execute("ALTER TABLE resource_statuses ADD CONSTRAINT fk_resource_statuses_report_id FOREIGN KEY (report_id) REFERENCES reports(id) ON DELETE CASCADE;") rake aborted! An error has occurred, all later migrations canceled: Mysql::Error: The table ''#sql-5675_a'' is full: ALTER TABLE resource_statuses ADD CONSTRAINT fk_resource_statuses_report_id FOREIGN KEY (report_id) REFERENCES reports(id) ON DELETE CASCADE; -- end -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Daniel Pittman
2012-Jan-04 18:47 UTC
Re: [Puppet Users] Puppet Dashboard: db:migrate failure (1.2.2 to 1.2.4)
On Tue, Jan 3, 2012 at 18:31, Russell Van Tassell <russellvt@gmail.com> wrote:> On Tue, Jan 3, 2012 at 4:30 PM, Daniel Pittman <daniel@puppetlabs.com> wrote: >> On Tue, Jan 3, 2012 at 16:01, Russell Van Tassell <russellvt@gmail.com> wrote: >> >> > For completeness sake, the subsequent error (with trace) is thus... I >> > think >> > at this point, I''m likely stuck -- at least short of wiping the DB and >> > starting over, fresh (which I''d prefer to not do). >> >> Both of those look like MySQL level errors - not specifically about >> this database; the first should be resolvable by rebuilding the table >> to reduce the number of locks required, according to the MySQL people. > > ...I just bumped the innodb_buffer_pool_size as recommended by an old bug. > That seemed to make that go away... 8M is a little silly, anyway...Great. :)> Okay... looks like I''m in an iterative process, now... this has allowed me > to get past the reports table.. > > Mysql::Error: The table ''#sql-5675_a'' is full: ALTER TABLE resource_statuses > ADD CONSTRAINT fk_resource_statuses_report_id FOREIGN KEY (report_id) > REFERENCES reports(id) ON DELETE CASCADE;That error is documented here: http://dev.mysql.com/doc/refman/5.0/en/full-table.html According to that, either your disk is full, or you have hit the OS dictated size limit for the file storing that data. (Unless you are running it on FAT32 or something, presumably the former. :) Daniel -- ⎋ Puppet Labs Developer – http://puppetlabs.com ♲ Made with 100 percent post-consumer electrons -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.