Héctor Rivas Gándara
2010-Aug-30 11:04 UTC
[Puppet Users] How report errors and random errors in clients
Hello, I am using puppet 2.6.1rc3 in a test environment with AIX, Suse, Debians. Right now there are 5 clients... I am running puppet from cron each 30m (using random minute per host). I have prepared a configuration ready to deploy puppet in all our infrastructure. If I deploy and use it everywhere, need to known each error. That is my first question How is the best way to report failures in puppet configuration? I was using the email aproach, but I am having lots of random errors in clients that make the puppet runs fail and send a report error. For 5 hosts I am receiving around 150 emails/day. I do not known if this is normal. The errors appear once or two times, randomly, and if I execute the client again it works properly. They are usually these kind of errors: Thu Aug 26 18:05:10 +0200 2010 //puppetclient.myhost.com//Stage[main]/Cgx_unixserver::Ad_integration::Debian/Cgx_unixserver::Ad_integration::Debian::Pam_file[common-password]/File[/etc/pam.d/common-password] (err): Could not evaluate: SSL_connect SYSCALL returned=5 errno=0 state=SSLv2/v3 read server hello A Could not retrieve file metadata for puppet:///modules/cgx_unixserver/Linux.ad_integration/debian.pam.d/common-password: SSL_connect SYSCALL returned=5 errno=0 state=SSLv2/v3 read server hello A at /cgx1/puppet/data/development/services/cgx_unixserver/manifests/ad_integration/debian.pp:24 Thu Aug 26 17:49:12 +0200 2010 //puppetclient.myhost.com//Stage[main]/Monit::Base/File[/srv/monit/monit/monit.d] (err): Failed to generate additional resources using ''eval_generate'': end of file reached Mon Aug 30 10:36:45 +0200 2010 //puppetclient.myhost.com/Puppet (err): Could not retrieve catalog from remote server: execution expired Mon Aug 30 10:36:45 +0200 2010 //puppetclient.myhost.com/Puppet (err): Cached catalog for puppetclient.myhost.com failed: Could not parse YAML data for catalog puppetclient.myhost.com: allocator undefined for Proc Mon Aug 30 10:36:45 +0200 2010 //puppetclient.myhost.com/Puppet (err): Could not retrieve catalog; skipping run Mon Aug 30 09:57:59 +0200 2010 //puppetclient.myhost.com/Puppet (err): Could not retrieve catalog from remote server: Error 400 on SERVER: SQLite3::BusyException: database is locked: DELETE FROM "fact_values" WHERE ("id" IN (205405,205406,205407,205408,205409,205410,205411,205412,205413,205414,205415,205416,205417,205418,205419,205420,205421,205422,205423,205424,205425,205426,205427,205428,205429,205430,205431,205432,205433,205434,205435,205436,205437,205438,205439,205440,205441)) Mon Aug 30 09:57:59 +0200 2010 //puppetclient.myhost.com/Puppet (err): Cached catalog for puppetclient.myhost.com failed: Could not parse YAML data for catalog puppetclient.myhost.com: allocator undefined for Proc Mon Aug 30 09:57:59 +0200 2010 //puppetclient.myhost.com/Puppet (err): Could not retrieve catalog; skipping run Mon Aug 30 12:57:18 +0200 2010 //puppetclient.myhost.com/Puppet (err): Could not save yaml puppetclient.myhost.com: class or module required I also get sometimes some errors from cron output like these: Could not run: method `directory?'' called on terminated object (0x2005051c) /usr/local/lib/ruby/site_ruby/1.8/puppet/util/zaml.rb:243: [BUG] Segmentation fault ruby 1.8.7 (2009-06-12 patchlevel 174) [rs6000-aix] /srv/scripts/puppet/puppet.ctl.sh: line 117: 319712 IOT/Abort trap (core dumped) $PUPPETD ${PUPPET_OPTS} ${PUPPET_EXTRA_OPTS} --onetime --no-daemonize --verbose Is this normal? How is the best way to monitor all the puppet network? -- Atentamente Héctor Rivas -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Daniel Pittman
2010-Aug-30 13:41 UTC
Re: [Puppet Users] How report errors and random errors in clients
Héctor Rivas Gándara <keymon@gmail.com> writes:> I am using puppet 2.6.1rc3 in a test environment with AIX, Suse, > Debians. Right now there are 5 clients... I am running puppet from cron > each 30m (using random minute per host). I have prepared a configuration > ready to deploy puppet in all our infrastructure. > > If I deploy and use it everywhere, need to known each error. That is my > first question How is the best way to report failures in puppet > configuration?We monitor the logs the same way we monitor for any other failures in our system. I would generally suggest you take the same approach, since it means you don''t have two distinct processes for detecting problems. (In my case, that involves both ''logcheck'' as per the Debian package, and Splunk, together with a network-wide reliable log collection facility.)> I was using the email aproach, but I am having lots of random errors > in clients that make the puppet runs fail and send a report error. For > 5 hosts I am receiving around 150 emails/day. I do not known if this > is normal.No. You should receive zero error messages per day.> The errors appear once or two times, randomly, and if I execute the > client again it works properly. They are usually these kind of errors:I bet that you are still using the default webrick puppetmaster configuration, and I see you are using sqlite with storedconfigs. Neither of those is really very good at scale, and "scale" doesn''t mean very many nodes I am afraid. You will probably find life improved by deploying another mode of operation for the puppetmaster daemon, and using a real database (I use PostgreSQL) for storedconfigs. [...]> Mon Aug 30 12:57:18 +0200 2010 //puppetclient.myhost.com/Puppet (err): > Could not save yaml puppetclient.myhost.com: class or module requiredThat looks troublingly like a local Ruby problem of some sort.> I also get sometimes some errors from cron output like these: > > Could not run: method `directory?'' called on terminated object (0x2005051c) > /usr/local/lib/ruby/site_ruby/1.8/puppet/util/zaml.rb:243: [BUG] > Segmentation fault ruby 1.8.7 (2009-06-12 patchlevel 174) [rs6000-aix] > /srv/scripts/puppet/puppet.ctl.sh: line 117: 319712 IOT/Abort trap > (core dumped) $PUPPETD ${PUPPET_OPTS} ${PUPPET_EXTRA_OPTS} > --onetime --no-daemonize --verbose > > Is this normal?Core dumps in Ruby are absolutely not normal, and are a sign you should look to upstream Ruby support for fixing the problem with your interpreter. It is hard to say how many of the other errors (other than sqlite locking) might be caused by underlying Ruby problems, so that sounds like a really good place to start on resolving this trouble. Regards, Daniel -- ✣ Daniel Pittman ✉ daniel@rimspace.net ☎ +61 401 155 707 ♽ made with 100 percent post-consumer electrons -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Héctor Rivas Gándara
2010-Aug-30 18:05 UTC
Re: [Puppet Users] How report errors and random errors in clients
> We monitor the logs the same way we monitor for any other failures in our > system. I would generally suggest you take the same approach, since it means > you don''t have two distinct processes for detecting problems. > (In my case, that involves both ''logcheck'' as per the Debian package, and > Splunk, together with a network-wide reliable log collection facility.)Yes, probably that is the best aproach. I was thinking on use email+puppet-dashboard, but email is a bad idea and puppet-dashboard is too slow (at least my experience)>> The errors appear once or two times, randomly, and if I execute the >> client again it works properly. They are usually these kind of errors: > I bet that you are still using the default webrick puppetmaster configuration, > and I see you are using sqlite with storedconfigs. > Neither of those is really very good at scale, and "scale" doesn''t mean very > many nodes I am afraid. You will probably find life improved by deploying > another mode of operation for the puppetmaster daemon, and using a real > database (I use PostgreSQL) for storedconfigs.You are right, I am using both of them... but I thougth that It could scale to 20 clients... I will setup mysql+mongrel. I will tell you if this solves the problem.>> Segmentation fault ruby 1.8.7 (2009-06-12 patchlevel 174) [rs6000-aix] >> /srv/scripts/puppet/puppet.ctl.sh: line 117: 319712 IOT/Abort trap >> (core dumped) $PUPPETD ${PUPPET_OPTS} ${PUPPET_EXTRA_OPTS} >> --onetime --no-daemonize --verbose >> >> Is this normal? > > Core dumps in Ruby are absolutely not normal, and are a sign you should look > to upstream Ruby support for fixing the problem with your interpreter.Well, it was on a AIX system and coredumps does not appear too often. I will try to send them the core. -- Regards Hector Rivas -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Héctor Rivas Gándara
2010-Sep-01 12:28 UTC
Re: [Puppet Users] How report errors and random errors in clients
2010/8/30 Héctor Rivas Gándara <keymon@gmail.com>:>>> The errors appear once or two times, randomly, and if I execute the >>> client again it works properly. They are usually these kind of errors: >> You will probably find life improved by deploying >> another mode of operation for the puppetmaster daemon, and using a real >> database (I use PostgreSQL) for storedconfigs. > > You are right, I am using both of them... but I thougth that It could > scale to 20 clients... I will setup mysql+mongrel. > I will tell you if this solves the problem.I just configured mysql for storeconfigs and Apache+mongrel. I still have to test it for a pair of days, but here are my first conclusions: Just using mysql solved most of the problems, but I still had some "random" errors with ruby syntax. Setting mongrel also reduced, and I think that the executions are now much more faster. But there it still get some random "syntax errors" with ruby on AIX nodes. specially if I run it with --ignorecache --no-usecacheonfailure. But now these errors are far not so often. Some of them are: //puppetclient.myhost.com//Stage[main]/Cgx_unixserver::Srv_tree/File[/srv] (err): Failed to generate additional resources using ''eval_generate'': Invalid parameter 0(0) at /cgx1/puppet/data/test/modules/stow/manifests/package.pp:61 //puppetclient.myhost.com/Puppet (err): Could not retrieve catalog from remote server: Could not intern from pson: private method `gsub'' called for 0:Fixnum Wed Sep 01 13:59:56 +0200 2010 //puppetclient.myhost.com/Puppet (err): Could not retrieve catalog; skipping run //puppetclient.myhost.com//Stage[main]/Cgx_unixserver::Profile_local/File[/srv/conf/profile.d] (err): Failed to generate additional resources using ''eval_generate'': can''t convert Fixnum into String //puppetclient.myhost.com//Stage[main]/Cgx_unixserver::Profile_local/Exec[hook_etc_profile] (err): Could not evaluate: Could not create resource status: undefined method `path'' for false:FalseClass at /cgx1/puppet/data/test/services/cgx_unixserver/manifests/profile_local.pp:10 I still have random "Segmentation Faults" with ruby on AIX. I will ignore them for the moment until I will compile last ruby source (may be 1.9.x) and test it. Maybe all this problems are related to ruby on AIX. Anybody running AIX has same problems? Daniel: Thank you for your reply and suggestion. -- Atentamente Héctor Rivas -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Patrick
2010-Sep-01 14:55 UTC
Re: [Puppet Users] How report errors and random errors in clients
On Sep 1, 2010, at 5:28 AM, Héctor Rivas Gándara wrote:> I still have random "Segmentation Faults" with ruby on AIX. I will > ignore them for the moment until I will compile last ruby source (may > be 1.9.x) and test it.I remember reading somewhere that puppet isn''t compatible with 1.9.x versions of ruby. I don''t know if this changed with puppet 2.6.x. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Peter Meier
2010-Sep-01 15:08 UTC
Re: [Puppet Users] How report errors and random errors in clients
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1> On Sep 1, 2010, at 5:28 AM, Héctor Rivas Gándara wrote: >> I still have random "Segmentation Faults" with ruby on AIX. I will >> ignore them for the moment until I will compile last ruby source >> (may be 1.9.x) and test it. > > > I remember reading somewhere that puppet isn''t compatible with 1.9.x > versions of ruby. I don''t know if this changed with puppet 2.6.x.this hasn''t changed. pete -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkx+bFMACgkQbwltcAfKi3+4ywCffIiqzeZMyyErSbVCRSm8IAyt 6DoAnR4jhlvYFAyAMWRJqlIFiQmJRA63 =Hwtj -----END PGP SIGNATURE----- -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Héctor Rivas Gándara
2010-Sep-01 15:53 UTC
Re: [Puppet Users] How report errors and random errors in clients
On Wed, Sep 1, 2010 at 4:55 PM, Patrick <kc7zzv@gmail.com> wrote:> > On Sep 1, 2010, at 5:28 AM, Héctor Rivas Gándara wrote: >> I still have random "Segmentation Faults" with ruby on AIX. I will >> ignore them for the moment until I will compile last ruby source (may >> be 1.9.x) and test it. > I remember reading somewhere that puppet isn''t compatible with 1.9.x versions of ruby. I don''t know if this changed with puppet 2.6.x.Yes, 0.25 was not compatible, but I thought that 2.6.x were compatible... Then I will keep 1.8.7 version. -- Atentamente Héctor Rivas -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.