Louise Baker
2013-Oct-24 05:54 UTC
[Puppet Users] Random Internal Server Error after upgrading from 2.7.19 to 3.3.1
Hello, I have a rhel 6 puppet master with the following packages installed: facter.x86_64 1:1.7.3-1.el6 hiera.noarch 1.2.1-1.el6 puppet.noarch 3.3.1-1.el6 puppet-server.noarch 3.3.1-1.el6 ruby.x86_64 1.8.7.352-12.el6_4 I have recently upgraded the puppet master from 2.7.19 to 3.3.1, downloaded from the puppetlabs yum repo. I am now randomly seeing the following errors: 1. On the node I get: … Debug: catalog supports formats: b64_zlib_yaml dot pson raw yaml; using pson Error: Could not retrieve catalog from remote server: Error 500 on SERVER: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Internal Server Error</title> </head><body> <h1>Internal Server Error</h1> <p>The server encountered an internal error or misconfiguration and was unable to complete your request.</p> <p>Please contact the server administrator, root@localhost and inform them of the time the error occurred, and anything you might have done that may have caused the error.</p> <p>More information about this error may be available in the server error log.</p> <hr> <address>Apache/2.2.15 (Red Hat) Server at <pmaster> Port 8140</address> </body></html> /opt/csw/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:185:in `is_http_200?'' /opt/csw/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:100:in `find'' /opt/csw/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:197:in `find'' /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:243:in `retrieve_new_catalog'' /opt/csw/lib/ruby/site_ruby/1.8/puppet/util.rb:351:in `thinmark'' /opt/csw/lib/ruby/1.8/benchmark.rb:308:in `realtime'' /opt/csw/lib/ruby/site_ruby/1.8/puppet/util.rb:350:in `thinmark'' /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:242:in `retrieve_new_catalog'' /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:67:in `retrieve_catalog'' /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:107:in `prepare_and_retrieve_catalog'' /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:159:in `run'' /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:45:in `run'' /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent/locker.rb:20:in `lock'' /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:45:in `run'' /opt/csw/lib/ruby/1.8/sync.rb:230:in `synchronize'' /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:45:in `run'' /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:119:in `with_client'' /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:42:in `run'' /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:84:in `run_in_fork'' /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:41:in `run'' /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:179:in `call'' /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:179:in `controlled_run'' /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:39:in `run'' /opt/csw/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:353:in `onetime'' /opt/csw/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:327:in `run_command'' /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'' /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:456:in `plugin_hook'' /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'' /opt/csw/lib/ruby/site_ruby/1.8/puppet/util.rb:504:in `exit_on_fail'' /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'' /opt/csw/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:132:in `run'' /opt/csw/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:86:in `execute'' /opt/csw/bin/puppet:4 Warning: Not using cache on failed catalog Error: Could not retrieve catalog; skipping run 2. In /var/log/messages I see errors such as: Oct 24 14:01:57 <pmaster> puppet-master[13114]: Compiled catalog for <node> in environment production in 33.30 seconds Oct 24 14:02:11 <pmaster> puppet-master[13114]: YAML in network requests is deprecated and will be removed in a future version. See http://links.puppetlabs.com/deprecate_yaml_on_network Oct 24 14:02:11 <pmaster> puppet-master[13114]: (at /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:252:in `response_formatter_for'') Oct 24 14:02:11 <pmaster> puppet-master[13114]: YAML in network requests is deprecated and will be removed in a future version. See http://links.puppetlabs.com/deprecate_yaml_on_network Oct 24 14:02:11 <pmaster> puppet-master[13114]: (at /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:65:in `request_format'') 3. And in the apache error log I get errors such as: [Thu Oct 24 14:04:50 2013] [error] [client xx.xx.xx.xx] Premature end of script headers: <node1> [ pid=8205 thr=140243091982304 file=ext/apache2/Hooks.cpp:819 time=2013-10-24 14:04:50.263 ]: The backend application (process 13114) did not send a valid HTTP response; instead, it sent nothing at all. It is possible that it has crashed; please check whether there are crashing bugs in this application. /usr/lib/ruby/site_ruby/1.8/puppet/util/tagging.rb:42: [BUG] rb_gc_mark(): unknown data type 0x20(0x495a580) non object ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux] [Thu Oct 24 14:05:58 2013] [error] [client xx.xx.xx.xx] Premature end of script headers: <node2> [ pid=8269 thr=140243091982304 file=ext/apache2/Hooks.cpp:819 time=2013-10-24 14:05:58.762 ]: The backend application (process 11392) did not send a valid HTTP response; instead, it sent nothing at all. It is possible that it has crashed; please check whether there are crashing bugs in this application. /usr/lib/ruby/site_ruby/1.8/puppet/util/tagging.rb:42: [BUG] Segmentation fault ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux] --- The nodes are a mix of rhel5 and 6 and solaris 10 servers, mostly running 2.7.19. I have upgraded some rhel6 nodes to 3.3.1, and some solaris 10 servers to 3.2.4 (as that is the latest csw package available). The vast majority of the errors occur with the solaris servers running 2.7.19, but I have had a 3.2.4 fail consistently with the same error. Annoyingly I have had the error on a rhel server once, so I can’t say for certain the Solaris servers are the problem :/ I have searched for similar errors but cannot find anything exact. Any help would be greatly appreciated. Many thanks, Lou -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
Laurent Domb
2013-Nov-29 04:27 UTC
[Puppet Users] Re: Random Internal Server Error after upgrading from 2.7.19 to 3.3.1
I am running into the exact same issue with 3.3.1 Did you find a solution for it? On Thursday, October 24, 2013 1:54:28 AM UTC-4, Lou wrote:> > Hello, > > I have a rhel 6 puppet master with the following packages installed: > > facter.x86_64 1:1.7.3-1.el6 > hiera.noarch 1.2.1-1.el6 > puppet.noarch 3.3.1-1.el6 > puppet-server.noarch 3.3.1-1.el6 > ruby.x86_64 1.8.7.352-12.el6_4 > > I have recently upgraded the puppet master from 2.7.19 to 3.3.1, > downloaded from the puppetlabs yum repo. > > I am now randomly seeing the following errors: > > 1. On the node I get: > … > Debug: catalog supports formats: b64_zlib_yaml dot pson raw yaml; using > pson > Error: Could not retrieve catalog from remote server: Error 500 on SERVER: > <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> > <html><head> > <title>500 Internal Server Error</title> > </head><body> > <h1>Internal Server Error</h1> > <p>The server encountered an internal error or > misconfiguration and was unable to complete > your request.</p> > <p>Please contact the server administrator, > root@localhost and inform them of the time the error occurred, > and anything you might have done that may have > caused the error.</p> > <p>More information about this error may be available > in the server error log.</p> > <hr> > <address>Apache/2.2.15 (Red Hat) Server at <pmaster> Port 8140</address> > </body></html> > > /opt/csw/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:185:in > `is_http_200?'' > /opt/csw/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:100:in `find'' > /opt/csw/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:197:in > `find'' > /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:243:in > `retrieve_new_catalog'' > /opt/csw/lib/ruby/site_ruby/1.8/puppet/util.rb:351:in `thinmark'' > /opt/csw/lib/ruby/1.8/benchmark.rb:308:in `realtime'' > /opt/csw/lib/ruby/site_ruby/1.8/puppet/util.rb:350:in `thinmark'' > /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:242:in > `retrieve_new_catalog'' > /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:67:in > `retrieve_catalog'' > /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:107:in > `prepare_and_retrieve_catalog'' > /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:159:in `run'' > /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:45:in `run'' > /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent/locker.rb:20:in `lock'' > /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:45:in `run'' > /opt/csw/lib/ruby/1.8/sync.rb:230:in `synchronize'' > /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:45:in `run'' > /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:119:in `with_client'' > /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:42:in `run'' > /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:84:in `run_in_fork'' > /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:41:in `run'' > /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:179:in `call'' > /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:179:in > `controlled_run'' > /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:39:in `run'' > /opt/csw/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:353:in > `onetime'' > /opt/csw/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:327:in > `run_command'' > /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'' > /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:456:in `plugin_hook'' > /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'' > /opt/csw/lib/ruby/site_ruby/1.8/puppet/util.rb:504:in `exit_on_fail'' > /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'' > /opt/csw/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:132:in `run'' > /opt/csw/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:86:in `execute'' > /opt/csw/bin/puppet:4 > Warning: Not using cache on failed catalog > Error: Could not retrieve catalog; skipping run > > 2. In /var/log/messages I see errors such as: > > Oct 24 14:01:57 <pmaster> puppet-master[13114]: Compiled catalog for > <node> in environment production in 33.30 seconds > Oct 24 14:02:11 <pmaster> puppet-master[13114]: YAML in network requests > is deprecated and will be removed in a future version. See > http://links.puppetlabs.com/deprecate_yaml_on_network > Oct 24 14:02:11 <pmaster> puppet-master[13114]: (at > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:252:in > `response_formatter_for'') > Oct 24 14:02:11 <pmaster> puppet-master[13114]: YAML in network requests > is deprecated and will be removed in a future version. See > http://links.puppetlabs.com/deprecate_yaml_on_network > Oct 24 14:02:11 <pmaster> puppet-master[13114]: (at > /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:65:in > `request_format'') > > 3. And in the apache error log I get errors such as: > > [Thu Oct 24 14:04:50 2013] [error] [client xx.xx.xx.xx] Premature end of > script headers: <node1> > [ pid=8205 thr=140243091982304 file=ext/apache2/Hooks.cpp:819 > time=2013-10-24 14:04:50.263 ]: The backend application (process 13114) did > not send a valid HTTP response; instead, it sent nothing at all. It is > possible that it has crashed; please check whether there are crashing bugs > in this application. > /usr/lib/ruby/site_ruby/1.8/puppet/util/tagging.rb:42: [BUG] rb_gc_mark(): > unknown data type 0x20(0x495a580) non object > ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux] > > [Thu Oct 24 14:05:58 2013] [error] [client xx.xx.xx.xx] Premature end of > script headers: <node2> > [ pid=8269 thr=140243091982304 file=ext/apache2/Hooks.cpp:819 > time=2013-10-24 14:05:58.762 ]: The backend application (process 11392) did > not send a valid HTTP response; instead, it sent nothing at all. It is > possible that it has crashed; please check whether there are crashing bugs > in this application. > /usr/lib/ruby/site_ruby/1.8/puppet/util/tagging.rb:42: [BUG] Segmentation > fault > ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux] > > --- > The nodes are a mix of rhel5 and 6 and solaris 10 servers, mostly running > 2.7.19. I have upgraded some rhel6 nodes to 3.3.1, and some solaris 10 > servers to 3.2.4 (as that is the latest csw package available). > > The vast majority of the errors occur with the solaris servers running > 2.7.19, but I have had a 3.2.4 fail consistently with the same error. > Annoyingly I have had the error on a rhel server once, so I can’t say for > certain the Solaris servers are the problem :/ > > I have searched for similar errors but cannot find anything exact. Any > help would be greatly appreciated. > > Many thanks, > Lou >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/9660c713-e03f-4d60-ab99-d0be1792e7fa%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
james.eckersall@fasthosts.com
2013-Nov-29 12:39 UTC
[Puppet Users] Re: Random Internal Server Error after upgrading from 2.7.19 to 3.3.1
Hi, For reference, I''ve just upgraded my puppet masters from 2.7.22 to 3.3.2 and haven''t seen any errors of this kind. I presume you are running with passenger? I am too. CentOS EL6 masters. Maybe there is a change between 3.3.1 and 3.3.2 that will resolve this for you both. I have seen one nasty other error though. If you aren''t setting the vardir explicitly in all clients puppet.conf, I''d suggest doing so before upgrading. After my upgrade, I had to manually intervene on almost all client nodes because puppet was failing to run. Bug filed at http://projects.puppetlabs.com/issues/23311 J On Friday, 29 November 2013 04:27:20 UTC, Laurent Domb wrote:> > I am running into the exact same issue with 3.3.1 Did you find a solution > for it? > > On Thursday, October 24, 2013 1:54:28 AM UTC-4, Lou wrote: >> >> Hello, >> >> I have a rhel 6 puppet master with the following packages installed: >> >> facter.x86_64 1:1.7.3-1.el6 >> hiera.noarch 1.2.1-1.el6 >> puppet.noarch 3.3.1-1.el6 >> puppet-server.noarch 3.3.1-1.el6 >> ruby.x86_64 1.8.7.352-12.el6_4 >> >> I have recently upgraded the puppet master from 2.7.19 to 3.3.1, >> downloaded from the puppetlabs yum repo. >> >> I am now randomly seeing the following errors: >> >> 1. On the node I get: >> … >> Debug: catalog supports formats: b64_zlib_yaml dot pson raw yaml; using >> pson >> Error: Could not retrieve catalog from remote server: Error 500 on >> SERVER: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> >> <html><head> >> <title>500 Internal Server Error</title> >> </head><body> >> <h1>Internal Server Error</h1> >> <p>The server encountered an internal error or >> misconfiguration and was unable to complete >> your request.</p> >> <p>Please contact the server administrator, >> root@localhost and inform them of the time the error occurred, >> and anything you might have done that may have >> caused the error.</p> >> <p>More information about this error may be available >> in the server error log.</p> >> <hr> >> <address>Apache/2.2.15 (Red Hat) Server at <pmaster> Port 8140</address> >> </body></html> >> >> /opt/csw/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:185:in >> `is_http_200?'' >> /opt/csw/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:100:in `find'' >> /opt/csw/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:197:in >> `find'' >> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:243:in >> `retrieve_new_catalog'' >> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util.rb:351:in `thinmark'' >> /opt/csw/lib/ruby/1.8/benchmark.rb:308:in `realtime'' >> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util.rb:350:in `thinmark'' >> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:242:in >> `retrieve_new_catalog'' >> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:67:in >> `retrieve_catalog'' >> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:107:in >> `prepare_and_retrieve_catalog'' >> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:159:in `run'' >> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:45:in `run'' >> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent/locker.rb:20:in `lock'' >> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:45:in `run'' >> /opt/csw/lib/ruby/1.8/sync.rb:230:in `synchronize'' >> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:45:in `run'' >> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:119:in `with_client'' >> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:42:in `run'' >> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:84:in `run_in_fork'' >> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:41:in `run'' >> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:179:in `call'' >> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:179:in >> `controlled_run'' >> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:39:in `run'' >> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:353:in >> `onetime'' >> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:327:in >> `run_command'' >> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'' >> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:456:in `plugin_hook'' >> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'' >> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util.rb:504:in `exit_on_fail'' >> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'' >> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:132:in `run'' >> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:86:in >> `execute'' >> /opt/csw/bin/puppet:4 >> Warning: Not using cache on failed catalog >> Error: Could not retrieve catalog; skipping run >> >> 2. In /var/log/messages I see errors such as: >> >> Oct 24 14:01:57 <pmaster> puppet-master[13114]: Compiled catalog for >> <node> in environment production in 33.30 seconds >> Oct 24 14:02:11 <pmaster> puppet-master[13114]: YAML in network requests >> is deprecated and will be removed in a future version. See >> http://links.puppetlabs.com/deprecate_yaml_on_network >> Oct 24 14:02:11 <pmaster> puppet-master[13114]: (at >> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:252:in >> `response_formatter_for'') >> Oct 24 14:02:11 <pmaster> puppet-master[13114]: YAML in network requests >> is deprecated and will be removed in a future version. See >> http://links.puppetlabs.com/deprecate_yaml_on_network >> Oct 24 14:02:11 <pmaster> puppet-master[13114]: (at >> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:65:in >> `request_format'') >> >> 3. And in the apache error log I get errors such as: >> >> [Thu Oct 24 14:04:50 2013] [error] [client xx.xx.xx.xx] Premature end of >> script headers: <node1> >> [ pid=8205 thr=140243091982304 file=ext/apache2/Hooks.cpp:819 >> time=2013-10-24 14:04:50.263 ]: The backend application (process 13114) did >> not send a valid HTTP response; instead, it sent nothing at all. It is >> possible that it has crashed; please check whether there are crashing bugs >> in this application. >> /usr/lib/ruby/site_ruby/1.8/puppet/util/tagging.rb:42: [BUG] >> rb_gc_mark(): unknown data type 0x20(0x495a580) non object >> ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux] >> >> [Thu Oct 24 14:05:58 2013] [error] [client xx.xx.xx.xx] Premature end of >> script headers: <node2> >> [ pid=8269 thr=140243091982304 file=ext/apache2/Hooks.cpp:819 >> time=2013-10-24 14:05:58.762 ]: The backend application (process 11392) did >> not send a valid HTTP response; instead, it sent nothing at all. It is >> possible that it has crashed; please check whether there are crashing bugs >> in this application. >> /usr/lib/ruby/site_ruby/1.8/puppet/util/tagging.rb:42: [BUG] Segmentation >> fault >> ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux] >> >> --- >> The nodes are a mix of rhel5 and 6 and solaris 10 servers, mostly running >> 2.7.19. I have upgraded some rhel6 nodes to 3.3.1, and some solaris 10 >> servers to 3.2.4 (as that is the latest csw package available). >> >> The vast majority of the errors occur with the solaris servers running >> 2.7.19, but I have had a 3.2.4 fail consistently with the same error. >> Annoyingly I have had the error on a rhel server once, so I can’t say for >> certain the Solaris servers are the problem :/ >> >> I have searched for similar errors but cannot find anything exact. Any >> help would be greatly appreciated. >> >> Many thanks, >> Lou >> >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/1c43515d-a4a8-4594-a59d-bdd6b785a4ec%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Louise Baker
2013-Dec-10 22:29 UTC
Re: [Puppet Users] Re: Random Internal Server Error after upgrading from 2.7.19 to 3.3.1
Hi, Apologies for the delay in replying. I completely missed the posts! I am yet to find a solution. I ended up reverting to 2.7.23 until I had more time to investigate the issue. Laurent, have you had any success in the past week or so? Thanks for the heads up James. I''m not explicitly setting vardir, but will certainly do so. The puppet master is running passenger on RHEL 6. I will try 3.3.2, however if I can''t be assured the problem won''t occur again when I roll it into production, I am hesitant. Because of the randomness it is really hard to replicate, and didn''t show up in my test environment. Perhaps it relates to load on the puppet master??? Lou On Fri, Nov 29, 2013 at 10:39 PM, <james.eckersall@fasthosts.com> wrote:> Hi, > > For reference, I''ve just upgraded my puppet masters from 2.7.22 to 3.3.2 > and haven''t seen any errors of this kind. > > I presume you are running with passenger? I am too. CentOS EL6 masters. > > Maybe there is a change between 3.3.1 and 3.3.2 that will resolve this for > you both. > > I have seen one nasty other error though. > > If you aren''t setting the vardir explicitly in all clients puppet.conf, > I''d suggest doing so before upgrading. > After my upgrade, I had to manually intervene on almost all client nodes > because puppet was failing to run. Bug filed at > http://projects.puppetlabs.com/issues/23311 > > J > > On Friday, 29 November 2013 04:27:20 UTC, Laurent Domb wrote: >> >> I am running into the exact same issue with 3.3.1 Did you find a solution >> for it? >> >> On Thursday, October 24, 2013 1:54:28 AM UTC-4, Lou wrote: >>> >>> Hello, >>> >>> I have a rhel 6 puppet master with the following packages installed: >>> >>> facter.x86_64 1:1.7.3-1.el6 >>> hiera.noarch 1.2.1-1.el6 >>> puppet.noarch 3.3.1-1.el6 >>> puppet-server.noarch 3.3.1-1.el6 >>> ruby.x86_64 1.8.7.352-12.el6_4 >>> >>> I have recently upgraded the puppet master from 2.7.19 to 3.3.1, >>> downloaded from the puppetlabs yum repo. >>> >>> I am now randomly seeing the following errors: >>> >>> 1. On the node I get: >>> … >>> Debug: catalog supports formats: b64_zlib_yaml dot pson raw yaml; using >>> pson >>> Error: Could not retrieve catalog from remote server: Error 500 on >>> SERVER: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> >>> <html><head> >>> <title>500 Internal Server Error</title> >>> </head><body> >>> <h1>Internal Server Error</h1> >>> <p>The server encountered an internal error or >>> misconfiguration and was unable to complete >>> your request.</p> >>> <p>Please contact the server administrator, >>> root@localhost and inform them of the time the error occurred, >>> and anything you might have done that may have >>> caused the error.</p> >>> <p>More information about this error may be available >>> in the server error log.</p> >>> <hr> >>> <address>Apache/2.2.15 (Red Hat) Server at <pmaster> Port 8140</address> >>> </body></html> >>> >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:185:in >>> `is_http_200?'' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:100:in `find'' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:197:in >>> `find'' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:243:in >>> `retrieve_new_catalog'' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util.rb:351:in `thinmark'' >>> /opt/csw/lib/ruby/1.8/benchmark.rb:308:in `realtime'' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util.rb:350:in `thinmark'' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:242:in >>> `retrieve_new_catalog'' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:67:in >>> `retrieve_catalog'' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:107:in >>> `prepare_and_retrieve_catalog'' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:159:in `run'' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:45:in `run'' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent/locker.rb:20:in `lock'' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:45:in `run'' >>> /opt/csw/lib/ruby/1.8/sync.rb:230:in `synchronize'' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:45:in `run'' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:119:in `with_client'' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:42:in `run'' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:84:in `run_in_fork'' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:41:in `run'' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:179:in `call'' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:179:in >>> `controlled_run'' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:39:in `run'' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:353:in >>> `onetime'' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:327:in >>> `run_command'' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:456:in >>> `plugin_hook'' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util.rb:504:in `exit_on_fail'' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:132:in `run'' >>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:86:in >>> `execute'' >>> /opt/csw/bin/puppet:4 >>> Warning: Not using cache on failed catalog >>> Error: Could not retrieve catalog; skipping run >>> >>> 2. In /var/log/messages I see errors such as: >>> >>> Oct 24 14:01:57 <pmaster> puppet-master[13114]: Compiled catalog for >>> <node> in environment production in 33.30 seconds >>> Oct 24 14:02:11 <pmaster> puppet-master[13114]: YAML in network requests >>> is deprecated and will be removed in a future version. See >>> http://links.puppetlabs.com/deprecate_yaml_on_network >>> Oct 24 14:02:11 <pmaster> puppet-master[13114]: (at >>> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:252:in >>> `response_formatter_for'') >>> Oct 24 14:02:11 <pmaster> puppet-master[13114]: YAML in network requests >>> is deprecated and will be removed in a future version. See >>> http://links.puppetlabs.com/deprecate_yaml_on_network >>> Oct 24 14:02:11 <pmaster> puppet-master[13114]: (at >>> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:65:in >>> `request_format'') >>> >>> 3. And in the apache error log I get errors such as: >>> >>> [Thu Oct 24 14:04:50 2013] [error] [client xx.xx.xx.xx] Premature end of >>> script headers: <node1> >>> [ pid=8205 thr=140243091982304 file=ext/apache2/Hooks.cpp:819 >>> time=2013-10-24 14:04:50.263 ]: The backend application (process 13114) did >>> not send a valid HTTP response; instead, it sent nothing at all. It is >>> possible that it has crashed; please check whether there are crashing bugs >>> in this application. >>> /usr/lib/ruby/site_ruby/1.8/puppet/util/tagging.rb:42: [BUG] >>> rb_gc_mark(): unknown data type 0x20(0x495a580) non object >>> ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux] >>> >>> [Thu Oct 24 14:05:58 2013] [error] [client xx.xx.xx.xx] Premature end of >>> script headers: <node2> >>> [ pid=8269 thr=140243091982304 file=ext/apache2/Hooks.cpp:819 >>> time=2013-10-24 14:05:58.762 ]: The backend application (process 11392) did >>> not send a valid HTTP response; instead, it sent nothing at all. It is >>> possible that it has crashed; please check whether there are crashing bugs >>> in this application. >>> /usr/lib/ruby/site_ruby/1.8/puppet/util/tagging.rb:42: [BUG] >>> Segmentation fault >>> ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux] >>> >>> --- >>> The nodes are a mix of rhel5 and 6 and solaris 10 servers, mostly >>> running 2.7.19. I have upgraded some rhel6 nodes to 3.3.1, and some solaris >>> 10 servers to 3.2.4 (as that is the latest csw package available). >>> >>> The vast majority of the errors occur with the solaris servers running >>> 2.7.19, but I have had a 3.2.4 fail consistently with the same error. >>> Annoyingly I have had the error on a rhel server once, so I can’t say for >>> certain the Solaris servers are the problem :/ >>> >>> I have searched for similar errors but cannot find anything exact. Any >>> help would be greatly appreciated. >>> >>> Many thanks, >>> Lou >>> >> -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscribe@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/puppet-users/1c43515d-a4a8-4594-a59d-bdd6b785a4ec%40googlegroups.com > . > > For more options, visit https://groups.google.com/groups/opt_out. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAPWqdEZVxtqSH2jmg1OkJBggcYmjCU7iL59Od0XRQFgoUAbfsA%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Laurent Domb
2013-Dec-10 22:47 UTC
Re: [Puppet Users] Re: Random Internal Server Error after upgrading from 2.7.19 to 3.3.1
Hi Louise, In my case the error was triggered by a faulty manifest which contained a variable ::$fqdn instead of $::fqdn. So it might be worth looking at the manifests and check for faulty vars. Laurent On Dec 10, 2013 5:29 PM, "Louise Baker" <louise.baker01@gmail.com> wrote:> > Hi, > > Apologies for the delay in replying. I completely missed the posts! > > I am yet to find a solution. I ended up reverting to 2.7.23 until I had > more time to investigate the issue. Laurent, have you had any success in > the past week or so? > > Thanks for the heads up James. I''m not explicitly setting vardir, but > will certainly do so. The puppet master is running passenger on RHEL 6. > I will try 3.3.2, however if I can''t be assured the problem won''t occur > again when I roll it into production, I am hesitant. Because of the > randomness it is really hard to replicate, and didn''t show up in my test > environment. Perhaps it relates to load on the puppet master??? > > Lou > > On Fri, Nov 29, 2013 at 10:39 PM, <james.eckersall@fasthosts.com> wrote: > >> Hi, >> >> For reference, I''ve just upgraded my puppet masters from 2.7.22 to 3.3.2 >> and haven''t seen any errors of this kind. >> >> I presume you are running with passenger? I am too. CentOS EL6 masters. >> >> Maybe there is a change between 3.3.1 and 3.3.2 that will resolve this >> for you both. >> >> I have seen one nasty other error though. >> >> If you aren''t setting the vardir explicitly in all clients puppet.conf, >> I''d suggest doing so before upgrading. >> After my upgrade, I had to manually intervene on almost all client nodes >> because puppet was failing to run. Bug filed at >> http://projects.puppetlabs.com/issues/23311 >> >> J >> >> On Friday, 29 November 2013 04:27:20 UTC, Laurent Domb wrote: >>> >>> I am running into the exact same issue with 3.3.1 Did you find a >>> solution for it? >>> >>> On Thursday, October 24, 2013 1:54:28 AM UTC-4, Lou wrote: >>>> >>>> Hello, >>>> >>>> I have a rhel 6 puppet master with the following packages installed: >>>> >>>> facter.x86_64 1:1.7.3-1.el6 >>>> hiera.noarch 1.2.1-1.el6 >>>> puppet.noarch 3.3.1-1.el6 >>>> puppet-server.noarch 3.3.1-1.el6 >>>> ruby.x86_64 1.8.7.352-12.el6_4 >>>> >>>> I have recently upgraded the puppet master from 2.7.19 to 3.3.1, >>>> downloaded from the puppetlabs yum repo. >>>> >>>> I am now randomly seeing the following errors: >>>> >>>> 1. On the node I get: >>>> … >>>> Debug: catalog supports formats: b64_zlib_yaml dot pson raw yaml; using >>>> pson >>>> Error: Could not retrieve catalog from remote server: Error 500 on >>>> SERVER: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> >>>> <html><head> >>>> <title>500 Internal Server Error</title> >>>> </head><body> >>>> <h1>Internal Server Error</h1> >>>> <p>The server encountered an internal error or >>>> misconfiguration and was unable to complete >>>> your request.</p> >>>> <p>Please contact the server administrator, >>>> root@localhost and inform them of the time the error occurred, >>>> and anything you might have done that may have >>>> caused the error.</p> >>>> <p>More information about this error may be available >>>> in the server error log.</p> >>>> <hr> >>>> <address>Apache/2.2.15 (Red Hat) Server at <pmaster> Port 8140</address> >>>> </body></html> >>>> >>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:185:in >>>> `is_http_200?'' >>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:100:in `find'' >>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:197:in >>>> `find'' >>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:243:in >>>> `retrieve_new_catalog'' >>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util.rb:351:in `thinmark'' >>>> /opt/csw/lib/ruby/1.8/benchmark.rb:308:in `realtime'' >>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util.rb:350:in `thinmark'' >>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:242:in >>>> `retrieve_new_catalog'' >>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:67:in >>>> `retrieve_catalog'' >>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:107:in >>>> `prepare_and_retrieve_catalog'' >>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:159:in `run'' >>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:45:in `run'' >>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent/locker.rb:20:in `lock'' >>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:45:in `run'' >>>> /opt/csw/lib/ruby/1.8/sync.rb:230:in `synchronize'' >>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:45:in `run'' >>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:119:in `with_client'' >>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:42:in `run'' >>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:84:in `run_in_fork'' >>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:41:in `run'' >>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:179:in `call'' >>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:179:in >>>> `controlled_run'' >>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:39:in `run'' >>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:353:in >>>> `onetime'' >>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:327:in >>>> `run_command'' >>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'' >>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:456:in >>>> `plugin_hook'' >>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'' >>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util.rb:504:in `exit_on_fail'' >>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'' >>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:132:in >>>> `run'' >>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:86:in >>>> `execute'' >>>> /opt/csw/bin/puppet:4 >>>> Warning: Not using cache on failed catalog >>>> Error: Could not retrieve catalog; skipping run >>>> >>>> 2. In /var/log/messages I see errors such as: >>>> >>>> Oct 24 14:01:57 <pmaster> puppet-master[13114]: Compiled catalog for >>>> <node> in environment production in 33.30 seconds >>>> Oct 24 14:02:11 <pmaster> puppet-master[13114]: YAML in network >>>> requests is deprecated and will be removed in a future version. See >>>> http://links.puppetlabs.com/deprecate_yaml_on_network >>>> Oct 24 14:02:11 <pmaster> puppet-master[13114]: (at >>>> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:252:in >>>> `response_formatter_for'') >>>> Oct 24 14:02:11 <pmaster> puppet-master[13114]: YAML in network >>>> requests is deprecated and will be removed in a future version. See >>>> http://links.puppetlabs.com/deprecate_yaml_on_network >>>> Oct 24 14:02:11 <pmaster> puppet-master[13114]: (at >>>> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:65:in >>>> `request_format'') >>>> >>>> 3. And in the apache error log I get errors such as: >>>> >>>> [Thu Oct 24 14:04:50 2013] [error] [client xx.xx.xx.xx] Premature end >>>> of script headers: <node1> >>>> [ pid=8205 thr=140243091982304 file=ext/apache2/Hooks.cpp:819 >>>> time=2013-10-24 14:04:50.263 ]: The backend application (process 13114) did >>>> not send a valid HTTP response; instead, it sent nothing at all. It is >>>> possible that it has crashed; please check whether there are crashing bugs >>>> in this application. >>>> /usr/lib/ruby/site_ruby/1.8/puppet/util/tagging.rb:42: [BUG] >>>> rb_gc_mark(): unknown data type 0x20(0x495a580) non object >>>> ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux] >>>> >>>> [Thu Oct 24 14:05:58 2013] [error] [client xx.xx.xx.xx] Premature end >>>> of script headers: <node2> >>>> [ pid=8269 thr=140243091982304 file=ext/apache2/Hooks.cpp:819 >>>> time=2013-10-24 14:05:58.762 ]: The backend application (process 11392) did >>>> not send a valid HTTP response; instead, it sent nothing at all. It is >>>> possible that it has crashed; please check whether there are crashing bugs >>>> in this application. >>>> /usr/lib/ruby/site_ruby/1.8/puppet/util/tagging.rb:42: [BUG] >>>> Segmentation fault >>>> ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux] >>>> >>>> --- >>>> The nodes are a mix of rhel5 and 6 and solaris 10 servers, mostly >>>> running 2.7.19. I have upgraded some rhel6 nodes to 3.3.1, and some solaris >>>> 10 servers to 3.2.4 (as that is the latest csw package available). >>>> >>>> The vast majority of the errors occur with the solaris servers running >>>> 2.7.19, but I have had a 3.2.4 fail consistently with the same error. >>>> Annoyingly I have had the error on a rhel server once, so I can’t say for >>>> certain the Solaris servers are the problem :/ >>>> >>>> I have searched for similar errors but cannot find anything exact. Any >>>> help would be greatly appreciated. >>>> >>>> Many thanks, >>>> Lou >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to puppet-users+unsubscribe@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/puppet-users/1c43515d-a4a8-4594-a59d-bdd6b785a4ec%40googlegroups.com >> . >> >> For more options, visit https://groups.google.com/groups/opt_out. >> > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Puppet Users" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/puppet-users/yXXuVN3Bb0w/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > puppet-users+unsubscribe@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/puppet-users/CAPWqdEZVxtqSH2jmg1OkJBggcYmjCU7iL59Od0XRQFgoUAbfsA%40mail.gmail.com > . > For more options, visit https://groups.google.com/groups/opt_out. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAL4%2BNOeEy1UgpOS2jA2%3DrO8y56o%3DsEMr1n%2BAe_89ur8J_qF3jQ%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Louise Baker
2013-Dec-10 23:11 UTC
Re: [Puppet Users] Re: Random Internal Server Error after upgrading from 2.7.19 to 3.3.1
Thanks for the response ... will do. Just to clarify though, were you getting the internal server errors every time the agent ran for a particular node, or were they sometimes successful? Lou On Wed, Dec 11, 2013 at 8:47 AM, Laurent Domb <laurent.domb@gmail.com>wrote:> Hi Louise, > > In my case the error was triggered by a faulty manifest which contained a > variable ::$fqdn instead of $::fqdn. So it might be worth looking at the > manifests and check for faulty vars. > > Laurent > On Dec 10, 2013 5:29 PM, "Louise Baker" <louise.baker01@gmail.com> wrote: > >> >> Hi, >> >> Apologies for the delay in replying. I completely missed the posts! >> >> I am yet to find a solution. I ended up reverting to 2.7.23 until I had >> more time to investigate the issue. Laurent, have you had any success in >> the past week or so? >> >> Thanks for the heads up James. I''m not explicitly setting vardir, but >> will certainly do so. The puppet master is running passenger on RHEL6. I will try 3.3.2, however if I can''t be assured the problem won''t occur >> again when I roll it into production, I am hesitant. Because of the >> randomness it is really hard to replicate, and didn''t show up in my test >> environment. Perhaps it relates to load on the puppet master??? >> >> Lou >> >> On Fri, Nov 29, 2013 at 10:39 PM, <james.eckersall@fasthosts.com> wrote: >> >>> Hi, >>> >>> For reference, I''ve just upgraded my puppet masters from 2.7.22 to 3.3.2 >>> and haven''t seen any errors of this kind. >>> >>> I presume you are running with passenger? I am too. CentOS EL6 masters. >>> >>> Maybe there is a change between 3.3.1 and 3.3.2 that will resolve this >>> for you both. >>> >>> I have seen one nasty other error though. >>> >>> If you aren''t setting the vardir explicitly in all clients puppet.conf, >>> I''d suggest doing so before upgrading. >>> After my upgrade, I had to manually intervene on almost all client nodes >>> because puppet was failing to run. Bug filed at >>> http://projects.puppetlabs.com/issues/23311 >>> >>> J >>> >>> On Friday, 29 November 2013 04:27:20 UTC, Laurent Domb wrote: >>>> >>>> I am running into the exact same issue with 3.3.1 Did you find a >>>> solution for it? >>>> >>>> On Thursday, October 24, 2013 1:54:28 AM UTC-4, Lou wrote: >>>>> >>>>> Hello, >>>>> >>>>> I have a rhel 6 puppet master with the following packages installed: >>>>> >>>>> facter.x86_64 1:1.7.3-1.el6 >>>>> hiera.noarch 1.2.1-1.el6 >>>>> puppet.noarch 3.3.1-1.el6 >>>>> puppet-server.noarch 3.3.1-1.el6 >>>>> ruby.x86_64 1.8.7.352-12.el6_4 >>>>> >>>>> I have recently upgraded the puppet master from 2.7.19 to 3.3.1, >>>>> downloaded from the puppetlabs yum repo. >>>>> >>>>> I am now randomly seeing the following errors: >>>>> >>>>> 1. On the node I get: >>>>> … >>>>> Debug: catalog supports formats: b64_zlib_yaml dot pson raw yaml; >>>>> using pson >>>>> Error: Could not retrieve catalog from remote server: Error 500 on >>>>> SERVER: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> >>>>> <html><head> >>>>> <title>500 Internal Server Error</title> >>>>> </head><body> >>>>> <h1>Internal Server Error</h1> >>>>> <p>The server encountered an internal error or >>>>> misconfiguration and was unable to complete >>>>> your request.</p> >>>>> <p>Please contact the server administrator, >>>>> root@localhost and inform them of the time the error occurred, >>>>> and anything you might have done that may have >>>>> caused the error.</p> >>>>> <p>More information about this error may be available >>>>> in the server error log.</p> >>>>> <hr> >>>>> <address>Apache/2.2.15 (Red Hat) Server at <pmaster> Port >>>>> 8140</address> >>>>> </body></html> >>>>> >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:185:in >>>>> `is_http_200?'' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:100:in >>>>> `find'' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:197:in >>>>> `find'' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:243:in >>>>> `retrieve_new_catalog'' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util.rb:351:in `thinmark'' >>>>> /opt/csw/lib/ruby/1.8/benchmark.rb:308:in `realtime'' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util.rb:350:in `thinmark'' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:242:in >>>>> `retrieve_new_catalog'' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:67:in >>>>> `retrieve_catalog'' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:107:in >>>>> `prepare_and_retrieve_catalog'' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:159:in `run'' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:45:in `run'' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent/locker.rb:20:in `lock'' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:45:in `run'' >>>>> /opt/csw/lib/ruby/1.8/sync.rb:230:in `synchronize'' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:45:in `run'' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:119:in `with_client'' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:42:in `run'' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:84:in `run_in_fork'' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:41:in `run'' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:179:in `call'' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:179:in >>>>> `controlled_run'' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:39:in `run'' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:353:in >>>>> `onetime'' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:327:in >>>>> `run_command'' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:456:in >>>>> `plugin_hook'' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util.rb:504:in `exit_on_fail'' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:132:in >>>>> `run'' >>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:86:in >>>>> `execute'' >>>>> /opt/csw/bin/puppet:4 >>>>> Warning: Not using cache on failed catalog >>>>> Error: Could not retrieve catalog; skipping run >>>>> >>>>> 2. In /var/log/messages I see errors such as: >>>>> >>>>> Oct 24 14:01:57 <pmaster> puppet-master[13114]: Compiled catalog for >>>>> <node> in environment production in 33.30 seconds >>>>> Oct 24 14:02:11 <pmaster> puppet-master[13114]: YAML in network >>>>> requests is deprecated and will be removed in a future version. See >>>>> http://links.puppetlabs.com/deprecate_yaml_on_network >>>>> Oct 24 14:02:11 <pmaster> puppet-master[13114]: (at >>>>> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:252:in >>>>> `response_formatter_for'') >>>>> Oct 24 14:02:11 <pmaster> puppet-master[13114]: YAML in network >>>>> requests is deprecated and will be removed in a future version. See >>>>> http://links.puppetlabs.com/deprecate_yaml_on_network >>>>> Oct 24 14:02:11 <pmaster> puppet-master[13114]: (at >>>>> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:65:in >>>>> `request_format'') >>>>> >>>>> 3. And in the apache error log I get errors such as: >>>>> >>>>> [Thu Oct 24 14:04:50 2013] [error] [client xx.xx.xx.xx] Premature end >>>>> of script headers: <node1> >>>>> [ pid=8205 thr=140243091982304 file=ext/apache2/Hooks.cpp:819 >>>>> time=2013-10-24 14:04:50.263 ]: The backend application (process 13114) did >>>>> not send a valid HTTP response; instead, it sent nothing at all. It is >>>>> possible that it has crashed; please check whether there are crashing bugs >>>>> in this application. >>>>> /usr/lib/ruby/site_ruby/1.8/puppet/util/tagging.rb:42: [BUG] >>>>> rb_gc_mark(): unknown data type 0x20(0x495a580) non object >>>>> ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux] >>>>> >>>>> [Thu Oct 24 14:05:58 2013] [error] [client xx.xx.xx.xx] Premature end >>>>> of script headers: <node2> >>>>> [ pid=8269 thr=140243091982304 file=ext/apache2/Hooks.cpp:819 >>>>> time=2013-10-24 14:05:58.762 ]: The backend application (process 11392) did >>>>> not send a valid HTTP response; instead, it sent nothing at all. It is >>>>> possible that it has crashed; please check whether there are crashing bugs >>>>> in this application. >>>>> /usr/lib/ruby/site_ruby/1.8/puppet/util/tagging.rb:42: [BUG] >>>>> Segmentation fault >>>>> ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux] >>>>> >>>>> --- >>>>> The nodes are a mix of rhel5 and 6 and solaris 10 servers, mostly >>>>> running 2.7.19. I have upgraded some rhel6 nodes to 3.3.1, and some solaris >>>>> 10 servers to 3.2.4 (as that is the latest csw package available). >>>>> >>>>> The vast majority of the errors occur with the solaris servers running >>>>> 2.7.19, but I have had a 3.2.4 fail consistently with the same error. >>>>> Annoyingly I have had the error on a rhel server once, so I can’t say for >>>>> certain the Solaris servers are the problem :/ >>>>> >>>>> I have searched for similar errors but cannot find anything exact. >>>>> Any help would be greatly appreciated. >>>>> >>>>> Many thanks, >>>>> Lou >>>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Puppet Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to puppet-users+unsubscribe@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/puppet-users/1c43515d-a4a8-4594-a59d-bdd6b785a4ec%40googlegroups.com >>> . >>> >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Puppet Users" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/puppet-users/yXXuVN3Bb0w/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> puppet-users+unsubscribe@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/puppet-users/CAPWqdEZVxtqSH2jmg1OkJBggcYmjCU7iL59Od0XRQFgoUAbfsA%40mail.gmail.com >> . >> >> For more options, visit https://groups.google.com/groups/opt_out. >> > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscribe@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/puppet-users/CAL4%2BNOeEy1UgpOS2jA2%3DrO8y56o%3DsEMr1n%2BAe_89ur8J_qF3jQ%40mail.gmail.com > . > > For more options, visit https://groups.google.com/groups/opt_out. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAPWqdEZdBt_46NYvY5ViJAzq57PNS1JFzoEseozgqaCm_%3DG_Cg%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Laurent Domb
2013-Dec-10 23:23 UTC
Re: [Puppet Users] Re: Random Internal Server Error after upgrading from 2.7.19 to 3.3.1
No the error occurred randomly. Once I found the module which seemed to cause the issue and excluded all other module from the node declaration I could reproduce the error on every single puppetrun. On Tue, Dec 10, 2013 at 6:11 PM, Louise Baker <louise.baker01@gmail.com>wrote:> Thanks for the response ... will do. Just to clarify though, were you > getting the internal server errors every time the agent ran for a > particular node, or were they sometimes successful? > > Lou > > > On Wed, Dec 11, 2013 at 8:47 AM, Laurent Domb <laurent.domb@gmail.com>wrote: > >> Hi Louise, >> >> In my case the error was triggered by a faulty manifest which contained a >> variable ::$fqdn instead of $::fqdn. So it might be worth looking at the >> manifests and check for faulty vars. >> >> Laurent >> On Dec 10, 2013 5:29 PM, "Louise Baker" <louise.baker01@gmail.com> wrote: >> >>> >>> Hi, >>> >>> Apologies for the delay in replying. I completely missed the posts! >>> >>> I am yet to find a solution. I ended up reverting to 2.7.23 until I had >>> more time to investigate the issue. Laurent, have you had any success in >>> the past week or so? >>> >>> Thanks for the heads up James. I''m not explicitly setting vardir, but >>> will certainly do so. The puppet master is running passenger on RHEL6. I will try 3.3.2, however if I can''t be assured the problem won''t occur >>> again when I roll it into production, I am hesitant. Because of the >>> randomness it is really hard to replicate, and didn''t show up in my test >>> environment. Perhaps it relates to load on the puppet master??? >>> >>> Lou >>> >>> On Fri, Nov 29, 2013 at 10:39 PM, <james.eckersall@fasthosts.com> wrote: >>> >>>> Hi, >>>> >>>> For reference, I''ve just upgraded my puppet masters from 2.7.22 to >>>> 3.3.2 and haven''t seen any errors of this kind. >>>> >>>> I presume you are running with passenger? I am too. CentOS EL6 >>>> masters. >>>> >>>> Maybe there is a change between 3.3.1 and 3.3.2 that will resolve this >>>> for you both. >>>> >>>> I have seen one nasty other error though. >>>> >>>> If you aren''t setting the vardir explicitly in all clients puppet.conf, >>>> I''d suggest doing so before upgrading. >>>> After my upgrade, I had to manually intervene on almost all client >>>> nodes because puppet was failing to run. Bug filed at >>>> http://projects.puppetlabs.com/issues/23311 >>>> >>>> J >>>> >>>> On Friday, 29 November 2013 04:27:20 UTC, Laurent Domb wrote: >>>>> >>>>> I am running into the exact same issue with 3.3.1 Did you find a >>>>> solution for it? >>>>> >>>>> On Thursday, October 24, 2013 1:54:28 AM UTC-4, Lou wrote: >>>>>> >>>>>> Hello, >>>>>> >>>>>> I have a rhel 6 puppet master with the following packages installed: >>>>>> >>>>>> facter.x86_64 1:1.7.3-1.el6 >>>>>> hiera.noarch 1.2.1-1.el6 >>>>>> puppet.noarch 3.3.1-1.el6 >>>>>> puppet-server.noarch 3.3.1-1.el6 >>>>>> ruby.x86_64 1.8.7.352-12.el6_4 >>>>>> >>>>>> I have recently upgraded the puppet master from 2.7.19 to 3.3.1, >>>>>> downloaded from the puppetlabs yum repo. >>>>>> >>>>>> I am now randomly seeing the following errors: >>>>>> >>>>>> 1. On the node I get: >>>>>> … >>>>>> Debug: catalog supports formats: b64_zlib_yaml dot pson raw yaml; >>>>>> using pson >>>>>> Error: Could not retrieve catalog from remote server: Error 500 on >>>>>> SERVER: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> >>>>>> <html><head> >>>>>> <title>500 Internal Server Error</title> >>>>>> </head><body> >>>>>> <h1>Internal Server Error</h1> >>>>>> <p>The server encountered an internal error or >>>>>> misconfiguration and was unable to complete >>>>>> your request.</p> >>>>>> <p>Please contact the server administrator, >>>>>> root@localhost and inform them of the time the error occurred, >>>>>> and anything you might have done that may have >>>>>> caused the error.</p> >>>>>> <p>More information about this error may be available >>>>>> in the server error log.</p> >>>>>> <hr> >>>>>> <address>Apache/2.2.15 (Red Hat) Server at <pmaster> Port >>>>>> 8140</address> >>>>>> </body></html> >>>>>> >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:185:in >>>>>> `is_http_200?'' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:100:in >>>>>> `find'' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:197:in >>>>>> `find'' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:243:in >>>>>> `retrieve_new_catalog'' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util.rb:351:in `thinmark'' >>>>>> /opt/csw/lib/ruby/1.8/benchmark.rb:308:in `realtime'' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util.rb:350:in `thinmark'' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:242:in >>>>>> `retrieve_new_catalog'' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:67:in >>>>>> `retrieve_catalog'' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:107:in >>>>>> `prepare_and_retrieve_catalog'' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/configurer.rb:159:in `run'' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:45:in `run'' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent/locker.rb:20:in `lock'' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:45:in `run'' >>>>>> /opt/csw/lib/ruby/1.8/sync.rb:230:in `synchronize'' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:45:in `run'' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:119:in `with_client'' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:42:in `run'' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:84:in `run_in_fork'' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:41:in `run'' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:179:in `call'' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:179:in >>>>>> `controlled_run'' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/agent.rb:39:in `run'' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:353:in >>>>>> `onetime'' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:327:in >>>>>> `run_command'' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:456:in >>>>>> `plugin_hook'' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util.rb:504:in `exit_on_fail'' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:132:in >>>>>> `run'' >>>>>> /opt/csw/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:86:in >>>>>> `execute'' >>>>>> /opt/csw/bin/puppet:4 >>>>>> Warning: Not using cache on failed catalog >>>>>> Error: Could not retrieve catalog; skipping run >>>>>> >>>>>> 2. In /var/log/messages I see errors such as: >>>>>> >>>>>> Oct 24 14:01:57 <pmaster> puppet-master[13114]: Compiled catalog for >>>>>> <node> in environment production in 33.30 seconds >>>>>> Oct 24 14:02:11 <pmaster> puppet-master[13114]: YAML in network >>>>>> requests is deprecated and will be removed in a future version. See >>>>>> http://links.puppetlabs.com/deprecate_yaml_on_network >>>>>> Oct 24 14:02:11 <pmaster> puppet-master[13114]: (at >>>>>> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:252:in >>>>>> `response_formatter_for'') >>>>>> Oct 24 14:02:11 <pmaster> puppet-master[13114]: YAML in network >>>>>> requests is deprecated and will be removed in a future version. See >>>>>> http://links.puppetlabs.com/deprecate_yaml_on_network >>>>>> Oct 24 14:02:11 <pmaster> puppet-master[13114]: (at >>>>>> /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:65:in >>>>>> `request_format'') >>>>>> >>>>>> 3. And in the apache error log I get errors such as: >>>>>> >>>>>> [Thu Oct 24 14:04:50 2013] [error] [client xx.xx.xx.xx] Premature end >>>>>> of script headers: <node1> >>>>>> [ pid=8205 thr=140243091982304 file=ext/apache2/Hooks.cpp:819 >>>>>> time=2013-10-24 14:04:50.263 ]: The backend application (process 13114) did >>>>>> not send a valid HTTP response; instead, it sent nothing at all. It is >>>>>> possible that it has crashed; please check whether there are crashing bugs >>>>>> in this application. >>>>>> /usr/lib/ruby/site_ruby/1.8/puppet/util/tagging.rb:42: [BUG] >>>>>> rb_gc_mark(): unknown data type 0x20(0x495a580) non object >>>>>> ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux] >>>>>> >>>>>> [Thu Oct 24 14:05:58 2013] [error] [client xx.xx.xx.xx] Premature end >>>>>> of script headers: <node2> >>>>>> [ pid=8269 thr=140243091982304 file=ext/apache2/Hooks.cpp:819 >>>>>> time=2013-10-24 14:05:58.762 ]: The backend application (process 11392) did >>>>>> not send a valid HTTP response; instead, it sent nothing at all. It is >>>>>> possible that it has crashed; please check whether there are crashing bugs >>>>>> in this application. >>>>>> /usr/lib/ruby/site_ruby/1.8/puppet/util/tagging.rb:42: [BUG] >>>>>> Segmentation fault >>>>>> ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux] >>>>>> >>>>>> --- >>>>>> The nodes are a mix of rhel5 and 6 and solaris 10 servers, mostly >>>>>> running 2.7.19. I have upgraded some rhel6 nodes to 3.3.1, and some solaris >>>>>> 10 servers to 3.2.4 (as that is the latest csw package available). >>>>>> >>>>>> The vast majority of the errors occur with the solaris servers >>>>>> running 2.7.19, but I have had a 3.2.4 fail consistently with the same >>>>>> error. Annoyingly I have had the error on a rhel server once, so I can’t >>>>>> say for certain the Solaris servers are the problem :/ >>>>>> >>>>>> I have searched for similar errors but cannot find anything exact. >>>>>> Any help would be greatly appreciated. >>>>>> >>>>>> Many thanks, >>>>>> Lou >>>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Puppet Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to puppet-users+unsubscribe@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/puppet-users/1c43515d-a4a8-4594-a59d-bdd6b785a4ec%40googlegroups.com >>>> . >>>> >>>> For more options, visit https://groups.google.com/groups/opt_out. >>>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "Puppet Users" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/puppet-users/yXXuVN3Bb0w/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> puppet-users+unsubscribe@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/puppet-users/CAPWqdEZVxtqSH2jmg1OkJBggcYmjCU7iL59Od0XRQFgoUAbfsA%40mail.gmail.com >>> . >>> >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to puppet-users+unsubscribe@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/puppet-users/CAL4%2BNOeEy1UgpOS2jA2%3DrO8y56o%3DsEMr1n%2BAe_89ur8J_qF3jQ%40mail.gmail.com >> . >> >> For more options, visit https://groups.google.com/groups/opt_out. >> > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Puppet Users" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/puppet-users/yXXuVN3Bb0w/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > puppet-users+unsubscribe@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/puppet-users/CAPWqdEZdBt_46NYvY5ViJAzq57PNS1JFzoEseozgqaCm_%3DG_Cg%40mail.gmail.com > . > > For more options, visit https://groups.google.com/groups/opt_out. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAL4%2BNOcP4mz4LNy6FWkkv7tOJjuPPa0HvohFN91idngRF2ZMUQ%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Felix Frank
2013-Dec-10 23:25 UTC
Re: [Puppet Users] Re: Random Internal Server Error after upgrading from 2.7.19 to 3.3.1
On 11/29/2013 01:39 PM, james.eckersall@fasthosts.com wrote:> > If you aren''t setting the vardir explicitly in all clients puppet.conf, > I''d suggest doing so before upgrading. > After my upgrade, I had to manually intervene on almost all client nodes > because puppet was failing to run. Bug filed > at http://projects.puppetlabs.com/issues/23311I had some pokes at it. Unfortunately, it got duplicated in issue 23349. Fortunately, that newer one has already been solved, so 3.4 should be good. Thanks, Felix -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/52A7A2F5.7050106%40Alumni.TU-Berlin.de. For more options, visit https://groups.google.com/groups/opt_out.
Louise Baker
2013-Dec-10 23:36 UTC
Re: [Puppet Users] Re: Random Internal Server Error after upgrading from 2.7.19 to 3.3.1
Thank you both for your responses :) On Wed, Dec 11, 2013 at 9:25 AM, Felix Frank < Felix.Frank@alumni.tu-berlin.de> wrote:> On 11/29/2013 01:39 PM, james.eckersall@fasthosts.com wrote: > > > > If you aren''t setting the vardir explicitly in all clients puppet.conf, > > I''d suggest doing so before upgrading. > > After my upgrade, I had to manually intervene on almost all client nodes > > because puppet was failing to run. Bug filed > > at http://projects.puppetlabs.com/issues/23311 > > I had some pokes at it. Unfortunately, it got duplicated in issue 23349. > Fortunately, that newer one has already been solved, so 3.4 should be good. > > Thanks, > Felix > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscribe@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/puppet-users/52A7A2F5.7050106%40Alumni.TU-Berlin.de > . > For more options, visit https://groups.google.com/groups/opt_out. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAPWqdEbVeEW%3D3DJVLreV4wLS0uoK1wae2rT8uJu7BSj-0SnY9w%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
James Eckersall
2013-Dec-10 23:37 UTC
RE: [Puppet Users] Re: Random Internal Server Error after upgrading from 2.7.19 to 3.3.1
And thanks from me. Felix, thank you for your investigation on this issue, it''s really appreciated :) From: puppet-users@googlegroups.com [mailto:puppet-users@googlegroups.com] On Behalf Of Louise Baker Sent: 10 December 2013 23:36 To: puppet-users@googlegroups.com Subject: Re: [Puppet Users] Re: Random Internal Server Error after upgrading from 2.7.19 to 3.3.1 Thank you both for your responses :) On Wed, Dec 11, 2013 at 9:25 AM, Felix Frank <Felix.Frank@alumni.tu-berlin.de<mailto:Felix.Frank@alumni.tu-berlin.de>> wrote: On 11/29/2013 01:39 PM, james.eckersall@fasthosts.com<mailto:james.eckersall@fasthosts.com> wrote:> > If you aren''t setting the vardir explicitly in all clients puppet.conf, > I''d suggest doing so before upgrading. > After my upgrade, I had to manually intervene on almost all client nodes > because puppet was failing to run. Bug filed > at http://projects.puppetlabs.com/issues/23311I had some pokes at it. Unfortunately, it got duplicated in issue 23349. Fortunately, that newer one has already been solved, so 3.4 should be good. Thanks, Felix -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com<mailto:puppet-users%2Bunsubscribe@googlegroups.com>. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/52A7A2F5.7050106%40Alumni.TU-Berlin.de. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/yXXuVN3Bb0w/unsubscribe. To unsubscribe from this group and all its topics, send an email to puppet-users+unsubscribe@googlegroups.com<mailto:puppet-users+unsubscribe@googlegroups.com>. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAPWqdEbVeEW%3D3DJVLreV4wLS0uoK1wae2rT8uJu7BSj-0SnY9w%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/62250036DF392146A154CA8F5F2CF3FC0CFCD2A0AC%40fh-exch07-01. For more options, visit https://groups.google.com/groups/opt_out.