I''m running PuppetDB 1.1.1 with Puppet 3.1, both on the same server. Occasionally nodes report this error: err: Could not retrieve catalog from remote server: Error 400 on SERVER:> Could not retrieve resources from the PuppetDB at > puppet.mydomain.local:8081: SSL_connect SYSCALL returned=5 errno=0 > state=SSLv3 read finished A on node client-2.mydomain.localIt doesn''t happen on every agent run - most times it runs just fine. The names on my SSL certificates appear to be correct, aka its the same one that is used in puppet.conf. Any ideas on things I can at least check to try and track down and get rid of this error? I could understand it if the error occurred on every run, but its bugging me that its just occasional. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
On Friday, March 22, 2013 10:47:15 AM UTC, Russell Parsloe wrote:> I''m running PuppetDB 1.1.1 with Puppet 3.1, both on the same server. > Occasionally nodes report this error: > > err: Could not retrieve catalog from remote server: Error 400 on SERVER: >> Could not retrieve resources from the PuppetDB at >> puppet.mydomain.local:8081: SSL_connect SYSCALL returned=5 errno=0 >> state=SSLv3 read finished A on node client-2.mydomain.local > > > It doesn''t happen on every agent run - most times it runs just fine. > > The names on my SSL certificates appear to be correct, aka its the same > one that is used in puppet.conf. > > Any ideas on things I can at least check to try and track down and get rid > of this error? I could understand it if the error occurred on every run, > but its bugging me that its just occasional. >I''m also seeing intermittent failure reports from nodes, with the same error message from the master. It''s normally accompanied by a message in the PuppetDB logs: WARN [qtp1193921293-27694] [io.nio] javax.net.ssl.SSLHandshakeException: Invalid Padding length: 114 or WARN [qtp1193921293-39] [io.nio] javax.net.ssl.SSLHandshakeException: Invalid TLS padding data -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Russel: Can you confirm the same error message that Hugh is receiving in your own puppetdb.log? Hugh: I''d suggest raising a bug with all the details: http://projects.puppetlabs.com/projects/puppetdb/issues/new ... Russell, if the problem looks the same I''d confirm it in the same ticket so we can correlate. The more people seeing the issue that can help with debugging the problem the better. At first glance any details around command/query frequency would be useful as well, so we can understand if its load related (or not). Also - details around exact Ruby version and distro/version, version of OpenSSL linked to Ruby and exact flavour/version of Java you are using with PuppetDB might help. Anyone else getting this same error? I have not seen it before so I would be curious to hear if its prevalent. ken. On Fri, Mar 22, 2013 at 6:12 PM, Hugh Cole-Baker <hugh@fanduel.com> wrote:> On Friday, March 22, 2013 10:47:15 AM UTC, Russell Parsloe wrote: >> >> I''m running PuppetDB 1.1.1 with Puppet 3.1, both on the same server. >> Occasionally nodes report this error: >> >>> err: Could not retrieve catalog from remote server: Error 400 on SERVER: >>> Could not retrieve resources from the PuppetDB at >>> puppet.mydomain.local:8081: SSL_connect SYSCALL returned=5 errno=0 >>> state=SSLv3 read finished A on node client-2.mydomain.local >> >> >> It doesn''t happen on every agent run - most times it runs just fine. >> >> The names on my SSL certificates appear to be correct, aka its the same >> one that is used in puppet.conf. >> >> Any ideas on things I can at least check to try and track down and get rid >> of this error? I could understand it if the error occurred on every run, but >> its bugging me that its just occasional. > > > I''m also seeing intermittent failure reports from nodes, with the same error > message from the master. It''s normally accompanied by a message in the > PuppetDB logs: > WARN [qtp1193921293-27694] [io.nio] javax.net.ssl.SSLHandshakeException: > Invalid Padding length: 114 > or > WARN [qtp1193921293-39] [io.nio] javax.net.ssl.SSLHandshakeException: > Invalid TLS padding data > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
I was helping someone on IRC with a similar issue the other day...it looks like there may be a bug in very recent 1.7 OpenJDK versions that cause this to happen. Reverting to an earlier JDK version resolved the issue. As Ken mentioned, it would be most helpful if we could get the Ruby/OpenSSL/JDK versions from your masters and puppetdb servers. Thanks! On Sat, Mar 23, 2013 at 2:04 AM, Ken Barber <ken@puppetlabs.com> wrote:> Russel: Can you confirm the same error message that Hugh is receiving > in your own puppetdb.log? > > Hugh: I''d suggest raising a bug with all the details: > http://projects.puppetlabs.com/projects/puppetdb/issues/new ... > Russell, if the problem looks the same I''d confirm it in the same > ticket so we can correlate. The more people seeing the issue that can > help with debugging the problem the better. At first glance any > details around command/query frequency would be useful as well, so we > can understand if its load related (or not). Also - details around > exact Ruby version and distro/version, version of OpenSSL linked to > Ruby and exact flavour/version of Java you are using with PuppetDB > might help. > > Anyone else getting this same error? I have not seen it before so I > would be curious to hear if its prevalent. > > ken. > > On Fri, Mar 22, 2013 at 6:12 PM, Hugh Cole-Baker <hugh@fanduel.com> wrote: > > On Friday, March 22, 2013 10:47:15 AM UTC, Russell Parsloe wrote: > >> > >> I''m running PuppetDB 1.1.1 with Puppet 3.1, both on the same server. > >> Occasionally nodes report this error: > >> > >>> err: Could not retrieve catalog from remote server: Error 400 on > SERVER: > >>> Could not retrieve resources from the PuppetDB at > >>> puppet.mydomain.local:8081: SSL_connect SYSCALL returned=5 errno=0 > >>> state=SSLv3 read finished A on node client-2.mydomain.local > >> > >> > >> It doesn''t happen on every agent run - most times it runs just fine. > >> > >> The names on my SSL certificates appear to be correct, aka its the same > >> one that is used in puppet.conf. > >> > >> Any ideas on things I can at least check to try and track down and get > rid > >> of this error? I could understand it if the error occurred on every > run, but > >> its bugging me that its just occasional. > > > > > > I''m also seeing intermittent failure reports from nodes, with the same > error > > message from the master. It''s normally accompanied by a message in the > > PuppetDB logs: > > WARN [qtp1193921293-27694] [io.nio] javax.net.ssl.SSLHandshakeException: > > Invalid Padding length: 114 > > or > > WARN [qtp1193921293-39] [io.nio] javax.net.ssl.SSLHandshakeException: > > Invalid TLS padding data > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Puppet Users" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to puppet-users+unsubscribe@googlegroups.com. > > To post to this group, send email to puppet-users@googlegroups.com. > > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > > For more options, visit https://groups.google.com/groups/opt_out. > > > > > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Awesome news Deepak, I suspected "SSL something" which is why I was asking for JDK and OpenSSL details ... be good to nail down exact revisions (and distros in case of special patching they might do) so we can get an errata out somehow - so if anyone has seen this - be kind & report! :-). On Sat, Mar 23, 2013 at 8:32 PM, Deepak Giridharagopal <deepak@puppetlabs.com> wrote:> I was helping someone on IRC with a similar issue the other day...it looks > like there may be a bug in very recent 1.7 OpenJDK versions that cause this > to happen. Reverting to an earlier JDK version resolved the issue. As Ken > mentioned, it would be most helpful if we could get the Ruby/OpenSSL/JDK > versions from your masters and puppetdb servers. Thanks! > > > On Sat, Mar 23, 2013 at 2:04 AM, Ken Barber <ken@puppetlabs.com> wrote: >> >> Russel: Can you confirm the same error message that Hugh is receiving >> in your own puppetdb.log? >> >> Hugh: I''d suggest raising a bug with all the details: >> http://projects.puppetlabs.com/projects/puppetdb/issues/new ... >> Russell, if the problem looks the same I''d confirm it in the same >> ticket so we can correlate. The more people seeing the issue that can >> help with debugging the problem the better. At first glance any >> details around command/query frequency would be useful as well, so we >> can understand if its load related (or not). Also - details around >> exact Ruby version and distro/version, version of OpenSSL linked to >> Ruby and exact flavour/version of Java you are using with PuppetDB >> might help. >> >> Anyone else getting this same error? I have not seen it before so I >> would be curious to hear if its prevalent. >> >> ken. >> >> On Fri, Mar 22, 2013 at 6:12 PM, Hugh Cole-Baker <hugh@fanduel.com> wrote: >> > On Friday, March 22, 2013 10:47:15 AM UTC, Russell Parsloe wrote: >> >> >> >> I''m running PuppetDB 1.1.1 with Puppet 3.1, both on the same server. >> >> Occasionally nodes report this error: >> >> >> >>> err: Could not retrieve catalog from remote server: Error 400 on >> >>> SERVER: >> >>> Could not retrieve resources from the PuppetDB at >> >>> puppet.mydomain.local:8081: SSL_connect SYSCALL returned=5 errno=0 >> >>> state=SSLv3 read finished A on node client-2.mydomain.local >> >> >> >> >> >> It doesn''t happen on every agent run - most times it runs just fine. >> >> >> >> The names on my SSL certificates appear to be correct, aka its the same >> >> one that is used in puppet.conf. >> >> >> >> Any ideas on things I can at least check to try and track down and get >> >> rid >> >> of this error? I could understand it if the error occurred on every >> >> run, but >> >> its bugging me that its just occasional. >> > >> > >> > I''m also seeing intermittent failure reports from nodes, with the same >> > error >> > message from the master. It''s normally accompanied by a message in the >> > PuppetDB logs: >> > WARN [qtp1193921293-27694] [io.nio] >> > javax.net.ssl.SSLHandshakeException: >> > Invalid Padding length: 114 >> > or >> > WARN [qtp1193921293-39] [io.nio] javax.net.ssl.SSLHandshakeException: >> > Invalid TLS padding data >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups >> > "Puppet Users" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> > an >> > email to puppet-users+unsubscribe@googlegroups.com. >> > To post to this group, send email to puppet-users@googlegroups.com. >> > Visit this group at http://groups.google.com/group/puppet-users?hl=en. >> > For more options, visit https://groups.google.com/groups/opt_out. >> > >> > >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to puppet-users+unsubscribe@googlegroups.com. >> To post to this group, send email to puppet-users@googlegroups.com. >> Visit this group at http://groups.google.com/group/puppet-users?hl=en. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
I''ve filed a bug report http://projects.puppetlabs.com/issues/19884 with some info on the OpenJDK / Ruby / OpenSSL versions we''re using. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Thanks Hugh, can you confirm if switching to openjdk-6 fixes it? On Mon, Mar 25, 2013 at 1:35 PM, Hugh Cole-Baker <hugh@fanduel.com> wrote:> I''ve filed a bug report http://projects.puppetlabs.com/issues/19884 with > some info on the OpenJDK / Ruby / OpenSSL versions we''re using. > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Russell and Hugh - any luck downgrading to openjdk-6? An alternative thing to try - I found this in the openssl changelog: http://changelogs.ubuntu.com/changelogs/pool/main/o/openssl/openssl_1.0.1-4ubuntu5.8/changelog. Looks like the patch for CVE-2013-0169 was reverted due to a bug, but it has now been re-enabled with a fix for the regression. Can you try upgrading to 1.0.1-4ubuntu5.8 (combined with openjdk-7) to see if this helps? ken. On Mon, Mar 25, 2013 at 1:59 PM, Ken Barber <ken@puppetlabs.com> wrote:> Thanks Hugh, can you confirm if switching to openjdk-6 fixes it? > > On Mon, Mar 25, 2013 at 1:35 PM, Hugh Cole-Baker <hugh@fanduel.com> wrote: >> I''ve filed a bug report http://projects.puppetlabs.com/issues/19884 with >> some info on the OpenJDK / Ruby / OpenSSL versions we''re using. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to puppet-users+unsubscribe@googlegroups.com. >> To post to this group, send email to puppet-users@googlegroups.com. >> Visit this group at http://groups.google.com/group/puppet-users?hl=en. >> For more options, visit https://groups.google.com/groups/opt_out. >> >>-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
To pitch my two cents as information. I was searching about this error. In my case I am not running puppetDB. Here are the versions that I am using, puppet-server-3.0.2-1.el5 puppet-3.0.2-1.el5 java-1.6.0-openjdk-1.6.0.0-1.27.1.10.8.el5_8 ruby-1.8.7.352-5.el5 openssl-0.9.8e-22.el5_8.4 Error occurs when some of the client nodes try to access files that is in "/etc/puppet/modules/<modulename>/files/<filename>". It happens for random nodes, so I assume it might be related to SSL timeout or something similar in my case. Puppet (err): Could not retrieve catalog from remote server: execution expired Puppet (notice): Using cached catalog /File[/etc/security/http/key.pem] (err): Could not evaluate: SSL_connect SYSCALL returned=5 errno=0 state=SSLv2/v3 read server hello A Could not retrieve file metadata for puppet:///modules/certs/key.pem: SSL_connect SYSCALL returned=5 errno=0 state=SSLv2/v3 read server hello A -------- -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
> Puppet (err): Could not retrieve catalog from remote server: execution > expired > Puppet (notice): Using cached catalog > > /File[/etc/security/http/key.pem] (err): Could not evaluate: SSL_connect > SYSCALL returned=5 errno=0 state=SSLv2/v3 read server hello A Could not > retrieve file metadata for puppet:///modules/certs/key.pem: SSL_connect > SYSCALL returned=5 errno=0 state=SSLv2/v3 read server hello AI don''t think this is the same error is it? ken. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.