Several times a day I receive the below error from Puppet runs on my nodes: err Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to submit ''replace catalog'' command for i-7b51c334.example.com to PuppetDB at puppet.internal.example.com:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A err Could not retrieve catalog; skipping run notice Using cached catalog Either ''replace catalog'' or ''replace facts'' commands fail. Every time this happens, the /var/log/puppetdb/puppetdb.log shows one of two types of errors: ''Invalid Padding length'' or ''Invalid TLS padding data''. Here are some recent examples: 2013-04-30 04:15:56,874 WARN [qtp1818558055-40] [io.nio] javax.net.ssl.SSLHandshakeException: Invalid Padding length: 94 2013-04-30 13:15:30,580 WARN [qtp1818558055-40] [io.nio] javax.net.ssl.SSLHandshakeException: Invalid Padding length: 178 2013-04-30 13:39:31,403 WARN [qtp1818558055-38] [io.nio] javax.net.ssl.SSLHandshakeException: Invalid TLS padding data 2013-04-30 17:16:17,170 WARN [qtp1818558055-37] [io.nio] javax.net.ssl.SSLHandshakeException: Invalid TLS padding data 2013-04-30 20:17:59,374 WARN [qtp1818558055-40] [io.nio] javax.net.ssl.SSLHandshakeException: Invalid Padding length: 187 2013-04-30 20:50:59,536 WARN [qtp1818558055-37] [io.nio] javax.net.ssl.SSLHandshakeException: Invalid Padding length: 179 2013-04-30 22:25:56,914 WARN [qtp1818558055-38] [io.nio] javax.net.ssl.SSLHandshakeException: Invalid Padding length: 242 2013-05-01 05:02:54,751 WARN [qtp1818558055-40] [io.nio] javax.net.ssl.SSLHandshakeException: Invalid Padding length: 214 It seems to happen with all nodes, regardless of time of day, whether they''re inside or outside our WAN firewall, etc. I''ve found no pattern in their timing. It''s always just one run that fails, while other runs from other nodes that occur a minute earlier or later report no problems. PuppetDB runs on the same host as the Puppet master. Connectivity seems to be fine. Ubuntu packages are up-to-date: ii puppetdb 1.2.0-1puppetlabs1 PuppetDB Centralized Storage. ii puppetdb-terminus 1.2.0-1puppetlabs1 Connect Puppet to PuppetDB by setting up a terminus for PuppetDB. ii puppetmaster-common 3.1.1-1puppetlabs1 Puppet master common scripts ii puppetmaster-passenger 3.1.1-1puppetlabs1 Centralised configuration management - master setup to run under mod passenger Any idea what could be causing these errors? Regards, Martijn -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Ken Barber
2013-May-02 12:52 UTC
Re: [Puppet Users] Occasional SSL_connect errors to PuppetDB
Are you running version 7 of the JDK? We believe this was a bug introduced during a CVE fix into a minor revision of Java 7: http://projects.puppetlabs.com/issues/19884 If you downgrade to JDK 6 the issue should disappear. ken. On Thu, May 2, 2013 at 11:32 AM, Martijn <martijn@heemels.com> wrote:> Several times a day I receive the below error from Puppet runs on my nodes: > > err Could not retrieve catalog from remote server: Error 400 on SERVER: > Failed to submit ''replace catalog'' command for i-7b51c334.example.com to > PuppetDB at puppet.internal.example.com:8081: SSL_connect SYSCALL returned=5 > errno=0 state=SSLv3 read finished A > err Could not retrieve catalog; skipping run > notice Using cached catalog > > Either ''replace catalog'' or ''replace facts'' commands fail. > > Every time this happens, the /var/log/puppetdb/puppetdb.log shows one of two > types of errors: ''Invalid Padding length'' or ''Invalid TLS padding data''. > Here are some recent examples: > > 2013-04-30 04:15:56,874 WARN [qtp1818558055-40] [io.nio] > javax.net.ssl.SSLHandshakeException: Invalid Padding length: 94 > 2013-04-30 13:15:30,580 WARN [qtp1818558055-40] [io.nio] > javax.net.ssl.SSLHandshakeException: Invalid Padding length: 178 > 2013-04-30 13:39:31,403 WARN [qtp1818558055-38] [io.nio] > javax.net.ssl.SSLHandshakeException: Invalid TLS padding data > 2013-04-30 17:16:17,170 WARN [qtp1818558055-37] [io.nio] > javax.net.ssl.SSLHandshakeException: Invalid TLS padding data > 2013-04-30 20:17:59,374 WARN [qtp1818558055-40] [io.nio] > javax.net.ssl.SSLHandshakeException: Invalid Padding length: 187 > 2013-04-30 20:50:59,536 WARN [qtp1818558055-37] [io.nio] > javax.net.ssl.SSLHandshakeException: Invalid Padding length: 179 > 2013-04-30 22:25:56,914 WARN [qtp1818558055-38] [io.nio] > javax.net.ssl.SSLHandshakeException: Invalid Padding length: 242 > 2013-05-01 05:02:54,751 WARN [qtp1818558055-40] [io.nio] > javax.net.ssl.SSLHandshakeException: Invalid Padding length: 214 > > It seems to happen with all nodes, regardless of time of day, whether > they''re inside or outside our WAN firewall, etc. I''ve found no pattern in > their timing. It''s always just one run that fails, while other runs from > other nodes that occur a minute earlier or later report no problems. > > PuppetDB runs on the same host as the Puppet master. Connectivity seems to > be fine. Ubuntu packages are up-to-date: > > ii puppetdb 1.2.0-1puppetlabs1 > PuppetDB Centralized Storage. > ii puppetdb-terminus 1.2.0-1puppetlabs1 > Connect Puppet to PuppetDB by setting up a terminus for PuppetDB. > ii puppetmaster-common 3.1.1-1puppetlabs1 > Puppet master common scripts > ii puppetmaster-passenger 3.1.1-1puppetlabs1 > Centralised configuration management - master setup to run under mod > passenger > > Any idea what could be causing these errors? > > Regards, Martijn > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Martijn
2013-May-03 12:46 UTC
Re: [Puppet Users] Occasional SSL_connect errors to PuppetDB
Hi Ken, I was indeed using openjdk-7. I''ve now downgraded to 6 and will see if this helps. Thanks for the advice, Martijn Op donderdag 2 mei 2013 14:52:27 UTC+2 schreef Ken Barber het volgende:> > Are you running version 7 of the JDK? We believe this was a bug > introduced during a CVE fix into a minor revision of Java 7: > > http://projects.puppetlabs.com/issues/19884 > > If you downgrade to JDK 6 the issue should disappear. > > ken. > > On Thu, May 2, 2013 at 11:32 AM, Martijn <mar...@heemels.com <javascript:>> > wrote: > > Several times a day I receive the below error from Puppet runs on my > nodes: > > > > err Could not retrieve catalog from remote server: Error 400 on > SERVER: > > Failed to submit ''replace catalog'' command for i-7b51c334.example.comto > > PuppetDB at puppet.internal.example.com:8081: SSL_connect SYSCALL > returned=5 > > errno=0 state=SSLv3 read finished A > > err Could not retrieve catalog; skipping run > > notice Using cached catalog > > > > Either ''replace catalog'' or ''replace facts'' commands fail. > > > > Every time this happens, the /var/log/puppetdb/puppetdb.log shows one of > two > > types of errors: ''Invalid Padding length'' or ''Invalid TLS padding data''. > > Here are some recent examples: > > > > 2013-04-30 04:15:56,874 WARN [qtp1818558055-40] [io.nio] > > javax.net.ssl.SSLHandshakeException: Invalid Padding length: 94 > > 2013-04-30 13:15:30,580 WARN [qtp1818558055-40] [io.nio] > > javax.net.ssl.SSLHandshakeException: Invalid Padding length: 178 > > 2013-04-30 13:39:31,403 WARN [qtp1818558055-38] [io.nio] > > javax.net.ssl.SSLHandshakeException: Invalid TLS padding data > > 2013-04-30 17:16:17,170 WARN [qtp1818558055-37] [io.nio] > > javax.net.ssl.SSLHandshakeException: Invalid TLS padding data > > 2013-04-30 20:17:59,374 WARN [qtp1818558055-40] [io.nio] > > javax.net.ssl.SSLHandshakeException: Invalid Padding length: 187 > > 2013-04-30 20:50:59,536 WARN [qtp1818558055-37] [io.nio] > > javax.net.ssl.SSLHandshakeException: Invalid Padding length: 179 > > 2013-04-30 22:25:56,914 WARN [qtp1818558055-38] [io.nio] > > javax.net.ssl.SSLHandshakeException: Invalid Padding length: 242 > > 2013-05-01 05:02:54,751 WARN [qtp1818558055-40] [io.nio] > > javax.net.ssl.SSLHandshakeException: Invalid Padding length: 214 > > > > It seems to happen with all nodes, regardless of time of day, whether > > they''re inside or outside our WAN firewall, etc. I''ve found no pattern > in > > their timing. It''s always just one run that fails, while other runs from > > other nodes that occur a minute earlier or later report no problems. > > > > PuppetDB runs on the same host as the Puppet master. Connectivity seems > to > > be fine. Ubuntu packages are up-to-date: > > > > ii puppetdb 1.2.0-1puppetlabs1 > > PuppetDB Centralized Storage. > > ii puppetdb-terminus 1.2.0-1puppetlabs1 > > Connect Puppet to PuppetDB by setting up a terminus for PuppetDB. > > ii puppetmaster-common 3.1.1-1puppetlabs1 > > Puppet master common scripts > > ii puppetmaster-passenger 3.1.1-1puppetlabs1 > > Centralised configuration management - master setup to run under mod > > passenger > > > > Any idea what could be causing these errors? > > > > Regards, Martijn > > > > -- > > You received this message because you are subscribed to the Google > Groups > > "Puppet Users" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to puppet-users...@googlegroups.com <javascript:>. > > To post to this group, send email to puppet...@googlegroups.com<javascript:>. > > > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > > For more options, visit https://groups.google.com/groups/opt_out. > > > > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Martijn
2013-May-07 19:22 UTC
Re: [Puppet Users] Occasional SSL_connect errors to PuppetDB
After 4 days without errors, I can safely say that using the latest Java 6 works fine. Thanks again Ken. Regards, Martijn Op vrijdag 3 mei 2013 14:46:25 UTC+2 schreef Martijn het volgende:> > Hi Ken, > > I was indeed using openjdk-7. I''ve now downgraded to 6 and will see if > this helps. > > Thanks for the advice, > Martijn > > Op donderdag 2 mei 2013 14:52:27 UTC+2 schreef Ken Barber het volgende: >> >> Are you running version 7 of the JDK? We believe this was a bug >> introduced during a CVE fix into a minor revision of Java 7: >> >> http://projects.puppetlabs.com/issues/19884 >> >> If you downgrade to JDK 6 the issue should disappear. >> >> ken. >> >>-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.