Justin Lambert
2013-May-14 22:55 UTC
[Puppet Users] PuppetDB "Failed to submit ''replace facts'' command"
I have a small environment with a single puppetmaster and puppetdb on the same node that I get sporadic "Failed to submit ''replace facts'' command" errors from nodes. It isn''t consistent about which nodes are causing the error and I can''t find anything else going on at the same time on either the master or the agent that these errors happen. Here are the logs from a previous event The puppet client log: Retrieving plugin Loading facts in /var/lib/puppet/lib/facter/pe_version.rb Loading facts in /var/lib/puppet/lib/facter/root_home.rb Loading facts in /var/lib/puppet/lib/facter/classes.rb Loading facts in /var/lib/puppet/lib/facter/puppet_enabled.rb Loading facts in /var/lib/puppet/lib/facter/puppetdb_keystore_pw.rb Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb Loading facts in /var/lib/puppet/lib/facter/disposition.rb Loading facts in /var/lib/puppet/lib/facter/ec2_security_groups_cached.rb Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb Loading facts in /var/lib/puppet/lib/facter/iptables_version.rb Loading facts in /var/lib/puppet/lib/facter/jenkins_plugver.rb Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb Loading facts in /var/lib/puppet/lib/facter/ip6tables_version.rb Loading facts in /var/lib/puppet/lib/facter/ec2_instance_id_cached.rb Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to submit ''replace facts'' command for ue1a-devweb02.mycompany.com to PuppetDB atpuppet.mycompany.com:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A Not using cache on failed catalog Could not retrieve catalog; skipping run The puppet master: May 13 21:46:24 ue1a-util01 puppet-master[28860]: Compiled catalog for ue1a-devweb02.mycompany.com in environment production in 2.36 seconds PuppetDB: 2013-05-13 21:46:22,424 INFO [command-proc-46] [puppetdb.command] [e393b600-02c6-4b60-886e-7056516afef7] [replace facts] ue1a-devweb02.mycompany.com 2013-05-13 21:46:31,497 INFO [command-proc-46] [puppetdb.command] [b6f94e37-6732-44fa-b369-1bc36ed2ef54] [replace catalog] ue1a-devweb02.mycompany.com I turned on debug logging for puppetdb and looked through the logs when an event happened and saw this in the logs: 2013-05-14 20:19:19,237 WARN [qtp874952411-436] [io.nio] javax.net.ssl.SSLHandshakeException: Invalid Padding length: 41 I''m not sure if that''s the actual issue or not, but it''s what jumped out at me. Anyone else having this issue or know of what to fix? I can provide more logs if needed - I''m just not sure what all is relevant out of the puppetdb debug logs. Puppet 3.1.1, PuppetDB 1.3.0, CentOS 6.4. Thanks, Justin -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Ken Barber
2013-May-15 13:06 UTC
Re: [Puppet Users] PuppetDB "Failed to submit ''replace facts'' command"
Justin, Take a look at this: http://projects.puppetlabs.com/issues/19884 My colleague Deepak produced a patch for this problem and it has been merged in, but its not yet released. In the meantime you can work-around the problem by downgrading to JDK 1.6, which seems to do the trick. ken. On Tue, May 14, 2013 at 11:55 PM, Justin Lambert <jlambert@infiniteviewtech.com> wrote:> I have a small environment with a single puppetmaster and puppetdb on the > same node that I get sporadic "Failed to submit ''replace facts'' command" > errors from nodes. It isn''t consistent about which nodes are causing the > error and I can''t find anything else going on at the same time on either the > master or the agent that these errors happen. > > Here are the logs from a previous event > > The puppet client log: > Retrieving plugin > Loading facts in /var/lib/puppet/lib/facter/pe_version.rb > Loading facts in /var/lib/puppet/lib/facter/root_home.rb > Loading facts in /var/lib/puppet/lib/facter/classes.rb > Loading facts in /var/lib/puppet/lib/facter/puppet_enabled.rb > Loading facts in /var/lib/puppet/lib/facter/puppetdb_keystore_pw.rb > Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb > Loading facts in /var/lib/puppet/lib/facter/disposition.rb > Loading facts in /var/lib/puppet/lib/facter/ec2_security_groups_cached.rb > Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb > Loading facts in /var/lib/puppet/lib/facter/iptables_version.rb > Loading facts in /var/lib/puppet/lib/facter/jenkins_plugver.rb > Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb > Loading facts in /var/lib/puppet/lib/facter/ip6tables_version.rb > Loading facts in /var/lib/puppet/lib/facter/ec2_instance_id_cached.rb > Could not retrieve catalog from remote server: Error 400 on SERVER: Failed > to submit ''replace facts'' command for ue1a-devweb02.mycompany.com to > PuppetDB atpuppet.mycompany.com:8081: SSL_connect SYSCALL returned=5 errno=0 > state=SSLv3 read finished A > Not using cache on failed catalog > Could not retrieve catalog; skipping run > > The puppet master: > May 13 21:46:24 ue1a-util01 puppet-master[28860]: Compiled catalog for > ue1a-devweb02.mycompany.com in environment production in 2.36 seconds > > PuppetDB: > 2013-05-13 21:46:22,424 INFO [command-proc-46] [puppetdb.command] > [e393b600-02c6-4b60-886e-7056516afef7] [replace facts] > ue1a-devweb02.mycompany.com > 2013-05-13 21:46:31,497 INFO [command-proc-46] [puppetdb.command] > [b6f94e37-6732-44fa-b369-1bc36ed2ef54] [replace catalog] > ue1a-devweb02.mycompany.com > > > I turned on debug logging for puppetdb and looked through the logs when an > event happened and saw this in the logs: > 2013-05-14 20:19:19,237 WARN [qtp874952411-436] [io.nio] > javax.net.ssl.SSLHandshakeException: Invalid Padding length: 41 > > I''m not sure if that''s the actual issue or not, but it''s what jumped out at > me. Anyone else having this issue or know of what to fix? I can provide > more logs if needed - I''m just not sure what all is relevant out of the > puppetdb debug logs. > > Puppet 3.1.1, PuppetDB 1.3.0, CentOS 6.4. > > Thanks, > Justin > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Justin Lambert
2013-May-16 12:23 UTC
Re: [Puppet Users] PuppetDB "Failed to submit ''replace facts'' command"
Thanks for this Ken, I checked the mailing list but not the bug tracker - should have done that. Do you know if there is an anticipated 1.3.1 release for puppetdb? I haven''t seen any mention of what the next release is going to be. On May 15, 2013, at 7:06 AM, Ken Barber <ken@puppetlabs.com> wrote:> Justin, > > Take a look at this: http://projects.puppetlabs.com/issues/19884 > > My colleague Deepak produced a patch for this problem and it has been > merged in, but its not yet released. In the meantime you can > work-around the problem by downgrading to JDK 1.6, which seems to do > the trick. > > ken. > > On Tue, May 14, 2013 at 11:55 PM, Justin Lambert > <jlambert@infiniteviewtech.com> wrote: >> I have a small environment with a single puppetmaster and puppetdb on the >> same node that I get sporadic "Failed to submit ''replace facts'' command" >> errors from nodes. It isn''t consistent about which nodes are causing the >> error and I can''t find anything else going on at the same time on either the >> master or the agent that these errors happen. >> >> Here are the logs from a previous event >> >> The puppet client log: >> Retrieving plugin >> Loading facts in /var/lib/puppet/lib/facter/pe_version.rb >> Loading facts in /var/lib/puppet/lib/facter/root_home.rb >> Loading facts in /var/lib/puppet/lib/facter/classes.rb >> Loading facts in /var/lib/puppet/lib/facter/puppet_enabled.rb >> Loading facts in /var/lib/puppet/lib/facter/puppetdb_keystore_pw.rb >> Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb >> Loading facts in /var/lib/puppet/lib/facter/disposition.rb >> Loading facts in /var/lib/puppet/lib/facter/ec2_security_groups_cached.rb >> Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb >> Loading facts in /var/lib/puppet/lib/facter/iptables_version.rb >> Loading facts in /var/lib/puppet/lib/facter/jenkins_plugver.rb >> Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb >> Loading facts in /var/lib/puppet/lib/facter/ip6tables_version.rb >> Loading facts in /var/lib/puppet/lib/facter/ec2_instance_id_cached.rb >> Could not retrieve catalog from remote server: Error 400 on SERVER: Failed >> to submit ''replace facts'' command for ue1a-devweb02.mycompany.com to >> PuppetDB atpuppet.mycompany.com:8081: SSL_connect SYSCALL returned=5 errno=0 >> state=SSLv3 read finished A >> Not using cache on failed catalog >> Could not retrieve catalog; skipping run >> >> The puppet master: >> May 13 21:46:24 ue1a-util01 puppet-master[28860]: Compiled catalog for >> ue1a-devweb02.mycompany.com in environment production in 2.36 seconds >> >> PuppetDB: >> 2013-05-13 21:46:22,424 INFO [command-proc-46] [puppetdb.command] >> [e393b600-02c6-4b60-886e-7056516afef7] [replace facts] >> ue1a-devweb02.mycompany.com >> 2013-05-13 21:46:31,497 INFO [command-proc-46] [puppetdb.command] >> [b6f94e37-6732-44fa-b369-1bc36ed2ef54] [replace catalog] >> ue1a-devweb02.mycompany.com >> >> >> I turned on debug logging for puppetdb and looked through the logs when an >> event happened and saw this in the logs: >> 2013-05-14 20:19:19,237 WARN [qtp874952411-436] [io.nio] >> javax.net.ssl.SSLHandshakeException: Invalid Padding length: 41 >> >> I''m not sure if that''s the actual issue or not, but it''s what jumped out at >> me. Anyone else having this issue or know of what to fix? I can provide >> more logs if needed - I''m just not sure what all is relevant out of the >> puppetdb debug logs. >> >> Puppet 3.1.1, PuppetDB 1.3.0, CentOS 6.4. >> >> Thanks, >> Justin >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to puppet-users+unsubscribe@googlegroups.com. >> To post to this group, send email to puppet-users@googlegroups.com. >> Visit this group at http://groups.google.com/group/puppet-users?hl=en. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Ken Barber
2013-May-16 14:06 UTC
Re: [Puppet Users] PuppetDB "Failed to submit ''replace facts'' command"
Actually there is an anticipated 1.3.1, and it should have the fix for JDK 1.7. We''re in the middle of sorting that out now. On Thu, May 16, 2013 at 1:23 PM, Justin Lambert <jlambert@infiniteviewtech.com> wrote:> Thanks for this Ken, I checked the mailing list but not the bug tracker - should have done that. > > Do you know if there is an anticipated 1.3.1 release for puppetdb? I haven''t seen any mention of what the next release is going to be. > > On May 15, 2013, at 7:06 AM, Ken Barber <ken@puppetlabs.com> wrote: > >> Justin, >> >> Take a look at this: http://projects.puppetlabs.com/issues/19884 >> >> My colleague Deepak produced a patch for this problem and it has been >> merged in, but its not yet released. In the meantime you can >> work-around the problem by downgrading to JDK 1.6, which seems to do >> the trick. >> >> ken. >> >> On Tue, May 14, 2013 at 11:55 PM, Justin Lambert >> <jlambert@infiniteviewtech.com> wrote: >>> I have a small environment with a single puppetmaster and puppetdb on the >>> same node that I get sporadic "Failed to submit ''replace facts'' command" >>> errors from nodes. It isn''t consistent about which nodes are causing the >>> error and I can''t find anything else going on at the same time on either the >>> master or the agent that these errors happen. >>> >>> Here are the logs from a previous event >>> >>> The puppet client log: >>> Retrieving plugin >>> Loading facts in /var/lib/puppet/lib/facter/pe_version.rb >>> Loading facts in /var/lib/puppet/lib/facter/root_home.rb >>> Loading facts in /var/lib/puppet/lib/facter/classes.rb >>> Loading facts in /var/lib/puppet/lib/facter/puppet_enabled.rb >>> Loading facts in /var/lib/puppet/lib/facter/puppetdb_keystore_pw.rb >>> Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb >>> Loading facts in /var/lib/puppet/lib/facter/disposition.rb >>> Loading facts in /var/lib/puppet/lib/facter/ec2_security_groups_cached.rb >>> Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb >>> Loading facts in /var/lib/puppet/lib/facter/iptables_version.rb >>> Loading facts in /var/lib/puppet/lib/facter/jenkins_plugver.rb >>> Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb >>> Loading facts in /var/lib/puppet/lib/facter/ip6tables_version.rb >>> Loading facts in /var/lib/puppet/lib/facter/ec2_instance_id_cached.rb >>> Could not retrieve catalog from remote server: Error 400 on SERVER: Failed >>> to submit ''replace facts'' command for ue1a-devweb02.mycompany.com to >>> PuppetDB atpuppet.mycompany.com:8081: SSL_connect SYSCALL returned=5 errno=0 >>> state=SSLv3 read finished A >>> Not using cache on failed catalog >>> Could not retrieve catalog; skipping run >>> >>> The puppet master: >>> May 13 21:46:24 ue1a-util01 puppet-master[28860]: Compiled catalog for >>> ue1a-devweb02.mycompany.com in environment production in 2.36 seconds >>> >>> PuppetDB: >>> 2013-05-13 21:46:22,424 INFO [command-proc-46] [puppetdb.command] >>> [e393b600-02c6-4b60-886e-7056516afef7] [replace facts] >>> ue1a-devweb02.mycompany.com >>> 2013-05-13 21:46:31,497 INFO [command-proc-46] [puppetdb.command] >>> [b6f94e37-6732-44fa-b369-1bc36ed2ef54] [replace catalog] >>> ue1a-devweb02.mycompany.com >>> >>> >>> I turned on debug logging for puppetdb and looked through the logs when an >>> event happened and saw this in the logs: >>> 2013-05-14 20:19:19,237 WARN [qtp874952411-436] [io.nio] >>> javax.net.ssl.SSLHandshakeException: Invalid Padding length: 41 >>> >>> I''m not sure if that''s the actual issue or not, but it''s what jumped out at >>> me. Anyone else having this issue or know of what to fix? I can provide >>> more logs if needed - I''m just not sure what all is relevant out of the >>> puppetdb debug logs. >>> >>> Puppet 3.1.1, PuppetDB 1.3.0, CentOS 6.4. >>> >>> Thanks, >>> Justin >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Puppet Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to puppet-users+unsubscribe@googlegroups.com. >>> To post to this group, send email to puppet-users@googlegroups.com. >>> Visit this group at http://groups.google.com/group/puppet-users?hl=en. >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >>> >> >> -- >> You received this message because you are subscribed to the Google Groups "Puppet Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. >> To post to this group, send email to puppet-users@googlegroups.com. >> Visit this group at http://groups.google.com/group/puppet-users?hl=en. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.