Hello, Stuck in a weird place here. I am trying to do ''puppet master --compile foo.com'', however I am not getting the catalog json. So far, I have noticed two sort of outputs: 1) The above command results in : notice: Compiled catalog for foo.com in environment production in 10.60 seconds Failed to submit ''replace catalog'' command for foo.com to PuppetDB at puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A Notice that I don''t actually get any json & of course, the ssl error. 2) Other output: Failed when searching for node bar.com: Could not retrieve facts for bar.com: Failed to find facts from PuppetDB at puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A Disabling storeconfig allows me to get a proper catalog json. Regarding the ssl errors, we''ve established the correctness of certificates in the relevant places - and done tests using curl and using ruby net/http. Here''s a gist of for the curl and ruby snippet. https://gist.github.com/alcy/5262866 Any suggestions what could be happening here ? Perhaps in indirector/facts/puppetdb.rb, http_get doesn''t get the correct ssl params or something ? -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Mohit Chawla
2013-Mar-28 12:55 UTC
[Puppet Users] Re: Catalog compilation, puppetdb and ssl
Forgot mentioning the env details: [user@puppetmaster ~]# rpm -qa | grep puppet puppetlabs-release-6-6.noarch puppetdb-terminus-1.0.4-1.el6.noarch mcollective-puppet-agent-1.4.1-1.noarch puppet-2.7.20-1.el6.noarch hiera-puppet-1.0.0-1.el6.noarch puppet-server-2.7.20-1.el6.noarch mcollective-puppet-common-1.4.1-1.noarch [user@puppetmaster ~]# cat /etc/centos-release CentOS release 6.3 (Final) On Thu, Mar 28, 2013 at 6:23 PM, Mohit Chawla <mohit.chawla.binary@gmail.com> wrote:> Hello, > > Stuck in a weird place here. I am trying to do ''puppet master > --compile foo.com'', however I am not getting the catalog json. So far, > I have noticed two sort of outputs: > > 1) The above command results in : > notice: Compiled catalog for foo.com in environment production in 10.60 seconds > Failed to submit ''replace catalog'' command for foo.com to PuppetDB at > puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read > finished A > > Notice that I don''t actually get any json & of course, the ssl error. > > 2) Other output: > Failed when searching for node bar.com: Could not retrieve facts for > bar.com: Failed to find facts from PuppetDB at puppetdb:8081: > SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A > > Disabling storeconfig allows me to get a proper catalog json. > > Regarding the ssl errors, we''ve established the correctness of > certificates in the relevant places - and done tests using curl and > using ruby net/http. Here''s a gist of for the curl and ruby snippet. > https://gist.github.com/alcy/5262866 > > Any suggestions what could be happening here ? Perhaps in > indirector/facts/puppetdb.rb, http_get doesn''t get the correct ssl > params or something ?-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Ken Barber
2013-Mar-28 13:12 UTC
Re: [Puppet Users] Re: Catalog compilation, puppetdb and ssl
So I have some questions, as the error could mean a number of things: What version of PuppetDB are you running? And what exact version of Java is it using? Can you take a look at puppetdb.log and tell me if you see any meaningful error messages? Without trying to compile a catalog in this manner - are you getting any trouble with replace facts/replace catalogs generally? ie. Just trying to running puppet agent -t for example? Have you tried re-initialisation your SSL setup with puppetdb-ssl-setup at all yet? Does the hostname you have specified in /etc/puppet/puppetdb.conf to talk to your puppetdb server match the SSL certificate assigned to the puppetdb host itself? On Thu, Mar 28, 2013 at 12:55 PM, Mohit Chawla <mohit.chawla.binary@gmail.com> wrote:> Forgot mentioning the env details: > > [user@puppetmaster ~]# rpm -qa | grep puppet > puppetlabs-release-6-6.noarch > puppetdb-terminus-1.0.4-1.el6.noarch > mcollective-puppet-agent-1.4.1-1.noarch > puppet-2.7.20-1.el6.noarch > hiera-puppet-1.0.0-1.el6.noarch > puppet-server-2.7.20-1.el6.noarch > mcollective-puppet-common-1.4.1-1.noarch > > [user@puppetmaster ~]# cat /etc/centos-release > CentOS release 6.3 (Final) > > On Thu, Mar 28, 2013 at 6:23 PM, Mohit Chawla > <mohit.chawla.binary@gmail.com> wrote: >> Hello, >> >> Stuck in a weird place here. I am trying to do ''puppet master >> --compile foo.com'', however I am not getting the catalog json. So far, >> I have noticed two sort of outputs: >> >> 1) The above command results in : >> notice: Compiled catalog for foo.com in environment production in 10.60 seconds >> Failed to submit ''replace catalog'' command for foo.com to PuppetDB at >> puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read >> finished A >> >> Notice that I don''t actually get any json & of course, the ssl error. >> >> 2) Other output: >> Failed when searching for node bar.com: Could not retrieve facts for >> bar.com: Failed to find facts from PuppetDB at puppetdb:8081: >> SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A >> >> Disabling storeconfig allows me to get a proper catalog json. >> >> Regarding the ssl errors, we''ve established the correctness of >> certificates in the relevant places - and done tests using curl and >> using ruby net/http. Here''s a gist of for the curl and ruby snippet. >> https://gist.github.com/alcy/5262866 >> >> Any suggestions what could be happening here ? Perhaps in >> indirector/facts/puppetdb.rb, http_get doesn''t get the correct ssl >> params or something ? > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Mohit Chawla
2013-Mar-28 13:17 UTC
Re: [Puppet Users] Re: Catalog compilation, puppetdb and ssl
Hello Ken, Thanks for the response. On Thu, Mar 28, 2013 at 6:42 PM, Ken Barber <ken@puppetlabs.com> wrote:> So I have some questions, as the error could mean a number of things: > > What version of PuppetDB are you running? And what exact version of > Java is it using?puppetdb version is puppetdb-1.0.4-1.el6.noarch. $ java -version java version "1.6.0_24" OpenJDK Runtime Environment (IcedTea6 1.11.5) (rhel-1.50.1.11.5.el6_3-x86_64) OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)> > Can you take a look at puppetdb.log and tell me if you see any > meaningful error messages?I took a look, no error messages, only info messages.> > Without trying to compile a catalog in this manner - are you getting > any trouble with replace facts/replace catalogs generally? ie. Just > trying to running puppet agent -t for example?Nope, those runs are working fine.> > Have you tried re-initialisation your SSL setup with > puppetdb-ssl-setup at all yet?Yes, and confirmed the validity of certificates and their fingerprints in the truststore and master ca.> > Does the hostname you have specified in /etc/puppet/puppetdb.conf to > talk to your puppetdb server match the SSL certificate assigned to the > puppetdb host itself?Yes.> > On Thu, Mar 28, 2013 at 12:55 PM, Mohit Chawla > <mohit.chawla.binary@gmail.com> wrote: >> Forgot mentioning the env details: >> >> [user@puppetmaster ~]# rpm -qa | grep puppet >> puppetlabs-release-6-6.noarch >> puppetdb-terminus-1.0.4-1.el6.noarch >> mcollective-puppet-agent-1.4.1-1.noarch >> puppet-2.7.20-1.el6.noarch >> hiera-puppet-1.0.0-1.el6.noarch >> puppet-server-2.7.20-1.el6.noarch >> mcollective-puppet-common-1.4.1-1.noarch >> >> [user@puppetmaster ~]# cat /etc/centos-release >> CentOS release 6.3 (Final) >> >> On Thu, Mar 28, 2013 at 6:23 PM, Mohit Chawla >> <mohit.chawla.binary@gmail.com> wrote: >>> Hello, >>> >>> Stuck in a weird place here. I am trying to do ''puppet master >>> --compile foo.com'', however I am not getting the catalog json. So far, >>> I have noticed two sort of outputs: >>> >>> 1) The above command results in : >>> notice: Compiled catalog for foo.com in environment production in 10.60 seconds >>> Failed to submit ''replace catalog'' command for foo.com to PuppetDB at >>> puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read >>> finished A >>> >>> Notice that I don''t actually get any json & of course, the ssl error. >>> >>> 2) Other output: >>> Failed when searching for node bar.com: Could not retrieve facts for >>> bar.com: Failed to find facts from PuppetDB at puppetdb:8081: >>> SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A >>> >>> Disabling storeconfig allows me to get a proper catalog json. >>> >>> Regarding the ssl errors, we''ve established the correctness of >>> certificates in the relevant places - and done tests using curl and >>> using ruby net/http. Here''s a gist of for the curl and ruby snippet. >>> https://gist.github.com/alcy/5262866 >>> >>> Any suggestions what could be happening here ? Perhaps in >>> indirector/facts/puppetdb.rb, http_get doesn''t get the correct ssl >>> params or something ? >> >> -- >> You received this message because you are subscribed to the Google Groups "Puppet Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. >> To post to this group, send email to puppet-users@googlegroups.com. >> Visit this group at http://groups.google.com/group/puppet-users?hl=en. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Ken Barber
2013-Mar-28 13:37 UTC
Re: [Puppet Users] Re: Catalog compilation, puppetdb and ssl
I''m just trying to run up the same environment so I can try to replicate it, as yet I can''t replicate it on the newer environment. What does your puppet.conf look like on the host you are trying to run puppet master --compile btw? I presume you are trying to run the command as root, is there a ~/.puppet directory for that user at all? ken. On Thu, Mar 28, 2013 at 1:17 PM, Mohit Chawla <mohit.chawla.binary@gmail.com> wrote:> Hello Ken, > > Thanks for the response. > > On Thu, Mar 28, 2013 at 6:42 PM, Ken Barber <ken@puppetlabs.com> wrote: >> So I have some questions, as the error could mean a number of things: >> >> What version of PuppetDB are you running? And what exact version of >> Java is it using? > > puppetdb version is puppetdb-1.0.4-1.el6.noarch. > > $ java -version > java version "1.6.0_24" > OpenJDK Runtime Environment (IcedTea6 1.11.5) (rhel-1.50.1.11.5.el6_3-x86_64) > OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode) > >> >> Can you take a look at puppetdb.log and tell me if you see any >> meaningful error messages? > > I took a look, no error messages, only info messages. >> >> Without trying to compile a catalog in this manner - are you getting >> any trouble with replace facts/replace catalogs generally? ie. Just >> trying to running puppet agent -t for example? > > Nope, those runs are working fine. > >> >> Have you tried re-initialisation your SSL setup with >> puppetdb-ssl-setup at all yet? > > Yes, and confirmed the validity of certificates and their fingerprints > in the truststore and master ca. > >> >> Does the hostname you have specified in /etc/puppet/puppetdb.conf to >> talk to your puppetdb server match the SSL certificate assigned to the >> puppetdb host itself? > > Yes. > >> >> On Thu, Mar 28, 2013 at 12:55 PM, Mohit Chawla >> <mohit.chawla.binary@gmail.com> wrote: >>> Forgot mentioning the env details: >>> >>> [user@puppetmaster ~]# rpm -qa | grep puppet >>> puppetlabs-release-6-6.noarch >>> puppetdb-terminus-1.0.4-1.el6.noarch >>> mcollective-puppet-agent-1.4.1-1.noarch >>> puppet-2.7.20-1.el6.noarch >>> hiera-puppet-1.0.0-1.el6.noarch >>> puppet-server-2.7.20-1.el6.noarch >>> mcollective-puppet-common-1.4.1-1.noarch >>> >>> [user@puppetmaster ~]# cat /etc/centos-release >>> CentOS release 6.3 (Final) >>> >>> On Thu, Mar 28, 2013 at 6:23 PM, Mohit Chawla >>> <mohit.chawla.binary@gmail.com> wrote: >>>> Hello, >>>> >>>> Stuck in a weird place here. I am trying to do ''puppet master >>>> --compile foo.com'', however I am not getting the catalog json. So far, >>>> I have noticed two sort of outputs: >>>> >>>> 1) The above command results in : >>>> notice: Compiled catalog for foo.com in environment production in 10.60 seconds >>>> Failed to submit ''replace catalog'' command for foo.com to PuppetDB at >>>> puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read >>>> finished A >>>> >>>> Notice that I don''t actually get any json & of course, the ssl error. >>>> >>>> 2) Other output: >>>> Failed when searching for node bar.com: Could not retrieve facts for >>>> bar.com: Failed to find facts from PuppetDB at puppetdb:8081: >>>> SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A >>>> >>>> Disabling storeconfig allows me to get a proper catalog json. >>>> >>>> Regarding the ssl errors, we''ve established the correctness of >>>> certificates in the relevant places - and done tests using curl and >>>> using ruby net/http. Here''s a gist of for the curl and ruby snippet. >>>> https://gist.github.com/alcy/5262866 >>>> >>>> Any suggestions what could be happening here ? Perhaps in >>>> indirector/facts/puppetdb.rb, http_get doesn''t get the correct ssl >>>> params or something ? >>> >>> -- >>> You received this message because you are subscribed to the Google Groups "Puppet Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. >>> To post to this group, send email to puppet-users@googlegroups.com. >>> Visit this group at http://groups.google.com/group/puppet-users?hl=en. >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >>> >> >> -- >> You received this message because you are subscribed to the Google Groups "Puppet Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. >> To post to this group, send email to puppet-users@googlegroups.com. >> Visit this group at http://groups.google.com/group/puppet-users?hl=en. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Mohit Chawla
2013-Mar-28 20:54 UTC
Re: [Puppet Users] Re: Catalog compilation, puppetdb and ssl
Hi, Not at the workstation right now, but regarding puppet.conf I cant think of any peculiar settings apart from this being one of the two puppet masters apart from a separate ca server (we took care of having the ca server''s certs being available at these masters). And afaik right now, there wasn''t any ~/.puppet dir for root, however I need to confirm this. On Thu, Mar 28, 2013 at 7:07 PM, Ken Barber <ken@puppetlabs.com> wrote:> I''m just trying to run up the same environment so I can try to > replicate it, as yet I can''t replicate it on the newer environment. > > What does your puppet.conf look like on the host you are trying to run > puppet master --compile btw? I presume you are trying to run the > command as root, is there a ~/.puppet directory for that user at all? > > ken. > > On Thu, Mar 28, 2013 at 1:17 PM, Mohit Chawla > <mohit.chawla.binary@gmail.com> wrote: >> Hello Ken, >> >> Thanks for the response. >> >> On Thu, Mar 28, 2013 at 6:42 PM, Ken Barber <ken@puppetlabs.com> wrote: >>> So I have some questions, as the error could mean a number of things: >>> >>> What version of PuppetDB are you running? And what exact version of >>> Java is it using? >> >> puppetdb version is puppetdb-1.0.4-1.el6.noarch. >> >> $ java -version >> java version "1.6.0_24" >> OpenJDK Runtime Environment (IcedTea6 1.11.5) (rhel-1.50.1.11.5.el6_3-x86_64) >> OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode) >> >>> >>> Can you take a look at puppetdb.log and tell me if you see any >>> meaningful error messages? >> >> I took a look, no error messages, only info messages. >>> >>> Without trying to compile a catalog in this manner - are you getting >>> any trouble with replace facts/replace catalogs generally? ie. Just >>> trying to running puppet agent -t for example? >> >> Nope, those runs are working fine. >> >>> >>> Have you tried re-initialisation your SSL setup with >>> puppetdb-ssl-setup at all yet? >> >> Yes, and confirmed the validity of certificates and their fingerprints >> in the truststore and master ca. >> >>> >>> Does the hostname you have specified in /etc/puppet/puppetdb.conf to >>> talk to your puppetdb server match the SSL certificate assigned to the >>> puppetdb host itself? >> >> Yes. >> >>> >>> On Thu, Mar 28, 2013 at 12:55 PM, Mohit Chawla >>> <mohit.chawla.binary@gmail.com> wrote: >>>> Forgot mentioning the env details: >>>> >>>> [user@puppetmaster ~]# rpm -qa | grep puppet >>>> puppetlabs-release-6-6.noarch >>>> puppetdb-terminus-1.0.4-1.el6.noarch >>>> mcollective-puppet-agent-1.4.1-1.noarch >>>> puppet-2.7.20-1.el6.noarch >>>> hiera-puppet-1.0.0-1.el6.noarch >>>> puppet-server-2.7.20-1.el6.noarch >>>> mcollective-puppet-common-1.4.1-1.noarch >>>> >>>> [user@puppetmaster ~]# cat /etc/centos-release >>>> CentOS release 6.3 (Final) >>>> >>>> On Thu, Mar 28, 2013 at 6:23 PM, Mohit Chawla >>>> <mohit.chawla.binary@gmail.com> wrote: >>>>> Hello, >>>>> >>>>> Stuck in a weird place here. I am trying to do ''puppet master >>>>> --compile foo.com'', however I am not getting the catalog json. So far, >>>>> I have noticed two sort of outputs: >>>>> >>>>> 1) The above command results in : >>>>> notice: Compiled catalog for foo.com in environment production in 10.60 seconds >>>>> Failed to submit ''replace catalog'' command for foo.com to PuppetDB at >>>>> puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read >>>>> finished A >>>>> >>>>> Notice that I don''t actually get any json & of course, the ssl error. >>>>> >>>>> 2) Other output: >>>>> Failed when searching for node bar.com: Could not retrieve facts for >>>>> bar.com: Failed to find facts from PuppetDB at puppetdb:8081: >>>>> SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A >>>>> >>>>> Disabling storeconfig allows me to get a proper catalog json. >>>>> >>>>> Regarding the ssl errors, we''ve established the correctness of >>>>> certificates in the relevant places - and done tests using curl and >>>>> using ruby net/http. Here''s a gist of for the curl and ruby snippet. >>>>> https://gist.github.com/alcy/5262866 >>>>> >>>>> Any suggestions what could be happening here ? Perhaps in >>>>> indirector/facts/puppetdb.rb, http_get doesn''t get the correct ssl >>>>> params or something ? >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups "Puppet Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. >>>> To post to this group, send email to puppet-users@googlegroups.com. >>>> Visit this group at http://groups.google.com/group/puppet-users?hl=en. >>>> For more options, visit https://groups.google.com/groups/opt_out. >>>> >>>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups "Puppet Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. >>> To post to this group, send email to puppet-users@googlegroups.com. >>> Visit this group at http://groups.google.com/group/puppet-users?hl=en. >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >>> >> >> -- >> You received this message because you are subscribed to the Google Groups "Puppet Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. >> To post to this group, send email to puppet-users@googlegroups.com. >> Visit this group at http://groups.google.com/group/puppet-users?hl=en. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Ken Barber
2013-Mar-28 21:44 UTC
Re: [Puppet Users] Re: Catalog compilation, puppetdb and ssl
Yeah, it does seem very odd though ... if agent works - and the master is able to talk to PuppetDB no problem, then its weird that running puppet master on the command line doesn''t seem to work. What is strange is that the SSL error is very very unspecific: Failed to submit ''replace catalog'' command for foo.com to PuppetDB at puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A It doesn''t speak of a particular problem, ie. remote host doesn''t match certificate, CA not verified etc. etc. it just says ''read finished A''. I think this is significant, the last time I''ve seen it is when there was an SSL bug, and the transport is cut short - which is why I wanted to see what you had in your logs. If you try running the command with --debug --trace do you get any more information? If you tcpdump the connection to the puppetdb server do you see the TCP packets hit the server on port 8081? Try running puppetdb-foreground --debug on your puppetdb server and see what happens when you attempt to compile like this. You''d obviously want to try these things when the other activity on the server is disabled :-). ken. On Thu, Mar 28, 2013 at 8:54 PM, Mohit Chawla <mohit.chawla.binary@gmail.com> wrote:> Hi, > > Not at the workstation right now, but regarding puppet.conf I cant > think of any peculiar settings apart from this being one of the two > puppet masters apart from a separate ca server (we took care of having > the ca server''s certs being available at these masters). And afaik > right now, there wasn''t any ~/.puppet dir for root, however I need to > confirm this. > > On Thu, Mar 28, 2013 at 7:07 PM, Ken Barber <ken@puppetlabs.com> wrote: >> I''m just trying to run up the same environment so I can try to >> replicate it, as yet I can''t replicate it on the newer environment. >> >> What does your puppet.conf look like on the host you are trying to run >> puppet master --compile btw? I presume you are trying to run the >> command as root, is there a ~/.puppet directory for that user at all? >> >> ken. >> >> On Thu, Mar 28, 2013 at 1:17 PM, Mohit Chawla >> <mohit.chawla.binary@gmail.com> wrote: >>> Hello Ken, >>> >>> Thanks for the response. >>> >>> On Thu, Mar 28, 2013 at 6:42 PM, Ken Barber <ken@puppetlabs.com> wrote: >>>> So I have some questions, as the error could mean a number of things: >>>> >>>> What version of PuppetDB are you running? And what exact version of >>>> Java is it using? >>> >>> puppetdb version is puppetdb-1.0.4-1.el6.noarch. >>> >>> $ java -version >>> java version "1.6.0_24" >>> OpenJDK Runtime Environment (IcedTea6 1.11.5) (rhel-1.50.1.11.5.el6_3-x86_64) >>> OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode) >>> >>>> >>>> Can you take a look at puppetdb.log and tell me if you see any >>>> meaningful error messages? >>> >>> I took a look, no error messages, only info messages. >>>> >>>> Without trying to compile a catalog in this manner - are you getting >>>> any trouble with replace facts/replace catalogs generally? ie. Just >>>> trying to running puppet agent -t for example? >>> >>> Nope, those runs are working fine. >>> >>>> >>>> Have you tried re-initialisation your SSL setup with >>>> puppetdb-ssl-setup at all yet? >>> >>> Yes, and confirmed the validity of certificates and their fingerprints >>> in the truststore and master ca. >>> >>>> >>>> Does the hostname you have specified in /etc/puppet/puppetdb.conf to >>>> talk to your puppetdb server match the SSL certificate assigned to the >>>> puppetdb host itself? >>> >>> Yes. >>> >>>> >>>> On Thu, Mar 28, 2013 at 12:55 PM, Mohit Chawla >>>> <mohit.chawla.binary@gmail.com> wrote: >>>>> Forgot mentioning the env details: >>>>> >>>>> [user@puppetmaster ~]# rpm -qa | grep puppet >>>>> puppetlabs-release-6-6.noarch >>>>> puppetdb-terminus-1.0.4-1.el6.noarch >>>>> mcollective-puppet-agent-1.4.1-1.noarch >>>>> puppet-2.7.20-1.el6.noarch >>>>> hiera-puppet-1.0.0-1.el6.noarch >>>>> puppet-server-2.7.20-1.el6.noarch >>>>> mcollective-puppet-common-1.4.1-1.noarch >>>>> >>>>> [user@puppetmaster ~]# cat /etc/centos-release >>>>> CentOS release 6.3 (Final) >>>>> >>>>> On Thu, Mar 28, 2013 at 6:23 PM, Mohit Chawla >>>>> <mohit.chawla.binary@gmail.com> wrote: >>>>>> Hello, >>>>>> >>>>>> Stuck in a weird place here. I am trying to do ''puppet master >>>>>> --compile foo.com'', however I am not getting the catalog json. So far, >>>>>> I have noticed two sort of outputs: >>>>>> >>>>>> 1) The above command results in : >>>>>> notice: Compiled catalog for foo.com in environment production in 10.60 seconds >>>>>> Failed to submit ''replace catalog'' command for foo.com to PuppetDB at >>>>>> puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read >>>>>> finished A >>>>>> >>>>>> Notice that I don''t actually get any json & of course, the ssl error. >>>>>> >>>>>> 2) Other output: >>>>>> Failed when searching for node bar.com: Could not retrieve facts for >>>>>> bar.com: Failed to find facts from PuppetDB at puppetdb:8081: >>>>>> SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A >>>>>> >>>>>> Disabling storeconfig allows me to get a proper catalog json. >>>>>> >>>>>> Regarding the ssl errors, we''ve established the correctness of >>>>>> certificates in the relevant places - and done tests using curl and >>>>>> using ruby net/http. Here''s a gist of for the curl and ruby snippet. >>>>>> https://gist.github.com/alcy/5262866 >>>>>> >>>>>> Any suggestions what could be happening here ? Perhaps in >>>>>> indirector/facts/puppetdb.rb, http_get doesn''t get the correct ssl >>>>>> params or something ? >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups "Puppet Users" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. >>>>> To post to this group, send email to puppet-users@googlegroups.com. >>>>> Visit this group at http://groups.google.com/group/puppet-users?hl=en. >>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>> >>>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups "Puppet Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. >>>> To post to this group, send email to puppet-users@googlegroups.com. >>>> Visit this group at http://groups.google.com/group/puppet-users?hl=en. >>>> For more options, visit https://groups.google.com/groups/opt_out. >>>> >>>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups "Puppet Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. >>> To post to this group, send email to puppet-users@googlegroups.com. >>> Visit this group at http://groups.google.com/group/puppet-users?hl=en. >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >>> >> >> -- >> You received this message because you are subscribed to the Google Groups "Puppet Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. >> To post to this group, send email to puppet-users@googlegroups.com. >> Visit this group at http://groups.google.com/group/puppet-users?hl=en. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Mohit Chawla
2013-Apr-01 07:49 UTC
Re: [Puppet Users] Re: Catalog compilation, puppetdb and ssl
Hi Ken, Got a trace from running puppetdb-foreground --debug - https://gist.github.com/alcy/5283661. Weird that this doesn''t happen during standard puppet runs as opposed to doing a --compile. Here''s the puppet trace when doing --compile - https://gist.github.com/alcy/5283712. On Fri, Mar 29, 2013 at 3:14 AM, Ken Barber <ken@puppetlabs.com> wrote:> Yeah, it does seem very odd though ... if agent works - and the master > is able to talk to PuppetDB no problem, then its weird that running > puppet master on the command line doesn''t seem to work. > > What is strange is that the SSL error is very very unspecific: > > Failed to submit ''replace catalog'' command for foo.com to PuppetDB at > puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read > finished A > > It doesn''t speak of a particular problem, ie. remote host doesn''t > match certificate, CA not verified etc. etc. it just says ''read > finished A''. I think this is significant, the last time I''ve seen it > is when there was an SSL bug, and the transport is cut short - which > is why I wanted to see what you had in your logs. > > If you try running the command with --debug --trace do you get any > more information? If you tcpdump the connection to the puppetdb server > do you see the TCP packets hit the server on port 8081? Try running > puppetdb-foreground --debug on your puppetdb server and see what > happens when you attempt to compile like this. You''d obviously want to > try these things when the other activity on the server is disabled > :-). > > ken. > > On Thu, Mar 28, 2013 at 8:54 PM, Mohit Chawla > <mohit.chawla.binary@gmail.com> wrote: > > Hi, > > > > Not at the workstation right now, but regarding puppet.conf I cant > > think of any peculiar settings apart from this being one of the two > > puppet masters apart from a separate ca server (we took care of having > > the ca server''s certs being available at these masters). And afaik > > right now, there wasn''t any ~/.puppet dir for root, however I need to > > confirm this. > > > > On Thu, Mar 28, 2013 at 7:07 PM, Ken Barber <ken@puppetlabs.com> wrote: > >> I''m just trying to run up the same environment so I can try to > >> replicate it, as yet I can''t replicate it on the newer environment. > >> > >> What does your puppet.conf look like on the host you are trying to run > >> puppet master --compile btw? I presume you are trying to run the > >> command as root, is there a ~/.puppet directory for that user at all? > >> > >> ken. > >> > >> On Thu, Mar 28, 2013 at 1:17 PM, Mohit Chawla > >> <mohit.chawla.binary@gmail.com> wrote: > >>> Hello Ken, > >>> > >>> Thanks for the response. > >>> > >>> On Thu, Mar 28, 2013 at 6:42 PM, Ken Barber <ken@puppetlabs.com> > wrote: > >>>> So I have some questions, as the error could mean a number of things: > >>>> > >>>> What version of PuppetDB are you running? And what exact version of > >>>> Java is it using? > >>> > >>> puppetdb version is puppetdb-1.0.4-1.el6.noarch. > >>> > >>> $ java -version > >>> java version "1.6.0_24" > >>> OpenJDK Runtime Environment (IcedTea6 1.11.5) > (rhel-1.50.1.11.5.el6_3-x86_64) > >>> OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode) > >>> > >>>> > >>>> Can you take a look at puppetdb.log and tell me if you see any > >>>> meaningful error messages? > >>> > >>> I took a look, no error messages, only info messages. > >>>> > >>>> Without trying to compile a catalog in this manner - are you getting > >>>> any trouble with replace facts/replace catalogs generally? ie. Just > >>>> trying to running puppet agent -t for example? > >>> > >>> Nope, those runs are working fine. > >>> > >>>> > >>>> Have you tried re-initialisation your SSL setup with > >>>> puppetdb-ssl-setup at all yet? > >>> > >>> Yes, and confirmed the validity of certificates and their fingerprints > >>> in the truststore and master ca. > >>> > >>>> > >>>> Does the hostname you have specified in /etc/puppet/puppetdb.conf to > >>>> talk to your puppetdb server match the SSL certificate assigned to the > >>>> puppetdb host itself? > >>> > >>> Yes. > >>> > >>>> > >>>> On Thu, Mar 28, 2013 at 12:55 PM, Mohit Chawla > >>>> <mohit.chawla.binary@gmail.com> wrote: > >>>>> Forgot mentioning the env details: > >>>>> > >>>>> [user@puppetmaster ~]# rpm -qa | grep puppet > >>>>> puppetlabs-release-6-6.noarch > >>>>> puppetdb-terminus-1.0.4-1.el6.noarch > >>>>> mcollective-puppet-agent-1.4.1-1.noarch > >>>>> puppet-2.7.20-1.el6.noarch > >>>>> hiera-puppet-1.0.0-1.el6.noarch > >>>>> puppet-server-2.7.20-1.el6.noarch > >>>>> mcollective-puppet-common-1.4.1-1.noarch > >>>>> > >>>>> [user@puppetmaster ~]# cat /etc/centos-release > >>>>> CentOS release 6.3 (Final) > >>>>> > >>>>> On Thu, Mar 28, 2013 at 6:23 PM, Mohit Chawla > >>>>> <mohit.chawla.binary@gmail.com> wrote: > >>>>>> Hello, > >>>>>> > >>>>>> Stuck in a weird place here. I am trying to do ''puppet master > >>>>>> --compile foo.com'', however I am not getting the catalog json. So > far, > >>>>>> I have noticed two sort of outputs: > >>>>>> > >>>>>> 1) The above command results in : > >>>>>> notice: Compiled catalog for foo.com in environment production in > 10.60 seconds > >>>>>> Failed to submit ''replace catalog'' command for foo.com to PuppetDB > at > >>>>>> puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 > read > >>>>>> finished A > >>>>>> > >>>>>> Notice that I don''t actually get any json & of course, the ssl > error. > >>>>>> > >>>>>> 2) Other output: > >>>>>> Failed when searching for node bar.com: Could not retrieve facts > for > >>>>>> bar.com: Failed to find facts from PuppetDB at puppetdb:8081: > >>>>>> SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A > >>>>>> > >>>>>> Disabling storeconfig allows me to get a proper catalog json. > >>>>>> > >>>>>> Regarding the ssl errors, we''ve established the correctness of > >>>>>> certificates in the relevant places - and done tests using curl and > >>>>>> using ruby net/http. Here''s a gist of for the curl and ruby snippet. > >>>>>> https://gist.github.com/alcy/5262866 > >>>>>> > >>>>>> Any suggestions what could be happening here ? Perhaps in > >>>>>> indirector/facts/puppetdb.rb, http_get doesn''t get the correct ssl > >>>>>> params or something ? > >>>>> > >>>>> -- > >>>>> You received this message because you are subscribed to the Google > Groups "Puppet Users" group. > >>>>> To unsubscribe from this group and stop receiving emails from it, > send an email to puppet-users+unsubscribe@googlegroups.com. > >>>>> To post to this group, send email to puppet-users@googlegroups.com. > >>>>> Visit this group at > http://groups.google.com/group/puppet-users?hl=en. > >>>>> For more options, visit https://groups.google.com/groups/opt_out. > >>>>> > >>>>> > >>>> > >>>> -- > >>>> You received this message because you are subscribed to the Google > Groups "Puppet Users" group. > >>>> To unsubscribe from this group and stop receiving emails from it, > send an email to puppet-users+unsubscribe@googlegroups.com. > >>>> To post to this group, send email to puppet-users@googlegroups.com. > >>>> Visit this group at http://groups.google.com/group/puppet-users?hl=en > . > >>>> For more options, visit https://groups.google.com/groups/opt_out. > >>>> > >>>> > >>> > >>> -- > >>> You received this message because you are subscribed to the Google > Groups "Puppet Users" group. > >>> To unsubscribe from this group and stop receiving emails from it, send > an email to puppet-users+unsubscribe@googlegroups.com. > >>> To post to this group, send email to puppet-users@googlegroups.com. > >>> Visit this group at http://groups.google.com/group/puppet-users?hl=en. > >>> For more options, visit https://groups.google.com/groups/opt_out. > >>> > >>> > >> > >> -- > >> You received this message because you are subscribed to the Google > Groups "Puppet Users" group. > >> To unsubscribe from this group and stop receiving emails from it, send > an email to puppet-users+unsubscribe@googlegroups.com. > >> To post to this group, send email to puppet-users@googlegroups.com. > >> Visit this group at http://groups.google.com/group/puppet-users?hl=en. > >> For more options, visit https://groups.google.com/groups/opt_out. > >> > >> > > > > -- > > You received this message because you are subscribed to the Google > Groups "Puppet Users" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to puppet-users+unsubscribe@googlegroups.com. > > To post to this group, send email to puppet-users@googlegroups.com. > > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > > For more options, visit https://groups.google.com/groups/opt_out. > > > > > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Mohit Chawla
2013-Apr-01 08:33 UTC
Re: [Puppet Users] Re: Catalog compilation, puppetdb and ssl
Hi, Got it working by passing --certname param (the ca server hostname, instead of this master). On Mon, Apr 1, 2013 at 1:19 PM, Mohit Chawla <mohit.chawla.binary@gmail.com>wrote:> Hi Ken, > > Got a trace from running puppetdb-foreground --debug - > https://gist.github.com/alcy/5283661. Weird that this doesn''t happen > during standard puppet runs as opposed to doing a --compile. Here''s the > puppet trace when doing --compile - https://gist.github.com/alcy/5283712. > > > > > On Fri, Mar 29, 2013 at 3:14 AM, Ken Barber <ken@puppetlabs.com> wrote: > >> Yeah, it does seem very odd though ... if agent works - and the master >> is able to talk to PuppetDB no problem, then its weird that running >> puppet master on the command line doesn''t seem to work. >> >> What is strange is that the SSL error is very very unspecific: >> >> Failed to submit ''replace catalog'' command for foo.com to PuppetDB at >> puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read >> finished A >> >> It doesn''t speak of a particular problem, ie. remote host doesn''t >> match certificate, CA not verified etc. etc. it just says ''read >> finished A''. I think this is significant, the last time I''ve seen it >> is when there was an SSL bug, and the transport is cut short - which >> is why I wanted to see what you had in your logs. >> >> If you try running the command with --debug --trace do you get any >> more information? If you tcpdump the connection to the puppetdb server >> do you see the TCP packets hit the server on port 8081? Try running >> puppetdb-foreground --debug on your puppetdb server and see what >> happens when you attempt to compile like this. You''d obviously want to >> try these things when the other activity on the server is disabled >> :-). >> >> ken. >> >> On Thu, Mar 28, 2013 at 8:54 PM, Mohit Chawla >> <mohit.chawla.binary@gmail.com> wrote: >> > Hi, >> > >> > Not at the workstation right now, but regarding puppet.conf I cant >> > think of any peculiar settings apart from this being one of the two >> > puppet masters apart from a separate ca server (we took care of having >> > the ca server''s certs being available at these masters). And afaik >> > right now, there wasn''t any ~/.puppet dir for root, however I need to >> > confirm this. >> > >> > On Thu, Mar 28, 2013 at 7:07 PM, Ken Barber <ken@puppetlabs.com> wrote: >> >> I''m just trying to run up the same environment so I can try to >> >> replicate it, as yet I can''t replicate it on the newer environment. >> >> >> >> What does your puppet.conf look like on the host you are trying to run >> >> puppet master --compile btw? I presume you are trying to run the >> >> command as root, is there a ~/.puppet directory for that user at all? >> >> >> >> ken. >> >> >> >> On Thu, Mar 28, 2013 at 1:17 PM, Mohit Chawla >> >> <mohit.chawla.binary@gmail.com> wrote: >> >>> Hello Ken, >> >>> >> >>> Thanks for the response. >> >>> >> >>> On Thu, Mar 28, 2013 at 6:42 PM, Ken Barber <ken@puppetlabs.com> >> wrote: >> >>>> So I have some questions, as the error could mean a number of things: >> >>>> >> >>>> What version of PuppetDB are you running? And what exact version of >> >>>> Java is it using? >> >>> >> >>> puppetdb version is puppetdb-1.0.4-1.el6.noarch. >> >>> >> >>> $ java -version >> >>> java version "1.6.0_24" >> >>> OpenJDK Runtime Environment (IcedTea6 1.11.5) >> (rhel-1.50.1.11.5.el6_3-x86_64) >> >>> OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode) >> >>> >> >>>> >> >>>> Can you take a look at puppetdb.log and tell me if you see any >> >>>> meaningful error messages? >> >>> >> >>> I took a look, no error messages, only info messages. >> >>>> >> >>>> Without trying to compile a catalog in this manner - are you getting >> >>>> any trouble with replace facts/replace catalogs generally? ie. Just >> >>>> trying to running puppet agent -t for example? >> >>> >> >>> Nope, those runs are working fine. >> >>> >> >>>> >> >>>> Have you tried re-initialisation your SSL setup with >> >>>> puppetdb-ssl-setup at all yet? >> >>> >> >>> Yes, and confirmed the validity of certificates and their fingerprints >> >>> in the truststore and master ca. >> >>> >> >>>> >> >>>> Does the hostname you have specified in /etc/puppet/puppetdb.conf to >> >>>> talk to your puppetdb server match the SSL certificate assigned to >> the >> >>>> puppetdb host itself? >> >>> >> >>> Yes. >> >>> >> >>>> >> >>>> On Thu, Mar 28, 2013 at 12:55 PM, Mohit Chawla >> >>>> <mohit.chawla.binary@gmail.com> wrote: >> >>>>> Forgot mentioning the env details: >> >>>>> >> >>>>> [user@puppetmaster ~]# rpm -qa | grep puppet >> >>>>> puppetlabs-release-6-6.noarch >> >>>>> puppetdb-terminus-1.0.4-1.el6.noarch >> >>>>> mcollective-puppet-agent-1.4.1-1.noarch >> >>>>> puppet-2.7.20-1.el6.noarch >> >>>>> hiera-puppet-1.0.0-1.el6.noarch >> >>>>> puppet-server-2.7.20-1.el6.noarch >> >>>>> mcollective-puppet-common-1.4.1-1.noarch >> >>>>> >> >>>>> [user@puppetmaster ~]# cat /etc/centos-release >> >>>>> CentOS release 6.3 (Final) >> >>>>> >> >>>>> On Thu, Mar 28, 2013 at 6:23 PM, Mohit Chawla >> >>>>> <mohit.chawla.binary@gmail.com> wrote: >> >>>>>> Hello, >> >>>>>> >> >>>>>> Stuck in a weird place here. I am trying to do ''puppet master >> >>>>>> --compile foo.com'', however I am not getting the catalog json. So >> far, >> >>>>>> I have noticed two sort of outputs: >> >>>>>> >> >>>>>> 1) The above command results in : >> >>>>>> notice: Compiled catalog for foo.com in environment production in >> 10.60 seconds >> >>>>>> Failed to submit ''replace catalog'' command for foo.com to >> PuppetDB at >> >>>>>> puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 >> read >> >>>>>> finished A >> >>>>>> >> >>>>>> Notice that I don''t actually get any json & of course, the ssl >> error. >> >>>>>> >> >>>>>> 2) Other output: >> >>>>>> Failed when searching for node bar.com: Could not retrieve facts >> for >> >>>>>> bar.com: Failed to find facts from PuppetDB at puppetdb:8081: >> >>>>>> SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A >> >>>>>> >> >>>>>> Disabling storeconfig allows me to get a proper catalog json. >> >>>>>> >> >>>>>> Regarding the ssl errors, we''ve established the correctness of >> >>>>>> certificates in the relevant places - and done tests using curl and >> >>>>>> using ruby net/http. Here''s a gist of for the curl and ruby >> snippet. >> >>>>>> https://gist.github.com/alcy/5262866 >> >>>>>> >> >>>>>> Any suggestions what could be happening here ? Perhaps in >> >>>>>> indirector/facts/puppetdb.rb, http_get doesn''t get the correct ssl >> >>>>>> params or something ? >> >>>>> >> >>>>> -- >> >>>>> You received this message because you are subscribed to the Google >> Groups "Puppet Users" group. >> >>>>> To unsubscribe from this group and stop receiving emails from it, >> send an email to puppet-users+unsubscribe@googlegroups.com. >> >>>>> To post to this group, send email to puppet-users@googlegroups.com. >> >>>>> Visit this group at >> http://groups.google.com/group/puppet-users?hl=en. >> >>>>> For more options, visit https://groups.google.com/groups/opt_out. >> >>>>> >> >>>>> >> >>>> >> >>>> -- >> >>>> You received this message because you are subscribed to the Google >> Groups "Puppet Users" group. >> >>>> To unsubscribe from this group and stop receiving emails from it, >> send an email to puppet-users+unsubscribe@googlegroups.com. >> >>>> To post to this group, send email to puppet-users@googlegroups.com. >> >>>> Visit this group at >> http://groups.google.com/group/puppet-users?hl=en. >> >>>> For more options, visit https://groups.google.com/groups/opt_out. >> >>>> >> >>>> >> >>> >> >>> -- >> >>> You received this message because you are subscribed to the Google >> Groups "Puppet Users" group. >> >>> To unsubscribe from this group and stop receiving emails from it, >> send an email to puppet-users+unsubscribe@googlegroups.com. >> >>> To post to this group, send email to puppet-users@googlegroups.com. >> >>> Visit this group at http://groups.google.com/group/puppet-users?hl=en >> . >> >>> For more options, visit https://groups.google.com/groups/opt_out. >> >>> >> >>> >> >> >> >> -- >> >> You received this message because you are subscribed to the Google >> Groups "Puppet Users" group. >> >> To unsubscribe from this group and stop receiving emails from it, send >> an email to puppet-users+unsubscribe@googlegroups.com. >> >> To post to this group, send email to puppet-users@googlegroups.com. >> >> Visit this group at http://groups.google.com/group/puppet-users?hl=en. >> >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> >> >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups "Puppet Users" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> an email to puppet-users+unsubscribe@googlegroups.com. >> > To post to this group, send email to puppet-users@googlegroups.com. >> > Visit this group at http://groups.google.com/group/puppet-users?hl=en. >> > For more options, visit https://groups.google.com/groups/opt_out. >> > >> > >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to puppet-users+unsubscribe@googlegroups.com. >> To post to this group, send email to puppet-users@googlegroups.com. >> Visit this group at http://groups.google.com/group/puppet-users?hl=en. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Mohit Chawla
2013-Apr-01 08:57 UTC
Re: [Puppet Users] Re: Catalog compilation, puppetdb and ssl
Slightly OT here - shouldn''t setting ca_name be enough for --compile to not fail ? Why does it need the --certname param ? On Mon, Apr 1, 2013 at 2:03 PM, Mohit Chawla <mohit.chawla.binary@gmail.com>wrote:> Hi, > > Got it working by passing --certname param (the ca server hostname, > instead of this master). > > > > On Mon, Apr 1, 2013 at 1:19 PM, Mohit Chawla < > mohit.chawla.binary@gmail.com> wrote: > >> Hi Ken, >> >> Got a trace from running puppetdb-foreground --debug - >> https://gist.github.com/alcy/5283661. Weird that this doesn''t happen >> during standard puppet runs as opposed to doing a --compile. Here''s the >> puppet trace when doing --compile - https://gist.github.com/alcy/5283712. >> >> >> >> >> >> On Fri, Mar 29, 2013 at 3:14 AM, Ken Barber <ken@puppetlabs.com> wrote: >> >>> Yeah, it does seem very odd though ... if agent works - and the master >>> is able to talk to PuppetDB no problem, then its weird that running >>> puppet master on the command line doesn''t seem to work. >>> >>> What is strange is that the SSL error is very very unspecific: >>> >>> Failed to submit ''replace catalog'' command for foo.com to PuppetDB at >>> puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read >>> finished A >>> >>> It doesn''t speak of a particular problem, ie. remote host doesn''t >>> match certificate, CA not verified etc. etc. it just says ''read >>> finished A''. I think this is significant, the last time I''ve seen it >>> is when there was an SSL bug, and the transport is cut short - which >>> is why I wanted to see what you had in your logs. >>> >>> If you try running the command with --debug --trace do you get any >>> more information? If you tcpdump the connection to the puppetdb server >>> do you see the TCP packets hit the server on port 8081? Try running >>> puppetdb-foreground --debug on your puppetdb server and see what >>> happens when you attempt to compile like this. You''d obviously want to >>> try these things when the other activity on the server is disabled >>> :-). >>> >>> ken. >>> >>> On Thu, Mar 28, 2013 at 8:54 PM, Mohit Chawla >>> <mohit.chawla.binary@gmail.com> wrote: >>> > Hi, >>> > >>> > Not at the workstation right now, but regarding puppet.conf I cant >>> > think of any peculiar settings apart from this being one of the two >>> > puppet masters apart from a separate ca server (we took care of having >>> > the ca server''s certs being available at these masters). And afaik >>> > right now, there wasn''t any ~/.puppet dir for root, however I need to >>> > confirm this. >>> > >>> > On Thu, Mar 28, 2013 at 7:07 PM, Ken Barber <ken@puppetlabs.com> >>> wrote: >>> >> I''m just trying to run up the same environment so I can try to >>> >> replicate it, as yet I can''t replicate it on the newer environment. >>> >> >>> >> What does your puppet.conf look like on the host you are trying to run >>> >> puppet master --compile btw? I presume you are trying to run the >>> >> command as root, is there a ~/.puppet directory for that user at all? >>> >> >>> >> ken. >>> >> >>> >> On Thu, Mar 28, 2013 at 1:17 PM, Mohit Chawla >>> >> <mohit.chawla.binary@gmail.com> wrote: >>> >>> Hello Ken, >>> >>> >>> >>> Thanks for the response. >>> >>> >>> >>> On Thu, Mar 28, 2013 at 6:42 PM, Ken Barber <ken@puppetlabs.com> >>> wrote: >>> >>>> So I have some questions, as the error could mean a number of >>> things: >>> >>>> >>> >>>> What version of PuppetDB are you running? And what exact version of >>> >>>> Java is it using? >>> >>> >>> >>> puppetdb version is puppetdb-1.0.4-1.el6.noarch. >>> >>> >>> >>> $ java -version >>> >>> java version "1.6.0_24" >>> >>> OpenJDK Runtime Environment (IcedTea6 1.11.5) >>> (rhel-1.50.1.11.5.el6_3-x86_64) >>> >>> OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode) >>> >>> >>> >>>> >>> >>>> Can you take a look at puppetdb.log and tell me if you see any >>> >>>> meaningful error messages? >>> >>> >>> >>> I took a look, no error messages, only info messages. >>> >>>> >>> >>>> Without trying to compile a catalog in this manner - are you getting >>> >>>> any trouble with replace facts/replace catalogs generally? ie. Just >>> >>>> trying to running puppet agent -t for example? >>> >>> >>> >>> Nope, those runs are working fine. >>> >>> >>> >>>> >>> >>>> Have you tried re-initialisation your SSL setup with >>> >>>> puppetdb-ssl-setup at all yet? >>> >>> >>> >>> Yes, and confirmed the validity of certificates and their >>> fingerprints >>> >>> in the truststore and master ca. >>> >>> >>> >>>> >>> >>>> Does the hostname you have specified in /etc/puppet/puppetdb.conf to >>> >>>> talk to your puppetdb server match the SSL certificate assigned to >>> the >>> >>>> puppetdb host itself? >>> >>> >>> >>> Yes. >>> >>> >>> >>>> >>> >>>> On Thu, Mar 28, 2013 at 12:55 PM, Mohit Chawla >>> >>>> <mohit.chawla.binary@gmail.com> wrote: >>> >>>>> Forgot mentioning the env details: >>> >>>>> >>> >>>>> [user@puppetmaster ~]# rpm -qa | grep puppet >>> >>>>> puppetlabs-release-6-6.noarch >>> >>>>> puppetdb-terminus-1.0.4-1.el6.noarch >>> >>>>> mcollective-puppet-agent-1.4.1-1.noarch >>> >>>>> puppet-2.7.20-1.el6.noarch >>> >>>>> hiera-puppet-1.0.0-1.el6.noarch >>> >>>>> puppet-server-2.7.20-1.el6.noarch >>> >>>>> mcollective-puppet-common-1.4.1-1.noarch >>> >>>>> >>> >>>>> [user@puppetmaster ~]# cat /etc/centos-release >>> >>>>> CentOS release 6.3 (Final) >>> >>>>> >>> >>>>> On Thu, Mar 28, 2013 at 6:23 PM, Mohit Chawla >>> >>>>> <mohit.chawla.binary@gmail.com> wrote: >>> >>>>>> Hello, >>> >>>>>> >>> >>>>>> Stuck in a weird place here. I am trying to do ''puppet master >>> >>>>>> --compile foo.com'', however I am not getting the catalog json. >>> So far, >>> >>>>>> I have noticed two sort of outputs: >>> >>>>>> >>> >>>>>> 1) The above command results in : >>> >>>>>> notice: Compiled catalog for foo.com in environment production >>> in 10.60 seconds >>> >>>>>> Failed to submit ''replace catalog'' command for foo.com to >>> PuppetDB at >>> >>>>>> puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 >>> read >>> >>>>>> finished A >>> >>>>>> >>> >>>>>> Notice that I don''t actually get any json & of course, the ssl >>> error. >>> >>>>>> >>> >>>>>> 2) Other output: >>> >>>>>> Failed when searching for node bar.com: Could not retrieve facts >>> for >>> >>>>>> bar.com: Failed to find facts from PuppetDB at puppetdb:8081: >>> >>>>>> SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A >>> >>>>>> >>> >>>>>> Disabling storeconfig allows me to get a proper catalog json. >>> >>>>>> >>> >>>>>> Regarding the ssl errors, we''ve established the correctness of >>> >>>>>> certificates in the relevant places - and done tests using curl >>> and >>> >>>>>> using ruby net/http. Here''s a gist of for the curl and ruby >>> snippet. >>> >>>>>> https://gist.github.com/alcy/5262866 >>> >>>>>> >>> >>>>>> Any suggestions what could be happening here ? Perhaps in >>> >>>>>> indirector/facts/puppetdb.rb, http_get doesn''t get the correct >>> ssl >>> >>>>>> params or something ? >>> >>>>> >>> >>>>> -- >>> >>>>> You received this message because you are subscribed to the Google >>> Groups "Puppet Users" group. >>> >>>>> To unsubscribe from this group and stop receiving emails from it, >>> send an email to puppet-users+unsubscribe@googlegroups.com. >>> >>>>> To post to this group, send email to puppet-users@googlegroups.com >>> . >>> >>>>> Visit this group at >>> http://groups.google.com/group/puppet-users?hl=en. >>> >>>>> For more options, visit https://groups.google.com/groups/opt_out. >>> >>>>> >>> >>>>> >>> >>>> >>> >>>> -- >>> >>>> You received this message because you are subscribed to the Google >>> Groups "Puppet Users" group. >>> >>>> To unsubscribe from this group and stop receiving emails from it, >>> send an email to puppet-users+unsubscribe@googlegroups.com. >>> >>>> To post to this group, send email to puppet-users@googlegroups.com. >>> >>>> Visit this group at >>> http://groups.google.com/group/puppet-users?hl=en. >>> >>>> For more options, visit https://groups.google.com/groups/opt_out. >>> >>>> >>> >>>> >>> >>> >>> >>> -- >>> >>> You received this message because you are subscribed to the Google >>> Groups "Puppet Users" group. >>> >>> To unsubscribe from this group and stop receiving emails from it, >>> send an email to puppet-users+unsubscribe@googlegroups.com. >>> >>> To post to this group, send email to puppet-users@googlegroups.com. >>> >>> Visit this group at >>> http://groups.google.com/group/puppet-users?hl=en. >>> >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >>> >>> >>> >>> >> >>> >> -- >>> >> You received this message because you are subscribed to the Google >>> Groups "Puppet Users" group. >>> >> To unsubscribe from this group and stop receiving emails from it, >>> send an email to puppet-users+unsubscribe@googlegroups.com. >>> >> To post to this group, send email to puppet-users@googlegroups.com. >>> >> Visit this group at http://groups.google.com/group/puppet-users?hl=en >>> . >>> >> For more options, visit https://groups.google.com/groups/opt_out. >>> >> >>> >> >>> > >>> > -- >>> > You received this message because you are subscribed to the Google >>> Groups "Puppet Users" group. >>> > To unsubscribe from this group and stop receiving emails from it, send >>> an email to puppet-users+unsubscribe@googlegroups.com. >>> > To post to this group, send email to puppet-users@googlegroups.com. >>> > Visit this group at http://groups.google.com/group/puppet-users?hl=en. >>> > For more options, visit https://groups.google.com/groups/opt_out. >>> > >>> > >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Puppet Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to puppet-users+unsubscribe@googlegroups.com. >>> To post to this group, send email to puppet-users@googlegroups.com. >>> Visit this group at http://groups.google.com/group/puppet-users?hl=en. >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >>> >>> >> >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Ken Barber
2013-Apr-02 10:10 UTC
Re: [Puppet Users] Re: Catalog compilation, puppetdb and ssl
If you specify ''certname'' it will use the local certificate with that name as apposed to using the certificate with the same name as the boxes fqdn (ie. facter fqdn). For example, if I use ''puppet master --compile puppetdb1.vm'' on its own, it uses my local fqdn (determined with `facter fqdn`) as an indicator of where the certificate is: /var/lib/puppet/ssl/certs/puppetdb1.vm.pem. If I specify --certname foo it uses /var/lib/puppet/ssl/certs/foo.pem. I presume that the certificate that matches the fqdn on the host you are testing is probably _not_ signed by the CA that is loaded into PuppetDB. Its just a guess, but we should confirm this. Lets take a step back a bit and do an exercise. On your PuppetDB node, in /etc/puppetdb/ssl/ you have three files: * truststore.jks * keystore.jks * puppetdb_keystore_pw.txt If you look inside truststore.jks, you''ll see that it contains the CA certificate: # keytool -list -keystore /etc/puppetdb/ssl/truststore.jks Enter keystore password: (this is contained inside puppetdb_keystore_pw.txt) Keystore type: JKS Keystore provider: SUN Your keystore contains 1 entry puppetdb ca, 28-Mar-2013, trustedCertEntry, Certificate fingerprint (SHA1): B1:5E:8C:2C:BE:BB:97:5C:09:42:34:7A:F2:02:3F:7B:06:C6:A2:C1 This CA certificate is the same as what matches my CA file here, we can test this by matching the fingerprint: # openssl x509 -noout -in /var/lib/puppet/ssl/ca/ca_crt.pem -fingerprint SHA1 Fingerprint=B1:5E:8C:2C:BE:BB:97:5C:09:42:34:7A:F2:02:3F:7B:06:C6:A2:C1 For client validation, this is really what matters - the client certificate must be signed by this CA certificate to be accepted by PuppetDB. It doesn''t matter what the client thinks its CA is in Puppet configuration or whatever (so setting --ca_name is not enough here), all that matters is the client certificate must have been signed by that CA, and it is verified using OpenSSL techniques on the PuppetDB server during SSL negotiation. You can do this manually for example. Lets confirm this on my master - we know that my local CA is confirmed as the thing loaded into the PuppetDB trust store, so now we just run the verify routine against my certificate: # openssl verify -CAfile /var/lib/puppet/ssl/ca/ca_crt.pem `puppet master --configprint hostcert` /var/lib/puppet/ssl/certs/puppetdb1.vm.pem: OK You must be sure to test this host certificate against the exact same CA loaded into PuppetDB ... not just the local CA (in case they differ - you can check this with the -fingerprint command). If the certificate on the other hand is NOT signed by the CA loaded into PuppetDB you get something like: # openssl verify -CAfile /var/lib/puppet/ssl/ca/ca_crt.pem /var/lib/puppet/ssl/certs/puppetdb2.vm.pem /var/lib/puppet/ssl/certs/puppetdb2.vm.pem: CN = puppetdb2.vm error 20 at 0 depth lookup:unable to get local issuer certificate Which is by design for this example - that certificate ''puppetdb2.vm'' was a certificate I copied from another Puppetmaster which has a separate CA, and thus it fails verification. In the real world, puppetdb2.vm would get reject by PuppetDB - which is the correct behaviour. ken. On Mon, Apr 1, 2013 at 9:57 AM, Mohit Chawla <mohit.chawla.binary@gmail.com> wrote:> Slightly OT here - shouldn''t setting ca_name be enough for --compile to not > fail ? Why does it need the --certname param ? > > > On Mon, Apr 1, 2013 at 2:03 PM, Mohit Chawla <mohit.chawla.binary@gmail.com> > wrote: >> >> Hi, >> >> Got it working by passing --certname param (the ca server hostname, >> instead of this master). >> >> >> >> On Mon, Apr 1, 2013 at 1:19 PM, Mohit Chawla >> <mohit.chawla.binary@gmail.com> wrote: >>> >>> Hi Ken, >>> >>> Got a trace from running puppetdb-foreground --debug - >>> https://gist.github.com/alcy/5283661. Weird that this doesn''t happen during >>> standard puppet runs as opposed to doing a --compile. Here''s the puppet >>> trace when doing --compile - https://gist.github.com/alcy/5283712. >>> >>> >>> >>> >>> On Fri, Mar 29, 2013 at 3:14 AM, Ken Barber <ken@puppetlabs.com> wrote: >>>> >>>> Yeah, it does seem very odd though ... if agent works - and the master >>>> is able to talk to PuppetDB no problem, then its weird that running >>>> puppet master on the command line doesn''t seem to work. >>>> >>>> What is strange is that the SSL error is very very unspecific: >>>> >>>> Failed to submit ''replace catalog'' command for foo.com to PuppetDB at >>>> puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read >>>> finished A >>>> >>>> It doesn''t speak of a particular problem, ie. remote host doesn''t >>>> match certificate, CA not verified etc. etc. it just says ''read >>>> finished A''. I think this is significant, the last time I''ve seen it >>>> is when there was an SSL bug, and the transport is cut short - which >>>> is why I wanted to see what you had in your logs. >>>> >>>> If you try running the command with --debug --trace do you get any >>>> more information? If you tcpdump the connection to the puppetdb server >>>> do you see the TCP packets hit the server on port 8081? Try running >>>> puppetdb-foreground --debug on your puppetdb server and see what >>>> happens when you attempt to compile like this. You''d obviously want to >>>> try these things when the other activity on the server is disabled >>>> :-). >>>> >>>> ken. >>>> >>>> On Thu, Mar 28, 2013 at 8:54 PM, Mohit Chawla >>>> <mohit.chawla.binary@gmail.com> wrote: >>>> > Hi, >>>> > >>>> > Not at the workstation right now, but regarding puppet.conf I cant >>>> > think of any peculiar settings apart from this being one of the two >>>> > puppet masters apart from a separate ca server (we took care of having >>>> > the ca server''s certs being available at these masters). And afaik >>>> > right now, there wasn''t any ~/.puppet dir for root, however I need to >>>> > confirm this. >>>> > >>>> > On Thu, Mar 28, 2013 at 7:07 PM, Ken Barber <ken@puppetlabs.com> >>>> > wrote: >>>> >> I''m just trying to run up the same environment so I can try to >>>> >> replicate it, as yet I can''t replicate it on the newer environment. >>>> >> >>>> >> What does your puppet.conf look like on the host you are trying to >>>> >> run >>>> >> puppet master --compile btw? I presume you are trying to run the >>>> >> command as root, is there a ~/.puppet directory for that user at all? >>>> >> >>>> >> ken. >>>> >> >>>> >> On Thu, Mar 28, 2013 at 1:17 PM, Mohit Chawla >>>> >> <mohit.chawla.binary@gmail.com> wrote: >>>> >>> Hello Ken, >>>> >>> >>>> >>> Thanks for the response. >>>> >>> >>>> >>> On Thu, Mar 28, 2013 at 6:42 PM, Ken Barber <ken@puppetlabs.com> >>>> >>> wrote: >>>> >>>> So I have some questions, as the error could mean a number of >>>> >>>> things: >>>> >>>> >>>> >>>> What version of PuppetDB are you running? And what exact version of >>>> >>>> Java is it using? >>>> >>> >>>> >>> puppetdb version is puppetdb-1.0.4-1.el6.noarch. >>>> >>> >>>> >>> $ java -version >>>> >>> java version "1.6.0_24" >>>> >>> OpenJDK Runtime Environment (IcedTea6 1.11.5) >>>> >>> (rhel-1.50.1.11.5.el6_3-x86_64) >>>> >>> OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode) >>>> >>> >>>> >>>> >>>> >>>> Can you take a look at puppetdb.log and tell me if you see any >>>> >>>> meaningful error messages? >>>> >>> >>>> >>> I took a look, no error messages, only info messages. >>>> >>>> >>>> >>>> Without trying to compile a catalog in this manner - are you >>>> >>>> getting >>>> >>>> any trouble with replace facts/replace catalogs generally? ie. Just >>>> >>>> trying to running puppet agent -t for example? >>>> >>> >>>> >>> Nope, those runs are working fine. >>>> >>> >>>> >>>> >>>> >>>> Have you tried re-initialisation your SSL setup with >>>> >>>> puppetdb-ssl-setup at all yet? >>>> >>> >>>> >>> Yes, and confirmed the validity of certificates and their >>>> >>> fingerprints >>>> >>> in the truststore and master ca. >>>> >>> >>>> >>>> >>>> >>>> Does the hostname you have specified in /etc/puppet/puppetdb.conf >>>> >>>> to >>>> >>>> talk to your puppetdb server match the SSL certificate assigned to >>>> >>>> the >>>> >>>> puppetdb host itself? >>>> >>> >>>> >>> Yes. >>>> >>> >>>> >>>> >>>> >>>> On Thu, Mar 28, 2013 at 12:55 PM, Mohit Chawla >>>> >>>> <mohit.chawla.binary@gmail.com> wrote: >>>> >>>>> Forgot mentioning the env details: >>>> >>>>> >>>> >>>>> [user@puppetmaster ~]# rpm -qa | grep puppet >>>> >>>>> puppetlabs-release-6-6.noarch >>>> >>>>> puppetdb-terminus-1.0.4-1.el6.noarch >>>> >>>>> mcollective-puppet-agent-1.4.1-1.noarch >>>> >>>>> puppet-2.7.20-1.el6.noarch >>>> >>>>> hiera-puppet-1.0.0-1.el6.noarch >>>> >>>>> puppet-server-2.7.20-1.el6.noarch >>>> >>>>> mcollective-puppet-common-1.4.1-1.noarch >>>> >>>>> >>>> >>>>> [user@puppetmaster ~]# cat /etc/centos-release >>>> >>>>> CentOS release 6.3 (Final) >>>> >>>>> >>>> >>>>> On Thu, Mar 28, 2013 at 6:23 PM, Mohit Chawla >>>> >>>>> <mohit.chawla.binary@gmail.com> wrote: >>>> >>>>>> Hello, >>>> >>>>>> >>>> >>>>>> Stuck in a weird place here. I am trying to do ''puppet master >>>> >>>>>> --compile foo.com'', however I am not getting the catalog json. So >>>> >>>>>> far, >>>> >>>>>> I have noticed two sort of outputs: >>>> >>>>>> >>>> >>>>>> 1) The above command results in : >>>> >>>>>> notice: Compiled catalog for foo.com in environment production in >>>> >>>>>> 10.60 seconds >>>> >>>>>> Failed to submit ''replace catalog'' command for foo.com to >>>> >>>>>> PuppetDB at >>>> >>>>>> puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 >>>> >>>>>> read >>>> >>>>>> finished A >>>> >>>>>> >>>> >>>>>> Notice that I don''t actually get any json & of course, the ssl >>>> >>>>>> error. >>>> >>>>>> >>>> >>>>>> 2) Other output: >>>> >>>>>> Failed when searching for node bar.com: Could not retrieve facts >>>> >>>>>> for >>>> >>>>>> bar.com: Failed to find facts from PuppetDB at puppetdb:8081: >>>> >>>>>> SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished >>>> >>>>>> A >>>> >>>>>> >>>> >>>>>> Disabling storeconfig allows me to get a proper catalog json. >>>> >>>>>> >>>> >>>>>> Regarding the ssl errors, we''ve established the correctness of >>>> >>>>>> certificates in the relevant places - and done tests using curl >>>> >>>>>> and >>>> >>>>>> using ruby net/http. Here''s a gist of for the curl and ruby >>>> >>>>>> snippet. >>>> >>>>>> https://gist.github.com/alcy/5262866 >>>> >>>>>> >>>> >>>>>> Any suggestions what could be happening here ? Perhaps in >>>> >>>>>> indirector/facts/puppetdb.rb, http_get doesn''t get the correct >>>> >>>>>> ssl >>>> >>>>>> params or something ? >>>> >>>>> >>>> >>>>> -- >>>> >>>>> You received this message because you are subscribed to the Google >>>> >>>>> Groups "Puppet Users" group. >>>> >>>>> To unsubscribe from this group and stop receiving emails from it, >>>> >>>>> send an email to puppet-users+unsubscribe@googlegroups.com. >>>> >>>>> To post to this group, send email to >>>> >>>>> puppet-users@googlegroups.com. >>>> >>>>> Visit this group at >>>> >>>>> http://groups.google.com/group/puppet-users?hl=en. >>>> >>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>> >>>>> >>>> >>>>> >>>> >>>> >>>> >>>> -- >>>> >>>> You received this message because you are subscribed to the Google >>>> >>>> Groups "Puppet Users" group. >>>> >>>> To unsubscribe from this group and stop receiving emails from it, >>>> >>>> send an email to puppet-users+unsubscribe@googlegroups.com. >>>> >>>> To post to this group, send email to puppet-users@googlegroups.com. >>>> >>>> Visit this group at >>>> >>>> http://groups.google.com/group/puppet-users?hl=en. >>>> >>>> For more options, visit https://groups.google.com/groups/opt_out. >>>> >>>> >>>> >>>> >>>> >>> >>>> >>> -- >>>> >>> You received this message because you are subscribed to the Google >>>> >>> Groups "Puppet Users" group. >>>> >>> To unsubscribe from this group and stop receiving emails from it, >>>> >>> send an email to puppet-users+unsubscribe@googlegroups.com. >>>> >>> To post to this group, send email to puppet-users@googlegroups.com. >>>> >>> Visit this group at >>>> >>> http://groups.google.com/group/puppet-users?hl=en. >>>> >>> For more options, visit https://groups.google.com/groups/opt_out. >>>> >>> >>>> >>> >>>> >> >>>> >> -- >>>> >> You received this message because you are subscribed to the Google >>>> >> Groups "Puppet Users" group. >>>> >> To unsubscribe from this group and stop receiving emails from it, >>>> >> send an email to puppet-users+unsubscribe@googlegroups.com. >>>> >> To post to this group, send email to puppet-users@googlegroups.com. >>>> >> Visit this group at >>>> >> http://groups.google.com/group/puppet-users?hl=en. >>>> >> For more options, visit https://groups.google.com/groups/opt_out. >>>> >> >>>> >> >>>> > >>>> > -- >>>> > You received this message because you are subscribed to the Google >>>> > Groups "Puppet Users" group. >>>> > To unsubscribe from this group and stop receiving emails from it, send >>>> > an email to puppet-users+unsubscribe@googlegroups.com. >>>> > To post to this group, send email to puppet-users@googlegroups.com. >>>> > Visit this group at http://groups.google.com/group/puppet-users?hl=en. >>>> > For more options, visit https://groups.google.com/groups/opt_out. >>>> > >>>> > >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Puppet Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to puppet-users+unsubscribe@googlegroups.com. >>>> To post to this group, send email to puppet-users@googlegroups.com. >>>> Visit this group at http://groups.google.com/group/puppet-users?hl=en. >>>> For more options, visit https://groups.google.com/groups/opt_out. >>>> >>>> >>> >> > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Mohit Chawla
2013-Apr-03 06:41 UTC
Re: [Puppet Users] Re: Catalog compilation, puppetdb and ssl
Hey Ken, that definitely cleared some misconceptions, thanks ! I now know what the problem was. First, I assumed that the client must''ve the same certificate as in puppetdb''s truststore. I didn''t realize that any cert signed by that CA should be able to make calls. In our scenario, where we have separate certs for the agent and master component on the same server (under /etc/puppet/ssl & /var/lib/puppet/ssl respectively), invoking --compile without giving the certname meant no certificate at all ! And so, passing --ssldir to use the agent''s certs (signed by the same CA) worked. On Tue, Apr 2, 2013 at 3:40 PM, Ken Barber <ken@puppetlabs.com> wrote:> If you specify ''certname'' it will use the local certificate with that > name as apposed to using the certificate with the same name as the > boxes fqdn (ie. facter fqdn). > > For example, if I use ''puppet master --compile puppetdb1.vm'' on its > own, it uses my local fqdn (determined with `facter fqdn`) as an > indicator of where the certificate is: > /var/lib/puppet/ssl/certs/puppetdb1.vm.pem. If I specify --certname > foo it uses /var/lib/puppet/ssl/certs/foo.pem. > > I presume that the certificate that matches the fqdn on the host you > are testing is probably _not_ signed by the CA that is loaded into > PuppetDB. Its just a guess, but we should confirm this. > > Lets take a step back a bit and do an exercise. On your PuppetDB node, > in /etc/puppetdb/ssl/ you have three files: > > * truststore.jks > * keystore.jks > * puppetdb_keystore_pw.txt > > If you look inside truststore.jks, you''ll see that it contains the CA > certificate: > > # keytool -list -keystore /etc/puppetdb/ssl/truststore.jks > Enter keystore password: (this is contained inside > puppetdb_keystore_pw.txt) > > Keystore type: JKS > Keystore provider: SUN > > Your keystore contains 1 entry > > puppetdb ca, 28-Mar-2013, trustedCertEntry, > Certificate fingerprint (SHA1): > B1:5E:8C:2C:BE:BB:97:5C:09:42:34:7A:F2:02:3F:7B:06:C6:A2:C1 > > This CA certificate is the same as what matches my CA file here, we > can test this by matching the fingerprint: > > # openssl x509 -noout -in /var/lib/puppet/ssl/ca/ca_crt.pem -fingerprint > SHA1 > Fingerprint=B1:5E:8C:2C:BE:BB:97:5C:09:42:34:7A:F2:02:3F:7B:06:C6:A2:C1 > > For client validation, this is really what matters - the client > certificate must be signed by this CA certificate to be accepted by > PuppetDB. It doesn''t matter what the client thinks its CA is in Puppet > configuration or whatever (so setting --ca_name is not enough here), > all that matters is the client certificate must have been signed by > that CA, and it is verified using OpenSSL techniques on the PuppetDB > server during SSL negotiation. > > You can do this manually for example. Lets confirm this on my master - > we know that my local CA is confirmed as the thing loaded into the > PuppetDB trust store, so now we just run the verify routine against my > certificate: > > # openssl verify -CAfile /var/lib/puppet/ssl/ca/ca_crt.pem `puppet > master --configprint hostcert` > /var/lib/puppet/ssl/certs/puppetdb1.vm.pem: OK > > You must be sure to test this host certificate against the exact same > CA loaded into PuppetDB ... not just the local CA (in case they differ > - you can check this with the -fingerprint command). > > If the certificate on the other hand is NOT signed by the CA loaded > into PuppetDB you get something like: > > # openssl verify -CAfile /var/lib/puppet/ssl/ca/ca_crt.pem > /var/lib/puppet/ssl/certs/puppetdb2.vm.pem > /var/lib/puppet/ssl/certs/puppetdb2.vm.pem: CN = puppetdb2.vm > error 20 at 0 depth lookup:unable to get local issuer certificate > > Which is by design for this example - that certificate ''puppetdb2.vm'' > was a certificate I copied from another Puppetmaster which has a > separate CA, and thus it fails verification. In the real world, > puppetdb2.vm would get reject by PuppetDB - which is the correct > behaviour. > > ken. > > On Mon, Apr 1, 2013 at 9:57 AM, Mohit Chawla > <mohit.chawla.binary@gmail.com> wrote: > > Slightly OT here - shouldn''t setting ca_name be enough for --compile to > not > > fail ? Why does it need the --certname param ? > > > > > > On Mon, Apr 1, 2013 at 2:03 PM, Mohit Chawla < > mohit.chawla.binary@gmail.com> > > wrote: > >> > >> Hi, > >> > >> Got it working by passing --certname param (the ca server hostname, > >> instead of this master). > >> > >> > >> > >> On Mon, Apr 1, 2013 at 1:19 PM, Mohit Chawla > >> <mohit.chawla.binary@gmail.com> wrote: > >>> > >>> Hi Ken, > >>> > >>> Got a trace from running puppetdb-foreground --debug - > >>> https://gist.github.com/alcy/5283661. Weird that this doesn''t happen > during > >>> standard puppet runs as opposed to doing a --compile. Here''s the puppet > >>> trace when doing --compile - https://gist.github.com/alcy/5283712. > >>> > >>> > >>> > >>> > >>> On Fri, Mar 29, 2013 at 3:14 AM, Ken Barber <ken@puppetlabs.com> > wrote: > >>>> > >>>> Yeah, it does seem very odd though ... if agent works - and the master > >>>> is able to talk to PuppetDB no problem, then its weird that running > >>>> puppet master on the command line doesn''t seem to work. > >>>> > >>>> What is strange is that the SSL error is very very unspecific: > >>>> > >>>> Failed to submit ''replace catalog'' command for foo.com to PuppetDB at > >>>> puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read > >>>> finished A > >>>> > >>>> It doesn''t speak of a particular problem, ie. remote host doesn''t > >>>> match certificate, CA not verified etc. etc. it just says ''read > >>>> finished A''. I think this is significant, the last time I''ve seen it > >>>> is when there was an SSL bug, and the transport is cut short - which > >>>> is why I wanted to see what you had in your logs. > >>>> > >>>> If you try running the command with --debug --trace do you get any > >>>> more information? If you tcpdump the connection to the puppetdb server > >>>> do you see the TCP packets hit the server on port 8081? Try running > >>>> puppetdb-foreground --debug on your puppetdb server and see what > >>>> happens when you attempt to compile like this. You''d obviously want to > >>>> try these things when the other activity on the server is disabled > >>>> :-). > >>>> > >>>> ken. > >>>> > >>>> On Thu, Mar 28, 2013 at 8:54 PM, Mohit Chawla > >>>> <mohit.chawla.binary@gmail.com> wrote: > >>>> > Hi, > >>>> > > >>>> > Not at the workstation right now, but regarding puppet.conf I cant > >>>> > think of any peculiar settings apart from this being one of the two > >>>> > puppet masters apart from a separate ca server (we took care of > having > >>>> > the ca server''s certs being available at these masters). And afaik > >>>> > right now, there wasn''t any ~/.puppet dir for root, however I need > to > >>>> > confirm this. > >>>> > > >>>> > On Thu, Mar 28, 2013 at 7:07 PM, Ken Barber <ken@puppetlabs.com> > >>>> > wrote: > >>>> >> I''m just trying to run up the same environment so I can try to > >>>> >> replicate it, as yet I can''t replicate it on the newer environment. > >>>> >> > >>>> >> What does your puppet.conf look like on the host you are trying to > >>>> >> run > >>>> >> puppet master --compile btw? I presume you are trying to run the > >>>> >> command as root, is there a ~/.puppet directory for that user at > all? > >>>> >> > >>>> >> ken. > >>>> >> > >>>> >> On Thu, Mar 28, 2013 at 1:17 PM, Mohit Chawla > >>>> >> <mohit.chawla.binary@gmail.com> wrote: > >>>> >>> Hello Ken, > >>>> >>> > >>>> >>> Thanks for the response. > >>>> >>> > >>>> >>> On Thu, Mar 28, 2013 at 6:42 PM, Ken Barber <ken@puppetlabs.com> > >>>> >>> wrote: > >>>> >>>> So I have some questions, as the error could mean a number of > >>>> >>>> things: > >>>> >>>> > >>>> >>>> What version of PuppetDB are you running? And what exact version > of > >>>> >>>> Java is it using? > >>>> >>> > >>>> >>> puppetdb version is puppetdb-1.0.4-1.el6.noarch. > >>>> >>> > >>>> >>> $ java -version > >>>> >>> java version "1.6.0_24" > >>>> >>> OpenJDK Runtime Environment (IcedTea6 1.11.5) > >>>> >>> (rhel-1.50.1.11.5.el6_3-x86_64) > >>>> >>> OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode) > >>>> >>> > >>>> >>>> > >>>> >>>> Can you take a look at puppetdb.log and tell me if you see any > >>>> >>>> meaningful error messages? > >>>> >>> > >>>> >>> I took a look, no error messages, only info messages. > >>>> >>>> > >>>> >>>> Without trying to compile a catalog in this manner - are you > >>>> >>>> getting > >>>> >>>> any trouble with replace facts/replace catalogs generally? ie. > Just > >>>> >>>> trying to running puppet agent -t for example? > >>>> >>> > >>>> >>> Nope, those runs are working fine. > >>>> >>> > >>>> >>>> > >>>> >>>> Have you tried re-initialisation your SSL setup with > >>>> >>>> puppetdb-ssl-setup at all yet? > >>>> >>> > >>>> >>> Yes, and confirmed the validity of certificates and their > >>>> >>> fingerprints > >>>> >>> in the truststore and master ca. > >>>> >>> > >>>> >>>> > >>>> >>>> Does the hostname you have specified in /etc/puppet/puppetdb.conf > >>>> >>>> to > >>>> >>>> talk to your puppetdb server match the SSL certificate assigned > to > >>>> >>>> the > >>>> >>>> puppetdb host itself? > >>>> >>> > >>>> >>> Yes. > >>>> >>> > >>>> >>>> > >>>> >>>> On Thu, Mar 28, 2013 at 12:55 PM, Mohit Chawla > >>>> >>>> <mohit.chawla.binary@gmail.com> wrote: > >>>> >>>>> Forgot mentioning the env details: > >>>> >>>>> > >>>> >>>>> [user@puppetmaster ~]# rpm -qa | grep puppet > >>>> >>>>> puppetlabs-release-6-6.noarch > >>>> >>>>> puppetdb-terminus-1.0.4-1.el6.noarch > >>>> >>>>> mcollective-puppet-agent-1.4.1-1.noarch > >>>> >>>>> puppet-2.7.20-1.el6.noarch > >>>> >>>>> hiera-puppet-1.0.0-1.el6.noarch > >>>> >>>>> puppet-server-2.7.20-1.el6.noarch > >>>> >>>>> mcollective-puppet-common-1.4.1-1.noarch > >>>> >>>>> > >>>> >>>>> [user@puppetmaster ~]# cat /etc/centos-release > >>>> >>>>> CentOS release 6.3 (Final) > >>>> >>>>> > >>>> >>>>> On Thu, Mar 28, 2013 at 6:23 PM, Mohit Chawla > >>>> >>>>> <mohit.chawla.binary@gmail.com> wrote: > >>>> >>>>>> Hello, > >>>> >>>>>> > >>>> >>>>>> Stuck in a weird place here. I am trying to do ''puppet master > >>>> >>>>>> --compile foo.com'', however I am not getting the catalog > json. So > >>>> >>>>>> far, > >>>> >>>>>> I have noticed two sort of outputs: > >>>> >>>>>> > >>>> >>>>>> 1) The above command results in : > >>>> >>>>>> notice: Compiled catalog for foo.com in environment > production in > >>>> >>>>>> 10.60 seconds > >>>> >>>>>> Failed to submit ''replace catalog'' command for foo.com to > >>>> >>>>>> PuppetDB at > >>>> >>>>>> puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 > state=SSLv3 > >>>> >>>>>> read > >>>> >>>>>> finished A > >>>> >>>>>> > >>>> >>>>>> Notice that I don''t actually get any json & of course, the ssl > >>>> >>>>>> error. > >>>> >>>>>> > >>>> >>>>>> 2) Other output: > >>>> >>>>>> Failed when searching for node bar.com: Could not retrieve > facts > >>>> >>>>>> for > >>>> >>>>>> bar.com: Failed to find facts from PuppetDB at puppetdb:8081: > >>>> >>>>>> SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read > finished > >>>> >>>>>> A > >>>> >>>>>> > >>>> >>>>>> Disabling storeconfig allows me to get a proper catalog json. > >>>> >>>>>> > >>>> >>>>>> Regarding the ssl errors, we''ve established the correctness of > >>>> >>>>>> certificates in the relevant places - and done tests using curl > >>>> >>>>>> and > >>>> >>>>>> using ruby net/http. Here''s a gist of for the curl and ruby > >>>> >>>>>> snippet. > >>>> >>>>>> https://gist.github.com/alcy/5262866 > >>>> >>>>>> > >>>> >>>>>> Any suggestions what could be happening here ? Perhaps in > >>>> >>>>>> indirector/facts/puppetdb.rb, http_get doesn''t get the correct > >>>> >>>>>> ssl > >>>> >>>>>> params or something ? > >>>> >>>>> > >>>> >>>>> -- > >>>> >>>>> You received this message because you are subscribed to the > Google > >>>> >>>>> Groups "Puppet Users" group. > >>>> >>>>> To unsubscribe from this group and stop receiving emails from > it, > >>>> >>>>> send an email to puppet-users+unsubscribe@googlegroups.com. > >>>> >>>>> To post to this group, send email to > >>>> >>>>> puppet-users@googlegroups.com. > >>>> >>>>> Visit this group at > >>>> >>>>> http://groups.google.com/group/puppet-users?hl=en. > >>>> >>>>> For more options, visit > https://groups.google.com/groups/opt_out. > >>>> >>>>> > >>>> >>>>> > >>>> >>>> > >>>> >>>> -- > >>>> >>>> You received this message because you are subscribed to the > Google > >>>> >>>> Groups "Puppet Users" group. > >>>> >>>> To unsubscribe from this group and stop receiving emails from it, > >>>> >>>> send an email to puppet-users+unsubscribe@googlegroups.com. > >>>> >>>> To post to this group, send email to > puppet-users@googlegroups.com. > >>>> >>>> Visit this group at > >>>> >>>> http://groups.google.com/group/puppet-users?hl=en. > >>>> >>>> For more options, visit https://groups.google.com/groups/opt_out > . > >>>> >>>> > >>>> >>>> > >>>> >>> > >>>> >>> -- > >>>> >>> You received this message because you are subscribed to the Google > >>>> >>> Groups "Puppet Users" group. > >>>> >>> To unsubscribe from this group and stop receiving emails from it, > >>>> >>> send an email to puppet-users+unsubscribe@googlegroups.com. > >>>> >>> To post to this group, send email to > puppet-users@googlegroups.com. > >>>> >>> Visit this group at > >>>> >>> http://groups.google.com/group/puppet-users?hl=en. > >>>> >>> For more options, visit https://groups.google.com/groups/opt_out. > >>>> >>> > >>>> >>> > >>>> >> > >>>> >> -- > >>>> >> You received this message because you are subscribed to the Google > >>>> >> Groups "Puppet Users" group. > >>>> >> To unsubscribe from this group and stop receiving emails from it, > >>>> >> send an email to puppet-users+unsubscribe@googlegroups.com. > >>>> >> To post to this group, send email to puppet-users@googlegroups.com > . > >>>> >> Visit this group at > >>>> >> http://groups.google.com/group/puppet-users?hl=en. > >>>> >> For more options, visit https://groups.google.com/groups/opt_out. > >>>> >> > >>>> >> > >>>> > > >>>> > -- > >>>> > You received this message because you are subscribed to the Google > >>>> > Groups "Puppet Users" group. > >>>> > To unsubscribe from this group and stop receiving emails from it, > send > >>>> > an email to puppet-users+unsubscribe@googlegroups.com. > >>>> > To post to this group, send email to puppet-users@googlegroups.com. > >>>> > Visit this group at > http://groups.google.com/group/puppet-users?hl=en. > >>>> > For more options, visit https://groups.google.com/groups/opt_out. > >>>> > > >>>> > > >>>> > >>>> -- > >>>> You received this message because you are subscribed to the Google > >>>> Groups "Puppet Users" group. > >>>> To unsubscribe from this group and stop receiving emails from it, send > >>>> an email to puppet-users+unsubscribe@googlegroups.com. > >>>> To post to this group, send email to puppet-users@googlegroups.com. > >>>> Visit this group at http://groups.google.com/group/puppet-users?hl=en > . > >>>> For more options, visit https://groups.google.com/groups/opt_out. > >>>> > >>>> > >>> > >> > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Puppet Users" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to puppet-users+unsubscribe@googlegroups.com. > > To post to this group, send email to puppet-users@googlegroups.com. > > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > > For more options, visit https://groups.google.com/groups/opt_out. > > > > > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Ken Barber
2013-Apr-03 09:43 UTC
Re: [Puppet Users] Re: Catalog compilation, puppetdb and ssl
Yes, in this way it behaves a lot like the puppet masters. Don''t forget to check out certificate-whitelist (http://docs.puppetlabs.com/puppetdb/1.1/configure.html#certificate-whitelist) this gives you the ability to only allow certain hosts to connect if you desire it. On Wed, Apr 3, 2013 at 7:41 AM, Mohit Chawla <mohit.chawla.binary@gmail.com> wrote:> Hey Ken, that definitely cleared some misconceptions, thanks ! > > I now know what the problem was. First, I assumed that the client must''ve > the same certificate as in puppetdb''s truststore. I didn''t realize that any > cert signed by that CA should be able to make calls. In our scenario, where > we have separate certs for the agent and master component on the same server > (under /etc/puppet/ssl & /var/lib/puppet/ssl respectively), invoking > --compile without giving the certname meant no certificate at all ! And so, > passing --ssldir to use the agent''s certs (signed by the same CA) worked. > > On Tue, Apr 2, 2013 at 3:40 PM, Ken Barber <ken@puppetlabs.com> wrote: >> >> If you specify ''certname'' it will use the local certificate with that >> name as apposed to using the certificate with the same name as the >> boxes fqdn (ie. facter fqdn). >> >> For example, if I use ''puppet master --compile puppetdb1.vm'' on its >> own, it uses my local fqdn (determined with `facter fqdn`) as an >> indicator of where the certificate is: >> /var/lib/puppet/ssl/certs/puppetdb1.vm.pem. If I specify --certname >> foo it uses /var/lib/puppet/ssl/certs/foo.pem. >> >> I presume that the certificate that matches the fqdn on the host you >> are testing is probably _not_ signed by the CA that is loaded into >> PuppetDB. Its just a guess, but we should confirm this. >> >> Lets take a step back a bit and do an exercise. On your PuppetDB node, >> in /etc/puppetdb/ssl/ you have three files: >> >> * truststore.jks >> * keystore.jks >> * puppetdb_keystore_pw.txt >> >> If you look inside truststore.jks, you''ll see that it contains the CA >> certificate: >> >> # keytool -list -keystore /etc/puppetdb/ssl/truststore.jks >> Enter keystore password: (this is contained inside >> puppetdb_keystore_pw.txt) >> >> Keystore type: JKS >> Keystore provider: SUN >> >> Your keystore contains 1 entry >> >> puppetdb ca, 28-Mar-2013, trustedCertEntry, >> Certificate fingerprint (SHA1): >> B1:5E:8C:2C:BE:BB:97:5C:09:42:34:7A:F2:02:3F:7B:06:C6:A2:C1 >> >> This CA certificate is the same as what matches my CA file here, we >> can test this by matching the fingerprint: >> >> # openssl x509 -noout -in /var/lib/puppet/ssl/ca/ca_crt.pem -fingerprint >> SHA1 >> Fingerprint=B1:5E:8C:2C:BE:BB:97:5C:09:42:34:7A:F2:02:3F:7B:06:C6:A2:C1 >> >> For client validation, this is really what matters - the client >> certificate must be signed by this CA certificate to be accepted by >> PuppetDB. It doesn''t matter what the client thinks its CA is in Puppet >> configuration or whatever (so setting --ca_name is not enough here), >> all that matters is the client certificate must have been signed by >> that CA, and it is verified using OpenSSL techniques on the PuppetDB >> server during SSL negotiation. >> >> You can do this manually for example. Lets confirm this on my master - >> we know that my local CA is confirmed as the thing loaded into the >> PuppetDB trust store, so now we just run the verify routine against my >> certificate: >> >> # openssl verify -CAfile /var/lib/puppet/ssl/ca/ca_crt.pem `puppet >> master --configprint hostcert` >> /var/lib/puppet/ssl/certs/puppetdb1.vm.pem: OK >> >> You must be sure to test this host certificate against the exact same >> CA loaded into PuppetDB ... not just the local CA (in case they differ >> - you can check this with the -fingerprint command). >> >> If the certificate on the other hand is NOT signed by the CA loaded >> into PuppetDB you get something like: >> >> # openssl verify -CAfile /var/lib/puppet/ssl/ca/ca_crt.pem >> /var/lib/puppet/ssl/certs/puppetdb2.vm.pem >> /var/lib/puppet/ssl/certs/puppetdb2.vm.pem: CN = puppetdb2.vm >> error 20 at 0 depth lookup:unable to get local issuer certificate >> >> Which is by design for this example - that certificate ''puppetdb2.vm'' >> was a certificate I copied from another Puppetmaster which has a >> separate CA, and thus it fails verification. In the real world, >> puppetdb2.vm would get reject by PuppetDB - which is the correct >> behaviour. >> >> ken. >> >> On Mon, Apr 1, 2013 at 9:57 AM, Mohit Chawla >> <mohit.chawla.binary@gmail.com> wrote: >> > Slightly OT here - shouldn''t setting ca_name be enough for --compile to >> > not >> > fail ? Why does it need the --certname param ? >> > >> > >> > On Mon, Apr 1, 2013 at 2:03 PM, Mohit Chawla >> > <mohit.chawla.binary@gmail.com> >> > wrote: >> >> >> >> Hi, >> >> >> >> Got it working by passing --certname param (the ca server hostname, >> >> instead of this master). >> >> >> >> >> >> >> >> On Mon, Apr 1, 2013 at 1:19 PM, Mohit Chawla >> >> <mohit.chawla.binary@gmail.com> wrote: >> >>> >> >>> Hi Ken, >> >>> >> >>> Got a trace from running puppetdb-foreground --debug - >> >>> https://gist.github.com/alcy/5283661. Weird that this doesn''t happen >> >>> during >> >>> standard puppet runs as opposed to doing a --compile. Here''s the >> >>> puppet >> >>> trace when doing --compile - https://gist.github.com/alcy/5283712. >> >>> >> >>> >> >>> >> >>> >> >>> On Fri, Mar 29, 2013 at 3:14 AM, Ken Barber <ken@puppetlabs.com> >> >>> wrote: >> >>>> >> >>>> Yeah, it does seem very odd though ... if agent works - and the >> >>>> master >> >>>> is able to talk to PuppetDB no problem, then its weird that running >> >>>> puppet master on the command line doesn''t seem to work. >> >>>> >> >>>> What is strange is that the SSL error is very very unspecific: >> >>>> >> >>>> Failed to submit ''replace catalog'' command for foo.com to PuppetDB at >> >>>> puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 >> >>>> read >> >>>> finished A >> >>>> >> >>>> It doesn''t speak of a particular problem, ie. remote host doesn''t >> >>>> match certificate, CA not verified etc. etc. it just says ''read >> >>>> finished A''. I think this is significant, the last time I''ve seen it >> >>>> is when there was an SSL bug, and the transport is cut short - which >> >>>> is why I wanted to see what you had in your logs. >> >>>> >> >>>> If you try running the command with --debug --trace do you get any >> >>>> more information? If you tcpdump the connection to the puppetdb >> >>>> server >> >>>> do you see the TCP packets hit the server on port 8081? Try running >> >>>> puppetdb-foreground --debug on your puppetdb server and see what >> >>>> happens when you attempt to compile like this. You''d obviously want >> >>>> to >> >>>> try these things when the other activity on the server is disabled >> >>>> :-). >> >>>> >> >>>> ken. >> >>>> >> >>>> On Thu, Mar 28, 2013 at 8:54 PM, Mohit Chawla >> >>>> <mohit.chawla.binary@gmail.com> wrote: >> >>>> > Hi, >> >>>> > >> >>>> > Not at the workstation right now, but regarding puppet.conf I cant >> >>>> > think of any peculiar settings apart from this being one of the two >> >>>> > puppet masters apart from a separate ca server (we took care of >> >>>> > having >> >>>> > the ca server''s certs being available at these masters). And afaik >> >>>> > right now, there wasn''t any ~/.puppet dir for root, however I need >> >>>> > to >> >>>> > confirm this. >> >>>> > >> >>>> > On Thu, Mar 28, 2013 at 7:07 PM, Ken Barber <ken@puppetlabs.com> >> >>>> > wrote: >> >>>> >> I''m just trying to run up the same environment so I can try to >> >>>> >> replicate it, as yet I can''t replicate it on the newer >> >>>> >> environment. >> >>>> >> >> >>>> >> What does your puppet.conf look like on the host you are trying to >> >>>> >> run >> >>>> >> puppet master --compile btw? I presume you are trying to run the >> >>>> >> command as root, is there a ~/.puppet directory for that user at >> >>>> >> all? >> >>>> >> >> >>>> >> ken. >> >>>> >> >> >>>> >> On Thu, Mar 28, 2013 at 1:17 PM, Mohit Chawla >> >>>> >> <mohit.chawla.binary@gmail.com> wrote: >> >>>> >>> Hello Ken, >> >>>> >>> >> >>>> >>> Thanks for the response. >> >>>> >>> >> >>>> >>> On Thu, Mar 28, 2013 at 6:42 PM, Ken Barber <ken@puppetlabs.com> >> >>>> >>> wrote: >> >>>> >>>> So I have some questions, as the error could mean a number of >> >>>> >>>> things: >> >>>> >>>> >> >>>> >>>> What version of PuppetDB are you running? And what exact version >> >>>> >>>> of >> >>>> >>>> Java is it using? >> >>>> >>> >> >>>> >>> puppetdb version is puppetdb-1.0.4-1.el6.noarch. >> >>>> >>> >> >>>> >>> $ java -version >> >>>> >>> java version "1.6.0_24" >> >>>> >>> OpenJDK Runtime Environment (IcedTea6 1.11.5) >> >>>> >>> (rhel-1.50.1.11.5.el6_3-x86_64) >> >>>> >>> OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode) >> >>>> >>> >> >>>> >>>> >> >>>> >>>> Can you take a look at puppetdb.log and tell me if you see any >> >>>> >>>> meaningful error messages? >> >>>> >>> >> >>>> >>> I took a look, no error messages, only info messages. >> >>>> >>>> >> >>>> >>>> Without trying to compile a catalog in this manner - are you >> >>>> >>>> getting >> >>>> >>>> any trouble with replace facts/replace catalogs generally? ie. >> >>>> >>>> Just >> >>>> >>>> trying to running puppet agent -t for example? >> >>>> >>> >> >>>> >>> Nope, those runs are working fine. >> >>>> >>> >> >>>> >>>> >> >>>> >>>> Have you tried re-initialisation your SSL setup with >> >>>> >>>> puppetdb-ssl-setup at all yet? >> >>>> >>> >> >>>> >>> Yes, and confirmed the validity of certificates and their >> >>>> >>> fingerprints >> >>>> >>> in the truststore and master ca. >> >>>> >>> >> >>>> >>>> >> >>>> >>>> Does the hostname you have specified in >> >>>> >>>> /etc/puppet/puppetdb.conf >> >>>> >>>> to >> >>>> >>>> talk to your puppetdb server match the SSL certificate assigned >> >>>> >>>> to >> >>>> >>>> the >> >>>> >>>> puppetdb host itself? >> >>>> >>> >> >>>> >>> Yes. >> >>>> >>> >> >>>> >>>> >> >>>> >>>> On Thu, Mar 28, 2013 at 12:55 PM, Mohit Chawla >> >>>> >>>> <mohit.chawla.binary@gmail.com> wrote: >> >>>> >>>>> Forgot mentioning the env details: >> >>>> >>>>> >> >>>> >>>>> [user@puppetmaster ~]# rpm -qa | grep puppet >> >>>> >>>>> puppetlabs-release-6-6.noarch >> >>>> >>>>> puppetdb-terminus-1.0.4-1.el6.noarch >> >>>> >>>>> mcollective-puppet-agent-1.4.1-1.noarch >> >>>> >>>>> puppet-2.7.20-1.el6.noarch >> >>>> >>>>> hiera-puppet-1.0.0-1.el6.noarch >> >>>> >>>>> puppet-server-2.7.20-1.el6.noarch >> >>>> >>>>> mcollective-puppet-common-1.4.1-1.noarch >> >>>> >>>>> >> >>>> >>>>> [user@puppetmaster ~]# cat /etc/centos-release >> >>>> >>>>> CentOS release 6.3 (Final) >> >>>> >>>>> >> >>>> >>>>> On Thu, Mar 28, 2013 at 6:23 PM, Mohit Chawla >> >>>> >>>>> <mohit.chawla.binary@gmail.com> wrote: >> >>>> >>>>>> Hello, >> >>>> >>>>>> >> >>>> >>>>>> Stuck in a weird place here. I am trying to do ''puppet master >> >>>> >>>>>> --compile foo.com'', however I am not getting the catalog json. >> >>>> >>>>>> So >> >>>> >>>>>> far, >> >>>> >>>>>> I have noticed two sort of outputs: >> >>>> >>>>>> >> >>>> >>>>>> 1) The above command results in : >> >>>> >>>>>> notice: Compiled catalog for foo.com in environment production >> >>>> >>>>>> in >> >>>> >>>>>> 10.60 seconds >> >>>> >>>>>> Failed to submit ''replace catalog'' command for foo.com to >> >>>> >>>>>> PuppetDB at >> >>>> >>>>>> puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 >> >>>> >>>>>> state=SSLv3 >> >>>> >>>>>> read >> >>>> >>>>>> finished A >> >>>> >>>>>> >> >>>> >>>>>> Notice that I don''t actually get any json & of course, the ssl >> >>>> >>>>>> error. >> >>>> >>>>>> >> >>>> >>>>>> 2) Other output: >> >>>> >>>>>> Failed when searching for node bar.com: Could not retrieve >> >>>> >>>>>> facts >> >>>> >>>>>> for >> >>>> >>>>>> bar.com: Failed to find facts from PuppetDB at puppetdb:8081: >> >>>> >>>>>> SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read >> >>>> >>>>>> finished >> >>>> >>>>>> A >> >>>> >>>>>> >> >>>> >>>>>> Disabling storeconfig allows me to get a proper catalog json. >> >>>> >>>>>> >> >>>> >>>>>> Regarding the ssl errors, we''ve established the correctness of >> >>>> >>>>>> certificates in the relevant places - and done tests using >> >>>> >>>>>> curl >> >>>> >>>>>> and >> >>>> >>>>>> using ruby net/http. Here''s a gist of for the curl and ruby >> >>>> >>>>>> snippet. >> >>>> >>>>>> https://gist.github.com/alcy/5262866 >> >>>> >>>>>> >> >>>> >>>>>> Any suggestions what could be happening here ? Perhaps in >> >>>> >>>>>> indirector/facts/puppetdb.rb, http_get doesn''t get the >> >>>> >>>>>> correct >> >>>> >>>>>> ssl >> >>>> >>>>>> params or something ? >> >>>> >>>>> >> >>>> >>>>> -- >> >>>> >>>>> You received this message because you are subscribed to the >> >>>> >>>>> Google >> >>>> >>>>> Groups "Puppet Users" group. >> >>>> >>>>> To unsubscribe from this group and stop receiving emails from >> >>>> >>>>> it, >> >>>> >>>>> send an email to puppet-users+unsubscribe@googlegroups.com. >> >>>> >>>>> To post to this group, send email to >> >>>> >>>>> puppet-users@googlegroups.com. >> >>>> >>>>> Visit this group at >> >>>> >>>>> http://groups.google.com/group/puppet-users?hl=en. >> >>>> >>>>> For more options, visit >> >>>> >>>>> https://groups.google.com/groups/opt_out. >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>> >> >>>> >>>> -- >> >>>> >>>> You received this message because you are subscribed to the >> >>>> >>>> Google >> >>>> >>>> Groups "Puppet Users" group. >> >>>> >>>> To unsubscribe from this group and stop receiving emails from >> >>>> >>>> it, >> >>>> >>>> send an email to puppet-users+unsubscribe@googlegroups.com. >> >>>> >>>> To post to this group, send email to >> >>>> >>>> puppet-users@googlegroups.com. >> >>>> >>>> Visit this group at >> >>>> >>>> http://groups.google.com/group/puppet-users?hl=en. >> >>>> >>>> For more options, visit >> >>>> >>>> https://groups.google.com/groups/opt_out. >> >>>> >>>> >> >>>> >>>> >> >>>> >>> >> >>>> >>> -- >> >>>> >>> You received this message because you are subscribed to the >> >>>> >>> Google >> >>>> >>> Groups "Puppet Users" group. >> >>>> >>> To unsubscribe from this group and stop receiving emails from it, >> >>>> >>> send an email to puppet-users+unsubscribe@googlegroups.com. >> >>>> >>> To post to this group, send email to >> >>>> >>> puppet-users@googlegroups.com. >> >>>> >>> Visit this group at >> >>>> >>> http://groups.google.com/group/puppet-users?hl=en. >> >>>> >>> For more options, visit https://groups.google.com/groups/opt_out. >> >>>> >>> >> >>>> >>> >> >>>> >> >> >>>> >> -- >> >>>> >> You received this message because you are subscribed to the Google >> >>>> >> Groups "Puppet Users" group. >> >>>> >> To unsubscribe from this group and stop receiving emails from it, >> >>>> >> send an email to puppet-users+unsubscribe@googlegroups.com. >> >>>> >> To post to this group, send email to >> >>>> >> puppet-users@googlegroups.com. >> >>>> >> Visit this group at >> >>>> >> http://groups.google.com/group/puppet-users?hl=en. >> >>>> >> For more options, visit https://groups.google.com/groups/opt_out. >> >>>> >> >> >>>> >> >> >>>> > >> >>>> > -- >> >>>> > You received this message because you are subscribed to the Google >> >>>> > Groups "Puppet Users" group. >> >>>> > To unsubscribe from this group and stop receiving emails from it, >> >>>> > send >> >>>> > an email to puppet-users+unsubscribe@googlegroups.com. >> >>>> > To post to this group, send email to puppet-users@googlegroups.com. >> >>>> > Visit this group at >> >>>> > http://groups.google.com/group/puppet-users?hl=en. >> >>>> > For more options, visit https://groups.google.com/groups/opt_out. >> >>>> > >> >>>> > >> >>>> >> >>>> -- >> >>>> You received this message because you are subscribed to the Google >> >>>> Groups "Puppet Users" group. >> >>>> To unsubscribe from this group and stop receiving emails from it, >> >>>> send >> >>>> an email to puppet-users+unsubscribe@googlegroups.com. >> >>>> To post to this group, send email to puppet-users@googlegroups.com. >> >>>> Visit this group at >> >>>> http://groups.google.com/group/puppet-users?hl=en. >> >>>> For more options, visit https://groups.google.com/groups/opt_out. >> >>>> >> >>>> >> >>> >> >> >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups >> > "Puppet Users" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> > an >> > email to puppet-users+unsubscribe@googlegroups.com. >> > To post to this group, send email to puppet-users@googlegroups.com. >> > Visit this group at http://groups.google.com/group/puppet-users?hl=en. >> > For more options, visit https://groups.google.com/groups/opt_out. >> > >> > >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to puppet-users+unsubscribe@googlegroups.com. >> To post to this group, send email to puppet-users@googlegroups.com. >> Visit this group at http://groups.google.com/group/puppet-users?hl=en. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Mohit Chawla
2013-Apr-03 09:55 UTC
Re: [Puppet Users] Re: Catalog compilation, puppetdb and ssl
Ah nice, thanks for pointing that out. On Wed, Apr 3, 2013 at 3:13 PM, Ken Barber <ken@puppetlabs.com> wrote:> Yes, in this way it behaves a lot like the puppet masters. Don''t > forget to check out certificate-whitelist > ( > http://docs.puppetlabs.com/puppetdb/1.1/configure.html#certificate-whitelist > ) > this gives you the ability to only allow certain hosts to connect if > you desire it. > > On Wed, Apr 3, 2013 at 7:41 AM, Mohit Chawla > <mohit.chawla.binary@gmail.com> wrote: > > Hey Ken, that definitely cleared some misconceptions, thanks ! > > > > I now know what the problem was. First, I assumed that the client must''ve > > the same certificate as in puppetdb''s truststore. I didn''t realize that > any > > cert signed by that CA should be able to make calls. In our scenario, > where > > we have separate certs for the agent and master component on the same > server > > (under /etc/puppet/ssl & /var/lib/puppet/ssl respectively), invoking > > --compile without giving the certname meant no certificate at all ! And > so, > > passing --ssldir to use the agent''s certs (signed by the same CA) worked. > > > > On Tue, Apr 2, 2013 at 3:40 PM, Ken Barber <ken@puppetlabs.com> wrote: > >> > >> If you specify ''certname'' it will use the local certificate with that > >> name as apposed to using the certificate with the same name as the > >> boxes fqdn (ie. facter fqdn). > >> > >> For example, if I use ''puppet master --compile puppetdb1.vm'' on its > >> own, it uses my local fqdn (determined with `facter fqdn`) as an > >> indicator of where the certificate is: > >> /var/lib/puppet/ssl/certs/puppetdb1.vm.pem. If I specify --certname > >> foo it uses /var/lib/puppet/ssl/certs/foo.pem. > >> > >> I presume that the certificate that matches the fqdn on the host you > >> are testing is probably _not_ signed by the CA that is loaded into > >> PuppetDB. Its just a guess, but we should confirm this. > >> > >> Lets take a step back a bit and do an exercise. On your PuppetDB node, > >> in /etc/puppetdb/ssl/ you have three files: > >> > >> * truststore.jks > >> * keystore.jks > >> * puppetdb_keystore_pw.txt > >> > >> If you look inside truststore.jks, you''ll see that it contains the CA > >> certificate: > >> > >> # keytool -list -keystore /etc/puppetdb/ssl/truststore.jks > >> Enter keystore password: (this is contained inside > >> puppetdb_keystore_pw.txt) > >> > >> Keystore type: JKS > >> Keystore provider: SUN > >> > >> Your keystore contains 1 entry > >> > >> puppetdb ca, 28-Mar-2013, trustedCertEntry, > >> Certificate fingerprint (SHA1): > >> B1:5E:8C:2C:BE:BB:97:5C:09:42:34:7A:F2:02:3F:7B:06:C6:A2:C1 > >> > >> This CA certificate is the same as what matches my CA file here, we > >> can test this by matching the fingerprint: > >> > >> # openssl x509 -noout -in /var/lib/puppet/ssl/ca/ca_crt.pem -fingerprint > >> SHA1 > >> Fingerprint=B1:5E:8C:2C:BE:BB:97:5C:09:42:34:7A:F2:02:3F:7B:06:C6:A2:C1 > >> > >> For client validation, this is really what matters - the client > >> certificate must be signed by this CA certificate to be accepted by > >> PuppetDB. It doesn''t matter what the client thinks its CA is in Puppet > >> configuration or whatever (so setting --ca_name is not enough here), > >> all that matters is the client certificate must have been signed by > >> that CA, and it is verified using OpenSSL techniques on the PuppetDB > >> server during SSL negotiation. > >> > >> You can do this manually for example. Lets confirm this on my master - > >> we know that my local CA is confirmed as the thing loaded into the > >> PuppetDB trust store, so now we just run the verify routine against my > >> certificate: > >> > >> # openssl verify -CAfile /var/lib/puppet/ssl/ca/ca_crt.pem `puppet > >> master --configprint hostcert` > >> /var/lib/puppet/ssl/certs/puppetdb1.vm.pem: OK > >> > >> You must be sure to test this host certificate against the exact same > >> CA loaded into PuppetDB ... not just the local CA (in case they differ > >> - you can check this with the -fingerprint command). > >> > >> If the certificate on the other hand is NOT signed by the CA loaded > >> into PuppetDB you get something like: > >> > >> # openssl verify -CAfile /var/lib/puppet/ssl/ca/ca_crt.pem > >> /var/lib/puppet/ssl/certs/puppetdb2.vm.pem > >> /var/lib/puppet/ssl/certs/puppetdb2.vm.pem: CN = puppetdb2.vm > >> error 20 at 0 depth lookup:unable to get local issuer certificate > >> > >> Which is by design for this example - that certificate ''puppetdb2.vm'' > >> was a certificate I copied from another Puppetmaster which has a > >> separate CA, and thus it fails verification. In the real world, > >> puppetdb2.vm would get reject by PuppetDB - which is the correct > >> behaviour. > >> > >> ken. > >> > >> On Mon, Apr 1, 2013 at 9:57 AM, Mohit Chawla > >> <mohit.chawla.binary@gmail.com> wrote: > >> > Slightly OT here - shouldn''t setting ca_name be enough for --compile > to > >> > not > >> > fail ? Why does it need the --certname param ? > >> > > >> > > >> > On Mon, Apr 1, 2013 at 2:03 PM, Mohit Chawla > >> > <mohit.chawla.binary@gmail.com> > >> > wrote: > >> >> > >> >> Hi, > >> >> > >> >> Got it working by passing --certname param (the ca server hostname, > >> >> instead of this master). > >> >> > >> >> > >> >> > >> >> On Mon, Apr 1, 2013 at 1:19 PM, Mohit Chawla > >> >> <mohit.chawla.binary@gmail.com> wrote: > >> >>> > >> >>> Hi Ken, > >> >>> > >> >>> Got a trace from running puppetdb-foreground --debug - > >> >>> https://gist.github.com/alcy/5283661. Weird that this doesn''t > happen > >> >>> during > >> >>> standard puppet runs as opposed to doing a --compile. Here''s the > >> >>> puppet > >> >>> trace when doing --compile - https://gist.github.com/alcy/5283712. > >> >>> > >> >>> > >> >>> > >> >>> > >> >>> On Fri, Mar 29, 2013 at 3:14 AM, Ken Barber <ken@puppetlabs.com> > >> >>> wrote: > >> >>>> > >> >>>> Yeah, it does seem very odd though ... if agent works - and the > >> >>>> master > >> >>>> is able to talk to PuppetDB no problem, then its weird that running > >> >>>> puppet master on the command line doesn''t seem to work. > >> >>>> > >> >>>> What is strange is that the SSL error is very very unspecific: > >> >>>> > >> >>>> Failed to submit ''replace catalog'' command for foo.com to > PuppetDB at > >> >>>> puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 > >> >>>> read > >> >>>> finished A > >> >>>> > >> >>>> It doesn''t speak of a particular problem, ie. remote host doesn''t > >> >>>> match certificate, CA not verified etc. etc. it just says ''read > >> >>>> finished A''. I think this is significant, the last time I''ve seen > it > >> >>>> is when there was an SSL bug, and the transport is cut short - > which > >> >>>> is why I wanted to see what you had in your logs. > >> >>>> > >> >>>> If you try running the command with --debug --trace do you get any > >> >>>> more information? If you tcpdump the connection to the puppetdb > >> >>>> server > >> >>>> do you see the TCP packets hit the server on port 8081? Try running > >> >>>> puppetdb-foreground --debug on your puppetdb server and see what > >> >>>> happens when you attempt to compile like this. You''d obviously want > >> >>>> to > >> >>>> try these things when the other activity on the server is disabled > >> >>>> :-). > >> >>>> > >> >>>> ken. > >> >>>> > >> >>>> On Thu, Mar 28, 2013 at 8:54 PM, Mohit Chawla > >> >>>> <mohit.chawla.binary@gmail.com> wrote: > >> >>>> > Hi, > >> >>>> > > >> >>>> > Not at the workstation right now, but regarding puppet.conf I > cant > >> >>>> > think of any peculiar settings apart from this being one of the > two > >> >>>> > puppet masters apart from a separate ca server (we took care of > >> >>>> > having > >> >>>> > the ca server''s certs being available at these masters). And > afaik > >> >>>> > right now, there wasn''t any ~/.puppet dir for root, however I > need > >> >>>> > to > >> >>>> > confirm this. > >> >>>> > > >> >>>> > On Thu, Mar 28, 2013 at 7:07 PM, Ken Barber <ken@puppetlabs.com> > >> >>>> > wrote: > >> >>>> >> I''m just trying to run up the same environment so I can try to > >> >>>> >> replicate it, as yet I can''t replicate it on the newer > >> >>>> >> environment. > >> >>>> >> > >> >>>> >> What does your puppet.conf look like on the host you are trying > to > >> >>>> >> run > >> >>>> >> puppet master --compile btw? I presume you are trying to run the > >> >>>> >> command as root, is there a ~/.puppet directory for that user at > >> >>>> >> all? > >> >>>> >> > >> >>>> >> ken. > >> >>>> >> > >> >>>> >> On Thu, Mar 28, 2013 at 1:17 PM, Mohit Chawla > >> >>>> >> <mohit.chawla.binary@gmail.com> wrote: > >> >>>> >>> Hello Ken, > >> >>>> >>> > >> >>>> >>> Thanks for the response. > >> >>>> >>> > >> >>>> >>> On Thu, Mar 28, 2013 at 6:42 PM, Ken Barber < > ken@puppetlabs.com> > >> >>>> >>> wrote: > >> >>>> >>>> So I have some questions, as the error could mean a number of > >> >>>> >>>> things: > >> >>>> >>>> > >> >>>> >>>> What version of PuppetDB are you running? And what exact > version > >> >>>> >>>> of > >> >>>> >>>> Java is it using? > >> >>>> >>> > >> >>>> >>> puppetdb version is puppetdb-1.0.4-1.el6.noarch. > >> >>>> >>> > >> >>>> >>> $ java -version > >> >>>> >>> java version "1.6.0_24" > >> >>>> >>> OpenJDK Runtime Environment (IcedTea6 1.11.5) > >> >>>> >>> (rhel-1.50.1.11.5.el6_3-x86_64) > >> >>>> >>> OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode) > >> >>>> >>> > >> >>>> >>>> > >> >>>> >>>> Can you take a look at puppetdb.log and tell me if you see any > >> >>>> >>>> meaningful error messages? > >> >>>> >>> > >> >>>> >>> I took a look, no error messages, only info messages. > >> >>>> >>>> > >> >>>> >>>> Without trying to compile a catalog in this manner - are you > >> >>>> >>>> getting > >> >>>> >>>> any trouble with replace facts/replace catalogs generally? ie. > >> >>>> >>>> Just > >> >>>> >>>> trying to running puppet agent -t for example? > >> >>>> >>> > >> >>>> >>> Nope, those runs are working fine. > >> >>>> >>> > >> >>>> >>>> > >> >>>> >>>> Have you tried re-initialisation your SSL setup with > >> >>>> >>>> puppetdb-ssl-setup at all yet? > >> >>>> >>> > >> >>>> >>> Yes, and confirmed the validity of certificates and their > >> >>>> >>> fingerprints > >> >>>> >>> in the truststore and master ca. > >> >>>> >>> > >> >>>> >>>> > >> >>>> >>>> Does the hostname you have specified in > >> >>>> >>>> /etc/puppet/puppetdb.conf > >> >>>> >>>> to > >> >>>> >>>> talk to your puppetdb server match the SSL certificate > assigned > >> >>>> >>>> to > >> >>>> >>>> the > >> >>>> >>>> puppetdb host itself? > >> >>>> >>> > >> >>>> >>> Yes. > >> >>>> >>> > >> >>>> >>>> > >> >>>> >>>> On Thu, Mar 28, 2013 at 12:55 PM, Mohit Chawla > >> >>>> >>>> <mohit.chawla.binary@gmail.com> wrote: > >> >>>> >>>>> Forgot mentioning the env details: > >> >>>> >>>>> > >> >>>> >>>>> [user@puppetmaster ~]# rpm -qa | grep puppet > >> >>>> >>>>> puppetlabs-release-6-6.noarch > >> >>>> >>>>> puppetdb-terminus-1.0.4-1.el6.noarch > >> >>>> >>>>> mcollective-puppet-agent-1.4.1-1.noarch > >> >>>> >>>>> puppet-2.7.20-1.el6.noarch > >> >>>> >>>>> hiera-puppet-1.0.0-1.el6.noarch > >> >>>> >>>>> puppet-server-2.7.20-1.el6.noarch > >> >>>> >>>>> mcollective-puppet-common-1.4.1-1.noarch > >> >>>> >>>>> > >> >>>> >>>>> [user@puppetmaster ~]# cat /etc/centos-release > >> >>>> >>>>> CentOS release 6.3 (Final) > >> >>>> >>>>> > >> >>>> >>>>> On Thu, Mar 28, 2013 at 6:23 PM, Mohit Chawla > >> >>>> >>>>> <mohit.chawla.binary@gmail.com> wrote: > >> >>>> >>>>>> Hello, > >> >>>> >>>>>> > >> >>>> >>>>>> Stuck in a weird place here. I am trying to do ''puppet > master > >> >>>> >>>>>> --compile foo.com'', however I am not getting the catalog > json. > >> >>>> >>>>>> So > >> >>>> >>>>>> far, > >> >>>> >>>>>> I have noticed two sort of outputs: > >> >>>> >>>>>> > >> >>>> >>>>>> 1) The above command results in : > >> >>>> >>>>>> notice: Compiled catalog for foo.com in environment > production > >> >>>> >>>>>> in > >> >>>> >>>>>> 10.60 seconds > >> >>>> >>>>>> Failed to submit ''replace catalog'' command for foo.com to > >> >>>> >>>>>> PuppetDB at > >> >>>> >>>>>> puppetdb:8081: SSL_connect SYSCALL returned=5 errno=0 > >> >>>> >>>>>> state=SSLv3 > >> >>>> >>>>>> read > >> >>>> >>>>>> finished A > >> >>>> >>>>>> > >> >>>> >>>>>> Notice that I don''t actually get any json & of course, the > ssl > >> >>>> >>>>>> error. > >> >>>> >>>>>> > >> >>>> >>>>>> 2) Other output: > >> >>>> >>>>>> Failed when searching for node bar.com: Could not retrieve > >> >>>> >>>>>> facts > >> >>>> >>>>>> for > >> >>>> >>>>>> bar.com: Failed to find facts from PuppetDB at > puppetdb:8081: > >> >>>> >>>>>> SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read > >> >>>> >>>>>> finished > >> >>>> >>>>>> A > >> >>>> >>>>>> > >> >>>> >>>>>> Disabling storeconfig allows me to get a proper catalog > json. > >> >>>> >>>>>> > >> >>>> >>>>>> Regarding the ssl errors, we''ve established the correctness > of > >> >>>> >>>>>> certificates in the relevant places - and done tests using > >> >>>> >>>>>> curl > >> >>>> >>>>>> and > >> >>>> >>>>>> using ruby net/http. Here''s a gist of for the curl and ruby > >> >>>> >>>>>> snippet. > >> >>>> >>>>>> https://gist.github.com/alcy/5262866 > >> >>>> >>>>>> > >> >>>> >>>>>> Any suggestions what could be happening here ? Perhaps in > >> >>>> >>>>>> indirector/facts/puppetdb.rb, http_get doesn''t get the > >> >>>> >>>>>> correct > >> >>>> >>>>>> ssl > >> >>>> >>>>>> params or something ? > >> >>>> >>>>> > >> >>>> >>>>> -- > >> >>>> >>>>> You received this message because you are subscribed to the > >> >>>> >>>>> Google > >> >>>> >>>>> Groups "Puppet Users" group. > >> >>>> >>>>> To unsubscribe from this group and stop receiving emails from > >> >>>> >>>>> it, > >> >>>> >>>>> send an email to puppet-users+unsubscribe@googlegroups.com. > >> >>>> >>>>> To post to this group, send email to > >> >>>> >>>>> puppet-users@googlegroups.com. > >> >>>> >>>>> Visit this group at > >> >>>> >>>>> http://groups.google.com/group/puppet-users?hl=en. > >> >>>> >>>>> For more options, visit > >> >>>> >>>>> https://groups.google.com/groups/opt_out. > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>> > >> >>>> >>>> -- > >> >>>> >>>> You received this message because you are subscribed to the > >> >>>> >>>> Google > >> >>>> >>>> Groups "Puppet Users" group. > >> >>>> >>>> To unsubscribe from this group and stop receiving emails from > >> >>>> >>>> it, > >> >>>> >>>> send an email to puppet-users+unsubscribe@googlegroups.com. > >> >>>> >>>> To post to this group, send email to > >> >>>> >>>> puppet-users@googlegroups.com. > >> >>>> >>>> Visit this group at > >> >>>> >>>> http://groups.google.com/group/puppet-users?hl=en. > >> >>>> >>>> For more options, visit > >> >>>> >>>> https://groups.google.com/groups/opt_out. > >> >>>> >>>> > >> >>>> >>>> > >> >>>> >>> > >> >>>> >>> -- > >> >>>> >>> You received this message because you are subscribed to the > >> >>>> >>> Google > >> >>>> >>> Groups "Puppet Users" group. > >> >>>> >>> To unsubscribe from this group and stop receiving emails from > it, > >> >>>> >>> send an email to puppet-users+unsubscribe@googlegroups.com. > >> >>>> >>> To post to this group, send email to > >> >>>> >>> puppet-users@googlegroups.com. > >> >>>> >>> Visit this group at > >> >>>> >>> http://groups.google.com/group/puppet-users?hl=en. > >> >>>> >>> For more options, visit > https://groups.google.com/groups/opt_out. > >> >>>> >>> > >> >>>> >>> > >> >>>> >> > >> >>>> >> -- > >> >>>> >> You received this message because you are subscribed to the > Google > >> >>>> >> Groups "Puppet Users" group. > >> >>>> >> To unsubscribe from this group and stop receiving emails from > it, > >> >>>> >> send an email to puppet-users+unsubscribe@googlegroups.com. > >> >>>> >> To post to this group, send email to > >> >>>> >> puppet-users@googlegroups.com. > >> >>>> >> Visit this group at > >> >>>> >> http://groups.google.com/group/puppet-users?hl=en. > >> >>>> >> For more options, visit > https://groups.google.com/groups/opt_out. > >> >>>> >> > >> >>>> >> > >> >>>> > > >> >>>> > -- > >> >>>> > You received this message because you are subscribed to the > Google > >> >>>> > Groups "Puppet Users" group. > >> >>>> > To unsubscribe from this group and stop receiving emails from it, > >> >>>> > send > >> >>>> > an email to puppet-users+unsubscribe@googlegroups.com. > >> >>>> > To post to this group, send email to > puppet-users@googlegroups.com. > >> >>>> > Visit this group at > >> >>>> > http://groups.google.com/group/puppet-users?hl=en. > >> >>>> > For more options, visit https://groups.google.com/groups/opt_out > . > >> >>>> > > >> >>>> > > >> >>>> > >> >>>> -- > >> >>>> You received this message because you are subscribed to the Google > >> >>>> Groups "Puppet Users" group. > >> >>>> To unsubscribe from this group and stop receiving emails from it, > >> >>>> send > >> >>>> an email to puppet-users+unsubscribe@googlegroups.com. > >> >>>> To post to this group, send email to puppet-users@googlegroups.com > . > >> >>>> Visit this group at > >> >>>> http://groups.google.com/group/puppet-users?hl=en. > >> >>>> For more options, visit https://groups.google.com/groups/opt_out. > >> >>>> > >> >>>> > >> >>> > >> >> > >> > > >> > -- > >> > You received this message because you are subscribed to the Google > >> > Groups > >> > "Puppet Users" group. > >> > To unsubscribe from this group and stop receiving emails from it, send > >> > an > >> > email to puppet-users+unsubscribe@googlegroups.com. > >> > To post to this group, send email to puppet-users@googlegroups.com. > >> > Visit this group at http://groups.google.com/group/puppet-users?hl=en > . > >> > For more options, visit https://groups.google.com/groups/opt_out. > >> > > >> > > >> > >> -- > >> You received this message because you are subscribed to the Google > Groups > >> "Puppet Users" group. > >> To unsubscribe from this group and stop receiving emails from it, send > an > >> email to puppet-users+unsubscribe@googlegroups.com. > >> To post to this group, send email to puppet-users@googlegroups.com. > >> Visit this group at http://groups.google.com/group/puppet-users?hl=en. > >> For more options, visit https://groups.google.com/groups/opt_out. > >> > >> > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Puppet Users" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to puppet-users+unsubscribe@googlegroups.com. > > To post to this group, send email to puppet-users@googlegroups.com. > > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > > For more options, visit https://groups.google.com/groups/opt_out. > > > > > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.