I''m trying to set up a puppetmaster under unicorn using the ubuntu maverick packages (currently at version 2.6.1rc2), and I''m getting the following error: root@test.puppet.cms.ucar.edu $ puppetd -t err: Could not retrieve catalog from remote server: Error 403 on SERVER: Forbidden request: test.puppet.cms.ucar.edu(128.117.224.193) access to /catalog/test.puppet.cms.ucar.edu [find] at line 98 warning: Not using cache on failed catalog err: Could not retrieve catalog; skipping run I''m using the standard auth.conf, but if I turn off auth by adding this to the top of the file everything works: path / auth no allow * Has anyone seen this, or know of a workaround? -- Dan Urist durist@ucar.edu 303-497-2459 -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Brice Figureau
2010-Aug-26 20:34 UTC
Re: [Puppet Users] auth failure under unicorn with 2.6.1rc2
On 26/08/10 21:55, Dan Urist wrote:> I''m trying to set up a puppetmaster under unicorn using the ubuntu > maverick packages (currently at version 2.6.1rc2), and I''m getting the > following error: > > root@test.puppet.cms.ucar.edu $ puppetd -t > err: Could not retrieve catalog from remote server: Error 403 on SERVER: Forbidden request: test.puppet.cms.ucar.edu(128.117.224.193) access to /catalog/test.puppet.cms.ucar.edu [find] at line 98 > warning: Not using cache on failed catalog > err: Could not retrieve catalog; skipping run > > I''m using the standard auth.conf, but if I turn off auth by adding this > to the top of the file everything works: > > path / > auth no > allow *Of course you understand the security risk if you run with this auth.conf :)> Has anyone seen this, or know of a workaround?The usual cause is that the SSL end point didn''t propagate to the master the fact that this node''s certificate validates. This is usally done by adding some HTTP headers in the request, and you need to tell puppet what those headers are. For rack you need to set: [puppetmasterd] ssl_client_header = SSL_CLIENT_S_DN ssl_client_verify_header = SSL_CLIENT_VERIFY Off course you also need to configure the ssl endpoint to set those headers when the cerficate is valid (and also when it''s invalid). You didn''t mention what was the SLL endpoint in your configuration so I can''t really help for this. -- Brice Figureau My Blog: http://www.masterzen.fr/ -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Mathias Gug
2010-Aug-26 20:58 UTC
Re: [Puppet Users] auth failure under unicorn with 2.6.1rc2
Hi, Excerpts from Brice Figureau''s message of Thu Aug 26 16:34:59 -0400 2010:> On 26/08/10 21:55, Dan Urist wrote: > > I''m trying to set up a puppetmaster under unicorn using the ubuntu > > maverick packages (currently at version 2.6.1rc2)Starting from maverick you may wanna look at the puppetmaster-passenger package. Installing this package will automatically setup puppetmaster to run under apache2 + mod_ssl + mod_passenger. apache2 ssl configuration is automatically done by the package itself. -- Mathias Gug Ubuntu Developer http://www.ubuntu.com -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Dan Urist
2010-Aug-26 21:09 UTC
Re: [Puppet Users] auth failure under unicorn with 2.6.1rc2
On Thu, 26 Aug 2010 22:34:59 +0200 Brice Figureau <brice-puppet@daysofwonder.com> wrote:> On 26/08/10 21:55, Dan Urist wrote: > > I''m trying to set up a puppetmaster under unicorn using the ubuntu > > maverick packages (currently at version 2.6.1rc2), and I''m getting > > the following error: > > > > root@test.puppet.cms.ucar.edu $ puppetd -t > > err: Could not retrieve catalog from remote server: Error 403 on > > SERVER: Forbidden request: > > test.puppet.cms.ucar.edu(128.117.224.193) access > > to /catalog/test.puppet.cms.ucar.edu [find] at line 98 warning: Not > > using cache on failed catalog err: Could not retrieve catalog; > > skipping run > > > > I''m using the standard auth.conf, but if I turn off auth by adding > > this to the top of the file everything works: > > > > path / > > auth no > > allow * > > Of course you understand the security risk if you run with this > auth.conf :)Yes, I just tried this for testing.> > Has anyone seen this, or know of a workaround? > > The usual cause is that the SSL end point didn''t propagate to the > master the fact that this node''s certificate validates. > > This is usally done by adding some HTTP headers in the request, and > you need to tell puppet what those headers are. > For rack you need to set: > > [puppetmasterd] > ssl_client_header = SSL_CLIENT_S_DN > ssl_client_verify_header = SSL_CLIENT_VERIFYI have this, but it''s under "master" rather than "puppetmasterd". I''ve tried it under "puppetmasterd" and I''m getting the same failure.> Off course you also need to configure the ssl endpoint to set those > headers when the cerficate is valid (and also when it''s invalid). > You didn''t mention what was the SLL endpoint in your configuration so > I can''t really help for this.I''m using nginx, and I''ve followed the docs at: http://projects.puppetlabs.com/projects/1/wiki/Using_Unicorn The relevant parts of my nginx config, per the doc, are:> proxy_set_header Host $host; > proxy_set_header X-Real-IP $remote_addr; > proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; > proxy_set_header X-Client-Verify $ssl_client_verify; > proxy_set_header X-Client-DN $ssl_client_s_dn; > proxy_set_header X-SSL-Issuer $ssl_client_i_dn; > proxy_read_timeout 120;So as far as I can see, those headers are being set. Any hints on debugging this? -- Dan Urist durist@ucar.edu 303-497-2459 -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Dan Urist
2010-Aug-26 21:17 UTC
Re: [Puppet Users] auth failure under unicorn with 2.6.1rc2
On Thu, 26 Aug 2010 16:58:19 -0400 Mathias Gug <mathiaz@ubuntu.com> wrote:> Hi, > > Excerpts from Brice Figureau''s message of Thu Aug 26 16:34:59 -0400 > 2010: > > On 26/08/10 21:55, Dan Urist wrote: > > > I''m trying to set up a puppetmaster under unicorn using the ubuntu > > > maverick packages (currently at version 2.6.1rc2) > > Starting from maverick you may wanna look at the > puppetmaster-passenger package. Installing this package will > automatically setup puppetmaster to run under apache2 + mod_ssl + > mod_passenger. apache2 ssl configuration is automatically done by the > package itself. >I have a working puppet server running under apache/passenger for 0.25.4; I''d really like to get it running under nginx for performance and consistency, though, since that''s what most of the rest of my site is using. -- Dan Urist durist@ucar.edu 303-497-2459 -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Brice Figureau
2010-Aug-27 07:34 UTC
Re: [Puppet Users] auth failure under unicorn with 2.6.1rc2
On Thu, 2010-08-26 at 15:09 -0600, Dan Urist wrote:> On Thu, 26 Aug 2010 22:34:59 +0200 > Brice Figureau <brice-puppet@daysofwonder.com> wrote: > > > On 26/08/10 21:55, Dan Urist wrote: > > > I''m trying to set up a puppetmaster under unicorn using the ubuntu > > > maverick packages (currently at version 2.6.1rc2), and I''m getting > > > the following error: > > > > > > root@test.puppet.cms.ucar.edu $ puppetd -t > > > err: Could not retrieve catalog from remote server: Error 403 on > > > SERVER: Forbidden request: > > > test.puppet.cms.ucar.edu(128.117.224.193) access > > > to /catalog/test.puppet.cms.ucar.edu [find] at line 98 warning: Not > > > using cache on failed catalog err: Could not retrieve catalog; > > > skipping run > > > > > > I''m using the standard auth.conf, but if I turn off auth by adding > > > this to the top of the file everything works: > > > > > > path / > > > auth no > > > allow * > > > > Of course you understand the security risk if you run with this > > auth.conf :) > > Yes, I just tried this for testing.OK, I prefer to check :)> > > Has anyone seen this, or know of a workaround? > > > > The usual cause is that the SSL end point didn''t propagate to the > > master the fact that this node''s certificate validates. > > > > This is usally done by adding some HTTP headers in the request, and > > you need to tell puppet what those headers are. > > For rack you need to set: > > > > [puppetmasterd] > > ssl_client_header = SSL_CLIENT_S_DN > > ssl_client_verify_header = SSL_CLIENT_VERIFY > > I have this, but it''s under "master" rather than "puppetmasterd". I''ve > tried it under "puppetmasterd" and I''m getting the same failure.Yes, you should use master for 2.6, but puppetmasterd for 0.25.> > Off course you also need to configure the ssl endpoint to set those > > headers when the cerficate is valid (and also when it''s invalid). > > You didn''t mention what was the SLL endpoint in your configuration so > > I can''t really help for this. > > I''m using nginx, and I''ve followed the docs at: > http://projects.puppetlabs.com/projects/1/wiki/Using_Unicorn > > The relevant parts of my nginx config, per the doc, are: > > > proxy_set_header Host $host; > > proxy_set_header X-Real-IP $remote_addr; > > proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; > > proxy_set_header X-Client-Verify $ssl_client_verify; > > proxy_set_header X-Client-DN $ssl_client_s_dn;Note that the config snippets I sent you refers those headers as SSL_CLIENT_VERIFY and not X_CLIENT_VERIFY. Either correct the configuration or nginx, but both should use the same header names.> > proxy_set_header X-SSL-Issuer $ssl_client_i_dn; > > proxy_read_timeout 120; > > So as far as I can see, those headers are being set. Any hints on > debugging this?There are several possibilities: * check puppet uses the correct $ssldir. I''ve already seen people using a different $ssldir when running the master differently, in which case the master regenerates a CA, and client certs are not compatible anymore. * check that the client cert is valid (ie it was signed by your master current $ssldir CA). This can be done with openssl * run nginx in debug mode to check it sets correctly the upstream headers * use tcpdump/wireshard to capture the http traffic between nginx and unicorn and check the headers are there and correct. * add some Puppet.notice() statements in puppet ruby rack adapter (in lib/puppet/network/http/rack/rest.rb) around line 93 to print the various values and which branch of the if is taken. Hope that helps, -- Brice Figureau Follow the latest Puppet Community evolutions on www.planetpuppet.org! -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Dan Urist
2010-Aug-27 20:32 UTC
Re: [Puppet Users] auth failure under unicorn with 2.6.1rc2
I took your suggestion and tried Puppet.notice() statements in rest.rb and was able to dump out the whole request (after a crash course in ruby syntax). I didn''t realize there was a mapping from the nginx config file parameters to the actual request parameters; this appears to have fixed it in puppet.conf (these are the actual request parameters): ssl_client_header = HTTP_X_CLIENT_DN ssl_client_verify_header = HTTP_X_CLIENT_VERIFY The relevant lines from my nginx config are: proxy_set_header X-Client-Verify $ssl_client_verify; proxy_set_header X-Client-DN $ssl_client_s_dn; Thanks for your help. On Fri, 27 Aug 2010 09:34:03 +0200 Brice Figureau <brice-puppet@daysofwonder.com> wrote:> On Thu, 2010-08-26 at 15:09 -0600, Dan Urist wrote: > > On Thu, 26 Aug 2010 22:34:59 +0200 > > Brice Figureau <brice-puppet@daysofwonder.com> wrote: > > > > > On 26/08/10 21:55, Dan Urist wrote: > > > > I''m trying to set up a puppetmaster under unicorn using the > > > > ubuntu maverick packages (currently at version 2.6.1rc2), and > > > > I''m getting the following error: > > > > > > > > root@test.puppet.cms.ucar.edu $ puppetd -t > > > > err: Could not retrieve catalog from remote server: Error 403 on > > > > SERVER: Forbidden request: > > > > test.puppet.cms.ucar.edu(128.117.224.193) access > > > > to /catalog/test.puppet.cms.ucar.edu [find] at line 98 warning: > > > > Not using cache on failed catalog err: Could not retrieve > > > > catalog; skipping run > > > > > > > > I''m using the standard auth.conf, but if I turn off auth by > > > > adding this to the top of the file everything works: > > > > > > > > path / > > > > auth no > > > > allow * > > > > > > Of course you understand the security risk if you run with this > > > auth.conf :) > > > > Yes, I just tried this for testing. > > OK, I prefer to check :) > > > > > Has anyone seen this, or know of a workaround? > > > > > > The usual cause is that the SSL end point didn''t propagate to the > > > master the fact that this node''s certificate validates. > > > > > > This is usally done by adding some HTTP headers in the request, > > > and you need to tell puppet what those headers are. > > > For rack you need to set: > > > > > > [puppetmasterd] > > > ssl_client_header = SSL_CLIENT_S_DN > > > ssl_client_verify_header = SSL_CLIENT_VERIFY > > > > I have this, but it''s under "master" rather than "puppetmasterd". > > I''ve tried it under "puppetmasterd" and I''m getting the same > > failure. > > Yes, you should use master for 2.6, but puppetmasterd for 0.25. > > > > Off course you also need to configure the ssl endpoint to set > > > those headers when the cerficate is valid (and also when it''s > > > invalid). You didn''t mention what was the SLL endpoint in your > > > configuration so I can''t really help for this. > > > > I''m using nginx, and I''ve followed the docs at: > > http://projects.puppetlabs.com/projects/1/wiki/Using_Unicorn > > > > The relevant parts of my nginx config, per the doc, are: > > > > > proxy_set_header Host $host; > > > proxy_set_header X-Real-IP $remote_addr; > > > proxy_set_header X-Forwarded-For > > > $proxy_add_x_forwarded_for; proxy_set_header X-Client-Verify > > > $ssl_client_verify; proxy_set_header X-Client-DN $ssl_client_s_dn; > > Note that the config snippets I sent you refers those headers as > SSL_CLIENT_VERIFY and not X_CLIENT_VERIFY. > Either correct the configuration or nginx, but both should use the > same header names. > > > > proxy_set_header X-SSL-Issuer $ssl_client_i_dn; > > > proxy_read_timeout 120; > > > > So as far as I can see, those headers are being set. Any hints on > > debugging this? > > There are several possibilities: > > * check puppet uses the correct $ssldir. I''ve already seen people > using a different $ssldir when running the master differently, in > which case the master regenerates a CA, and client certs are not > compatible anymore. > > * check that the client cert is valid (ie it was signed by your master > current $ssldir CA). This can be done with openssl > > * run nginx in debug mode to check it sets correctly the upstream > headers > > * use tcpdump/wireshard to capture the http traffic between nginx and > unicorn and check the headers are there and correct. > > * add some Puppet.notice() statements in puppet ruby rack adapter (in > lib/puppet/network/http/rack/rest.rb) around line 93 to print the > various values and which branch of the if is taken. > > Hope that helps,-- Dan Urist durist@ucar.edu 303-497-2459 -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.