I am trying to establish what looks like a common pattern for scaling puppet. My main departure is that I''m using an F5 rather than an apache load balancer. Namely, I want to have my puppet agents go through the F5 to a pool of "master only" systems, and any Certificate activity to get proxied by those masters through to one single Certificate Authority. That CA system is not part of the F5 pool, it''s role is to provide CA, Puppetdb and Postgresql. It is configured as a master because that was the easiest way to get a CA stood up, but I don''t intend to use it as a master in normal operation (and in fact I don''t plan to have it hosting any modules). I''m using RHEL 6, Apache, and Passenger, and Open Source Puppet. I initially set up passenger using puppetlabs/passenger from the Forge, (which got me most of the way there but not fully configured). All of these steps worked fine for the CA system to configure it as a working master (I have tested by registering systems with it, but then done puppet cert clean and wiped the test systems'' ssl directories). I then set up my first master-only system the same way, except I didn''t actually start the master service (as the docs say) until after I had set ca = false and ca_server = $MY_CA_SERVER into /etc/puppet.conf. I also made the necessary changes listed at http://docs.puppetlabs.com/guides/scaling_multiple_masters.html, including the certificate access on the CA system, the SSLProxyEngine on and ProxyPassMatch lines in the VHost definition in /etc/httpd/conf.d/puppetmaster.conf. I''m positive I followed all the steps in the docs in order, but I''m not having any luck with external agents. If I run puppet agent -t on the master-only system (with it''s "server" in puppet.conf set to itself) it works fine--it can talk to the CA and talk to itself, and all is right with the world. If I run puppet agent -t on a client host, pointing at the load balancer''s address (or even pointing direclty at the master-only system''s real hostname), I get: [root@elmer ~]# puppet agent -t Info: Creating a new SSL key for elmer.allstate.com Error: Could not request certificate: Error 400 on SERVER: this master is not a CA Exiting; failed to retrieve certificate and waitforcert is disabled I''ve looked at the logs, enabled debug logging in the webserver with LogLevel, dug around everywhere I can think of, and I see no sign of any actual proxying going on. tcpdump certainly shows no attempt by the master-only system to contact the CA. What it LOOKS like is happening is that apache is not actually proxying anything, the request gets passed to the puppet master app running under passenger, and it (rightly) says "I''m not a CA" because /etc/puppet/puppet.conf says so. I do not see any errors in the logs about proxy attempts failing for this agent. I do see workers being attached for proxy purposes: [Tue Oct 01 13:48:26 2013] [debug] proxy_util.c(1833): proxy: grabbed scoreboard slot 0 in child 27434 for worker https://caserver.allstate.com:8140/$1 [Tue Oct 01 13:48:26 2013] [debug] proxy_util.c(1852): proxy: worker https://caserver.allstate.com:8140/$1 already initialized [Tue Oct 01 13:48:26 2013] [debug] proxy_util.c(1949): proxy: initialized single connection worker 0 in child 27434 for caserver.allstate.com) I''ve repeatedly re-checked the settings in /etc/puppet.conf /etc/httpd/conf.d/passenger.conf, /et/chttpd/conf.d/puppetmaster.conf etc against the documentation and I am not seeing any errors. This seems like I have to be overlooking something really basic, and I''m going to feel stupid when I find it, but it''s right in my critical path right now and I can''t see it. Anyone have any suggestions? I can provide config files and log files if need be, but I''m trying to avoid all the redacting I''d need to do (my server is not literally named "caserver" etc). Thanks Pete -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
Pete Hartman
2013-Oct-02 02:34 UTC
Re: [Puppet Users] Pulling my hair out with CA proxying
I have to do more testing to determine for certain, but it appears to have been some combination of 1) the order in which modules were loaded, and 2) not having mod_proxy_http loaded. On Tue, Oct 1, 2013 at 2:39 PM, Pete Hartman <pete.hartman@gmail.com> wrote:> I am trying to establish what looks like a common pattern for scaling > puppet. My main departure is that I''m using an F5 rather than an apache load > balancer. Namely, I want to have my puppet agents go through the F5 to a > pool of "master only" systems, and any Certificate activity to get proxied > by those masters through to one single Certificate Authority. That CA > system is not part of the F5 pool, it''s role is to provide CA, Puppetdb and > Postgresql. It is configured as a master because that was the easiest way > to get a CA stood up, but I don''t intend to use it as a master in normal > operation (and in fact I don''t plan to have it hosting any modules). > > I''m using RHEL 6, Apache, and Passenger, and Open Source Puppet. > > I initially set up passenger using puppetlabs/passenger from the Forge, > (which got me most of the way there but not fully configured). All of these > steps worked fine for the CA system to configure it as a working master (I > have tested by registering systems with it, but then done puppet cert clean > and wiped the test systems'' ssl directories). > > I then set up my first master-only system the same way, except I didn''t > actually start the master service (as the docs say) until after I had set ca > = false and ca_server = $MY_CA_SERVER into /etc/puppet.conf. I also made > the necessary changes listed at > http://docs.puppetlabs.com/guides/scaling_multiple_masters.html, including > the certificate access on the CA system, the SSLProxyEngine on and > ProxyPassMatch lines in the VHost definition in > /etc/httpd/conf.d/puppetmaster.conf. I''m positive I followed all the steps > in the docs in order, but I''m not having any luck with external agents. > > If I run puppet agent -t on the master-only system (with it''s "server" in > puppet.conf set to itself) it works fine--it can talk to the CA and talk to > itself, and all is right with the world. > > If I run puppet agent -t on a client host, pointing at the load balancer''s > address (or even pointing direclty at the master-only system''s real > hostname), I get: > > [root@elmer ~]# puppet agent -t > Info: Creating a new SSL key for elmer.allstate.com > Error: Could not request certificate: Error 400 on SERVER: this master is > not a CA > Exiting; failed to retrieve certificate and waitforcert is disabled > > > I''ve looked at the logs, enabled debug logging in the webserver with > LogLevel, dug around everywhere I can think of, and I see no sign of any > actual proxying going on. tcpdump certainly shows no attempt by the > master-only system to contact the CA. > > What it LOOKS like is happening is that apache is not actually proxying > anything, the request gets passed to the puppet master app running under > passenger, and it (rightly) says "I''m not a CA" because > /etc/puppet/puppet.conf says so. > > I do not see any errors in the logs about proxy attempts failing for this > agent. I do see workers being attached for proxy purposes: > > [Tue Oct 01 13:48:26 2013] [debug] proxy_util.c(1833): proxy: grabbed > scoreboard slot 0 in child 27434 for worker > https://caserver.allstate.com:8140/$1 > [Tue Oct 01 13:48:26 2013] [debug] proxy_util.c(1852): proxy: worker > https://caserver.allstate.com:8140/$1 already initialized > [Tue Oct 01 13:48:26 2013] [debug] proxy_util.c(1949): proxy: initialized > single connection worker 0 in child 27434 for caserver.allstate.com) > > > I''ve repeatedly re-checked the settings in /etc/puppet.conf > /etc/httpd/conf.d/passenger.conf, /et/chttpd/conf.d/puppetmaster.conf etc > against the documentation and I am not seeing any errors. > > This seems like I have to be overlooking something really basic, and I''m > going to feel stupid when I find it, but it''s right in my critical path > right now and I can''t see it. Anyone have any suggestions? I can provide > config files and log files if need be, but I''m trying to avoid all the > redacting I''d need to do (my server is not literally named "caserver" etc). > > Thanks > > Pete > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Puppet Users" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/puppet-users/xY5xnOU09Qg/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users. > For more options, visit https://groups.google.com/groups/opt_out.-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
Gavin Williams
2013-Oct-02 07:08 UTC
Re: [Puppet Users] Pulling my hair out with CA proxying
Pete I''ve not done this before, however am familiar with Puppet, and know a lot more about F5s... I note that you say that you''re expecting apache on the masters to proxy onto the CA server. Is there any reason you couldn''t use the F5 to select the CA server for any CA requests? Should be a fairly straight forward iRule to do pool selection based on the URI. Thoughts? Gav -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
Pete Hartman
2013-Oct-02 12:35 UTC
Re: [Puppet Users] Pulling my hair out with CA proxying
I do not have responsibility for the F5''s and I''m not sure what my networking team would be willing to do in terms of custom rules no matter how simple. The use of the apache proxy service on the masters is a configuration documented and recommended (at least as one alternative) by PuppetLabs; now that I have found what I was missing, I plan to stick with that. On Wed, Oct 2, 2013 at 2:08 AM, Gavin Williams <fatmcgav@gmail.com> wrote:> Pete > > I''ve not done this before, however am familiar with Puppet, and know a lot more about F5s... > > I note that you say that you''re expecting apache on the masters to proxy onto the CA server. > Is there any reason you couldn''t use the F5 to select the CA server for any CA requests? > Should be a fairly straight forward iRule to do pool selection based on the URI. > > Thoughts? > > Gav > > -- > You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group. > To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/xY5xnOU09Qg/unsubscribe. > To unsubscribe from this group and all its topics, send an email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users. > For more options, visit https://groups.google.com/groups/opt_out.-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
Felipe Salum
2013-Oct-02 19:27 UTC
Re: [Puppet Users] Pulling my hair out with CA proxying
Can you paste your /etc/httpd/conf.d/puppetmaster.conf ? On Wednesday, October 2, 2013 5:35:58 AM UTC-7, Pete Hartman wrote:> > I do not have responsibility for the F5''s and I''m not sure what my > networking team would be willing to do in terms of custom rules no > matter how simple. > > The use of the apache proxy service on the masters is a configuration > documented and recommended (at least as one alternative) by > PuppetLabs; now that I have found what I was missing, I plan to stick > with that. > > On Wed, Oct 2, 2013 at 2:08 AM, Gavin Williams <fatm...@gmail.com<javascript:>> > wrote: > > Pete > > > > I''ve not done this before, however am familiar with Puppet, and know a > lot more about F5s... > > > > I note that you say that you''re expecting apache on the masters to proxy > onto the CA server. > > Is there any reason you couldn''t use the F5 to select the CA server for > any CA requests? > > Should be a fairly straight forward iRule to do pool selection based on > the URI. > > > > Thoughts? > > > > Gav > > > > -- > > You received this message because you are subscribed to a topic in the > Google Groups "Puppet Users" group. > > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/puppet-users/xY5xnOU09Qg/unsubscribe. > > To unsubscribe from this group and all its topics, send an email to > puppet-users...@googlegroups.com <javascript:>. > > To post to this group, send email to puppet...@googlegroups.com<javascript:>. > > > Visit this group at http://groups.google.com/group/puppet-users. > > For more options, visit https://groups.google.com/groups/opt_out. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
Pete Hartman
2013-Oct-02 20:05 UTC
Re: [Puppet Users] Pulling my hair out with CA proxying
I tried to update this, but apparently failed. Problem was my own misunderstanding of apache. 1) the passenger module was loaded before the proxy module, so the app was responding before apache could proxy the request 2) I didn''t recognize this as a working fix at first because I also omitted mod_proxy_http which was needed in addition to mod_proxy Thanks... Pete On Wed, Oct 2, 2013 at 2:27 PM, Felipe Salum <fsalum@gmail.com> wrote:> Can you paste your /etc/httpd/conf.d/puppetmaster.conf ? > > > On Wednesday, October 2, 2013 5:35:58 AM UTC-7, Pete Hartman wrote: >> >> I do not have responsibility for the F5''s and I''m not sure what my >> networking team would be willing to do in terms of custom rules no >> matter how simple. >> >> The use of the apache proxy service on the masters is a configuration >> documented and recommended (at least as one alternative) by >> PuppetLabs; now that I have found what I was missing, I plan to stick >> with that. >> >> On Wed, Oct 2, 2013 at 2:08 AM, Gavin Williams <fatm...@gmail.com> wrote: >> > Pete >> > >> > I''ve not done this before, however am familiar with Puppet, and know a >> > lot more about F5s... >> > >> > I note that you say that you''re expecting apache on the masters to proxy >> > onto the CA server. >> > Is there any reason you couldn''t use the F5 to select the CA server for >> > any CA requests? >> > Should be a fairly straight forward iRule to do pool selection based on >> > the URI. >> > >> > Thoughts? >> > >> > Gav >> > >> > -- >> > You received this message because you are subscribed to a topic in the >> > Google Groups "Puppet Users" group. >> > To unsubscribe from this topic, visit >> > https://groups.google.com/d/topic/puppet-users/xY5xnOU09Qg/unsubscribe. >> > To unsubscribe from this group and all its topics, send an email to >> > puppet-users...@googlegroups.com. >> > To post to this group, send email to puppet...@googlegroups.com. >> > Visit this group at http://groups.google.com/group/puppet-users. >> > For more options, visit https://groups.google.com/groups/opt_out. > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Puppet Users" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/puppet-users/xY5xnOU09Qg/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users. > For more options, visit https://groups.google.com/groups/opt_out.-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.