I''m new to puppet, fixin'' to (in Texas parlance) setup a largish installation, and want to make sure I get things right the first time. I have a colo plus several satellite locations, with many more on the way. I''ll be using open-source Puppet, as we have way too many nodes to manage (potentially 275+ at each site) to afford PE. I''m thinking about setting up a master in the colo with a slaved master at each site, following this document: http://docs.puppetlabs.com/guides/scaling_multiple_masters.html Each master will run passenger/apache. I''ll have puppetDB running on the primary master at the colo (with an eye to moving it to a separate server if needed). I''ll probably use Subversion (but maybe git) for synchronization of data among masters. I''d like to use mcollective and hiera, so those will probably get setup on the primary master. (Do I need them on each master?) One of our sites is a lab, so I''ll be doing testing there.I''ll need to setup some kind of dev/test/prod environment system. I''ll be testing both dashboard and foreman, although I need to do automated bare-metal installs, so foreman will probably win out there. Each site also has around 100 nodes that break and get swapped out frequently, so (depending on what foreman can do for me in the installation process) I might need to be relatively lax about certificates to avoid revoking and regenerating them constantly. (We''re talking 2-3 nodes a week getting pulled, worked on, and replaced or reimaged.) So that''s where I am right now. Any thoughts, suggestions, tips, how-tos would be greatly appreciated. Paul -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
With hiera, you''ll obviously need to sync your hiera data and hierarchy, along with your modules and whatever else, between your masters. For mcollective, There are ways to set up a network of brokers, both to distribute load and to make sure that if a site gets separated, mcollective will still work locally. http://docs.puppetlabs.com/mcollective/deploy/middleware/activemq.html#settings-for-networks-of-brokers. That entire page is actually pretty useful... On Friday, August 23, 2013 9:03:41 AM UTC-7, Paul Archer wrote:> > I''m new to puppet, fixin'' to (in Texas parlance) setup a largish > installation, and want to make sure I get things right the first time. > > I have a colo plus several satellite locations, with many more on the way. > > I''ll be using open-source Puppet, as we have way too many nodes to manage > (potentially 275+ at each site) to afford PE. > > I''m thinking about setting up a master in the colo with a slaved master at > each site, following this document: > http://docs.puppetlabs.com/guides/scaling_multiple_masters.html > Each master will run passenger/apache. I''ll have puppetDB running on the > primary master at the colo (with an eye to moving it to a separate server > if needed). I''ll probably use Subversion (but maybe git) for > synchronization of data among masters. > I''d like to use mcollective and hiera, so those will probably get setup on > the primary master. (Do I need them on each master?) > > One of our sites is a lab, so I''ll be doing testing there.I''ll need to > setup some kind of dev/test/prod environment system. > > I''ll be testing both dashboard and foreman, although I need to do > automated bare-metal installs, so foreman will probably win out there. > > Each site also has around 100 nodes that break and get swapped out > frequently, so (depending on what foreman can do for me in the installation > process) I might need to be relatively lax about certificates to avoid > revoking and regenerating them constantly. (We''re talking 2-3 nodes a week > getting pulled, worked on, and replaced or reimaged.) > > > So that''s where I am right now. Any thoughts, suggestions, tips, how-tos > would be greatly appreciated. > > > Paul >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
Martin Langhoff
2013-Aug-23 16:51 UTC
Re: [Puppet Users] Best practices for infrastructure
On Fri, Aug 23, 2013 at 12:03 PM, Paul Archer <geek65535@gmail.com> wrote:> I''m thinking about setting up a master in the colo with a slaved master at > each site, >I would strongly recommend using "master-less" recipes, which are actually "a git repository as a master, and cronjobs running puppet apply as client". On that track, I have recently implemented just that, rolling out to 4 sites totalling a couple thousand client nodes. I have posted on this list about my glue / tools, which I am publishing at http://repo.or.cz/w/puppet-git.git . They allow you to feed the reports to a puppet dashboard (something that you usually lose in "master-less" setups. An addition to puppet-git being triggered via cron, I have a configuration that sets up an incrond rule, so if we need an immediate rollout, touching a file in a magic directory triggers the clients to update their config right now. My puppet-git is good, I recommend it :-) -- but YMMV on that. But using git as a master is, IMHO, best practice at scale. cheers, m -- martin.langhoff@gmail.com - ask interesting questions - don''t get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
Much of what I would suggest depends on the exact setup of your servers. Here are a few generic suggestions: You already want to use hiera. That''s great, but like any tool, if you do not use it properly, you can booger it all up. I started using puppet before hiera and I can personally vouch for how great it is to pull all the per-server data out of the main "code" base. Hiera is built into puppet-master, so no worries there. I cannot speak about mcollective as I do not use it (small buncha servers) Use the hierarchy and priority to organize your servers by functional groups. In my location, I started with a generic server (dev), then got a QA server, and a production server from that. Production servers split into internal and external (dmz). I hope this helps you get started. Try not to do it all at once. Take it in smaller steps. “Sometimes I think the surest sign that intelligent life exists elsewhere in the universe is that none of it has tried to contact us.” Bill Waterson (Calvin & Hobbes) ----- Original Message ----- From: "Paul Archer" <geek65535@gmail.com> To: puppet-users@googlegroups.com Sent: Friday, August 23, 2013 12:03:41 PM Subject: [Puppet Users] Best practices for infrastructure I''m new to puppet, fixin'' to (in Texas parlance) setup a largish installation, and want to make sure I get things right the first time. I have a colo plus several satellite locations, with many more on the way. I''ll be using open-source Puppet, as we have way too many nodes to manage (potentially 275+ at each site) to afford PE. I''m thinking about setting up a master in the colo with a slaved master at each site, following this document: http://docs.puppetlabs.com/guides/scaling_multiple_masters.html Each master will run passenger/apache. I''ll have puppetDB running on the primary master at the colo (with an eye to moving it to a separate server if needed). I''ll probably use Subversion (but maybe git) for synchronization of data among masters. I''d like to use mcollective and hiera, so those will probably get setup on the primary master. (Do I need them on each master?) One of our sites is a lab, so I''ll be doing testing there.I''ll need to setup some kind of dev/test/prod environment system. I''ll be testing both dashboard and foreman, although I need to do automated bare-metal installs, so foreman will probably win out there. Each site also has around 100 nodes that break and get swapped out frequently, so (depending on what foreman can do for me in the installation process) I might need to be relatively lax about certificates to avoid revoking and regenerating them constantly. (We''re talking 2-3 nodes a week getting pulled, worked on, and replaced or reimaged.) So that''s where I am right now. Any thoughts, suggestions, tips, how-tos would be greatly appreciated. Paul -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users . For more options, visit https://groups.google.com/groups/opt_out . -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
Thanks for your input. There are some good suggestions and advice here. Martin, I will definitely look into a masterless setup, although I think I''ll start with the more traditional multi-master setup to start with. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
Speaking in security terms, could be masterless puppet configuration less secure? I mean, the puppet code is in *all* the clients. On the other hand, the puppet code is only in the master, which I think is more secure (you can isolate it on a restricted VLAN, private network, etc). If the security of one client is vulnerated the hacker gets nothing, otherwise he would be able to read the whole puppet code. El viernes, 23 de agosto de 2013 18:51:07 UTC+2, Martin Langhoff escribió:> > On Fri, Aug 23, 2013 at 12:03 PM, Paul Archer <geek...@gmail.com<javascript:> > > wrote: > >> I''m thinking about setting up a master in the colo with a slaved master >> at each site, >> > > I would strongly recommend using "master-less" recipes, which are actually > "a git repository as a master, and cronjobs running puppet apply as client". > > On that track, I have recently implemented just that, rolling out to 4 > sites totalling a couple thousand client nodes. I have posted on this list > about my glue / tools, which I am publishing at > http://repo.or.cz/w/puppet-git.git . They allow you to feed the reports > to a puppet dashboard (something that you usually lose in "master-less" > setups. > > An addition to puppet-git being triggered via cron, I have a configuration > that sets up an incrond rule, so if we need an immediate rollout, touching > a file in a magic directory triggers the clients to update their config > right now. > > My puppet-git is good, I recommend it :-) -- but YMMV on that. But using > git as a master is, IMHO, best practice at scale. > > cheers, > > > > m > -- > martin....@gmail.com <javascript:> > - ask interesting questions > - don''t get distracted with shiny stuff - working code first > ~ http://docs.moodle.org/en/User:Martin_Langhoff >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
Martin Langhoff
2013-Aug-24 14:13 UTC
Re: [Puppet Users] Best practices for infrastructure
On Sat, Aug 24, 2013 at 6:33 AM, Félix Barbeira <fbarbeira@gmail.com> wrote:> Speaking in security terms, could be masterless puppet configuration less > secure? I mean, the puppet code is in *all* the clients. On the other hand, > the puppet code is only in the master, which I think is more secure (you > can isolate it on a restricted VLAN, private network, etc). If the security > of one client is vulnerated the hacker gets nothing, otherwise he would be > able to read the whole puppet code. >The difference is minimal. The master will happily serve any config to any host. The puppet server relies on the self-reported hostname, so a compromised host can go "fishing" for configurations. In my git-as-a-master configurations I use ssh to connect to the master. Yes, all hosts using the same master see the "full" set of configs. If I ever have a clearly separate security domain of sorts, plan would be to set up a separate git master. I think that makes sense too with a puppet master. cheers, m -- martin.langhoff@gmail.com - ask interesting questions - don''t get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
On 08/24/2013 04:13 PM, Martin Langhoff wrote:> On Sat, Aug 24, 2013 at 6:33 AM, Félix Barbeira <fbarbeira@gmail.com > <mailto:fbarbeira@gmail.com>> wrote: > > Speaking in security terms, could be masterless puppet configuration > less secure? I mean, the puppet code is in *all* the clients. On the > other hand, the puppet code is only in the master, which I think is > more secure (you can isolate it on a restricted VLAN, private > network, etc). If the security of one client is vulnerated the > hacker gets nothing, otherwise he would be able to read the whole > puppet code. > > > The difference is minimal. The master will happily serve any config to > any host. The puppet server relies on the self-reported hostname, so a > compromised host can go "fishing" for configurations.Only if you use autosign option. After the certificate is signed, agents report certname and not hostname. In that regard, puppet master is safer option, but also less scalable. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
Martin Langhoff
2013-Aug-25 12:17 UTC
Re: [Puppet Users] Best practices for infrastructure
On Sat, Aug 24, 2013 at 5:18 PM, Jakov Sosic <jsosic@srce.hr> wrote:> Only if you use autosign option. After the certificate is signed, agents > report certname and not hostname.Well-behaved clients report certname. A malicious client could use one cert, but report a different name. AIUI the puppet master checks the certificate to allow connection, but uses the client-reported name to pick the configuration served. cheers, m -- martin.langhoff@gmail.com - ask interesting questions - don''t get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
On 23/08/13 12:51 PM, Martin Langhoff wrote:> On Fri, Aug 23, 2013 at 12:03 PM, Paul Archer <geek65535@gmail.com > <mailto:geek65535@gmail.com>> wrote: > > I''m thinking about setting up a master in the colo with a slaved > master at each site, > > > I would strongly recommend using "master-less" recipes, which are > actually "a git repository as a master, and cronjobs running puppet > apply as client".keep in mind if you implement a masterless setup that you can''t aggregate values for other servers to consume (e.g. exported resources -- or option storeconfigs), so if you''d like to automate configuration of a service based on what is managed in other nodes, you''re gonna have to find weird hacks to do this. an interesting read on masterless setups: http://current.workingdirectory.net/posts/2011/puppet-without-masters/ -- Gabriel Filion
On 08/25/2013 02:17 PM, Martin Langhoff wrote:> On Sat, Aug 24, 2013 at 5:18 PM, Jakov Sosic <jsosic@srce.hr> wrote: >> Only if you use autosign option. After the certificate is signed, agents >> report certname and not hostname. > > Well-behaved clients report certname. A malicious client could use one > cert, but report a different name. AIUI the puppet master checks the > certificate to allow connection, but uses the client-reported name to > pick the configuration served.Hm, are you really sure about that? Can you demonstrate it? I''m asking out of curiosity cause I''ve thought it can''t be done. -- Jakov Sosic www.srce.unizg.hr -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
On Sunday, August 25, 2013 7:17:21 AM UTC-5, Martin Langhoff wrote:> > On Sat, Aug 24, 2013 at 5:18 PM, Jakov Sosic <jso...@srce.hr <javascript:>> > wrote: > > Only if you use autosign option. After the certificate is signed, agents > > report certname and not hostname. > > Well-behaved clients report certname. A malicious client could use one > cert, but report a different name. AIUI the puppet master checks the > certificate to allow connection, but uses the client-reported name to > pick the configuration served. > >Puppet identifies client nodes by the certname on the SSL certificates they present. This is what is passed to an ENC (if configured) and what is matched against defined node blocks, so it controls what configuration is served. The client can provide a $::hostname fact that is different from the certname it presents, but that is perfectly valid and expected under some circumstances. It is possible that a client doing so is thereby able to exploit weaknesses in (user-provided) manifest files required anyway for its catalog, thereby extracting information to which it is not intended to have access, but that is possible to some degree or another with any fact. It does not constitute a flaw in Puppet itself, but rather in the manifests in question. John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
Martin Langhoff
2013-Aug-27 18:47 UTC
Re: [Puppet Users] Best practices for infrastructure
On Tue, Aug 27, 2013 at 2:41 PM, jcbollinger <John.Bollinger@stjude.org> wrote:> The client can provide a $::hostname fact that is different from the > certname it presents, but that is perfectly valid and expected under some > circumstances. It is possible that a client doing so is thereby able to > exploit weaknesses in (user-provided) manifest files required anyway for its > catalog, thereby extracting information to which it is not intended to have > access, but that is possible to some degree or another with any fact. It > does not constitute a flaw in Puppet itself, but rather in the manifests in > question.That''s roughly what I recall. So, in less words: from a security perspective, do not count on Puppet only serving the right config for the host. It is a very flimsy security boundary. cheers, m -- martin.langhoff@gmail.com - ask interesting questions - don''t get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
On Tuesday, August 27, 2013 1:47:02 PM UTC-5, Martin Langhoff wrote:> > On Tue, Aug 27, 2013 at 2:41 PM, jcbollinger <John.Bo...@stjude.org<javascript:>> > wrote: > > The client can provide a $::hostname fact that is different from the > > certname it presents, but that is perfectly valid and expected under > some > > circumstances. It is possible that a client doing so is thereby able to > > exploit weaknesses in (user-provided) manifest files required anyway for > its > > catalog, thereby extracting information to which it is not intended to > have > > access, but that is possible to some degree or another with any fact. > It > > does not constitute a flaw in Puppet itself, but rather in the manifests > in > > question. > > That''s roughly what I recall. > > So, in less words: from a security perspective, do not count on Puppet > only serving the right config for the host. It is a very flimsy > security boundary. > >If the objective is to render it into a small number of words, then I think a fairer characterization would be: Puppet securely identifies clients and provides that information, but it cannot not prevent ENCs and manifests from relying on falsifiable information. That''s not any different from most other security-sensitive information services, such as an https web application running on Apache. Apache can be made to identify me via my own certificate and to provide that identity to the webapp, but if the application ignores my securely-established identity when making decisions about what information to serve up to me then I may be able to trick it out of information I''m not supposed to have. Puppet makes client certnames available implicitly in node blocks and explicitly everywhere (via a fact), so a Puppet site (master + ENC / manifests / data + agents) can be secured reasonably well. It is misleading to complain that the master alone cannot provide sufficient security without acknowledging that it''s because the master is only one part of the picture. John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
Martin Langhoff
2013-Aug-28 17:00 UTC
Re: [Puppet Users] Best practices for infrastructure
On Wed, Aug 28, 2013 at 9:31 AM, jcbollinger <John.Bollinger@stjude.org> wrote:> If the objective is to render it into a small number of words,Just to double-check my understanding is right. If the client-reported $::hostname does not match the certname, _and_ I am only using ''node "fqdn"'' entries in my Puppet manifests, puppet will use... certname or client-reported $::hostname? cheers, m -- martin.langhoff@gmail.com - ask interesting questions - don''t get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
On Wednesday, August 28, 2013 12:00:21 PM UTC-5, Martin Langhoff wrote:> > On Wed, Aug 28, 2013 at 9:31 AM, jcbollinger <John.Bo...@stjude.org<javascript:>> > wrote: > > If the objective is to render it into a small number of words, > > Just to double-check my understanding is right. If the client-reported > $::hostname does not match the certname, _and_ I am only using ''node > "fqdn"'' entries in my Puppet manifests, puppet will use... certname or > client-reported $::hostname? > >The master will always choose the node block to use based on the client''s SSL certname (spelled $::clientcert in Puppet DSL). Likewise, if you use an ENC then it is the certname that the master passes to it when requesting a node classification. It is common for $::hostname to have the same value as the certname, but the master itself in no way depends on such a coincidence. Indeed, some sites intentionally assign certnames that do not match hostnames. For example, they may use MAC addresses instead. That has considerable advantages when machines do not have stable hostnames. John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
Martin Langhoff
2013-Aug-30 13:27 UTC
Re: [Puppet Users] Best practices for infrastructure
On Thu, Aug 29, 2013 at 9:02 AM, jcbollinger <John.Bollinger@stjude.org> wrote:> The master will always choose the node block to use based on the client''s > SSL certname (spelled $::clientcert in Puppet DSL).Oh, that is considerably safer than what I feared. Thanks for the clarification. My comments earlier in this thread were under the mistaken understanding that the Puppet master in its default behaviour would allow match nodename based on $::hostname. thank you! m -- martin.langhoff@gmail.com - ask interesting questions - don''t get distracted with shiny stuff - working code first ~ http://docs.moodle.org/en/User:Martin_Langhoff -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.