Hi all. We''re building our puppet infrastructure from scratch, and need to decide on how to organize our modules. As the puppet best practice document suggests, we''re going to put all building blocks modules in the "modules" area, and make use of the services and clients areas to make up server configurations. But how does others build server configurations from these modules in a dynamic and structured way? How to you handle situations where one have multiple projects, environments environments (dev, test, qass, prod), and so forth - do you go for a "<project>::<environment>::<role>" type of module/class structure? Before starting to model our servers in the services and clients areas I''d like to make sure we don''t start out wrong. Best regards, Kenneth Holter -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Hi Kenneth, I''m also creating a setup from scratch and want to do similar things like you. Therefore i cannot provide a big experience report. However we disussed quite a lot internally how we want to setup it correctly. One outcome was to create a generic node which holds all module available on all machines (dns, nagios daemons, ntp,...). A node therefore builds together from that generic node and the node specific modules. In order to define different setups for each module (test, dev,...) we are going to use tags. You can then use the central puppet run and pass a list of tags you want deploy. You should check the tag section of the puppet documentation. I hope that helps a bit. I would be interested if experienced puppet user would agree with this setup suggestion. Christian On 16 Jul., 12:40, Kenneth Holter <kenneho....@gmail.com> wrote:> Hi all. > > We''re building our puppet infrastructure from scratch, and need to > decide on how to organize our modules. As the puppet best practice > document suggests, we''re going to put all building blocks modules in > the "modules" area, and make use of the services and clients areas to > make up server configurations. > > But how does others build server configurations from these modules in > a dynamic and structured way? How to you handle situations where one > have multiple projects, environments environments (dev, test, qass, > prod), and so forth - do you go for a > "<project>::<environment>::<role>" type of module/class structure? > Before starting to model our servers in the services and clients areas > I''d like to make sure we don''t start out wrong. > > Best regards, > Kenneth Holter-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Kenneth Holter
2010-Aug-02 11:50 UTC
Re: [Puppet Users] Re: Best practices on organizing modules
Thank you for your reply. It was not clear to me exactly how you will be using tags - could you please elaborate? How will your "clients" and "services" areas be organized? Let me briefly mention that we''re going to use external nodes to tell which environment (prod, qass, ...) a node belongs to, and use the build in puppet enironments to separate code between the environments (production clients will pull modules from a different module area than clients in for example qass, and so forth). - Kenneth On Fri, Jul 16, 2010 at 1:46 PM, Christian <berwangerchristian@googlemail.com> wrote:> Hi Kenneth, > > I''m also creating a setup from scratch and want to do similar things > like you. Therefore i cannot provide a big experience report. > > However we disussed quite a lot internally how we want to setup it > correctly. > One outcome was to create a generic node which holds all module > available on all machines (dns, nagios daemons, ntp,...). > A node therefore builds together from that generic node and the node > specific modules. > > In order to define different setups for each module (test, dev,...) we > are going to use tags. > You can then use the central puppet run and pass a list of tags you > want deploy. You should check the tag section of the puppet > documentation. > > I hope that helps a bit. I would be interested if experienced puppet > user would agree with this setup suggestion. > > Christian > > > > > > On 16 Jul., 12:40, Kenneth Holter <kenneho....@gmail.com> wrote: >> Hi all. >> >> We''re building our puppet infrastructure from scratch, and need to >> decide on how to organize our modules. As the puppet best practice >> document suggests, we''re going to put all building blocks modules in >> the "modules" area, and make use of the services and clients areas to >> make up server configurations. >> >> But how does others build server configurations from these modules in >> a dynamic and structured way? How to you handle situations where one >> have multiple projects, environments environments (dev, test, qass, >> prod), and so forth - do you go for a >> "<project>::<environment>::<role>" type of module/class structure? >> Before starting to model our servers in the services and clients areas >> I''d like to make sure we don''t start out wrong. >> >> Best regards, >> Kenneth Holter > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Jeff McCune
2010-Aug-03 05:37 UTC
Re: [Puppet Users] Re: Best practices on organizing modules
On Mon, Aug 2, 2010 at 4:50 AM, Kenneth Holter <kenneho.ndu@gmail.com> wrote:> > Let me briefly mention that we''re going to use external nodes to tell > which environment (prod, qass, ...) a node belongs to, and use the > build in puppet enironments to separate code between the environments > (production clients will pull modules from a different module area > than clients in for example qass, and so forth).The organization of using environments in puppet with module paths referencing multiple, different working copies of the same version control repository checked out to different branches works quite well for many people. Git is particularly good in this case since it''s designed to quickly switch the working copy among branches quickly and easily. Doing so in git doesn''t result in a different filesystem path which is convenient when setting the modulepath in each environment. -- Jeff McCune http://www.puppetlabs.com/ -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Kenneth Holter
2010-Aug-03 11:00 UTC
Re: [Puppet Users] Re: Best practices on organizing modules
We''re a subversion shop, so we''re using subversion for this type of thing (although I''d like to try out Git). We''ve created a few simple scripts for tagging modules and such, so I belive we have a neat solution to version control of our code. But structuring the code itself (as was the original topic for this thread) is of course equally important. We have multiple projects each consisting of multiple server types/roles and environments (dev, test, qass, prod), and need our clients and services areas to reflect this in a dynamic and structured way. Having a base/generic class like Christian proposed is the way to go, but I would like to hear from others how they have organized the clients and services areas. - Kenneth On Tue, Aug 3, 2010 at 7:37 AM, Jeff McCune <jeff@puppetlabs.com> wrote:> On Mon, Aug 2, 2010 at 4:50 AM, Kenneth Holter <kenneho.ndu@gmail.com> wrote: >> >> Let me briefly mention that we''re going to use external nodes to tell >> which environment (prod, qass, ...) a node belongs to, and use the >> build in puppet enironments to separate code between the environments >> (production clients will pull modules from a different module area >> than clients in for example qass, and so forth). > > The organization of using environments in puppet with module paths > referencing multiple, different working copies of the same version > control repository checked out to different branches works quite well > for many people. > > Git is particularly good in this case since it''s designed to quickly > switch the working copy among branches quickly and easily. Doing so > in git doesn''t result in a different filesystem path which is > convenient when setting the modulepath in each environment. > > -- > Jeff McCune > http://www.puppetlabs.com/ > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Ohad Levy
2010-Aug-03 11:16 UTC
Re: [Puppet Users] Re: Best practices on organizing modules
maybe this thread can help: http://groups.google.com/group/puppet-users/browse_thread/thread/838480ef7394bc79/72b48b4b1ea6a0e5?lnk=gst&q=Ideas/Best+Practices+for+module+versionin#72b48b4b1ea6a0e5 <http://groups.google.com/group/puppet-users/browse_thread/thread/838480ef7394bc79/72b48b4b1ea6a0e5?lnk=gst&q=Ideas/Best+Practices+for+module+versionin#72b48b4b1ea6a0e5> Ohad On Tue, Aug 3, 2010 at 7:00 PM, Kenneth Holter <kenneho.ndu@gmail.com>wrote:> We''re a subversion shop, so we''re using subversion for this type of > thing (although I''d like to try out Git). We''ve created a few simple > scripts for tagging modules and such, so I belive we have a neat > solution to version control of our code. > > But structuring the code itself (as was the original topic for this > thread) is of course equally important. We have multiple projects each > consisting of multiple server types/roles and environments (dev, test, > qass, prod), and need our clients and services areas to reflect this > in a dynamic and structured way. Having a base/generic class like > Christian proposed is the way to go, but I would like to hear from > others how they have organized the clients and services areas. > > > - Kenneth > > On Tue, Aug 3, 2010 at 7:37 AM, Jeff McCune <jeff@puppetlabs.com> wrote: > > On Mon, Aug 2, 2010 at 4:50 AM, Kenneth Holter <kenneho.ndu@gmail.com> > wrote: > >> > >> Let me briefly mention that we''re going to use external nodes to tell > >> which environment (prod, qass, ...) a node belongs to, and use the > >> build in puppet enironments to separate code between the environments > >> (production clients will pull modules from a different module area > >> than clients in for example qass, and so forth). > > > > The organization of using environments in puppet with module paths > > referencing multiple, different working copies of the same version > > control repository checked out to different branches works quite well > > for many people. > > > > Git is particularly good in this case since it''s designed to quickly > > switch the working copy among branches quickly and easily. Doing so > > in git doesn''t result in a different filesystem path which is > > convenient when setting the modulepath in each environment. > > > > -- > > Jeff McCune > > http://www.puppetlabs.com/ > > > > -- > > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > > To post to this group, send email to puppet-users@googlegroups.com. > > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com> > . > > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > > > > > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Kenneth Holter
2010-Aug-03 13:30 UTC
Re: [Puppet Users] Re: Best practices on organizing modules
Thanks. So does your environment folder structure only contain modules, or have you put the clients and services module areas into the environment folder structure too? What does your clients and services areas look like? For now I''ve created different environment areas only for modules, and have thought about maybe going for something like "c_<project>::<environment>::<serverRole>" type of class structure, so that I can build different server configurations for the different projects, environments and server roles. On other words, my clients area would look something like this: - "c_projectA", in which the classes "c_projectA::prod::webserver", "c_projectA::qass::webserver" and so forth would be implemented (and added to the node definition for the relevant servers) - "c_projectB", in which the classes "c_projectB::dev::database", "c_projectB::testing::database" and so forth would be implemented (and added to the node definition for the relevant servers) - etc I haven''t thought out all the details yet, but I believe something like this would make having multiple server setups manageable. Any thoughts on this kind of setup? Kenneth On Tue, Aug 3, 2010 at 1:16 PM, Ohad Levy <ohadlevy@gmail.com> wrote:> maybe this thread can > help:http://groups.google.com/group/puppet-users/browse_thread/thread/838480ef7394bc79/72b48b4b1ea6a0e5?lnk=gst&q=Ideas/Best+Practices+for+module+versionin#72b48b4b1ea6a0e5> Ohad > > On Tue, Aug 3, 2010 at 7:00 PM, Kenneth Holter <kenneho.ndu@gmail.com> > wrote: >> >> We''re a subversion shop, so we''re using subversion for this type of >> thing (although I''d like to try out Git). We''ve created a few simple >> scripts for tagging modules and such, so I belive we have a neat >> solution to version control of our code. >> >> But structuring the code itself (as was the original topic for this >> thread) is of course equally important. We have multiple projects each >> consisting of multiple server types/roles and environments (dev, test, >> qass, prod), and need our clients and services areas to reflect this >> in a dynamic and structured way. Having a base/generic class like >> Christian proposed is the way to go, but I would like to hear from >> others how they have organized the clients and services areas. >> >> >> - Kenneth >> >> On Tue, Aug 3, 2010 at 7:37 AM, Jeff McCune <jeff@puppetlabs.com> wrote: >> > On Mon, Aug 2, 2010 at 4:50 AM, Kenneth Holter <kenneho.ndu@gmail.com> >> > wrote: >> >> >> >> Let me briefly mention that we''re going to use external nodes to tell >> >> which environment (prod, qass, ...) a node belongs to, and use the >> >> build in puppet enironments to separate code between the environments >> >> (production clients will pull modules from a different module area >> >> than clients in for example qass, and so forth). >> > >> > The organization of using environments in puppet with module paths >> > referencing multiple, different working copies of the same version >> > control repository checked out to different branches works quite well >> > for many people. >> > >> > Git is particularly good in this case since it''s designed to quickly >> > switch the working copy among branches quickly and easily. Doing so >> > in git doesn''t result in a different filesystem path which is >> > convenient when setting the modulepath in each environment. >> > >> > -- >> > Jeff McCune >> > http://www.puppetlabs.com/ >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups "Puppet Users" group. >> > To post to this group, send email to puppet-users@googlegroups.com. >> > To unsubscribe from this group, send email to >> > puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>.>> > For more options, visit this group at >> > http://groups.google.com/group/puppet-users?hl=en. >> > >> > >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To post to this group, send email to puppet-users@googlegroups.com. >> To unsubscribe from this group, send email to >> puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>.>> For more options, visit this group at >> http://groups.google.com/group/puppet-users?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>.> For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Eric Sorenson
2010-Aug-03 17:45 UTC
Re: [Puppet Users] Re: Best practices on organizing modules
On Aug 3, 2010, at 6:30 AM, Kenneth Holter wrote:> On other words, my clients area would look something like this: > • "c_projectA", in which the classes "c_projectA::prod::webserver", "c_projectA::qass::webserver" and so forth would be implemented (and added to the node definition for the relevant servers) > • "c_projectB", in which the classes "c_projectB::dev::database", "c_projectB::testing::database" and so forth would be implemented (and added to the node definition for the relevant servers) > • etc > I haven''t thought out all the details yet, but I believe something like this would make having multiple server setups manageable. Any thoughts on this kind of setup? >This seems like it would lead to a confusing multiplicity of manifests. Unless (in your example) webservers are completely divergent and have nothing in common between qass and prod environments, I''d put all the webserver code in one class and use decision-making conditionals inside the class to change aspects of the resources. That way when someone else comes along and wants to modify webserver behaviour they have only one place to look instead of four or five. You''re starting out with an external node tool, so setting variables at top-level scope for projects and environments will make this easy. I''ve ended up with a common structure among the different modules: typically there''s a case statement at the top which sets class-scope variables based on the globals everyone on the team knows about and uses, then the actual resources below which use those class-scope vars inline. EG: class infrastructure::sudoers { case $env { # globally set by external node tool dev,qa,perf: { $sudoersfile = "sudoers.preprod" } prod: { $sudoersfile = "sudoers.prod" } default: { # always make a safe default in case $env is unset $sudoersfile = "sudoers.minimal" } } file { "sudoers": path => $operatingsystem ? { solaris => "/usr/local/etc/sudoers", default => "/etc/sudoers", }, owner => "root", group => 0, mode => "0440", source => "puppet:///external/sudo/$sudoersfile", } } Does that make sense? Not sure if the use-case maps up exactly to yours but it sounds close enough that this might work for you and end up being a lot simpler. - Eric Sorenson - N37 17.255 W121 55.738 - http://twitter.com/ahpook - -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Kenneth Holter
2010-Oct-07 12:01 UTC
Re: [Puppet Users] Re: Best practices on organizing modules
On Tue, Aug 3, 2010 at 7:45 PM, Eric Sorenson <eric.sorenson@me.com> wrote:> On Aug 3, 2010, at 6:30 AM, Kenneth Holter wrote: > > > On other words, my clients area would look something like this: > > • "c_projectA", in which the classes "c_projectA::prod::webserver", > "c_projectA::qass::webserver" and so forth would be implemented (and added > to the node definition for the relevant servers) > > • "c_projectB", in which the classes "c_projectB::dev::database", > "c_projectB::testing::database" and so forth would be implemented (and added > to the node definition for the relevant servers) > > • etc > > I haven''t thought out all the details yet, but I believe something like > this would make having multiple server setups manageable. Any thoughts on > this kind of setup? > > > > This seems like it would lead to a confusing multiplicity of manifests. > > Unless (in your example) webservers are completely divergent and have > nothing in common between qass and prod environments, I''d put all the > webserver code in one class and use decision-making conditionals inside the > class to change aspects of the resources. That way when someone else comes > along and wants to modify webserver behaviour they have only one place to > look instead of four or five. You''re starting out with an external node > tool, so setting variables at top-level scope for projects and environments > will make this easy. I''ve ended up with a common structure among the > different modules: typically there''s a case statement at the top which sets > class-scope variables based on the globals everyone on the team knows about > and uses, then the actual resources below which use those class-scope vars > inline. EG: > > class infrastructure::sudoers { > case $env { # globally set by external node tool > dev,qa,perf: { > $sudoersfile = "sudoers.preprod" > } > prod: { > $sudoersfile = "sudoers.prod" > } > default: { # always make a safe default in case $env is unset > $sudoersfile = "sudoers.minimal" > } > > } > > file { "sudoers": > path => $operatingsystem ? { > solaris => "/usr/local/etc/sudoers", > default => "/etc/sudoers", > }, > owner => "root", > group => 0, > mode => "0440", > source => "puppet:///external/sudo/$sudoersfile", > } > } > > > Does that make sense? Not sure if the use-case maps up exactly to yours but > it sounds close enough that this might work for you and end up being a lot > simpler. > > - Eric Sorenson - N37 17.255 W121 55.738 - http://twitter.com/ahpook - > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > >Thanks for your detailed answer. It make perfect sense to implement thing that way, by including different resources based on which environment the client are defined in. But if the different projects had different requirements for their servers, would you extend your example to check on things like "if $evn == ''dev'' and $project == ''projectA'' then ....", or even "if $fqdn == ''client1.example.com'' then ..."? In a real request I just got from on the the projects at work they wanted to include a couple of classes just for one specific server, so I basically did something like this: -- code start -- class c_appserver::projectA { include baseclass include c_appserver if ($fqdn == ''client1.dev.example.com'') { include some::class } } -- code start -- If the "some::class" class were to be included on all development servers for the project I''d be using "if ($env == ''dev'') { ... }" instead. Comments? - Kenneth -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.