In the course of training and consulting with Puppet, the question of change management best practices has come up over and over again. On the edges, we have small teams that can get away with simply version controlling their code using an SCM as an incremental backups while rolling out change in a fairly adhoc fashion and larger teams that need branches, QA, and DEV environments, and perhaps even separate repositories for each module. There is also the issues of roll back and testing. We are curious how the community approaches these problems in hopes of developing some best practices. So what do you guys/gals do? -- Teyo Tyree :: www.reductivelabs.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Fri, Jul 17, 2009 at 1:06 PM, Teyo Tyree <teyo@reductivelabs.com> wrote:> In the course of training and consulting with Puppet, the question of > change management best practices has come up over and over again. On the > edges, we have small teams that can get away with simply version controlling > their code using an SCM as an incremental backups while rolling out change > in a fairly adhoc fashion and larger teams that need branches, QA, and DEV > environments, and perhaps even separate repositories for each module. There > is also the issues of roll back and testing. We are curious how the > community approaches these problems in hopes of developing some best > practices. So what do you guys/gals do? >The policy that I set up at my previous workplace with some measure of success involved a "dev" and "release" branch. You could make as many changes as you wanted to the dev release, but when it came time to promote it, you would have to send a ticket, have someone else review/test the release, then promote it to release using a custom tool that created a pointer (using svn externals) setting it as the current release tag. This worked pretty well, but it had some deficiencies too. For example, if no one else was available to review your release, it slowed things down, and sometimes tickets got lost. And sometimes bad releases DID get out. --Russell> > -- > Teyo Tyree :: www.reductivelabs.com > > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Hi,> In the course of training and consulting with Puppet, the question of > change management best practices has come up over and over again. On > the edges, we have small teams that can get away with simply version > controlling their code using an SCM as an incremental backups while > rolling out change in a fairly adhoc fashion and larger teams that > need branches, QA, and DEV environments, and perhaps even separate > repositories for each module. There is also the issues of roll back > and testing. We are curious how the community approaches these > problems in hopes of developing some best practices. So what do you > guys/gals do?Here we don''t (yet) have different code bases for production and development, but are considering it. Instead, we each have a clone of the manifests in our home-dirs and test new stuff by running: puppetd -t --environment <username> on relevant dev machines, then push/pull the changes into the central repository on the puppetmaster once everything seems ok. As we have different puppetmaster servers (more or less one for each customer), we try to share the most we can by putting almost everything in modules, stored in seperate repositories on github. Then using git-submodule (currently testing git-subtree [1] as a replacement) to glue them all together in one big repository on each puppetmaster. This forces us to write cross-platform manifests, in a "one application one module" fashion. Marc [1] http://github.com/apenwarr/git-subtree/tree/master --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
----- "Teyo Tyree" <teyo@reductivelabs.com> wrote:> In the course of training and consulting with Puppet, the question of > change management best practices has come up over and over again. On > the edges, we have small teams that can get away with simply version > controlling their code using an SCM as an incremental backups while > rolling out change in a fairly adhoc fashion and larger teams that > need branches, QA, and DEV environments, and perhaps even separate > repositories for each module. There is also the issues of roll back > and testing. We are curious how the community approaches these > problems in hopes of developing some best practices. So what do you > guys/gals do?We''ll be developing extensive practices in this area when we move to 0.25 and start using the environment support (that''s why we wanted it, actually: our change management practices are difficult without them). Generally speaking we plan to have all servers pointing to a "stable" environment where only bugfixes or emergency changes are introduced. We will develop in a testing/unstable branch and when we finish our testing, we will tag that as stable and move servers over to using this environment as the departments approve of the change. -- Digant C Kasundra <digant@stanford.edu> Technical Lead, ITS Unix Systems and Applications, Stanford University --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Fri, Jul 17, 2009 at 1:06 PM, Teyo Tyree<teyo@reductivelabs.com> wrote:> In the course of training and consulting with Puppet, the question of change > management best practices has come up over and over again. On the edges, we > have small teams that can get away with simply version controlling their > code using an SCM as an incremental backups while rolling out change in a > fairly adhoc fashion and larger teams that need branches, QA, and DEV > environments, and perhaps even separate repositories for each module. There > is also the issues of roll back and testing. We are curious how the > community approaches these problems in hopes of developing some best > practices. So what do you guys/gals do?We use environments extensively. We store all our configurations in VCS. All edits are done to a "common" branch. Teams responsible for specific platforms (say "mac") perform a VCS integrate from the common branch to a mac_unstable branch. The team members tend to live on their own unstable branch, and regularly bug fix in common and repeat the integrates to "mac_unstable". Once that has settled, a VCS integrate is performed from "mac_unstable" to "mac_testing". A larger group of pilot users not in the immediate team group live on this branch/environment. Once that has settled, a VCS integrate is performed from "mac_testing" to "mac_stable". This allows all the teams to follow their own release cycles, yet all still share a common code base. We have an extensive series of presubmit hooks that do things like run puppet --parseonly --detailed-exitcodes on each .pp file, that validate that file source urls like puppet:///module/file actually point to a real file, style guide enforcement like trailing commas, whitespace, tabs, etc etc. We also have test servers where we can do things like redirect all environments to refer to the "common" development environment for ease of testing fixes without going through the whole release process. We sit down every so often and look back at the developed code base to see where we can combine solutions from different groups and share as much configuration as possible. It''s actually a bit more complicated than this because our environments are more fine-grained than just "mac_unstable" so we can target different OS versions differently and different hardware types (workstation, laptop) differently, but the same principles all hold. Oh, and we also have matching apt repositories for the unstable/testing/stable environment suffixes, and perform simultaneous releases/promotions in apt and puppet.> -- > Teyo Tyree :: www.reductivelabs.com > > > >-- Nigel Kersten nigelk@google.com System Administrator Google, Inc. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Ok, I''ll bite :) warning: this is going to be long and complicated, i think this is the part which was most challenging in setting up our puppet infrastructure. some background and assumptions: - we have many sites / locations / data centers. - we have many different host types (e.g. server profiles) - we have site administrators (responseble only for their site) - we have global administrators, each one is responsible for a different topic globally. - we usually have a very short maintenance window on our servers (and its never on the same time across the different sites). - each host type might have a different maintenance window. we were looking for a way, of how we can have the same setup in all of our locations, but still managed by puppet so we came up with the following requirements: - every change should be done by puppet (unless really really critical and bla bla - never happened). - "global administrators" (or I don''t know how else to call them, in contrast to site administrators) are the only one to control their respective parts in the manifest. - site administrators should have a way to rollout changes immediately and override global settings if something breaks (like they used to before puppet). - everything must be visible to every administrator - every change must be tracked etc. - must provide a release cycle. - must be easy to audit configuration database / repository - Consists of all Global (aka standard) configuration. - Consists of all Sites customizations (or global configuration overrides). - Access Control, Change Log - Using Version Control -Ticketing System, Milestone management (e.g. downtimes) etc puppet modules ( we defined 3 types of modules ) - Host Type - primary purpose of a host --- Each host is associated with one host type --- Always inherits the Host base class for common policy - Service modules - used (shared) by other modules, e.g. ssh, syslog, apache, x11 etc. ---Avoids repeating the same code again and again ---Breaks down the scope of each module, allowing different module owners per topic ---Changes requires more attention as many host types might use the same service module ---Should be plug-and-play - Sites Modules - sites customizations --- The only place a site can edit his settings (Sites can not change global modules) --- Each site can only change his own site module --- Site modules provide a mechanism to override the global modules in a visible and auditable way --- As Site modules are not tagged (versioned) any change in the site specific settings, will be rolled out immediately. on top of all that we tag the modules, and associate them with a version: Versioning – The ability to refer (tag) to a certain state/PiT Host Type and Service modules are associated with a version, e.g. ssh v1.0 Site Modules are not tagged in order to reduce complexity Why ? Allows each module owner to “release” testing and stable versions of his module Sites subscribes to modules in a certain version, allowing each site to select which and when changes will be applied Environments – The ability to select a group of modules in a certain version Each site is associated with at least two environments(usually more) – Production and Testing: Testing – an environment which refer to versioned modules under a test or productive tag. Production – A selection of modules in a certain version that has already been approved for Production usage. Development environment – The state where the modules are in the latest check-in version, but are not tagged (e.g. not a testing candidate yet). Development --> Testing --> Production (Per Host Type) Each module owner should review the local sites adaptation, to find out the reasons for change - Improve communications - Improve host-type / service module quality - Auditing ensures that our infrastructure remains harmonized on top of that, we also have global environments, this is needed for services which are identical across the sites but and not under the site control, for example puppetmasters, ldap servers etc. hope it helps :) Ohad On Sat, Jul 18, 2009 at 4:06 AM, Teyo Tyree <teyo@reductivelabs.com> wrote:> In the course of training and consulting with Puppet, the question of > change management best practices has come up over and over again. On the > edges, we have small teams that can get away with simply version controlling > their code using an SCM as an incremental backups while rolling out change > in a fairly adhoc fashion and larger teams that need branches, QA, and DEV > environments, and perhaps even separate repositories for each module. There > is also the issues of roll back and testing. We are curious how the > community approaches these problems in hopes of developing some best > practices. So what do you guys/gals do? > -- > Teyo Tyree :: www.reductivelabs.com > > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
I try to explain how we implemented our workflow. We developed some strategies to make puppet module development easy to test and we have a way to rollout our configuration to different environments by simply taging the versions we think they are stable enough. Basically we use the power of the DVCS Git to get that done. Development or changes of puppet manifests: If we build a new Server we can choose between two different post install scrips. The standard script simply installs puppet and register the node on the puppetmaster for one of the three environments we have (testing->integration->production) The other one installs a puppetmaster and an empty git repo with a post-receive hook. This way it can be used as a development-server. And this is how it works: - take your copy of your puppet-modules repo and commit your changes (don''t push them to the real puppetmaster) - add the new development-server as a remote target to git. - push the branch to the development-server This is where the post-recieve hook kicks in. it copies the HEAD to the modules directory of the development-server (he has his own puppetmaster) and runs "puppetd --test --debug". That''s it! Just make sure the development-server includes all classes you want to test. You can also do cosmetics on your commits while you fix some syntax errors or small bugs, because you never pushed your changes to a public repo. Rollout: If the new version is ready and we believe everything is right, we push our changes to the puppetmaster (if there are changes and you have to merge, go back to your development-server and check if everythis is still ok). On the puppetmaster there is another post-receive hook that copies the HEAD to the testing environment. If everything went fine, we simply tag the version with a integration-YYYYMMDDxx tag and push that tag. the post-receive hook then copies that to the integration environment... and so on.. Rollback: This comes for free, simply tag an older version with a newer tag (and restore the damn file that ensure => absent killed from the backup :-) ). We don''t have that much servers in puppet at the moment ~80 and only 2 people working on the manifests so i don''t know if this scales if numbers grow. Maybe someone here with experience on larger setups has some suggestions on this? would be very welcome. I hope someone understands my poor english. Andy Am Friday 17 July 2009 22:06:09 schrieb Teyo Tyree:> In the course of training and consulting with Puppet, the question of > change management best practices has come up over and over again. On the > edges, we have small teams that can get away with simply version > controlling their code using an SCM as an incremental backups while rolling > out change in a fairly adhoc fashion and larger teams that need branches, > QA, and DEV environments, and perhaps even separate repositories for each > module. There is also the issues of roll back and testing. We are curious > how the community approaches these problems in hopes of developing some > best practices. So what do you guys/gals do?--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
I wanted to post sooner ... oh well... This is resonably complicated and
easier understood on the whiteboard in my office :)
Our build processes uses 3 environments, noop setting, SVN, inventory fakedb
(csv file), and custom reports
*ENVs*
1. *dev* - separate server, you can do whatever you want here, puppet can
rebuild this server in case you go to far :)
2. *test* environment - low impact machines - a place to apply changes to
a large group of machines without getting fired if something goes wrong
(although developers might yell at you)
3. *prod - *high impact machines - only when you are absolutely sure ....
*SVN:*
3 separately visioned directories
- *conf *- manifests + site configuration file
- *dev* - services and modules
- *global* - everything not supported by environments (tools=[external
node classifier, custom reports, build process code], plugins)
*INVENTORY* - defines node attributes for external_node_classifier (and
provisioning process)
*environment column - *the external_node_classifier will send a node to
this new environment if he is not currently there.
*Build Process:
*1. identify changes
2. move test agents to dev puppet server (via inventory fake db)
*DEV ENV:*
3. Iterative development
4. test
5. create a release
*TEST ENV:
*6. checkout latest test release (automatically forces everything into
SAFE MODE, noop=true)
7. activate environment (turn safe mode = false)
8. create production release
*PROD ENV:
** *6. checkout latest prod release (automatically forces everything into
SAFE MODE, noop=true)
7. activate environment (turn safe mode = false)*
*
*Reporting:*
create new reports dir ENV/RELEASE/SAFE_MODE/
for each report type (ENV/RELEASE/SAFE_MODE/) - one report per host
if the report already exists, then the puppetd run does not coincide
with a new release and emails should be sent if anything changes.
*Limitations:
*- puppet servers manage part of the process by changing the puppet.conf
file
- environments do not support plugins, external_node_classifiers, etc..
- noop can only be set on the client side(not per environment)
*
MIsc:
*there are some other minor points not mentioned.
- svn commit hooks do things like verify code and notify admins when
releases are created
- conf and dev repos are separate so that users can make conf changes on
the test environment (skipping the dev steps in the process) and so that
conf changes can be triggered by scripts (EX: rootpwchange script), and
maybe eventually a web interface.
- when relevent, puppet server checks for the hosts in its environment
(inventory file) and broadcasts a puppetrun call.
- other...
*Phase 2:*
1. Tickets are created in some kind of a ticketing system
2. Code releases reference tickets
3. system associates all infrastructure changes(from reports) - with the
corresponding build IDs. Probably by autogenerating tiki pages in the build
management system (trac or whatever)
For any change in the infrastructure, you can see the ticket, the source
changes and the resulting changes in the infrastructure. Then the
infrastrucutre change process becomes a well understood and auditable
process.
enjoy!
Dan
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---
We''ve started to feel the pain of this so have just got a process running. The main puppet tree lives in svn and consists of the majority of service modules, we''re only deploying to CentOS boxes so this makes it relatively easy. We have: * two environments - development, production * three puppetmasters - dev, testing, production * Jira system to log changes/requests/additions to application and service modules. Various application manifests/modules are svn:externals to the related project, that way ops team can own service/generic modules while application teams can develop app specific modules. The puppet trunk can be checked out, modules changed or different versioned externals pulled in, build a distribution, then release to the dev environment and test. Once a puppet release to production is required, a release is built from the puppet trunk and deployed to the testing master, then finally to the production master. Rollback is in the form of deploying the previous distribution. Issues -- The development master can be a bottleneck. -- Concurrent work streams on application / service modules can cause a release to be bad. This is our first stab at it, and i''d like to investigate how people are using environments and tags to always have a releasable puppet trunk. 2009/7/17 Teyo Tyree <teyo@reductivelabs.com>:> In the course of training and consulting with Puppet, the question of change > management best practices has come up over and over again. On the edges, we > have small teams that can get away with simply version controlling their > code using an SCM as an incremental backups while rolling out change in a > fairly adhoc fashion and larger teams that need branches, QA, and DEV > environments, and perhaps even separate repositories for each module. There > is also the issues of roll back and testing. We are curious how the > community approaches these problems in hopes of developing some best > practices. So what do you guys/gals do? > -- > Teyo Tyree :: www.reductivelabs.com > > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---