So, the usual problem -- we want to be able to test our manifests before putting them into full-scale no-holds-barred oh-shit-we-broke-teh-intarweb production. The two possible solutions I''ve considered so far are: 1) Set a tag on objects we only want to be run on our test machines, and run Puppet on the test machines with "please run objects tagged ''test''". I haven''t researched tags well enough to know if this is actually going to work as I expect it will, though. 2) Have two separate branches in bzr, and two separate Puppetmasters -- one for the testing machines to use, and one for production. Once things are tested, then merge from the ''test'' to the ''prod'' branch. I know the latter will work, and I''ve done it in the past (in a limited way) for CFEngine, but I''m really not liking the duplicated infrastructure and necessity of a merge for deployment. Tagging would be *so* much neater and tidier, but might also suffer from not being able to properly test *changes* to objects, rather than just new ones (duplicating the object and renaming it would probably fix this, though). Does anyone have any comments on my solutions, or better ways to solve this problem? - Matt -- when SuSE are doing better than you at publishing the tools they use, it''s a hint that maybe you suck. -- Andrew Suffield, debian-devel
On 05/09/06, Matthew Palmer <mpalmer@hezmatt.org> wrote:> So, the usual problem -- we want to be able to test our manifests before > putting them into full-scale no-holds-barred oh-shit-we-broke-teh-intarweb > production. The two possible solutions I''ve considered so far are: > > 1) Set a tag on objects we only want to be run on our test machines, and run > Puppet on the test machines with "please run objects tagged ''test''". I > haven''t researched tags well enough to know if this is actually going to > work as I expect it will, though. > > 2) Have two separate branches in bzr, and two separate Puppetmasters -- one > for the testing machines to use, and one for production. Once things are > tested, then merge from the ''test'' to the ''prod'' branch. > > I know the latter will work, and I''ve done it in the past (in a limited way) > for CFEngine, but I''m really not liking the duplicated infrastructure and > necessity of a merge for deployment. Tagging would be *so* much neater and > tidier, but might also suffer from not being able to properly test *changes* > to objects, rather than just new ones (duplicating the object and renaming > it would probably fix this, though). > > Does anyone have any comments on my solutions, or better ways to solve this > problem? > > - MattI would go for #2. My colleagues an I have been discussing how to do this in a larger environment where there are potentially hundreds/thousands of hosts and many change requests being made and each request needing to be scheduled for a certain date and time (along with customer notifications etc etc). Our idea, implemented with SVN, was to have a a repo structure something like: requests/ yyyymmdd/1234/ 5678/ approved/ yyyymmdd/5566/ 6789/ release/ The idea being that a change request would be made and assigned an ID. That ID would be an ''svn copy release requests/<ID>'' then the changes would be made to that ''request/<ID>'' branch, reviewed and once approved moved to the "approved" branch where a person or process would merge it into the "release" branch for deployment on the date required. I have to say this is all theory at the moment and it may not scale well and is definitely overkill for smaller environments. That said, I really like the idea of using svn/bzr to keep the test and prod environments seperate until changes are tested and "approved". regards matthew> -- > when SuSE are doing better than you at publishing the tools they use, it''s a > hint that maybe you suck. > -- Andrew Suffield, debian-devel > _______________________________________________ > Puppet-users mailing list > Puppet-users@madstop.com > https://mail.madstop.com/mailman/listinfo/puppet-users >
Matthew Palmer wrote:> > Does anyone have any comments on my solutions, or better ways to solve this > problem?The short answer, as always, is "it depends". What are your service quality requirements? If 10 seconds of downtime or misconfiguration will cost you $100billion, then I''d use a separate architecture, but if it''s more annoying than deadly, I''d use tagging. I haven''t run a production infrastructure in a while, but when I did (back in the cfengine days) I used classes in the way you''re talking about using tags, and it worked really well, with one caveat: Only very competent people can handle it, from what I can tell. That is, you might have no trouble doing this, but I wouldn''t plan on being able to hand it off to your junior admins to deal with. I would usually tag a few hosts with something like "deployment_testing_thingie", and then only deploy the test class on those hosts. Then I would slowly increase the number of hosts that were tagged this way until I was confident everything was working. Do realize, though, that you have to tag the hosts before you include the class, which in this case means you can''t tag them in the node definition unless that''s the same place doing the including -- if you tag in the node def and include in the parent node def, that won''t work. -- A computer lets you make more mistakes faster than any invention in human history--with the possible exceptions of handguns and tequila. -- Mitch Ratcliffe --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
Matthew Flanagan wrote:> > That said, I really like the idea of using svn/bzr to keep the test > and prod environments seperate until changes are tested and > "approved".I can''t speak to the suitability of the solution you''re discussing, but in general I think this is the Next Big Problem. In many ways, my primary goal for Puppet is for the actual administration to be so easy that this is now your biggest problem. I don''t purport to be trying to solve this problem (generally called Change Management) with Puppet, but I''d love to have some standard, accepted solutions to it. One thing I''ve also thought about, although this would only work on a smaller scale in more paranoid environments, is to always run Puppet in noop mode but then support a kind of local merge operation, where you could stepwise tell Puppet to make the changes it sees as necessary and then check its work. I''ve seen people who don''t trust the tools at all, and this would provide them a way to let Puppet help them but to still keep hands-on control. I do really like the idea of merging, either local state toward the central config, or vice-versa. -- A citizen of America will cross the ocean to fight for democracy, but won''t cross the street to vote in a national election. --Bill Vaughan --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com