So I''ve got puppet running, and I''m quite pleased with it... but I''ve got one (probably stupid) question. Management is nervous about Puppet making changes to production boxes without a human overseeing its behavior. Is there a way to have the clients report differences in state without making changes, and then allowing operators to active those changes via some mechanism (puppetrun)? Pointers to articles or faq entries appreciated... Thanks Mark Farver
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Mark Farver wrote: | So I''ve got puppet running, and I''m quite pleased with it... but I''ve | got one (probably stupid) question. | | Management is nervous about Puppet making changes to production boxes | without a human overseeing its behavior. Is there a way to have the | clients report differences in state without making changes, and then | allowing operators to active those changes via some mechanism | (puppetrun)? | The --noop option will tell you what Puppet intends to do without doing it. You can also use a functions like Tags (see the Wiki - http://reductivelabs.com/trac/puppet/wiki/UsingTags) to selectively manage resources. The new multiple environments function might also be worth examining - http://reductivelabs.com/trac/puppet/wiki/UsingMultipleEnvironments Hope that helps. Regards James Turnbull - -- James Turnbull (james@lovedthanlost.net) - -- Author of: - - Pulling Strings with Puppet (http://www.amazon.com/gp/product/1590599780/) - - Pro Nagios 2.0 (http://www.amazon.com/gp/product/1590596099/) - - Hardening Linux (http://www.amazon.com/gp/product/1590594444/) -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHoiwT9hTGvAxC30ARAq++AJ0dIe+/FH+uy4swzD8E2YQlGLOEDACZAZYY N4p6IfjgOVL/L9D2TnJjKsU=+6lP -----END PGP SIGNATURE-----
Mark Farver wrote:> So I''ve got puppet running, and I''m quite pleased with it... but I''ve > got one (probably stupid) question. > > Management is nervous about Puppet making changes to production boxes > without a human overseeing its behavior. Is there a way to have the > clients report differences in state without making changes, and then > allowing operators to active those changes via some mechanism > (puppetrun)?This is simply a matter of QA''ing the changes IMHO. If no QA process exists, then the concerns of Management are justified. If on the other hand, changes can be tested before they are associated with production nodes and Management is still nervous, then I''d say that there''s nothing puppet can do in that environment. After all, the whole idea of puppet is to provide a road leading toward minimal human interaction thereby reducing or eliminating the risks associated with that style of management. So back to QA, I personally believe the best approach is to run another instance of puppetmasterd on a non-standard port (8141 in my case) that checks out a testing branch of whatever version control is being used. Dedicated clients then pull from that instance of puppetmasterd and in turn, provide an environment to test development code (e.g. manifests). Once integrity is verified, the testing branch is then merged with the production branch which is checked out by the production instance of puppetmasterd. http://reductivelabs.com/trac/puppet/wiki/BranchTesting Best, Adam> > Pointers to articles or faq entries appreciated... > > Thanks > Mark Farver > _______________________________________________ > Puppet-users mailing list > Puppet-users@madstop.com > https://mail.madstop.com/mailman/listinfo/puppet-users > >
I beg to differ... There may be circumstances in which this is simply impossible. When you are applying changes designed by one department onto a server managed by another department serving an application built by a third department supported by a fourth department you find you have a lot of cooks in the kitchen. Complex manifests tend to have a lot of conditional processing which can (i) break compilation or (ii) "fix" changes^Wproblems other people have made on the server. We run Puppet in --noop mode by default. Then we use the YAML reporting functionality to generate "Proposed Work" reports. Then, once approved Puppet is run in --no-noop mode. This gives all interested parties a chance to look at the proposed work before it is carried out. It also creates a closed feedback loop between the plethora of interested parties, something previously unheard of here. Using tags unfortunately means that the untagged classes never make it into the reports, otherwise we''d be using them. We also use a very similar system to Adam for QA in addition to the above - for the very same reasons. Regards, Derek -----Original Message----- From: puppet-users-bounces@madstop.com [mailto:puppet-users-bounces@madstop.com] On Behalf Of Adam Kosmin Sent: 31 January 2008 20:38 To: Puppet User Discussion Subject: Re: [Puppet-users] Avoiding Automated changes Mark Farver wrote:> So I''ve got puppet running, and I''m quite pleased with it... but I''ve > got one (probably stupid) question. > > Management is nervous about Puppet making changes to production boxes > without a human overseeing its behavior. Is there a way to have the > clients report differences in state without making changes, and then > allowing operators to active those changes via some mechanism > (puppetrun)?This is simply a matter of QA''ing the changes IMHO. If no QA process exists, then the concerns of Management are justified. If on the other hand, changes can be tested before they are associated with production nodes and Management is still nervous, then I''d say that there''s nothing puppet can do in that environment. After all, the whole idea of puppet is to provide a road leading toward minimal human interaction thereby reducing or eliminating the risks associated with that style of management. So back to QA, I personally believe the best approach is to run another instance of puppetmasterd on a non-standard port (8141 in my case) that checks out a testing branch of whatever version control is being used. Dedicated clients then pull from that instance of puppetmasterd and in turn, provide an environment to test development code (e.g. manifests). Once integrity is verified, the testing branch is then merged with the production branch which is checked out by the production instance of puppetmasterd. http://reductivelabs.com/trac/puppet/wiki/BranchTesting Best, Adam> > Pointers to articles or faq entries appreciated... > > Thanks > Mark Farver > _______________________________________________ > Puppet-users mailing list > Puppet-users@madstop.com > https://mail.madstop.com/mailman/listinfo/puppet-users > >_______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users ------------------------------------------------------------------------ For important statutory and regulatory disclosures and more information about Barclays Capital, please visit our web site at http://www.barcap.com. Internet communications are not secure and therefore the Barclays Group does not accept legal responsibility for the contents of this message. Although the Barclays Group operates anti-virus programmes, it does not accept responsibility for any damage whatsoever that is caused by viruses being passed. Any views or opinions presented are solely those of the author and do not necessarily represent those of the Barclays Group. Replies to this email may be monitored by the Barclays Group for operational or business reasons. Barclays Capital is the investment banking division of Barclays Bank PLC, a company registered in England (number 1026167) with its registered office at 1 Churchill Place, London, E14 5HP. This email may relate to or be sent from other members of the Barclays Group. ------------------------------------------------------------------------
On Feb 1, 2008 1:35 AM, <Derek.Whayman@barclayscapital.com> wrote:> We also use a very similar system to Adam for QA in addition to the > above - for the very same reasons.We''ve still got a primarily "development" or "testing" environment, with only a few real production systems. However when we upgrade to Puppet 0.24.1, which will be soon, we will start using multiple environments: http://reductivelabs.com/trac/puppet/wiki/UsingMultipleEnvironments With our current environment though, our machines are all redundant pairs. We typically test changes on node ''b'', make sure everything is functional, and then apply to node ''a''. Its a bit more manual than I''d like, but its a stop gap until we have the environments configured with the new puppet version. Its a growing (and learning) process.
Hello...
----- "Joshua Timberman" <joshua.timberman@gmail.com> wrote:
| On Feb 1, 2008 1:35 AM, <Derek.Whayman@barclayscapital.com> wrote:
| > We also use a very similar system to Adam for QA in addition to the
| > above - for the very same reasons.
|
| We''ve still got a primarily "development" or
"testing" environment,
| with only a few real production systems. However when we upgrade to
| Puppet 0.24.1, which will be soon, we will start using multiple
| environments:
|
| http://reductivelabs.com/trac/puppet/wiki/UsingMultipleEnvironments
|
| With our current environment though, our machines are all redundant
| pairs. We typically test changes on node ''b'', make sure
everything is
| functional, and then apply to node ''a''. Its a bit more
manual than
| I''d
| like, but its a stop gap until we have the environments configured
| with the new puppet version. Its a growing (and learning) process.
|
For file serving I do something similar to:
fact:
# adapted to your setup
# assumes you name servers like
# database-a and database-b
# otherserver-a otherserver-b
# salt to taste
Facter.add("typeofserver") do
setcode do
case type=%x{hostname}.chomp
when /[a-z]-a$/
"Aserver"
when /[a-z]-b$/
"Bserver"
...
class somefile {
file { "somefile":
path => /etc/dir/file.conf
source => [
"puppet:///dist/file.conf.$hostname",
"puppet:///dist/file.conf.$typeofserver",
"puppet:///dist/file.conf"
]
...
then in your puppet dist directory:
dist/file.conf # production conf
dist/file.conf.Bserver # test on "B" servers first, if OK move to
file.conf
dist/file.conf.database-b # test on one specific server
Make sense?
_______________________________________________
| Puppet-users mailing list
| Puppet-users@madstop.com
| https://mail.madstop.com/mailman/listinfo/puppet-users
--
Christopher McCrory
"The guy that keeps the servers running"
To the optimist, the glass is half full.
To the pessimist, the glass is half empty.
To the engineer, the glass is twice as big as it needs to be.
Basically, this is what we''re doing:
Myself and Tony have sort rules which sort these noop runs into folders in
Outlook, where the changes from a puppetd noop run are emailed. We only ever
expect changes to systems we''re actively changing, and have a process
outlined for making changes to live systems. Generally no changes are ever made,
or ''runpuppet'' is ran shortly after commiting them to the
/etc/puppet repository.
~/puppet/files/runpuppet:
aj@gatekeeper <mailto:aj@gatekeeper> ~/puppet/files$ cat runpuppet
#!/bin/bash
# this file is installed via Puppet, edits will be overwritten
hostname=`hostname -f`
echo "running puppet on $hostname, standby."
/usr/sbin/puppetd --test --pluginsync --vardir /var/lib/puppet --ssldir
/var/lib/puppet/ssl --factpath /var/lib/puppet/lib/facter
~/puppet/files/runpuppet-noop:
#!/bin/bash
# this file is installed via Puppet, edits will be overwritten
hostname=`hostname -f`
echo "running puppet on $hostname, standby."
/usr/sbin/puppetd -t --pluginsync --puppetdlockfile
/var/lib/puppet/state/puppetdnooplock --vardir /var/lib/puppet --ssldir
/var/lib/puppet/ssl --factpath /var/lib/puppet/lib/facter --noop --color
false|grep noop|egrep -v ''(checksum|Git)''|mail -s "puppet
noop run on $hostname" -e puppet
relevant manifests:
cron {
puppetd:
command => "/usr/sbin/runpuppet-noop",
hour => "*/1", minute => "0", user =>
root,
environment => [ "MAILTO=\"\"",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" ],
require => [ File["/usr/sbin/runpuppet-noop"],
Mailalias["puppet"] ];
}
file {
"/usr/sbin/runpuppet":
source => "puppet:///files/runpuppet",
ensure => present,
owner => root, group => root, mode => 755;
"/usr/sbin/runpuppet-noop":
source => "puppet:///files/runpuppet-noop",
ensure => present,
owner => root, group => root, mode => 755;
}
mailalias {
puppet:
ensure => present,
target => "/etc/aliases",
recipient => [ "arjuna.christensen@maxnet.co.nz
<mailto:arjuna.christensen@maxnet.co.nz> ", "
<mailto:tony.jenness@maxnet> tony.jenness@maxnet
<mailto:tony.jenness@maxnet.co.nz> .co.nz" ],
notify => Exec["newaliases"];
}
exec {
"newaliases":
refreshonly => true;
}
for example:
puppet noop run on mailaccess02.maxnet.net.nz
root [root@mailaccess02.maxnet.net.nz]
To: puppet@mailaccess02.maxnet.net.nz
Cc:
notice:
//Node[mailaccess02]/mail::access::node/mailproxy/File[/usr/local/bin/get_perdition_mapping.pl]/source:
is {md5}28d14902b0c7cce819a587843dac2cfa, should be
puppet:///files/get_perdition_mapping.pl (noop)
Regards,
AJ (Fujin)
Sorry about the formatting - OWA
________________________________
From: puppet-users-bounces@madstop.com on behalf of Christopher McCrory
Sent: Sat 2/02/2008 7:18 a.m.
To: Puppet User Discussion
Subject: Re: [Puppet-users] Avoiding Automated changes
Hello...
----- "Joshua Timberman" <joshua.timberman@gmail.com> wrote:
| On Feb 1, 2008 1:35 AM, <Derek.Whayman@barclayscapital.com> wrote:
| > We also use a very similar system to Adam for QA in addition to the
| > above - for the very same reasons.
|
| We''ve still got a primarily "development" or
"testing" environment,
| with only a few real production systems. However when we upgrade to
| Puppet 0.24.1, which will be soon, we will start using multiple
| environments:
|
| http://reductivelabs.com/trac/puppet/wiki/UsingMultipleEnvironments
|
| With our current environment though, our machines are all redundant
| pairs. We typically test changes on node ''b'', make sure
everything is
| functional, and then apply to node ''a''. Its a bit more
manual than
| I''d
| like, but its a stop gap until we have the environments configured
| with the new puppet version. Its a growing (and learning) process.
|
For file serving I do something similar to:
fact:
# adapted to your setup
# assumes you name servers like
# database-a and database-b
# otherserver-a otherserver-b
# salt to taste
Facter.add("typeofserver") do
setcode do
case type=%x{hostname}.chomp
when /[a-z]-a$/
"Aserver"
when /[a-z]-b$/
"Bserver"
...
class somefile {
file { "somefile":
path => /etc/dir/file.conf
source => [
"puppet:///dist/file.conf.$hostname",
"puppet:///dist/file.conf.$typeofserver",
"puppet:///dist/file.conf"
]
...
then in your puppet dist directory:
dist/file.conf # production conf
dist/file.conf.Bserver # test on "B" servers first, if OK move to
file.conf
dist/file.conf.database-b # test on one specific server
Make sense?
_______________________________________________
| Puppet-users mailing list
| Puppet-users@madstop.com
| https://mail.madstop.com/mailman/listinfo/puppet-users
--
Christopher McCrory
"The guy that keeps the servers running"
To the optimist, the glass is half full.
To the pessimist, the glass is half empty.
To the engineer, the glass is twice as big as it needs to be.
_______________________________________________
Puppet-users mailing list
Puppet-users@madstop.com
https://mail.madstop.com/mailman/listinfo/puppet-users
_______________________________________________
Puppet-users mailing list
Puppet-users@madstop.com
https://mail.madstop.com/mailman/listinfo/puppet-users
The scripts are a good solution, I assume you trigger then using ssh? What I''d really love is for puppetrun to have an option that overrides noop mode on the client. Then the clients could have puppetd operating as a daemon and operators could trigger/approve updates using the puppetrun command. I''ll have to take a look at the code... Thanks Mark Farver On Feb 2, 2008 6:22 PM, Arjuna Christensen <arjuna.christensen@maxnet.co.nz> wrote:> > > > Basically, this is what we''re doing: > > Myself and Tony have sort rules which sort these noop runs into folders in > Outlook, where the changes from a puppetd noop run are emailed. We only ever > expect changes to systems we''re actively changing, and have a process > outlined for making changes to live systems. Generally no changes are ever > made, or ''runpuppet'' is ran shortly after commiting them to the /etc/puppet > repository. >
Yes, manual runs are triggered via ssh (for i in `cat nodes`; do ssh $i sudo runpuppet; done). Automated noop runs are handled via cron. I *also* would for puppetrun to override noop mode on the client, that''d be awesome. Arjuna Christensen | Systems Engineer Maximum Internet Ltd 7a Parkhead Pl, Albany, North Shore, 0632 | PO Box 8006, Auckland, 1150, NZ DDI: + 64 9 913 9683 | Ph: +64 9 915 1825 | Fax:: +64 9 300 7227 arjuna.christensen@maxnet.co.nz| www.maxnet.co.nz ________________________________ Maxnet | mission critical internet ________________________________ This email (including any attachments) is confidential and intended only for the person to whom it is addressed. If you have received this email in error, please notify the sender immediately and erase all copies of this message and attachments. The views expressed in this email do not necessarily reflect those held by Maxnet. -----Original Message----- From: puppet-users-bounces@madstop.com [mailto:puppet-users-bounces@madstop.com] On Behalf Of Mark Farver Sent: Thursday, 7 February 2008 3:00 p.m. To: Puppet User Discussion Subject: Re: [Puppet-users] Avoiding Automated changes The scripts are a good solution, I assume you trigger then using ssh? What I''d really love is for puppetrun to have an option that overrides noop mode on the client. Then the clients could have puppetd operating as a daemon and operators could trigger/approve updates using the puppetrun command. I''ll have to take a look at the code... Thanks Mark Farver On Feb 2, 2008 6:22 PM, Arjuna Christensen <arjuna.christensen@maxnet.co.nz> wrote:> > > > Basically, this is what we''re doing: > > Myself and Tony have sort rules which sort these noop runs into folders in > Outlook, where the changes from a puppetd noop run are emailed. We only ever > expect changes to systems we''re actively changing, and have a process > outlined for making changes to live systems. Generally no changes are ever > made, or ''runpuppet'' is ran shortly after commiting them to the /etc/puppet > repository. >_______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users