Puppet users - May 2012 - Using Puppet to manage preexisting servers

Good Afternoon Puppet List,

I''m investigating the possibility of using puppet to take over an
already existing cluster of servers. Ordinarily this doesn''t sound like
anything out the ordinary, but I''ve had my hands tied by the cluster
owner saying he doesn''t want anything to break/fail while I''m
wrestling the cluster to be under my control.

Ordinarily, I''d say lets upgrade your working practices while
we''re at it, and properly manage your whole workflow, manage access and
manage who can make changes. Lets start by rebuilding your boxes to
specification, and migrating your existing data over. Sounds great, "Client
says no" huh? - Now I need to rethink my strategy.

I was planning on using my current manifests, but I think I''ll end up
over writing some config file that had a critical setting in it, or change a
password somewhere. Usually, I''m not too worried about that, but it did
occur to me that I don''t have any real sanity checks in my modules.

	• Are there any recommended ways of using puppet to take over a production
environment?
	• How can I ensure that when I''m spreading my manifests across the
globe, I mean cluster, that any changes can be reverted?
		• I understand that every file puppet changes, gets backed up? Where?
(embarrassed he doesn''t know!)
	• Finally, can I force puppet to replace changed files, or is this a manual
process?

Thanks,
-- Mister Guru

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 16/05/12 13:07, Mister IT Guru wrote:
> Good Afternoon Puppet List,
>
> I''m investigating the possibility of using puppet to take over an
already existing cluster of servers. Ordinarily this doesn''t sound like
anything out the ordinary, but I''ve had my hands tied by the cluster
owner saying he doesn''t want anything to break/fail while I''m
wrestling the cluster to be under my control.
>
> Ordinarily, I''d say lets upgrade your working practices while
we''re at it, and properly manage your whole workflow, manage access and
manage who can make changes. Lets start by rebuilding your boxes to
specification, and migrating your existing data over. Sounds great, "Client
says no" huh? - Now I need to rethink my strategy.
>
> I was planning on using my current manifests, but I think I''ll end
up over writing some config file that had a critical setting in it, or change a
password somewhere. Usually, I''m not too worried about that, but it did
occur to me that I don''t have any real sanity checks in my modules.
>
> 	• Are there any recommended ways of using puppet to take over a production
environment?
I''ve done this a lot. Unfortunately the only way you''ll be
absolutely
sure that Puppet isn''t breaking something is to look at the log/report 
of each host. You can do breadth first: get Puppet everywhere just 
managing itself, then slowly add managed resources to Puppet. Or you can 
do depth first: get Puppet managing everything on one machine, prove to 
your cluster manager that this one machine still works, then teach other 
people how to convert a cluster node to Puppet and check the logs (might 
be faster in your case).
> 	• How can I ensure that when I''m spreading my manifests across
the globe, I mean cluster, that any changes can be reverted?
Not easily... Puppet has no inbuilt ''revert'' functionality and
how do
you know what state everything is in to start with? You could use Ralsh 
to query the state of all the resources you want to manage before you 
apply any changes. The log of a run will show a resource''s previous 
state and the state Puppet changed it to as well. Also be aware that the 
log is dependent on how the provider reports what it''s done. Most of
the
native Puppet resource types are ok (Service changed ''stopped''
to
''running'') but other native types (I can''t think of
any off hand) could
simply say "Changed ''absent'' to
''present''" and that could be a bit
ambiguous depending on the provider''s definition of "absent".
> 		• I understand that every file puppet changes, gets backed up? Where?
(embarrassed he doesn''t know!)
Look at the filebucket feature, generally stored on the Puppet Master in 
/var/lib/puppet/bucket: 
http://docs.puppetlabs.com/references/2.7.0/type.html#filebucket
> 	• Finally, can I force puppet to replace changed files, or is this a
manual process?
I''m not sure what you mean by changed files. If you have Puppet manage
a
file''s contents with the File type, it will continually put in that
file
what it thinks should be in there. Other providers can make inline edits 
of files, like Augeas or PuppetLab''s Parsedfile.
>
> Thanks,
> -- Mister Guru
>

-- 
Luke Bigum

Information Systems
Ph: +44 (0) 20 3192 2520
luke.bigum@lmax.com | http://www.lmax.com
LMAX, Yellow Building, 1A Nicholas Road, London W11 4AN


FX and CFDs are leveraged products that can result in losses exceeding
your deposit.  They are not suitable for everyone so please ensure you
fully understand the risks involved.  The information in this email is not
directed at residents of the United States of America or any other
jurisdiction where trading in CFDs and/or FX is restricted or prohibited
by local laws or regulations.

The information in this email and any attachment is confidential and is
intended only for the named recipient(s). The email may not be disclosed
or used by any person other than the addressee, nor may it be copied in
any way. If you are not the intended recipient please notify the sender
immediately and delete any copies of this message. Any unauthorised
copying, disclosure or distribution of the material in this e-mail is
strictly forbidden.

LMAX operates a multilateral trading facility.  Authorised and regulated 
by the Financial Services Authority (firm registration number 509778) and
is registered in England and Wales (number 06505809). 
Our registered address is Yellow Building, 1A Nicholas Road, London, W11
4AN.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

My first experience with puppet was puppetizing an already deployed 500-node HPC
cluster. The cluster admin was kind enough to pull a node out of the cluster so
I could test my manifest/modules on it without affecting production nodes. Once
I was confident I had it dialed in, I converted while racks at a time until all
nodes were under puppet''s watchful eye.

If this is not possible for you, you could use VMware or Virtualbox and clone a
server to a VM during a maintenance window and use that for your development.
Snapshots are very nice to have when testing your manifests as they allow rapid
and complete undos.

As for reverting changes on non-virtual servers, you can use puppet''s
filebucket -- it is well documented. I, however, am a great fan of having
regular, non-puppet backups from which to restore original files.

-- Peter (from phone)

On May 16, 2012, at 8:07 AM, Mister IT Guru <misteritguru@gmx.com> wrote:
> Good Afternoon Puppet List,
> 
> I''m investigating the possibility of using puppet to take over an
already existing cluster of servers. Ordinarily this doesn''t sound like
anything out the ordinary, but I''ve had my hands tied by the cluster
owner saying he doesn''t want anything to break/fail while I''m
wrestling the cluster to be under my control.
> 
> Ordinarily, I''d say lets upgrade your working practices while
we''re at it, and properly manage your whole workflow, manage access and
manage who can make changes. Lets start by rebuilding your boxes to
specification, and migrating your existing data over. Sounds great, "Client
says no" huh? - Now I need to rethink my strategy.
> 
> I was planning on using my current manifests, but I think I''ll end
up over writing some config file that had a critical setting in it, or change a
password somewhere. Usually, I''m not too worried about that, but it did
occur to me that I don''t have any real sanity checks in my modules.
> 
>    • Are there any recommended ways of using puppet to take over a
production environment?
>    • How can I ensure that when I''m spreading my manifests across
the globe, I mean cluster, that any changes can be reverted?
>        • I understand that every file puppet changes, gets backed up?
Where? (embarrassed he doesn''t know!)
>    • Finally, can I force puppet to replace changed files, or is this a
manual process?
> 
> Thanks,
> -- Mister Guru
> 
> -- 
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
> 
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Wed, May 16, 2012 at 5:07 AM, Mister IT Guru <misteritguru@gmx.com>
wrote:
> Ordinarily, I''d say lets upgrade your working practices while
we''re at it, and properly manage your whole workflow, manage access and
manage who can make changes. Lets start by rebuilding your boxes to
specification, and migrating your existing data over. Sounds great, "Client
says no" huh? - Now I need to rethink my strategy.

One of the tools that''s helpful in this situation is Puppet''s
no-op
mode.  This gives you the ability to report in advance on what would
change during a Puppet run.  It''s helpful for reassuring angsty
customers as well as making a list of what might need to be reverted
in an emergency.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Wed, May 16, 2012 at 12:08 PM, Aaron Grewell
<aaron.grewell@gmail.com>wrote:
> On Wed, May 16, 2012 at 5:07 AM, Mister IT Guru
<misteritguru@gmx.com>
> wrote:
> > Ordinarily, I''d say lets upgrade your working practices while
we''re at
> it, and properly manage your whole workflow, manage access and manage who
> can make changes. Lets start by rebuilding your boxes to specification, and
> migrating your existing data over. Sounds great, "Client says no"
huh? -
> Now I need to rethink my strategy.
>
>
> One of the tools that''s helpful in this situation is
Puppet''s no-op
> mode.  This gives you the ability to report in advance on what would
> change during a Puppet run.  It''s helpful for reassuring angsty
> customers as well as making a list of what might need to be reverted
> in an emergency.
>
 I concur to the advice all posters have given. I''ll add that the three
tools we''ve used to do stuff like this are:

1) puppet resource
2) blueprint
http://pratikamin.wordpress.com/2011/03/22/generate-puppet-recipe-from-running-system/(These
recipes that get generated really need hand tuning, as they are
massive.)
3) puppet in --noop mode to validate.

Generally we start with the manage nothing and go through and add resources
and services one at a time. It''s slow and tedious, and we
don''t like to do
it, but you sometimes have to.

Cheers,
Brian

--
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>

-- 
<http://aws.amazon.com/solutions/solution-providers/brandorr/>

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.