Puppet users - Sep 2010 - Need advice on managing large Centos environment

I''m working on rolling out Puppet, but I''m stuck and I know
somebody has
solved this problem...

We have an engineering environment of around 200 Centos servers, plus a
production environment of roughly the same size. Currently, when we roll out
a new server, we do a ''yum update'' so the new server has the
latest
packages; however this means that just about every server has a different
set of package versions - a system rolled out today will have different
versions from one rolled out last month, and that will have different
versions from one rolled out last year.

This has bitten me in the past, where a feature developed on a recent system
failed to run on an older server, so I''m looking for a solution. I am
in the
middle of rolling out Puppet, and we have private mirrors of the yum repos,
so a solution could build on these.

I can see several possible solutions:
- manage changes to the yum repos, and use puppet to make sure that every
server is up to date w.r.t. the yum repo. However this makes it difficult to
roll out changes in a controlled way,or to rollback changes.
- use puppet to list all package versions, and manage the versions
explicitly in puppet, however this means that puppet is going to have
hundreds of entries, and I can imagine a situation where I upgrade one
package in puppet without realizing that it depends on a newer version of
another package, thus puppet and yum start fighting about which version of
the other package should be installed.
- always make sure that all systems are updated to the latest, which means a
slightly less stable environment, but avoids incompatibilities between
systems, and particularly the possibility that a feature developed on a
newer system will be asked to run on an older system.

Has anybody else been faced with this problem, and if so, how did you
resolve it?

Lezz Giles

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Sep 22, 2010, at 8:06 AM, Leslie Giles wrote:
> We have an engineering environment of around 200 Centos servers, plus a
production environment of roughly the same size. Currently, when we roll out a
new server, we do a ''yum update'' so the new server has the
latest packages; however this means that just about every server has a different
set of package versions - a system rolled out today will have different versions
from one rolled out last month, and that will have different versions from one
rolled out last year.
> 
> This has bitten me in the past, where a feature developed on a recent
system failed to run on an older server, so I''m looking for a solution.
I am in the middle of rolling out Puppet, and we have private mirrors of the yum
repos, so a solution could build on these.

My understanding is that RHEL (and CentOS by extension) won''t add or
remove any features within a major release. I have a similar situation with ~100
RHEL5 systems all at different versions of various RPMs and I''ve never
had a problem with anything I''m doing in Puppet.

The only thing that''s ever caused problems for me was when Puppet
and/or Augeas were updated in EPEL and a new system got the newer version before
the rest of the environment was really set up for it. If you''ve got all
these packages in a local repo (which is what I should be doing), you
won''t have to worry about that.

-- 
Rob McBroom
<http://www.skurfer.com/>

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi Lezz,

Another approach is to maintain your own repos by freezing them over time,
e.g. monthly pull the latest packages for your repos and put them as your
frozen repo, provide these frozen repos through puppet and once you have
validated all the updates in the frozen repos you can relase those to all
your servers.

Roll back is harder to maintain so you better make sure the updates in the
frozen repo works before pushing into production.  You don''t have to
list
each package in puppet but only those that you use/need and you can set them
to be "latest" this way the control relies on your repository.

Thanks.

On Wed, Sep 22, 2010 at 7:26 AM, Rob McBroom
<mailinglist0@skurfer.com>wrote:
> On Sep 22, 2010, at 8:06 AM, Leslie Giles wrote:
>
> We have an engineering environment of around 200 Centos servers, plus a
> production environment of roughly the same size. Currently, when we roll
out
> a new server, we do a ''yum update'' so the new server has
the latest
> packages; however this means that just about every server has a different
> set of package versions - a system rolled out today will have different
> versions from one rolled out last month, and that will have different
> versions from one rolled out last year.
>
> This has bitten me in the past, where a feature developed on a recent
> system failed to run on an older server, so I''m looking for a
solution. I am
> in the middle of rolling out Puppet, and we have private mirrors of the yum
> repos, so a solution could build on these.
>
>
> My understanding is that RHEL (and CentOS by extension) won''t add
or remove
> any features within a major release. I have a similar situation with ~100
> RHEL5 systems all at different versions of various RPMs and I''ve
never had a
> problem with anything I''m doing in Puppet.
>
> The only thing that''s ever caused problems for me was when Puppet
and/or
> Augeas were updated in EPEL and a new system got the newer version before
> the rest of the environment was really set up for it. If you''ve
got all
> these packages in a local repo (which is what I *should* be doing), you
> won''t have to worry about that.
>
> --
> Rob McBroom
> <http://www.skurfer.com/>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
>
puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>


-- 
Tony
http://blog.tonyskapunk.net

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Wed, 22 Sep 2010, Leslie Giles wrote:
> We have an engineering environment of around 200 Centos servers, plus a
> production environment of roughly the same size. Currently, when we roll
out
> a new server, we do a ''yum update'' so the new server has
the latest
> packages; however this means that just about every server has a different
> set of package versions - a system rolled out today will have different
> versions from one rolled out last month, and that will have different
> versions from one rolled out last year.
  ...
> Has anybody else been faced with this problem, and if so, how did you
> resolve it?
Let''s consider just the problem of ''package version skew; and 
solve it

1. Set up a local mirror of the centos external mirrors, and 
call it ''incoming''

2. Optionally, set a sub-mirror of ''incoming'' called
''vault'',
and mirror in a fashion that does NOT delete old content no 
longer present on ''incoming''

3. set a third mirror called ''testing'', which ''picks
and
chooses'' selected packages to test, and their dependencies 
(see the package: yum-utils for some tools to permit 
confirming that one has ''closure'' of those dependencies)

4. Test on your pre-deployment ''bench'' against
''testing''
until you have a change-set you wish to deploy throughout the 
universe of your boxes under management.  Obviously, several 
''testing'' mirrors can be set up, for differing classes of 
machines

5. FINALLY, have a master distribution mirror called ''rtm'' 
that has a change-set from a ''testing'' mirror deployed to it. 
Remove the stock repository specification files from
 	/etc/yum.repos.d/
and deploy local variants to taste, that point at ''rtm''. 
Again, several several ''rtm'' mirrors can be set up, for 
differing classes of machines

Something like this to ensure coherency of a enterprise wide 
deployment is usually mandated by a Change Control Board 
(explicitly, or implicitly).  Obviously, other aspects of an 
IT policy document will attend to getting the various mirrors 
properly recoverable in one''s backup strategy.  [there, the 
''testing'' mirrors are often NOT covered, as they are 
ephemeral as to their usefulness, and recoverable out of 
''vault'' (top down) or from a ''rtm'' (bottom
up)]

-- Russ herrold

end
=================================  .-- -... ---.. ... -.- -.--
Copyright (C) 2010 R P Herrold
       herrold@owlriver.com
    My words are not deathless prose,
       but they are mine.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
> Has anybody else been faced with this problem, and if so, how did you
> resolve it?
use Spacewalk for patch management. At least at your scale this makes
sense. It makes manual repository management a lot easier.

cheers pete
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkybaoUACgkQbwltcAfKi3/lsgCgsMztzUKp6lfmlRTfRG1Tx2nJ
BQwAn13Sinshi72yPLf4etnPWrSSGDID
=uLl+
-----END PGP SIGNATURE-----

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

I can also recommend using frozen repos, with a testing cycle every time you
update them...
I a while ago I wrote a small web app[1] to collect all versions of all
packages across all of your hosts, you might find it useful to know which
package is installed on which server... in anycase, if you freeze your
repos, a yum update on all of your servers will also ensure compliance :)

[1] http://github.com/ohadlevy/dish

Ohad

On Thu, Sep 23, 2010 at 4:56 PM, Peter Meier <peter.meier@immerda.ch>
wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> > Has anybody else been faced with this problem, and if so, how did you
> > resolve it?
>
> use Spacewalk for patch management. At least at your scale this makes
> sense. It makes manual repository management a lot easier.
>
> cheers pete
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAkybaoUACgkQbwltcAfKi3/lsgCgsMztzUKp6lfmlRTfRG1Tx2nJ
> BQwAn13Sinshi72yPLf4etnPWrSSGDID
> =uLl+
> -----END PGP SIGNATURE-----
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
>
puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Thu, Sep 23, 2010 at 12:34 PM, Ohad Levy <ohadlevy@gmail.com> wrote:
> I can also recommend using frozen repos, with a testing cycle every time
> you update them...
> I a while ago I wrote a small web app[1] to collect all versions of all
> packages across all of your hosts, you might find it useful to know which
> package is installed on which server... in anycase, if you freeze your
> repos, a yum update on all of your servers will also ensure compliance :)
>
> +1 for this.
We manage 750 CentOS boxes this way. We create a new server and/or compute
node using SystemImager. That image includes /etc/puppet, but not
/etc/puppet/ssl. /etc/rc.local is used to launch puppet, which then --when
the key is signed-- begins the process of bringing the box in synch with the
rest of the cluster.

We yum-update the boxes from a frozen repo, and when the update is done a
new image is taken and added to SytemImager. We step very lightly about
yum-updates, especially if a reboot is required.

~Charles~

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Sep 22, 2010, at 14:06 , Leslie Giles wrote:
> I''m working on rolling out Puppet, but I''m stuck and I
know somebody has solved this problem...
> 
> We have an engineering environment of around 200 Centos servers, plus a
production environment of roughly the same size. Currently, when we roll out a
new server, we do a ''yum update'' so the new server has the
latest packages; however this means that just about every server has a different
set of package versions - a system rolled out today will have different versions
from one rolled out last month, and that will have different versions from one
rolled out last year.
This might be a bit OT on this list but....

Usually operating system vendors do stable releases for that purpose. They
invest a lot of effort to provide a stable version of a certain set of
packages. Once released, they tend to not update software versions, except for
updates which will fix security patches.

So really, if you need a stable environment, use the stable version of your
distribution and DO make security updates. Doing so never caused any harm to my
systems.

The next thing is that you have to somehow migrate your systems from stable
release to stable release. Thats where puppet comes into play.  You simply
apply your manifests on the distribution''s new release, and test the
major
service provided by the server.
> This has bitten me in the past, where a feature developed on a recent
system failed to run on an older server, so I''m looking for a solution.
I am in the middle of rolling out Puppet, and we have private mirrors of the yum
repos, so a solution could build on these.
> I can see several possible solutions:
> - manage changes to the yum repos, and use puppet to make sure that every
server is up to date w.r.t. the yum repo. However this makes it difficult to
roll out changes in a controlled way,or to rollback changes.
Same here, there shouldn''t be any issues with updates as long as you
don''t
upgrade a system from one major distribution release to the next.  If a
security update causes any harm to your application, you should fix it too.
> - use puppet to list all package versions, and manage the versions
explicitly in puppet, however this means that puppet is going to have hundreds
of entries, and I can imagine a situation where I upgrade one package in puppet
without realizing that it depends on a newer version of another package, thus
puppet and yum start fighting about which version of the other package should be
installed.
Specifying each installed package''s version manually in your puppet
manifest doesn''t sound like a wise idea to me.
> - always make sure that all systems are updated to the latest, which means
a slightly less stable environment, but avoids incompatibilities between
systems, and particularly the possibility that a feature developed on a newer
system will be asked to run on an older system.
I think you should invest some work to provide an environment to your
developers which better fits your production environment. If they use
applications or libraries provided by your operating system, they need to make
sure, that as soon as you upgrade your system, their code is still in a
workable state.

If they don''t want to do that, either you or them needs to ship their
application bundled with the appropriate set of libraries, so you decouple your
application release cycles from your distribution''s one.


Regards, Stefan.










-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi,

You can use stuff like cobbler with puppet.  The cobbler server has
all the repositories configured via its Web interface and the client
nodes will have the repositories that is present on the cobbler
server. You then only one system (cobbler server)  that manages the
repository. The client machine just gets the updated file once cobbler
does the reposync from the External Repo. I have tested it and it
works fine for me.

Regards,
Kevin

On Sep 24, 8:46 am, Stefan Schlesinger <s...@ono.at>
wrote:
> On Sep 22, 2010, at 14:06 , Leslie Giles wrote:
>
> > I''m working on rolling out Puppet, but I''m stuck and
I know somebody has solved this problem...
>
> > We have an engineering environment of around 200 Centos servers, plus
a production environment of roughly the same size. Currently, when we roll out a
new server, we do a ''yum update'' so the new server has the
latest packages; however this means that just about every server has a different
set of package versions - a system rolled out today will have different versions
from one rolled out last month, and that will have different versions from one
rolled out last year.
>
> This might be a bit OT on this list but....
>
> Usually operating system vendors do stable releases for that purpose. They
> invest a lot of effort to provide a stable version of a certain set of
> packages. Once released, they tend to not update software versions, except
for
> updates which will fix security patches.
>
> So really, if you need a stable environment, use the stable version of your
> distribution and DO make security updates. Doing so never caused any harm
to my
> systems.
>
> The next thing is that you have to somehow migrate your systems from stable
> release to stable release. Thats where puppet comes into play.  You simply
> apply your manifests on the distribution''s new release, and test
the major
> service provided by the server.
>
> > This has bitten me in the past, where a feature developed on a recent
system failed to run on an older server, so I''m looking for a solution.
I am in the middle of rolling out Puppet, and we have private mirrors of the yum
repos, so a solution could build on these.
> > I can see several possible solutions:
> > - manage changes to the yum repos, and use puppet to make sure that
every server is up to date w.r.t. the yum repo. However this makes it difficult
to roll out changes in a controlled way,or to rollback changes.
>
> Same here, there shouldn''t be any issues with updates as long as
you don''t
> upgrade a system from one major distribution release to the next.  If a
> security update causes any harm to your application, you should fix it too.
>
> > - use puppet to list all package versions, and manage the versions
explicitly in puppet, however this means that puppet is going to have hundreds
of entries, and I can imagine a situation where I upgrade one package in puppet
without realizing that it depends on a newer version of another package, thus
puppet and yum start fighting about which version of the other package should be
installed.
>
> Specifying each installed package''s version manually in your
puppet
> manifest doesn''t sound like a wise idea to me.
>
> > - always make sure that all systems are updated to the latest, which
means a slightly less stable environment, but avoids incompatibilities between
systems, and particularly the possibility that a feature developed on a newer
system will be asked to run on an older system.
>
> I think you should invest some work to provide an environment to your
> developers which better fits your production environment. If they use
> applications or libraries provided by your operating system, they need to
make
> sure, that as soon as you upgrade your system, their code is still in a
> workable state.
>
> If they don''t want to do that, either you or them needs to ship
their
> application bundled with the appropriate set of libraries, so you decouple
your
> application release cycles from your distribution''s one.
>
> Regards, Stefan.
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Thu, Sep 23, 2010 at 12:34 PM, Ohad Levy
<ohadlevy@gmail.com<mailto:ohadlevy@gmail.com>> wrote:
I can also recommend using frozen repos, with a testing cycle every time you
update them...
I a while ago I wrote a small web app[1] to collect all versions of all packages
across all of your hosts, you might find it useful to know which package is
installed on which server... in anycase, if you freeze your repos, a yum update
on all of your servers will also ensure compliance :)


most likely OT:
you can also use func (https://fedorahosted.org/func/) which has a nifty
inventory app (https://fedorahosted.org/func/wiki/FuncInventory) that can list
all installed pkgs plus much more.  the output can be stored in git, making it
easy to see changes in pkg versions and such.  func has much more functionality
than just inventorying and can be quite a useful tool.  i find it useful for
running one-off type commands on an arbitrary group of systems.  it''s
quite easy to deploy func with puppet or via your provisioning process.


byron

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.