Jake - USPS
2011-Nov-18 15:26 UTC
[Puppet Users] Managing a few resources on a few systems
We have an environment of thousands of servers. Up to this point we have been using puppet to manage resources common to all systems (ntp, ssh, etc) which are managed automatically (without manually assigning class/module to a system) and also resources for applications that make up a majority of our environment (Oracle DB, WebSphere, etc) which are assigned via ENC to specific systems (foreman hostgroups). What I am now running into are requests to modify a small amount of systems for a few resources. Just recently a small new app needs 2 limits.conf entries on like 8 systems. It seems like a waste of time to me to craft and manage a recipe for this app to manage these resources. But, it would still be nice to have this managed by puppet somehow for all the benefits puppet brings (standardization, documentation, etc). Also, I see lag time between crafting a recipe after a request and then pushing the recipe out to the environments before it would be applied to a system so that the app can then run/ install (although I know we can add it manually in the meantime, but then your sort of doubling up on work). How do other people handle managing ''one offs'' like this? I''m wondering if there is a way to create a generic limits.conf module that could take parameters from an ENC somehow and then manage the resources? That way I do not need to make new modules for app that maybe just modify limits.conf or sysctl.conf with a couple of lines, and then I have the changes documented and managed. Or do people just create a new module for every little app that comes along? Thanks! Jake -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Christopher Wood
2011-Nov-18 15:54 UTC
Re: [Puppet Users] Managing a few resources on a few systems
inline On Fri, Nov 18, 2011 at 07:26:45AM -0800, Jake - USPS wrote:> We have an environment of thousands of servers. Up to this point we > have been using puppet to manage resources common to all systems (ntp, > ssh, etc) which are managed automatically (without manually assigning > class/module to a system) and also resources for applications that > make up a majority of our environment (Oracle DB, WebSphere, etc) > which are assigned via ENC to specific systems (foreman hostgroups). > > What I am now running into are requests to modify a small amount of > systems for a few resources. Just recently a small new app needs 2 > limits.conf entries on like 8 systems. It seems like a waste of time > to me to craft and manage a recipe for this app to manage these > resources. But, it would still be nice to have this managed by puppet > somehow for all the benefits puppet brings (standardization, > documentation, etc). Also, I see lag time between crafting a recipe > after a request and then pushing the recipe out to the environments > before it would be applied to a system so that the app can then run/ > install (although I know we can add it manually in the meantime, but > then your sort of doubling up on work).If you put the effort into revision-controlling and puppetizing any specific configuration item, then in two years when somebody wonders what is configured on a system they will be able to dig into your manifests and know. It''s well worth the initial wait. You can always, depending on your config, "puppet kick" the hosts which need changing right now. Or somehow orchestrate that these reapply the current config right now. If you aren''t able to orchestrate, then the cheap cousin might be uploading the puppet manifest to the hosts and having puppet configure that. When the regular run comes around it''ll do nothing since limits.conf is already the new one. (This option is vastly less preferable to orchestration.) More specifically, there''s an example for modifying limits.conf with the augeas type: http://projects.puppetlabs.com/projects/1/wiki/Puppet_Augeas#/etc/security/limits.conf> How do other people handle managing ''one offs'' like this? I''m > wondering if there is a way to create a generic limits.conf module > that could take parameters from an ENC somehow and then manage the > resources? That way I do not need to make new modules for app that > maybe just modify limits.conf or sysctl.conf with a couple of lines, > and then I have the changes documented and managed. Or do people just > create a new module for every little app that comes along?node "specialserver.mine.com" { # everything that a generic host needs in our environment # snmp, ssh, ldap auth, puppet, apt/yum sources, etc. class { ''standardhost'': } # special shiny thing that only this subset of hosts does # inside this module would be your lookups, I suppose class { ''limits'': } } This doesn''t scale way, way up without automagic manifest generation, but it''s useful at our scale and number of special servers. It''s also fairly easy to see in the puppet tree which nodes get the limits class/module. On the other hand, your limits class could go everywhere and check if it''s supposed to apply the new limits.conf file. That might mean puppet cycles everywhere for only eight servers and perhaps not worth the cost. (I take it by your domain that you scale way, way, way up. I''m not sure of your values for "might" and "perhaps".) To arrange networking on the initial server powerup I use a magic variable in the node definition which the networking class applies. It''s a bit horrible, but then some hosts have fairly twisty networking.> Thanks! > Jake > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Luke Bigum
2011-Nov-18 15:54 UTC
Re: [Puppet Users] Managing a few resources on a few systems
I would create a custom define to add/edit limits.conf and sysctl.conf lines into some "base" module, then call that define in a simple module. I don''t know much about your internal processes but it shouldn''t really be that much work - you''ll probably spend more time documenting it than writing it ;-) If you make the limits.conf and sysctl.conf defines reusable enough you can avoid problems down the track with a dozen different admins editing these files a dozen different ways (exec sed vs augeas vs file resource, etc). Personally I''ve used augeas for something only to find out someone using sed to change the same field a few weeks later - makes for interesting debugging. Also, even though it might be 8 nodes now, you have the added bonus that if you lost those servers in a disaster you know how to rebuild at least this part of them. What starts as 8 now may be more in 6 months time - though I''m pretty sure I don''t need to tell you of the quadruple digit infrastructure this phenomenon ;-) On 18/11/11 15:26, Jake - USPS wrote:> We have an environment of thousands of servers. Up to this point we > have been using puppet to manage resources common to all systems (ntp, > ssh, etc) which are managed automatically (without manually assigning > class/module to a system) and also resources for applications that > make up a majority of our environment (Oracle DB, WebSphere, etc) > which are assigned via ENC to specific systems (foreman hostgroups). > > What I am now running into are requests to modify a small amount of > systems for a few resources. Just recently a small new app needs 2 > limits.conf entries on like 8 systems. It seems like a waste of time > to me to craft and manage a recipe for this app to manage these > resources. But, it would still be nice to have this managed by puppet > somehow for all the benefits puppet brings (standardization, > documentation, etc). Also, I see lag time between crafting a recipe > after a request and then pushing the recipe out to the environments > before it would be applied to a system so that the app can then run/ > install (although I know we can add it manually in the meantime, but > then your sort of doubling up on work). > > How do other people handle managing ''one offs'' like this? I''m > wondering if there is a way to create a generic limits.conf module > that could take parameters from an ENC somehow and then manage the > resources? That way I do not need to make new modules for app that > maybe just modify limits.conf or sysctl.conf with a couple of lines, > and then I have the changes documented and managed. Or do people just > create a new module for every little app that comes along? > > Thanks! > Jake >-- Luke Bigum Information Systems +44 (0) 20 3192 2520 Luke.Bigum@lmax.com | http://www.lmax.com LMAX, Yellow Building, 1A Nicholas Road, London W11 4AN The information in this e-mail and any attachment is confidential and is intended only for the named recipient(s). The e-mail may not be disclosed or used by any person other than the addressee, nor may it be copied in any way. If you are not a named recipient please notify the sender immediately and delete any copies of this message. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Any view or opinions presented are solely those of the author and do not necessarily represent those of the company. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Jake - USPS
2011-Nov-18 16:11 UTC
[Puppet Users] Re: Managing a few resources on a few systems
Thank you both for your responses. I do already have a define for adding items to limits.conf and sysctl.conf (using augeas). So making new modules for this wouldn''t take a lot of time (probably more then just logging in and adding the values though because of version control/testing/code promition/etc). I''m just worried that we would eventually have hundreds of modules for all these little one offs. We use foreman as an ENC. This means that when choosing additional classes to assign to a system someone would have to shift through hundreds of classes for the one they want. I get that having the manifest is good documentation. That is why I think being able to utilize parameters in an ENC that maybe a class is able to ''parse'' to manage it would be good. That way we have it documented for the host in foreman with the parameters (which allows easy rebuilds as well in case of a system blowing up), its managed by puppet, and we don''t need to wait to push out an update throughout our puppet environments to add small resources like this. But maybe this isn''t possible? Thanks again for your comments! Jake On Nov 18, 9:54 am, Christopher Wood <christopher_w...@pobox.com> wrote:> inline > > > > > > > > > > On Fri, Nov 18, 2011 at 07:26:45AM -0800, Jake - USPS wrote: > > We have an environment of thousands of servers. Up to this point we > > have been using puppet to manage resources common to all systems (ntp, > > ssh, etc) which are managed automatically (without manually assigning > > class/module to a system) and also resources for applications that > > make up a majority of our environment (Oracle DB, WebSphere, etc) > > which are assigned via ENC to specific systems (foreman hostgroups). > > > What I am now running into are requests to modify a small amount of > > systems for a few resources. Just recently a small new app needs 2 > > limits.conf entries on like 8 systems. It seems like a waste of time > > to me to craft and manage a recipe for this app to manage these > > resources. But, it would still be nice to have this managed by puppet > > somehow for all the benefits puppet brings (standardization, > > documentation, etc). Also, I see lag time between crafting a recipe > > after a request and then pushing the recipe out to the environments > > before it would be applied to a system so that the app can then run/ > > install (although I know we can add it manually in the meantime, but > > then your sort of doubling up on work). > > If you put the effort into revision-controlling and puppetizing any specific configuration item, then in two years when somebody wonders what is configured on a system they will be able to dig into your manifests and know. It''s well worth the initial wait. > > You can always, depending on your config, "puppet kick" the hosts which need changing right now. Or somehow orchestrate that these reapply the current config right now. > > If you aren''t able to orchestrate, then the cheap cousin might be uploading the puppet manifest to the hosts and having puppet configure that. When the regular run comes around it''ll do nothing since limits.conf is already the new one. (This option is vastly less preferable to orchestration.) > > More specifically, there''s an example for modifying limits.conf with the augeas type: > > http://projects.puppetlabs.com/projects/1/wiki/Puppet_Augeas#/etc/sec... > > > How do other people handle managing ''one offs'' like this? I''m > > wondering if there is a way to create a generic limits.conf module > > that could take parameters from an ENC somehow and then manage the > > resources? That way I do not need to make new modules for app that > > maybe just modify limits.conf or sysctl.conf with a couple of lines, > > and then I have the changes documented and managed. Or do people just > > create a new module for every little app that comes along? > > node "specialserver.mine.com" { > > # everything that a generic host needs in our environment > # snmp, ssh, ldap auth, puppet, apt/yum sources, etc. > class { ''standardhost'': } > > # special shiny thing that only this subset of hosts does > # inside this module would be your lookups, I suppose > class { ''limits'': } > > } > > This doesn''t scale way, way up without automagic manifest generation, but it''s useful at our scale and number of special servers. It''s also fairly easy to see in the puppet tree which nodes get the limits class/module. > > On the other hand, your limits class could go everywhere and check if it''s supposed to apply the new limits.conf file. That might mean puppet cycles everywhere for only eight servers and perhaps not worth the cost. (I take it by your domain that you scale way, way, way up. I''m not sure of your values for "might" and "perhaps".) > > To arrange networking on the initial server powerup I use a magic variable in the node definition which the networking class applies. It''s a bit horrible, but then some hosts have fairly twisty networking. > > > > > > > > > Thanks! > > Jake > > > -- > > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > > To post to this group, send email to puppet-users@googlegroups.com. > > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > > For more options, visit this group athttp://groups.google.com/group/puppet-users?hl=en.-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Jake - USPS
2011-Nov-18 19:45 UTC
[Puppet Users] Re: Managing a few resources on a few systems
So what I got that is functionally what I want is:
Define parameter in foreman ...
limits: "user type item value"
Can have multiple ''entries'' such as:
limits: "user1 type1 item1 value1,john soft nproc 1234"
Then I have class ''limits'':
class limits {
define set_limits() {
$user = inline_template("<%= name.split(\" \")[0]
%>")
$type = inline_template("<%= name.split(\" \")[1]
%>")
$item = inline_template("<%= name.split(\" \")[2]
%>")
$value = inline_template("<%= name.split(\" \")[3]
%>")
if $user != "" and $user != nil and $type != "" and
$type != nil
and $item != "" and $itme != nil and $value != "" and $value
!= nil {
include aug_conf
aug_conf::etc_security_limits_conf {
"${user}-${type}-${item}": domain => "$user",
type => "$type",
item => "$item", value => "$value";
}
}
}
$limits_arr = split($limits,",")
set_limits{ $limits_arr: }
}
And I include class limits on all systems. If parameter doesn''t exist
nothing happens. If parameter exists then it checks/adds the items.
This seems to work for me. Not sure its the best way though. I did
see http://projects.puppetlabs.com/issues/1858 and
http://projects.puppetlabs.com/issues/8670 which are very similar to
what I am trying to do, but it seems foreman does not allow parameters
for classes yet nor can I figure out how to have a ''parameter
hash'' in
foreman.
So functionally this is pretty close to what I am looking for. Not
sure if there is a better way to implement with foreman (a question
perhaps for that group), or if someone has a better solution overall
disregarding what I''ve done.
Thanks!
Jake
On Nov 18, 10:11 am, Jake - USPS <jacob.m.mcc...@usps.gov>
wrote:> Thank you both for your responses.
>
> I do already have a define for adding items to limits.conf and
> sysctl.conf (using augeas). So making new modules for this
wouldn''t
> take a lot of time (probably more then just logging in and adding the
> values though because of version control/testing/code promition/etc).
> I''m just worried that we would eventually have hundreds of modules
for
> all these little one offs. We use foreman as an ENC. This means that
> when choosing additional classes to assign to a system someone would
> have to shift through hundreds of classes for the one they want.
>
> I get that having the manifest is good documentation. That is why I
> think being able to utilize parameters in an ENC that maybe a class is
> able to ''parse'' to manage it would be good. That way we
have it
> documented for the host in foreman with the parameters (which allows
> easy rebuilds as well in case of a system blowing up), its managed by
> puppet, and we don''t need to wait to push out an update throughout
our
> puppet environments to add small resources like this. But maybe this
> isn''t possible?
>
> Thanks again for your comments!
> Jake
>
> On Nov 18, 9:54 am, Christopher Wood <christopher_w...@pobox.com>
> wrote:
>
>
>
>
>
>
>
> > inline
>
> > On Fri, Nov 18, 2011 at 07:26:45AM -0800, Jake - USPS wrote:
> > > We have an environment of thousands of servers. Up to this point
we
> > > have been using puppet to manage resources common to all systems
(ntp,
> > > ssh, etc) which are managed automatically (without manually
assigning
> > > class/module to a system) and also resources for applications
that
> > > make up a majority of our environment (Oracle DB, WebSphere, etc)
> > > which are assigned via ENC to specific systems (foreman
hostgroups).
>
> > > What I am now running into are requests to modify a small amount
of
> > > systems for a few resources. Just recently a small new app needs
2
> > > limits.conf entries on like 8 systems. It seems like a waste of
time
> > > to me to craft and manage a recipe for this app to manage these
> > > resources. But, it would still be nice to have this managed by
puppet
> > > somehow for all the benefits puppet brings (standardization,
> > > documentation, etc). Also, I see lag time between crafting a
recipe
> > > after a request and then pushing the recipe out to the
environments
> > > before it would be applied to a system so that the app can then
run/
> > > install (although I know we can add it manually in the meantime,
but
> > > then your sort of doubling up on work).
>
> > If you put the effort into revision-controlling and puppetizing any
specific configuration item, then in two years when somebody wonders what is
configured on a system they will be able to dig into your manifests and know.
It''s well worth the initial wait.
>
> > You can always, depending on your config, "puppet kick" the
hosts which need changing right now. Or somehow orchestrate that these reapply
the current config right now.
>
> > If you aren''t able to orchestrate, then the cheap cousin
might be uploading the puppet manifest to the hosts and having puppet configure
that. When the regular run comes around it''ll do nothing since
limits.conf is already the new one. (This option is vastly less preferable to
orchestration.)
>
> > More specifically, there''s an example for modifying
limits.conf with the augeas type:
>
>
>http://projects.puppetlabs.com/projects/1/wiki/Puppet_Augeas#/etc/sec...
>
> > > How do other people handle managing ''one offs''
like this? I''m
> > > wondering if there is a way to create a generic limits.conf
module
> > > that could take parameters from an ENC somehow and then manage
the
> > > resources? That way I do not need to make new modules for app
that
> > > maybe just modify limits.conf or sysctl.conf with a couple of
lines,
> > > and then I have the changes documented and managed. Or do people
just
> > > create a new module for every little app that comes along?
>
> > node "specialserver.mine.com" {
>
> > # everything that a generic host needs in our environment
> > # snmp, ssh, ldap auth, puppet, apt/yum sources, etc.
> > class { ''standardhost'': }
>
> > # special shiny thing that only this subset of hosts does
> > # inside this module would be your lookups, I suppose
> > class { ''limits'': }
>
> > }
>
> > This doesn''t scale way, way up without automagic manifest
generation, but it''s useful at our scale and number of special servers.
It''s also fairly easy to see in the puppet tree which nodes get the
limits class/module.
>
> > On the other hand, your limits class could go everywhere and check if
it''s supposed to apply the new limits.conf file. That might mean puppet
cycles everywhere for only eight servers and perhaps not worth the cost. (I take
it by your domain that you scale way, way, way up. I''m not sure of your
values for "might" and "perhaps".)
>
> > To arrange networking on the initial server powerup I use a magic
variable in the node definition which the networking class applies.
It''s a bit horrible, but then some hosts have fairly twisty networking.
>
> > > Thanks!
> > > Jake
>
> > > --
> > > You received this message because you are subscribed to the
Google Groups "Puppet Users" group.
> > > To post to this group, send email to
puppet-users@googlegroups.com.
> > > To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> > > For more options, visit this group
athttp://groups.google.com/group/puppet-users?hl=en.
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.