How do I avoid a situation where all of my Linux servers execute a service restart at the same time upon receiving a new configuration change via Puppet? I am trying to avoid any possibility that the service would be unavailable for any length of time. The servers are behind a load balancer. At least one node needs to remain available. Any idea how I might configure Puppet to work in this HA environment? Thank you for your feedback! -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
How about turning off the puppet agent on all the machines, make the change, then turning them back on a few at a time. Matt On Thu, Aug 29, 2013 at 4:50 PM, <rjbutler3@gmail.com> wrote:> How do I avoid a situation where all of my Linux servers execute a service > restart at the same time upon receiving a new configuration change via > Puppet? I am trying to avoid any possibility that the service would be > unavailable for any length of time. The servers are behind a load > balancer. At least one node needs to remain available. Any idea how I > might configure Puppet to work in this HA environment?**** > > Thank you for your feedback! > > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users. > For more options, visit https://groups.google.com/groups/opt_out. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
I think there should be a randomizer built-in to prevent this without these kinds of manual requirements. On Aug 29, 2013, at 5:16 PM, Matt Shields <matt@mattshields.org> wrote:> How about turning off the puppet agent on all the machines, make the change, then turning them back on a few at a time. > > Matt > > > On Thu, Aug 29, 2013 at 4:50 PM, <rjbutler3@gmail.com> wrote: > How do I avoid a situation where all of my Linux servers execute a service restart at the same time upon receiving a new configuration change via Puppet? I am trying to avoid any possibility that the service would be unavailable for any length of time. The servers are behind a load balancer. At least one node needs to remain available. Any idea how I might configure Puppet to work in this HA environment? > > Thank you for your feedback! > > > > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users. > For more options, visit https://groups.google.com/groups/opt_out.-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
There *is* a (semi-)randomizer. Look at the splay option in the puppet.conf http://docs.puppetlabs.com/references/latest/configuration.html#splay -- Peter (from phone) On Aug 29, 2013, at 10:21 PM, Stuart Cracraft <smcracraft@me.com> wrote:> I think there should be a randomizer built-in to prevent this without these kinds of manual requirements. > > On Aug 29, 2013, at 5:16 PM, Matt Shields <matt@mattshields.org> wrote: > >> How about turning off the puppet agent on all the machines, make the change, then turning them back on a few at a time. >> >> Matt >> >> >> On Thu, Aug 29, 2013 at 4:50 PM, <rjbutler3@gmail.com> wrote: >>> How do I avoid a situation where all of my Linux servers execute a service restart at the same time upon receiving a new configuration change via Puppet? I am trying to avoid any possibility that the service would be unavailable for any length of time. The servers are behind a load balancer. At least one node needs to remain available. Any idea how I might configure Puppet to work in this HA environment? >>> >>> Thank you for your feedback! >>> >>> >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups "Puppet Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. >>> To post to this group, send email to puppet-users@googlegroups.com. >>> Visit this group at http://groups.google.com/group/puppet-users. >>> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> -- >> You received this message because you are subscribed to the Google Groups "Puppet Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. >> To post to this group, send email to puppet-users@googlegroups.com. >> Visit this group at http://groups.google.com/group/puppet-users. >> For more options, visit https://groups.google.com/groups/opt_out. > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users. > For more options, visit https://groups.google.com/groups/opt_out.-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
In your situation, I''d be tempted to not run the puppet agent in daemon mode at all so that you can retain full control of when the agents will check in. I can''t see how the splay option will help avoid concurrent checkins: Imagine HA node 1 is rebooted for whatever reason and comes back up at 12:00 at which point the puppet agent checks in due to its boot script running. Its splay parameter is set to true, its runinterval is the default 30m and its psuedo-random splay time is 5 mins. It will therefore next check in at 12:35. Now, HA node 2 just so happens to have checked in last at 11:55 and its pseudo-random splay time is 10 mins, so it''s going to next check in at 12:35. As you, by definition, cannot control the pseudo-random delay time you cannot guarantee the availability of services managed by Puppet. So, instead of running puppet in daemon mode, I''d look to use something like mcollective to control when the agents check in with the master. Kind Regards, Matt. On 29 August 2013 21:50, <rjbutler3@gmail.com> wrote:> How do I avoid a situation where all of my Linux servers execute a service > restart at the same time upon receiving a new configuration change via > Puppet? I am trying to avoid any possibility that the service would be > unavailable for any length of time. The servers are behind a load > balancer. At least one node needs to remain available. Any idea how I > might configure Puppet to work in this HA environment?**** > > Thank you for your feedback! > > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users. > For more options, visit https://groups.google.com/groups/opt_out. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
On 30 August 2013 10:42, Matthew Burgess <matthew.2.burgess@gmail.com>wrote:> > So, instead of running puppet in daemon mode, I''d look to use something > like mcollective to control when the agents check in with the master. >You could of course just set up a cron job on each host, ensuring they check in at different times. Kind Regards, Matt. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
You can use different *runinterval *for each client. This can be done using editing of "/etc/puppet/puppet.conf" on each client machine. Set the following (add a new line if it''s not already present) in the [agent] section of the file: runinterval=XXX where, XXX is the time in seconds(default is 180), Thanks and Regards, Rahul Khengare, NTT DATA OSS Center, Pune, India. On Friday, August 30, 2013 2:20:36 AM UTC+5:30, rjbu...@gmail.com wrote:> > How do I avoid a situation where all of my Linux servers execute a service > restart at the same time upon receiving a new configuration change via > Puppet? I am trying to avoid any possibility that the service would be > unavailable for any length of time. The servers are behind a load > balancer. At least one node needs to remain available. Any idea how I > might configure Puppet to work in this HA environment? > > Thank you for your feedback! > > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
How can this be randomized within a range? I believe someone mentioned "splay" ? My fear is that all the boxes will request at a similar some day, by chance and send a tidal wave over to the master. On Sep 1, 2013, at 10:27 PM, Rahul Khengare <rahulk1306@gmail.com> wrote:> You can use different runinterval for each client. > This can be done using editing of "/etc/puppet/puppet.conf" on each client machine. > Set the following (add a new line if it''s not already present) > in the [agent] section of the file: > > runinterval=XXX > > where, XXX is the time in seconds(default is 180), > > Thanks and Regards, > Rahul Khengare, > NTT DATA OSS Center, Pune, India. > > > On Friday, August 30, 2013 2:20:36 AM UTC+5:30, rjbu...@gmail.com wrote: > How do I avoid a situation where all of my Linux servers execute a service restart at the same time upon receiving a new configuration change via Puppet? I am trying to avoid any possibility that the service would be unavailable for any length of time. The servers are behind a load balancer. At least one node needs to remain available. Any idea how I might configure Puppet to work in this HA environment? > > Thank you for your feedback! > > > > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users. > For more options, visit https://groups.google.com/groups/opt_out.-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
On 2 September 2013 17:01, Stuart Cracraft <smcracraft@me.com> wrote:> How can this be randomized within a range? > >I don''t think it can; it suffers from the same issue as splay does, which I explained in some detail last week.> > My fear is that all the boxes will request at a similar some day, by chance > and send a tidal wave over to the master. >I think the only way to allay that fear is to maintain strict control over when each client (or group of clients) checks in, either using cron or mcollective. Or you could set up multiple masters in a load balanced configuration so that your puppet masters can cope with the maximum possible load that can be thrown at them. Regards, Matt. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
Stuart, If I''m understanding your needs correctly, this may be what you''re looking for: http://www.devco.net/archives/2010/03/17/scheduling_puppet_with_mcollective.php On Monday, September 2, 2013 11:01:46 AM UTC-5, Stuart Cracraft wrote:> > How can this be randomized within a range? > > I believe someone mentioned "splay" ? > > My fear is that all the boxes will request at a similar some day, by chance > and send a tidal wave over to the master. > > On Sep 1, 2013, at 10:27 PM, Rahul Khengare <rahul...@gmail.com<javascript:>> > wrote: > > You can use different *runinterval *for each client. > This can be done using editing of "/etc/puppet/puppet.conf" on each > client machine. > Set the following (add a new line if it''s not already present) > in the [agent] section of the file: > > runinterval=XXX > > where, XXX is the time in seconds(default is 180), > > Thanks and Regards, > Rahul Khengare, > NTT DATA OSS Center, Pune, India. > > > On Friday, August 30, 2013 2:20:36 AM UTC+5:30, rjbu...@gmail.com wrote: >> >> How do I avoid a situation where all of my Linux servers execute a >> service restart at the same time upon receiving a new configuration change >> via Puppet? I am trying to avoid any possibility that the service would be >> unavailable for any length of time. The servers are behind a load >> balancer. At least one node needs to remain available. Any idea how I >> might configure Puppet to work in this HA environment? >> >> Thank you for your feedback! >> >> >> > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users...@googlegroups.com <javascript:>. > To post to this group, send email to puppet...@googlegroups.com<javascript:> > . > Visit this group at http://groups.google.com/group/puppet-users. > For more options, visit https://groups.google.com/groups/opt_out. > > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
On Monday, September 2, 2013 12:01:46 PM UTC-4, Stuart Cracraft wrote:> > How can this be randomized within a range? > > I believe someone mentioned "splay" ? > > My fear is that all the boxes will request at a similar some day, by chance > and send a tidal wave over to the master. > > On Sep 1, 2013, at 10:27 PM, Rahul Khengare <rahul...@gmail.com<javascript:>> > wrote: > > You can use different *runinterval *for each client. > This can be done using editing of "/etc/puppet/puppet.conf" on each > client machine. > Set the following (add a new line if it''s not already present) > in the [agent] section of the file: > > runinterval=XXX > > where, XXX is the time in seconds(default is 180), > > Thanks and Regards, > Rahul Khengare, > NTT DATA OSS Center, Pune, India. > > > On Friday, August 30, 2013 2:20:36 AM UTC+5:30, rjbu...@gmail.com wrote: >> >> How do I avoid a situation where all of my Linux servers execute a >> service restart at the same time upon receiving a new configuration change >> via Puppet? I am trying to avoid any possibility that the service would be >> unavailable for any length of time. The servers are behind a load >> balancer. At least one node needs to remain available. Any idea how I >> might configure Puppet to work in this HA environment? >> >> Thank you for your feedback! >> >It is pretty common to run the puppet agent from cron as opposed to daemon mode. From there you can randomize when the agent runs using the fqdn as a seed with the fqdn_rand[1] function. Here is an example[2] of this in a manifest. [1] - http://docs.puppetlabs.com/references/latest/function.html#fqdnrand [2] - https://github.com/ghoneycutt/puppet-module-puppet/blob/master/manifests/agent.pp#L57 Best regards, -g -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
One of my former co-workers did a talk about Mcollective at PuppetConf. People involved in this thread might find it useful :) He gives some examples about doing rolling restarts, you could do something very similar with orchestrating agent runs. http://puppetlabs.com/presentations/intro-systems-orchestration-mcollective On Tue, Sep 3, 2013 at 10:13 AM, Brad <darkerosxx@gmail.com> wrote:> Stuart, > > If I''m understanding your needs correctly, this may be what you''re looking > for: > > > http://www.devco.net/archives/2010/03/17/scheduling_puppet_with_mcollective.php > > > On Monday, September 2, 2013 11:01:46 AM UTC-5, Stuart Cracraft wrote: > >> How can this be randomized within a range? >> >> I believe someone mentioned "splay" ? >> >> My fear is that all the boxes will request at a similar some day, by >> chance >> and send a tidal wave over to the master. >> >> On Sep 1, 2013, at 10:27 PM, Rahul Khengare <rahul...@gmail.com> wrote: >> >> You can use different *runinterval *for each client. >> This can be done using editing of "/etc/puppet/puppet.conf" on each >> client machine. >> Set the following (add a new line if it''s not already present) >> in the [agent] section of the file: >> >> runinterval=XXX >> >> where, XXX is the time in seconds(default is 180), >> >> Thanks and Regards, >> Rahul Khengare, >> NTT DATA OSS Center, Pune, India. >> >> >> On Friday, August 30, 2013 2:20:36 AM UTC+5:30, rjbu...@gmail.com wrote: >>> >>> How do I avoid a situation where all of my Linux servers execute a >>> service restart at the same time upon receiving a new configuration change >>> via Puppet? I am trying to avoid any possibility that the service would be >>> unavailable for any length of time. The servers are behind a load >>> balancer. At least one node needs to remain available. Any idea how I >>> might configure Puppet to work in this HA environment? >>> >>> Thank you for your feedback! >>> >>> >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to puppet-users...@**googlegroups.com. >> To post to this group, send email to puppet...@googlegroups.com. >> >> Visit this group at http://groups.google.com/**group/puppet-users<http://groups.google.com/group/puppet-users> >> . >> For more options, visit https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out> >> . >> >> >> -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users. > For more options, visit https://groups.google.com/groups/opt_out. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.