For a long time it appeared that run cycles were fairly balanced -- a few every 30 seconds over the 30 minute period. Right now I''m seeing more than 100 systems hit in the same minute: 28 and 58 minutes after the hour. Is there some way to alter the spread of these systems back to even out the load? Or passenger options which could limit the effects of this? -- Jo Rhett Net Consonance : consonant endings by net philanthropy, open source and other randomness -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On Thu, Nov 3, 2011 at 8:36 AM, Jo Rhett <jrhett@netconsonance.com> wrote:> For a long time it appeared that run cycles were fairly balanced -- a few > every 30 seconds over the 30 minute period. Right now I''m seeing more than > 100 systems hit in the same minute: 28 and 58 minutes after the hour. Is > there some way to alter the spread of these systems back to even out the > load? > > Or passenger options which could limit the effects of this? >In your puppet.conf agent block: # How often puppet agent applies the client configuration; in seconds. # Note that a runinterval of 0 means "run continuously" rather than # "never run." If you want puppet agent to never run, you should start # it with the `--no-client` option. # The default value is ''1800''. runinterval = 1800 ... # The maximum time to delay before runs. Defaults to being the same as the # run interval. # The default value is ''$runinterval''. splaylimit = 1800 ... # Whether to sleep for a pseudo-random (but consistent) amount of time before # a run. splay = false> > -- > Jo Rhett > Net Consonance : consonant endings by net philanthropy, open source and > other randomness > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > >-- Nigel Kersten Product Manager, Puppet Labs -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On Thu, Nov 3, 2011 at 8:38 AM, Nigel Kersten <nigel@puppetlabs.com> wrote:> In your puppet.conf agent block: > ... > # The maximum time to delay before runs. Defaults to being the same as the > # run interval. > # The default value is ''$runinterval''. > splaylimit = 1800Having this uncommented as this value is a noop, right?> ... > # Whether to sleep for a pseudo-random (but consistent) amount of time before > # a run. > splay = falseI think we should document this better; I don''t feel any closer to being able to solve this problem. I would probably set "splay" to "true" and hope that it worked, but it wouldn''t surprise me if I had to go back and tweak something else. I also can''t tell if you''re just telling Jo to read the fine manual, or if you''re suggesting configuration values which will solve the problem. (Don''t anyone take this as a flame on Nigel; he''s sitting 6 feet away from me right now :) r -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Christopher Wood
2011-Nov-03 15:53 UTC
Re: [Puppet Users] can we alter the 30-min run time?
(splay inline) On Thu, Nov 03, 2011 at 08:38:39AM -0700, Nigel Kersten wrote:> On Thu, Nov 3, 2011 at 8:36 AM, Jo Rhett <[1]jrhett@netconsonance.com> > wrote: > > For a long time it appeared that run cycles were fairly balanced -- a > few every 30 seconds over the 30 minute period. �Right now I''m seeing > more than 100 systems hit in the same minute: 28 and 58 minutes after > the hour. �Is there some way to alter the spread of these systems back > to even out the load? > > Or passenger options which could limit the effects of this? > > In your puppet.conf agent block: > � � # How often puppet agent applies the client configuration; in seconds. > � � # Note that a runinterval of 0 means "run continuously" rather than > � � # "never run." If you want puppet agent to never run, you should start > � � # it with the `--no-client` option. > � � # The default value is ''1800''. > � � runinterval = 1800 > ... > � � # The maximum time to delay before runs. �Defaults to being the same > as the > � � # run interval. > � � # The default value is ''$runinterval''. > � � splaylimit = 1800 > ... > � � # Whether to sleep for a pseudo-random (but consistent) amount of time > before > � � # a run. > � � splay = falseOn this side, my difficulty in using splay is that it interferes with "puppet kick": http://projects.puppetlabs.com/issues/1100 I work around this by not using play, instead templating my puppet.conf and making my runinterval different on each host: ---------------------------------------- [agent] runinterval=<% # sixty minutes plus up to sixty minutes # average runinterval might be ninety minutes srand MD5.new(fqdn).to_s.hex num = 3600 + rand(3600) %> ---------------------------------------- They''ll intersect at some point, but I''m not worried if they only do it occasionally.> -- > Jo Rhett > Net Consonance : consonant endings by net philanthropy, open source and > other randomness > > -- > You received this message because you are subscribed to the Google > Groups "Puppet Users" group. > To post to this group, send email to [2]puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > [3]puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > [4]http://groups.google.com/group/puppet-users?hl=en. > > -- > Nigel Kersten > Product Manager, Puppet Labs > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > > References > > Visible links > 1. mailto:jrhett@netconsonance.com > 2. mailto:puppet-users@googlegroups.com > 3. mailto:puppet-users%2Bunsubscribe@googlegroups.com > 4. http://groups.google.com/group/puppet-users?hl=en-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On Thu, Nov 3, 2011 at 8:52 AM, Randall Hansen <randall@puppetlabs.com>wrote:> On Thu, Nov 3, 2011 at 8:38 AM, Nigel Kersten <nigel@puppetlabs.com> > wrote: > > > In your puppet.conf agent block: > > ... > > # The maximum time to delay before runs. Defaults to being the same > as the > > # run interval. > > # The default value is ''$runinterval''. > > splaylimit = 1800 > > Having this uncommented as this value is a noop, right? >I was somewhat misleading here. I ran "puppet agent --genconfig" to get the config, and manually uncommented the value so that it was obvious which bits were comments, and which bits weren''t. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Nigel, As you''ve said, the time chosen for the run cycle will be consistent. All of these settings are already set -- this isn''t a question of how to change how often to run, it''s how to affect the chosen runtime? I''ve got an awful lot of systems (> 100) which have decided to all roll at 28 and 58 minutes after the hour. How can I rebalance them? On Nov 3, 2011, at 8:38 AM, Nigel Kersten wrote:> On Thu, Nov 3, 2011 at 8:36 AM, Jo Rhett <jrhett@netconsonance.com> wrote: > For a long time it appeared that run cycles were fairly balanced -- a few every 30 seconds over the 30 minute period. Right now I''m seeing more than 100 systems hit in the same minute: 28 and 58 minutes after the hour. Is there some way to alter the spread of these systems back to even out the load? > > Or passenger options which could limit the effects of this? > > In your puppet.conf agent block: > > # How often puppet agent applies the client configuration; in seconds. > # Note that a runinterval of 0 means "run continuously" rather than > # "never run." If you want puppet agent to never run, you should start > # it with the `--no-client` option. > # The default value is ''1800''. > runinterval = 1800 > > ... > > # The maximum time to delay before runs. Defaults to being the same as the > # run interval. > # The default value is ''$runinterval''. > splaylimit = 1800 > > ... > > # Whether to sleep for a pseudo-random (but consistent) amount of time before > # a run. > splay = false > > > > > > > -- > Jo Rhett > Net Consonance : consonant endings by net philanthropy, open source and other randomness > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. > > > > > -- > Nigel Kersten > Product Manager, Puppet Labs > > > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.-- Jo Rhett Net Consonance : consonant endings by net philanthropy, open source and other randomness -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On Thu, Nov 3, 2011 at 12:41 PM, Jo Rhett <jrhett@netconsonance.com> wrote:> Nigel, As you''ve said, the time chosen for the run cycle will be consistent. > All of these settings are already set -- this isn''t a question of how to > change how often to run, it''s how to affect the chosen runtime? > I''ve got an awful lot of systems (> 100) which have decided to all roll at > 28 and 58 minutes after the hour. How can I rebalance them?This is exactly why I use cron instead of the daemon. In my module that configures puppet I have: # These are used to set when the puppet_run cron job runs. $time1 = fqdn_rand(30) $time2 = $time1 + 30 cron { "puppet_run": command => "$common::data::bindir/puppet_run.sh", user => root, minute => [$time1, $time2], ensure => present, require => File["$common::data::bindir/puppet_run.sh"]; } The fqdn_rand gives me a consistent random number per host, and so far this has been much better than letting the daemon try to splay things on it''s own. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Christopher Wood
2011-Nov-03 20:08 UTC
Re: [Puppet Users] can we alter the 30-min run time?
On Thu, Nov 03, 2011 at 12:41:50PM -0700, Jo Rhett wrote:> Nigel, As you''ve said, the time chosen for the run cycle will be > consistent. All of these settings are already set -- this isn''t a > question of how to change how often to run, it''s how to affect the chosen > runtime?I thought the chosen run time was selected on the agent. It sounds like you''ll have to reconfigure the agent to have it decide differently. So far your options seem to be, from reading this thread: -run via puppetized fqdn_rand() cron -use splay (if you don''t mind living without puppet kick) -change the runinterval for each puppet agent There''s possibly other cheap/dirty ways like having a cron job restart the puppet agent at an interval shorter than when they converge, or setting an at job to do it, or similar. Perhaps you could use a cronjob on the puppet master to check logs for run times and kick the groups that are teaming up. And so forth.> I''ve got an awful lot of systems (> 100) which have decided to all roll at > 28 and 58 minutes after the hour. How can I rebalance them? > On Nov 3, 2011, at 8:38 AM, Nigel Kersten wrote: > > On Thu, Nov 3, 2011 at 8:36 AM, Jo Rhett <[1]jrhett@netconsonance.com> > wrote: > > For a long time it appeared that run cycles were fairly balanced -- a > few every 30 seconds over the 30 minute period. Right now I''m seeing > more than 100 systems hit in the same minute: 28 and 58 minutes after > the hour. Is there some way to alter the spread of these systems back > to even out the load? > > Or passenger options which could limit the effects of this? > > In your puppet.conf agent block: > # How often puppet agent applies the client configuration; in > seconds. > # Note that a runinterval of 0 means "run continuously" rather than > # "never run." If you want puppet agent to never run, you should > start > # it with the `--no-client` option. > # The default value is ''1800''. > runinterval = 1800 > ... > # The maximum time to delay before runs. Defaults to being the same > as the > # run interval. > # The default value is ''$runinterval''. > splaylimit = 1800 > ... > # Whether to sleep for a pseudo-random (but consistent) amount of > time before > # a run. > splay = false > > > -- > Jo Rhett > Net Consonance : consonant endings by net philanthropy, open source > and other randomness > > -- > You received this message because you are subscribed to the Google > Groups "Puppet Users" group. > To post to this group, send email to [2]puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > [3]puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > [4]http://groups.google.com/group/puppet-users?hl=en. > > -- > Nigel Kersten > Product Manager, Puppet Labs > > -- > You received this message because you are subscribed to the Google > Groups "Puppet Users" group. > To post to this group, send email to [5]puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > [6]puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > [7]http://groups.google.com/group/puppet-users?hl=en. > > -- > Jo Rhett > Net Consonance : consonant endings by net philanthropy, open source and > other randomness > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > > References > > Visible links > 1. mailto:jrhett@netconsonance.com > 2. mailto:puppet-users@googlegroups.com > 3. mailto:puppet-users%2Bunsubscribe@googlegroups.com > 4. http://groups.google.com/group/puppet-users?hl=en > 5. mailto:puppet-users@googlegroups.com > 6. mailto:puppet-users+unsubscribe@googlegroups.com > 7. http://groups.google.com/group/puppet-users?hl=en-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
PuppetCommander is another option: http://projects.puppetlabs.com/projects/mcollective-plugins/wiki/ToolPuppetcommander It deals with ''splay'' globally instead of on a per node basis with random sleeps. ken. On Thu, Nov 3, 2011 at 8:08 PM, Christopher Wood <christopher_wood@pobox.com> wrote:> On Thu, Nov 03, 2011 at 12:41:50PM -0700, Jo Rhett wrote: >> Nigel, As you''ve said, the time chosen for the run cycle will be >> consistent. All of these settings are already set -- this isn''t a >> question of how to change how often to run, it''s how to affect the chosen >> runtime? > > I thought the chosen run time was selected on the agent. It sounds like you''ll have to reconfigure the agent to have it decide differently. So far your options seem to be, from reading this thread: > > -run via puppetized fqdn_rand() cron > -use splay (if you don''t mind living without puppet kick) > -change the runinterval for each puppet agent > > There''s possibly other cheap/dirty ways like having a cron job restart the puppet agent at an interval shorter than when they converge, or setting an at job to do it, or similar. Perhaps you could use a cronjob on the puppet master to check logs for run times and kick the groups that are teaming up. And so forth. > >> I''ve got an awful lot of systems (> 100) which have decided to all roll at >> 28 and 58 minutes after the hour. How can I rebalance them? >> On Nov 3, 2011, at 8:38 AM, Nigel Kersten wrote: >> >> On Thu, Nov 3, 2011 at 8:36 AM, Jo Rhett <[1]jrhett@netconsonance.com> >> wrote: >> >> For a long time it appeared that run cycles were fairly balanced -- a >> few every 30 seconds over the 30 minute period. Right now I''m seeing >> more than 100 systems hit in the same minute: 28 and 58 minutes after >> the hour. Is there some way to alter the spread of these systems back >> to even out the load? >> >> Or passenger options which could limit the effects of this? >> >> In your puppet.conf agent block: >> # How often puppet agent applies the client configuration; in >> seconds. >> # Note that a runinterval of 0 means "run continuously" rather than >> # "never run." If you want puppet agent to never run, you should >> start >> # it with the `--no-client` option. >> # The default value is ''1800''. >> runinterval = 1800 >> ... >> # The maximum time to delay before runs. Defaults to being the same >> as the >> # run interval. >> # The default value is ''$runinterval''. >> splaylimit = 1800 >> ... >> # Whether to sleep for a pseudo-random (but consistent) amount of >> time before >> # a run. >> splay = false >> >> >> -- >> Jo Rhett >> Net Consonance : consonant endings by net philanthropy, open source >> and other randomness >> >> -- >> You received this message because you are subscribed to the Google >> Groups "Puppet Users" group. >> To post to this group, send email to [2]puppet-users@googlegroups.com. >> To unsubscribe from this group, send email to >> [3]puppet-users+unsubscribe@googlegroups.com. >> For more options, visit this group at >> [4]http://groups.google.com/group/puppet-users?hl=en. >> >> -- >> Nigel Kersten >> Product Manager, Puppet Labs >> >> -- >> You received this message because you are subscribed to the Google >> Groups "Puppet Users" group. >> To post to this group, send email to [5]puppet-users@googlegroups.com. >> To unsubscribe from this group, send email to >> [6]puppet-users+unsubscribe@googlegroups.com. >> For more options, visit this group at >> [7]http://groups.google.com/group/puppet-users?hl=en. >> >> -- >> Jo Rhett >> Net Consonance : consonant endings by net philanthropy, open source and >> other randomness >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To post to this group, send email to puppet-users@googlegroups.com. >> To unsubscribe from this group, send email to >> puppet-users+unsubscribe@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/puppet-users?hl=en. >> >> References >> >> Visible links >> 1. mailto:jrhett@netconsonance.com >> 2. mailto:puppet-users@googlegroups.com >> 3. mailto:puppet-users%2Bunsubscribe@googlegroups.com >> 4. http://groups.google.com/group/puppet-users?hl=en >> 5. mailto:puppet-users@googlegroups.com >> 6. mailto:puppet-users+unsubscribe@googlegroups.com >> 7. http://groups.google.com/group/puppet-users?hl=en > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On Nov 3, 2011, at 1:01 PM, Adam Crews wrote:> This is exactly why I use cron instead of the daemon.we "puppet kick" a lot. -- Jo Rhett Net Consonance : consonant endings by net philanthropy, open source and other randomness -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Why not document how the client decides the runtime, and how to adjust it? On Nov 3, 2011, at 1:08 PM, Christopher Wood wrote:> I thought the chosen run time was selected on the agent. It sounds like you''ll have to reconfigure the agent to have it decide differently. So far your options seem to be, from reading this thread: > > -run via puppetized fqdn_rand() cron > -use splay (if you don''t mind living without puppet kick) > -change the runinterval for each puppet agent > > There''s possibly other cheap/dirty ways like having a cron job restart the puppet agent at an interval shorter than when they converge, or setting an at job to do it, or similar. Perhaps you could use a cronjob on the puppet master to check logs for run times and kick the groups that are teaming up. And so forth. > >> I''ve got an awful lot of systems (> 100) which have decided to all roll at >> 28 and 58 minutes after the hour. How can I rebalance them? >> On Nov 3, 2011, at 8:38 AM, Nigel Kersten wrote: >> >> On Thu, Nov 3, 2011 at 8:36 AM, Jo Rhett <[1]jrhett@netconsonance.com> >> wrote: >> >> For a long time it appeared that run cycles were fairly balanced -- a >> few every 30 seconds over the 30 minute period. Right now I''m seeing >> more than 100 systems hit in the same minute: 28 and 58 minutes after >> the hour. Is there some way to alter the spread of these systems back >> to even out the load? >> >> Or passenger options which could limit the effects of this? >> >> In your puppet.conf agent block: >> # How often puppet agent applies the client configuration; in >> seconds. >> # Note that a runinterval of 0 means "run continuously" rather than >> # "never run." If you want puppet agent to never run, you should >> start >> # it with the `--no-client` option. >> # The default value is ''1800''. >> runinterval = 1800 >> ... >> # The maximum time to delay before runs. Defaults to being the same >> as the >> # run interval. >> # The default value is ''$runinterval''. >> splaylimit = 1800 >> ... >> # Whether to sleep for a pseudo-random (but consistent) amount of >> time before >> # a run. >> splay = false >> >> >> -- >> Jo Rhett >> Net Consonance : consonant endings by net philanthropy, open source >> and other randomness >> >> -- >> You received this message because you are subscribed to the Google >> Groups "Puppet Users" group. >> To post to this group, send email to [2]puppet-users@googlegroups.com. >> To unsubscribe from this group, send email to >> [3]puppet-users+unsubscribe@googlegroups.com. >> For more options, visit this group at >> [4]http://groups.google.com/group/puppet-users?hl=en. >> >> -- >> Nigel Kersten >> Product Manager, Puppet Labs >> >> -- >> You received this message because you are subscribed to the Google >> Groups "Puppet Users" group. >> To post to this group, send email to [5]puppet-users@googlegroups.com. >> To unsubscribe from this group, send email to >> [6]puppet-users+unsubscribe@googlegroups.com. >> For more options, visit this group at >> [7]http://groups.google.com/group/puppet-users?hl=en. >> >> -- >> Jo Rhett >> Net Consonance : consonant endings by net philanthropy, open source and >> other randomness >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To post to this group, send email to puppet-users@googlegroups.com. >> To unsubscribe from this group, send email to >> puppet-users+unsubscribe@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/puppet-users?hl=en. >> >> References >> >> Visible links >> 1. mailto:jrhett@netconsonance.com >> 2. mailto:puppet-users@googlegroups.com >> 3. mailto:puppet-users%2Bunsubscribe@googlegroups.com >> 4. http://groups.google.com/group/puppet-users?hl=en >> 5. mailto:puppet-users@googlegroups.com >> 6. mailto:puppet-users+unsubscribe@googlegroups.com >> 7. http://groups.google.com/group/puppet-users?hl=en > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. >-- Jo Rhett Net Consonance : consonant endings by net philanthropy, open source and other randomness -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On Thu, Nov 3, 2011 at 12:41 PM, Jo Rhett <jrhett@netconsonance.com> wrote:> Nigel, As you''ve said, the time chosen for the run cycle will be > consistent. All of these settings are already set -- this isn''t a question > of how to change how often to run, it''s how to affect the chosen runtime? > > I''ve got an awful lot of systems (> 100) which have decided to all roll at > 28 and 58 minutes after the hour. How can I rebalance them? >This should be what the splay settings do for you Jo. Even though those agents all roll at 28/58 minutes past the hour, if you set splay to true, they''ll then wait a random amount of time up to "splaylimit" before they *actually* perform the run. The default setting for "splay" is off, so you won''t see this behavior unless you explicitly turn it on. Does that make more sense?> > On Nov 3, 2011, at 8:38 AM, Nigel Kersten wrote: > > On Thu, Nov 3, 2011 at 8:36 AM, Jo Rhett <jrhett@netconsonance.com> wrote: > >> For a long time it appeared that run cycles were fairly balanced -- a few >> every 30 seconds over the 30 minute period. Right now I''m seeing more than >> 100 systems hit in the same minute: 28 and 58 minutes after the hour. Is >> there some way to alter the spread of these systems back to even out the >> load? >> >> Or passenger options which could limit the effects of this? >> > > In your puppet.conf agent block: > > # How often puppet agent applies the client configuration; in seconds. > # Note that a runinterval of 0 means "run continuously" rather than > # "never run." If you want puppet agent to never run, you should start > # it with the `--no-client` option. > # The default value is ''1800''. > runinterval = 1800 > > ... > > # The maximum time to delay before runs. Defaults to being the same > as the > # run interval. > # The default value is ''$runinterval''. > splaylimit = 1800 > > ... > > # Whether to sleep for a pseudo-random (but consistent) amount of time > before > # a run. > splay = false > > > > > > >> >> -- >> Jo Rhett >> Net Consonance : consonant endings by net philanthropy, open source and >> other randomness >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To post to this group, send email to puppet-users@googlegroups.com. >> To unsubscribe from this group, send email to >> puppet-users+unsubscribe@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/puppet-users?hl=en. >> >> > > > -- > Nigel Kersten > Product Manager, Puppet Labs > > > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > > > -- > Jo Rhett > Net Consonance : consonant endings by net philanthropy, open source and > other randomness > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. >-- Nigel Kersten Product Manager, Puppet Labs -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Hi , How about an option for throttling number of max puppet clients being active - then puppet client time to run has come it registers at the puppetmaster - and once there is below max allowed active puppet clients running - puppetmaster initiates the client run (puppet client needs to have "listen=true" for that to work). Priorities can also be added easily. This also gives you easy visibility on then running puppet every X minutes becomes a bottleneck. This ca be probably done with mcollective - but requires a separate daemon process to manage the queue waiting puppet clients. The advanate of separate daemon is that it can have a view of more than one puppetmaster. The disadvantage is that puppet clients can still connect directly puppetmaster avoiding the queue altogether. So it looks like a builtin puppet queue functionality is a better choice. wdyt? Alex On 11/05/2011 06:42 PM, Nigel Kersten wrote:> On Thu, Nov 3, 2011 at 12:41 PM, Jo Rhett <jrhett@netconsonance.com > <mailto:jrhett@netconsonance.com>> wrote: > > Nigel, As you''ve said, the time chosen for the run cycle will be > consistent. All of these settings are already set -- this isn''t a > question of how to change how often to run, it''s how to affect the > chosen runtime? > > I''ve got an awful lot of systems (> 100) which have decided to all > roll at 28 and 58 minutes after the hour. How can I rebalance them? > > > This should be what the splay settings do for you Jo. > > Even though those agents all roll at 28/58 minutes past the hour, if > you set splay to true, they''ll then wait a random amount of time up to > "splaylimit" before they *actually* perform the run. > > The default setting for "splay" is off, so you won''t see this behavior > unless you explicitly turn it on. > > Does that make more sense? > > > > On Nov 3, 2011, at 8:38 AM, Nigel Kersten wrote: >> On Thu, Nov 3, 2011 at 8:36 AM, Jo Rhett >> <jrhett@netconsonance.com <mailto:jrhett@netconsonance.com>> wrote: >> >> For a long time it appeared that run cycles were fairly >> balanced -- a few every 30 seconds over the 30 minute period. >> Right now I''m seeing more than 100 systems hit in the same >> minute: 28 and 58 minutes after the hour. Is there some way >> to alter the spread of these systems back to even out the load? >> >> Or passenger options which could limit the effects of this? >> >> >> In your puppet.conf agent block: >> >> # How often puppet agent applies the client configuration; in >> seconds. >> # Note that a runinterval of 0 means "run continuously" >> rather than >> # "never run." If you want puppet agent to never run, you >> should start >> # it with the `--no-client` option. >> # The default value is ''1800''. >> runinterval = 1800 >> >> ... >> >> # The maximum time to delay before runs. Defaults to being >> the same as the >> # run interval. >> # The default value is ''$runinterval''. >> splaylimit = 1800 >> >> ... >> >> # Whether to sleep for a pseudo-random (but consistent) >> amount of time before >> # a run. >> splay = false >> >> >> >> >> >> -- >> Jo Rhett >> Net Consonance : consonant endings by net philanthropy, open >> source and other randomness >> >> -- >> You received this message because you are subscribed to the >> Google Groups "Puppet Users" group. >> To post to this group, send email to >> puppet-users@googlegroups.com >> <mailto:puppet-users@googlegroups.com>. >> To unsubscribe from this group, send email to >> puppet-users+unsubscribe@googlegroups.com >> <mailto:puppet-users%2Bunsubscribe@googlegroups.com>. >> For more options, visit this group at >> http://groups.google.com/group/puppet-users?hl=en. >> >> >> >> >> -- >> Nigel Kersten >> Product Manager, Puppet Labs >> >> >> >> -- >> You received this message because you are subscribed to the >> Google Groups "Puppet Users" group. >> To post to this group, send email to >> puppet-users@googlegroups.com <mailto:puppet-users@googlegroups.com>. >> To unsubscribe from this group, send email to >> puppet-users+unsubscribe@googlegroups.com >> <mailto:puppet-users+unsubscribe@googlegroups.com>. >> For more options, visit this group at >> http://groups.google.com/group/puppet-users?hl=en. > > -- > Jo Rhett > Net Consonance : consonant endings by net philanthropy, open > source and other randomness > > -- > You received this message because you are subscribed to the Google > Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com > <mailto:puppet-users@googlegroups.com>. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com > <mailto:puppet-users%2Bunsubscribe@googlegroups.com>. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > > > > > -- > Nigel Kersten > Product Manager, Puppet Labs > > > -- > You received this message because you are subscribed to the Google > Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en.-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On Sun, Nov 6, 2011 at 5:57 AM, Piavlo <lolitushka@gmail.com> wrote:> > Hi , > > How about an option for throttling number of max puppet clients being > active - then puppet client time to run has come it registers at the > puppetmaster - and once there is below max allowed active puppet clients > running - puppetmaster initiates the client run (puppet client needs to > have "listen=true" for that to work). Priorities can also be added easily. > This also gives you easy visibility on then running puppet every X minutes > becomes a bottleneck. > This ca be probably done with mcollective - but requires a separate daemon > process to manage the queue waiting puppet clients. > The advanate of separate daemon is that it can have a view of more than > one puppetmaster. > The disadvantage is that puppet clients can still connect directly > puppetmaster avoiding the queue altogether. > So it looks like a builtin puppet queue functionality is a better choice. > > wdyt? >So you need a queue *somewhere* to do this, and MCollective will do this for you now with the "mco puppetd runall X" agent. Your point about agents still being able to connect if they want to is a good one though. I''d love it if you filed this as a feature request though Piavlo, it''s important to capture this somewhere outside the mailing lists. -- Nigel Kersten Product Manager, Puppet Labs -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On Nov 5, 2011, at 9:42 AM, Nigel Kersten wrote:> On Thu, Nov 3, 2011 at 12:41 PM, Jo Rhett <jrhett@netconsonance.com> wrote: > Nigel, As you''ve said, the time chosen for the run cycle will be consistent. All of these settings are already set -- this isn''t a question of how to change how often to run, it''s how to affect the chosen runtime? > > I''ve got an awful lot of systems (> 100) which have decided to all roll at 28 and 58 minutes after the hour. How can I rebalance them? > > This should be what the splay settings do for you Jo. > > Even though those agents all roll at 28/58 minutes past the hour, if you set splay to true, they''ll then wait a random amount of time up to "splaylimit" before they *actually* perform the run.splay breaks many things, including puppet kick. It simply won''t work until you break splay out to only affect normal running. I tried enabling it, and found that it simply didn''t affect the systems enough. I am deeply curious as to why you are avoiding the main question here: 1. How does puppet decide when its 30 minutes is up? Is this stored in a file somewhere? 2. How can we change its calculation? (can we edit a file, run a command, …?)> On Nov 3, 2011, at 8:38 AM, Nigel Kersten wrote: >> On Thu, Nov 3, 2011 at 8:36 AM, Jo Rhett <jrhett@netconsonance.com> wrote: >> For a long time it appeared that run cycles were fairly balanced -- a few every 30 seconds over the 30 minute period. Right now I''m seeing more than 100 systems hit in the same minute: 28 and 58 minutes after the hour. Is there some way to alter the spread of these systems back to even out the load? >> >> Or passenger options which could limit the effects of this? >> >> In your puppet.conf agent block: >> >> # How often puppet agent applies the client configuration; in seconds. >> # Note that a runinterval of 0 means "run continuously" rather than >> # "never run." If you want puppet agent to never run, you should start >> # it with the `--no-client` option. >> # The default value is ''1800''. >> runinterval = 1800 >> >> ... >> >> # The maximum time to delay before runs. Defaults to being the same as the >> # run interval. >> # The default value is ''$runinterval''. >> splaylimit = 1800 >> >> ... >> >> # Whether to sleep for a pseudo-random (but consistent) amount of time before >> # a run. >> splay = false >> >> >> >> >> >> >> -- >> Jo Rhett >> Net Consonance : consonant endings by net philanthropy, open source and other randomness >> >> -- >> You received this message because you are subscribed to the Google Groups "Puppet Users" group. >> To post to this group, send email to puppet-users@googlegroups.com. >> To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. >> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. >> >> >> >> >> -- >> Nigel Kersten >> Product Manager, Puppet Labs >> >> >> >> -- >> You received this message because you are subscribed to the Google Groups "Puppet Users" group. >> To post to this group, send email to puppet-users@googlegroups.com. >> To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. >> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. > > -- > Jo Rhett > Net Consonance : consonant endings by net philanthropy, open source and other randomness > > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. > > > > -- > Nigel Kersten > Product Manager, Puppet Labs > > > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.-- Jo Rhett Net Consonance : consonant endings by net philanthropy, open source and other randomness -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Christopher Wood
2011-Nov-15 20:09 UTC
Re: [Puppet Users] can we alter the 30-min run time?
(Without jumping too hard in the middle of your conversation... inline.) On Tue, Nov 15, 2011 at 11:30:40AM -0800, Jo Rhett wrote:> On Nov 5, 2011, at 9:42 AM, Nigel Kersten wrote: > > On Thu, Nov 3, 2011 at 12:41 PM, Jo Rhett <[1]jrhett@netconsonance.com> > wrote: > > Nigel, As you''ve said, the time chosen for the run cycle will be > consistent. All of these settings are already set -- this isn''t a > question of how to change how often to run, it''s how to affect the > chosen runtime? > I''ve got an awful lot of systems (> 100) which have decided to all > roll at 28 and 58 minutes after the hour. How can I rebalance them? > > This should be what the splay settings do for you Jo. > Even though those agents all roll at 28/58 minutes past the hour, if you > set splay to true, they''ll then wait a random amount of time up to > "splaylimit" before they *actually* perform the run. > > splay breaks many things, including puppet kick. It simply won''t work > until you break splay out to only affect normal running. I tried enabling > it, and found that it simply didn''t affect the systems enough. > I am deeply curious as to why you are avoiding the main question here: > 1. How does puppet decide when its 30 minutes is up? Is this stored in > a file somewhere?Grep shows me (on Debian Stable, puppet 2.6.2) that agent.rb sets up an event loop: /usr/lib/ruby/1.8/puppet/agent.rb /usr/lib/ruby/1.8/puppet/external/event-loop/event-loop.rb> 2. How can we change its calculation? (can we edit a file, run a > command, �?)Subclass the event loop or the agent perhaps? (I''m assuming that the several existing techniques for changing agent run times are insufficient for you.)> On Nov 3, 2011, at 8:38 AM, Nigel Kersten wrote: > > On Thu, Nov 3, 2011 at 8:36 AM, Jo Rhett > <[2]jrhett@netconsonance.com> wrote: > > For a long time it appeared that run cycles were fairly balanced > -- a few every 30 seconds over the 30 minute period. Right now > I''m seeing more than 100 systems hit in the same minute: 28 and 58 > minutes after the hour. Is there some way to alter the spread of > these systems back to even out the load? > > Or passenger options which could limit the effects of this? > > In your puppet.conf agent block: > # How often puppet agent applies the client configuration; in > seconds. > # Note that a runinterval of 0 means "run continuously" rather > than > # "never run." If you want puppet agent to never run, you should > start > # it with the `--no-client` option. > # The default value is ''1800''. > runinterval = 1800 > ... > # The maximum time to delay before runs. Defaults to being the > same as the > # run interval. > # The default value is ''$runinterval''. > splaylimit = 1800 > ... > # Whether to sleep for a pseudo-random (but consistent) amount > of time before > # a run. > splay = false > > > -- > Jo Rhett > Net Consonance : consonant endings by net philanthropy, open > source and other randomness > > -- > You received this message because you are subscribed to the Google > Groups "Puppet Users" group. > To post to this group, send email to > [3]puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > [4]puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > [5]http://groups.google.com/group/puppet-users?hl=en. > > -- > Nigel Kersten > Product Manager, Puppet Labs > > -- > You received this message because you are subscribed to the Google > Groups "Puppet Users" group. > To post to this group, send email to > [6]puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > [7]puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > [8]http://groups.google.com/group/puppet-users?hl=en. > > -- > Jo Rhett > Net Consonance : consonant endings by net philanthropy, open source > and other randomness > -- > You received this message because you are subscribed to the Google > Groups "Puppet Users" group. > To post to this group, send email to [9]puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > [10]puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > [11]http://groups.google.com/group/puppet-users?hl=en. > > -- > Nigel Kersten > Product Manager, Puppet Labs > > -- > You received this message because you are subscribed to the Google > Groups "Puppet Users" group. > To post to this group, send email to [12]puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > [13]puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > [14]http://groups.google.com/group/puppet-users?hl=en. > > -- > Jo Rhett > Net Consonance : consonant endings by net philanthropy, open source and > other randomness > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > > References > > Visible links > 1. mailto:jrhett@netconsonance.com > 2. mailto:jrhett@netconsonance.com > 3. mailto:puppet-users@googlegroups.com > 4. mailto:puppet-users%2Bunsubscribe@googlegroups.com > 5. http://groups.google.com/group/puppet-users?hl=en > 6. mailto:puppet-users@googlegroups.com > 7. mailto:puppet-users+unsubscribe@googlegroups.com > 8. http://groups.google.com/group/puppet-users?hl=en > 9. mailto:puppet-users@googlegroups.com > 10. mailto:puppet-users%2Bunsubscribe@googlegroups.com > 11. http://groups.google.com/group/puppet-users?hl=en > 12. mailto:puppet-users@googlegroups.com > 13. mailto:puppet-users+unsubscribe@googlegroups.com > 14. http://groups.google.com/group/puppet-users?hl=en-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.