Puppet users - Jan 2009 - Feature request runinterval + $random

Hello...

I have an existing puppet setup that works very well except for one
thing.  Occasionally the puppetmaster is busy and the clients tend to
"bunch up".  Then on the next client run they tend to "bunch
up" again
causing the puppetmaster to be slow down due to the high load which
causes more slow downs.  This example should show what is happending:

shell> grep ": Compiled catalog for " all.log | cut -d: -f 1-2 |
uniq -c
   1 Jan  8 10:59
   8 Jan  8 11:00
  14 Jan  8 11:01
   2 Jan  8 11:02
   1 Jan  8 11:08
  15 Jan  8 11:10
  12 Jan  8 11:11
   1 Jan  8 11:20

The above shows an idle time followed by a rush of puppet clients
hitting the puppetmaster followed by another idle time.

This is probably worse when clients run a logrotate script with a client
restart from a cronjob. 

I would suggest something like the runinterval be $runinterval +
$(random number of seconds between 1 and 120) || $(random number of
seconds between 1 and $runinterval*0.1 )

That should help spread out the load on the server.




-- 
Christopher <chrismcc@pricegrabber.com>


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

I believe the client side ''splay'' option takes care of this.

Best,
Adam

On Jan 8, 3:17 pm, Christopher <chris...@pricegrabber.com>
wrote:
> Hello...
>
> I have an existing puppet setup that works very well except for one
> thing.  Occasionally the puppetmaster is busy and the clients tend to
> "bunch up".  Then on the next client run they tend to "bunch
up" again
> causing the puppetmaster to be slow down due to the high load which
> causes more slow downs.  This example should show what is happending:
>
> shell> grep ": Compiled catalog for " all.log | cut -d: -f 1-2
| uniq -c
>    1 Jan  8 10:59
>    8 Jan  8 11:00
>   14 Jan  8 11:01
>    2 Jan  8 11:02
>    1 Jan  8 11:08
>   15 Jan  8 11:10
>   12 Jan  8 11:11
>    1 Jan  8 11:20
>
> The above shows an idle time followed by a rush of puppet clients
> hitting the puppetmaster followed by another idle time.
>
> This is probably worse when clients run a logrotate script with a client
> restart from a cronjob.
>
> I would suggest something like the runinterval be $runinterval +
> $(random number of seconds between 1 and 120) || $(random number of
> seconds between 1 and $runinterval*0.1 )
>
> That should help spread out the load on the server.
>
> --
> Christopher <chris...@pricegrabber.com>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Have you checked out the splay option?

Alternatively, I''ve since moved on from splay to use a cron job to run 
puppetd.  I have a meta package that pulls in puppet and drops in a root 
cron job entry based on the mac address of the host''s primary interface
during the post-install step of rpm.

Cheers,
Ryan

On 1/8/09 12:17 PM, Christopher wrote:
> Hello...
>
> I have an existing puppet setup that works very well except for one
> thing.  Occasionally the puppetmaster is busy and the clients tend to
> "bunch up".  Then on the next client run they tend to "bunch
up" again
> causing the puppetmaster to be slow down due to the high load which
> causes more slow downs.  This example should show what is happending:
>
> shell>  grep ": Compiled catalog for " all.log | cut -d: -f
1-2 | uniq -c
>     1 Jan  8 10:59
>     8 Jan  8 11:00
>    14 Jan  8 11:01
>     2 Jan  8 11:02
>     1 Jan  8 11:08
>    15 Jan  8 11:10
>    12 Jan  8 11:11
>     1 Jan  8 11:20
>
> The above shows an idle time followed by a rush of puppet clients
> hitting the puppetmaster followed by another idle time.
>
> This is probably worse when clients run a logrotate script with a client
> restart from a cronjob.
>
> I would suggest something like the runinterval be $runinterval +
> $(random number of seconds between 1 and 120) || $(random number of
> seconds between 1 and $runinterval*0.1 )
>
> That should help spread out the load on the server.
>
>
>
>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

I also use cron, but I found using the IP address last bit as a more
effective way to "randomize" it.

more info here:
http://reductivelabs.com/trac/puppet/wiki/Recipes/cron

Cheers,
Ohad

On Fri, Jan 9, 2009 at 9:21 AM, Ryan Dooley <ryan.dooley@gmail.com> wrote:
>
> Have you checked out the splay option?
>
> Alternatively, I''ve since moved on from splay to use a cron job to
run
> puppetd.  I have a meta package that pulls in puppet and drops in a root
> cron job entry based on the mac address of the host''s primary
interface
> during the post-install step of rpm.
>
> Cheers,
> Ryan
>
> On 1/8/09 12:17 PM, Christopher wrote:
> > Hello...
> >
> > I have an existing puppet setup that works very well except for one
> > thing.  Occasionally the puppetmaster is busy and the clients tend to
> > "bunch up".  Then on the next client run they tend to
"bunch up" again
> > causing the puppetmaster to be slow down due to the high load which
> > causes more slow downs.  This example should show what is happending:
> >
> > shell>  grep ": Compiled catalog for " all.log | cut -d:
-f 1-2 | uniq -c
> >     1 Jan  8 10:59
> >     8 Jan  8 11:00
> >    14 Jan  8 11:01
> >     2 Jan  8 11:02
> >     1 Jan  8 11:08
> >    15 Jan  8 11:10
> >    12 Jan  8 11:11
> >     1 Jan  8 11:20
> >
> > The above shows an idle time followed by a rush of puppet clients
> > hitting the puppetmaster followed by another idle time.
> >
> > This is probably worse when clients run a logrotate script with a
client
> > restart from a cronjob.
> >
> > I would suggest something like the runinterval be $runinterval +
> > $(random number of seconds between 1 and 120) || $(random number of
> > seconds between 1 and $runinterval*0.1 )
> >
> > That should help spread out the load on the server.
> >
> >
> >
> >
>
>
> >
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

I''ve also gone for cron, but I really randomize it by running
puppetd from a wrapper script:

#!/bin/bash

RANDOM=$$$(date +%s)
d=$[ ( $RANDOM % 900 ) + 1 ]

sleep $d
/opt/sysadmin/bin/puppetd -t

I know it''s not absolutely 100% random, but it''s random enough
for this.  The risk of course is that you could still get clumps
but I''ve found that four puppetmasters running with mongrel+apache
are enough to not get anywhere near overloaded with ~200 nodes.

I use the ''sleep'' command rather than --splay because --splay
wasn''t working for me this time last year.  Don''t recall what
the
problem was any more, but this works pretty well.

Matt

Ohad Levy wrote:
> I also use cron, but I found using the IP address last bit as a more
> effective way to "randomize" it.
> 
> more info here:
> http://reductivelabs.com/trac/puppet/wiki/Recipes/cron
> 
> Cheers,
> Ohad
> 
> On Fri, Jan 9, 2009 at 9:21 AM, Ryan Dooley <ryan.dooley@gmail.com>
wrote:
> 
> >
> > Have you checked out the splay option?
> >
> > Alternatively, I''ve since moved on from splay to use a cron
job to run
> > puppetd.  I have a meta package that pulls in puppet and drops in a
root
> > cron job entry based on the mac address of the host''s primary
interface
> > during the post-install step of rpm.
> >
> > Cheers,
> > Ryan
> >
> > On 1/8/09 12:17 PM, Christopher wrote:
> > > Hello...
> > >
> > > I have an existing puppet setup that works very well except for
one
> > > thing.  Occasionally the puppetmaster is busy and the clients
tend to
> > > "bunch up".  Then on the next client run they tend to
"bunch up" again
> > > causing the puppetmaster to be slow down due to the high load
which
> > > causes more slow downs.  This example should show what is
happending:
> > >
> > > shell>  grep ": Compiled catalog for " all.log | cut
-d: -f 1-2 | uniq -c
> > >     1 Jan  8 10:59
> > >     8 Jan  8 11:00
> > >    14 Jan  8 11:01
> > >     2 Jan  8 11:02
> > >     1 Jan  8 11:08
> > >    15 Jan  8 11:10
> > >    12 Jan  8 11:11
> > >     1 Jan  8 11:20
> > >
> > > The above shows an idle time followed by a rush of puppet clients
> > > hitting the puppetmaster followed by another idle time.
> > >
> > > This is probably worse when clients run a logrotate script with a
client
> > > restart from a cronjob.
> > >
> > > I would suggest something like the runinterval be $runinterval +
> > > $(random number of seconds between 1 and 120) || $(random number
of
> > > seconds between 1 and $runinterval*0.1 )
> > >
> > > That should help spread out the load on the server.
> > >
> > >
> > >
> > >
> >
> >
> > >
> >
> 
> > 
-- 
* Matt McLeod | mail: matt@boggle.org | blog: http://abortrephrase.com/ *
     --- People can do the work, so machines have time to think ---

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

+1 for sleep, splay loads puppet into memory (which can be a more than a few
MB in mem)

On Fri, Jan 9, 2009 at 2:45 PM, Matt McLeod <matt@boggle.org> wrote:
>
> I''ve also gone for cron, but I really randomize it by running
> puppetd from a wrapper script:
>
> #!/bin/bash
>
> RANDOM=$$$(date +%s)
> d=$[ ( $RANDOM % 900 ) + 1 ]
>
> sleep $d
> /opt/sysadmin/bin/puppetd -t
>
> I know it''s not absolutely 100% random, but it''s random
enough
> for this.  The risk of course is that you could still get clumps
> but I''ve found that four puppetmasters running with mongrel+apache
> are enough to not get anywhere near overloaded with ~200 nodes.
>
> I use the ''sleep'' command rather than --splay because
--splay
> wasn''t working for me this time last year.  Don''t recall
what the
> problem was any more, but this works pretty well.
>
> Matt
>
> Ohad Levy wrote:
> > I also use cron, but I found using the IP address last bit as a more
> > effective way to "randomize" it.
> >
> > more info here:
> > http://reductivelabs.com/trac/puppet/wiki/Recipes/cron
> >
> > Cheers,
> > Ohad
> >
> > On Fri, Jan 9, 2009 at 9:21 AM, Ryan Dooley
<ryan.dooley@gmail.com>
> wrote:
> >
> > >
> > > Have you checked out the splay option?
> > >
> > > Alternatively, I''ve since moved on from splay to use a
cron job to run
> > > puppetd.  I have a meta package that pulls in puppet and drops in
a
> root
> > > cron job entry based on the mac address of the host''s
primary interface
> > > during the post-install step of rpm.
> > >
> > > Cheers,
> > > Ryan
> > >
> > > On 1/8/09 12:17 PM, Christopher wrote:
> > > > Hello...
> > > >
> > > > I have an existing puppet setup that works very well except
for one
> > > > thing.  Occasionally the puppetmaster is busy and the
clients tend to
> > > > "bunch up".  Then on the next client run they tend
to "bunch up"
> again
> > > > causing the puppetmaster to be slow down due to the high
load which
> > > > causes more slow downs.  This example should show what is
happending:
> > > >
> > > > shell>  grep ": Compiled catalog for " all.log
| cut -d: -f 1-2 |
> uniq -c
> > > >     1 Jan  8 10:59
> > > >     8 Jan  8 11:00
> > > >    14 Jan  8 11:01
> > > >     2 Jan  8 11:02
> > > >     1 Jan  8 11:08
> > > >    15 Jan  8 11:10
> > > >    12 Jan  8 11:11
> > > >     1 Jan  8 11:20
> > > >
> > > > The above shows an idle time followed by a rush of puppet
clients
> > > > hitting the puppetmaster followed by another idle time.
> > > >
> > > > This is probably worse when clients run a logrotate script
with a
> client
> > > > restart from a cronjob.
> > > >
> > > > I would suggest something like the runinterval be
$runinterval +
> > > > $(random number of seconds between 1 and 120) || $(random
number of
> > > > seconds between 1 and $runinterval*0.1 )
> > > >
> > > > That should help spread out the load on the server.
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > > >
> > >
> >
> > >
>
> --
> * Matt McLeod | mail: matt@boggle.org | blog: http://abortrephrase.com/ *
>     --- People can do the work, so machines have time to think ---
>
> >
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---