Puppet users - Oct 2008 - Puppetmaster and clients running really slow

Hello list.


We''re running puppet version 0.24.4, and are having trouble with
puppetmaster running really slow. The puppetmaster has been running fine
until about a couple of day ago, and for no apparent reason it just lost
performance. I''ve also noticed that the puppet daemon on many of our
clients
a dying. Both master and clients run very slow, but I''m not sure if the
client runs slow because of the master.

The symptoms I''ve seen are that running puppet on a client often takes
a
very long time (one of the nodes used close to 800 seconds to run the
config), or even times out. Once in a while, however, things seems back to
normal, and the run is executed in a few seconds. On the server side the
time between "Compiled configuration for ..." messages takes a very
long
time (often several minutes), in contrast to normal operations where
configurations get compiled quite frequently.

What puzzles me is that there are no signs of resource starvation on the
server - we have low CPU and memory utilization. So there shouldn''t be,
as
far as I can see, any hardware limitations causing puppetmaster to run slow.


I don''t have the exact number of active puppet nodes, but it should be
well
under 100 nodes.

Other than that, I can mention that "storeconfig" is set to
"true"
in puppet.conf.

And another question/observation: I started a client run more than five
minutes ago by issuing "puppet --server puppetmaster --test", and it
is
either running very slow or something has crashed. The last message on the
screen is "notice: Starting catalogue run", and it''s been
like that since I
started the run. Is this a symptom of a problem at the client side, or at
the master side? I mean, when I get to the message just mentions, I''d
think
that the puppetmaster has done it''s part, and now it''s up to
the client to
execute. So maybe this indicates that there is something wrong at the client
side also?


Anyways, any tip on how to figure this out will be very appreciated, as
we''re quite dependent on puppet in our daily work.


Regards,
Kenneth Holter

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Are you using Webrick? If so, use Mongrel!

There''s a wiki page on it (UsingMongrel iirc). I''d recommend
nginx as
your proxy loadbalancer.

On 11/10/2008, at 12:3 AM, "Kenneth Holter"
<kenneho.ndu@gmail.com>
wrote:
>
> Hello list.
>
>
> We''re running puppet version 0.24.4, and are having trouble with  
> puppetmaster running really slow. The puppetmaster has been running  
> fine until about a couple of day ago, and for no apparent reason it  
> just lost performance. I''ve also noticed that the puppet daemon on
> many of our clients a dying. Both master and clients run very slow,  
> but I''m not sure if the client runs slow because of the master.
>
> The symptoms I''ve seen are that running puppet on a client often  
> takes a very long time (one of the nodes used close to 800 seconds  
> to run the config), or even times out. Once in a while, however,  
> things seems back to normal, and the run is executed in a few  
> seconds. On the server side the time between "Compiled configuration  
> for ..." messages takes a very long time (often several minutes), in  
> contrast to normal operations where configurations get compiled  
> quite frequently.
>
> What puzzles me is that there are no signs of resource starvation on  
> the server - we have low CPU and memory utilization. So there  
> shouldn''t be, as far as I can see, any hardware limitations
causing
> puppetmaster to run slow.
>
> I don''t have the exact number of active puppet nodes, but it
should
> be well under 100 nodes.
>
> Other than that, I can mention that "storeconfig" is set to
"true"
> in puppet.conf.
>
> And another question/observation: I started a client run more than  
> five minutes ago by issuing "puppet --server puppetmaster
--test",
> and it is either running very slow or something has crashed. The  
> last message on the screen is "notice: Starting catalogue run",
and
> it''s been like that since I started the run. Is this a symptom of
a
> problem at the client side, or at the master side? I mean, when I  
> get to the message just mentions, I''d think that the puppetmaster
> has done it''s part, and now it''s up to the client to
execute. So
> maybe this indicates that there is something wrong at the client  
> side also?
>
>
> Anyways, any tip on how to figure this out will be very appreciated,  
> as we''re quite dependent on puppet in our daily work.
>
>
> Regards,
> Kenneth Holter
>
> >
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Fri, Oct 10, 2008 at 4:39 AM, Kenneth Holter <kenneho.ndu@gmail.com>
wrote:
>
> Hello list.
>
>
> We''re running puppet version 0.24.4, and are having trouble with
> puppetmaster running really slow. The puppetmaster has been running fine
> until about a couple of day ago, and for no apparent reason it just lost
> performance. I''ve also noticed that the puppet daemon on many of
our clients
> a dying. Both master and clients run very slow, but I''m not sure
if the
> client runs slow because of the master.
>
> The symptoms I''ve seen are that running puppet on a client often
takes a
> very long time (one of the nodes used close to 800 seconds to run the
> config), or even times out. Once in a while, however, things seems back to
> normal, and the run is executed in a few seconds. On the server side the
> time between "Compiled configuration for ..." messages takes a
very long
> time (often several minutes), in contrast to normal operations where
> configurations get compiled quite frequently.
>
> What puzzles me is that there are no signs of resource starvation on the
> server - we have low CPU and memory utilization. So there
shouldn''t be, as
> far as I can see, any hardware limitations causing puppetmaster to run
slow.
>
> I don''t have the exact number of active puppet nodes, but it
should be well
> under 100 nodes.
>
> Other than that, I can mention that "storeconfig" is set to
"true"
> in puppet.conf.
<snip>

This is the telling point for me. I ran into the same problem myself.
The storeconfig database has some terrible issues. I wish I could tell
you how to solve this; we ended up just abandoning stored configs.
Others have had better luck.

One thing you might try is removing stale entries from your stored
config database. There''s a script floating around out there...

--Paul

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Tue, 2008-10-14 at 10:22 -0700, Paul Lathrop wrote:
> On Fri, Oct 10, 2008 at 4:39 AM, Kenneth Holter
<kenneho.ndu@gmail.com> wrote:
> >
> > Hello list.
> >
> >
> > We''re running puppet version 0.24.4, and are having trouble
with
> > puppetmaster running really slow. The puppetmaster has been running
fine
> > until about a couple of day ago, and for no apparent reason it just
lost
> > performance. I''ve also noticed that the puppet daemon on many
of our clients
> > a dying. Both master and clients run very slow, but I''m not
sure if the
> > client runs slow because of the master.
> >
> > The symptoms I''ve seen are that running puppet on a client
often takes a
> > very long time (one of the nodes used close to 800 seconds to run the
> > config), or even times out. Once in a while, however, things seems
back to
> > normal, and the run is executed in a few seconds. On the server side
the
> > time between "Compiled configuration for ..." messages takes
a very long
> > time (often several minutes), in contrast to normal operations where
> > configurations get compiled quite frequently.
> >
> > What puzzles me is that there are no signs of resource starvation on
the
> > server - we have low CPU and memory utilization. So there
shouldn''t be, as
> > far as I can see, any hardware limitations causing puppetmaster to run
slow.
> >
> > I don''t have the exact number of active puppet nodes, but it
should be well
> > under 100 nodes.
> >
> > Other than that, I can mention that "storeconfig" is set to
"true"
> > in puppet.conf.
> 
> <snip>
> 
> This is the telling point for me. I ran into the same problem myself.
> The storeconfig database has some terrible issues. I wish I could tell
> you how to solve this; we ended up just abandoning stored configs.
> Others have had better luck.
> 
> One thing you might try is removing stale entries from your stored
> config database. There''s a script floating around out there...
0.24.6 will contain (actually 0.24.6-rc1 contains) a patch that fixes
several performance issues with storeconfig. 
Peter Meier even reported that it reduced its master (or sql) load by
half :
http://groups.google.com/group/puppet-dev/browse_thread/thread/6a74e8103eff4313/363d90830880690c#363d90830880690c

My test show that on my master, catalog compilation for a 500 resources
node takes now 6s instead of 15s before my patch (storeconfig was about
60% of the compilation time).

The complete information about the patch is here:
http://projects.reductivelabs.com/issues/show/1502

I also submitted a patch to refine the current db schema which uses
multiple unused index (it is also part of 0.24.6-rc1). You need to
migrate the schema with --dbmigrate:
http://projects.reductivelabs.com/issues/show/1506

-- 
Brice Figureau <brice-puppet@daysofwonder.com>


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---