Puppet users - Jul 2011 - puppet never finishing when there are thousands of resources

We are using puppet to export nagios resources (now we''re using file
resources to represent them) and the number of resources is probably
in the thousands -- our puppet runs take 6GB (on the client side) and
then just sort of spin forever (it''s been running for a couple of
hours now -- and no, we''re not paging.)

We''re using puppet 2.6.4 and ruby 1.8.6 -- upgrading is on our
roadmap, but we need to fight a few other fires first.

Anyone have any suggestions for fixing this?  Or how to mitigate this?
 I wonder if something isn''t scaling -- like maybe the number of edges
in the puppet DAG is increasing exponentially....

Rich

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Tue, Jul 12, 2011 at 7:53 PM, Rich Rauenzahn <rrauenza@gmail.com>
wrote:
> We are using puppet to export nagios resources (now we''re using
file
> resources to represent them) and the number of resources is probably
> in the thousands -- our puppet runs take 6GB (on the client side) and
> then just sort of spin forever (it''s been running for a couple of
> hours now -- and no, we''re not paging.)
>
> We''re using puppet 2.6.4 and ruby 1.8.6 -- upgrading is on our
> roadmap, but we need to fight a few other fires first.
>
> Anyone have any suggestions for fixing this?  Or how to mitigate this?
>  I wonder if something isn''t scaling -- like maybe the number of
edges
> in the puppet DAG is increasing exponentially....
>
>
I would absolutely test 2.7.x as fast as you can. It should be a drop in
replacement for both master and agent, and may end up being your fastest
mitigation path.

Even 2.6.9 if you can try it, there have been a few changes since 2.6.4.

I tend to find that a visual scan of a dot file from a --graph run is a good
way to look for explosions in complexity.


-- 
Nigel Kersten
Product Manager, Puppet Labs
Twitter: @nigelkersten

*Join us for **PuppetConf *<http://www.bit.ly/puppetconfsig>
September 22nd and 23rd in Portland, Oregon, USA.
*
*

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Jul 12, 9:53 pm, Rich Rauenzahn <rraue...@gmail.com>
wrote:
> We are using puppet to export nagios resources (now we''re using
file
> resources to represent them) and the number of resources is probably
> in the thousands -- our puppet runs take 6GB (on the client side) and
> then just sort of spin forever (it''s been running for a couple of
> hours now -- and no, we''re not paging.)
>
> We''re using puppet 2.6.4 and ruby 1.8.6 -- upgrading is on our
> roadmap, but we need to fight a few other fires first.
>
> Anyone have any suggestions for fixing this?  Or how to mitigate this?
>  I wonder if something isn''t scaling -- like maybe the number of
edges
> in the puppet DAG is increasing exponentially....

I don''t know whether the size of the DAG is actually your problem, but
it sounds like it might be.  On that topic, then ...

Relationships involving whole classes, especially of one whole class
on another, tend to add a lot of edges that are not strictly
necessary, because they are reduced to multiple inter-resource
dependencies.  That is, given Class[''a''] ->
Class[''b''], you get a
relationship between each of the n resources of Class[''a''] and
each of
the m resources of Class[''b''], for n x m DAG edges.

Run stages magnify that effect, because they establish relationships
between each class in a given stage and each class in each later
stage.  If there is a total of r resources under management, then the
number of DAG edges can scale as poorly as O(r*r).  It would be a rare
manifest set indeed where all those relationships are actually needed.

If you must use run stages (not a given!) then keeping as many classes
as possible in the main stage will tend to improve DAG scaling.  The
scaling effect for individual class relationships can be somewhat
managed by careful factoring of your classes into those intended for
other modules to depend on (these contain only a few key resources)
and those intended for use only inside the same module, coupled with
using only resource-level relationships intra-module.

For the most precise control and minimum number of DAG edges, however,
use only resource-level relationships everywhere, and audit those
relationships ruthlessly.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Wed, Jul 13, 2011 at 8:27 AM, jcbollinger
<John.Bollinger@stjude.org>wrote:
>
> Relationships involving whole classes, especially of one whole class
> on another, tend to add a lot of edges that are not strictly
> necessary, because they are reduced to multiple inter-resource
> dependencies.  That is, given Class[''a''] ->
Class[''b''], you get a
> relationship between each of the n resources of
Class[''a''] and each of
> the m resources of Class[''b''], for n x m DAG edges.
>
Note that 2.7.x resolved this relationship explosion, which is one reason I
suggested trying that.

-- 
Nigel Kersten
Product Manager, Puppet Labs
Twitter: @nigelkersten

*Join us for **PuppetConf *<http://www.bit.ly/puppetconfsig>
September 22nd and 23rd in Portland, Oregon, USA.
*
*

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.