Puppet users - Jan 2013 - Terrible exported resources performance

Hi, 

Since the below is a little long, I put my question at the top: how do I 
troubleshoot awful exported resources performance in puppet and is there 
anything I can tweak to get it to run under 10 minutes in larger 
environments?

I have a fairly modest environment (118 nodes, but prod will be at least 
twice as large). I''m trying to move my distributed nagios setup to one 
based on exported resources and it works brilliantly in a smaller env (35 
nodes), but with 118 nodes it''s not so great anymore. 

All the hosts export a nagios host and services (up to 20) with 
corresponding commands, plus files for them. It''s all based on this: 
https://github.com/camptocamp/puppet-nagios , but fixed so that it''s 
working :). I appreciate it''s not the most efficient way of doing
things,
but it makes my setup insanely easy to configure, comparing to the previous 
one, with a combination of static files, templates, multiple
''profiles'',
duplicated configs, etc.

In the small env puppet agent takes about 2 minutes to run on nagios server 
and collect resources.

In the larger env it takes about 70 minutes, if it manages to finish at 
all. Initially, as a "quick" test, I was running puppetdb without
postgres
and had to give it 5GB to get it to finish at all (70 mins). With postgres 
8.4, load on the puppetmaster is significantly reduced, but with 512MB for 
puppetdb (128 + 1MB per node, and then double it for good measure) puppetdb 
still runs out of memory. I set it to 1GB and puppedb just crashed again 
(I''ve got dumps). Trying with 2GB now. I haven''t fiddled with
thread
settings, but my puppet agents aren''t deamonized or
''croned'', I run them
using mcollective or manually. So there''s only a single puppet agent 
running during this test, on the core nagios server. It seems that
there''s
a ruby process taking 100% of one core during this run and nothing else 
"dramatic" seems to be happening (except for puppetdb dying of
course).

It''s all on centos 5.8 with puppetlabs ruby and puppet packages, on 
apache+passenger:

ruby-1.8.7.370-1.el5
ruby-rdoc-1.8.7.370-1.el5
rubygem-stomp-1.2.2-1.el5
rubygem-json-1.4.6-2.el5
rubygem-fastthread-1.0.7-1
rubygem-rake-0.8.7-2
rubygem-daemon_controller-0.2.5-1
rubygem-passenger-native-3.0.12-1
puppetdb-terminus-1.0.5-1.el5
puppet-server-3.0.2-1.el5
ruby-libs-1.8.7.370-1.el5
ruby-irb-1.8.7.370-1.el5
rubygems-1.3.7-1.el5
ruby-shadow-1.4.1-7
libselinux-ruby-1.33.4-5.7.el5
ruby-augeas-0.4.1-1
puppet-3.0.2-1.el5
rubygem-rack-1.1.0-2.el5
rubygem-passenger-3.0.12-1
rubygem-passenger-native-libs-3.0.12-1_1.8.7.370
puppetdb-1.0.5-1.el5

Passenger config:

    PassengerHighPerformance on
    PassengerMaxPoolSize 12
    PassengerPoolIdleTime 1500
    PassengerMaxRequests 1000
    PassengerStatThrottleRate 120


Regards,
Daniel

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/bLnpN2PTnQsJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 21.01.2013 14:05, Daniel wrote:
> Hi,
>
> Since the below is a little long, I put my question at the top: how do I
> troubleshoot awful exported resources performance in puppet and is there
> anything I can tweak to get it to run under 10 minutes in larger
> environments?
There is a "simple" answer: puppetdb.

There''ll still be bottlenecks to wring, but collection performance
won''t
be one of them.


Best Regards, David

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi,

Is there a way to define a variable  in puppet.conf file ?

Something like  :
My_var = my_value in puppet.conf

Then to be able to use it in manifest with something like :
   file {custo.txt'':
    path=> $my_value,
    ensure=> present,
    mode=> 0640,
    content=> "this content",
 }

Sincerly,

Bernard Granier

#
" This e-mail and any attached documents may contain confidential or
proprietary information. If you are not the intended recipient, you are notified
that any dissemination, copying of this e-mail and any attachments thereto or
use of their contents by any means whatsoever is strictly prohibited. If you
have received this e-mail in error, please advise the sender immediately and
delete this e-mail and all attached documents from your computer system."
#

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Something like this would typically go in your manifest files, not the
puppet.conf file.  I think you will find the Puppet language guide
instructive, specifically the page on variables:
    http://docs.puppetlabs.com/puppet/2.7/reference/lang_variables.html

You should think of Puppet as more of a development language than a
tool that you simply configure using a .conf file.  The site.pp is
simply the entry point into the structure you build with the
manifests, and you can do quite a lot with the language.


❧ Brian Mathis


On Mon, Jan 21, 2013 at 11:58 AM, GRANIER Bernard (MORPHO)
<bernard.granier@morpho.com> wrote:
> Hi,
>
> Is there a way to define a variable  in puppet.conf file ?
>
> Something like  :
> My_var = my_value in puppet.conf
>
> Then to be able to use it in manifest with something like :
>    file {custo.txt'':
>     path=> $my_value,
>     ensure=> present,
>     mode=> 0640,
>     content=> "this content",
>  }
>
> Sincerly,
>
> Bernard Granier
>
> #
> " This e-mail and any attached documents may contain confidential or
proprietary information. If you are not the intended recipient, you are notified
that any dissemination, copying of this e-mail and any attachments thereto or
use of their contents by any means whatsoever is strictly prohibited. If you
have received this e-mail in error, please advise the sender immediately and
delete this e-mail and all attached documents from your computer system."
> #
>
> --
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Thanks for the advice, I will use variables to define specific values,

Sincerely,


Bernard Granier


-----Original Message-----
From: puppet-users@googlegroups.com [mailto:puppet-users@googlegroups.com] On
Behalf Of Brian Mathis
Sent: Monday, January 21, 2013 6:09 PM
To: puppet-users@googlegroups.com
Subject: Re: [Puppet Users] puppet configuration and variables ?

Something like this would typically go in your manifest files, not the
puppet.conf file.  I think you will find the Puppet language guide instructive,
specifically the page on variables:
    http://docs.puppetlabs.com/puppet/2.7/reference/lang_variables.html

You should think of Puppet as more of a development language than a tool that
you simply configure using a .conf file.  The site.pp is simply the entry point
into the structure you build with the manifests, and you can do quite a lot with
the language.


❧ Brian Mathis


On Mon, Jan 21, 2013 at 11:58 AM, GRANIER Bernard (MORPHO)
<bernard.granier@morpho.com> wrote:
> Hi,
>
> Is there a way to define a variable  in puppet.conf file ?
>
> Something like  :
> My_var = my_value in puppet.conf
>
> Then to be able to use it in manifest with something like :
>    file {custo.txt'':
>     path=> $my_value,
>     ensure=> present,
>     mode=> 0640,
>     content=> "this content",
>  }
>
> Sincerly,
>
> Bernard Granier
>
> #
> " This e-mail and any attached documents may contain confidential or
proprietary information. If you are not the intended recipient, you are notified
that any dissemination, copying of this e-mail and any attachments thereto or
use of their contents by any means whatsoever is strictly prohibited. If you
have received this e-mail in error, please advise the sender immediately and
delete this e-mail and all attached documents from your computer system."
> #
>
> --
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
>
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

#
" This e-mail and any attached documents may contain confidential or
proprietary information. If you are not the intended recipient, you are notified
that any dissemination, copying of this e-mail and any attachments thereto or
use of their contents by any means whatsoever is strictly prohibited. If you
have received this e-mail in error, please advise the sender immediately and
delete this e-mail and all attached documents from your computer system."
#

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi Daniel,

On Monday, January 21, 2013 1:05:26 PM UTC, Daniel
wrote:
>
> In the larger env it takes about 70 minutes, if it manages to finish at 
> all. Initially, as a "quick" test, I was running puppetdb without
postgres
> and had to give it 5GB to get it to finish at all (70 mins). With postgres 
> 8.4, load on the puppetmaster is significantly reduced, but with 512MB for 
> puppetdb (128 + 1MB per node, and then double it for good measure) puppetdb
> still runs out of memory. I set it to 1GB and puppedb just crashed again 
> (I''ve got dumps). Trying with 2GB now. I haven''t fiddled
with thread
> settings, but my puppet agents aren''t deamonized or
''croned'', I run them
> using mcollective or manually. So there''s only a single puppet
agent
> running during this test, on the core nagios server. It seems that
there''s
> a ruby process taking 100% of one core during this run and nothing else 
> "dramatic" seems to be happening (except for puppetdb dying of
course).
>
>
Given enough RAM it doesn''t sound like PuppetDB is the problem any
more, is
that correct?

The Ruby process is most likely a Puppet Master thread doing the catalog 
construction. I think your suffering from a similar problem that we had 
recently, where it''s not specifically resource collection
that''s taking up
all the time, it''s the Puppet Master turning the exported resources 
information into one enormous catalog that takes too long.

We got around this by bypassing exported resources and querying the 
information from PuppetDB directly and using that information in a 
template. I suggested the following to another user a few days ago in this 
thread:

https://groups.google.com/forum/#!topic/puppet-users/X6Lm-0_etbA

-Luke

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/GOV7Nh_co1EJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Monday, January 21, 2013 1:05:26 PM UTC, Daniel
wrote:
>
> In the larger env it takes about 70 minutes, if it manages to finish at 
>> all. Initially, as a "quick" test, I was running puppetdb
without postgres
>> and had to give it 5GB to get it to finish at all (70 mins). With
postgres
>> 8.4, load on the puppetmaster is significantly reduced, but with 512MB
for
>> puppetdb (128 + 1MB per node, and then double it for good measure)
puppetdb
>> still runs out of memory. I set it to 1GB and puppedb just crashed
again
>> (I''ve got dumps). Trying with 2GB now. I haven''t
fiddled with thread
>> settings, but my puppet agents aren''t deamonized or
''croned'', I run them
>> using mcollective or manually. So there''s only a single puppet
agent
>> running during this test, on the core nagios server. It seems that
there''s
>> a ruby process taking 100% of one core during this run and nothing else
>> "dramatic" seems to be happening (except for puppetdb dying
of course).
>>
>>
> Given enough RAM it doesn''t sound like PuppetDB is the problem any
more,
> is that correct?
>
> The Ruby process is most likely a Puppet Master thread doing the catalog 
> construction. I think your suffering from a similar problem that we had 
> recently, where it''s not specifically resource collection
that''s taking up
> all the time, it''s the Puppet Master turning the exported
resources
> information into one enormous catalog that takes too long.
>
> We got around this by bypassing exported resources and querying the 
> information from PuppetDB directly and using that information in a 
> template. I suggested the following to another user a few days ago in this 
> thread:
>
> https://groups.google.com/forum/#!topic/puppet-users/X6Lm-0_etbA
>
>
Hi Luke, 

This sounds like a sensible workaround, I will definitely have a look. I 
haven''t yet had enough time to look at the issue properly, but it seems
that this very long time is indeed consumed by catalog construction. 
Puppetdb fails after this is finished, so it seems that it dies when nagios 
host tries to report its catalog back.

To be honest it''s very disappointing that puppet and puppetdb
can''t handle
few thousand exported resources and that workarounds like that are needed. 
I might need to look at alternatives.


Regards,
Daniel

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/uXWPXZLe_FgJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

> This sounds like a sensible workaround, I will definitely have a look. I
> haven''t yet had enough time to look at the issue properly, but it
seems that
> this very long time is indeed consumed by catalog construction. Puppetdb
> fails after this is finished, so it seems that it dies when nagios host
> tries to report its catalog back.
Do you mean it dies from an OOM when it tries to report the catalogue back?

ken.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Tue, Jan 22, 2013 at 3:04 PM, Ken Barber <ken@puppetlabs.com>
wrote:
>> This sounds like a sensible workaround, I will definitely have a look.
I
>> haven''t yet had enough time to look at the issue properly, but
it seems that
>> this very long time is indeed consumed by catalog construction.
Puppetdb
>> fails after this is finished, so it seems that it dies when nagios host
>> tries to report its catalog back.
>
> Do you mean it dies from an OOM when it tries to report the catalogue back?
Yes, that''s what it looks like. Of course I can prevent it by giving
it more memory (which I did), but I already have postgres backed
puppetdb and had to give puppetdb 3GB, or a puppet agent run on a
single host (OK, with thousands of exported resources to collect and
process) that takes about 70 minutes can still kill it. This waiting
70 minutes for it to die is an insult to injury... Overall not great.
I''m happy to redo this setup if I''m doing something wrong, but
it just
seems like this is exponential (30-odd nodes - 2 minutes, 100-odd
nodes, 70 minutes).

Regards,
Daniel

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

How many resources are we talking for the 100 nodes?

If you''re running puppetdb one way to do this is to get a report on
the nagios server that does the collection:

curl -G -H ''Accept: application/json''
''http://puppetdb:8080/resources''
--data-urlencode ''query=["=", ["node",
"name"], "nagiosserver"]'' | p
resource | wc -l

(this at least works on Puppetdb 1.0.5 - as JSON output is pretty-printed)

It would be interesting to see if this is an artifact of collection or
the nagios resource specifically, although the only good way I know to
do this is to perhaps replace the nagios exported resource cases with
something simple - such as a noop style resource like ''whit''
or
''anchor''.

When you run puppet agent -t ... if you do a summarize what is the
result? You should get something like this:

# puppet agent -t --summarize
info: Caching catalog for puppetdb1.vm
info: Applying configuration version ''1358813527''
notice: exported
notice: /Stage[main]//Node[puppetdb1.vm]/Notify[exported]/message:
defined ''message'' as ''exported''
notice: asdfasdfsasdf
notice: /Stage[main]//Node[puppetdb1.vm]/Notify[foo]/message: defined
''message'' as ''asdfasdfsasdf''
notice: Finished catalog run in 0.03 seconds
Changes:
            Total: 2
Events:
            Total: 2
          Success: 2
Resources:
          Changed: 2
      Out of sync: 2
          Skipped: 6
            Total: 9
Time:
       Filebucket: 0.00
           Notify: 0.00
   Config retrieval: 0.90
            Total: 0.90
         Last run: 1358813994
Version:
           Config: 1358813527
           Puppet: 2.7.18

I think Luke''s suggestion about tapping the information in puppetdb
using functions is not a bad work-around however, but its
disappointing that exported resources in this case isn''t just
_working_ :-(.

ken.

On Tue, Jan 22, 2013 at 3:19 PM, Daniel Siechniewicz
<daniel@nulldowntime.com> wrote:
> On Tue, Jan 22, 2013 at 3:04 PM, Ken Barber <ken@puppetlabs.com>
wrote:
>>> This sounds like a sensible workaround, I will definitely have a
look. I
>>> haven''t yet had enough time to look at the issue properly,
but it seems that
>>> this very long time is indeed consumed by catalog construction.
Puppetdb
>>> fails after this is finished, so it seems that it dies when nagios
host
>>> tries to report its catalog back.
>>
>> Do you mean it dies from an OOM when it tries to report the catalogue
back?
>
> Yes, that''s what it looks like. Of course I can prevent it by
giving
> it more memory (which I did), but I already have postgres backed
> puppetdb and had to give puppetdb 3GB, or a puppet agent run on a
> single host (OK, with thousands of exported resources to collect and
> process) that takes about 70 minutes can still kill it. This waiting
> 70 minutes for it to die is an insult to injury... Overall not great.
> I''m happy to redo this setup if I''m doing something
wrong, but it just
> seems like this is exponential (30-odd nodes - 2 minutes, 100-odd
> nodes, 70 minutes).
>
> Regards,
> Daniel
>
> --
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Tuesday, January 22, 2013 9:19:02 AM UTC-6, Daniel
wrote:
>
> On Tue, Jan 22, 2013 at 3:04 PM, Ken Barber
<k...@puppetlabs.com<javascript:>>
> wrote: 
> >> This sounds like a sensible workaround, I will definitely have a
look.
> I 
> >> haven''t yet had enough time to look at the issue
properly, but it seems
> that 
> >> this very long time is indeed consumed by catalog construction. 
> Puppetdb 
> >> fails after this is finished, so it seems that it dies when nagios
host
> >> tries to report its catalog back. 
> > 
> > Do you mean it dies from an OOM when it tries to report the catalogue 
> back? 
>
> Yes, that''s what it looks like. Of course I can prevent it by
giving
> it more memory (which I did), but I already have postgres backed 
> puppetdb and had to give puppetdb 3GB, or a puppet agent run on a 
> single host (OK, with thousands of exported resources to collect and 
> process) that takes about 70 minutes can still kill it. This waiting 
> 70 minutes for it to die is an insult to injury... Overall not great. 
> I''m happy to redo this setup if I''m doing something
wrong, but it just
> seems like this is exponential (30-odd nodes - 2 minutes, 100-odd 
> nodes, 70 minutes). 
>
>
This sounds like a memory-management problem.  If you''re running up
against
(or close to) a memory limit, then the Java-based puppetdb will start doing 
a lot of garbage collection as available heap memory becomes scarce.  That 
will greatly slow down the whole Java VM, especially so if the GC runs 
don''t free much memory.  Also, if you give the VM more memory than 
available RAM then you will have slowdowns related to paging parts of the 
heap between physical and virtual memory; depending on memory access 
patterns, that could happen frequently and be very costly.

On the other hand, some versions of the Java VM have problems with large 
amounts of heap memory.  One would hope that such problems have been 
resolved by now, but you could anyway try reducing the heap memory to about 
1.5G.  That''s close to the maximum that you can safely use with some VM
implementations.

I certainly agree that puppetdb has a serious problem if it requires such 
large amounts of memory to handle resource collections with a few thousand 
elements.  If roughly 3000 (at ~30 resources * ~100 nodes) exported 
resources require in excess of 3GB then that''s in the vicinity of 1MB
per
resource -- absurd!

On the other hand, that''s *so* absurd that I suspect something else is 
going on here.  It smacks of a combinatoric problem, either in your 
manifests or in the puppetdb implementation.  There aren''t enough data 
points to be sure, but your run times could be scaling with the square of 
the number of nodes.  Is there any chance that the number of exported 
resources is scaling the same way?  That would explain both time and memory 
consumption.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/25e0ER5_C4gJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

This is now reported here:

http://projects.puppetlabs.com/issues/18804

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/ZpyFiFkYjawJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Seems that chaining exported resources might not be too efficient and 
produces lots of data that could be the reason for puppetdb crashing.

The culprits being these two lines in two manifest files:

./nsca/server.pp:  #File <<| tag == $get_tag |>> -> Nagios_host
<<| tag == $get_tag |>>
./nrpe/server.pp:  #File <<| tag == $get_tag |>> -> Nagios_host
<<| tag == $get_tag |>>

replacing them with unchained:

File <<| tag == $get_tag |>>
Nagios_host <<| tag == $get_tag |>>

causes it to run even with 1GB for puppetdb (still a 16GB vm) in under 10 
mins, which is acceptable.

This is tracked under bug #18804

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/S-md_OBZEF8J.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Thursday, January 24, 2013 5:24:58 AM UTC-6, Daniel
wrote:
>
> Seems that chaining exported resources might not be too efficient and 
> produces lots of data that could be the reason for puppetdb crashing.
>
> The culprits being these two lines in two manifest files:
>
> ./nsca/server.pp:  #File <<| tag == $get_tag |>> ->
Nagios_host <<| tag == $get_tag |>>
> ./nrpe/server.pp:  #File <<| tag == $get_tag |>> ->
Nagios_host <<| tag == $get_tag |>>
>
>
And there may be your combinatorial problem.  Supposing that the tags are 
not specific to the exporting node, the numbers of Files and Nagios_hosts 
increase linearly with the number of nodes, but the number of relationships 
among them increases with the square of the number of nodes.
 
> replacing them with unchained:
>
> File <<| tag == $get_tag |>>
> Nagios_host <<| tag == $get_tag |>>
>
> causes it to run even with 1GB for puppetdb (still a 16GB vm) in under 10 
> mins, which is acceptable.
>
Do you in fact need the order of application constraints that the chaining 
declared?  All of them?

This is an area that seems to give people problems.  On one hand, Puppet 
provides a couple of mechanisms that allow people to compactly declare 
large numbers of ordering relationships.  On the other hand, people 
sometimes lose sight of the distinction between functional dependency and 
application-order dependency.  Either one of these can cause trouble, as 
unneeded declarations of any kind slow Puppet and make it consume more 
resources.  They also make you manifests more susceptible to dependency 
cycles.  The combination is worse, however, because the number of unneeded 
relationships produced can be multiplicative.

Depending on what relationships you actually need, you have various options:

1) If it doesn''t actually matter whether Puppet syncs given Files
before or
after it syncs any particular Nagios_host, then you never needed any 
relationships at all, and simply removing the chaining as you did is the 
best solution.

2) If specific Files must be synced before specific Nagios_hosts, then you 
should express those relationships and not any others.  In particular, if 
you only need relationships among Files and Nagios_hosts exported by the 
same node, then you should declare the relationships on the exporting side 
instead of on the collecting side.  (And if they always come in such pairs 
then perhaps you should wrap then in a defined type and export that 
instead).

3) If you really need something along the lines that the chaining expressed 
-- all Files with a given tag synced before all Nagios_hosts bearing that 
tag -- then you can break the combinatorial problem by introducing an 
artificial barrier resource for each tag:

define mymodule::barrier() {
  # no content
}

mymodule::barrier { $get_tag: }

File<<| tag == $get_tag |>> -> Mymodule::Barrier[ $get_tag ]
-> Nagios_host
<<| tag == $get_tag |>>

That gives you (number(Files) + number(Nagios_host)) relationships for each 
tag instead of (number(Files) * number(Nagios_host)) relationships.

Of course, you can use any single resource as a barrier; it doesn''t
need to
be of a defined type.  In some cases there might be a real resource that 
naturally fills the role.  If you have Puppet''s stdlib module then the 
Anchor resource would be a good fit; it is intended for a similar purpose 
anyway.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/2_AvVt-Qm4oJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

I think John is on to something here, Daniel - I haven''t seen your
full content yet, but are you creating a file for each nagios_host, so
that you can use that as the ''target''? Thus creating a single
file for
each nagios_host entry?

If this is the case, the John is spot-on ... and you''re creating more
relationships than necessary. You probably want one file to one
nagios_* resource, and thus one ''edge'' for each, not this
many-to-many
case you seem to be creating at collection time.

Of course I''m just guessing, not having seen your content.

On Thu, Jan 24, 2013 at 2:41 PM, jcbollinger <John.Bollinger@stjude.org>
wrote:
> On Thursday, January 24, 2013 5:24:58 AM UTC-6, Daniel wrote:
>>
>> Seems that chaining exported resources might not be too efficient and
>> produces lots of data that could be the reason for puppetdb crashing.
>>
>> The culprits being these two lines in two manifest files:
>>
>> ./nsca/server.pp:  #File <<| tag == $get_tag |>> ->
Nagios_host <<| tag =>> $get_tag |>>
>> ./nrpe/server.pp:  #File <<| tag == $get_tag |>> ->
Nagios_host <<| tag =>> $get_tag |>>
>
>
>
> And there may be your combinatorial problem.  Supposing that the tags are
> not specific to the exporting node, the numbers of Files and Nagios_hosts
> increase linearly with the number of nodes, but the number of relationships
> among them increases with the square of the number of nodes.
>
>>
>> replacing them with unchained:
>>
>> File <<| tag == $get_tag |>>
>> Nagios_host <<| tag == $get_tag |>>
>>
>> causes it to run even with 1GB for puppetdb (still a 16GB vm) in under
10
>> mins, which is acceptable.
>
>
> Do you in fact need the order of application constraints that the chaining
> declared?  All of them?
>
> This is an area that seems to give people problems.  On one hand, Puppet
> provides a couple of mechanisms that allow people to compactly declare
large
> numbers of ordering relationships.  On the other hand, people sometimes
lose
> sight of the distinction between functional dependency and
application-order
> dependency.  Either one of these can cause trouble, as unneeded
declarations
> of any kind slow Puppet and make it consume more resources.  They also make
> you manifests more susceptible to dependency cycles.  The combination is
> worse, however, because the number of unneeded relationships produced can
be
> multiplicative.
>
> Depending on what relationships you actually need, you have various
options:
>
> 1) If it doesn''t actually matter whether Puppet syncs given Files
before or
> after it syncs any particular Nagios_host, then you never needed any
> relationships at all, and simply removing the chaining as you did is the
> best solution.
>
> 2) If specific Files must be synced before specific Nagios_hosts, then you
> should express those relationships and not any others.  In particular, if
> you only need relationships among Files and Nagios_hosts exported by the
> same node, then you should declare the relationships on the exporting side
> instead of on the collecting side.  (And if they always come in such pairs
> then perhaps you should wrap then in a defined type and export that
> instead).
>
> 3) If you really need something along the lines that the chaining expressed
> -- all Files with a given tag synced before all Nagios_hosts bearing that
> tag -- then you can break the combinatorial problem by introducing an
> artificial barrier resource for each tag:
>
> define mymodule::barrier() {
>   # no content
> }
>
> mymodule::barrier { $get_tag: }
>
> File<<| tag == $get_tag |>> -> Mymodule::Barrier[ $get_tag ]
-> Nagios_host
> <<| tag == $get_tag |>>
>
> That gives you (number(Files) + number(Nagios_host)) relationships for each
> tag instead of (number(Files) * number(Nagios_host)) relationships.
>
> Of course, you can use any single resource as a barrier; it
doesn''t need to
> be of a defined type.  In some cases there might be a real resource that
> naturally fills the role.  If you have Puppet''s stdlib module then
the
> Anchor resource would be a good fit; it is intended for a similar purpose
> anyway.
>
>
> John
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/puppet-users/-/2_AvVt-Qm4oJ.
>
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Thursday, January 24, 2013 2:41:53 PM UTC, jcbollinger
wrote:
>
> On Thursday, January 24, 2013 5:24:58 AM UTC-6, Daniel wrote:
>>
>> Seems that chaining exported resources might not be too efficient and 
>> produces lots of data that could be the reason for puppetdb crashing.
>>
>> The culprits being these two lines in two manifest files:
>>
>> ./nsca/server.pp:  #File <<| tag == $get_tag |>> ->
Nagios_host <<| tag == $get_tag |>>
>> ./nrpe/server.pp:  #File <<| tag == $get_tag |>> ->
Nagios_host <<| tag == $get_tag |>>
>>
>>
>
> And there may be your combinatorial problem.  Supposing that the tags are 
> not specific to the exporting node, the numbers of Files and Nagios_hosts 
> increase linearly with the number of nodes, but the number of relationships
> among them increases with the square of the number of nodes.
>
Do you in fact need the order of application constraints that the chaining
> declared?  All of them?
>
I don''t need all of them. The reason I added it was to fix problems
with
files not being created when puppet tried to realize services, etc., thus 
producing an incorrect nagios setup. It worked in my small environment and 
only gave me a pause after days of trying to figure out what the hell is 
going on in the larger one. 

Looks like I didn''t understand implications of this configuration, but
to
be honest I haven''t found much in terms of documentation on chaining 
exported resources, potential implications and problems. Now I know, so 
"I''ve learnt something today" :). 

One suggestion - I would have a "WFT?" moment a lot sooner if
--summarize
displayed a number of relationships generated by my crazy config. I
don''t
know if it''s possible/easy to implement, but may be worth having a look
at.

Depending on what relationships you actually need, you have various
options:
>
> 1) If it doesn''t actually matter whether Puppet syncs given Files
before
> or after it syncs any particular Nagios_host, then you never needed any 
> relationships at all, and simply removing the chaining as you did is the 
> best solution.
>
 
As stated above, I''ve added it after failures, so I do need some sort
of
relationships.


2) If specific Files must be synced before specific Nagios_hosts, then you
> should express those relationships and not any others.  In particular, if 
> you only need relationships among Files and Nagios_hosts exported by the 
> same node, then you should declare the relationships on the exporting side 
> instead of on the collecting side.  (And if they always come in such pairs 
> then perhaps you should wrap then in a defined type and export that 
> instead).
>
That''s the most likely approach I''m going to implement in the
"next
iteration" of this module. 


3) If you really need something along the lines that the chaining expressed
> -- all Files with a given tag synced before all Nagios_hosts bearing that 
> tag -- then you can break the combinatorial problem by introducing an 
> artificial barrier resource for each tag:
>
I most likely don''t need this, but will have it in mind for future 
experiments with exported resources.

Thanks for your comprehensive answer. 


Regards,
Daniel

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.