Puppet users - May 2013 - Puppet/Nagios/PuppetDB slow performance

Hi all,

For months now I''ve been using Puppet with PuppetDB backend to manage
my
Nagios configs. We now have 1200+ services being checked and including 
servicedependency, host, hostdependency, command and other Nagios 
resources being managed, that''s about 3000 resources in total. 
Unfortunately this means the monitoring server takes about 8 minutes to 
perform a Puppet run.

I''m not exactly sure what the bottleneck is - whether it''s
doing 3000
calls to PuppetDB (although the load on the PuppetDB server and its 
Postgres database are low), or whether it is parsing nagios_service.cfg 
that causes the slowdown. It''s about 500kb and 11,000 lines.

I know it is possible to squirt the Nagios config into several smaller 
files when the resources are collected on the monitoring server, but I 
gather this also means that Nagios resources cannot be purged if the 
default file locations are not used. The ability to purge old configs 
automatically is essential to our environment.

So does anyone have any suggestions on how to improve the performance of 
Puppet when handling large Nagios configs?

Cheers,
Jonathan

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Its probably the nagios resource, its well known for not scaling when
the files contain a lot of entries. That would explain why your
PuppetDB is showing low load - as its probably not the bottleneck.

The most performant work-around I''ve seen for this is that you can use
the nagios_* resources to export, but then use custom collectors that
query PuppetDB (using Dalen''s puppetdbquery for example:
https://github.com/dalen/puppet-puppetdbquery) that generates custom
templates for your nagios_*.cfg files. If nagios_service.cfg is the
main bottleneck, you can focus on this one to start with.

This way you continue to use the model of the nagios_* resources, but
at collection time you improve performance.

ken.

On Fri, May 24, 2013 at 1:53 PM, Jonathan Gazeley
<Jonathan.Gazeley@bristol.ac.uk> wrote:
> Hi all,
>
> For months now I''ve been using Puppet with PuppetDB backend to
manage my
> Nagios configs. We now have 1200+ services being checked and including
> servicedependency, host, hostdependency, command and other Nagios resources
> being managed, that''s about 3000 resources in total. Unfortunately
this
> means the monitoring server takes about 8 minutes to perform a Puppet run.
>
> I''m not exactly sure what the bottleneck is - whether
it''s doing 3000 calls
> to PuppetDB (although the load on the PuppetDB server and its Postgres
> database are low), or whether it is parsing nagios_service.cfg that causes
> the slowdown. It''s about 500kb and 11,000 lines.
>
> I know it is possible to squirt the Nagios config into several smaller
files
> when the resources are collected on the monitoring server, but I gather
this
> also means that Nagios resources cannot be purged if the default file
> locations are not used. The ability to purge old configs automatically is
> essential to our environment.
>
> So does anyone have any suggestions on how to improve the performance of
> Puppet when handling large Nagios configs?
>
> Cheers,
> Jonathan
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscribe@googlegroups.com.
> To post to this group, send email to puppet-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/puppet-users?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On 24.05.2013 16:58, Ken Barber wrote:
> Its probably the nagios resource, its well known for not scaling when
> the files contain a lot of entries. That would explain why your
> PuppetDB is showing low load - as its probably not the bottleneck.
I second that. We have mysql storeconfigs, about 5000 Services, 2,5 MB 
nagios_* files and >50k lines of config. The puppet run takes aprox 45 
minutes. I have heard people on irc say that its not uncommon.

  - alex

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On 2013-05-25 10:11, Alexander Bien wrote:
> On 24.05.2013 16:58, Ken Barber wrote:
>> Its probably the nagios resource, its well known for not scaling when
>> the files contain a lot of entries. That would explain why your
>> PuppetDB is showing low load - as its probably not the bottleneck.
>
> I second that. We have mysql storeconfigs, about 5000 Services, 2,5 MB
> nagios_* files and >50k lines of config. The puppet run takes aprox 45
> minutes. I have heard people on irc say that its not uncommon.
You can cut quite a bit off that time by using templates and files (one 
per service/host) and purging on the base directory.

Regards, David


-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On 25/05/13 17:24, David Schmitt wrote:
> On 2013-05-25 10:11, Alexander Bien wrote:
>> On 24.05.2013 16:58, Ken Barber wrote:
>>> Its probably the nagios resource, its well known for not scaling
when
>>> the files contain a lot of entries. That would explain why your
>>> PuppetDB is showing low load - as its probably not the bottleneck.
>>
>> I second that. We have mysql storeconfigs, about 5000 Services, 2,5 MB
>> nagios_* files and >50k lines of config. The puppet run takes aprox
45
>> minutes. I have heard people on irc say that its not uncommon.
>
> You can cut quite a bit off that time by using templates and files 
> (one per service/host) and purging on the base directory.
>
Hi David,

Sorry to reply to a stale thread. Please could you explain a bit more on 
your solution? Presumably I need to pas some extra arguments to my 
spaceship operator that collects the Nagios resources, but how is the 
directory-level purging done?

Thanks,
Jonathan

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On 2013-06-05 15:53, Jonathan Gazeley wrote:
> On 25/05/13 17:24, David Schmitt wrote:
>> On 2013-05-25 10:11, Alexander Bien wrote:
>>> On 24.05.2013 16:58, Ken Barber wrote:
>>>> Its probably the nagios resource, its well known for not
scaling when
>>>> the files contain a lot of entries. That would explain why your
>>>> PuppetDB is showing low load - as its probably not the
bottleneck.
>>>
>>> I second that. We have mysql storeconfigs, about 5000 Services, 2,5
MB
>>> nagios_* files and >50k lines of config. The puppet run takes
aprox 45
>>> minutes. I have heard people on irc say that its not uncommon.
>>
>> You can cut quite a bit off that time by using templates and files
>> (one per service/host) and purging on the base directory.
>>
>
> Hi David,
>
> Sorry to reply to a stale thread. Please could you explain a bit more on
> your solution? Presumably I need to pas some extra arguments to my
> spaceship operator that collects the Nagios resources, but how is the
> directory-level purging done?
I''m doing it this way:

file {
	"/etc/nagios/conf.d":
		ensure => directory,
		source => "puppet:///modules/nagios/empty",
		ignore => [ ''.gitkeep'' ],
		recurse => true,
		force => true,
		purge => true;
}

File<<| tag == ''nagios::service'' |>>

define nagios::service(...) {
	@@file {
		"/etc/nagios/conf.d/${host}_service_${name}":
			content => template("nagios/service.erb"),
			tag => ''nagios::service'',
			...;
}

This way you collect a minimal number of simple resources. This still is 
dog slow when you have a couple of thousand services, but it sure beats 
the heck out of parsing the complete nagios config for each nagios_* 
resource.



Regards, David

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

If you are looking to implement it via the puppetdbquery module that Ken 
was referring to earlier, here is some code to do so with.  I recently 
implemented this approach with my own setup and it worked out great.  About 
2500 nagios_service resources went from taking 20-30 minutes to about 2. 
 Only issue i ran into was with puppet dashboard when the size of the 
report log became too big to save for one run.  I had to alter the mysql DB 
columns from text to mediumtext and now its all good.

https://gist.github.com/richardc/4714701

On Wednesday, June 5, 2013 3:57:22 PM UTC-4, David Schmitt
wrote:
>
> On 2013-06-05 15:53, Jonathan Gazeley wrote: 
> > On 25/05/13 17:24, David Schmitt wrote: 
> >> On 2013-05-25 10:11, Alexander Bien wrote: 
> >>> On 24.05.2013 16:58, Ken Barber wrote: 
> >>>> Its probably the nagios resource, its well known for not
scaling when
> >>>> the files contain a lot of entries. That would explain why
your
> >>>> PuppetDB is showing low load - as its probably not the
bottleneck.
> >>> 
> >>> I second that. We have mysql storeconfigs, about 5000
Services, 2,5 MB
> >>> nagios_* files and >50k lines of config. The puppet run
takes aprox 45
> >>> minutes. I have heard people on irc say that its not uncommon.
> >> 
> >> You can cut quite a bit off that time by using templates and files
> >> (one per service/host) and purging on the base directory. 
> >> 
> > 
> > Hi David, 
> > 
> > Sorry to reply to a stale thread. Please could you explain a bit more
on
> > your solution? Presumably I need to pas some extra arguments to my 
> > spaceship operator that collects the Nagios resources, but how is the 
> > directory-level purging done? 
>
> I''m doing it this way: 
>
> file { 
>         "/etc/nagios/conf.d": 
>                 ensure => directory, 
>                 source => "puppet:///modules/nagios/empty", 
>                 ignore => [ ''.gitkeep'' ], 
>                 recurse => true, 
>                 force => true, 
>                 purge => true; 
> } 
>
> File<<| tag == ''nagios::service'' |>> 
>
> define nagios::service(...) { 
>         @@file { 
>                 "/etc/nagios/conf.d/${host}_service_${name}": 
>                         content =>
template("nagios/service.erb"),
>                         tag => ''nagios::service'', 
>                         ...; 
> } 
>
> This way you collect a minimal number of simple resources. This still is 
> dog slow when you have a couple of thousand services, but it sure beats 
> the heck out of parsing the complete nagios config for each nagios_* 
> resource. 
>
>
>
> Regards, David 
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/2858c151-1647-4bcd-bd3d-6071451aedf1%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.