Puppet users - Dec 2012 - inspect resources that are already added to a manifest

Hi all,

Does anyone know of a way to inspect resources that are already parsed in a 
node''s manifest during catalog compilation? This would certainly need
some
serious Ruby Fu.

As an example, imagine I have a number of arbitrary files defined by 
multiple classes and it is impossible to get an all encompassing list of 
these files:

file { ''woof'': }
file { ''cows'': }
file { ''meow'': }
...
$all_files = inline_template(...)

I would like to be able to gather those file names into a Puppet variable - 
this would be parse order dependent. It would be fantastic if it could 
handle exported resources that have just been collected as well.

Thanks,

-Luke

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/SIz5R_2fmNwJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Saturday, December 22, 2012 12:20:10 PM UTC-6, Luke Bigum
wrote:
>
> Hi all,
>
> Does anyone know of a way to inspect resources that are already parsed in 
> a node''s manifest during catalog compilation? This would certainly
need
> some serious Ruby Fu.
>

This is a bad idea.  If your the Puppet circuits in your brain didn''t
trip
over "inspect", they certainly should have sounded the alarm over
"serious
Ruby Fu".  You are fighting against the tool.

 
>
> As an example, imagine I have a number of arbitrary files defined by 
> multiple classes and it is impossible to get an all encompassing list of 
> these files:
>
> file { ''woof'': }
> file { ''cows'': }
> file { ''meow'': }
> ...
> $all_files = inline_template(...)
>
> I would like to be able to gather those file names into a Puppet variable 
> - this would be parse order dependent. It would be fantastic if it could 
> handle exported resources that have just been collected as well.
>

And "parse-order dependent"?  Of course it is.  You need a
Puppet-bogometer.

So what configuration objective are you actually trying to accomplish 
here?  There is likely a more robust, less Rubyriffic way to accomplish it.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/IP4c-a9fVJcJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi John!

On Wed, Jan 02, 2013 at 07:51:37AM -0800, jcbollinger
wrote:
> You need a Puppet-bogometer.
I need one, too! Where can I order it?

Cheers,
Matthias Viehweger
-- 
Serververwaltung und Softwareentwicklung
 
https://www.heute-kaufen.de
Prinzessinnenstraße 20 - 10969 Berlin

On Wednesday, January 2, 2013 3:51:37 PM UTC, jcbollinger wrote:
>
>
> On Saturday, December 22, 2012 12:20:10 PM UTC-6, Luke Bigum wrote:
>>
>> Hi all,
>>
>> Does anyone know of a way to inspect resources that are already parsed
in
>> a node''s manifest during catalog compilation? This would
certainly need
>> some serious Ruby Fu.
>>
>
>
> This is a bad idea.  If your the Puppet circuits in your brain
didn''t trip
> over "inspect", they certainly should have sounded the alarm over
"serious
> Ruby Fu".  You are fighting against the tool.
>
 
>
>>
>> As an example, imagine I have a number of arbitrary files defined by 
>> multiple classes and it is impossible to get an all encompassing list
of
>> these files:
>>
>> file { ''woof'': }
>> file { ''cows'': }
>> file { ''meow'': }
>> ...
>> $all_files = inline_template(...)
>>
>> I would like to be able to gather those file names into a Puppet
variable
>> - this would be parse order dependent. It would be fantastic if it
could
>> handle exported resources that have just been collected as well.
>>
>
>
> And "parse-order dependent"?  Of course it is.  You need a 
> Puppet-bogometer.
>
> So what configuration objective are you actually trying to accomplish 
> here?  There is likely a more robust, less Rubyriffic way to accomplish it.
>
>
Ohh don''t worry, John, my bogometer was going off like crazy, the
needle
almost broke ;-)

I''m taking shortcuts in my spare time with a tool that''s
probably 70% right
for the job. It''s for monitoring - I really like the idea of a Puppet 
module to describe or advertise how to monitor itself, it keeps them very 
self contained.

Just a bit more on this - I generally see three categories of monitoring 
tools. Ones that are configured separately from your CRM and end up being a 
source of truth on their own are in my mind the worst. The next level up 
are ones either defined from or derived from your CRM. The best are 
auto-discovery, but they cost an absolute fortune. I''m trying to move
my
team from the first one to the second one with as little "new tools"
as
possible, which is where the "70% right for the job" comment comes
from.

I''m using exported resources to describe how modules are monitored. The
problem is that exported resources are not the equivalent of raw 
information passing. So when I want to start doing trickier things like 
group and analyse what is collected, exported resources don''t cut it 
because it''s not what they are designed for.

Specifically what I was trying to do was collect exported resources of the 
same type and group them on the monitoring server. There is no predefined 
list of service names anywhere (unless you parse the node definitions) so 
that''s why I wanted to go from resource collection to Array of Strings.
A
colleague has managed to reduce my 300 lines to 50 though so the need for 
craziness is reduced somewhat. We still need to do the "Export a File"
trick and run a script on the monitoring server to build the complex 
configuration that exported resources are not designed to handle.

The next iteration of this work might be to scrap resource collection in 
favour of querying PuppetDB directly to figure out what to monitor, but 
that''s a lot more work than I''m prepared to do at this stage.
Maybe in a
few months... ;-)

-Luke

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/OLpl0Bx1q5kJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Thursday, January 3, 2013 4:08:54 AM UTC-6, Luke Bigum
wrote:
>
>
> I''m taking shortcuts in my spare time with a tool that''s
probably 70%
> right for the job. It''s for monitoring - I really like the idea of
a Puppet
> module to describe or advertise how to monitor itself, it keeps them very 
> self contained.
>
> Just a bit more on this - I generally see three categories of monitoring 
> tools. Ones that are configured separately from your CRM and end up being a
> source of truth on their own are in my mind the worst. The next level up 
> are ones either defined from or derived from your CRM. The best are 
> auto-discovery, but they cost an absolute fortune. I''m trying to
move my
> team from the first one to the second one with as little "new
tools" as
> possible, which is where the "70% right for the job" comment
comes from.
>
> I''m using exported resources to describe how modules are
monitored. The
> problem is that exported resources are not the equivalent of raw 
> information passing. So when I want to start doing trickier things like 
> group and analyse what is collected, exported resources don''t cut
it
> because it''s not what they are designed for.
>
> Specifically what I was trying to do was collect exported resources of the 
> same type and group them on the monitoring server. There is no predefined 
> list of service names anywhere (unless you parse the node definitions) so 
> that''s why I wanted to go from resource collection to Array of
Strings. A
> colleague has managed to reduce my 300 lines to 50 though so the need for 
> craziness is reduced somewhat. We still need to do the "Export a
File"
> trick and run a script on the monitoring server to build the complex 
> configuration that exported resources are not designed to handle.
>
> The next iteration of this work might be to scrap resource collection in 
> favour of querying PuppetDB directly to figure out what to monitor, but 
> that''s a lot more work than I''m prepared to do at this
stage. Maybe in a
> few months... ;-)
>
>
Querying PuppetDB in fact sounds like a good approach.  One alternative I 
see would be to extend the main Puppet engine.  Possibly that could take 
the form of a new Puppet face (which, now that I think of it, might just 
end up a front end to the DB query).

You could also inspect the master''s catalog cache instead of hooking 
directly into compilation.  That would give you all the information you 
want, without parse-order issues.  You could monitor the cache directory 
for changes to make the system nearly as immediate as what you you''re
doing
now.

Or if you have a list of known systems to monitor, then you can generate 
catalogs for them at any time via the "puppet catalog" face.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/BbIJAdmSA7gJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 3 January 2013 20:08, Luke Bigum <Luke.Bigum@lmax.com> wrote:
> On Wednesday, January 2, 2013 3:51:37 PM UTC, jcbollinger wrote:
>
>>
>>
>> On Saturday, December 22, 2012 12:20:10 PM UTC-6, Luke Bigum wrote:
>>>
>>> Hi all,
>>>
>>> Does anyone know of a way to inspect resources that are already
parsed
>>> in a node''s manifest during catalog compilation? This
would certainly need
>>> some serious Ruby Fu.
>>>
>>
>>
>> This is a bad idea.  If your the Puppet circuits in your brain
didn''t
>> trip over "inspect", they certainly should have sounded the
alarm over
>> "serious Ruby Fu".  You are fighting against the tool.
>>
>
>>
>>>
>>> As an example, imagine I have a number of arbitrary files defined
by
>>> multiple classes and it is impossible to get an all encompassing
list of
>>> these files:
>>>
>>> file { ''woof'': }
>>> file { ''cows'': }
>>> file { ''meow'': }
>>> ...
>>> $all_files = inline_template(...)
>>>
>>> I would like to be able to gather those file names into a Puppet
>>> variable - this would be parse order dependent. It would be
fantastic if it
>>> could handle exported resources that have just been collected as
well.
>>>
>>
>>
>> And "parse-order dependent"?  Of course it is.  You need a
>> Puppet-bogometer.
>>
>> So what configuration objective are you actually trying to accomplish
>> here?  There is likely a more robust, less Rubyriffic way to accomplish
it.
>>
>>
> Ohh don''t worry, John, my bogometer was going off like crazy, the
needle
> almost broke ;-)
>
> I''m taking shortcuts in my spare time with a tool that''s
probably 70%
> right for the job. It''s for monitoring - I really like the idea of
a Puppet
> module to describe or advertise how to monitor itself, it keeps them very
> self contained.
>
> Just a bit more on this - I generally see three categories of monitoring
> tools. Ones that are configured separately from your CRM and end up being a
> source of truth on their own are in my mind the worst. The next level up
> are ones either defined from or derived from your CRM. The best are
> auto-discovery, but they cost an absolute fortune. I''m trying to
move my
> team from the first one to the second one with as little "new
tools" as
> possible, which is where the "70% right for the job" comment
comes from.
>
> I''m using exported resources to describe how modules are
monitored. The
> problem is that exported resources are not the equivalent of raw
> information passing. So when I want to start doing trickier things like
> group and analyse what is collected, exported resources don''t cut
it
> because it''s not what they are designed for.
>
> Specifically what I was trying to do was collect exported resources of the
> same type and group them on the monitoring server. There is no predefined
> list of service names anywhere (unless you parse the node definitions) so
> that''s why I wanted to go from resource collection to Array of
Strings. A
> colleague has managed to reduce my 300 lines to 50 though so the need for
> craziness is reduced somewhat. We still need to do the "Export a
File"
> trick and run a script on the monitoring server to build the complex
> configuration that exported resources are not designed to handle.
>

You have made me curious.

I am going to make some wild assumptions and probably leap to some
incorrect conclusions.

Querying PupptDB seems like the wrong way to achieve this in my opinion.

I put my monitoring (and firewall rules incidentally) into each class on a
node and import into my monitoring server (nagios currently but intend on
switching to icinga soon) them based on tags.
I was using global vars but are currently rewriting my modules to use heira
to set the monitoring server and a few other settings.
This is working nicely for me.
I had a few different ideas along the way but my current iteration is
getting close to awesome.
It also gives me fine grained control over whether a node gets sms alerts
or escalation and such.

Is this the kind of thing you are attempting to achieve?

I can provide extra complexity to how it works if you need it.

> The next iteration of this work might be to scrap resource collection in
> favour of querying PuppetDB directly to figure out what to monitor, but
> that''s a lot more work than I''m prepared to do at this
stage. Maybe in a
> few months... ;-)
>
> -Luke
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/puppet-users/-/OLpl0Bx1q5kJ.
>
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Sunday, January 6, 2013 9:23:17 PM UTC-6, Pete wrote:
>
>
> I put my monitoring (and firewall rules incidentally) into each class on a 
> node and import into my monitoring server (nagios currently but intend on 
> switching to icinga soon) them based on tags.
> I was using global vars but are currently rewriting my modules to use 
> heira to set the monitoring server and a few other settings.
> This is working nicely for me.
> I had a few different ideas along the way but my current iteration is 
> getting close to awesome.
> It also gives me fine grained control over whether a node gets sms alerts 
> or escalation and such.
>
>
Something along those lines would be my preferred way to do it if I were 
willing and able to modify the classes involved.  My inference from
Luke''s
comments was that he wanted to avoid doings so, and perhaps that he was 
specifically looking for a solution that did not rely on the cooperation of 
the modules involved.  That would be an eminently reasonable objective 
where third-party modules are in the picture, but not an unreasonable one 
even where all modules were built in-house.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/L5WxTz43T8sJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 8 January 2013 00:19, jcbollinger <John.Bollinger@stjude.org> wrote:
>
>
> On Sunday, January 6, 2013 9:23:17 PM UTC-6, Pete wrote:
>>
>>
>> I put my monitoring (and firewall rules incidentally) into each class
on
>> a node and import into my monitoring server (nagios currently but
intend on
>> switching to icinga soon) them based on tags.
>> I was using global vars but are currently rewriting my modules to use
>> heira to set the monitoring server and a few other settings.
>> This is working nicely for me.
>> I had a few different ideas along the way but my current iteration is
>> getting close to awesome.
>> It also gives me fine grained control over whether a node gets sms
alerts
>> or escalation and such.
>>
>>
> Something along those lines would be my preferred way to do it if I were
> willing and able to modify the classes involved.  My inference from
Luke''s
> comments was that he wanted to avoid doings so, and perhaps that he was
> specifically looking for a solution that did not rely on the cooperation of
> the modules involved.  That would be an eminently reasonable objective
> where third-party modules are in the picture, but not an unreasonable one
> even where all modules were built in-house.
>
I wrote most of the modules in my setup myself and that makes it easier.

I am attempting to be tricky with my monitoring modules and make them
extensible so adding extra monitoring classes should be pretty simple.
I am also intending on releasing them when I am happy with them.

>
>
> John
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/puppet-users/-/L5WxTz43T8sJ.
>
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.