Puppet users - Jan 2011 - Best Practices/Style: add stuff to a file for each host?

(I''m going a bit more for philosophical discussion than practicality
here, maybe.  Do at least feel free to think in terms of what Puppet
*should* do rather than tha fastest way to solve this problem.)

There''s a pattern I''ve run into a lot recently mhere a config
file
needs to be built based on information from a number of puppet
managed hosts.  Assume here than I mean "node" in the puppet sense
when I say "host".  *shrug*

Use cases:

  - backups, where each host has directories that need to be backed
    up, and things need to be done on the individual backup client
    hosts to handle that, and *also* stuff needs to be done on the
    backup master

  - deploy configuration, where each host has a deploy role and it
    makes most sense to talk about the deploy role in each host''s
    puppet config, but the file that manages the deployment is on
    the deploy master host

  - VM configuration, where information about a VM needs to affect
    data/configuration stored on the host that holds that VM

All the same general pattern of action-at-a-distance: configuration
on a number of hosts affecting the master config file on a single
host.

Also, multi-part config files: in at least some of these cases, the
information from each host generates some configuration file output,
perhaps from a template, but all of those bits of config file need
to be merged together into one master config file.

These don''t seem, to me, to be things puppet handles natively, and
(see my other posts) new types don''t seem to be being added all that
often.

So, my question is this: what''s the right way to handle this sort of
situation?

Feel free to break the action-at-a-distance bit and the
multipart-config-file-merge bit into seperate pieces, if you like.

Thanks so much for your time!

-Robin

-- 
http://singinst.org/ :  Our last, best hope for a fantastic future.
Lojban (http://www.lojban.org/): The language in which "this parrot
is dead" is "ti poi spitaki cu morsi", but "this sentence is
false"
is "na nei".   My personal page: http://www.digitalkingdom.org/rlp/

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi

On Tuesday, January 11, 2011 5:51:07 AM UTC+1, Robin Lee Powell
wrote:
>
> (I''m going a bit more for philosophical discussion than
practicality
> here, maybe.  Do at least feel free to think in terms of what Puppet
> *should* do rather than tha fastest way to solve this problem.)
>
> There''s a pattern I''ve run into a lot recently mhere a
config file
> needs to be built based on information from a number of puppet
> managed hosts.  Assume here than I mean "node" in the puppet
sense
> when I say "host".  *shrug*
>
> Use cases:
>
>   - backups, where each host has directories that need to be backed
>     up, and things need to be done on the individual backup client
>     hosts to handle that, and *also* stuff needs to be done on the
>     backup master
>
>   - deploy configuration, where each host has a deploy role and it
>     makes most sense to talk about the deploy role in each host''s
>     puppet config, but the file that manages the deployment is on
>     the deploy master host
>
>   - VM configuration, where information about a VM needs to affect
>     data/configuration stored on the host that holds that VM
>
> All the same general pattern of action-at-a-distance: configuration
> on a number of hosts affecting the master config file on a single
> host.
>
This is exactly the case when you need exported resources and, therefore, 
storeconfigs.
http://projects.puppetlabs.com/projects/1/wiki/Exported_Resources

Also, multi-part config files: in at least some of these cases,
the
> information from each host generates some configuration file output,
> perhaps from a template, but all of those bits of config file need
> to be merged together into one master config file.
>
You can build a file based on different "fragments" at least in 2
ways:
- When you specify an array of templates , when using the content => 
argument, these templates are actually appended in the defined order. 
- With the puppet-concat module by Rip 
https://github.com/ripienaar/puppet-concat you can build up files based on 
different fragments (and you can define single fragmensts as exported 
resources so that they fit the need you expressed). Incidentally, I''ve 
started to use it just yesterday, to build named.conf in a bind module, and 
does exactly what it says.

Hope it helps
Af

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Al @ Lab42:
> You can build a file based on different "fragments" at least in 2
ways:
> - When you specify an array of templates , when using the content =>
> argument, these templates are actually appended in the defined order. 
> - With the puppet-concat module by Rip
> https://github.com/ripienaar/puppet-concat you can build up files
> based on different fragments (and you can define single fragments as
> exported resources so that they fit the need you expressed). 
I recently adopted puppet-concat, and found it my new big hammer for
almost everything.  I had previously set up a define for a simple
header/body/footer structure where only those who cared would supply a
custom body template name as an argument.  When it came time to add more
smarts, it was trivial to swap out the file resource in the define and
insert both a concat and a concat::fragment.  

I now still have the old three-template API as well as a "provide any
fragments you like, but know that the header/body/footer have order
01/50/99" mechanism available.

One could also use Augeas, but I do not feel that it would fit the
original poster''s desire to find a strong and helpful standard.
Augeas doesn''t do well when you need to assert the state of a file
overall (which is at the heart of puppet''s declarative nature) but with
responsibility divided among systems.  It does much better when you need
to surgically assert one or two lines or stanzas in an otherwise
unmanaged file.

I''ll just say that Augeas is named after a man who let his stables fill
up with so much crap that Hercules had to divert two whole rivers to
pressure-hose it clean again.  You can''t make this stuff up.

-- 
"These people program the way Victorians dress.
It takes two hours and three assistants to put on
your clothes, and you have to change before dinner.
But everything is modular."    -- Miles Nordin, on PAM

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Tue, Jan 11, 2011 at 05:20:38AM -0800, Al @ Lab42
wrote:
>
> You can build a file based on different "fragments" at least in 2
> ways:
>
> - When you specify an array of templates , when using the content
> => argument, these templates are actually appended in the defined
> order. 
Can you use exported resources to generate such an array?

-Robin

-- 
http://singinst.org/ :  Our last, best hope for a fantastic future.
Lojban (http://www.lojban.org/): The language in which "this parrot
is dead" is "ti poi spitaki cu morsi", but "this sentence is
false"
is "na nei".   My personal page: http://www.digitalkingdom.org/rlp/

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Tue, Jan 11, 2011 at 12:07:30PM -0800, Robin Lee Powell
wrote:
> On Tue, Jan 11, 2011 at 05:20:38AM -0800, Al @ Lab42 wrote:
> >
> > You can build a file based on different "fragments" at least
in
> > 2 ways:
> >
> > - When you specify an array of templates , when using the
> > content => argument, these templates are actually appended in
> > the defined order. 
> 
> Can you use exported resources to generate such an array?
Anybody?  I can see how to generate resources from lots of hosts and
run it on one host, but I can''t see how to generate a config file
out of all of that information.

-Robin

-- 
http://singinst.org/ :  Our last, best hope for a fantastic future.
Lojban (http://www.lojban.org/): The language in which "this parrot
is dead" is "ti poi spitaki cu morsi", but "this sentence is
false"
is "na nei".   My personal page: http://www.digitalkingdom.org/rlp/

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 01/12/2011 05:08 PM, Robin Lee Powell wrote:
> On Tue, Jan 11, 2011 at 12:07:30PM -0800, Robin Lee Powell wrote:
>> On Tue, Jan 11, 2011 at 05:20:38AM -0800, Al @ Lab42 wrote:
>>>
>>> You can build a file based on different "fragments" at
least in
>>> 2 ways:
>>>
>>> - When you specify an array of templates , when using the
>>> content => argument, these templates are actually appended in
>>> the defined order. 
>>
>> Can you use exported resources to generate such an array?
> 
> Anybody?  I can see how to generate resources from lots of hosts and
> run it on one host, but I can''t see how to generate a config file
> out of all of that information.
If you use a concat mechanism as described in this thread, you can have
each of your machines export a respective snippet and the target machine
realizes those and concatenates them to your config file.

HTH,
Felix

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Wednesday, January 12, 2011 5:08:07 PM UTC+1, Robin Lee Powell
wrote:
>
> On Tue, Jan 11, 2011 at 12:07:30PM -0800, Robin Lee Powell wrote:
> > On Tue, Jan 11, 2011 at 05:20:38AM -0800, Al @ Lab42 wrote:
> > >
> > > You can build a file based on different "fragments" at
least in
> > > 2 ways:
> > >
> > > - When you specify an array of templates , when using the
> > > content => argument, these templates are actually appended in
> > > the defined order. 
> > 
> > Can you use exported resources to generate such an array?
>
> Anybody?  I can see how to generate resources from lots of hosts and
> run it on one host, but I can''t see how to generate a config file
> out of all of that information.
>
> -Robin
>
I give you an example I use for Nagios.

On each node I''ve something like:
nagios::host { $ fqdn }

This calls the custom define (ignore the various frills such the grouptag 
selector that manages different Nagios servers according to custom logic or 
the various nagios::params variables, that are just module''s internal 
variables defined in the separated nagios::params class and concentrate on 
the fact this this defines just exports a file resource: @@file ):
define nagios::host (
    $ip = $fqdn,
    $short_alias = $fqdn,
    $use = ''generic-host'',
    $nagios_parent = '''',
    $ensure = ''present'',
    $hostgroups = ''all'' ) {

    require nagios::params

    @@file { "${nagios::params::customconfigdir}/hosts/${name}.cfg":
        mode    => "${nagios::params::configfile_mode}",
        owner   => "${nagios::params::configfile_owner}",
        group   => "${nagios::params::configfile_group}",
        ensure  => "${ensure}",
        require => Class["nagios::extra"],
        notify  => Service["nagios"],
        content => template( "nagios/host.erb" ),
        tag     => "${nagios::params::grouptag}" ? {
            ''''       => "nagios_host",
            default  => "nagios_host_$nagios::params::grouptag",
        },
    }

}


On the Nagios server I ''ve (in order to collect all the exported
reources
with the relevant tag)
    case $nagios::params::grouptag {
        "": {
        File <<| tag == "nagios_host" |>>
        File <<| tag == "nagios_service" |>>
#       File <<| tag == "nagios_hostgroup" |>>
        }
        default: {
        File <<| tag == "nagios_host_$nagios::params::grouptag"
|>>
        File <<| tag ==
"nagios_service_$nagios::params::grouptag" |>>
#       File <<| tag ==
"nagios_hostgroup_$nagios::params::grouptag" |>>
        }
    }

Now, I could place (and maybe I will) , in the nagios::host define something 
like:

    @@concat::fragment { "nagios_hosts_${name}.cfg":
        target => 
"${nagios::params::customconfigdir}/hosts/nagios_hosts.cfg",
        ensure  => "${ensure}",
        notify  => Service["nagios"],
        content => template( "nagios/host.erb" ),
        tag     => "${nagios::params::grouptag}" ? {
            ''''       => "nagios_host",
            default  => "nagios_host_$nagios::params::grouptag",
        },

and in the nagios server class collect these fragments with:
    case $nagios::params::grouptag {
        "": {
        Concat::fragment <<| tag == "nagios_host" |>>
        Concat::fragment <<| tag == "nagios_service" |>>
#       Concat::fragment <<| tag == "nagios_hostgroup" |>>
        }
        default: {
        Concat::fragment <<| tag ==
"nagios_host_$nagios::params::grouptag"
|>>
        Concat::fragment <<| tag == 
"nagios_service_$nagios::params::grouptag" |>>
#       Concat::fragment <<| tag == 
"nagios_hostgroup_$nagios::params::grouptag" |>>
        }
    }

In the same server I should also define the base concat file with something 
like:
concat { "${nagios::params::customconfigdir}/hosts/nagios_hosts.cfg":
        mode => 644,
        owner => root,
        group => root,
}


Consider that I''ve not tested this , and there could be syntax or other
errors around .
Also I''m not so sure that is safe to have the double colons in the
define
name when collecting it ( Concat::fragment ) but the principle is that.

Remember also that all the variable or facts you use when exporting a 
resource (such as $fqdn ) are referred and valued on host that exports it 
and not to the one that collects them.

Hope it helped without adding confusion

Al

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.