Puppet users - Sep 2006 - How to handle config files used by a combination of classes?

Hi,

How should I handle the combinatorics when one config file must be
built for a combination of possible classes?

Take for example the amd automounter''s configuration file, which on
Debian is found at /etc/am-utils/amd.conf.  This is an INI style file
which looks like:

[fsname1]
  param1=foo
  param2=bar
[fsname2]
  param1=baz
  param2=quag

In my case I have one class of machines that should automount
"fsname1", one that should mount "fsname2" and one that
should mount
both.

In CFEngine I have used the file editting commands (eg
"BeginGroupIfNoSuchLine") to add in the appropriate stanza for those
classes of machines that need it.  But in puppet I don''t quite see
what to do.

Any ideas?

Thanks,
-Brett.

On Thu, Sep 07, 2006 at 04:38:05PM -0400, Brett Viren
wrote:
> How should I handle the combinatorics when one config file must be
> built for a combination of possible classes?
> 
> Take for example the amd automounter''s configuration file, which
on
> Debian is found at /etc/am-utils/amd.conf.  This is an INI style file
> which looks like:
> 
> [fsname1]
>   param1=foo
>   param2=bar
> [fsname2]
>   param1=baz
>   param2=quag
> 
> In my case I have one class of machines that should automount
> "fsname1", one that should mount "fsname2" and one that
should mount
> both.
I haven''t yet had to deploy this solution in anger (so it might be
complete
bollocks), but I *have* thought about how I''ll do it when I need to,
and my
answer has been thus:

* Create an overall class (or use the appropriate existing one) that does
the basics (installs the package, etc) but also creates a "special"
directory somewhere (/etc/am-utils/amd.conf.d, fer instance)

* As part of the general class, you create an exec object that looks
something like this:

exec { rebuild_amdconf:
	command => "/usr/local/sbin/rebuild-conf-file /etc/am-utils/amd.conf.d
/etc/am-utils/amd.conf",
	refreshonly => true
}

(The creation and installation of the rebuild-conf-file script is left as an
exercise for the reader <grin>)

* Each class that needs to define something in amd.conf then drops a file
into /etc/am-utils/amd.conf.d and subscribes itself to
"rebuild_amdconf".

Et voila!

(This might turn out to be a common-enough pattern to warrant some sort of
in-Puppet support eventually, but for now it seems quite reasonable just to
fudge it like this).

- Matt

On Thu, 2006-09-07 at 16:38 -0400, Brett Viren wrote:
> How should I handle the combinatorics when one config file must be
> built for a combination of possible classes?
>
> Take for example the amd automounter''s configuration file, which
on
> Debian is found at /etc/am-utils/amd.conf.  This is an INI style file
> which looks like:
> 
> [fsname1]
>   param1=foo
>   param2=bar
> [fsname2]
>   param1=baz
>   param2=quag
> 
> In my case I have one class of machines that should automount
> "fsname1", one that should mount "fsname2" and one that
should mount
> both.
If this is really just about suppressing parts of a file, you could use
puppet''s templating. As an example, save this manifest as
/tmp/templ.pp:

        define amd-mounts($mounts) {
          file { "$name":
             content => template("/tmp/templ.erb")
          }
        }
        
        amd-mounts { "/tmp/templ1.txt": mounts =>
"fsname1" }
        amd-mounts { "/tmp/templ2.txt": mounts =>
"fsname1,fsname2" }

and this as /tmp/templ.erb:
        <% mntarr = mounts.split(",") -%>
        <% if mntarr.include?("fsname1") -%>
        [fsname1]
          param1=foo
          param2=bar
        <% end -%>
        <% if mntarr.include?("fsname2") -%>
        [fsname2]
          param1=baz
          param2=quag
        <% end -%>

When you run ''puppet /tmp/templ.pp'', you''ll get two
files, with
hopefully predictable results. Ideally, of course, there would be a
puppet type to describe automount maps ...

David

Hi Matt and David,

These are both good ideas.  I like Matt''s in that it avoids making a
single file that must know about all classes but I like David''s that
it does the work on the server and doesn''t require an extra script on
the client.  

Choices, choices....  Better to have them then not.  Thanks!

-Brett.

On Fri, Sep 08, 2006 at 08:01:39AM +1000, Matthew Palmer
wrote:
> 
> I haven''t yet had to deploy this solution in anger (so it might be
complete
> bollocks), but I *have* thought about how I''ll do it when I need
to, and my
> answer has been thus:
> 
> * Create an overall class (or use the appropriate existing one) that does
> the basics (installs the package, etc) but also creates a
"special"
> directory somewhere (/etc/am-utils/amd.conf.d, fer instance)
> 
> * As part of the general class, you create an exec object that looks
> something like this:
> 
> exec { rebuild_amdconf:
> 	command => "/usr/local/sbin/rebuild-conf-file
/etc/am-utils/amd.conf.d /etc/am-utils/amd.conf",
> 	refreshonly => true
> }
> 
> (The creation and installation of the rebuild-conf-file script is left as
an
> exercise for the reader <grin>)
> 
Heh, you stole my idea ... Just kidding. I already set up a little
python-script (yes, I know, shame on me) that does this for local files.
Now that I am on vacation (tomorrow is day number 1) I will work on it,
so it will be able to handle local files plus files via SSH ...
> * Each class that needs to define something in amd.conf then drops a file
> into /etc/am-utils/amd.conf.d and subscribes itself to
"rebuild_amdconf".
> 
Yes, this solution is very handy, at least for "simple" config files
(used by most linux programs ...

If anyone is interested in playing with it (though it probably isn''t
even in alpha-state) gimme a word ....

Cheers,

Waldemar la Tendresse

On Sep 7, 2006, at 3:01 PM, Matthew Palmer wrote:
>
> I haven''t yet had to deploy this solution in anger (so it might be
> complete
> bollocks), but I *have* thought about how I''ll do it when I need  
> to, and my
> answer has been thus:
>
> * Create an overall class (or use the appropriate existing one)  
> that does
> the basics (installs the package, etc) but also creates a
"special"
> directory somewhere (/etc/am-utils/amd.conf.d, fer instance)
>
> * As part of the general class, you create an exec object that looks
> something like this:
>
> exec { rebuild_amdconf:
> 	command => "/usr/local/sbin/rebuild-conf-file /etc/am-utils/ 
> amd.conf.d /etc/am-utils/amd.conf",
> 	refreshonly => true
> }
>
> (The creation and installation of the rebuild-conf-file script is  
> left as an
> exercise for the reader <grin>)
>
> * Each class that needs to define something in amd.conf then drops  
> a file
> into /etc/am-utils/amd.conf.d and subscribes itself to  
> "rebuild_amdconf".
This is exactly what I would recommend.  I expect there will be many  
configuration files that we will munge through this kind of process,  
and it''d be great if we could find a simple way to provide support  
for it within Puppet.
> Et voila!
>
> (This might turn out to be a common-enough pattern to warrant some  
> sort of
> in-Puppet support eventually, but for now it seems quite reasonable  
> just to
> fudge it like this).
Yes, I think such support is important.  I spent something like a  
month last year trying to make it easy to build bidirectional  
grammars within the Puppet language itself, so that you specify a  
filetype, the record separator, and then field separator, but I never  
got it to work well.  Now I''m planning on heavily modifying the  
ParsedType.rb class, to create a DSL just for generating and parsing  
files.  However, I just realized we could create a simple
''snippet''
type to get part-way there:

   snippet { "mount1": content => "...", file =>
"/etc/am-utils/
amd.conf" }

Then Puppet could just collect all of the snippets for a given file.   
This messes things up a little bit, though, because then you''ve got a  
three-way tuple determining uniqueness (type, name, and file), which  
Puppet cannot deal with ATM.

I''d love to hear others'' ideas on how Puppet could support
something
like this.

-- 
Luke Kanies
http://madstop.com | http://reductivelabs.com | 615-594-8199

Matthew Palmer <mpalmer@hezmatt.org> writes:
> exec { rebuild_amdconf:
> 	command => "/usr/local/sbin/rebuild-conf-file
/etc/am-utils/amd.conf.d /etc/am-utils/amd.conf",
> 	refreshonly => true
> }
I''ve tried implementing this idea.  My "exercise to the
reader" was to
just use "cat" for the command.  I needed to remove the
"refreshonly",
otherwise this would never trigger.

Below shows what I came up with.  I stole Luke''s use of the word
"snippet"...

Please point out any improvements or blatant stupidity.  One thing I
already see lacking is that once giving a snippet to a client it will
never go away.  Probably best is just to remove the entire .d/
directory in "mergesnippets".


Thanks,
-Brett.


#########
# Say we want to build 
#
#   client:/etc/foo.conf
# 
# from several chunks of files in 
# 
#   server:/etc/puppet/dist/etc/foo.conf.NAME
#
# we do so by first copying the foo.conf.NAME files to
#
#   client:/etc/foo.conf.d/XX.foo.conf.NAME
# 
# where XX is a two digit ordering number and 
# followed up with a simple "cat" command to merge them 

# make sure the temp .d directory exists
define dirsnippet 
{
    file { $name : ensure => directory }
}

# Place a file snippet for a file
define filesnippet(destpath, sourcepath, order,
    mode = 644, owner = root, group = root,
    server = "home.phy.bnl.gov", backup = false) 
{
    file { "$destpath/$order.$name" :
	mode => $mode,
	owner => $owner,
	group => $group,
	backup => $backup,
	source => "puppet://$server/$sourcepath/$name",
	before => exec["build_snippet_file"],
	require => file[$destpath]
    }
}

# merge existing snippets into a single file
define mergesnippets(mode = 644, owner = root, group = root)
{
    exec { build_snippet_file:
	command => "/bin/cat $name.d/[0-9][0-9].* > $name",
    }
    file { $name:
	mode => $mode,
	owner => $owner,
	group => $group
    }
}


##########
# Below is to test.  It assumes fileserver.conf has an "[etc]"
# distribution area.  This builds /etc/foo.conf from two snippets
##########

define foosnippet (order = 50) {
    filesnippet { $name :
	order => $order,
	destpath => "/etc/foo.conf.d",
	sourcepath => "etc",
    }
}

class testclass {

    $conf = "foo.conf"

    dirsnippet { "/etc/$conf.d": }

    foosnippet { "$conf.a" : order => 10 }
    foosnippet { "$conf.b" : order => 20 }

    mergesnippets { "/etc/$conf": }
}
######## end test

On Sep 12, 2006, at 1:52 PM, Brett Viren wrote:
>
> I''ve tried implementing this idea.  My "exercise to the
reader" was to
> just use "cat" for the command.  I needed to remove the
"refreshonly",
> otherwise this would never trigger.
If you used ''notify'' instead of ''before'',
then ''refreshonly'' should
work fine.
> Below shows what I came up with.  I stole Luke''s use of the word
> "snippet"...
Very cool; looks like a good implementation.
> Please point out any improvements or blatant stupidity.  One thing I
> already see lacking is that once giving a snippet to a client it will
> never go away.  Probably best is just to remove the entire .d/
> directory in "mergesnippets".
I''ve just added purging to files in 0.19.0, so you should be able to  
enable recursive purging in that directory:

   file { "/etc/$name": ensure => directory, purge => true,
recurse
=> true }

With this, any files that aren''t being managed by Puppet will get  
removed.

Note that this includes files that are created via ''exec'' -- I
know
that your exec is generating the file outside of the conf directory,  
which is clearly the right choice, but it''s still worth pointing out.

-- 
Luke Kanies
http://madstop.com | http://reductivelabs.com | 615-594-8199

On Tue, Sep 12, 2006 at 02:52:06PM -0400, Brett Viren
wrote:
> Matthew Palmer <mpalmer@hezmatt.org> writes:
> 
> > exec { rebuild_amdconf:
> > 	command => "/usr/local/sbin/rebuild-conf-file
/etc/am-utils/amd.conf.d /etc/am-utils/amd.conf",
> > 	refreshonly => true
> > }
> 
> I''ve tried implementing this idea.  My "exercise to the
reader" was to
> just use "cat" for the command.  I needed to remove the
"refreshonly",
> otherwise this would never trigger.
> 
> Below shows what I came up with.  I stole Luke''s use of the word
> "snippet"...
I used Brett''s mergesnippets idea, but I''m having a problem
triggering a
refresh when the file is created from snippets.
Here''s an example:

define mergesnippets(mode = 644, owner = root, group = root, refreshonly = true)
{
    exec { "build_snippet_file-$name":
        command => "/bin/cat $name.d/* > $name",
        refreshonly => $refreshonly,
    }
    file { $name:
        mode => $mode,
        notify => $notify,
        owner => $owner,
        group => $group
    }
}

service { heartbeat:
    ensure => running,
    restart => "/etc/init.d/heartbeat restart"
}

mergesnippets { "/etc/ha.d/haresources": notify =>
service[heartbeat] }

$ip = "192.168.15.248"
file {
  "/etc/ha.d/haresources.d/$ip":
     content => "lb1   IPaddr2::$ip\n",
     backup => false,
     notify => mergesnippets["/etc/ha.d/haresources"];
}


And the output of a test:

lb1:/tmp# rm /etc/ha.d/haresources.d/192.168.15.248
lb1:/tmp# puppet --debug /tmp/snippet_test.pp
debug: Creating interpreter
debug: getting config
debug: Loaded state in 0.00 seconds
debug: Our client is local
debug: Creating default schedules
debug: Default service type is debian
debug: Defining notify on /etc/ha.d/haresources of type mergesnippets
debug: //mergesnippets[/etc/ha.d/haresources]/file=/etc/ha.d/haresources:
subscribes to heartbeat
debug: //mergesnippets[/etc/ha.d/haresources]: subscribes to heartbeat
debug: //file=/etc/ha.d/haresources.d/192.168.15.248: subscribes to
component(mergesnippets[/etc/ha.d/haresources])
debug: Loaded state in 0.00 seconds
debug: //file=/etc/ha.d/haresources.d/192.168.15.248: File does not exist
debug: //file=/etc/ha.d/haresources.d/192.168.15.248: Changing content
debug: //file=/etc/ha.d/haresources.d/192.168.15.248: 1 change(s)
notice: //file=/etc/ha.d/haresources.d/192.168.15.248/content: synced
info: //file=/etc/ha.d/haresources.d/192.168.15.248: Scheduling refresh of
component[mergesnippets[/etc/ha.d/haresources]]
debug: //mergesnippets[/etc/ha.d/haresources]:
file[/etc/ha.d/haresources.d/192.168.15.248] results in triggering refresh
info: //mergesnippets[/etc/ha.d/haresources]: Triggering
''refresh'' from 1 dependencies
debug:
//mergesnippets[/etc/ha.d/haresources]/exec=build_snippet_file-/etc/ha.d/haresources:
Executing ''/bin/cat /etc/ha.d/haresources.d/* >
/etc/ha.d/haresources''
notice:
//mergesnippets[/etc/ha.d/haresources]/exec=build_snippet_file-/etc/ha.d/haresources:
triggering refresh
debug: //service=heartbeat: Executing ''ps -ef''
debug: //service=heartbeat: PID is 18096
debug: Finishing transaction -743166792 with 1 changes
debug: puppetconfig/puppet/file=/var/puppet/log: Autorequiring file /var/puppet
debug: puppetconfig/puppet/file=/var/puppet/log: subscribes to /var/puppet
debug: puppetconfig/puppet/file=/var/puppet/state: Autorequiring file
/var/puppet
debug: puppetconfig/puppet/file=/var/puppet/state: subscribes to /var/puppet
debug: puppetconfig/puppet/file=/var/puppet/run: Autorequiring file /var/puppet
debug: puppetconfig/puppet/file=/var/puppet/run: subscribes to /var/puppet
debug: puppetconfig/puppet/file=/var/puppet/locks: Autorequiring file
/var/puppet
debug: puppetconfig/puppet/file=/var/puppet/locks: subscribes to /var/puppet
debug: puppetconfig/puppet/file=/var/puppet/state/state.yaml: Autorequiring file
/var/puppet
debug: puppetconfig/puppet/file=/var/puppet/state/state.yaml: Autorequiring file
/var/puppet/state
debug: puppetconfig/puppet/file=/var/puppet/state/state.yaml: subscribes to
/var/puppet
debug: puppetconfig/puppet/file=/var/puppet/state/state.yaml: subscribes to
/var/puppet/state
debug: puppetconfig/puppet/file=/etc/puppet/ssl: Autorequiring file /etc/puppet
debug: puppetconfig/puppet/file=/etc/puppet/ssl: subscribes to /etc/puppet
debug: puppetconfig/puppet/file=/etc/puppet/puppet.conf: Autorequiring file
/etc/puppet
debug: puppetconfig/puppet/file=/etc/puppet/puppet.conf: subscribes to
/etc/puppet
debug: puppetconfig/puppet/file=/etc/puppet/namespaceauth.conf: Autorequiring
file /etc/puppet
debug: puppetconfig/puppet/file=/etc/puppet/namespaceauth.conf: subscribes to
/etc/puppet
debug: puppetconfig/puppet/file=/var/puppet/templates: Autorequiring file
/var/puppet
debug: puppetconfig/puppet/file=/var/puppet/templates: subscribes to /var/puppet
debug: puppetconfig/puppet/file=/var/puppet/plugins: Autorequiring file
/var/puppet
debug: puppetconfig/puppet/file=/var/puppet/plugins: subscribes to /var/puppet
debug: puppetconfig/puppet/file=/var/puppet/state/state.yaml: Changing mode
debug: puppetconfig/puppet/file=/var/puppet/state/state.yaml: 1 change(s)
debug: puppetconfig/puppet/file=/var/puppet/state/state.yaml/mode: mode changed
''640'' to ''660''
debug: puppetconfig/puppet/file=/etc/puppet/puppet.conf: File does not exist
debug: puppetconfig/puppet/file=/etc/puppet/namespaceauth.conf: File does not
exist
debug: puppetconfig/puppet/file=/var/puppet/plugins: File does not exist
debug: Finishing transaction -740376260 with 1 changes
debug: Storing state
debug: Stored state in 0.01 seconds


Any suggestions?

Thanks,
Christian

Christian G. Warden wrote:
> 
> I used Brett''s mergesnippets idea, but I''m having a
problem triggering a
> refresh when the file is created from snippets.
> Here''s an example:
It looks to me like you''re getting a refresh:

[SNIP]
> debug: Defining notify on /etc/ha.d/haresources of type mergesnippets
> debug: //mergesnippets[/etc/ha.d/haresources]/file=/etc/ha.d/haresources:
subscribes to heartbeat
> debug: //mergesnippets[/etc/ha.d/haresources]: subscribes to heartbeat
> debug: //file=/etc/ha.d/haresources.d/192.168.15.248: subscribes to
component(mergesnippets[/etc/ha.d/haresources])
> debug: Loaded state in 0.00 seconds
> debug: //file=/etc/ha.d/haresources.d/192.168.15.248: File does not exist
> debug: //file=/etc/ha.d/haresources.d/192.168.15.248: Changing content
> debug: //file=/etc/ha.d/haresources.d/192.168.15.248: 1 change(s)
> notice: //file=/etc/ha.d/haresources.d/192.168.15.248/content: synced
> info: //file=/etc/ha.d/haresources.d/192.168.15.248: Scheduling refresh of
component[mergesnippets[/etc/ha.d/haresources]]
> debug: //mergesnippets[/etc/ha.d/haresources]:
file[/etc/ha.d/haresources.d/192.168.15.248] results in triggering refresh
> info: //mergesnippets[/etc/ha.d/haresources]: Triggering
''refresh'' from 1 dependencies
> debug:
//mergesnippets[/etc/ha.d/haresources]/exec=build_snippet_file-/etc/ha.d/haresources:
Executing ''/bin/cat /etc/ha.d/haresources.d/* >
/etc/ha.d/haresources''
> notice:
//mergesnippets[/etc/ha.d/haresources]/exec=build_snippet_file-/etc/ha.d/haresources:
triggering refresh
Isn''t that what you want?

-- 
If I want your opinion, I''ll read your entrails.
              --Doug Shewfelt
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com

On Wed, Sep 20, 2006 at 01:11:15PM -0500, Luke Kanies
wrote:
> Christian G. Warden wrote:
> > 
> > I used Brett''s mergesnippets idea, but I''m having a
problem triggering a
> > refresh when the file is created from snippets.
> > Here''s an example:
> 
> It looks to me like you''re getting a refresh:
> 
> [SNIP]
> > debug: Defining notify on /etc/ha.d/haresources of type mergesnippets
> > debug:
//mergesnippets[/etc/ha.d/haresources]/file=/etc/ha.d/haresources: subscribes to
heartbeat
> > debug: //mergesnippets[/etc/ha.d/haresources]: subscribes to heartbeat
> > debug: //file=/etc/ha.d/haresources.d/192.168.15.248: subscribes to
component(mergesnippets[/etc/ha.d/haresources])
> > debug: Loaded state in 0.00 seconds
> > debug: //file=/etc/ha.d/haresources.d/192.168.15.248: File does not
exist
> > debug: //file=/etc/ha.d/haresources.d/192.168.15.248: Changing content
> > debug: //file=/etc/ha.d/haresources.d/192.168.15.248: 1 change(s)
> > notice: //file=/etc/ha.d/haresources.d/192.168.15.248/content: synced
> > info: //file=/etc/ha.d/haresources.d/192.168.15.248: Scheduling
refresh of component[mergesnippets[/etc/ha.d/haresources]]
> > debug: //mergesnippets[/etc/ha.d/haresources]:
file[/etc/ha.d/haresources.d/192.168.15.248] results in triggering refresh
> > info: //mergesnippets[/etc/ha.d/haresources]: Triggering
''refresh'' from 1 dependencies
> > debug:
//mergesnippets[/etc/ha.d/haresources]/exec=build_snippet_file-/etc/ha.d/haresources:
Executing ''/bin/cat /etc/ha.d/haresources.d/* >
/etc/ha.d/haresources''
> > notice:
//mergesnippets[/etc/ha.d/haresources]/exec=build_snippet_file-/etc/ha.d/haresources:
triggering refresh
> 
> Isn''t that what you want?
The heartbeat service isn''t being restarted after the new file is
generated.

Christian

Christian G. Warden wrote:
> 
> The heartbeat service isn''t being restarted after the new file is
> generated.
Hmm, apparently this hasn''t come up before.

The problem is that Puppet doesn''t consider ''refresh''
to be a change, so
if an object gets refreshed then its dependencies don''t also get
refreshed.

I''d guess that this is *probably* a bug; any disagreement from anyone?

-- 
Levy''s Law:
    The truth is always more interesting than your preconception of
    what it might be.
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com

Christian G. Warden wrote:
> 
> The heartbeat service isn''t being restarted after the new file is
> generated.
This now works.  Refreshed objects generate an event, which can trigger 
further refreshes:

$ cat bin/test.pp
file { "/tmp/testing": content => "something" }

exec { first:
     command => "/bin/echo first",
     subscribe => file["/tmp/testing"],
     refreshonly => true
}

exec { second:
     command => "/bin/echo second",
     subscribe => exec[first],
     refreshonly => true
}
$ puppet -v bin/test.pp
notice: //file=/tmp/testing/content: synced
info: //file=/tmp/testing: Scheduling refresh of exec[first]
notice: //exec=/bin/echo first: Triggering ''refresh'' from 1
dependencies
info: //exec=/bin/echo first: Scheduling refresh of exec[second]
notice: //exec=/bin/echo second: Triggering ''refresh'' from 1
dependencies

This''ll be in 0.19.2, which I''m putting out, um, right now.

-- 
I used to get high on life but lately I''ve built up a resistance.
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com

On Fri, Sep 22, 2006 at 12:03:29AM -0500, Luke Kanies
wrote:
> Christian G. Warden wrote:
> > 
> > The heartbeat service isn''t being restarted after the new
file is
> > generated.
> 
> This now works.  Refreshed objects generate an event, which can trigger 
> further refreshes:
Great.  Thanks, Luke.

Christian