Puppet users - Aug 2007 - Re: [puppet] #779: access to inherited variables from within a template

puppet schrieb:
> #779: access to inherited variables from within a template
>
-----------------------------------------+----------------------------------
>      Reporter:  simu                     |           Owner:  luke      
>          Type:  enhancement              |          Status:  new       
>      Priority:  normal                   |       Milestone:            
>     Component:  language                 |         Version:  0.23.1    
>      Severity:  normal                   |      Resolution:            
>      Keywords:  templating, inheritance  |           Stage:  Unreviewed
>      Approval:  Unnecessary              |           Patch:  None      
>    Complexity:  Unknown                  |   Compatibility:  Unknown   
> Specification:  Unnecessary              |  
>
-----------------------------------------+----------------------------------
> Comment (by mpalmer):
>
>  This is where "Puppet isn't Ruby" bites hard.  I have no
idea what a
>  fully-qualified Puppet variable might look like in ERb, but I suspect it
>  wouldn't be pretty.
>
>  I think you've got a bigger problem than ERb syntax, though -- the
idea of
>  "one host [configuring] another one's config files" just
scares the
>  willies out of me, and I suggest you rethink your manifest design
>  somewhat.
>
>   
Well - this just is reality - i have interacting middleware-components
that have a bit of each others configuration in their configfiles.
They are not neccessarily on different hosts but might be and the
described problem is not really related to inter-host communication but
to inter-class communication in general.

And btw: This "one host [configuring] another one's config files"
is one
of the features I´m looking forward to the most as we have a hell of a
lot of machines with even more services that all interact in a way, so I
*want* to be able to just change that one service in which the change
occurs and not have to look at all "may be somehow related" classes to
check if I didn´t forget to change them as well to reflect that change.
When you look at the "handling of dynamic filecontent creation" mail I
wrote this night there is an example of how I have to build our
services, the reason for my wanting that feature might becom clearer to you.

I really put a lot of thought in that whole thing by now and this is the
best solution i came up with. And I still cant believe that I am the
only one wanting such a thing - I am quite flabbergasted by that..
_______________________________________________
Puppet-users mailing list
Puppet-users@madstop.com
https://mail.madstop.com/mailman/listinfo/puppet-users

On Wed, Aug 22, 2007 at 11:13:00AM +0200, Simon Mügge
wrote:
> Well - this just is reality - i have interacting middleware-components
> that have a bit of each others configuration in their configfiles.
Custom types that manage these configuration files are far better than
trying to pull everything together into a monster template.  I tried that
with Nagios configs for a couple of days.  Madness.
> They are not neccessarily on different hosts but might be and the
> described problem is not really related to inter-host communication but
> to inter-class communication in general.
Exported/virtual resources are what you want for inter-host/class
communication -- you define a resource to be usable elsewhere, and the
destination host applies it to it''s configuration.  It''s
really quite neat.
At the moment exported resources are a bit limp, but there''s no reason
why
they can''t be a hell of a lot more powerful.
> I really put a lot of thought in that whole thing by now and this is the
> best solution i came up with. And I still cant believe that I am the
> only one wanting such a thing - I am quite flabbergasted by that..
"I can''t believe nobody else wants to do this" is often
translatable to
"everyone else has a different way of doing this" -- and, at the risk
of
falling foul of a "fallacy of popularity", if everyone else is doing
it a
different way, it''s probably worth at least considering the popular
wisdom
before possibly digging yourself a deep hole.

In my experience, other code smells include:

* Use of exec;

* Use of templates;

* A need for global (or cross-scope) variables.

You''re shooting at least 3 for 4 so far.  <grin>

(If you''re not familiar with the concept of "code smells",
try
http://c2.com/xp/CodeSmell.html for some basic background)

Puppet isn''t "there" yet in the sense of you necessarily
being able to avoid
these smells ($DEITY knows my manifests reek, particularly of exec calls
everywhere), but hacks around limitations in Puppet are certainly nothing to
celebrate.

- Matt

Matthew Palmer schrieb:
> On Wed, Aug 22, 2007 at 11:13:00AM +0200, Simon Mügge wrote:
>   
>> Well - this just is reality - i have interacting middleware-components
>> that have a bit of each others configuration in their configfiles.
>>     
>
> Custom types that manage these configuration files are far better than
> trying to pull everything together into a monster template.  I tried that
> with Nagios configs for a couple of days.  Madness.
>   
I´ll take a look at that, thanks for the hint.
>   
>> They are not neccessarily on different hosts but might be and the
>> described problem is not really related to inter-host communication but
>> to inter-class communication in general.
>>     
>
> Exported/virtual resources are what you want for inter-host/class
> communication -- you define a resource to be usable elsewhere, and the
> destination host applies it to it''s configuration.  It''s
really quite neat.
> At the moment exported resources are a bit limp, but there''s no
reason why
> they can''t be a hell of a lot more powerful.
>   
Yeah I got that, and just to clear things up a bit - I´m not saying I
don´t (want) to use exported/virtual ressources, I just think .. oh
wait, than can be done whe I use a custom type.. I´ll think about
that.
>   
>> I really put a lot of thought in that whole thing by now and this is
the
>> best solution i came up with. And I still cant believe that I am the
>> only one wanting such a thing - I am quite flabbergasted by that..
>>     
>
> "I can''t believe nobody else wants to do this" is often
translatable to
> "everyone else has a different way of doing this" -- and, at the
risk of
> falling foul of a "fallacy of popularity", if everyone else is
doing it a
> different way, it''s probably worth at least considering the
popular wisdom
> before possibly digging yourself a deep hole.
>   
The thing just is: I don´t *know* how everyone else is doing this as
cases like these are not documented anywhere, so calling that "popular
wisdom" is just plain wrong.. I  did by no means say that  I want to
stick to my "solution", no matter what everyone else does - once again
there is just no "best practices" or anything for this sort of thing
so
the only chance ive got is to make up my own mind and then ask everyones
opinion - which is what i did here - or in least intended to
do.
> In my experience, other code smells include:
>
> * Use of exec;
>
> * Use of templates;
>
> * A need for global (or cross-scope) variables.
>
> You''re shooting at least 3 for 4 so far.  <grin>
>
> (If you''re not familiar with the concept of "code
smells", try
> http://c2.com/xp/CodeSmell.html for some basic background)
>
> Puppet isn''t "there" yet in the sense of you necessarily
being able to avoid
> these smells ($DEITY knows my manifests reek, particularly of exec calls
> everywhere), but hacks around limitations in Puppet are certainly nothing
to
> celebrate.
> - Matt
>   
Thanks a bunch for the input - I´ll report back!

Simon
> _______________________________________________
> Puppet-users mailing list
> Puppet-users@madstop.com
> https://mail.madstop.com/mailman/listinfo/puppet-users
>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wednesday 22 August 2007, Matthew Palmer wrote:
> "I can''t believe nobody else wants to do this" is often
translatable to
> "everyone else has a different way of doing this" -- and, at the
risk of
> falling foul of a "fallacy of popularity", if everyone else is
doing it a
> different way, it''s probably worth at least considering the
popular wisdom
> before possibly digging yourself a deep hole.
>
> In my experience, other code smells include:
>
> * Use of exec;
>
> * Use of templates;
>
> * A need for global (or cross-scope) variables.
>
> You''re shooting at least 3 for 4 so far.  <grin>
>
> (If you''re not familiar with the concept of "code
smells", try
> http://c2.com/xp/CodeSmell.html for some basic background)
>
> Puppet isn''t "there" yet in the sense of you necessarily
being able to
> avoid these smells ($DEITY knows my manifests reek, particularly of exec
> calls everywhere), but hacks around limitations in Puppet are certainly
> nothing to celebrate.
You''re so right:

| david@zion:~/Work/puppet/manifests$ git grep exec | wc -l
| 78
| david@zion:~/Work/puppet/manifests$ 


I contest your general denounciation of templates though, could you elaborate 
please?

| david@zion:~/Work/puppet/manifests$ git grep template | wc -l
| 48
| david@zion:~/Work/puppet/manifests$ 


*sigh*

Regards, David
- -- 
The primary freedom of open source is not the freedom from cost, but the free-
dom to shape software to do what you want. This freedom is /never/ exercised
without cost, but is available /at all/ only by accepting the very different
costs associated with open source, costs not in money, but in time and effort.
- -- http://www.schierer.org/~luke/log/20070710-1129/on-forks-and-forking
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFG0eQf/Pp1N6Uzh0URApIOAKCeH6nORENbkAinnIDjcmW1doQ3RgCfcPB4
6zhv0L6hcWxeLFlEd3t/1ho=v5Fw
-----END PGP SIGNATURE-----

On Sun, Aug 26, 2007 at 10:35:39PM +0200, David Schmitt
wrote:
> On Wednesday 22 August 2007, Matthew Palmer wrote:
> > "I can''t believe nobody else wants to do this" is
often translatable to
> > "everyone else has a different way of doing this" -- and, at
the risk of
> > falling foul of a "fallacy of popularity", if everyone else
is doing it a
> > different way, it''s probably worth at least considering the
popular wisdom
> > before possibly digging yourself a deep hole.
> >
> > In my experience, other code smells include:
> >
> > * Use of exec;
> >
> > * Use of templates;
>
> I contest your general denounciation of templates though, could you
elaborate
> please?
Templates quickly become ridiculously complex to manage, as you extend the
parameters that the template is supposed to manage.  Suddenly your defines
start having to take massive numbers of parameters and your templates become
more and more filled with ERb gunk.

Take, for instance, the puppet.conf file.  To begin with, you have a static
one that you ship with everything:

file { ''/etc/puppet/puppet.conf'':
	source => "puppet://puppetfiles/config/classes/any/puppet.conf"
}

But then someone decides they want to have different values for the ssldir:

define puppet_conf($ssldir = ''/var/lib/puppet/ssl'') {
	file { ''/etc/puppet/puppet.conf'':
		content => template(''puppet.conf'')
	}
}

Then, for some crack-ass reason, we decide that we want different ssldirs
depending for the Puppetmaster as opposed to the clients:

define puppet_conf($main_ssldir = ''/var/lib/puppet/ssl'',
                   $puppetmaster_ssldir =
''/var/lib/puppet/ssl'') {
[...]

Then someone wants to start testing out pluginsync, but doesn''t want to
enable it for everyone just yet:

define puppet_conf($main_ssldir = ''/var/lib/puppet/ssl'',
                   $puppetmaster_ssldir =
''/var/lib/puppet/ssl'',
                   $enable_pluginsync = false) {
[...]

And then different types of machines want to sync different sets of plugins
(please don''t ask me why; this is (a) an example, and (b) people do
stupid
stuff like this all the time):

define puppet_conf($main_ssldir = ''/var/lib/puppet/ssl'',
                   $puppetmaster_ssldir =
''/var/lib/puppet/ssl'',
                   $enable_pluginsync = false,
                   $pluginsource =
''puppet://puppetfiles/config/plugins'') {
[...]

And so it goes.  As people dream up more and more bong stuff to jam into
the templated config file, there''s more and more parameters to the
define
and more ERb to try and handle in your template.

The standard refactoring for this is to provide a type that handles the
configuration file natively:

puppet_conf { set_main_ssldir:
	section => main,
	key => ssldir,
	value => ''/var/lib/puppet/ssl''
}

puppet_conf { set_puppetmaster_ssldir:
	section => puppetmasterd,
	key => ssldir,
	value => ''/var/lib/puppetmaster/ssl''
}

puppet_conf { set_pluginsync:
	section => main,
	key => pluginsync,
	value => true
}

And so on.

The ultimate in template perversion, I think, is amply demonstrated by this
chunk of help text we''ve got for one of our classes:

#   directories: a Ruby hash of directory trees and their options.
#     Each "directory tree" is a domain name, which will be the root
of an
#     LDAP directory tree (we only support dc-style trees).  The
''key'' of
#     each hash entry is the domain name, as a string.
#
#     The value of the entry is another hash, which specifies the replication
#     slaves and ACLs of the tree.
#
#     ''slaves'' is an array of hashes, each of which specifies
the
#     hostname (''hostname''), replication DN
(''binddn'') and replication
#     password (''bindpw'') for a single replication slave.
#
#     ''acls'' is an array of hashes, each of which specifies a
full DN
#     (''who'') and an access level (''access'')
for the DN.
#
#     This is ridiculously complex, and will remain so until Puppet gets
#     real data structures.  In the meantime, only an example could come close
to
#     explaining what the hell we''re on about:
#
#     directories => "
#       {''example.org'' => {''slaves'' =>
[{''hostname'' => ''ldap1.example.org'',
#                                        ''binddn'' =>
''cn=repl,dc=example,dc=org'',
#                                        ''bindpw'' =>
''s3kr1t''},
#                                       {''hostname'' =>
''ldap2.example.org'',
#                                        ''binddn'' =>
''cn=repl,dc=example,dc=org'',
#                                        ''bindpw'' =>
''s3kr1t''}
#                                      ],
#                          ''acls'' =>
[{''who'' =>
''uid=auser,ou=People,dc=example,dc=org'',
#                                      ''access'' =>
''write''}
#                                    ]
#                         }
#       }"
#
# Remember the quotes!

*That* is why templates are a code smell.  That monstrosity grew from a
simple whitespace-separated list of directories to initialise.

- Matt

--
"The user-friendly computer is a red herring. The user-friendliness of a
book just makes it easier to turn pages. There''s nothing user-friendly
about
learning to read."
		-- Alan Kay

Matthew Palmer wrote:
> On Sun, Aug 26, 2007 at 10:35:39PM +0200, David Schmitt wrote:
>> On Wednesday 22 August 2007, Matthew Palmer wrote:
>>> "I can''t believe nobody else wants to do this"
is often translatable to
>>> "everyone else has a different way of doing this" -- and,
at the risk of
>>> falling foul of a "fallacy of popularity", if everyone
else is doing it a
>>> different way, it''s probably worth at least considering
the popular wisdom
>>> before possibly digging yourself a deep hole.
>>>
>>> In my experience, other code smells include:
>>>
>>> * Use of exec;
>>>
>>> * Use of templates;
>> I contest your general denounciation of templates though, could you
elaborate
>> please?
>
> Templates quickly become ridiculously complex to manage, as you extend the
> parameters that the template is supposed to manage.  Suddenly your defines
> start having to take massive numbers of parameters and your templates
become
> more and more filled with ERb gunk.
>
> Take, for instance, the puppet.conf file.  To begin with, you have a static
> one that you ship with everything:
>
> file { ''/etc/puppet/puppet.conf'':
> 	source =>
"puppet://puppetfiles/config/classes/any/puppet.conf"
> }
>
> But then someone decides they want to have different values for the ssldir:
>
> define puppet_conf($ssldir = ''/var/lib/puppet/ssl'') {
> 	file { ''/etc/puppet/puppet.conf'':
> 		content => template(''puppet.conf'')
> 	}
> }
>
> Then, for some crack-ass reason, we decide that we want different ssldirs
> depending for the Puppetmaster as opposed to the clients:
>
> define puppet_conf($main_ssldir = ''/var/lib/puppet/ssl'',
>                    $puppetmaster_ssldir =
''/var/lib/puppet/ssl'') {
> [...]
>
> Then someone wants to start testing out pluginsync, but doesn''t
want to
> enable it for everyone just yet:
>
> define puppet_conf($main_ssldir = ''/var/lib/puppet/ssl'',
>                    $puppetmaster_ssldir =
''/var/lib/puppet/ssl'',
>                    $enable_pluginsync = false) {
> [...]
>
> And then different types of machines want to sync different sets of plugins
> (please don''t ask me why; this is (a) an example, and (b) people
do stupid
> stuff like this all the time):
>
> define puppet_conf($main_ssldir = ''/var/lib/puppet/ssl'',
>                    $puppetmaster_ssldir =
''/var/lib/puppet/ssl'',
>                    $enable_pluginsync = false,
>                    $pluginsource =
''puppet://puppetfiles/config/plugins'') {
> [...]
>
> And so it goes.  As people dream up more and more bong stuff to jam into
> the templated config file, there''s more and more parameters to the
define
> and more ERb to try and handle in your template.
>
> The standard refactoring for this is to provide a type that handles the
> configuration file natively:
>
> puppet_conf { set_main_ssldir:
> 	section => main,
> 	key => ssldir,
> 	value => ''/var/lib/puppet/ssl''
> }
>
> puppet_conf { set_puppetmaster_ssldir:
> 	section => puppetmasterd,
> 	key => ssldir,
> 	value => ''/var/lib/puppetmaster/ssl''
> }
>
> puppet_conf { set_pluginsync:
> 	section => main,
> 	key => pluginsync,
> 	value => true
> }
>
> And so on.
>
> The ultimate in template perversion, I think, is amply demonstrated by this
> chunk of help text we''ve got for one of our classes:
>
> #   directories: a Ruby hash of directory trees and their options.
> #     Each "directory tree" is a domain name, which will be the
root of an
> #     LDAP directory tree (we only support dc-style trees).  The
''key'' of
> #     each hash entry is the domain name, as a string.
> #
> #     The value of the entry is another hash, which specifies the
replication
> #     slaves and ACLs of the tree.
> #
> #     ''slaves'' is an array of hashes, each of which
specifies the
> #     hostname (''hostname''), replication DN
(''binddn'') and replication
> #     password (''bindpw'') for a single replication slave.
> #
> #     ''acls'' is an array of hashes, each of which
specifies a full DN
> #     (''who'') and an access level
(''access'') for the DN.
> #
> #     This is ridiculously complex, and will remain so until Puppet gets
> #     real data structures.  In the meantime, only an example could come
close to
> #     explaining what the hell we''re on about:
> #
> #     directories => "
> #       {''example.org'' => {''slaves''
=> [{''hostname'' =>
''ldap1.example.org'',
> #                                        ''binddn'' =>
''cn=repl,dc=example,dc=org'',
> #                                        ''bindpw'' =>
''s3kr1t''},
> #                                       {''hostname'' =>
''ldap2.example.org'',
> #                                        ''binddn'' =>
''cn=repl,dc=example,dc=org'',
> #                                        ''bindpw'' =>
''s3kr1t''}
> #                                      ],
> #                          ''acls'' =>
[{''who'' =>
''uid=auser,ou=People,dc=example,dc=org'',
> #                                      ''access'' =>
''write''}
> #                                    ]
> #                         }
> #       }"
> #
> # Remember the quotes!
>
> *That* is why templates are a code smell.  That monstrosity grew from a
> simple whitespace-separated list of directories to initialise.
>
Isn''t this ultimately a demonstration that if you try to create an
abstraction that is as
flexible and expressive as the thing you''re trying to abstract away
from, you wind up with
the same thing... only screwed up?

--
Russell A. Jackson <raj at csub.edu>
Network Analyst
California State University, Bakersfield

QOTD:
	"It''s been Monday all week today."

On Aug 28, 2007, at 1:33 AM, Russell Jackson wrote:
> Isn''t this ultimately a demonstration that if you try to create an
> abstraction that is as
> flexible and expressive as the thing you''re trying to abstract
away
> from, you wind up with
> the same thing... only screwed up?
Remember that ''abstract'' means to generalize and simplify.

I would say actually that templates are often not used to make an
abstraction at all.  Part of the whole goal of the native types is to
build an enforce as simple an abstraction as possible.

With templates, though, you''re never really forced to create a
simple, generalized interface.

However, configuration files that are just long lists of parameters,
like puppet.conf or sshd_config, are never going to be possible to
abstract as a whole; if you''re lucky you might be able to do one
resource per parameter, but that''s about it.

  --
  This book fills a much-needed gap.      -- Moses Hadas
  ---------------------------------------------------------------------
  Luke Kanies | http://reductivelabs.com | http://madstop.com