Puppet users - Aug 2011 - Handling ENC / Class interface changes

Hi all,

Recently I''ve hit against this problem which is proving to be a
formidable foe. We have an in-house developed ENC and use
parameterised classes just about everywhere and as a result enjoy a
fairly flexible way of handing out configurations and very well-
defined interfaces between components. We also quite strictly version
our Puppet code but there isn''t a fixed versioning between the ENC
YAML and the Puppet code that it drives.

Whenever we want to add additional parameters to certain classes,
unless we update that class and the YAML that drives it atomically
everywhere, we end up in a situation where the YAML does not match the
parameterised class and causes errors. Perhaps it is best shown as
code:

YAML:
classes:
  foo:
    var1: foo
    var2: bar

Puppet:
class foo (
  $var1,
  $var2
) { ... }

Now we want to add a new parameter to this class for a certain subset
of nodes that need new functionality:
class foo (
  $var1,
  $var2,
  $var3 = ''somedefault''
) { ... }

Since we have module versioning, we only need to worry about the new
nodes getting this version of the class. Unfortunately the YAML that
drives them is the same. The problem now is that adding the new
parameter to the YAML will break any machine that uses the old class -
additional parameters that don''t exist in the class interface will
cause a failure.

We''ve thought of a few solutions:
  * pass all the parameters as a hash (breaks the interface
consistency and verification, requires additional hash parameter
handling)
  * hack Puppet to not complain about additional unexpected parameters
(breaks the concept of well-defined interfaces)
  * hack Puppet to take an arg* final parameter (an awful combination
of the above two)
  * version our YAML interfaces in lockstep with Puppet classes
(removes our ability to benefit from inheritance of common class
definitions in the ENC)
  * never add new parameters to anything unless we can atomically
change it everywhere at once
  * using class introspection, generate class interface YAML that is
read and used by our ENC instead of using a separate configuration
specification, store this with the versioned Puppet classes.

I guess this could constitute a fairly solid argument against the
parameterised class interface benefits I''ve been arguing for and a
theoretical vote for Hiera ;) Anyway any thoughts or abuse are welcome!

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hey Oliver :-).
>  * using class introspection, generate class interface YAML that is
> read and used by our ENC instead of using a separate configuration
> specification, store this with the versioned Puppet classes.
I guess resource_type is your friend in this case. Info should be
available in the REST API guide. Sounds tricky though.
> I guess this could constitute a fairly solid argument against the
> parameterised class interface benefits I''ve been arguing for and a
> theoretical vote for Hiera ;)
Indeed. Hiera or any function could be designed to be more forgiving
... since you can be in control of the class instantiation not Puppet
core. Really you would just be solving the problem in similar ways as
you defined in your email ... but without hacking core.
Benefits/drawback''s don''t change much - just the methodology
becomes
easier.

ken.

-- 
"Join us for PuppetConf, September 22nd and 23rd in Portland, OR:
http://bit.ly/puppetconfsig"

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Aug 8, 7:51 am, Oliver Hookins <ohook...@gmail.com>
wrote:
> Hi all,
>
> Recently I''ve hit against this problem which is proving to be a
> formidable foe. We have an in-house developed ENC and use
> parameterised classes just about everywhere and as a result enjoy a
> fairly flexible way of handing out configurations and very well-
> defined interfaces between components. We also quite strictly version
> our Puppet code but there isn''t a fixed versioning between the ENC
> YAML and the Puppet code that it drives.

That right there is what I would consider the root of your problem.
Well, combined with the fact that you''re using parameterized classes.
Generally speaking, a situation like yours is one of the few for which
I don''t much object to parameterized classes, but you do then need to
keep your ENC synchronized with your class interfaces, one way or
another.

> Whenever we want to add additional parameters to certain classes,
> unless we update that class and the YAML that drives it atomically
> everywhere, we end up in a situation where the YAML does not match the
> parameterised class and causes errors. Perhaps it is best shown as
> code:
>
> YAML:
> classes:
>   foo:
>     var1: foo
>     var2: bar
>
> Puppet:
> class foo (
>   $var1,
>   $var2
> ) { ... }
>
> Now we want to add a new parameter to this class for a certain subset
> of nodes that need new functionality:
> class foo (
>   $var1,
>   $var2,
>   $var3 = ''somedefault''
> ) { ... }
>
> Since we have module versioning, we only need to worry about the new
> nodes getting this version of the class. Unfortunately the YAML that
> drives them is the same. The problem now is that adding the new
> parameter to the YAML will break any machine that uses the old class -
> additional parameters that don''t exist in the class interface will
> cause a failure.

I''m not a big fan of module versioning, though I appreciate why some
shops would favor it.  In this case, your problem would be easier to
address if there were only one version of each module in use at a time
for all your nodes.

> We''ve thought of a few solutions:
>   * pass all the parameters as a hash (breaks the interface
> consistency and verification, requires additional hash parameter
> handling)
Yuck.
>   * hack Puppet to not complain about additional unexpected parameters
> (breaks the concept of well-defined interfaces)
Double yuck.
>   * hack Puppet to take an arg* final parameter (an awful combination
> of the above two)
You have got to be kidding.
>   * version our YAML interfaces in lockstep with Puppet classes
> (removes our ability to benefit from inheritance of common class
> definitions in the ENC)
I don''t follow you precisely, but you probably do need to do something
along these lines.
>   * never add new parameters to anything unless we can atomically
> change it everywhere at once
That sounds something like "[having] only one set of module versions
in use at a time for all your nodes," as I described above.  I would
not personally be worried about atomicity in the sense of changing the
configurations already applied, only about atomically changing the
configurations that nodes will get the the next time they request one
from the master.  That falls out naturally from having only one
version of the manifest for each class in use at a time.
>   * using class introspection, generate class interface YAML that is
> read and used by our ENC instead of using a separate configuration
> specification, store this with the versioned Puppet classes.

That seems like a special case of versioning the YAML interfaces in
lockstep with your classes.  It should work, but it sounds painful to
set up and maintain.

> I guess this could constitute a fairly solid argument against the
> parameterised class interface benefits I''ve been arguing for and a
> theoretical vote for Hiera ;) Anyway any thoughts or abuse are welcome!

I confess I find it both interesting and surprising that you have run
into problems using parameterized classes in the type of context you
describe, but I guess you have an unusually complex combination of
techniques and goals in play.  If have not played with Hiera, but I
like what I''ve read about it.  I would personally account it a design
win to separate nodes'' data from their classification, almost as much
as it is a win to separate the data from the classes themselves.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

> Whenever we want to add additional parameters to certain classes,
 > unless we update that class and the YAML that drives it atomically
 > everywhere, we end up in a situation where the YAML does not match the
 > parameterised class and causes errors.

Warning: brainstorm ahead.

As long as you can sensibly default your new parameters there shouldn''t
be a problem.
If puppet won''t allow the ENC to rely on defaults then that would be a 
bug in puppet,
I should think.  I haven''t tried it, though.  So I can''t
report from
experience.

If you think of your module names as interface handles your problem goes 
away.
When you change the interface you naturally change the  module name as well.

Did you depart from the foo interface?  Great, your new module is named 
foo2,
and gets its own YAML and ENC section.

Conversely, any change that does not change the interface (say adding 
another distro)
does not require a new name.

This introduces other problems, such as having canonical names for 
classes/modules.
Not sure how far this idea could be taken, but aliasing names is  sort 
of  possible:


vagn@nika:~/puppet-nika/patterns$ cat alias.pp
#! /usr/bin/puppet apply

class foo {
         notice("this is SPARTA!")
}

$provides_sparta = ''foo''
include $provides_sparta

vagn@nika:~/puppet-nika/patterns$ ./alias.pp
notice: Scope(Class[Foo]): this is SPARTA!
notice: Finished catalog run in 0.15 seconds

--
vagn

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Mon, Aug 8, 2011 at 5:51 AM, Oliver Hookins <ohookins@gmail.com> wrote:
> Hi all,
>
> Recently I''ve hit against this problem which is proving to be a
> formidable foe. We have an in-house developed ENC and use
> parameterised classes just about everywhere and as a result enjoy a
> fairly flexible way of handing out configurations and very well-
> defined interfaces between components. We also quite strictly version
> our Puppet code but there isn''t a fixed versioning between the ENC
> YAML and the Puppet code that it drives.
>
> Whenever we want to add additional parameters to certain classes,
> unless we update that class and the YAML that drives it atomically
> everywhere, we end up in a situation where the YAML does not match the
> parameterised class and causes errors. Perhaps it is best shown as
> code:
>
> YAML:
> classes:
>  foo:
>    var1: foo
>    var2: bar
>
> Puppet:
> class foo (
>  $var1,
>  $var2
> ) { ... }
>
> Now we want to add a new parameter to this class for a certain subset
> of nodes that need new functionality:
> class foo (
>  $var1,
>  $var2,
>  $var3 = ''somedefault''
> ) { ... }
>
> Since we have module versioning, we only need to worry about the new
> nodes getting this version of the class. Unfortunately the YAML that
> drives them is the same. The problem now is that adding the new
> parameter to the YAML will break any machine that uses the old class -
> additional parameters that don''t exist in the class interface will
> cause a failure.
>
> We''ve thought of a few solutions:
>  * pass all the parameters as a hash (breaks the interface
> consistency and verification, requires additional hash parameter
> handling)
>  * hack Puppet to not complain about additional unexpected parameters
> (breaks the concept of well-defined interfaces)
>  * hack Puppet to take an arg* final parameter (an awful combination
> of the above two)
>  * version our YAML interfaces in lockstep with Puppet classes
> (removes our ability to benefit from inheritance of common class
> definitions in the ENC)
>  * never add new parameters to anything unless we can atomically
> change it everywhere at once
>
this is something that may be worth considering. If you know all of your
possible environments in advance, you could copy your data hierarchy (or
more reasonably branch) and version it to match those environments. This
would ensure that the final resulting data should only match the current
interfaces of the environment.


>  * using class introspection, generate class interface YAML that is
> read and used by our ENC instead of using a separate configuration
> specification, store this with the versioned Puppet classes.
>
this would be my vote, the resource_type interface provides all of the
information you need to do this, and its all possible through a well defined
interface.

Since you are generating the environment on the fly, you should have all of
the information that you need to query for mismatches against the current
interfaces. The final data set could be munged and produce warnings if there
are data mismatches.


> I guess this could constitute a fairly solid argument against the
> parameterised class interface benefits I''ve been arguing for and a
> theoretical vote for Hiera ;) Anyway any thoughts or abuse are welcome!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>

-- 
"Join us for PuppetConf <http://bit.ly/puppetconfsig>, September 22nd
and
23rd in Portland, OR."
 <http://bit.ly/puppetconfsig>

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Thanks for the replies so far from all. It does seem like it is both a
problem with the fact that I decided to explicitly version our
application modules and provide a mechanism for tieing configuration
"blueprints" in the ENC side to the version of the application module
in use, but no such mechanism for our common modules which naively I
had expected to remain relatively static.

So the ultimate solution may be to provide versioning in a similar
way, but by utilising the class interface specifications already
generated by introspection. I''m not sure what I would get from
resource_type in addition to what I already have, since it means some
additional work for every class to be called... surely there will be a
performance overhead.

The only remaining question is how to then tie configuration data
which now is the only static data coming into the ENC from the user,
with the class interfaces coming from the Puppet code. I expect I''ll
puzzle over this over the next week and report back ;)

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.