Puppet users - Apr 2010 - Design issue (or: problem with virtual resources)

Hello .*,

sorry this is a bit lengthy:

I''m in the progress of setting up a puppet system for our Linux
cluster (SLES) and have a ''small'' design question, since
I''m not able
to realize my ideas with capabilities of puppet as I have them in
mind. At least up to now.

The general setup looks like this:
There is a central database where all information of the hosts are
stored. Next to special hardware layout and network configurations it
is stored which type of service a host is offering (e.g.
cluster-batchnode, cluter-loginnode, server-nagios, etc.).
Connected with this services are the puppet classes which have to be
realized on a specific host. This is passed down to puppet by a YAML
file with as a list of classes and parameters per host.

The classes are organized in modules with a visioning, e.g.
ssh-1::client, ssh-1::server, auth-3::ldap etc. In this way (in view
of change management), when changing something in the module, I copy
the whole module to a new version and (after some testing) just
migrate all the affected services to the new module version.

So far so good. Now my problem emerges, when a module depends on an
other module. This should be version independent (and even better
independent of the module name). For example:
I have a generic nas-<version>::virtual module, which provides
mountpoint as virtual resources. These can be realized in multiple
other modules, without explicitly prepending a Nas-<version>::Virtual,
so I am independent of version and module name.

Now the ''but'':
Typically this virtual resources would depend on other resources (like
a mount would depened on a directory to exists). But since
requirements are not automatically realizes, this does not work! The
virtual file resource for the  directory is not automatically realized
when realizing the mount.

There are multiple solution I thought of:
1. putting everything into a virutal define: but this does not work,
since the define is not globally known, so I have to reference it with
exlipcit module and version Nas-<version>::Virtual::<Define>...
2. changing the ''realize'' function to an rvalue and using it
in the
require parameter to realize the requirement. But this does not work,
since the requirements are already realized during ''reading''
the
definition of the main virtual resource and not just when realizing
it!
3. Taking away the version numbering in the module name and giving
each module an own module directory (in fileserver.conf like for the
files directory with a %h variable) and symlinking the required module
version into this per host directory. But this only works for file not
for modules,
...

So I''m quite lost at this point... The automated realization of
required resources would already solve everything here (Issue here:
<http://projects.puppetlabs.com/issues/3178>). But up to now I''m
stuck
here and some hundreds of nodes want to be puppified... :-)

I''m grateful for ideas. There might be a big design flaw as well....

Thanks a lot in advance for reading so much of my brain dump...

Bye,
Frederik

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Wed, Apr 14, 2010 at 8:33 AM, Frederik Wagner <fnerdwq@googlemail.com>
wrote:
> <snip>
> So far so good. Now my problem emerges, when a module depends on an
> other module. This should be version independent (and even better
> independent of the module name). For example:
> I have a generic nas-<version>::virtual module, which provides
> mountpoint as virtual resources. These can be realized in multiple
> other modules, without explicitly prepending a
Nas-<version>::Virtual,
> so I am independent of version and module name.
I don''t personally do a whole lot with the realize function, but the
way I lay out my modules may help with the version dependency problem
you''re facing.

In modules I write, my init.pp simply includes a manifest named the
same as the module.  For the ntp module I just have import "ntp.pp" in
init.pp

I place all classes in ntp.pp.  The first class named ntp is
responsible for doing a selection based on facts to determine the
specific class for the platform the module is configuring.  This looks
something like:

# modules/ntp/manifests/ntp.pp
class ntp {
  case $operatingsystem {
    "RedHat", "CentOS": { include ntp::linux-rh }
    "Solaris": { include ntp::solaris }
    default: { include ntp::unconfigured }
  }
}
class ntp::common { # do stuff }
class ntp::linux-rh inherits ntp::common { # override stuff }
class ntp::solaris inherits ntp::common { # override stuff }
class ntp::unconfigured { # throw warning }
# EOF

ntp::linux-rh and ntp::solaris both inherit from ntp::common and
provide platform specific overrides of the base class.  This allows me
to simply classify the node as "ntp" and let the module sort out the
platform specifics, or do nothing at all if the platform isn''t
supported.  (I do have the unconfigured classes throw warnings).

You should be able to do something similar with specific versions of
your modules.  Rather than copy your entire module directory, you
could have modules/nas/manifest/nas-{1,2,3,4,...}.pp, have init.pp do
a import "*.pp" and then have your node classification point at your
specific "versioned" class, e.g. nas::nas-3 (or some better naming
scheme).  To achieve your dependency on some consistent "non
versioned" class you could have all of your versioned classes include
or descend from another class in the module, say nas::common or
nas::nas.  This class wouldn''t necessarily do anything except provide
an anchor to reference in your other modules.

Does this make sense?  This is purely me thinking out loud while half
way through my first cup of coffee, so this may not actually solve the
problem but I believe puppet is currently able to provide what you
need, although it may be too much of a hack to be a desirable
solution.

Hope this helps,
--
Jeff McCune

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi Jeff,

thanks for your ideas!

On Wed, Apr 14, 2010 at 3:12 PM, Jeff McCune <mccune.jeff@gmail.com>
wrote:
> On Wed, Apr 14, 2010 at 8:33 AM, Frederik Wagner
<fnerdwq@googlemail.com> wrote:
>> <snip>
>> So far so good. Now my problem emerges, when a module depends on an
>> other module. This should be version independent (and even better
>> independent of the module name). For example:
>> I have a generic nas-<version>::virtual module, which provides
>> mountpoint as virtual resources. These can be realized in multiple
>> other modules, without explicitly prepending a
Nas-<version>::Virtual,
>> so I am independent of version and module name.
>
> I don''t personally do a whole lot with the realize function, but
the
> way I lay out my modules may help with the version dependency problem
> you''re facing.
>
> In modules I write, my init.pp simply includes a manifest named the
> same as the module.  For the ntp module I just have import
"ntp.pp" in
> init.pp
>
> I place all classes in ntp.pp.  The first class named ntp is
> responsible for doing a selection based on facts to determine the
> specific class for the platform the module is configuring.  This looks
> something like:
>
> # modules/ntp/manifests/ntp.pp
> class ntp {
>  case $operatingsystem {
>    "RedHat", "CentOS": { include ntp::linux-rh }
>    "Solaris": { include ntp::solaris }
>    default: { include ntp::unconfigured }
>  }
> }
> class ntp::common { # do stuff }
> class ntp::linux-rh inherits ntp::common { # override stuff }
> class ntp::solaris inherits ntp::common { # override stuff }
> class ntp::unconfigured { # throw warning }
> # EOF
>
> ntp::linux-rh and ntp::solaris both inherit from ntp::common and
> provide platform specific overrides of the base class.  This allows me
> to simply classify the node as "ntp" and let the module sort out
the
> platform specifics, or do nothing at all if the platform isn''t
> supported.  (I do have the unconfigured classes throw warnings).
>
> You should be able to do something similar with specific versions of
> your modules.  Rather than copy your entire module directory, you
> could have modules/nas/manifest/nas-{1,2,3,4,...}.pp, have init.pp do
> a import "*.pp" and then have your node classification point at
your
> specific "versioned" class, e.g. nas::nas-3 (or some better
naming
> scheme).  To achieve your dependency on some consistent "non
> versioned" class you could have all of your versioned classes include
> or descend from another class in the module, say nas::common or
> nas::nas.  This class wouldn''t necessarily do anything except
provide
> an anchor to reference in your other modules.
>
> Does this make sense?  This is purely me thinking out loud while half
> way through my first cup of coffee, so this may not actually solve the
> problem but I believe puppet is currently able to provide what you
> need, although it may be too much of a hack to be a desirable
> solution.
This makes sense. I cold have a ''wrapper'' for each module
including a
versioned class depending on some passed parameter, e.g. I pass
through my external node script the version of the nas module
$nas_version and define a class like:

class nas::mount {
   include nas::mount-$nas_version
}

in this way I could get rid of the naming scheme in the main class,
but - as far as I tried it - it''s not possible to include classes with
names composed of variables...

Further I''m not yet seeing how I get rid of my virtual resource
problem... or wait... I then could use an own class per mountpoint
instead of a virtual resource. So using include instead of realize.
hmm...

Thanks a lot so far.
Frederik
>
> Hope this helps,
> --
> Jeff McCune
>
> --
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Wed, Apr 14, 2010 at 4:06 PM, Frederik Wagner <fnerdwq@googlemail.com>
wrote:
> Hi Jeff,
>
> thanks for your ideas!
>
> On Wed, Apr 14, 2010 at 3:12 PM, Jeff McCune <mccune.jeff@gmail.com>
wrote:
>> On Wed, Apr 14, 2010 at 8:33 AM, Frederik Wagner
<fnerdwq@googlemail.com> wrote:
>>> <snip>
>>> So far so good. Now my problem emerges, when a module depends on an
>>> other module. This should be version independent (and even better
>>> independent of the module name). For example:
>>> I have a generic nas-<version>::virtual module, which
provides
>>> mountpoint as virtual resources. These can be realized in multiple
>>> other modules, without explicitly prepending a
Nas-<version>::Virtual,
>>> so I am independent of version and module name.
>>
>> I don''t personally do a whole lot with the realize function,
but the
>> way I lay out my modules may help with the version dependency problem
>> you''re facing.
>>
>> In modules I write, my init.pp simply includes a manifest named the
>> same as the module.  For the ntp module I just have import
"ntp.pp" in
>> init.pp
>>
>> I place all classes in ntp.pp.  The first class named ntp is
>> responsible for doing a selection based on facts to determine the
>> specific class for the platform the module is configuring.  This looks
>> something like:
>>
>> # modules/ntp/manifests/ntp.pp
>> class ntp {
>>  case $operatingsystem {
>>    "RedHat", "CentOS": { include ntp::linux-rh }
>>    "Solaris": { include ntp::solaris }
>>    default: { include ntp::unconfigured }
>>  }
>> }
>> class ntp::common { # do stuff }
>> class ntp::linux-rh inherits ntp::common { # override stuff }
>> class ntp::solaris inherits ntp::common { # override stuff }
>> class ntp::unconfigured { # throw warning }
>> # EOF
>>
>> ntp::linux-rh and ntp::solaris both inherit from ntp::common and
>> provide platform specific overrides of the base class.  This allows me
>> to simply classify the node as "ntp" and let the module sort
out the
>> platform specifics, or do nothing at all if the platform isn''t
>> supported.  (I do have the unconfigured classes throw warnings).
>>
>> You should be able to do something similar with specific versions of
>> your modules.  Rather than copy your entire module directory, you
>> could have modules/nas/manifest/nas-{1,2,3,4,...}.pp, have init.pp do
>> a import "*.pp" and then have your node classification point
at your
>> specific "versioned" class, e.g. nas::nas-3 (or some better
naming
>> scheme).  To achieve your dependency on some consistent "non
>> versioned" class you could have all of your versioned classes
include
>> or descend from another class in the module, say nas::common or
>> nas::nas.  This class wouldn''t necessarily do anything except
provide
>> an anchor to reference in your other modules.
>>
>> Does this make sense?  This is purely me thinking out loud while half
>> way through my first cup of coffee, so this may not actually solve the
>> problem but I believe puppet is currently able to provide what you
>> need, although it may be too much of a hack to be a desirable
>> solution.
>
> This makes sense. I cold have a ''wrapper'' for each module
including a
> versioned class depending on some passed parameter, e.g. I pass
> through my external node script the version of the nas module
> $nas_version and define a class like:
>
> class nas::mount {
>   include nas::mount-$nas_version
> }
>
> in this way I could get rid of the naming scheme in the main class,
> but - as far as I tried it - it''s not possible to include classes
with
> names composed of variables...
I think I have to correct myself on this part... It seems to work, not
as mentioned above but like

class nas::mount {
   $nasmount="nas::mount-${nas_version}"
   include $nasmount
}
>
> Further I''m not yet seeing how I get rid of my virtual resource
> problem... or wait... I then could use an own class per mountpoint
> instead of a virtual resource. So using include instead of realize.
> hmm...
>
> Thanks a lot so far.
> Frederik
>
>>
>> Hope this helps,
>> --
>> Jeff McCune
>>
>> --
>> You received this message because you are subscribed to the Google
Groups "Puppet Users" group.
>> To post to this group, send email to puppet-users@googlegroups.com.
>> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
>> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
>>
>>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.