Puppet users - Sep 2013 - Automation of puppet configuration code

Hello,

   I work for the CHTC (Center for High Throughput Computing) at the 
University of Wisconsin, Madison.  We utilize puppet for most of our 
configuration management.  Currently we have a web application that 
performs the tasks of user account management.  When a new user is 
registered with the application, the application does some very limited 
parsing of a puppet manifest that has resources like this one:

    @useraccount { ''johndoe'':
        uid => ''XXXXX'',
        fullname => ''John Doe'',
        email => ''johndoe@wisc.edu'',
        home => ''/home/johndoe'',
        password => ''SOMEHASH'',
        ensure => ''present'',
        group => ''mathdepartment'',
        tag => ''cluster-users''
    }

If a resource doesn''t exist for the new users, the application edits
the
file and creates the resource, populating the fields with the proper 
values.  The code that does this is very limited.  I would like it to do 
things like:

- Change fields that the user web application governs;  Leave fields alone 
that the web application is not aware of
- For fields that are arrays or hashes, allow for appending values to the 
array or hash, rather than replacing it
- Do resource dependency checking;  Does the group resource 
''mathdepartment'' exist? Will it be
''present''? Will the manifest that the
''mathdepartment'' group resource is in be read and the class
declared?
- Check to see if there are any puppet errors in the code at all.  If a 
client pulls the configuration for the manager, will it receive an error?

I would think the best way to accomplish these goals would be to use an 
existing API or library that can actually parse puppet code, save it into 
data structures, and run functions of the puppet engine on said data 
structures.  Basically, the functions of the parser the puppet engine 
itself uses.  Is it possible to use the puppet code/engine in such a way? 
 Even better would be if it could take puppet data structures and write 
them into a puppet manifest.

Google reveals that there are various third party open source projects that 
do something similar, written in various languages.  If one of these works 
well for you, I would like to hear which one and why you like it.  However, 
it would be advantageous for me to use the actual puppet engine since it 
will keep pace with updates to puppet.

I''d appreciate any help or advice you can give.

Cheers,
Aaron Moate
CHTC Infrastructure Team

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

I haven''t heard of any libraries for parsing or building puppet
resources.
But the syntax check should be fairly trivial - you can use the "puppet
parser validate <file.pp>" command. It''s fairly common
practice to set up a
pre-commit hook script to run that against all files modified in a git
repo, for instance.

If you aren''t having any luck looking for libraries to parse puppet
manifests, you might consider using hiera to define resources and
parameters. You can use yaml or json to structure the data, and there are
plenty of libraries for parsing that stuff. Then you could use
create_resources to generate the appropriate resources automatically during
catalog compilation. For example, if your hiera yaml had this:

classes:
  - users

users:
  tom:
    uid: 512
    home: ''/home/tom''
  dick:
    uid: 513
    home: ''/home/dick''


And then your modules/users/manifests/init.pp file contained this (that
classes array will auto-load the users module as long as you have
"hiera_include(''classes'')" somewhere like in your
site.pp - alternatively
you could just "include users" for the relevant nodes):

class users {
  $users_hash = hiera_hash(''users'', false)
  if ($users_hash) {
    create_resources(''new_user'', $users_hash)
  }
}

define new_user (
  $uid = undef,
  $home = undef,
) {
  user { $title:
    uid     => $uid,
    home => $home,
  }
}

It''s a bit odd and seems backwards at first, but it moves the
configuration
elements out into a pure structured format.

Chris


On Fri, Sep 20, 2013 at 10:48 AM, <moate@wisc.edu> wrote:
> Hello,
>
>    I work for the CHTC (Center for High Throughput Computing) at the
> University of Wisconsin, Madison.  We utilize puppet for most of our
> configuration management.  Currently we have a web application that
> performs the tasks of user account management.  When a new user is
> registered with the application, the application does some very limited
> parsing of a puppet manifest that has resources like this one:
>
>     @useraccount { ''johndoe'':
>         uid => ''XXXXX'',
>         fullname => ''John Doe'',
>         email => ''johndoe@wisc.edu'',
>         home => ''/home/johndoe'',
>         password => ''SOMEHASH'',
>         ensure => ''present'',
>         group => ''mathdepartment'',
>         tag => ''cluster-users''
>     }
>
> If a resource doesn''t exist for the new users, the application
edits the
> file and creates the resource, populating the fields with the proper
> values.  The code that does this is very limited.  I would like it to do
> things like:
>
> - Change fields that the user web application governs;  Leave fields alone
> that the web application is not aware of
> - For fields that are arrays or hashes, allow for appending values to the
> array or hash, rather than replacing it
> - Do resource dependency checking;  Does the group resource
> ''mathdepartment'' exist? Will it be
''present''? Will the manifest that the
> ''mathdepartment'' group resource is in be read and the
class declared?
> - Check to see if there are any puppet errors in the code at all.  If a
> client pulls the configuration for the manager, will it receive an error?
>
> I would think the best way to accomplish these goals would be to use an
> existing API or library that can actually parse puppet code, save it into
> data structures, and run functions of the puppet engine on said data
> structures.  Basically, the functions of the parser the puppet engine
> itself uses.  Is it possible to use the puppet code/engine in such a way?
>  Even better would be if it could take puppet data structures and write
> them into a puppet manifest.
>
> Google reveals that there are various third party open source projects
> that do something similar, written in various languages.  If one of these
> works well for you, I would like to hear which one and why you like it.
>  However, it would be advantageous for me to use the actual puppet engine
> since it will keep pace with updates to puppet.
>
> I''d appreciate any help or advice you can give.
>
> Cheers,
> Aaron Moate
> CHTC Infrastructure Team
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscribe@googlegroups.com.
> To post to this group, send email to puppet-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/puppet-users.
> For more options, visit https://groups.google.com/groups/opt_out.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.