Puppet users - Sep 2012 - Wood/trees/hiera discrimination problem

Hello.

I think I''m being an idiot (not for the first time), but given the
following hiera paragraph:

directorymap:
  www.blah.com:
    - source: dodgy-uploaded-jpegs
      target: www.blah.com/wp-content/uploads
    - source: disturbing-home-videos
      target: www.blah.com/wp-content/filth
  www.nice-site.co.uk:
   - source: kitten-pics
   - target: www.nice-site.co.uk/site-content/user-data

What''s the least-worst way of extracting that, such that puppet
pseudo-code along the lines of

''ln -s ${site}/${source} ${site}{$target}'' would work?

There is likely a Better Way To Do It...

-- 
John Hawkes-Reed
Systems Administrator. Future Publishing. x 2526

-- 
Future Publishing Limited (registered company number 2008885) is a wholly owned
subsidiary of Future plc (registered company number 3757874), both of which are
incorporated in England and Wales and share the same registered address at
Beauford Court, 30 Monmouth Street, Bath BA1 2BW.

This email and any files transmitted with it are confidential and intended
solely for the use of the individual or entity to which they are addressed. If
you have received this email in error please reply to this email and then delete
it. Please note that any views or opinions presented in this email are solely
those of the author and do not necessarily represent those of Future.

The recipient should check this email and any attachments for the presence of
viruses. Future accepts no liability for any damage caused by any virus
transmitted by this email.

Future may regularly and randomly monitor outgoing and incoming emails and other
telecommunications on its email and telecommunications systems. By replying to
this email you give your consent to such monitoring.

*****
Save resources: think before you print.


-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Friday, September 21, 2012 6:04:37 AM UTC-5, John Hawkes-Reed
wrote:
>
> Hello. 
>
> I think I''m being an idiot (not for the first time), but given the
> following hiera paragraph: 
>
> directorymap: 
>   www.blah.com: 
>     - source: dodgy-uploaded-jpegs 
>       target: www.blah.com/wp-content/uploads 
>     - source: disturbing-home-videos 
>       target: www.blah.com/wp-content/filth 
>   www.nice-site.co.uk: 
>    - source: kitten-pics 
>    - target: www.nice-site.co.uk/site-content/user-data 
>
> What''s the least-worst way of extracting that, such that puppet 
> pseudo-code along the lines of 
>
> ''ln -s ${site}/${source} ${site}{$target}'' would work? 
>

So, it looks like that would expand to commands such as

ln -s www.nice-site.co.uk/kitten-pics  www.nice-site.co.uk
www.nice-site.co.uk/site-content/user-data

I suspect the duplication of the site on the target side is not what you 
want, but it follows from your data.  Nevertheless, I guess your real 
question is about how to make Puppet digest your nested data structure.

(Note also that your YAML data structure under "nice-site" is
inconsistent
with its cousin under "blah"; it appears that the structure under
"blah" is
what you intended.  Not so interested in kitten pics?  :-) )
 
>
> There is likely a Better Way To Do It... 
>
>
In fact, although they appear to make sense from a data design point of 
view, the arrays at the second nested level of your data structure are 
going to cause you trouble in Puppet.  Puppet does understand arrays, but 
not iteration, so you would probably need to resort to Ruby DSL to do 
anything useful with them.  If you change them to hashes, however, then you 
may have an easier time.  I.e.:

directorymap: 
  www.blah.com: 
      www.blah.com0:
        source: dodgy-uploaded-jpegs 
        target: www.blah.com/wp-content/uploads 
      www.blah.com1:
        source: disturbing-home-videos 
        target: www.blah.com/wp-content/filth 
  www.nice-site.co.uk:
    www.nice-site.co.uk0: 
      source: kitten-pics 
      target: www.nice-site.co.uk/site-content/user-data 

That duplicates data even more, however.  You could improve it further 
(IMO) by flattening it a bit and pushing down the site info, such as:

directorymap: 
  artificial-id-0:
    source: dodgy-uploaded-jpegs 
    site: www.blah.com
    target: wp-content/uploads <http://www.blah.com/wp-content/uploads> 
  artificial-id-1:
    source: disturbing-home-videos
    site: www.blah.com
    target: wp-content/filth <http://www.blah.com/wp-content/filth> 
  artificial-id-2:
    source: kitten-pics 
    site: www.nice-site.co.uk
    target:
site-content/user-data<http://www.nice-site.co.uk/site-content/user-data>

Note that now the data structure is a hash whose keys are suitable 
identifiers for resources you want to manage, and whose values are hashes 
of the desired properties of those resources.  That''s exactly what you
need
for the create_resources() function.

To use create_resources(), you also need a resource type to accept those 
parameters, and since there is no built-in type that does what you want, 
your best bet would be a defined type:

define content::link($source, $site, target) {
  file { "${content::base_directory}/${site}/${target}":
    ensure => link,
    target => "${site}/${source}"
  }
}

Note that variable $content::base_directory must be defined elsewhere (in a 
separate class "content", in this case).  This is essential because
the
base directory is not included in your data, yet is needed to correctly 
identify the link''s full path.

With those pieces in place, you can tie it all together as simply as this:

create_resources(''content::link'',
hiera(''directorymap''))


If you don''t like the revised form of your data that I suggested, or
for
future problems, there are at least a few other tools you should have in 
your toolbox:

1) A Puppet function to extract an array of keys from a hash.  You can 
write your own if you like (it''s easy), but there is one ready-made in 
Puppetlabs''s useful ''stdlib'' module 
(https://github.com/puppetlabs/puppetlabs-stdlib).

2) Defined types.  Learn them, live them, love them.  And especially, know 
that inside a defined type body, you can refer to instances'' given
titles
via the automatic $name and/or $title variables.

3) Multiple resource declaration syntax.  If the title of a resource 
declaration is an array -- either a literal one, or a variable with an 
array value -- then it means one resource for each element of the array, 
with the corresponding element as resource title.  For instance,

file { [ ''/tmp/foo'', ''/tmp/bar'' ]: ensure
=> present }

is equivalent to

file { ''/tmp/foo'': ensure => present }
file { ''/tmp/bar'': ensure => present }

That can be combined in useful ways with (1) and (2).  In particular, you 
can extract an array of hash keys, declare instances of a defined type with 
those titles, and in the body of the defined type use the title to extract 
and work with one element of the original hash.  So, if you wanted to avoid 
the create_resources() function (but use the same data structure) then you 
could do something like this instead:

$keys = keys($content::directories)

content_link2 { $keys: }

define content::link2() {
  $source = $content::directories[$name][''source'']
  $site = $content::directories[$name][''site'']
  $target = $content::directories[$name][''target'']

  file { "${content::base_directory}/${site}/${target}":
    ensure => link,
    target => "${site}/${source}"
  }
}



John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/9eHSlGxYMKYJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 21 Sep 2012, at 15:42, jcbollinger wrote:
> 
> 
> On Friday, September 21, 2012 6:04:37 AM UTC-5, John Hawkes-Reed wrote:
> Hello. 
> 
> I think I''m being an idiot (not for the first time), but given the
following hiera paragraph:
> 
> directorymap: 
>   www.blah.com: 
>     - source: dodgy-uploaded-jpegs 
>       target: www.blah.com/wp-content/uploads 
>     - source: disturbing-home-videos 
>       target: www.blah.com/wp-content/filth 
>   www.nice-site.co.uk: 
>    - source: kitten-pics 
>    - target: www.nice-site.co.uk/site-content/user-data 
> 
> What''s the least-worst way of extracting that, such that puppet
pseudo-code along the lines of
> 
> ''ln -s ${site}/${source} ${site}{$target}'' would work? 
> 
> 
> So, it looks like that would expand to commands such as
> 
> ln -s www.nice-site.co.uk/kitten-pics 
www.nice-site.co.ukwww.nice-site.co.uk/site-content/user-data
> 
> I suspect the duplication of the site on the target side is not what you
want, but it follows from your data.  Nevertheless, I guess your real question
is about how to make Puppet digest your nested data structure.
Yep. I''ve been staring at the problem slightly too long...
> 
> (Note also that your YAML data structure under "nice-site" is
inconsistent with its cousin under "blah"; it appears that the
structure under "blah" is what you intended.  Not so interested in
kitten pics?  :-) )
:p
>  
> 
> There is likely a Better Way To Do It... 
> 
> 
> In fact, although they appear to make sense from a data design point of
view, the arrays at the second nested level of your data structure are going to
cause you trouble in Puppet.  Puppet does understand arrays, but not iteration,
so you would probably need to resort to Ruby DSL to do anything useful with
them.  If you change them to hashes, however, then you may have an easier time. 
I.e.:
> 
> directorymap: 
>   www.blah.com: 
>       www.blah.com0:
>         source: dodgy-uploaded-jpegs 
>         target: www.blah.com/wp-content/uploads 
>       www.blah.com1:
>         source: disturbing-home-videos 
>         target: www.blah.com/wp-content/filth 
>   www.nice-site.co.uk:
>     www.nice-site.co.uk0: 
>       source: kitten-pics 
>       target: www.nice-site.co.uk/site-content/user-data 
> 
> That duplicates data even more, however.  You could improve it further
(IMO) by flattening it a bit and pushing down the site info, such as:
> 
> directorymap: 
>   artificial-id-0:
>     source: dodgy-uploaded-jpegs 
>     site: www.blah.com
>     target: wp-content/uploads 
>   artificial-id-1:
>     source: disturbing-home-videos
>     site: www.blah.com
>     target: wp-content/filth 
>   artificial-id-2:
>     source: kitten-pics 
>     site: www.nice-site.co.uk
>     target: site-content/user-data 
> 
> Note that now the data structure is a hash whose keys are suitable
identifiers for resources you want to manage, and whose values are hashes of the
desired properties of those resources.  That''s exactly what you need
for the create_resources() function.
> 
> To use create_resources(), you also need a resource type to accept those
parameters, and since there is no built-in type that does what you want, your
best bet would be a defined type:
> 
> define content::link($source, $site, target) {
>   file { "${content::base_directory}/${site}/${target}":
>     ensure => link,
>     target => "${site}/${source}"
>   }
> }
> 
> Note that variable $content::base_directory must be defined elsewhere (in a
separate class "content", in this case).  This is essential because
the base directory is not included in your data, yet is needed to correctly
identify the link''s full path.
> 
> With those pieces in place, you can tie it all together as simply as this:
> 
> create_resources(''content::link'',
hiera(''directorymap''))
> 
> 
> If you don''t like the revised form of your data that I suggested,
or for future problems, there are at least a few other tools you should have in
your toolbox:
> 
> 1) A Puppet function to extract an array of keys from a hash.  You can
write your own if you like (it''s easy), but there is one ready-made in
Puppetlabs''s useful ''stdlib'' module
(https://github.com/puppetlabs/puppetlabs-stdlib).
> 
> 2) Defined types.  Learn them, live them, love them.  And especially, know
that inside a defined type body, you can refer to instances'' given
titles via the automatic $name and/or $title variables.
> 
> 3) Multiple resource declaration syntax.  If the title of a resource
declaration is an array -- either a literal one, or a variable with an array
value -- then it means one resource for each element of the array, with the
corresponding element as resource title.  For instance,
> 
> file { [ ''/tmp/foo'', ''/tmp/bar'' ]:
ensure => present }
> 
> is equivalent to
> 
> file { ''/tmp/foo'': ensure => present }
> file { ''/tmp/bar'': ensure => present }
> 
> That can be combined in useful ways with (1) and (2).  In particular, you
can extract an array of hash keys, declare instances of a defined type with
those titles, and in the body of the defined type use the title to extract and
work with one element of the original hash.  So, if you wanted to avoid the
create_resources() function (but use the same data structure) then you could do
something like this instead:
> 
> $keys = keys($content::directories)
> 
> content_link2 { $keys: }
> 
> define content::link2() {
>   $source = $content::directories[$name][''source'']
>   $site = $content::directories[$name][''site'']
>   $target = $content::directories[$name][''target'']
> 
>   file { "${content::base_directory}/${site}/${target}":
>     ensure => link,
>     target => "${site}/${source}"
>   }
> }
> 
That''s really quite wonderful. Thank you.

As a team, we''re in the middle of re-factoring a lot of old (in Puppet
terms) code, which means Barberising the modules, lots of Jenkins runs with
puppet-lint... And now discovering that we can throw away circa half the cruft
if we use hiera/create_resources right and spend some time looking carefully at
the design of the yaml in the hiera tree.


-- 
John Hawkes-Reed
Systems Administrator. Future Publishing. x 2526

-- 
Future Publishing Limited (registered company number 2008885) is a wholly owned
subsidiary of Future plc (registered company number 3757874), both of which are
incorporated in England and Wales and share the same registered address at
Beauford Court, 30 Monmouth Street, Bath BA1 2BW.

This email and any files transmitted with it are confidential and intended
solely for the use of the individual or entity to which they are addressed. If
you have received this email in error please reply to this email and then delete
it. Please note that any views or opinions presented in this email are solely
those of the author and do not necessarily represent those of Future.

The recipient should check this email and any attachments for the presence of
viruses. Future accepts no liability for any damage caused by any virus
transmitted by this email.

Future may regularly and randomly monitor outgoing and incoming emails and other
telecommunications on its email and telecommunications systems. By replying to
this email you give your consent to such monitoring.

*****
Save resources: think before you print.


-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.