thr3ads.net - Puppet users - [Puppet Users] Making dependencies work with variable resource names [Feb 2011]

If this information is useful, please help other people find it:
Share via:

Matthew Pounsett

2011-Feb-11 17:42 UTC

[Puppet Users] Making dependencies work with variable resource names

I''m having an issue solving dependencies inside defines, where the
paths to various resources are variable.  It seems like puppet isn''t
expanding all of the variables when it constructs the catalog, so it''s
unable to find the resources necessary to build things in the right order.

I''ve constructed a simple proof of concept to demonstrate the problem. 
I''m hoping someone can provide some advice on how this case should be
handled, because clearly I''ve misunderstood how puppet is intended to
be used to handle this sort of case.

Here is my site.pp, with a single machine loading up the module that builds a
standard service.  Each instance of the service has its own homedir with its own
data, logs, etc.

node "puppet-bsd2.virtual" {
    include service
    service::app {
        "first":
            service_num =>  "1";
        "second":
            service_num =>  "2";
    }
}

Here''s the init.pp manifest for the ''service'' module.
It includes the ''devices'' module which is used for creating
device files in a chroot environment, and sets up the homedir for each instance
of the service.

include devices
class service {
    file {
        "/opt":
            ensure => directory,
            owner => root,
            group => wheel,
            mode => 755;
        "/opt/home":
            ensure => directory,
            owner => root,
            group => wheel,
            mode => 755;
    }
    define app (
        $homedir = "/opt/home",
        $service_num
    ) {
        file {
            "$homedir/${service_num}":
                ensure  =>  directory,
                owner   =>  root,
                group   =>  wheel,
                mode    =>  0750;
            "$homedir/${service_num}/dev":
                ensure  =>  directory,
                owner   =>  root,
                group   =>  wheel,
                mode    =>  0750;
        }
        devices::device_node {
            "${homedir}/${service_num}/dev/null":
                dir     => "$homedir/${service_num}/dev/";
            "${homedir}/${service_num}/dev/random":
                dir     => "$homedir/${service_num}/dev/";
        }
    }
}

And finally, the devices module.  

class devices {
    define device_node (
        $dir
    ) {
        exec {
            "create-device-${name}":
                creates => "${name}",
                cwd => ${dir},
                command => "/usr/bin/touch ${name}",
        }
    }
}


The problem is one of order of operations.  With the above manifests, puppet
always tries to write the device files before it creates the
''var'' directory that contains them.  According to the
''require'' documentation, ''cwd'' inside an
exec clause should auto-require the directory referenced.  This wasn''t
working, and so I tried to change the ''cwd'' to a
''require => File...'' in order to make it more explicit. 
This is what exposed the real problem to me:

Feb 11 17:18:40 puppet-bsd2 puppet-agent[68963]: Could not run Puppet
configuration client: Could not find dependency File[/opt/home/2/dev/] for
Exec[create-device-/opt/home/2/dev/null] at
/usr/local/etc/puppet/production/modules/devices/manifests/init.pp:12

How do other people deal with these sorts of dependencies, where the files and
directories being created have variable paths based on the arguments passed to a
definition?  Is there a better way to get where I''m trying to go with
this?

Any clue is highly appreciated.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Ashley Gould

2011-Feb-13 06:38 UTC

head link

Re: [Puppet Users] Making dependencies work with variable resource names

On Fri, Feb 11, 2011 at 12:42:58PM -0500, Matthew Pounsett
wrote:> I''m having an issue solving dependencies inside defines, where the
paths to various resources are variable.  It seems like puppet isn''t
expanding all of the variables when it constructs the catalog, so it''s
unable to find the resources necessary to build things in the right order.
> 
> I''ve constructed a simple proof of concept to demonstrate the
problem.  I''m hoping someone can provide some advice on how this case
should be handled, because clearly I''ve misunderstood how puppet is
intended to be used to handle this sort of case.
> 
> Here is my site.pp, with a single machine loading up the module that builds
a standard service.  Each instance of the service has its own homedir with its
own data, logs, etc.
> 
> node "puppet-bsd2.virtual" {
>     include service
>     service::app {
>         "first":
>             service_num =>  "1";
>         "second":
>             service_num =>  "2";
>     }
> }
> 
> Here''s the init.pp manifest for the ''service''
module.  It includes the ''devices'' module which is used for
creating device files in a chroot environment, and sets up the homedir for each
instance of the service.
> 
> include devices
> class service {
>     file {
>         "/opt":
>             ensure => directory,
>             owner => root,
>             group => wheel,
>             mode => 755;
>         "/opt/home":
>             ensure => directory,
>             owner => root,
>             group => wheel,
>             mode => 755;
>     }
>     define app (
>         $homedir = "/opt/home",
>         $service_num
>     ) {
>         file {
>             "$homedir/${service_num}":
>                 ensure  =>  directory,
>                 owner   =>  root,
>                 group   =>  wheel,
>                 mode    =>  0750;
>             "$homedir/${service_num}/dev":
>                 ensure  =>  directory,
>                 owner   =>  root,
>                 group   =>  wheel,
>                 mode    =>  0750;
>         }
>         devices::device_node {
>             "${homedir}/${service_num}/dev/null":
>                 dir     => "$homedir/${service_num}/dev/";
>             "${homedir}/${service_num}/dev/random":
>                 dir     => "$homedir/${service_num}/dev/";
>         }
>     }
> }
> 
> And finally, the devices module.  
> 
> class devices {
>     define device_node (
>         $dir
>     ) {
>         exec {
>             "create-device-${name}":
>                 creates => "${name}",
>                 cwd => ${dir},
>                 command => "/usr/bin/touch ${name}",
>         }
>     }
> }
> 
> 
> The problem is one of order of operations.  With the above manifests,
puppet always tries to write the device files before it creates the
''var'' directory that contains them.  According to the
''require'' documentation, ''cwd'' inside an
exec clause should auto-require the directory referenced.  This wasn''t
working, and so I tried to change the ''cwd'' to a
''require => File...'' in order to make it more explicit. 
This is what exposed the real problem to me:
> 
> Feb 11 17:18:40 puppet-bsd2 puppet-agent[68963]: Could not run Puppet
configuration client: Could not find dependency File[/opt/home/2/dev/] for
Exec[create-device-/opt/home/2/dev/null] at
/usr/local/etc/puppet/production/modules/devices/manifests/init.pp:12
> 
> How do other people deal with these sorts of dependencies, where the files
and directories being created have variable paths based on the arguments passed
to a definition?  Is there a better way to get where I''m trying to go
with this?
> 
> Any clue is highly appreciated.
> 
caution - newbie replying

what if you create the device directories within
devices::device_node instead of service::app?  somehow you have to
parse out the path components of $dir and ensure them one by one.
Why is it puppet does not have a nice trick for "mkdir -p"?







> -- 
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
-- 

-ashley

Did you try poking at it with a stick?

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Nigel Kersten

2011-Feb-13 17:47 UTC

head link

Re: [Puppet Users] Making dependencies work with variable resource names

On Sat, Feb 12, 2011 at 10:38 PM, Ashley Gould <agould@ucop.edu> wrote:
> somehow you have to
> parse out the path components of $dir and ensure them one by one.
> Why is it puppet does not have a nice trick for "mkdir -p"?
>
>Because it''s really not clear how it should best behave with a bunch of
cases.

This works, and is sufficient for most people:

$my_dirs = [ "/foo", "/foo/bar", "/foo/bar/spam",
"/foo/bar/spam/eggs" ]

file {  $my_dirs:
  ensure => directory,
}

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Felix Frank

2011-Feb-16 08:44 UTC

head link

Re: [Puppet Users] Making dependencies work with variable resource names

Hi,

here''s what I''d do:
> include devices
> class service {
>     file {
>         "/opt":
>             ensure => directory,
>             owner => root,
>             group => wheel,
>             mode => 755;
>         "/opt/home":
>             ensure => directory,
>             owner => root,
>             group => wheel,
>             mode => 755;
>     }
>     define app (
>         $homedir = "/opt/home",
>         $service_num
>     ) {
>         file {
>             "$homedir/${service_num}":
>                 ensure  =>  directory,
>                 owner   =>  root,
>                 group   =>  wheel,
>                 mode    =>  0750;
>             "$homedir/${service_num}/dev":
>                 ensure  =>  directory,
>                 owner   =>  root,
>                 group   =>  wheel,
>                 mode    =>  0750;
>         }
          Devices::Device_node {
              dir     => "$homedir/${service_num}/dev/",
              require => File["$homedir/${service_num}/dev/"],
          }
>         devices::device_node {
>             "${homedir}/${service_num}/dev/null": ;
>             "${homedir}/${service_num}/dev/random": ;
>         }
>     }
> }
Although why your original approach didn''t work is not quite fathomable
to me. If the error persistes, try and make a simplified version of your
manifests that reproduces the same problem. Often, this will make a
mistake obvious (when it stops to reproduce after all).

If the problem persists, use the simplified manifest in a bug report.

HTH,
Felix

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

jcbollinger

2011-Feb-16 14:20 UTC

head link

[Puppet Users] Re: Making dependencies work with variable resource names

On Feb 16, 2:44 am, Felix Frank <felix.fr...@alumni.tu-berlin.de>
wrote:> here''s what I''d do:
>
> > include devices
> > class service {
> >     file {
> >         "/opt":
> >             ensure => directory,
> >             owner => root,
> >             group => wheel,
> >             mode => 755;
> >         "/opt/home":
> >             ensure => directory,
> >             owner => root,
> >             group => wheel,
> >             mode => 755;
> >     }
> >     define app (
> >         $homedir = "/opt/home",
> >         $service_num
> >     ) {
> >         file {
> >             "$homedir/${service_num}":
> >                 ensure  =>  directory,
> >                 owner   =>  root,
> >                 group   =>  wheel,
> >                 mode    =>  0750;
> >             "$homedir/${service_num}/dev":
> >                 ensure  =>  directory,
> >                 owner   =>  root,
> >                 group   =>  wheel,
> >                 mode    =>  0750;
> >         }
>
>           Devices::Device_node {
>               dir     => "$homedir/${service_num}/dev/",
>               require => File["$homedir/${service_num}/dev/"],
>           }
>
> >         devices::device_node {
> >             "${homedir}/${service_num}/dev/null": ;
> >             "${homedir}/${service_num}/dev/random": ;
> >         }
> >     }
> > }
> Although why your original approach didn''t work is not quite
fathomable
> to me. If the error persistes, try and make a simplified version of your
> manifests that reproduces the same problem. Often, this will make a
> mistake obvious (when it stops to reproduce after all).
In general, if Puppet attempts to apply resources in a sequence that
doesn''t work, then it means that there are resource relationships that
are not implicit and have not been declared.  That''s exactly the case
in the OP''s manifest: there is nothing in it that would create an
order relation between devices::device_node instances and the
directories named in their $dir parameters.  Exec resources are not
documented to (and evidently do not) autorequire any part of the path
to their ''cwd'' or ''creates'' parameters.  The
sequence of declarations
in the manifest is irrelevant.

Adding an appropriate requires => to Devices::device_node
instantiations, as suggested, should do the trick.  Alternatively, the
requires => could be put on one of the resources inside the defined
type (in this case, the only candidate is the Exec).  Personally, I
prefer the latter whenever it is feasible, which in this case it is.


Cheers,

John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Felix Frank

2011-Feb-16 15:56 UTC

head link

Re: [Puppet Users] Re: Making dependencies work with variable resource names

> Adding an appropriate requires => to Devices::device_node
> instantiations, as suggested, should do the trick.  Alternatively, the
> requires => could be put on one of the resources inside the defined
> type (in this case, the only candidate is the Exec).  Personally, I
> prefer the latter whenever it is feasible, which in this case it is.
Ah, but the OP tried the latter and received an error. Again, I don''t
see why that is (possibly the manifest was censored in an unbecoming
fashion).

Cheers,
Felix

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Matthew Pounsett

2011-Feb-16 17:50 UTC

head link

Re: [Puppet Users] Making dependencies work with variable resource names

On 2011/02/16, at 03:44, Felix Frank wrote:
> Hi,
> 
> here''s what I''d do:
> 
> 
>          Devices::Device_node {
>              dir     => "$homedir/${service_num}/dev/",
>              require => File["$homedir/${service_num}/dev/"],
>          }
I tried that as well (noted in the original post).  That''s what
generated the error that I included.

Feb 11 17:18:40 puppet-bsd2 puppet-agent[68963]: Could not run Puppet
configuration client: Could not find dependency File[/opt/home/2/dev/] for
Exec[create-device-/opt/home/2/dev/null] at
/usr/local/etc/puppet/production/modules/devices/manifests/init.pp:12

It seems that the problem is that Puppet is expanding the variable in the
''require'' but has not yet expanded the variable in the file{}
block, and so can''t find the file resource to create the dependency
tree.
> Although why your original approach didn''t work is not quite
fathomable
> to me. If the error persistes, try and make a simplified version of your
> manifests that reproduces the same problem. Often, this will make a
> mistake obvious (when it stops to reproduce after all).
This is exactly what I posted.  It''s my actual manifests stripped down
to just the basics required to demonstrate the problem.
> If the problem persists, use the simplified manifest in a bug report.
Okay.  Now that I know this is supposed to work, I''ll go ahead and file
a bug report.  Thanks.



-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Matthew Pounsett

2011-Feb-16 17:53 UTC

head link

Re: [Puppet Users] Re: Making dependencies work with variable resource names

On 2011/02/16, at 09:20, jcbollinger wrote:
> In general, if Puppet attempts to apply resources in a sequence that
> doesn''t work, then it means that there are resource relationships
that
> are not implicit and have not been declared.  That''s exactly the
case
> in the OP''s manifest: there is nothing in it that would create an
> order relation between devices::device_node instances and the
> directories named in their $dir parameters.  Exec resources are not
> documented to (and evidently do not) autorequire any part of the path
> to their ''cwd'' or ''creates'' parameters.
The sequence of declarations
> in the manifest is irrelevant.
Actually, yes they are documented to do that.  From the documentation for
''require''[1]:

     Currently, exec resources will autorequire their CWD (if it is specified)
     plus any fully qualified paths that appear in the command. For instance, 
     if you had an exec command that ran the myscript mentioned above, the 
     above code that pulls the file down would be automatically listed as a 
     requirement to the exec code, so that you would always be running againts 
     the most recent version.

And, as noted in my original post, I did try replacing the
''cwd'' with a ''require'', and tried moving the
''require'' to the call to devices::device_node rather than in
its definition.  Neither alternative worked.


-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Matthew Pounsett

2011-Feb-16 17:56 UTC

head link

Re: [Puppet Users] Re: Making dependencies work with variable resource names

On 2011/02/16, at 10:56, Felix Frank wrote:
>> Adding an appropriate requires => to Devices::device_node
>> instantiations, as suggested, should do the trick.  Alternatively, the
>> requires => could be put on one of the resources inside the defined
>> type (in this case, the only candidate is the Exec).  Personally, I
>> prefer the latter whenever it is feasible, which in this case it is.
> 
> Ah, but the OP tried the latter and received an error. Again, I
don''t
> see why that is (possibly the manifest was censored in an unbecoming
> fashion).
Nothing there was censored.  As the original post noted, that is a stripped down
but functioning proof of concept based on my original manifests.


-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Matthew Pounsett

2011-Feb-16 18:41 UTC

head link

Re: [Puppet Users] Re: Making dependencies work with variable resource names

On 2011/02/16, at 12:53, Matthew Pounsett wrote:
> Actually, yes they are documented to do that.  From the documentation for
''require''[1]:
Hrm.. I meant to include the URL there:

<http://docs.puppetlabs.com/references/stable/metaparameter.html>

I''ll also note that I''m not too surprised you didn''t
know that was there.  In trying to get myself up to speed I''ve noticed
a major failing in the Puppet documentation in that it is very badly organized. 
It''s not unusual for the important details of a particular behaviour to
be spread around different parts of the documentation, or for it to be very
difficult for a new user to even find the main reference to a particular piece
of information.



-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

jcbollinger

2011-Feb-17 14:04 UTC

head link

[Puppet Users] Re: Making dependencies work with variable resource names

On Feb 16, 11:50 am, Matthew Pounsett <m...@conundrum.com>
wrote:> On 2011/02/16, at 03:44, Felix Frank wrote:
[...]> I tried that as well (noted in the original post).  That''s what
generated the error that I included.
>
> Feb 11 17:18:40 puppet-bsd2 puppet-agent[68963]: Could not run Puppet
configuration client: Could not find dependency File[/opt/home/2/dev/] for
Exec[create-device-/opt/home/2/dev/null] at
/usr/local/etc/puppet/production/modules/devices/manifests/init.pp:12
>
> It seems that the problem is that Puppet is expanding the variable in the
''require'' but has not yet expanded the variable in the file{}
block, and so can''t find the file resource to create the dependency
tree.
>
> > Although why your original approach didn''t work is not quite
fathomable
> > to me. If the error persistes, try and make a simplified version of
your
> > manifests that reproduces the same problem. Often, this will make a
> > mistake obvious (when it stops to reproduce after all).
>
> This is exactly what I posted.  It''s my actual manifests stripped
down to just the basics required to demonstrate the problem.
Looking at your manifest more closely, I see that the File resources
you are managing are named without a trailing slash, for example
"$homedir/${service_num}/dev", but the directory parameters you are
passing to your define are specified *with* a trailing slash, i.e.
"$homedir/${service_num}/dev/".  The error message shows that the
missing resource is File[/opt/home/2/dev/] (with a trailing slash),
which indeed you have not declared.  Although the two forms are
generally handed equivalently by the shell, I can fully believe that
Puppet does not consider it safe to treat them equivalently.

Try removing the trailing slashes from your directory names.  I think
that will solve your problem.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Puppet users - Feb 2011 - Making dependencies work with variable resource names

[Puppet Users] Making dependencies work with variable resource names

Re: [Puppet Users] Making dependencies work with variable resource names

Re: [Puppet Users] Making dependencies work with variable resource names

Re: [Puppet Users] Making dependencies work with variable resource names

[Puppet Users] Re: Making dependencies work with variable resource names

Re: [Puppet Users] Re: Making dependencies work with variable resource names

Re: [Puppet Users] Making dependencies work with variable resource names

Re: [Puppet Users] Re: Making dependencies work with variable resource names

Re: [Puppet Users] Re: Making dependencies work with variable resource names

Re: [Puppet Users] Re: Making dependencies work with variable resource names

[Puppet Users] Re: Making dependencies work with variable resource names