Puppet users - Apr 2013 - defined function notparse order dependant

I have been backed into a corner, by they way puppet works, but some third 
party module.

Basically I have two resources defined:

a {$somevar:}
b::b {$somrvar:}

both have code that looks something like this:
    if (!defined(File[$name])) {
        file { $name:
...
    }

According to the documentation ''defined'' is dependent on parse
order.  So
resource definition ''a'' should be parsed first and win.  But
it doesn''t, no
matter how I order ''a'' and ''b::b'', resource
definition ''b::b'' always
defines the file.  I even tried puppet version 3.1.1.  Same problem.  My 
problem is that resource "b::b" does it horribly wrong.  It causes the
runtime of my manifest to balloon out from 2mins to over 10 minutes.  It 
uses the wrong group.  That would be okay if I could just put
''a'' first and
have ''b::b'' go silent.

Does anyone know why this is happening?  How is "b::b" being parsed
first
even though "a" is ahead in the file?

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

If the relevant code were just sitting there naked in your site manifest, I 
think you''d probably see a fairly simple parse-order dependency -- I
think
it''s the fact that they''re in defined types that''s
shifting things around.
Actually, one of the core team surprised me the other week by telling me 
that defined types are somehow late-binding when creating their resources, 
in a way that classes aren''t; I can''t remember why they
thought it had been
implemented that way, though.

The point is, this is EXACTLY why we say to not use `defined()` like that, 
is because it can cause havoc for downstream users like you. I''m afraid
you''re going to have to fork b::b and go in and muck with its 
implementation if you want any certainty around this behavior. 

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

The difference between ''a'' and ''b::b'' was
that ''a'' didn''t have a file
declaration, it had a resource declaration that had a file declaration.  So 
if I created a new resource that wrapped ''b::b'' then I got a
straight
forward parse-order dependency.  

I tend to use defined because it allows you to under gird something without 
that something actually knowing who you are.  In this case the plugin was 
git.  I was using the repo resource to checkout some code.  It was 
declaring the folder that was going to be checked out.  Defined works nice 
because when you read it, you understand what they are trying to do, even 
if puppet has a hard time producing that result.  Alternatives I have read 
involve defining a class and including it.  But that requires git to know 
about said class, also it doesn''t work for resources which applies in
this
case.  The other alternative I have read about is something like 
virtualized resources?  You manifest them at each point.  I have not really 
seen any good documentation on what it is, how it works, what it requires.  
The code I have seen does not make it clear to the user what is taking 
pace, it just doesn''t read well.    

If you declare things something like this psuedo code:
A
   C
      F1
B
   D
      F2

When people read it they are going to more commonly expect the order to go
A
C
F1
B
D
F2

Not
A
B
C
D
F1
F2


In git they set recurse to true.  Compound that with a resource default up 
the declaration stack and you have puppet spending a long time uselessly 
changing the ownership and permission on a ton a files when they get 
checked out.  I find the dynamic scope of resource defaults to be more of a 
surprise and a burden then a useful tool.  

Anyway I am resolved, thanks for the help.  

On Tuesday, April 23, 2013 4:09:02 PM UTC-7, Nick Fagerlund
wrote:
>
> If the relevant code were just sitting there naked in your site manifest, 
> I think you''d probably see a fairly simple parse-order dependency
-- I
> think it''s the fact that they''re in defined types
that''s shifting things
> around. Actually, one of the core team surprised me the other week by 
> telling me that defined types are somehow late-binding when creating their 
> resources, in a way that classes aren''t; I can''t remember
why they thought
> it had been implemented that way, though.
>
> The point is, this is EXACTLY why we say to not use `defined()` like that, 
> is because it can cause havoc for downstream users like you. I''m
afraid
> you''re going to have to fork b::b and go in and muck with its 
> implementation if you want any certainty around this behavior. 
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On 24 April 2013 06:18, Mike Power <dodtsair@gmail.com> wrote:
>
> In git they set recurse to true.  Compound that with a resource default up
> the declaration stack and you have puppet spending a long time uselessly
> changing the ownership and permission on a ton a files when they get
> checked out.  I find the dynamic scope of resource defaults to be more of a
> surprise and a burden then a useful tool.
>
>
A bit of a thread hijack, but I really agree about the dynamic scope for
resource defaults being just as (if not even more) annoying and
unpredictable than the dynamic scoping for variables was.

-- 
Erik Dalén

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On Tuesday, April 23, 2013 4:52:11 PM UTC-5, Mike Power
wrote:
>
> I have been backed into a corner, by they way puppet works, but some third 
> party module.
>
> Basically I have two resources defined:
>
> a {$somevar:}
> b::b {$somrvar:}
>
> both have code that looks something like this:
>     if (!defined(File[$name])) {
>         file { $name:
> ...
>     }
>
> According to the documentation ''defined'' is dependent on
parse order.  So
> resource definition ''a'' should be parsed first and win.


This is not a characteristic to manipulate, but rather a strong reason to 
avoid defined() altogether.  I cannot say this more strongly: DO NOT USE IT.

 
>   But it doesn''t, no matter how I order ''a'' and
''b::b'', resource
> definition ''b::b'' always defines the file.  I even tried
puppet version
> 3.1.1.  Same problem.  My problem is that resource "b::b" does it
horribly
> wrong.  It causes the runtime of my manifest to balloon out from 2mins to 
> over 10 minutes.  It uses the wrong group.  That would be okay if I could 
> just put ''a'' first and have ''b::b'' go
silent.
>
> Does anyone know why this is happening?  How is "b::b" being
parsed first
> even though "a" is ahead in the file?
>

The *declaration* of resource A[$somevar] is surely parsed before the 
declaration of B::B[$somevar], but that''s not at all the same thing as
the
body of defined type ''a'' being parsed before the body of
defined type
''b::b''.  It''s pretty much irrelevant why the parse
order you see happens --
Puppet guarantees parse order only within each manifest file, so manifests 
that depend on parse order across multiple files are flawed.  Even if you 
found a magic formulation that achieved the order you want on a given 
version of Puppet, there is no guarantee that it would continue to work 
even in the next maintenance release.

As I say here from time to time, your manifests should not attempt to query 
Puppet about what has already been declared.  There are a number of 
techniques by which that is possible, but generally they can be described 
as using "data-driven" approaches to manifest and manifest-set design.
As
an over-simplified example, you could set a boolean node variable 
$i_will_use_a, and then change the condition in b::b like so

define b::b (...) {
  if ! $i_will_use_a {
    file { $name:
      ...
    }
  }
  ...
}

There are a host of other ways to do essentially the same thing.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

I appreciate the reply but I think the thrust of my point is getting missed.

Any other solution is far more cryptic, some reader coming and reading the 
code will not understand the code nearly as well.  

The fact that puppet has a dimension (parse order) that is so convoluted 
that its behavior cannot be understood is not a virtue.  It is a very bad 
thing.  

On Wednesday, April 24, 2013 6:19:13 AM UTC-7, jcbollinger
wrote:
>
>
>
> On Tuesday, April 23, 2013 4:52:11 PM UTC-5, Mike Power wrote:
>>
>> I have been backed into a corner, by they way puppet works, but some 
>> third party module.
>>
>> Basically I have two resources defined:
>>
>> a {$somevar:}
>> b::b {$somrvar:}
>>
>> both have code that looks something like this:
>>     if (!defined(File[$name])) {
>>         file { $name:
>> ...
>>     }
>>
>> According to the documentation ''defined'' is dependent
on parse order.  So
>> resource definition ''a'' should be parsed first and
win.
>
>
>
> This is not a characteristic to manipulate, but rather a strong reason to 
> avoid defined() altogether.  I cannot say this more strongly: DO NOT USE
IT.
>
>  
>
>>   But it doesn''t, no matter how I order ''a''
and ''b::b'', resource
>> definition ''b::b'' always defines the file.  I even
tried puppet version
>> 3.1.1.  Same problem.  My problem is that resource "b::b"
does it horribly
>> wrong.  It causes the runtime of my manifest to balloon out from 2mins
to
>> over 10 minutes.  It uses the wrong group.  That would be okay if I
could
>> just put ''a'' first and have ''b::b''
go silent.
>>
>> Does anyone know why this is happening?  How is "b::b" being
parsed first
>> even though "a" is ahead in the file?
>>
>
>
> The *declaration* of resource A[$somevar] is surely parsed before the 
> declaration of B::B[$somevar], but that''s not at all the same
thing as the
> body of defined type ''a'' being parsed before the body of
defined type
> ''b::b''.  It''s pretty much irrelevant why the
parse order you see happens --
> Puppet guarantees parse order only within each manifest file, so manifests 
> that depend on parse order across multiple files are flawed.  Even if you 
> found a magic formulation that achieved the order you want on a given 
> version of Puppet, there is no guarantee that it would continue to work 
> even in the next maintenance release.
>
> As I say here from time to time, your manifests should not attempt to 
> query Puppet about what has already been declared.  There are a number of 
> techniques by which that is possible, but generally they can be described 
> as using "data-driven" approaches to manifest and manifest-set
design.  As
> an over-simplified example, you could set a boolean node variable 
> $i_will_use_a, and then change the condition in b::b like so
>
> define b::b (...) {
>   if ! $i_will_use_a {
>     file { $name:
>       ...
>     }
>   }
>   ...
> }
>
> There are a host of other ways to do essentially the same thing.
>
>
> John
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On Wednesday, April 24, 2013 9:06:02 AM UTC-5, Mike Power
wrote:
>
> I appreciate the reply but I think the thrust of my point is getting 
> missed.
>
> Any other solution is far more cryptic, some reader coming and reading the 
> code will not understand the code nearly as well.  
>

That''s a matter of opinion and style, of course.  And surely you 
over-generalize, even for people who are inclined to prefer the same style 
you apparently do.  You cannot possibly expect anyone to accept that using 
defined() as you describe is the clearest, easiest to understand (for 
anyone) solution to every single problem that can be cast in the very 
general form you presented.

Personally, I find usage of defined() to be very obfuscatory.  The local 
meaning is clear enough, but the overall contribution and implication is 
not clear at all. Under what circumstances will the given resource be 
already defined?  That''s especially important if the defined()
condition
affects more (or different) declarations than just the one resource it 
tests.  And I''m not talking about parse order here; even an easily 
predictable parse / evaluation order would not solve that -- it''s a 
question of scope.

Moreover, there are correctness and maintainability problems.  If you have 
two classes that may or may not both be used on the same node, and both 
rely on the same resource, then having each one declare that resource under 
protection of a defined() condition forces you to continually ensure that 
each declaration of that resource is consistent with the needs of *all* the 
classes that (may) rely on it.

> The fact that puppet has a dimension (parse order) that is so convoluted 
> that its behavior cannot be understood is not a virtue.  It is a very bad 
> thing.  
>
>
No one claimed that the behavior is virtuous.  That''s all irrelevant, 
however, if you follow my advice to avoid parse-order dependencies in the 
first place.  That''s widely accepted as a best practice.  Leading up to
Puppet 3, there was even an initiative to remove the defined() function 
from Puppet, or at least deprecate it, though that didn''t ultimately
make
it into (out of) the product.  Puppet has few other areas of parse-order 
dependency, and most that it does have can be handled very simply.

Inasmuch as you conclude that difficulty in predicting Puppet parse order 
is a serious flaw, I would think you''d agree that parse-order
dependencies
are better avoided.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.