Puppet users - Oct 2009 - template scope.compiler.classlist.include problems/flapping

Good Afternoon, all,

I believe I posted something on this in the past, but didn''t get many
replies.

I''m trying to manage some slightly different configs for different
classes of hosts within templates using
scope.compiler.classlist.include. Specifically, for example, all of our
hosts have the same httpd.conf file, with the exception of two hosts
(each of which have a unique class defined in them).

Something like:
<% if scope.compiler.classlist.include?("fooClass") then %>
foo
bar
baz
<% end %>

I originally tried this with both the template for httpd.conf and the
template for /etc/sysconfig/iptables.

On all of the hosts that use these methods, I''m seeing some strange
behavior where intermittently (sometimes as much as 50/50, exactly 1 on
1 off) the if statement does not evaluate, and the segment of code
within it doesn''t get applied. I''ve looked through the logs to
no end -
all I can see is that these files flap back and forth endlessly,
removing the generated file, replacing it with a new one, the replacing
it with the original (the MD5 sums also reveal the A-B-A-B pattern).

I''m using an external node classifier script, but have been dumping
its''
output to a timestamped log file, and the output is always correct (the
classes in question always appear in the YAML).

Any ideas?

I''m aware of the pervasive theory of using fragment-based file
creation,
but have a few issues:
1) I very much want to keep all code related to a certain file in one
place (i.e. the iptables module should test for presence of each service
and then create the iptables file the way it wants to, not including
iptables information in other modules). I''d prefer not to scatter code
related to one concept (iptables, httpd) in different modules.
2) For some of these files, order is *very* important (we run a ...
complex... iptables ruleset) and, if this code were split across
multiple modules, it would require reading through endless files to
figure out where lines X, Y and Z should go.

Has anyone else experienced any issues like this?

Thanks,
Jason Antman
Rutgers University

PS - All boxes are CentOS 5.3 x86_64 running Puppet 0.24.8 (0.24.8-4.el5).

-- 

Linux: The smack in the face that Windows gripers have been begging for these
past 10 years...

--

Jason Antman
www.jasonantman.com
jason@jasonantman.com
Cell: (201)906-7347

Systems Programmer
Rutgers University
OIT Central Systems & Services / NetOps
jantman@ccf.rutgers.edu



--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

That''s a weird one.

I''m assuming the other resources that are created for the fooClass get
created?

Possibly an issue with the scope lookup.

Is there an open issue in redmine?




On Wed, Oct 14, 2009 at 12:43 PM, Jason Antman
<jason@jasonantman.com>wrote:
>
> Good Afternoon, all,
>
> I believe I posted something on this in the past, but didn''t get
many
> replies.
>
> I''m trying to manage some slightly different configs for different
> classes of hosts within templates using
> scope.compiler.classlist.include. Specifically, for example, all of our
> hosts have the same httpd.conf file, with the exception of two hosts
> (each of which have a unique class defined in them).
>
> Something like:
> <% if scope.compiler.classlist.include?("fooClass") then %>
> foo
> bar
> baz
> <% end %>
>
> I originally tried this with both the template for httpd.conf and the
> template for /etc/sysconfig/iptables.
>
> On all of the hosts that use these methods, I''m seeing some
strange
> behavior where intermittently (sometimes as much as 50/50, exactly 1 on
> 1 off) the if statement does not evaluate, and the segment of code
> within it doesn''t get applied. I''ve looked through the
logs to no end -
> all I can see is that these files flap back and forth endlessly,
> removing the generated file, replacing it with a new one, the replacing
> it with the original (the MD5 sums also reveal the A-B-A-B pattern).
>
> I''m using an external node classifier script, but have been
dumping its''
> output to a timestamped log file, and the output is always correct (the
> classes in question always appear in the YAML).
>
> Any ideas?
>
> I''m aware of the pervasive theory of using fragment-based file
creation,
> but have a few issues:
> 1) I very much want to keep all code related to a certain file in one
> place (i.e. the iptables module should test for presence of each service
> and then create the iptables file the way it wants to, not including
> iptables information in other modules). I''d prefer not to scatter
code
> related to one concept (iptables, httpd) in different modules.
> 2) For some of these files, order is *very* important (we run a ...
> complex... iptables ruleset) and, if this code were split across
> multiple modules, it would require reading through endless files to
> figure out where lines X, Y and Z should go.
>
> Has anyone else experienced any issues like this?
>
> Thanks,
> Jason Antman
> Rutgers University
>
> PS - All boxes are CentOS 5.3 x86_64 running Puppet 0.24.8 (0.24.8-4.el5).
>
> --
>
> Linux: The smack in the face that Windows gripers have been begging for
> these past 10 years...
>
> --
>
> Jason Antman
> www.jasonantman.com
> jason@jasonantman.com
> Cell: (201)906-7347
>
> Systems Programmer
> Rutgers University
> OIT Central Systems & Services / NetOps
> jantman@ccf.rutgers.edu
>
>
>
> >
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Yes, the other resources get created fine.

As far as I can find, no. I didn''t want to open an issue until I got
some feedback from the community on whether or not anyone else has seen
this behavior (no need to open an issue for something that someone knows
was fixed).

If I don''t get any more responses in the next few days, I''ll
open a bug
report. I''ve seen this reproduced on all of my hosts (to the point of
having to disable puppetd on some of them, since it was alternately
applying correct and incorrect configs), all running 0.24.8.

-Jason

Andrew Shafer wrote:
>
> That''s a weird one.
>
> I''m assuming the other resources that are created for the fooClass
get
> created?
>
> Possibly an issue with the scope lookup.
>
> Is there an open issue in redmine?
>
>
>
>
> On Wed, Oct 14, 2009 at 12:43 PM, Jason Antman <jason@jasonantman.com
> <mailto:jason@jasonantman.com>> wrote:
>
>
>     Good Afternoon, all,
>
>     I believe I posted something on this in the past, but didn''t
get many
>     replies.
>
>     I''m trying to manage some slightly different configs for
different
>     classes of hosts within templates using
>     scope.compiler.classlist.include. Specifically, for example, all
>     of our
>     hosts have the same httpd.conf file, with the exception of two hosts
>     (each of which have a unique class defined in them).
>
>     Something like:
>     <% if scope.compiler.classlist.include?("fooClass") then
%>
>     foo
>     bar
>     baz
>     <% end %>
>
>     I originally tried this with both the template for httpd.conf and the
>     template for /etc/sysconfig/iptables.
>
>     On all of the hosts that use these methods, I''m seeing some
strange
>     behavior where intermittently (sometimes as much as 50/50, exactly
>     1 on
>     1 off) the if statement does not evaluate, and the segment of code
>     within it doesn''t get applied. I''ve looked through
the logs to no
>     end -
>     all I can see is that these files flap back and forth endlessly,
>     removing the generated file, replacing it with a new one, the
>     replacing
>     it with the original (the MD5 sums also reveal the A-B-A-B pattern).
>
>     I''m using an external node classifier script, but have been
>     dumping its''
>     output to a timestamped log file, and the output is always correct
>     (the
>     classes in question always appear in the YAML).
>
>     Any ideas?
>
>     I''m aware of the pervasive theory of using fragment-based file
>     creation,
>     but have a few issues:
>     1) I very much want to keep all code related to a certain file in one
>     place (i.e. the iptables module should test for presence of each
>     service
>     and then create the iptables file the way it wants to, not including
>     iptables information in other modules). I''d prefer not to
scatter code
>     related to one concept (iptables, httpd) in different modules.
>     2) For some of these files, order is *very* important (we run a ...
>     complex... iptables ruleset) and, if this code were split across
>     multiple modules, it would require reading through endless files to
>     figure out where lines X, Y and Z should go.
>
>     Has anyone else experienced any issues like this?
>
>     Thanks,
>     Jason Antman
>     Rutgers University
>
>     PS - All boxes are CentOS 5.3 x86_64 running Puppet 0.24.8
>     (0.24.8-4.el5).
>
>     --
>
>     Linux: The smack in the face that Windows gripers have been
>     begging for these past 10 years...
>
>     --
>
>     Jason Antman
>     www.jasonantman.com <http://www.jasonantman.com>
>     jason@jasonantman.com <mailto:jason@jasonantman.com>
>     Cell: (201)906-7347
>
>     Systems Programmer
>     Rutgers University
>     OIT Central Systems & Services / NetOps
>     jantman@ccf.rutgers.edu <mailto:jantman@ccf.rutgers.edu>
>
>
>
>
>
>
> >
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Oct 14, 2009, at 11:43 AM, Jason Antman wrote:
>
> Good Afternoon, all,
>
> I believe I posted something on this in the past, but didn''t get
many
> replies.
>
> I''m trying to manage some slightly different configs for different
> classes of hosts within templates using
> scope.compiler.classlist.include. Specifically, for example, all of  
> our
> hosts have the same httpd.conf file, with the exception of two hosts
> (each of which have a unique class defined in them).
>
> Something like:
> <% if scope.compiler.classlist.include?("fooClass") then %>
> foo
> bar
> baz
> <% end %>
>
> I originally tried this with both the template for httpd.conf and the
> template for /etc/sysconfig/iptables.
>
> On all of the hosts that use these methods, I''m seeing some
strange
> behavior where intermittently (sometimes as much as 50/50, exactly 1  
> on
> 1 off) the if statement does not evaluate, and the segment of code
> within it doesn''t get applied. I''ve looked through the
logs to no
> end -
> all I can see is that these files flap back and forth endlessly,
> removing the generated file, replacing it with a new one, the  
> replacing
> it with the original (the MD5 sums also reveal the A-B-A-B pattern).
>
> I''m using an external node classifier script, but have been
dumping
> its''
> output to a timestamped log file, and the output is always correct  
> (the
> classes in question always appear in the YAML).
>
> Any ideas?

To clarify - you''re looking for a class generated by your external  
node tool, and sometimes it shows up in this list and sometimes
doesn''t?

My guess, though, is that this is an ordering issue - the classlist is  
built as classes are evaluated, and the evaluation order can change  
somewhat arbitrarily.

I think we fixed a lot of this in 0.25, but really, I still don''t  
think it''s as deterministic as it could be.

At the least, I think it''s a bug that the entire class list from the  
external node isn''t put into the classlist variable immediately, and  
I''ve just confirmed that it isn''t - that would provide you
isolation
from any ordering issues for at least those classes that come from the  
external node.

Filed and fixed, to be included in 0.25.1.

-- 
If you can''t be a good example, then you''ll just have to be a
horrible warning. -- Catherine Aird
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Luke Kanies wrote:
> To clarify - you''re looking for a class generated by your external
> node tool, and sometimes it shows up in this list and sometimes
doesn''t?
>
> My guess, though, is that this is an ordering issue - the classlist is  
> built as classes are evaluated, and the evaluation order can change  
> somewhat arbitrarily.
>
> I think we fixed a lot of this in 0.25, but really, I still don''t
> think it''s as deterministic as it could be.
>
> At the least, I think it''s a bug that the entire class list from
the
> external node isn''t put into the classlist variable immediately,
and
> I''ve just confirmed that it isn''t - that would provide
you isolation
> from any ordering issues for at least those classes that come from the  
> external node.
>
> Filed and fixed, to be included in 0.25.1.
>
>   
Luke,

That''s great! Thanks so much! Yes, that seems to make sense to me,
though somehow it totally slipped out of my mind that this could have
been caused by ordering.

I guess this is also a good time for me to think about building the RPMs
locally...

Thanks,
Jason Antman
Rutgers University, NJ, USA


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---