Puppet users - Oct 2012 - Puppet spuriously removing classes from classes.txt

I''ve been asked to look at a problem on an overseas rig whereby certain
bits of config were going awry. This was down to the fact that they were 
querying the mcollective registration database and feeding back in to the 
configs, using queries based on class data. The mcollective + mongodb setup 
is working fine, but it would appear that the classes.txt file which 
mcollective reads to get configuration management classes is, on certain 
servers, spuriously being emptied of all classes other than
''settings''

Seeing as the settings class is being left in, I can only assume that it 
must be puppet doing the modification. Has anyone ever seen behaviour like 
this before? It''s truly bizarre, and even stranger is that after this
has
happened, sometimes the classes populate again. Every time I''ve run
puppet
manually  with puppetd --test or run it --noop, it''s run just fine and 
without noop it always seems to fix the problem, but later the node will 
show up as having only one class again and even later fix itself, so
it''s
next to impossible to pin down.

This is 2.6.7 on Ubuntu running against a centralised puppetmaster.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/zi9NP9ysFkcJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi Matt,

On Fri, Oct 19, 2012 at 8:55 AM, Matt Carroll <oholiab@gmail.com>
wrote:
> I''ve been asked to look at a problem on an overseas rig whereby
certain bits
> of config were going awry. This was down to the fact that they were
querying
> the mcollective registration database and feeding back in to the configs,
> using queries based on class data. The mcollective + mongodb setup is
> working fine, but it would appear that the classes.txt file which
> mcollective reads to get configuration management classes is, on certain
> servers, spuriously being emptied of all classes other than
''settings''
>
> Seeing as the settings class is being left in, I can only assume that it
> must be puppet doing the modification. Has anyone ever seen behaviour like
> this before? It''s truly bizarre, and even stranger is that after
this has
> happened, sometimes the classes populate again. Every time I''ve
run puppet
> manually  with puppetd --test or run it --noop, it''s run just fine
and
> without noop it always seems to fix the problem, but later the node will
> show up as having only one class again and even later fix itself, so
it''s
> next to impossible to pin down.
>
> This is 2.6.7 on Ubuntu running against a centralised puppetmaster.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/puppet-users/-/zi9NP9ysFkcJ.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
There was this issue (http://projects.puppetlabs.com/issues/4518),
which was fixed in early 2.6. However, I think RI has said he''s seen
this more recently. You may want to ask on the mcollective-users list
if you don''t get a reply here.

Josh

-- 
Josh Cooper
Developer, Puppet Labs

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hmm, I''m not sure that this is the same issue... the issue details the
classes.txt only containing a newline wheras I''m experiencing a problem
whereby the file sometimes only contains the default class
''settings''. It appears as if the node is compiling spuriously
against a different environment with an empty default class every now and again.

I''m going to look in to the possibility Of upgrading the rig, as it may
fix the problem, and seeing as it''s quite an old version it''s
not really too necessary to know exactly what was causing the behaviour.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/3NhJKOAfRjQJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

----- Original Message -----
> From: "Matt Carroll" <oholiab@gmail.com>
> To: puppet-users@googlegroups.com
> Sent: Monday, October 22, 2012 10:43:44 AM
> Subject: Re: [Puppet Users] Puppet spuriously removing classes from
classes.txt
> 
> Hmm, I''m not sure that this is the same issue... the issue details
> the classes.txt only containing a newline wheras I''m experiencing
a
> problem whereby the file sometimes only contains the default class
> ''settings''. It appears as if the node is compiling
spuriously
> against a different environment with an empty default class every
> now and again.
> 
> I''m going to look in to the possibility Of upgrading the rig, as
it
> may fix the problem, and seeing as it''s quite an old version
it''s
> not really too necessary to know exactly what was causing the
> behaviour.
> 
depending on version it also writes resources.txt, so you can determine
if when classes.txt is empty if it really was not managing any resources.

Being someone who cares deeply for classes.txt, I''ve not observed this
behavior

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Friday, October 19, 2012 10:55:03 AM UTC-5, Matt Carroll
wrote:
>
> I''ve been asked to look at a problem on an overseas rig whereby
certain
> bits of config were going awry. This was down to the fact that they were 
> querying the mcollective registration database and feeding back in to the 
> configs, using queries based on class data. The mcollective + mongodb setup
> is working fine, but it would appear that the classes.txt file which 
> mcollective reads to get configuration management classes is, on certain 
> servers, spuriously being emptied of all classes other than
''settings''
>
> Seeing as the settings class is being left in, I can only assume that it 
> must be puppet doing the modification. Has anyone ever seen behaviour like 
> this before? It''s truly bizarre, and even stranger is that after
this has
> happened, sometimes the classes populate again. Every time I''ve
run puppet
> manually  with puppetd --test or run it --noop, it''s run just fine
and
> without noop it always seems to fix the problem, but later the node will 
> show up as having only one class again and even later fix itself, so
it''s
> next to impossible to pin down.
>
> This is 2.6.7 on Ubuntu running against a centralised puppetmaster.
>

Are you certain the behavior is buggy?  The fact that Puppet rights itself 
makes me think that Puppet itself is not the problem.

What if there is a cron job (or a human admin) that periodically does 
something like "puppetd --onetime --no-daemonize --tags settings"?  Or
what
if the affected node''s manifests use schedules in a way that
occasionally
excludes all classes except ''settings''?  The former, at least,
should show
up in the system log.

You might also get some insight by wrapping puppetd in a script that makes 
a timestamped backup of classes.txt after each run.  Done right, that ought 
to establish whether Puppet is the one writing the unexpected content, and 
if so then the timestamps when the bad files appear might tell you 
something useful.  Perhaps you could dredge the same information out of 
your system logs, but the timestamped backups might be easier to analyze 
(and they would pinpoint where in your logs to look for additional 
information).


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/1RlbAp6sTdMJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

I''m not sure whether it''s buggy behaviour or not, but by
comparing the
daemon.log and an inotify left running on the file I can confirm that
it''s
puppet that''s making the changes. Whether or not it''s someone
running
puppet against a stupid environment or not remains to be seen, but I still 
need to find out what is going on...

Like I say, this is not directly my infrastructure or I''d have a better
idea.

On Monday, 22 October 2012 14:41:16 UTC+1, jcbollinger
wrote:
>
>
>
> On Friday, October 19, 2012 10:55:03 AM UTC-5, Matt Carroll wrote:
>>
>> I''ve been asked to look at a problem on an overseas rig
whereby certain
>> bits of config were going awry. This was down to the fact that they
were
>> querying the mcollective registration database and feeding back in to
the
>> configs, using queries based on class data. The mcollective + mongodb
setup
>> is working fine, but it would appear that the classes.txt file which 
>> mcollective reads to get configuration management classes is, on
certain
>> servers, spuriously being emptied of all classes other than
''settings''
>>
>> Seeing as the settings class is being left in, I can only assume that
it
>> must be puppet doing the modification. Has anyone ever seen behaviour
like
>> this before? It''s truly bizarre, and even stranger is that
after this has
>> happened, sometimes the classes populate again. Every time
I''ve run puppet
>> manually  with puppetd --test or run it --noop, it''s run just
fine and
>> without noop it always seems to fix the problem, but later the node
will
>> show up as having only one class again and even later fix itself, so
it''s
>> next to impossible to pin down.
>>
>> This is 2.6.7 on Ubuntu running against a centralised puppetmaster.
>>
>
>
> Are you certain the behavior is buggy?  The fact that Puppet rights itself 
> makes me think that Puppet itself is not the problem.
>
> What if there is a cron job (or a human admin) that periodically does 
> something like "puppetd --onetime --no-daemonize --tags
settings"?  Or what
> if the affected node''s manifests use schedules in a way that
occasionally
> excludes all classes except ''settings''?  The former, at
least, should show
> up in the system log.
>
> You might also get some insight by wrapping puppetd in a script that makes 
> a timestamped backup of classes.txt after each run.  Done right, that ought
> to establish whether Puppet is the one writing the unexpected content, and 
> if so then the timestamps when the bad files appear might tell you 
> something useful.  Perhaps you could dredge the same information out of 
> your system logs, but the timestamped backups might be easier to analyze 
> (and they would pinpoint where in your logs to look for additional 
> information).
>
>
> John
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/GWaww-ftBBsJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.