Puppet users - Apr 2013 - Help me (fully) clear out stored configs from PuppetDB Postgresql

Hi all,

I recently started using the Nagios resources to populate a monitoring 
server.  I cycle hosts fairly quickly in my environment so already I''ve
had
five hosts come and go from under Nagios.  After a fair amount of searching 
I discovered "puppet node deactivate" and performed same on the
removed
hosts -- I''m still trying to figure out how to make that part of the 
process going forward -- but somehow only one of my five hosts ended up 
being removed from my Nagios configs.  I am using resource { purge => true 
} for the affected resource types.  

So here''s where I''m at -- if I run puppet node status
<node> on any of
these missing hosts, they appear as "deactivated".  If I clear out my 
nagios config files and re-run puppet agent, the nodes and their services 
(I did have an exported nagios_service resource when these hosts were 
alive, which I''ve since removed -- in case that matters) will
re-appear.
 I''ve tried *puppet node clean*, *puppet node destroy*, *puppet node 
deactivate*, with and without terminus=puppetdb.  I can see in the puppetdb 
log that it has received multiple deactivate commands for these hosts. 
 Nonetheless, the items are still appearing when the nagios host performs a 
collection.  I''ve got to put an end to that! 

An example of puppet node status for one of the affected:

# puppet node status [badnodename]
[badnodename]
Deactivated at 2013-04-03T23:00:55.349Z
No catalog received
No facts received


One interesting bit.  First, when I run puppet agent --test, during catalog 
compilation I _was_ seeing the following for all four affected hosts:  

warning: Nagios_service check_ssh_[badhost] found in both naginator and 
naginator; skipping the naginator version


Out of frustration, I disabled storedconfigurations on the puppet master 
and restarted it.  After a few catalogs had processed, I updated Nagios 
after manually purging the hosts file.  As expected, all my host and 
service definitions disappeared.  I then re-enabled storedconfigurations, 
fingers-crossed that the old host defs would be gone -- to my dismay, they 
reappeared on the *second* catalog run after I brought stored_configs back 
online.  Now, I no longer get the error message above, but I do still get 
the deactivated hosts and associated services.  

At this point, I''d be happy to simply wipe all of my existing stored 
configurations, as I suspect these four hosts have gotten themselves into 
some kind of limbo.  However, while the instructions for removing stored 
config data from the old MySQL puppet db is quite straightforward, it 
doesn''t seem to map to the postgresql DB schema, and I can''t
find any
advice on how to go about wiping it there.  I''d prefer not to just
scrap my
database, but I''m getting closer to that point.

Any help would be appreciated.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

< exasperated sigh >

So, the hosts in question have, as part of their unique hostnames, a small 
hex string which is in uppercase.  In the Nagios view where I was seeing 
these offline hosts, the hostname''s case was preserved.  Within
PuppetDB
however, the hostname is all lowercase.  As a consequence, these hosts were 
not removed because instead of deactivating that host, a *new* certname 
entry was created, with my mixed-case value, and *that* entry was marked as 
deactivated.  The actual host was still humming right along.

Well, I guess I learned something.  Technically there''s a bug there --
but
I''ve lost a whole day on this issue so I''m going to refrain
from reporting
it at the moment.  Hope this helps someone else.  If you''re
deactivating a
host with a capital letter, odds are good you''re not deactivating what
you
think you''re deactivating.  


On Thursday, April 4, 2013 3:19:00 PM UTC-4, Michael O''Dea
wrote:
>
> Hi all,
>
> I recently started using the Nagios resources to populate a monitoring 
> server.  I cycle hosts fairly quickly in my environment so already
I''ve had
> five hosts come and go from under Nagios.  After a fair amount of searching
> I discovered "puppet node deactivate" and performed same on the
removed
> hosts -- I''m still trying to figure out how to make that part of
the
> process going forward -- but somehow only one of my five hosts ended up 
> being removed from my Nagios configs.  I am using resource { purge =>
true
> } for the affected resource types.  
>
> So here''s where I''m at -- if I run puppet node status
<node> on any of
> these missing hosts, they appear as "deactivated".  If I clear
out my
> nagios config files and re-run puppet agent, the nodes and their services 
> (I did have an exported nagios_service resource when these hosts were 
> alive, which I''ve since removed -- in case that matters) will
re-appear.
>  I''ve tried *puppet node clean*, *puppet node destroy*, *puppet
node
> deactivate*, with and without terminus=puppetdb.  I can see in the 
> puppetdb log that it has received multiple deactivate commands for these 
> hosts.  Nonetheless, the items are still appearing when the nagios host 
> performs a collection.  I''ve got to put an end to that! 
>
> An example of puppet node status for one of the affected:
>
> # puppet node status [badnodename]
> [badnodename]
> Deactivated at 2013-04-03T23:00:55.349Z
> No catalog received
> No facts received
>
>
> One interesting bit.  First, when I run puppet agent --test, during 
> catalog compilation I _was_ seeing the following for all four affected 
> hosts:  
>
> warning: Nagios_service check_ssh_[badhost] found in both naginator and 
> naginator; skipping the naginator version
>
>
> Out of frustration, I disabled storedconfigurations on the puppet master 
> and restarted it.  After a few catalogs had processed, I updated Nagios 
> after manually purging the hosts file.  As expected, all my host and 
> service definitions disappeared.  I then re-enabled storedconfigurations, 
> fingers-crossed that the old host defs would be gone -- to my dismay, they 
> reappeared on the *second* catalog run after I brought stored_configs back 
> online.  Now, I no longer get the error message above, but I do still get 
> the deactivated hosts and associated services.  
>
> At this point, I''d be happy to simply wipe all of my existing
stored
> configurations, as I suspect these four hosts have gotten themselves into 
> some kind of limbo.  However, while the instructions for removing stored 
> config data from the old MySQL puppet db is quite straightforward, it 
> doesn''t seem to map to the postgresql DB schema, and I
can''t find any
> advice on how to go about wiping it there.  I''d prefer not to just
scrap my
> database, but I''m getting closer to that point.
>
> Any help would be appreciated.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Yeah, sounds like a bug Michael or at least something we can improve
upon - DNS is case insensitive: https://tools.ietf.org/rfc/rfc4343.txt
- but to make things more complex you can override what the node name
is in Puppet (ie. node_name_fact and node_name_value), so I can only
imagine the fun debates on such an issue :-).

Raise a bug though, and we''ll ponder the problem:

https://projects.puppetlabs.com/projects/puppetdb/issues/new

ken.

On Thu, Apr 4, 2013 at 9:08 PM, Michael O''Dea <gmodea@gmail.com>
wrote:
> < exasperated sigh >
>
> So, the hosts in question have, as part of their unique hostnames, a small
> hex string which is in uppercase.  In the Nagios view where I was seeing
> these offline hosts, the hostname''s case was preserved.  Within
PuppetDB
> however, the hostname is all lowercase.  As a consequence, these hosts were
> not removed because instead of deactivating that host, a new certname entry
> was created, with my mixed-case value, and that entry was marked as
> deactivated.  The actual host was still humming right along.
>
> Well, I guess I learned something.  Technically there''s a bug
there -- but
> I''ve lost a whole day on this issue so I''m going to
refrain from reporting
> it at the moment.  Hope this helps someone else.  If you''re
deactivating a
> host with a capital letter, odds are good you''re not deactivating
what you
> think you''re deactivating.
>
>
> On Thursday, April 4, 2013 3:19:00 PM UTC-4, Michael O''Dea wrote:
>>
>> Hi all,
>>
>> I recently started using the Nagios resources to populate a monitoring
>> server.  I cycle hosts fairly quickly in my environment so already
I''ve had
>> five hosts come and go from under Nagios.  After a fair amount of
searching
>> I discovered "puppet node deactivate" and performed same on
the removed
>> hosts -- I''m still trying to figure out how to make that part
of the process
>> going forward -- but somehow only one of my five hosts ended up being
>> removed from my Nagios configs.  I am using resource { purge => true
} for
>> the affected resource types.
>>
>> So here''s where I''m at -- if I run puppet node status
<node> on any of
>> these missing hosts, they appear as "deactivated".  If I
clear out my nagios
>> config files and re-run puppet agent, the nodes and their services (I
did
>> have an exported nagios_service resource when these hosts were alive,
which
>> I''ve since removed -- in case that matters) will re-appear. 
I''ve tried
>> puppet node clean, puppet node destroy, puppet node deactivate, with
and
>> without terminus=puppetdb.  I can see in the puppetdb log that it has
>> received multiple deactivate commands for these hosts.  Nonetheless,
the
>> items are still appearing when the nagios host performs a collection. 
I''ve
>> got to put an end to that!
>>
>> An example of puppet node status for one of the affected:
>>
>> # puppet node status [badnodename]
>> [badnodename]
>> Deactivated at 2013-04-03T23:00:55.349Z
>> No catalog received
>> No facts received
>>
>>
>> One interesting bit.  First, when I run puppet agent --test, during
>> catalog compilation I _was_ seeing the following for all four affected
>> hosts:
>>
>> warning: Nagios_service check_ssh_[badhost] found in both naginator and
>> naginator; skipping the naginator version
>>
>>
>> Out of frustration, I disabled storedconfigurations on the puppet
master
>> and restarted it.  After a few catalogs had processed, I updated Nagios
>> after manually purging the hosts file.  As expected, all my host and
service
>> definitions disappeared.  I then re-enabled storedconfigurations,
>> fingers-crossed that the old host defs would be gone -- to my dismay,
they
>> reappeared on the *second* catalog run after I brought stored_configs
back
>> online.  Now, I no longer get the error message above, but I do still
get
>> the deactivated hosts and associated services.
>>
>> At this point, I''d be happy to simply wipe all of my existing
stored
>> configurations, as I suspect these four hosts have gotten themselves
into
>> some kind of limbo.  However, while the instructions for removing
stored
>> config data from the old MySQL puppet db is quite straightforward, it
>> doesn''t seem to map to the postgresql DB schema, and I
can''t find any advice
>> on how to go about wiping it there.  I''d prefer not to just
scrap my
>> database, but I''m getting closer to that point.
>>
>> Any help would be appreciated.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscribe@googlegroups.com.
> To post to this group, send email to puppet-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/puppet-users?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On Thursday, April 4, 2013 1:08:06 PM UTC-7, Michael O''Dea wrote:
> < exasperated sigh >
>
> So, the hosts in question have, as part of their unique hostnames, a small 
> hex string which is in uppercase.  In the Nagios view where I was seeing 
> these offline hosts, the hostname''s case was preserved.  Within
PuppetDB
> however, the hostname is all lowercase.  As a consequence, these hosts were
> not removed because instead of deactivating that host, a *new* certname 
> entry was created, with my mixed-case value, and *that* entry was marked 
> as deactivated.  The actual host was still humming right along.
>
> Well, I guess I learned something.  Technically there''s a bug
there -- but
> I''ve lost a whole day on this issue so I''m going to
refrain from reporting
> it at the moment.  Hope this helps someone else.  If you''re
deactivating a
> host with a capital letter, odds are good you''re not deactivating
what you
> think you''re deactivating.  
>
>
I''ll join in that exasperated sigh with you. :) This issue pops up
fairly
commonly (albeit usually with hostname vs. fqdn). I remember a conversation 
on irc about having the `puppet node deactivate` command first check if the 
node exists. It could then fail if the node doesn''t (because you
probably
just got the name wrong), or have an option to force (if for some reason 
you want to proactively tell PuppetDB a node exists, but is inactive). 
Potentially, we could also perform a regex query and make suggestions.
I''m
not sure what happened with that idea, but I guess it didn''t go
anywhere.
At any rate, I think the frustration of accidentally deactivating the wrong 
nodes far outweighs whatever rationale someone might have for deactivating 
a node that doesn''t exist. This needs to change.

I''m sorry for your trouble. I''ll make sure to follow up on
making that
change, this time.
 
>
> On Thursday, April 4, 2013 3:19:00 PM UTC-4, Michael O''Dea wrote:
>>
>> Hi all,
>>
>> I recently started using the Nagios resources to populate a monitoring 
>> server.  I cycle hosts fairly quickly in my environment so already
I''ve had
>> five hosts come and go from under Nagios.  After a fair amount of
searching
>> I discovered "puppet node deactivate" and performed same on
the removed
>> hosts -- I''m still trying to figure out how to make that part
of the
>> process going forward -- but somehow only one of my five hosts ended up
>> being removed from my Nagios configs.  I am using resource { purge
=> true
>> } for the affected resource types.  
>>
>> So here''s where I''m at -- if I run puppet node status
<node> on any of
>> these missing hosts, they appear as "deactivated".  If I
clear out my
>> nagios config files and re-run puppet agent, the nodes and their
services
>> (I did have an exported nagios_service resource when these hosts were 
>> alive, which I''ve since removed -- in case that matters) will
re-appear.
>>  I''ve tried *puppet node clean*, *puppet node destroy*,
*puppet node
>> deactivate*, with and without terminus=puppetdb.  I can see in the 
>> puppetdb log that it has received multiple deactivate commands for
these
>> hosts.  Nonetheless, the items are still appearing when the nagios host
>> performs a collection.  I''ve got to put an end to that! 
>>
>> An example of puppet node status for one of the affected:
>>
>> # puppet node status [badnodename]
>> [badnodename]
>> Deactivated at 2013-04-03T23:00:55.349Z
>> No catalog received
>> No facts received
>>
>>
>> One interesting bit.  First, when I run puppet agent --test, during 
>> catalog compilation I _was_ seeing the following for all four affected 
>> hosts:  
>>
>> warning: Nagios_service check_ssh_[badhost] found in both naginator and
>> naginator; skipping the naginator version
>>
>>
>> Out of frustration, I disabled storedconfigurations on the puppet
master
>> and restarted it.  After a few catalogs had processed, I updated Nagios
>> after manually purging the hosts file.  As expected, all my host and 
>> service definitions disappeared.  I then re-enabled
storedconfigurations,
>> fingers-crossed that the old host defs would be gone -- to my dismay,
they
>> reappeared on the *second* catalog run after I brought stored_configs
back
>> online.  Now, I no longer get the error message above, but I do still
get
>> the deactivated hosts and associated services.  
>>
>> At this point, I''d be happy to simply wipe all of my existing
stored
>> configurations, as I suspect these four hosts have gotten themselves
into
>> some kind of limbo.  However, while the instructions for removing
stored
>> config data from the old MySQL puppet db is quite straightforward, it 
>> doesn''t seem to map to the postgresql DB schema, and I
can''t find any
>> advice on how to go about wiping it there.  I''d prefer not to
just scrap my
>> database, but I''m getting closer to that point.
>>
>> Any help would be appreciated.
>>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.