Hi all,
We''ve come across a rather strange problem where the parameters of some
resources in PuppetDB are now empty.
We have a Nagios server collecting resources from PuppetDB and we''ve
started to get failures like this for one resource type:
Error: Could not retrieve catalog from remote server: Error 400 on SERVER:
Must pass host_alias to Nagios::Config::Host[hostname] on node nagiosserver
The Puppet manifest that defines that resources is this, it is impossible
to not populate host_alias:
*******************************
define nagios::host::host($host = $::fqdn, $tag = undef) {
@@nagios::config::host { $host:
host_alias => $host,
address => $host,
tag => $use_tag,
}
}
*******************************
If we query PuppetDB directly (redacted), there are indeed no parameters at
all on this resource:
*******************************
# curl -H ''Accept: application/json'' -X GET
''https://puppet:8081/v2/resources'' --cacert
/var/lib/puppet/ssl/ca/ca_crt.pem --cert
/var/lib/puppet/ssl/certs/puppet.pem --key
/var/lib/puppet/ssl/private_keys/puppet.pem --data-urlencode
''query=["=",
"type", "Nagios::Config::Host"]'' | jgrep
"certname=hostname"
[
{
"resource": "8ba4379c364b9dba9d18836ef52ce5f4f82d0468",
"parameters": {
},
"title": "hostname",
"exported": true,
"certname": "hostname",
"type": "Nagios::Config::Host",
"sourceline": 27,
"sourcefile":
"/etc/puppet/environments/production/modules/nagios/manifests/host/host.pp",
"tags": [
]
}
]
*******************************
After a Puppet run and a new catalog, this resource now looks normal:
*******************************
# curl -H ''Accept: application/json'' -X GET
''https://puppet:8081/v2/resources'' --cacert
/var/lib/puppet/ssl/ca/ca_crt.pem --cert
/var/lib/puppet/ssl/certs/puppet.pem --key
/var/lib/puppet/ssl/private_keys/puppet.pem --data-urlencode
''query=["=",
"type", "Nagios::Config::Host"]'' | jgrep
"certname=hostname"
[
{
"type": "Nagios::Config::Host",
"sourceline": 27,
"title": "hostname",
"certname": "hostname",
"resource": "8ba4379c364b9dba9d18836ef52ce5f4f82d0468",
"parameters": {
"address": "hostname",
"tag": "tag",
"host_alias": "hostname"
},
"exported": true,
"sourcefile":
"/etc/puppet/environments/production/modules/nagios/manifests/host/host.pp",
"tags": [
]
}
]
*******************************
These nodes do not have Puppet run on them regularly. We did upgrade from
PuppetDB 1.0.1-1.el6.noarch to 1.3.2-1.el6.noarch about 3 weeks ago. We
don''t do any automatic report or node expiry.
This started happening back on 2nd August, just halfway through the day the
Puppet runs on the Nagios server start failing with this error. Now if I
think back, at this time I think I had a broken Nagios module and a lot of
manifests were failing to compile, but I fixed this and re-ran the
failures, and everything was ok. PuppetDB only stores the last catalog, so
there''s no way a broken catalog could have stayed there, right?
I''ve fixed this by refreshing the catalog of all nodes in PuppetDB, but
I''ve got no idea how it got into this state. Any ideas?
Thanks,
-Luke
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.
Ken Barber
2013-Aug-08 11:48 UTC
Re: [Puppet Users] does PuppetDB expire resource parameters?
> We''ve come across a rather strange problem where the parameters of some > resources in PuppetDB are now empty. > > We have a Nagios server collecting resources from PuppetDB and we''ve started > to get failures like this for one resource type: > > Error: Could not retrieve catalog from remote server: Error 400 on SERVER: > Must pass host_alias to Nagios::Config::Host[hostname] on node nagiosserver > > The Puppet manifest that defines that resources is this, it is impossible to > not populate host_alias: > > ******************************* > define nagios::host::host($host = $::fqdn, $tag = undef) { > @@nagios::config::host { $host: > host_alias => $host, > address => $host, > tag => $use_tag, > } > } > ******************************* > > If we query PuppetDB directly (redacted), there are indeed no parameters at > all on this resource: > > ******************************* > # curl -H ''Accept: application/json'' -X GET > ''https://puppet:8081/v2/resources'' --cacert > /var/lib/puppet/ssl/ca/ca_crt.pem --cert > /var/lib/puppet/ssl/certs/puppet.pem --key > /var/lib/puppet/ssl/private_keys/puppet.pem --data-urlencode ''query=["=", > "type", "Nagios::Config::Host"]'' | jgrep "certname=hostname" > [ > { > "resource": "8ba4379c364b9dba9d18836ef52ce5f4f82d0468", > "parameters": { > }, > "title": "hostname", > "exported": true, > "certname": "hostname", > "type": "Nagios::Config::Host", > "sourceline": 27, > "sourcefile": > "/etc/puppet/environments/production/modules/nagios/manifests/host/host.pp", > "tags": [ > ] > } > ] > ******************************* > > After a Puppet run and a new catalog, this resource now looks normal: > > ******************************* > # curl -H ''Accept: application/json'' -X GET > ''https://puppet:8081/v2/resources'' --cacert > /var/lib/puppet/ssl/ca/ca_crt.pem --cert > /var/lib/puppet/ssl/certs/puppet.pem --key > /var/lib/puppet/ssl/private_keys/puppet.pem --data-urlencode ''query=["=", > "type", "Nagios::Config::Host"]'' | jgrep "certname=hostname" > [ > { > "type": "Nagios::Config::Host", > "sourceline": 27, > "title": "hostname", > "certname": "hostname", > "resource": "8ba4379c364b9dba9d18836ef52ce5f4f82d0468", > "parameters": { > "address": "hostname", > "tag": "tag", > "host_alias": "hostname" > }, > "exported": true, > "sourcefile": > "/etc/puppet/environments/production/modules/nagios/manifests/host/host.pp", > "tags": [ > ] > } > ] > ******************************* > > These nodes do not have Puppet run on them regularly. We did upgrade from > PuppetDB 1.0.1-1.el6.noarch to 1.3.2-1.el6.noarch about 3 weeks ago. We > don''t do any automatic report or node expiry. > > This started happening back on 2nd August, just halfway through the day the > Puppet runs on the Nagios server start failing with this error. Now if I > think back, at this time I think I had a broken Nagios module and a lot of > manifests were failing to compile, but I fixed this and re-ran the failures, > and everything was ok. PuppetDB only stores the last catalog, so there''s no > way a broken catalog could have stayed there, right? > > I''ve fixed this by refreshing the catalog of all nodes in PuppetDB, but I''ve > got no idea how it got into this state. Any ideas?No good idea yet, but there is something suspicious in your curl responses - the "resource" hash, did you obfuscate this yourself on purpose? The two hashes between the first and second requests are identical. That hash is calculated based on the sum of the resource, including parameters - so it seems impossible that PuppetDB arrived at the same hash with and without parameters. Maybe just maybe the responses were identical, and somehow PuppetDB was not returning parameters as a fault. This might indicate some sort of integrity problem, whereby the link to the parameters in the RDBMS was lost somehow, although this is the first time I''ve heard of it Luke. Maybe this was an upgrade schema change failure between 1.0.1 and 1.3.2? I''d have to consult what changed in the schema between those two points to determine if this is likely however. Does the timing of your upgrade, and the first time you saw this fault line up with such a possibility? Remember a schema change will only occur after a restart of PuppetDB ... (so perhaps consult your logs to see when this happened after upgrade). Let me at least try to replicate while I await your responses. ken. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
Nicolai Mollerup
2013-Aug-08 11:57 UTC
[Puppet Users] Re: does PuppetDB expire resource parameters?
Since you upgraded from 1.0 to 1.3.2 something has indeed happened in puppetdb so check your settings: http://docs.puppetlabs.com/puppetdb/1.3/release_notes.html#section-3 This talks about automatic node purging, which will deactivate exported resources aswell. /Nicolai -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
Ken Barber
2013-Aug-08 12:42 UTC
Re: [Puppet Users] does PuppetDB expire resource parameters?
Well FWIW, I can''t replicate this when I upgrade from 1.0.1 to 1.3.2, I even attempted to name things in the same way, no luck: https://gist.github.com/kbarber/6184213. Looking at the schema changes in code, not much has changed to do with parameters since 1.0.1 either: https://github.com/puppetlabs/puppetdb/blob/master/src/com/puppetlabs/puppetdb/scf/migrate.clj#L326-L331 So its probably not specifically a bug with the schema change. On Thu, Aug 8, 2013 at 12:48 PM, Ken Barber <ken@puppetlabs.com> wrote:>> We''ve come across a rather strange problem where the parameters of some >> resources in PuppetDB are now empty. >> >> We have a Nagios server collecting resources from PuppetDB and we''ve started >> to get failures like this for one resource type: >> >> Error: Could not retrieve catalog from remote server: Error 400 on SERVER: >> Must pass host_alias to Nagios::Config::Host[hostname] on node nagiosserver >> >> The Puppet manifest that defines that resources is this, it is impossible to >> not populate host_alias: >> >> ******************************* >> define nagios::host::host($host = $::fqdn, $tag = undef) { >> @@nagios::config::host { $host: >> host_alias => $host, >> address => $host, >> tag => $use_tag, >> } >> } >> ******************************* >> >> If we query PuppetDB directly (redacted), there are indeed no parameters at >> all on this resource: >> >> ******************************* >> # curl -H ''Accept: application/json'' -X GET >> ''https://puppet:8081/v2/resources'' --cacert >> /var/lib/puppet/ssl/ca/ca_crt.pem --cert >> /var/lib/puppet/ssl/certs/puppet.pem --key >> /var/lib/puppet/ssl/private_keys/puppet.pem --data-urlencode ''query=["=", >> "type", "Nagios::Config::Host"]'' | jgrep "certname=hostname" >> [ >> { >> "resource": "8ba4379c364b9dba9d18836ef52ce5f4f82d0468", >> "parameters": { >> }, >> "title": "hostname", >> "exported": true, >> "certname": "hostname", >> "type": "Nagios::Config::Host", >> "sourceline": 27, >> "sourcefile": >> "/etc/puppet/environments/production/modules/nagios/manifests/host/host.pp", >> "tags": [ >> ] >> } >> ] >> ******************************* >> >> After a Puppet run and a new catalog, this resource now looks normal: >> >> ******************************* >> # curl -H ''Accept: application/json'' -X GET >> ''https://puppet:8081/v2/resources'' --cacert >> /var/lib/puppet/ssl/ca/ca_crt.pem --cert >> /var/lib/puppet/ssl/certs/puppet.pem --key >> /var/lib/puppet/ssl/private_keys/puppet.pem --data-urlencode ''query=["=", >> "type", "Nagios::Config::Host"]'' | jgrep "certname=hostname" >> [ >> { >> "type": "Nagios::Config::Host", >> "sourceline": 27, >> "title": "hostname", >> "certname": "hostname", >> "resource": "8ba4379c364b9dba9d18836ef52ce5f4f82d0468", >> "parameters": { >> "address": "hostname", >> "tag": "tag", >> "host_alias": "hostname" >> }, >> "exported": true, >> "sourcefile": >> "/etc/puppet/environments/production/modules/nagios/manifests/host/host.pp", >> "tags": [ >> ] >> } >> ] >> ******************************* >> >> These nodes do not have Puppet run on them regularly. We did upgrade from >> PuppetDB 1.0.1-1.el6.noarch to 1.3.2-1.el6.noarch about 3 weeks ago. We >> don''t do any automatic report or node expiry. >> >> This started happening back on 2nd August, just halfway through the day the >> Puppet runs on the Nagios server start failing with this error. Now if I >> think back, at this time I think I had a broken Nagios module and a lot of >> manifests were failing to compile, but I fixed this and re-ran the failures, >> and everything was ok. PuppetDB only stores the last catalog, so there''s no >> way a broken catalog could have stayed there, right? >> >> I''ve fixed this by refreshing the catalog of all nodes in PuppetDB, but I''ve >> got no idea how it got into this state. Any ideas? > > No good idea yet, but there is something suspicious in your curl > responses - the "resource" hash, did you obfuscate this yourself on > purpose? The two hashes between the first and second requests are > identical. That hash is calculated based on the sum of the resource, > including parameters - so it seems impossible that PuppetDB arrived at > the same hash with and without parameters. > > Maybe just maybe the responses were identical, and somehow PuppetDB > was not returning parameters as a fault. This might indicate some sort > of integrity problem, whereby the link to the parameters in the RDBMS > was lost somehow, although this is the first time I''ve heard of it > Luke. Maybe this was an upgrade schema change failure between 1.0.1 > and 1.3.2? I''d have to consult what changed in the schema between > those two points to determine if this is likely however. Does the > timing of your upgrade, and the first time you saw this fault line up > with such a possibility? Remember a schema change will only occur > after a restart of PuppetDB ... (so perhaps consult your logs to see > when this happened after upgrade). > > Let me at least try to replicate while I await your responses. > > ken.-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
Luke Bigum
2013-Aug-08 13:03 UTC
Re: [Puppet Users] does PuppetDB expire resource parameters?
On Thursday, August 8, 2013 12:48:03 PM UTC+1, Ken Barber wrote:> > > No good idea yet, but there is something suspicious in your curl > responses - the "resource" hash, did you obfuscate this yourself on > purpose? The two hashes between the first and second requests are > identical. That hash is calculated based on the sum of the resource, > including parameters - so it seems impossible that PuppetDB arrived at > the same hash with and without parameters. >I think that''s just me being too sensorship heavy and abusing copy and paste, I would have copied some fields from the same example. Trust me that the resources dictionary was empty though ;-)> Maybe just maybe the responses were identical, and somehow PuppetDB > was not returning parameters as a fault. This might indicate some sort > of integrity problem, whereby the link to the parameters in the RDBMS > was lost somehow, although this is the first time I''ve heard of it > Luke. Maybe this was an upgrade schema change failure between 1.0.1 > and 1.3.2?We thought of that, however we upgraded 3 weeks ago and I saw working reports in my Dashboard on the Nagios server up until Friday last week, so it was able to collect valid resources for a long while after the DB migration. What happened on Friday though was I basically had this running: the database mco rpc shellout cmd cmd="puppet agent --test --noop" -T mcollective Then I realised we had broken manifests everywhere, so I fixed it and started MC again. From that point onward the Nagios server was unable to collect as it was getting these dodgy resources from PuppetDB. What doesn''t make sense is my subsequent site wide no-op should have replaced every manifest in PuppetDB, so I''m stumped. I''d have to consult what changed in the schema between> those two points to determine if this is likely however. Does the > timing of your upgrade, and the first time you saw this fault line up > with such a possibility? Remember a schema change will only occur > after a restart of PuppetDB ... (so perhaps consult your logs to see > when this happened after upgrade). >Just checked - PuppetDB restarted at the same time the RPMs were upgraded and hasn''t restarted since. Let me at least try to replicate while I await your responses.>Now if I was thinking smart I would have taken a Postgres backup before I re-freshed all the catalogs, but I didn''t, not sure if that would have helped much. I agree with subsequent posts as well - probably not a migration problem. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
Ken Barber
2013-Aug-08 13:14 UTC
Re: [Puppet Users] does PuppetDB expire resource parameters?
>> No good idea yet, but there is something suspicious in your curl >> responses - the "resource" hash, did you obfuscate this yourself on >> purpose? The two hashes between the first and second requests are >> identical. That hash is calculated based on the sum of the resource, >> including parameters - so it seems impossible that PuppetDB arrived at >> the same hash with and without parameters. > > > I think that''s just me being too sensorship heavy and abusing copy and > paste, I would have copied some fields from the same example. Trust me that > the resources dictionary was empty though ;-)So just to clarify, the resources hash ''8ba4379c364b9dba9d18836ef52ce5f4f82d0468'' was different or the same between the two examples?> Now if I was thinking smart I would have taken a Postgres backup before I > re-freshed all the catalogs, but I didn''t, not sure if that would have > helped much. I agree with subsequent posts as well - probably not a > migration problem.It might have helped. Are any other nodes and resources still exhibiting this strange behaviour? Maybe checking for any exported resources with no params might be worthwhile. ken. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
Luke Bigum
2013-Aug-08 14:20 UTC
Re: [Puppet Users] does PuppetDB expire resource parameters?
On Thursday, August 8, 2013 2:14:33 PM UTC+1, Ken Barber wrote:> > > I think that''s just me being too sensorship heavy and abusing copy and > > paste, I would have copied some fields from the same example. Trust me > that > > the resources dictionary was empty though ;-) > > So just to clarify, the resources hash > ''8ba4379c364b9dba9d18836ef52ce5f4f82d0468'' was different or the same > between the two examples? >Actually they are the same, my copy and paste skills remain rock solid for another day. I found some more broken resources belonging to some dev servers with a handy jgrep: curl -H ''Accept: application/json'' -X GET ''https://puppet:8081/v2/resources'' --cacert /var/lib/puppet/ssl/ca/ca_crt.pem --cert /var/lib/puppet/ssl/certs/puppet.pem --key /var/lib/puppet/ssl/private_keys/puppet.pem --data-urlencode ''query=["=", "type", "Nagios::Config::Host"]'' | jgrep "parameters.host_alias=null" This is the hostname redacted JSON before: *************************** [ { "type": "Nagios::Config::Host", "tags": [ "nagios::host", "default", "node", "config", "nagios::config::host", "hostname", "en1", "host", "nagios::host::host", "undef", "class", "nagios" ], "parameters": { }, "certname": "hostname", "title": "hostname", "resource": "3368824b20c1eb7052952f574bb5547ca0c95a50", "sourcefile": "/etc/puppet/environments/production/modules/nagios/manifests/host/host.pp", "sourceline": 27, "exported": true } ] *************************** And after a Puppet run to refresh the catalog: *************************** [ { "type": "Nagios::Config::Host", "sourceline": 27, "certname": "hostname", "resource": "3368824b20c1eb7052952f574bb5547ca0c95a50", "exported": true, "title": "hostname", "tags": [ "nagios::host", "node", "config", "nagios::config::host", "hostname", "en1", "host", "nagios::host::host", "undef", "base", "class", "nagios" ], "parameters": { "host_alias": "hostname", "tag": "en1", "address": "hostname" }, "sourcefile": "/etc/puppet/environments/production/modules/nagios/manifests/host/host.pp" } ] *************************** So there''s 12 resources with this problem remaining now. -Luke> Now if I was thinking smart I would have taken a Postgres backup before I > > re-freshed all the catalogs, but I didn''t, not sure if that would have > > helped much. I agree with subsequent posts as well - probably not a > > migration problem. > > It might have helped. Are any other nodes and resources still > exhibiting this strange behaviour? Maybe checking for any exported > resources with no params might be worthwhile. > > ken. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
Ken Barber
2013-Aug-08 14:22 UTC
Re: [Puppet Users] does PuppetDB expire resource parameters?
Wow. Good, don''t do anything ... I''ll reach out to you directly. On Thu, Aug 8, 2013 at 3:20 PM, Luke Bigum <Luke.Bigum@lmax.com> wrote:> > > On Thursday, August 8, 2013 2:14:33 PM UTC+1, Ken Barber wrote: >> >> > I think that''s just me being too sensorship heavy and abusing copy and >> > paste, I would have copied some fields from the same example. Trust me >> > that >> > the resources dictionary was empty though ;-) >> >> So just to clarify, the resources hash >> ''8ba4379c364b9dba9d18836ef52ce5f4f82d0468'' was different or the same >> between the two examples? > > > Actually they are the same, my copy and paste skills remain rock solid for > another day. > > I found some more broken resources belonging to some dev servers with a > handy jgrep: > > curl -H ''Accept: application/json'' -X GET ''https://puppet:8081/v2/resources'' > --cacert /var/lib/puppet/ssl/ca/ca_crt.pem --cert > /var/lib/puppet/ssl/certs/puppet.pem --key > /var/lib/puppet/ssl/private_keys/puppet.pem --data-urlencode ''query=["=", > "type", "Nagios::Config::Host"]'' | jgrep "parameters.host_alias=null" > > This is the hostname redacted JSON before: > > *************************** > [ > { > "type": "Nagios::Config::Host", > "tags": [ > "nagios::host", > "default", > "node", > "config", > "nagios::config::host", > "hostname", > "en1", > "host", > "nagios::host::host", > "undef", > "class", > "nagios" > ], > "parameters": { > }, > "certname": "hostname", > "title": "hostname", > "resource": "3368824b20c1eb7052952f574bb5547ca0c95a50", > "sourcefile": > "/etc/puppet/environments/production/modules/nagios/manifests/host/host.pp", > "sourceline": 27, > "exported": true > } > ] > *************************** > > And after a Puppet run to refresh the catalog: > > > *************************** > [ > { > "type": "Nagios::Config::Host", > "sourceline": 27, > "certname": "hostname", > "resource": "3368824b20c1eb7052952f574bb5547ca0c95a50", > "exported": true, > "title": "hostname", > "tags": [ > "nagios::host", > "node", > "config", > "nagios::config::host", > "hostname", > "en1", > "host", > "nagios::host::host", > "undef", > "base", > "class", > "nagios" > ], > "parameters": { > "host_alias": "hostname", > "tag": "en1", > "address": "hostname" > }, > "sourcefile": > "/etc/puppet/environments/production/modules/nagios/manifests/host/host.pp" > } > ] > *************************** > > So there''s 12 resources with this problem remaining now. > > -Luke > >> > Now if I was thinking smart I would have taken a Postgres backup before >> > I >> > re-freshed all the catalogs, but I didn''t, not sure if that would have >> > helped much. I agree with subsequent posts as well - probably not a >> > migration problem. >> >> It might have helped. Are any other nodes and resources still >> exhibiting this strange behaviour? Maybe checking for any exported >> resources with no params might be worthwhile. >> >> ken. > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users. > For more options, visit https://groups.google.com/groups/opt_out. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
Ken Barber
2013-Aug-08 16:51 UTC
Re: [Puppet Users] does PuppetDB expire resource parameters?
For those following along at home, Luke and I continued to dig on IRC ... he was actually using HSQLDB not Postgresql it turns out. Doing queries on the database resulted in missing rows from resource_params for some reason, which a puppet run restored. We can''t find a reasonable explanation for this yet, and are still looking. FWIW, this is the first time I''ve seen this behaviour. On Thu, Aug 8, 2013 at 3:22 PM, Ken Barber <ken@puppetlabs.com> wrote:> Wow. Good, don''t do anything ... I''ll reach out to you directly. > > On Thu, Aug 8, 2013 at 3:20 PM, Luke Bigum <Luke.Bigum@lmax.com> wrote: >> >> >> On Thursday, August 8, 2013 2:14:33 PM UTC+1, Ken Barber wrote: >>> >>> > I think that''s just me being too sensorship heavy and abusing copy and >>> > paste, I would have copied some fields from the same example. Trust me >>> > that >>> > the resources dictionary was empty though ;-) >>> >>> So just to clarify, the resources hash >>> ''8ba4379c364b9dba9d18836ef52ce5f4f82d0468'' was different or the same >>> between the two examples? >> >> >> Actually they are the same, my copy and paste skills remain rock solid for >> another day. >> >> I found some more broken resources belonging to some dev servers with a >> handy jgrep: >> >> curl -H ''Accept: application/json'' -X GET ''https://puppet:8081/v2/resources'' >> --cacert /var/lib/puppet/ssl/ca/ca_crt.pem --cert >> /var/lib/puppet/ssl/certs/puppet.pem --key >> /var/lib/puppet/ssl/private_keys/puppet.pem --data-urlencode ''query=["=", >> "type", "Nagios::Config::Host"]'' | jgrep "parameters.host_alias=null" >> >> This is the hostname redacted JSON before: >> >> *************************** >> [ >> { >> "type": "Nagios::Config::Host", >> "tags": [ >> "nagios::host", >> "default", >> "node", >> "config", >> "nagios::config::host", >> "hostname", >> "en1", >> "host", >> "nagios::host::host", >> "undef", >> "class", >> "nagios" >> ], >> "parameters": { >> }, >> "certname": "hostname", >> "title": "hostname", >> "resource": "3368824b20c1eb7052952f574bb5547ca0c95a50", >> "sourcefile": >> "/etc/puppet/environments/production/modules/nagios/manifests/host/host.pp", >> "sourceline": 27, >> "exported": true >> } >> ] >> *************************** >> >> And after a Puppet run to refresh the catalog: >> >> >> *************************** >> [ >> { >> "type": "Nagios::Config::Host", >> "sourceline": 27, >> "certname": "hostname", >> "resource": "3368824b20c1eb7052952f574bb5547ca0c95a50", >> "exported": true, >> "title": "hostname", >> "tags": [ >> "nagios::host", >> "node", >> "config", >> "nagios::config::host", >> "hostname", >> "en1", >> "host", >> "nagios::host::host", >> "undef", >> "base", >> "class", >> "nagios" >> ], >> "parameters": { >> "host_alias": "hostname", >> "tag": "en1", >> "address": "hostname" >> }, >> "sourcefile": >> "/etc/puppet/environments/production/modules/nagios/manifests/host/host.pp" >> } >> ] >> *************************** >> >> So there''s 12 resources with this problem remaining now. >> >> -Luke >> >>> > Now if I was thinking smart I would have taken a Postgres backup before >>> > I >>> > re-freshed all the catalogs, but I didn''t, not sure if that would have >>> > helped much. I agree with subsequent posts as well - probably not a >>> > migration problem. >>> >>> It might have helped. Are any other nodes and resources still >>> exhibiting this strange behaviour? Maybe checking for any exported >>> resources with no params might be worthwhile. >>> >>> ken. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to puppet-users+unsubscribe@googlegroups.com. >> To post to this group, send email to puppet-users@googlegroups.com. >> Visit this group at http://groups.google.com/group/puppet-users. >> For more options, visit https://groups.google.com/groups/opt_out. >> >>-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.