Hey all,
I''m exporting a Host resource across my puppet managed client for
various
synchronizations across my servers. Lately I''m seeing duplicate
exported
resources problems and have a hard time understanding what is causing this,
I''ll appreciate your help :)
This happens almost every day at various times, I see in the log file:
Oct 26 07:43:57 ubuntu puppet-master[6164]: Exported resource
Host[proxy1back] cannot override local resource on node lb5back
Oct 26 07:44:00 ubuntu puppet-master[6164]: Exported resource
Host[proxy1back] cannot override local resource on node lb3
Oct 26 07:44:00 ubuntu puppet-master[6164]: Exported resource
Host[proxy1back] cannot override local resource on node lb3
Oct 26 07:44:01 ubuntu puppet-master[6164]: Exported resource
Host[proxy1back] cannot override local resource on node lb3back
Oct 26 07:44:01 ubuntu puppet-master[6164]: Exported resource
Host[proxy1back] cannot override local resource on node lb3back
As you can see it happens on several servers and not on a specific one, the
duplicate resource happens on random servers and the error affects random
servers (some of my servers are operating fine and does not report
duplicate errors).
Looking at the puppet DB resources table I see this:
mysql> select * from resources where title = ''proxy1back'';
+--------+------------+---------+---------+----------------+----------+------+---------------------+---------------------+
| id | title | restype | host_id | source_file_id | exported |
line | updated_at | created_at |
+--------+------------+---------+---------+----------------+----------+------+---------------------+---------------------+
| 573048 | proxy1back | Host | 39640 | 26 | 1 |
63 | 2011-10-26 02:01:22 | 2011-10-26 02:01:22 |
| 576542 | proxy1back | Host | 40107 | 2 | 0 |
63 | 2011-10-26 04:01:05 | 2011-10-26 04:01:05 |
| 581182 | proxy1back | Host | 40844 | 26 | 1 |
63 | 2011-10-26 07:01:35 | 2011-10-26 07:01:35 |
+--------+------------+---------+---------+----------------+----------+------+---------------------+---------------------+
It seems like at some point the server is getting a different host_id and
then exporting the resource again with the same title, which causes the
duplicate. But I have no idea why..
My Host resource looks like this:
@@host { "$hostname":
comment => "$hostname",
ip => "$ec2_local_ipv4",
name => "stable_host$proxy_id",
tag => "production-proxy",
}
Thanks in advance,
Galed.
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/3Ntpn1-sN20J.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
On Oct 26, 3:01 am, Galed Friedmann <galed.friedm...@onavo.com> wrote:> Hey all, > I''m exporting a Host resource across my puppet managed client for various > synchronizations across my servers. Lately I''m seeing duplicate exported > resources problems and have a hard time understanding what is causing this, > I''ll appreciate your help :) > > This happens almost every day at various times, I see in the log file: > Oct 26 07:43:57 ubuntu puppet-master[6164]: Exported resource > Host[proxy1back] cannot override local resource on node lb5back > Oct 26 07:44:00 ubuntu puppet-master[6164]: Exported resource > Host[proxy1back] cannot override local resource on node lb3 > Oct 26 07:44:00 ubuntu puppet-master[6164]: Exported resource > Host[proxy1back] cannot override local resource on node lb3 > Oct 26 07:44:01 ubuntu puppet-master[6164]: Exported resource > Host[proxy1back] cannot override local resource on node lb3back > Oct 26 07:44:01 ubuntu puppet-master[6164]: Exported resource > Host[proxy1back] cannot override local resource on node lb3back > > As you can see it happens on several servers and not on a specific one, the > duplicate resource happens on random servers and the error affects random > servers (some of my servers are operating fine and does not report > duplicate errors). > > Looking at the puppet DB resources table I see this: > mysql> select * from resources where title = ''proxy1back''; > +--------+------------+---------+---------+----------------+----------+------+---------------------+---------------------+ > | id | title | restype | host_id | source_file_id | exported | > line | updated_at | created_at | > +--------+------------+---------+---------+----------------+----------+------+---------------------+---------------------+ > | 573048 | proxy1back | Host | 39640 | 26 | 1 | > 63 | 2011-10-26 02:01:22 | 2011-10-26 02:01:22 | > | 576542 | proxy1back | Host | 40107 | 2 | 0 | > 63 | 2011-10-26 04:01:05 | 2011-10-26 04:01:05 | > | 581182 | proxy1back | Host | 40844 | 26 | 1 | > 63 | 2011-10-26 07:01:35 | 2011-10-26 07:01:35 | > +--------+------------+---------+---------+----------------+----------+------+---------------------+---------------------+ > > It seems like at some point the server is getting a different host_id and > then exporting the resource again with the same title, which causes the > duplicate. But I have no idea why.. > > My Host resource looks like this: > @@host { "$hostname": > comment => "$hostname", > ip => "$ec2_local_ipv4", > name => "stable_host$proxy_id", > tag => "production-proxy", > > }I think you''re looking for the problem in the wrong place. The way I read the error messages, the problem is that you are trying to collect resources that conflict with ordinary resources already defined on the affected nodes. Check for declarations of host resources other than the one you show above in the manifests applicable to the affected nodes. John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Hmm..
I''m not setting up local Host resources any where in the manifests, my
nodes
only collect the exported resources.
Almost all of my nodes are exporting their Host resource, and I have several
hosts that collect the resource according to their tag name:
Host <<| tag == "production-proxy" |>> {
ensure => present,
}
Host <<| tag =="backup-proxy" |>> {
ensure => present,
}
This also happens to me for an Opsviewmonitored resource I''m exporting,
which is being collected only by one server that has no local resources like
this set..
On Wed, Oct 26, 2011 at 3:16 PM, jcbollinger
<John.Bollinger@stjude.org>wrote:
>
>
> On Oct 26, 3:01 am, Galed Friedmann <galed.friedm...@onavo.com>
wrote:
> > Hey all,
> > I''m exporting a Host resource across my puppet managed client
for various
> > synchronizations across my servers. Lately I''m seeing
duplicate exported
> > resources problems and have a hard time understanding what is causing
> this,
> > I''ll appreciate your help :)
> >
> > This happens almost every day at various times, I see in the log file:
> > Oct 26 07:43:57 ubuntu puppet-master[6164]: Exported resource
> > Host[proxy1back] cannot override local resource on node lb5back
> > Oct 26 07:44:00 ubuntu puppet-master[6164]: Exported resource
> > Host[proxy1back] cannot override local resource on node lb3
> > Oct 26 07:44:00 ubuntu puppet-master[6164]: Exported resource
> > Host[proxy1back] cannot override local resource on node lb3
> > Oct 26 07:44:01 ubuntu puppet-master[6164]: Exported resource
> > Host[proxy1back] cannot override local resource on node lb3back
> > Oct 26 07:44:01 ubuntu puppet-master[6164]: Exported resource
> > Host[proxy1back] cannot override local resource on node lb3back
> >
> > As you can see it happens on several servers and not on a specific
one,
> the
> > duplicate resource happens on random servers and the error affects
random
> > servers (some of my servers are operating fine and does not report
> > duplicate errors).
> >
> > Looking at the puppet DB resources table I see this:
> > mysql> select * from resources where title =
''proxy1back'';
> >
>
+--------+------------+---------+---------+----------------+----------+------+---------------------+---------------------+
> > | id | title | restype | host_id | source_file_id | exported
|
> > line | updated_at | created_at |
> >
>
+--------+------------+---------+---------+----------------+----------+------+---------------------+---------------------+
> > | 573048 | proxy1back | Host | 39640 | 26 | 1
|
> > 63 | 2011-10-26 02:01:22 | 2011-10-26 02:01:22 |
> > | 576542 | proxy1back | Host | 40107 | 2 | 0
|
> > 63 | 2011-10-26 04:01:05 | 2011-10-26 04:01:05 |
> > | 581182 | proxy1back | Host | 40844 | 26 | 1
|
> > 63 | 2011-10-26 07:01:35 | 2011-10-26 07:01:35 |
> >
>
+--------+------------+---------+---------+----------------+----------+------+---------------------+---------------------+
> >
> > It seems like at some point the server is getting a different host_id
and
> > then exporting the resource again with the same title, which causes
the
> > duplicate. But I have no idea why..
> >
> > My Host resource looks like this:
> > @@host { "$hostname":
> > comment => "$hostname",
> > ip => "$ec2_local_ipv4",
> > name => "stable_host$proxy_id",
> > tag => "production-proxy",
> >
> > }
>
>
> I think you''re looking for the problem in the wrong place. The
way I
> read the error messages, the problem is that you are trying to collect
> resources that conflict with ordinary resources already defined on the
> affected nodes. Check for declarations of host resources other than
> the one you show above in the manifests applicable to the affected
> nodes.
>
>
> John
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
On Oct 27, 2:23 am, Galed Friedmann <galed.friedm...@onavo.com> wrote:> Hmm.. > I''m not setting up local Host resources any where in the manifests, my nodes > only collect the exported resources. > > Almost all of my nodes are exporting their Host resource, and I have several > hosts that collect the resource according to their tag name: > > Host <<| tag == "production-proxy" |>> { > ensure => present, > } > > Host <<| tag =="backup-proxy" |>> { > ensure => present, > } > > This also happens to me for an Opsviewmonitored resource I''m exporting, > which is being collected only by one server that has no local resources like > this set..Then perhaps you''re collecting the same resource twice. I wouldn''t normally expect that to be a problem, but when you add property overrides to the mix then that could change things. Even if the overrides match. That also ties in even closer to the error messages, which mention overriding. Why do you need to override ''ensure'' anyway? Can''t you just export it with that ''ensure'' value in the first place? John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
I didn''t know there was such a difference between exporting the ensure
and
setting it when collecting, I thought it was supposed to be the same thing
...
Anyways, I fixed this but I still see the error happening from time to
time, got a few of those this morning:
Oct 30 07:34:16 ubuntu puppet-master[10501]: Exported resource Host[proxy4]
cannot override local resource on node lb2
Oct 30 07:34:16 ubuntu puppet-master[10501]: Exported resource Host[proxy4]
cannot override local resource on node lb2
Oct 30 07:34:17 ubuntu puppet-master[10337]: Exported resource Host[proxy4]
cannot override local resource on node lb5
Oct 30 07:34:17 ubuntu puppet-master[10337]: Exported resource Host[proxy4]
cannot override local resource on node lb5
Oct 30 07:34:19 ubuntu puppet-master[10337]: Exported resource Host[proxy4]
cannot override local resource on node lb3back
This is the new manifest:
proxy* servers:
@@host { "$hostname":
comment => "$hostname",
ip => "$ec2_local_ipv4",
name => "stable_host$proxy_id",
tag => "production-proxy",
ensure => present,
}
lb* servers:
Host <<| tag == "production-proxy" |>> {
}
Host <<| tag =="backup-proxy" |>> {
}
Still no idea why this is happening. I have some hosts that sometime change
their IP address and I have to sync this with the other servers. Is it
possible that the cause is that there is already a host record in the
/etc/hosts file that has another IP address in it? (Although I''d assume
puppet should just change it and not take it as a local resource...)
I''d appreciate the help, quite desperate about this...
Thanks!
On Thu, Oct 27, 2011 at 3:10 PM, jcbollinger
<John.Bollinger@stjude.org>wrote:
>
>
> On Oct 27, 2:23 am, Galed Friedmann <galed.friedm...@onavo.com>
wrote:
> > Hmm..
> > I''m not setting up local Host resources any where in the
manifests, my
> nodes
> > only collect the exported resources.
> >
> > Almost all of my nodes are exporting their Host resource, and I have
> several
> > hosts that collect the resource according to their tag name:
> >
> > Host <<| tag == "production-proxy" |>> {
> > ensure => present,
> > }
> >
> > Host <<| tag =="backup-proxy" |>> {
> > ensure => present,
> > }
> >
> > This also happens to me for an Opsviewmonitored resource I''m
exporting,
> > which is being collected only by one server that has no local
resources
> like
> > this set..
>
>
> Then perhaps you''re collecting the same resource twice. I
wouldn''t
> normally expect that to be a problem, but when you add property
> overrides to the mix then that could change things. Even if the
> overrides match. That also ties in even closer to the error messages,
> which mention overriding.
>
> Why do you need to override ''ensure'' anyway?
Can''t you just export it
> with that ''ensure'' value in the first place?
>
>
> John
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
On Oct 30, 2:42 am, Galed Friedmann <galed.friedm...@onavo.com> wrote:> I didn''t know there was such a difference between exporting the ensure and > setting it when collecting, I thought it was supposed to be the same thingAs I seem to be saying frequently of late, the effect on the client should be the same if the catalog compiles, but the effect on catalog compilation is not necessarily the same. Also, it''s cleaner design- wise to collect resources without overriding their properties. Overriding has legitimate uses, but you should prefer to export the full target state of a resource when you can do so.> ... > > Anyways, I fixed this but I still see the error happening from time to > time, got a few of those this morning: > Oct 30 07:34:16 ubuntu puppet-master[10501]: Exported resource Host[proxy4] > cannot override local resource on node lb2 > Oct 30 07:34:16 ubuntu puppet-master[10501]: Exported resource Host[proxy4] > cannot override local resource on node lb2 > Oct 30 07:34:17 ubuntu puppet-master[10337]: Exported resource Host[proxy4] > cannot override local resource on node lb5 > Oct 30 07:34:17 ubuntu puppet-master[10337]: Exported resource Host[proxy4] > cannot override local resource on node lb5 > Oct 30 07:34:19 ubuntu puppet-master[10337]: Exported resource Host[proxy4] > cannot override local resource on node lb3back > > This is the new manifest: > proxy* servers: > @@host { "$hostname": > comment => "$hostname", > ip => "$ec2_local_ipv4", > name => "stable_host$proxy_id", > tag => "production-proxy", > ensure => present, > } > > lb* servers: > Host <<| tag == "production-proxy" |>> { > } > > Host <<| tag =="backup-proxy" |>> { > }If you''re not performing any actual overrides then you should remove the empty braces, too. It seems unlikely that that would solve the problem, but not completely out of the question. I''d remove for clarity, though, if for no other reason.> Still no idea why this is happening. I have some hosts that sometime change > their IP address and I have to sync this with the other servers. Is it > possible that the cause is that there is already a host record in the > /etc/hosts file that has another IP address in it? (Although I''d assume > puppet should just change it and not take it as a local resource...)It is conceivable that Puppet gets confused if in /etc/hosts it finds separate records, one having the same host name as the collected resource but a different address, and the other having the same IP address but a different name. I agree, however, that that *shouldn''t* be a problem, and I would be surprised if it were. You should be able to test that case. Do you have hosts with duplicate hostnames (perhaps in different domains)? Are any of the affected hosts flipping between being primary and backup proxies, or have they done since you turned on storeconfigs? If you clear out your storeconfig database do the problems continue and/or come back later? John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Hi again, I''m always deleting exported resources for hosts that are no longer in use, also changed what you suggested. It seems like this problem is still happening once in a while but much less than it used to so maybe things are starting to work out here. Thanks a lot for help. On Mon, Oct 31, 2011 at 5:47 PM, jcbollinger <John.Bollinger@stjude.org>wrote:> > > On Oct 30, 2:42 am, Galed Friedmann <galed.friedm...@onavo.com> wrote: > > I didn''t know there was such a difference between exporting the ensure > and > > setting it when collecting, I thought it was supposed to be the same > thing > > > As I seem to be saying frequently of late, the effect on the client > should be the same if the catalog compiles, but the effect on catalog > compilation is not necessarily the same. Also, it''s cleaner design- > wise to collect resources without overriding their properties. > Overriding has legitimate uses, but you should prefer to export the > full target state of a resource when you can do so. > > > > ... > > > > Anyways, I fixed this but I still see the error happening from time to > > time, got a few of those this morning: > > Oct 30 07:34:16 ubuntu puppet-master[10501]: Exported resource > Host[proxy4] > > cannot override local resource on node lb2 > > Oct 30 07:34:16 ubuntu puppet-master[10501]: Exported resource > Host[proxy4] > > cannot override local resource on node lb2 > > Oct 30 07:34:17 ubuntu puppet-master[10337]: Exported resource > Host[proxy4] > > cannot override local resource on node lb5 > > Oct 30 07:34:17 ubuntu puppet-master[10337]: Exported resource > Host[proxy4] > > cannot override local resource on node lb5 > > Oct 30 07:34:19 ubuntu puppet-master[10337]: Exported resource > Host[proxy4] > > cannot override local resource on node lb3back > > > > This is the new manifest: > > proxy* servers: > > @@host { "$hostname": > > comment => "$hostname", > > ip => "$ec2_local_ipv4", > > name => "stable_host$proxy_id", > > tag => "production-proxy", > > ensure => present, > > } > > > > lb* servers: > > Host <<| tag == "production-proxy" |>> { > > } > > > > Host <<| tag =="backup-proxy" |>> { > > } > > > If you''re not performing any actual overrides then you should remove > the empty braces, too. It seems unlikely that that would solve the > problem, but not completely out of the question. I''d remove for > clarity, though, if for no other reason. > > > > Still no idea why this is happening. I have some hosts that sometime > change > > their IP address and I have to sync this with the other servers. Is it > > possible that the cause is that there is already a host record in the > > /etc/hosts file that has another IP address in it? (Although I''d assume > > puppet should just change it and not take it as a local resource...) > > > It is conceivable that Puppet gets confused if in /etc/hosts it finds > separate records, one having the same host name as the collected > resource but a different address, and the other having the same IP > address but a different name. I agree, however, that that *shouldn''t* > be a problem, and I would be surprised if it were. You should be able > to test that case. > > Do you have hosts with duplicate hostnames (perhaps in different > domains)? Are any of the affected hosts flipping between being > primary and backup proxies, or have they done since you turned on > storeconfigs? If you clear out your storeconfig database do the > problems continue and/or come back later? > > > John > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.