thr3ads.net - Puppet users - [Puppet Users] Pluginsync hangs indefinitely [Aug 2011]

If this information is useful, please help other people find it:
Share via:

Nathaniel Cook

2011-Aug-08 22:52 UTC

[Puppet Users] Pluginsync hangs indefinitely

I am having an issue with puppet hanging. My system is pretty simple I
have several modules but only one that loads some extra facts. I
configure the client to have pluginsync=true in puppet.conf. On the
first run it creates and signs the certificate and the copies the one
ruby script for the facts and logs:

info: Loading downloaded plugin /var/lib/puppet/lib/facter/facts.rb

Then it hangs. Logs and the master and the client never say anything
else. The puppet process uses up about 50% cpu and never recovers. If
I kill puppet and restart it, it still hangs.

Any ideas as to why this would be happening?

Could it be related to using autosign and it being the same run that
it signs the cert?

Thanks for your help,
Nathaniel Cook

I am using puppet 2.6.9 on CentOS 5

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Craig White

2011-Aug-09 21:04 UTC

head link

Re: [Puppet Users] Pluginsync hangs indefinitely

On Aug 8, 2011, at 3:52 PM, Nathaniel Cook wrote:
> I am having an issue with puppet hanging. My system is pretty simple I
> have several modules but only one that loads some extra facts. I
> configure the client to have pluginsync=true in puppet.conf. On the
> first run it creates and signs the certificate and the copies the one
> ruby script for the facts and logs:
> 
> info: Loading downloaded plugin /var/lib/puppet/lib/facter/facts.rb
> 
> Then it hangs. Logs and the master and the client never say anything
> else. The puppet process uses up about 50% cpu and never recovers. If
> I kill puppet and restart it, it still hangs.
> 
> Any ideas as to why this would be happening?
> 
> Could it be related to using autosign and it being the same run that
> it signs the cert?
> 
> Thanks for your help,
> Nathaniel Cook
> 
> I am using puppet 2.6.9 on CentOS 5----
sounds like there''s a problem with the code in your
''fact''

Suggest that you make sure it can run in irb first before you implement.

Otherwise, you probably want to show us what the code looks like

Craig

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Nathaniel Cook

2011-Aug-09 23:11 UTC

head link

[Puppet Users] Re: Pluginsync hangs indefinitely

I have run the code by hand and it works just fine. Here it is:

#################
require ''facter''


def addFact(name , cmd)
	matches = cmd.scan(/\$\{\w+\}/)
	if ! matches.empty?
		for i in 0...matches.length
			match = matches[i]
			match = match[2..-2]
			value = Facter[match].value()
			if value == nil
				value = ''''
			end
			cmd = cmd.gsub(/\$\{#{Regexp.escape(match)}\}/, value)
		end
	end
	Facter.add(name) do
		setcode do
			`#{cmd}`.chomp
		end
	end
end

#
# List of facts to define,
#
# Can have references to previous facts by using the ${var} syntax
#
facts = {
		''ec2_region'' => ''cat
/opt/aws/info/region'',
		''ec2_instance_id'' => ''cat
/opt/aws/info/instance-id'',
		''ec2_url'' => ''echo
"https://ec2.${ec2_region}.amazonaws.com"'',
		''ec2_zone'' => "ec2-metadata -z | awk
{''print $NF''}"
		}


if ! Facter[''ec2_ami_id'']
    puts "Not loading aws_facts"
else

	keys = facts.keys

	for key in 0...keys.length
		fact = keys[key]
		addFact(fact, facts[fact])
	end

end

##############

Its a simple script that sets up some facts about aws ec2 instances.
Hope this helps.

Nathaniel

On Aug 9, 3:04 pm, Craig White <craig.wh...@ttiltd.com>
wrote:> On Aug 8, 2011, at 3:52 PM, Nathaniel Cook wrote:
>
>
>
>
>
>
>
>
>
> > I am having an issue with puppet hanging. My system is pretty simple I
> > have several modules but only one that loads some extra facts. I
> > configure the client to have pluginsync=true in puppet.conf. On the
> > first run it creates and signs the certificate and the copies the one
> > ruby script for the facts and logs:
>
> > info: Loading downloaded plugin /var/lib/puppet/lib/facter/facts.rb
>
> > Then it hangs. Logs and the master and the client never say anything
> > else. The puppet process uses up about 50% cpu and never recovers. If
> > I kill puppet and restart it, it still hangs.
>
> > Any ideas as to why this would be happening?
>
> > Could it be related to using autosign and it being the same run that
> > it signs the cert?
>
> > Thanks for your help,
> > Nathaniel Cook
>
> > I am using puppet 2.6.9 on CentOS 5
>
> ----
> sounds like there''s a problem with the code in your
''fact''
>
> Suggest that you make sure it can run in irb first before you implement.
>
> Otherwise, you probably want to show us what the code looks like
>
> Craig
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

jcbollinger

2011-Aug-10 14:16 UTC

head link

[Puppet Users] Re: Pluginsync hangs indefinitely

I do not think your problem is related to autosign, but surely that
would be easy enough to test.  More below.

On Aug 9, 6:11 pm, Nathaniel Cook <nvcoo...@gmail.com>
wrote:> I have run the code by hand and it works just fine. Here it is:
>
> #################
> require ''facter''
>
> def addFact(name , cmd)
>         matches = cmd.scan(/\$\{\w+\}/)
>         if ! matches.empty?
>                 for i in 0...matches.length
>                         match = matches[i]
>                         match = match[2..-2]
>                         value = Facter[match].value()
>                         if value == nil
>                                 value = ''''
>                         end
>                         cmd = cmd.gsub(/\$\{#{Regexp.escape(match)}\}/,
value)
>                 end
>         end
>         Facter.add(name) do
>                 setcode do
>                         `#{cmd}`.chomp
>                 end
>         end
> end
>
> #
> # List of facts to define,
> #
> # Can have references to previous facts by using the ${var} syntax
> #
> facts = {
>                 ''ec2_region'' => ''cat
/opt/aws/info/region'',
>                 ''ec2_instance_id'' => ''cat
/opt/aws/info/instance-id'',
>                 ''ec2_url'' => ''echo
"https://ec2.${ec2_region}.amazonaws.com"'',
>                 ''ec2_zone'' => "ec2-metadata -z |
awk {''print $NF''}"
>                 }
>
> if ! Facter[''ec2_ami_id'']
>     puts "Not loading aws_facts"
> else
>
>         keys = facts.keys
>
>         for key in 0...keys.length
>                 fact = keys[key]
>                 addFact(fact, facts[fact])
>         end
>
> end
>
> ##############
>
> Its a simple script that sets up some facts about aws ec2 instances.
> Hope this helps.

I am not convinced that your fact code is in the clear, despite your
success running it outside a Puppet context.  Indeed there''s a lot in
it that I find suspicious, questionable, and / or unsafe, even if it
turns out not to be responsible for your hang problem:

1) You use a lot of variables with rather generic names, in scopes
where they conceivably could collide with variables belonging to
Facter proper or to Puppet.

2) I think your setup for interpolating fact values into other fact
values is too clever by half.  I recommend you reserve interpolation
for Puppet itself to perform.  In particular, the ec2_url fact
provides nothing whatever that wouldn''t more appropriately be done by
Puppet.

3) If you do retain Facter-side interpolation, then you probably need
to do it in the fact code itself (i.e. inside the setcode block), as I
would not expect it to be safe to assume that *any* fact values were
available at the time that your addFact() funciton is executed.

4) If you do retain Facter-side interpolation, then you should also
take the evaluation order of facts into consideration.  I do not think
it safe to assume that that order will be the same as the order in
which your facts are added, and it certainly is not safe to assume
that your facts will be added in the order they are listed in your
hash.  If you get lucky, then it might end up working for you --
perhaps even reliably -- but I could easily imagine that changing with
addition of more custom facts or with a Facter upgrade.  For me, this
would be a show-stopper.

5) The reasoning in (4) also suggests that it may not be safe to use
facts within Facter to determine whether to add other facts (which you
do).  Even if it happens to work now, I would never rely on such
behavior without solid API documentation supporting it.  (Is there
any?)

Here''s how I would write the facts you showed:

===
Facter.add(''ec2_region'') do
    setcode do
        `cat /opt/aws/info/region || echo`.chomp
    end
end

Facter.add(''ec2_instance_id'') do
    setcode do
        `cat /opt/aws/info/instance-id || echo`.chomp
    end
end

# ec2_url intentionally omitted

Facter.add(''ec2_zone'') do
    setcode do
        # Note: you should consider using an explicit path
        # to some or all of the system commands below:
        `ec2-metadata -z | awk {''print $NF''} || echo`.chomp
    end
end

===
It''s short, sweet, easy to read, and easy to maintain.  It also has no
risk of variable collisions, no evaluation-order dependencies, and it
works on all nodes, regardless of whether they are ec2.

So, does Puppet still hang if you remove the custom facts?  (You''ll
need to both disable pluginsync and manually remove the custom fact
code from the client, or else make the client sync up a no-op
version.)  If not, then does it hang with my version of them?  This
should give you a good start on debugging.

John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Nathaniel Cook

2011-Aug-10 15:08 UTC

head link

[Puppet Users] Re: Pluginsync hangs indefinitely

Thanks John that did it, your script worked great. Now that I think
about it I see how using facts to generate more facts could cause
these issues.

Nathaniel

On Aug 10, 8:16 am, jcbollinger <John.Bollin...@stJude.org>
wrote:> I do not think your problem is related to autosign, but surely that
> would be easy enough to test.  More below.
>
> On Aug 9, 6:11 pm, Nathaniel Cook <nvcoo...@gmail.com> wrote:
>
>
>
>
>
>
>
>
>
> > I have run the code by hand and it works just fine. Here it is:
>
> > #################
> > require ''facter''
>
> > def addFact(name , cmd)
> >         matches = cmd.scan(/\$\{\w+\}/)
> >         if ! matches.empty?
> >                 for i in 0...matches.length
> >                         match = matches[i]
> >                         match = match[2..-2]
> >                         value = Facter[match].value()
> >                         if value == nil
> >                                 value = ''''
> >                         end
> >                         cmd =
cmd.gsub(/\$\{#{Regexp.escape(match)}\}/, value)
> >                 end
> >         end
> >         Facter.add(name) do
> >                 setcode do
> >                         `#{cmd}`.chomp
> >                 end
> >         end
> > end
>
> > #
> > # List of facts to define,
> > #
> > # Can have references to previous facts by using the ${var} syntax
> > #
> > facts = {
> >                 ''ec2_region'' => ''cat
/opt/aws/info/region'',
> >                 ''ec2_instance_id'' =>
''cat /opt/aws/info/instance-id'',
> >                 ''ec2_url'' => ''echo
"https://ec2.${ec2_region}.amazonaws.com"'',
> >                 ''ec2_zone'' => "ec2-metadata
-z | awk {''print $NF''}"
> >                 }
>
> > if ! Facter[''ec2_ami_id'']
> >     puts "Not loading aws_facts"
> > else
>
> >         keys = facts.keys
>
> >         for key in 0...keys.length
> >                 fact = keys[key]
> >                 addFact(fact, facts[fact])
> >         end
>
> > end
>
> > ##############
>
> > Its a simple script that sets up some facts about aws ec2 instances.
> > Hope this helps.
>
> I am not convinced that your fact code is in the clear, despite your
> success running it outside a Puppet context.  Indeed there''s a lot
in
> it that I find suspicious, questionable, and / or unsafe, even if it
> turns out not to be responsible for your hang problem:
>
> 1) You use a lot of variables with rather generic names, in scopes
> where they conceivably could collide with variables belonging to
> Facter proper or to Puppet.
>
> 2) I think your setup for interpolating fact values into other fact
> values is too clever by half.  I recommend you reserve interpolation
> for Puppet itself to perform.  In particular, the ec2_url fact
> provides nothing whatever that wouldn''t more appropriately be done
by
> Puppet.
>
> 3) If you do retain Facter-side interpolation, then you probably need
> to do it in the fact code itself (i.e. inside the setcode block), as I
> would not expect it to be safe to assume that *any* fact values were
> available at the time that your addFact() funciton is executed.
>
> 4) If you do retain Facter-side interpolation, then you should also
> take the evaluation order of facts into consideration.  I do not think
> it safe to assume that that order will be the same as the order in
> which your facts are added, and it certainly is not safe to assume
> that your facts will be added in the order they are listed in your
> hash.  If you get lucky, then it might end up working for you --
> perhaps even reliably -- but I could easily imagine that changing with
> addition of more custom facts or with a Facter upgrade.  For me, this
> would be a show-stopper.
>
> 5) The reasoning in (4) also suggests that it may not be safe to use
> facts within Facter to determine whether to add other facts (which you
> do).  Even if it happens to work now, I would never rely on such
> behavior without solid API documentation supporting it.  (Is there
> any?)
>
> Here''s how I would write the facts you showed:
>
> ===>
> Facter.add(''ec2_region'') do
>     setcode do
>         `cat /opt/aws/info/region || echo`.chomp
>     end
> end
>
> Facter.add(''ec2_instance_id'') do
>     setcode do
>         `cat /opt/aws/info/instance-id || echo`.chomp
>     end
> end
>
> # ec2_url intentionally omitted
>
> Facter.add(''ec2_zone'') do
>     setcode do
>         # Note: you should consider using an explicit path
>         # to some or all of the system commands below:
>         `ec2-metadata -z | awk {''print $NF''} ||
echo`.chomp
>     end
> end
>
> ===>
> It''s short, sweet, easy to read, and easy to maintain.  It also
has no
> risk of variable collisions, no evaluation-order dependencies, and it
> works on all nodes, regardless of whether they are ec2.
>
> So, does Puppet still hang if you remove the custom facts?
 (You''ll
> need to both disable pluginsync and manually remove the custom fact
> code from the client, or else make the client sync up a no-op
> version.)  If not, then does it hang with my version of them?  This
> should give you a good start on debugging.
>
> John
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Puppet users - Aug 2011 - Pluginsync hangs indefinitely

[Puppet Users] Pluginsync hangs indefinitely

Re: [Puppet Users] Pluginsync hangs indefinitely

[Puppet Users] Re: Pluginsync hangs indefinitely

[Puppet Users] Re: Pluginsync hangs indefinitely

[Puppet Users] Re: Pluginsync hangs indefinitely