Puppet users - Mar 2010 - puppet very slow to launch after an upgrade?

I recently found a machine of mine that was still running puppet 0.24.x,
so I upgraded it to 0.25.4 (from the debian git repo). Now, puppet takes
ages and ages to launch.

On a machine that''s only ever had puppet 0.25.x installed:

  seph@newer$ puppet --version
  0.25.4
  seph@newer$ time puppet -v /dev/null
  info: Applying configuration version ''1269437981''

  real  0m1.355s
  user  0m0.780s
  sys   0m0.456s

Compared to this machine that started at 0.24:

  root@older# time puppet --version
  0.25.4

  real    0m41.683s
  user    0m0.352s
  sys     0m0.152s

  root@older# puppet -v /dev/null 
  info: Loading facts in virtual
  info: Loading facts in acpi_available
  info: Loading facts in interfaces
  sh: Syntax error: Bad fd number
  sh: Syntax error: Bad fd number
  info: Applying configuration version ''1269437572''

  real  9m13.255s
  user  0m0.616s
  sys   0m0.516s

Granted these machines are different, but this is beyond what I can
reasonable account for. 

I get pretty different looking things with --debug added. But on the
slow one, most of the time is spent before any output is printed. And
those "bad fd number" lines remind me of a 0.24 bug. 

So I wonder if there''s some droppings left from 0.24? Or if
there''s some
way to approach debugging this?

Anyone have a suggestion?

seph

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

> Granted these machines are different, but this is beyond what I can
> reasonable account for.
definitely
> I get pretty different looking things with --debug added. But on the
> slow one, most of the time is spent before any output is printed. And
> those "bad fd number" lines remind me of a 0.24 bug.
>
> So I wonder if there''s some droppings left from 0.24? Or if
there''s some
> way to approach debugging this?
>
> Anyone have a suggestion?
this looks more like a facter issue. did you upgrade facter as well?  
how fast is it to run facter?

cheers pete

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Peter Meier <peter.meier@immerda.ch> writes:
> this looks more like a facter issue. did you upgrade facter as well?
> how fast is it to run facter?
Excellent suggestion. The new machine takes .8s for a facter run, where
the old one takes just under 2min. Not sure if that''s enough to account
for it, but it seems related.

Both machines are running facter 1.5.7ubuntu1, which I don''t think is
very old.

seph


-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

>> this looks more like a facter issue. did you upgrade facter as well?
>> how fast is it to run facter?
>
> Excellent suggestion. The new machine takes .8s for a facter run, where
> the old one takes just under 2min. Not sure if that''s enough to
account
> for it, but it seems related.
>
> Both machines are running facter 1.5.7ubuntu1, which I don''t think
is
> very old.
actually the most recent one.

do you have any idea why collecting facts could take that long? I  
assume that it runs somewhere in a timeout. something like io-wait on  
a mount etc.

cheers pete

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Poking around a little more, I tried purging facter and puppet and
re-installing them. The re-installed facter and puppet is twice as fast,
but that''s still slow. (1min for a facter run, and 5min for a puppet
run)

Interestingly I cannot successfully retrieve the signed cert from the
puppetmaster. I was able to connect to submit one, and I can connect
with openssl, but puppet just reports failed to retrieve cert.

This leads me to suspect some kind of network/firewall issue, but I
don''t know why openssl would connect, and how the puppetmaster got the
cert request, if there''s a firewall issue.

seph

Peter Meier <peter.meier@immerda.ch> writes:
>>> this looks more like a facter issue. did you upgrade facter as
well?
>>> how fast is it to run facter?
>>
>> Excellent suggestion. The new machine takes .8s for a facter run, where
>> the old one takes just under 2min. Not sure if that''s enough
to account
>> for it, but it seems related.
>>
>> Both machines are running facter 1.5.7ubuntu1, which I don''t
think is
>> very old.
>
> actually the most recent one.
>
> do you have any idea why collecting facts could take that long? I
> assume that it runs somewhere in a timeout. something like io-wait on
> a mount etc.
>
> cheers pete
>
> -- 
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

(whoops, forgot to reply to your actual query)

Peter Meier <peter.meier@immerda.ch> writes:
> actually the most recent one.
well, mostly. There''s this is actually a couple patches back. I think
there''s a patch level that isn''t showing in the --version
output.
> do you have any idea why collecting facts could take that long? I
> assume that it runs somewhere in a timeout. something like io-wait on
> a mount etc.
No clue. I was poking at that. If I invoke facter asking for a specific
fact. Sometimes it''s slow, and sometimes it''s fast. Looking at
the
uptime fact, it takes about .1s to run on most of my machines. On this
older machine it takes 1s. 

Looking at uptime.rb, it seems to either shell out to uptime or cat
/proc/uptime. On all machines (both the newer and the older) those
commands return in .003s. 


Aaaaand, on the theory that all of this pointed to some weird system
stuff, I rebooted. Things seem to be running at normal speed. I''m kinda
weirded out by this fix.

seph

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 3/24/2010 5:26 PM, seph wrote:
> Aaaaand, on the theory that all of this pointed to some weird system
> stuff, I rebooted. Things seem to be running at normal speed. I''m
kinda
> weirded out by this fix.
Perhaps some rogue process which ate up resources (swap, cpu, io) in the 
background?

Regards, David

-- 
dasz.at OG              Tel: +43 (0)664 2602670     Web: http://dasz.at
Klosterneuburg                                         UID: ATU64260999

        FB-Nr.: FN 309285 g          FB-Gericht: LG Korneuburg

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

David Schmitt <david@dasz.at> writes:
> On 3/24/2010 5:26 PM, seph wrote:
>> Aaaaand, on the theory that all of this pointed to some weird system
>> stuff, I rebooted. Things seem to be running at normal speed.
I''m kinda
>> weirded out by this fix.
>
> Perhaps some rogue process which ate up resources (swap, cpu, io) in
> the background?
Clearly something like that must have been happening. Though things like
uptime, top, etc all reported fine. I might just chalk it up to a xen
kernel bug and move on. 

seph

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.