Puppet users - Nov 2012 - 1.8.7 => 1.9.3 migration, compilation times raised twice!?

i''ve made a migration on our puppetmasters yesterday, hoping it will
run
faster and/or make less load on machines with newer ruby. but actual result 
is just opposite, our compilation times went up a lot, ie:

Nov 27 14:30:12 punch puppet-master[15293]: Compiled catalog for lb1 in 
environment production in 183.42 seconds
Nov 27 15:38:42 punch puppet-master[24399]: Compiled catalog for lb1 in 
environment production in 158.49 seconds
Nov 27 15:51:19 punch puppet-master[11963]: Compiled catalog for lb1 in 
environment production in 175.00 seconds
Nov 27 16:04:15 punch puppet-master[15264]: Compiled catalog for lb1 in 
environment production in 120.36 seconds
Nov 27 19:25:52 punch puppet-master[552]: Compiled catalog for lb1 in 
environment production in 281.54 seconds
Nov 27 20:20:19 punch puppet-master[2273]: Compiled catalog for lb1 in 
environment production in 239.05 seconds
Nov 27 22:26:03 punch puppet-master[4919]: Compiled catalog for lb1 in 
environment production in 275.38 seconds
Nov 28 00:55:24 punch puppet-master[12293]: Compiled catalog for lb1 in 
environment production in 267.91 seconds

guess when migration has occured. this machine (lb1) has ~2000 resources.

i''m a bit worried about scalability... we can go back to 1.8.7 now, but
it
won''t be supported in 7 months (ie. no security updates).

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/NV-H40aVfrcJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Wed, Nov 28, 2012 at 10:34 AM, asq <asquelt@gmail.com> wrote:
> i''ve made a migration on our puppetmasters yesterday, hoping it
will run
> faster and/or make less load on machines with newer ruby. but actual result
> is just opposite, our compilation times went up a lot, ie:
>
> Nov 27 14:30:12 punch puppet-master[15293]: Compiled catalog for lb1 in
> environment production in 183.42 seconds
> Nov 27 15:38:42 punch puppet-master[24399]: Compiled catalog for lb1 in
> environment production in 158.49 seconds
> Nov 27 15:51:19 punch puppet-master[11963]: Compiled catalog for lb1 in
> environment production in 175.00 seconds
> Nov 27 16:04:15 punch puppet-master[15264]: Compiled catalog for lb1 in
> environment production in 120.36 seconds
> Nov 27 19:25:52 punch puppet-master[552]: Compiled catalog for lb1 in
> environment production in 281.54 seconds
> Nov 27 20:20:19 punch puppet-master[2273]: Compiled catalog for lb1 in
> environment production in 239.05 seconds
> Nov 27 22:26:03 punch puppet-master[4919]: Compiled catalog for lb1 in
> environment production in 275.38 seconds
> Nov 28 00:55:24 punch puppet-master[12293]: Compiled catalog for lb1 in
> environment production in 267.91 seconds
>
> guess when migration has occured. this machine (lb1) has ~2000 resources.
>
> i''m a bit worried about scalability... we can go back to 1.8.7
now, but it
> won''t be supported in 7 months (ie. no security updates).
>
>  What version of Puppet are you on? Where did you get your 1.9.3?
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/puppet-users/-/NV-H40aVfrcJ.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

W dniu środa, 28 listopada 2012 22:57:40 UTC+1 użytkownik Michael Stanhke 
napisał:
>
>  What version of Puppet are you on? Where did you get your 1.9.3? 
>
3.0.1 and 1.9.3p327 from rvm. 

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/t2QS0K1gkK0J.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Thu, Nov 29, 2012 at 12:24 AM, asq <asquelt@gmail.com>
wrote:
> W dniu środa, 28 listopada 2012 22:57:40 UTC+1 użytkownik Michael Stanhke
> napisał:
>>
>>  What version of Puppet are you on? Where did you get your 1.9.3?
>
> 3.0.1 and 1.9.3p327 from rvm.
Ouch, do you happen to have a large number of ruby Gem''s installed?

-Jeff

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

W dniu czwartek, 29 listopada 2012 17:42:48 UTC+1 użytkownik Jeff McCune 
napisał:
>
> On Thu, Nov 29, 2012 at 12:24 AM, asq <asq...@gmail.com
<javascript:>>
> wrote: 
> > W dniu środa, 28 listopada 2012 22:57:40 UTC+1 użytkownik Michael 
> Stanhke 
> > napisał: 
> >> 
> >>  What version of Puppet are you on? Where did you get your 1.9.3? 
> > 
> > 3.0.1 and 1.9.3p327 from rvm. 
>
> Ouch, do you happen to have a large number of ruby Gem''s
installed?
>
> -Jeff 
>
yes, a bunch of gems for Foreman...

 <5>root@punch(8)~$ gem stale | wc -l
91

but it''s the same pack that we''ve used for 1.8.7.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/JJhz2RiMjBIJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Thu, Nov 29, 2012 at 8:56 AM, asq <asquelt@gmail.com> wrote:
> W dniu czwartek, 29 listopada 2012 17:42:48 UTC+1 użytkownik Jeff McCune
> napisał:
>
>> On Thu, Nov 29, 2012 at 12:24 AM, asq <asq...@gmail.com> wrote:
>> > W dniu środa, 28 listopada 2012 22:57:40 UTC+1 użytkownik Michael
>> Stanhke
>> > napisał:
>> >>
>> >>  What version of Puppet are you on? Where did you get your
1.9.3?
>> >
>> > 3.0.1 and 1.9.3p327 from rvm.
>>
>> Ouch, do you happen to have a large number of ruby Gem''s
installed?
>>
>> -Jeff
>>
>
> yes, a bunch of gems for Foreman...
>
It might not be the cause, but one related change we made in Puppet 3 is
that we scan Ruby gem''s and will load extensions contained within them.
 This is #7788 [1]  This change in behavior could be causing the Puppet
autoloader to perform a _lot_ of additional stat() system calls.  Could you
run strace, truss, or dtrace and compare the number of system calls between
Puppet 2.7 and 3.0 in your environment?

[1] http://projects.puppetlabs.com/issues/7788

-Jeff

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

W dniu czwartek, 29 listopada 2012 18:23:23 UTC+1 użytkownik Jeff McCune 
napisał:
>
> It might not be the cause, but one related change we made in Puppet 3 is 
> that we scan Ruby gem''s and will load extensions contained within
them.
>  This is #7788 [1]  This change in behavior could be causing the Puppet 
> autoloader to perform a _lot_ of additional stat() system calls.  Could you
> run strace, truss, or dtrace and compare the number of system calls between
> Puppet 2.7 and 3.0 in your environment?

not sure if it''s about 2.7 => 3.0 migration, as we did it month ago
with no
aparent degradation (but also no such load decrease as on some slides from 
PC).
now we just switched to newer ruby and passenger (1.8.7 => 1.9.3p327 and 
3.0.2 => 3.0.17).

i''ll try to do some tests tomorrow but i need to get both instances up
at
the same server.

another thing that is involved (and suspected) is hiera, as compilation 
time deltas differ on different groups of nodes, and it might be related to 
how much of modules with hiera lookups are included on mostly affected 
nodes (ie. 120s => 240s on our load balancers).
another thing is puppetdb, but nodes that collect resources, which should 
impose highest load on puppetdb are not the ones which are much affected 
(ie. 320s => 360s increase on our nagios server).

is it possible to do evaltrace on puppetmaster?

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/ShLDKg4XRUkJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.