Something I''ve found out using Puppet during our code testing. When making use of the Class type with require => or before => statements puppet run time can dramatically increase. We''ve seen 30-60+ second increase in puppet run times because of require/before a Class type. This is because puppet will require running portions you specify in serial, instead of some items running in parallel. So if you are requiring specific items to run before your code, try to require/ before specific items within a Class, instead of the entire Class itself. Logically this makes sense but I assume most people haven''t thought about this. Hope this helps. -L -- Larry Ludwig Empowering Media 1-866-792-0489 x600 Managed Xen VPSes http://www.hostcube.com/ --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Tue, Jul 15, 2008 at 7:35 AM, Larry Ludwig <larrylud@gmail.com> wrote:> > Something I''ve found out using Puppet during our code testing. When > making use of the Class type with require => or before => statements > puppet run time can dramatically increase. We''ve seen 30-60+ second > increase in puppet run times because of require/before a Class type. > This is because puppet will require running portions you specify in > serial, instead of some items running in parallel. So if you are > requiring specific items to run before your code, try to require/ > before specific items within a Class, instead of the entire Class > itself. > > Logically this makes sense but I assume most people haven''t thought > about this. >Did not even realize you could require at the class level. Good to know you can do it and better to know what it can cost you. Evan --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Tue, Jul 15, 2008 at 5:35 AM, Larry Ludwig <larrylud@gmail.com> wrote:> > Something I''ve found out using Puppet during our code testing. When > making use of the Class type with require => or before => statements > puppet run time can dramatically increase. We''ve seen 30-60+ second > increase in puppet run times because of require/before a Class type. > This is because puppet will require running portions you specify in > serial, instead of some items running in parallel. So if you are > requiring specific items to run before your code, try to require/ > before specific items within a Class, instead of the entire Class > itself. > > Logically this makes sense but I assume most people haven''t thought > about this. > > Hope this helps.I''m pretty sure this is un-true. Puppet isn''t ever running anything in parallel at this point - it''s running them serially. What is happening is the require => Class[foo] is essentially requiring all the objects in the class, which is probably making it take longer to render the directed graph. I say this without any actual digging or benchmarking. How much of this is node compile time instead of run time? Adam> > -L > -- > Larry Ludwig > Empowering Media > 1-866-792-0489 x600 > Managed Xen VPSes > http://www.hostcube.com/ > > >-- HJK Solutions - We Launch Startups - http://www.hjksolutions.com Adam Jacob, Senior Partner T: (206) 508-4759 E: adam@hjksolutions.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
> I say this without any actual digging or benchmarking. How much of this is > node compile time instead of run time?compile time is only a 1 or 2 seconds. From my tests where we chained our many Classes together, the execute time took over 500+ seconds. All I changed was using From require=> Class to require=> File or Exec or Package, etc.. within that class. Execute time then came back down to our typical 60-70 seconds. Compile stayed the same. So something by using the Class function causes it. If not not executing in parallel, then it''s certainly executing faster. At least that''s what I saw watching the log file spew out the executed types. -L --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Sorry for our most advanced config (which I was testing against), 15-30 seconds was the max compile times. The compile times did not have any noticeable change when replacing Class. -L On Jul 15, 10:40 pm, Larry Ludwig <larry...@gmail.com> wrote:> > I say this without any actual digging or benchmarking. How much of this is > > node compile time instead of run time? > > compile time is only a 1 or 2 seconds. > > From my tests where we chained our many Classes together, the execute > time took over 500+ seconds. All I changed was using From require=> > Class to require=> File or Exec or Package, etc.. within that class. > Execute time then came back down to our typical 60-70 seconds. > > Compile stayed the same. > > So something by using the Class function causes it. If not not > executing in parallel, then it''s certainly executing faster. At least > that''s what I saw watching the log file spew out the executed types. > > -L--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Tue, Jul 15, 2008 at 7:40 PM, Larry Ludwig <larrylud@gmail.com> wrote:> compile time is only a 1 or 2 seconds. > > From my tests where we chained our many Classes together, the execute > time took over 500+ seconds. All I changed was using From require=> > Class to require=> File or Exec or Package, etc.. within that class. > Execute time then came back down to our typical 60-70 seconds. > > Compile stayed the same. > > So something by using the Class function causes it. If not not > executing in parallel, then it''s certainly executing faster. At least > that''s what I saw watching the log file spew out the executed types.This is a bug someplace, that''s causing an increase in run-time processing if every resource has multiple required resources. You should file it in Redmine, and we should track against it. The puppetd binary is in fact single threaded when it''s processing your compiled manifest, so it''s not any increased degree of parallelism that''s getting you this speed increase/decrease. Adam -- HJK Solutions - We Launch Startups - http://www.hjksolutions.com Adam Jacob, Senior Partner T: (206) 508-4759 E: adam@hjksolutions.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
On Jul 15, 2008, at 9:40 PM, Larry Ludwig wrote:> > >> I say this without any actual digging or benchmarking. How much of >> this is >> node compile time instead of run time? > > compile time is only a 1 or 2 seconds. > > From my tests where we chained our many Classes together, the execute > time took over 500+ seconds. All I changed was using From require=> > Class to require=> File or Exec or Package, etc.. within that class. > Execute time then came back down to our typical 60-70 seconds. > > Compile stayed the same. > > So something by using the Class function causes it. If not not > executing in parallel, then it''s certainly executing faster. At least > that''s what I saw watching the log file spew out the executed types.Is this 0.24.4? I replaced the graph library some time around 0.24.2 with one that performs dramatically, painfully better. I hope you''re running the old version... As Adam said, Puppet never runs anything in parallel, so the only thing this could affect is either the time it takes to sort the graph or the time it takes to manage passing events to dependencies. It should be straightforward to see if it''s the graph -- wrap the ''topsort'' call in Transaction#prepare with some timing statements. As Adam said, though, graph timing will always scale somewhat linearly with the number of relationships (unless your algorithm is particularly horrible), so using class relationshpis can kinda hurt if you do too many of them. -- You only have to be open minded if you''re wrong. --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
> Is this 0.24.4? > > I replaced the graph library some time around 0.24.2 with one that > performs dramatically, painfully better. I hope you''re running the > old version...I know you aren''t gonna want to hear this but it''s 0.24.4> > As Adam said, Puppet never runs anything in parallel, so the only > thing this could affect is either the time it takes to sort the graph > or the time it takes to manage passing events to dependencies. It > should be straightforward to see if it''s the graph -- wrap the > ''topsort'' call in Transaction#prepare with some timing statements.ok.> As Adam said, though, graph timing will always scale somewhat linearly > with the number of relationships (unless your algorithm is > particularly horrible), so using class relationshpis can kinda hurt if > you do too many of them.When I have the time I will submit the ticket. -L --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---