As many learning Puppet for the first time, the fact that the order of actions is undefined unless specific metaparameters like ''require'' are used. Fine. This got me to thinking. The GNU make program has the "-j" option, which allows make to start more than one action in parallel if the actions are at the same dependency level. I''ve used this option on a 48-core machine to great benefit. So, why can''t there be a similar option in the puppet agent? I can easily imagine how this could substantially reduce the length of time for a puppet run. (The make "-j" option allows an optional numeric value, which, if given, is the maximum number of actions that can be run in parallel. If no numeric is given, then there''s no limit to the number of parallel actions). I did a quick review of the Puppet manual but I didn''t see anything like this. Am I missing something? Is this a good idea? Cordially, Jon Forrest -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
An interesting thought. If you enable graph output in your puppet.conf
http://docs.puppetlabs.com/references/stable/configuration.html#graph
you can identify those sub-graphs that could run in parallel pretty
easily, just by eyeballing it. Parallel execution is wonderful for an
optimizing compiler because you''re strongly CPU limited. I''d
be
willing to bet that a parallel puppet agent would be of less use: once
the catalog is compiled on the master, puppet''s execution time is
going to be limited by network and disk IO. Probably. Running puppet
agent through DTrace/SystemTap would be instructive.
Also, some resources are implicitly exclusive: two Packages with
''apt''
providers cannot be run in parallel because Debian/Ubuntu keeps a
global lock on... hmm, I''ve forgotten what, exactly. It''s
there,
though. Ensuring that implicitly parallel resources block one another
would increase the implementation complexity of Puppet, of end-user
modules or, as is likely, both.
Aside from all other considerations, multi-core parallelism in MRI is
not so great. JRuby''s better, being hosted on the JVM, but, well,
MRI''s RAM consumption is already bad enough.
On Sun, Mar 4, 2012 at 10:53 PM, Jon Forrest <nobozo@gmail.com>
wrote:> As many learning Puppet for the first time, the fact that the
> order of actions is undefined unless specific metaparameters
> like ''require'' are used. Fine.
>
> This got me to thinking. The GNU make program has the "-j"
> option, which allows make to start more than one action
> in parallel if the actions are at the same dependency level.
> I''ve used this option on a 48-core machine to great benefit.
>
> So, why can''t there be a similar option in the puppet agent?
> I can easily imagine how this could substantially reduce the
> length of time for a puppet run.
>
> (The make "-j" option allows an optional numeric value, which, if
> given, is the maximum number of actions that can be run
> in parallel. If no numeric is given, then there''s no limit
> to the number of parallel actions).
>
> I did a quick review of the Puppet manual but I didn''t see
> anything like this. Am I missing something? Is this a good
> idea?
>
> Cordially,
> Jon Forrest
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
--
Brian L. Troutwine
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
On 3/4/2012 8:12 PM, Brian Troutwine wrote: (First of all, sorry for the jumbled first paragraph. I was trying to do two things at once).> An interesting thought. If you enable graph output in your puppet.conf > > http://docs.puppetlabs.com/references/stable/configuration.html#graph > > you can identify those sub-graphs that could run in parallel pretty > easily, just by eyeballing it. Parallel execution is wonderful for an > optimizing compiler because you''re strongly CPU limited. I''d be > willing to bet that a parallel puppet agent would be of less use: once > the catalog is compiled on the master, puppet''s execution time is > going to be limited by network and disk IO. Probably. Running puppet > agent through DTrace/SystemTap would be instructive.My intuition, albeit as a novice, makes me question this. Just look at how well parallel make improves things. There would probably be a point of diminishing returns, or even negative returns, but that''s true for "make -j" also. Just "puppet -j2" could result in enough of a benefit to make this approach worthwhile.> Also, some resources are implicitly exclusive: two Packages with ''apt'' > providers cannot be run in parallel because Debian/Ubuntu keeps a > global lock on... hmm, I''ve forgotten what, exactly. It''s there, > though. Ensuring that implicitly parallel resources block one another > would increase the implementation complexity of Puppet, of end-user > modules or, as is likely, both.I see what you mean. The same is true for yum, which is what I''m most familiar with. Maybe puppet could do something so that these kind of inherent sequential processes with yum/apt (and anything else) could be reflected in the graph.> Aside from all other considerations, multi-core parallelism in MRI is > not so great. JRuby''s better, being hosted on the JVM, but, well, > MRI''s RAM consumption is already bad enough.I''m talking about process-level parallelism so I don''t think the virtual machine implementation matters here. Jon -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.