Hello, I have been running multiple exec scripts that could run in parallel. Does puppet run as much as possible in parallel? If it does not, is there a way to specify that actions may be run in parallel? Also, is puppet a multiple or single thread application? Thank you, -- Rob -- ____________________________________________________________________________________ Sucker-punch spam with award-winning protection. Try the free Yahoo! Mail Beta. http://advision.webevents.yahoo.com/mailbeta/features_spam.html _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
On Jan 25, 2007, at 3:11 PM, Robert Mombro wrote:> Hello, > > I have been running multiple exec scripts that could run in > parallel. Does puppet run as much as possible in parallel? If it > does not, is there a way to specify that actions may be run in > parallel?Puppet makes no attempt to parallelize, although I''ve been fearing the day someone would ask that. At this point, the internal graph library is such that it would at least be possible, but I expect that it would be a ton of work and I''m unlikely to start it without someone sponsoring the effort.> Also, is puppet a multiple or single thread application?Single-threaded, essentially. Ruby does not yet have native thread support, so even if I acted like it were multi-threaded, it would all still run within a single native thread. -- Everything that is really great and inspiring is created by the individual who can labor in freedom. -- Albert Einstein --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Thu, Jan 25, 2007 at 03:16:01PM -0600, Luke Kanies wrote:> At this point, the internal graph library is such that it would at > least be possible, but I expect that it would be a ton of work and > I''m unlikely to start it without someone sponsoring the effort.> [...]> Single-threaded, essentially. Ruby does not yet have native thread > support, so even if I acted like it were multi-threaded, it would all > still run within a single native thread.If I were to do this myself, I''d personally use a "pre-forking", where the children would act as a worker pool, collecting jobs from, and reporting completion and success or failure to, the managing parent process. This manager process would then handle graph traversal. On the other hand, I''d personally say that if your puppet configuration is coupled in such a manner that you have lots of long-running tasks, then it might be worth re-thinking the way it''s done. For example, if you''ve a package with a post-install script that spends 5 minutes downloading updates, then it might be worth changing the way the package handles those updates. -- Ceri Storey <cez@necrofish.org.uk> ''What I really want is "apt-get smite"'' --Rob Partington http://unix.culti.st/
On Jan 25, 2007, at 4:33 PM, Ceri Storey wrote:> > If I were to do this myself, I''d personally use a "pre-forking", where > the children would act as a worker pool, collecting jobs from, and > reporting completion and success or failure to, the managing parent > process. > This manager process would then handle graph traversal.Yeah, that would work fine. It would end up looking a lot like make - j3 or whatever.> On the other hand, I''d personally say that if your puppet > configuration is > coupled in such a manner that you have lots of long-running tasks, > then it > might be worth re-thinking the way it''s done. For example, if > you''ve a package > with a post-install script that spends 5 minutes downloading > updates, then it > might be worth changing the way the package handles those updates.I''d agree in the general case, although there are definitely exceptions. I like the idea of allowing parallelization, but it does open up the possibility that Puppet could take a lot more CPU than people are interested in allowing it. At the least, it would have to be configurable like make is. Either way, though, I''m unlikely to spend the time on this unless someone cares enough to pay for it, although I would gladly accept patches that enabled it. -- It is dangerous to be right when the government is wrong. -- Voltaire --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Jan 26, 2007, at 10:39 AM, Luke Kanies wrote:> On Jan 25, 2007, at 4:33 PM, Ceri Storey wrote: >> >> If I were to do this myself, I''d personally use a "pre-forking", >> where >> the children would act as a worker pool, collecting jobs from, and >> reporting completion and success or failure to, the managing parent >> process. >> This manager process would then handle graph traversal. > > Yeah, that would work fine. It would end up looking a lot like make - > j3 or whatever. > >> On the other hand, I''d personally say that if your puppet >> configuration is >> coupled in such a manner that you have lots of long-running tasks, >> then it >> might be worth re-thinking the way it''s done. For example, if >> you''ve a package >> with a post-install script that spends 5 minutes downloading >> updates, then it >> might be worth changing the way the package handles those updates. > > I''d agree in the general case, although there are definitely > exceptions. > > I like the idea of allowing parallelization, but it does open up the > possibility that Puppet could take a lot more CPU than people are > interested in allowing it. At the least, it would have to be > configurable like make is.Should we add support for the new Ruby process limiting features in 1.8.5? This applies to Rails, but is ruby specific. http://blog.segment7.net/articles/2006/09/13/controlling-rails- process-size -Blake
On Jan 29, 2007, at 3:33 PM, Blake Barnett wrote:> > Should we add support for the new Ruby process limiting features in > 1.8.5? > > This applies to Rails, but is ruby specific. > http://blog.segment7.net/articles/2006/09/13/controlling-rails- > process-sizeI''m not opposed, although we''ve had so many problems with 1.8.5 that I haven''t yet done much testing on it. It''s not going to be on my todo list any time soon that I can see, but it seems like a good option. -- Work is not always required. There is such a thing as sacred idleness. -- George MacDonald --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On Mon, Jan 29, 2007 at 01:33:29PM -0800, Blake Barnett wrote:> Should we add support for the new Ruby process limiting features in > 1.8.5? > > This applies to Rails, but is ruby specific. > http://blog.segment7.net/articles/2006/09/13/controlling-rails- > process-sizeIf you''re concerned about that, then I''d say that a cheap way to do that would be to set those limits in the puppet init.d script[0], which would then be inherited by puppet itself. It''s worth bearing in mind that these limits would then be inherited by any sub-processes (not necesarrily a bad thing) by default (ie: unless altered). That said, you would then also need to modify puppet to behave appropriately when it exceeds such a resource limit, and what that behavior should be (halt and catch-fire, for example). I suppose that having this configurable might be useful in some cases, for example when someone decides that it''s a good idea to transfer a 200MB binary file over the puppet transfer mechanism, and your puppetmaster decides to take over most of your virtual memory. Although puppet does seem fairly mild-mannered, given that puppet''s workloads are far more predicatable than that of a web server, and a sprinkling of concurrency (eg: only using 2/3 worker threads, if implemented) would probably suffice here; and be less of a sledgehammer. [0] -- Just in case you''re not aware, this module provides an interface to the operating system interfaces of the same name. -- Ceri Storey <cez@necrofish.org.uk> ''What I really want is "apt-get smite"'' --Rob Partington http://unix.culti.st/