I recently noticed that the Windows host where I installed the Puppet agent for testing was thrashing the CPU. The culprit ended up being Ruby.exe *32 - "Ruby interpreter (CUI) 1.8.7.334 [i386-mingw32]". This is actually being launched by the puppet-agent service in Windows. The CPU on the host was pegged around 50% all day long. When I shut down the puppet-agent it went down to a reasonable level...hovering in the low single digits most of the time. Start puppet-agent and CPU Usage (on both CPUs in a 2vCPU host) immediately pegs to 40%-60%....stop puppet-agent and it drops to near zero. Is this a known issue? Puppet-agent eating up tons of CPU time (via the ruby interpreter)? This host is a Windows 2008 VMware VM (VSphere 5) with 2vCPUs and 4GB of RAM assigned. It has no CPU or Memory resource limits (set to unlimited). No reservations (set to 0). CPU and memory resource shares are set to "High". VMware Tools are installed, running, and current. Thanks, Matt -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Jeff McCune
2012-Mar-07 23:39 UTC
Re: [Puppet Users] Puppet Agent on Windows - High CPU Usage
On Wed, Mar 7, 2012 at 2:58 PM, Matt Mencel <matt@techminer.net> wrote:> I recently noticed that the Windows host where I installed the Puppet > agent for testing was thrashing the CPU. > > The culprit ended up being Ruby.exe *32 - "Ruby interpreter (CUI) > 1.8.7.334 [i386-mingw32]". This is actually being launched by the > puppet-agent service in Windows. The CPU on the host was pegged around 50% > all day long. When I shut down the puppet-agent it went down to a > reasonable level...hovering in the low single digits most of the time. > > Start puppet-agent and CPU Usage (on both CPUs in a 2vCPU host) > immediately pegs to 40%-60%....stop puppet-agent and it drops to near zero. > > Is this a known issue? Puppet-agent eating up tons of CPU time (via the > ruby interpreter)? >It''s new information to me, so I don''t think we''ve started tracking this specific issue. I notice you mention it''s ruby.exe that''s spinning. The service actually starts rubyw.exe (Note the w) which invokes a very small ruby program that runs the windows service event loop [1]. The daemon.rb event loop will then use Process.create to spawn the actual Puppet agent process using INSTALLDIR\bin\puppet.bat agent --onetime. The Puppet agent is run inside of ruby.exe instead of rubyw.exe. Could you use SysInternals Process Explorer to see if it''s actually the puppet agent process that''s spinning, of if it''s the service daemon? [1] https://github.com/puppetlabs/puppet_for_the_win/blob/master/conf/windows/service/daemon.rb Thanks, -Jeff -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Josh Cooper
2012-Mar-08 00:12 UTC
Re: [Puppet Users] Puppet Agent on Windows - High CPU Usage
Hi Matt, On Wed, Mar 7, 2012 at 2:58 PM, Matt Mencel <matt@techminer.net> wrote:> I recently noticed that the Windows host where I installed the Puppet > agent for testing was thrashing the CPU. > > The culprit ended up being Ruby.exe *32 - "Ruby interpreter (CUI) > 1.8.7.334 [i386-mingw32]". This is actually being launched by the > puppet-agent service in Windows. >Did you install the service as described in the wiki page with nssm? Or did you install the MSI, which installed the service for you? If it''s the former, what version of puppet are you using?> The CPU on the host was pegged around 50% all day long. When I shut down > the puppet-agent it went down to a reasonable level...hovering in the low > single digits most of the time. > > Start puppet-agent and CPU Usage (on both CPUs in a 2vCPU host) > immediately pegs to 40%-60%....stop puppet-agent and it drops to near zero. > > Is this a known issue? Puppet-agent eating up tons of CPU time (via the > ruby interpreter)? >What happens if you stop the puppet-agent service and launch cmd.exe with elevated privileges (Run as Administrator), then try running: 1. facter.bat --timing 2. puppet.bat agent --test --debug This host is a Windows 2008 VMware VM (VSphere 5) with 2vCPUs and 4GB of> RAM assigned. It has no CPU or Memory resource limits (set to unlimited). > No reservations (set to 0). CPU and memory resource shares are set to > "High". VMware Tools are installed, running, and current. >Josh -- Josh Cooper Developer, Puppet Labs -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Matt Mencel
2012-Mar-08 14:45 UTC
Re: [Puppet Users] Puppet Agent on Windows - High CPU Usage
Hi Josh, On Wed, Mar 7, 2012 at 6:12 PM, Josh Cooper <josh@puppetlabs.com> wrote:> Hi Matt, > > On Wed, Mar 7, 2012 at 2:58 PM, Matt Mencel <matt@techminer.net> wrote: > >> I recently noticed that the Windows host where I installed the Puppet >> agent for testing was thrashing the CPU. >> >> The culprit ended up being Ruby.exe *32 - "Ruby interpreter (CUI) >> 1.8.7.334 [i386-mingw32]". This is actually being launched by the >> puppet-agent service in Windows. >> > > Did you install the service as described in the wiki page with nssm? Or > did you install the MSI, which installed the service for you? If it''s the > former, what version of puppet are you using? >I installed Ruby from http://rubyforge.org/frs/download.php/74293/rubyinstaller-1.8.7-p334.exe using the instructions in the Wiki and then the two gem install lines listed just below that to install all the win32-* gems.> >> The CPU on the host was pegged around 50% all day long. When I shut >> down the puppet-agent it went down to a reasonable level...hovering in the >> low single digits most of the time. >> >> Start puppet-agent and CPU Usage (on both CPUs in a 2vCPU host) >> immediately pegs to 40%-60%....stop puppet-agent and it drops to near zero. >> >> Is this a known issue? Puppet-agent eating up tons of CPU time (via the >> ruby interpreter)? >> > > What happens if you stop the puppet-agent service and launch cmd.exe with > elevated privileges (Run as Administrator), then try running: > > 1. facter.bat --timing > > 2. puppet.bat agent --test --debug >Screenshots are attached from these two commands.> > This host is a Windows 2008 VMware VM (VSphere 5) with 2vCPUs and 4GB of >> RAM assigned. It has no CPU or Memory resource limits (set to unlimited). >> No reservations (set to 0). CPU and memory resource shares are set to >> "High". VMware Tools are installed, running, and current. >> > > Josh > -- > Josh Cooper > Developer, Puppet Labs > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Matt Mencel
2012-Mar-08 14:53 UTC
Re: [Puppet Users] Puppet Agent on Windows - High CPU Usage
Hi Jeff, See the attached snapshot from Process Explorer. It''s one of the ruby processes under nssm. Matt On Wed, Mar 7, 2012 at 5:39 PM, Jeff McCune <jeff@puppetlabs.com> wrote:> On Wed, Mar 7, 2012 at 2:58 PM, Matt Mencel <matt@techminer.net> wrote: > >> I recently noticed that the Windows host where I installed the Puppet >> agent for testing was thrashing the CPU. >> >> The culprit ended up being Ruby.exe *32 - "Ruby interpreter (CUI) >> 1.8.7.334 [i386-mingw32]". This is actually being launched by the >> puppet-agent service in Windows. The CPU on the host was pegged around 50% >> all day long. When I shut down the puppet-agent it went down to a >> reasonable level...hovering in the low single digits most of the time. >> >> Start puppet-agent and CPU Usage (on both CPUs in a 2vCPU host) >> immediately pegs to 40%-60%....stop puppet-agent and it drops to near zero. >> >> Is this a known issue? Puppet-agent eating up tons of CPU time (via the >> ruby interpreter)? >> > > It''s new information to me, so I don''t think we''ve started tracking this > specific issue. > > I notice you mention it''s ruby.exe that''s spinning. The service actually > starts rubyw.exe (Note the w) which invokes a very small ruby program that > runs the windows service event loop [1]. > > The daemon.rb event loop will then use Process.create to spawn the actual > Puppet agent process using INSTALLDIR\bin\puppet.bat agent --onetime. The > Puppet agent is run inside of ruby.exe instead of rubyw.exe. > > Could you use SysInternals Process Explorer to see if it''s actually the > puppet agent process that''s spinning, of if it''s the service daemon? > > [1] > https://github.com/puppetlabs/puppet_for_the_win/blob/master/conf/windows/service/daemon.rb > > Thanks, > -Jeff > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Matt Mencel
2012-Mar-08 15:25 UTC
Re: [Puppet Users] Puppet Agent on Windows - High CPU Usage
Here''s a 10 second movie of Process Explorer. Matt On Thu, Mar 8, 2012 at 8:53 AM, Matt Mencel <matt@techminer.net> wrote:> Hi Jeff, > > See the attached snapshot from Process Explorer. It''s one of the ruby > processes under nssm. > > Matt > > > On Wed, Mar 7, 2012 at 5:39 PM, Jeff McCune <jeff@puppetlabs.com> wrote: > >> On Wed, Mar 7, 2012 at 2:58 PM, Matt Mencel <matt@techminer.net> wrote: >> >>> I recently noticed that the Windows host where I installed the Puppet >>> agent for testing was thrashing the CPU. >>> >>> The culprit ended up being Ruby.exe *32 - "Ruby interpreter (CUI) >>> 1.8.7.334 [i386-mingw32]". This is actually being launched by the >>> puppet-agent service in Windows. The CPU on the host was pegged around 50% >>> all day long. When I shut down the puppet-agent it went down to a >>> reasonable level...hovering in the low single digits most of the time. >>> >>> Start puppet-agent and CPU Usage (on both CPUs in a 2vCPU host) >>> immediately pegs to 40%-60%....stop puppet-agent and it drops to near zero. >>> >>> Is this a known issue? Puppet-agent eating up tons of CPU time (via the >>> ruby interpreter)? >>> >> >> It''s new information to me, so I don''t think we''ve started tracking this >> specific issue. >> >> I notice you mention it''s ruby.exe that''s spinning. The service actually >> starts rubyw.exe (Note the w) which invokes a very small ruby program that >> runs the windows service event loop [1]. >> >> The daemon.rb event loop will then use Process.create to spawn the actual >> Puppet agent process using INSTALLDIR\bin\puppet.bat agent --onetime. The >> Puppet agent is run inside of ruby.exe instead of rubyw.exe. >> >> Could you use SysInternals Process Explorer to see if it''s actually the >> puppet agent process that''s spinning, of if it''s the service daemon? >> >> [1] >> https://github.com/puppetlabs/puppet_for_the_win/blob/master/conf/windows/service/daemon.rb >> >> Thanks, >> -Jeff >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To post to this group, send email to puppet-users@googlegroups.com. >> To unsubscribe from this group, send email to >> puppet-users+unsubscribe@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/puppet-users?hl=en. >> > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Josh Cooper
2012-Mar-08 16:47 UTC
Re: [Puppet Users] Puppet Agent on Windows - High CPU Usage
HI Matt, On Thu, Mar 8, 2012 at 6:45 AM, Matt Mencel <matt@techminer.net> wrote:> Hi Josh, > > On Wed, Mar 7, 2012 at 6:12 PM, Josh Cooper <josh@puppetlabs.com> wrote: > >> Hi Matt, >> >> On Wed, Mar 7, 2012 at 2:58 PM, Matt Mencel <matt@techminer.net> wrote: >> >>> I recently noticed that the Windows host where I installed the Puppet >>> agent for testing was thrashing the CPU. >>> >>> The culprit ended up being Ruby.exe *32 - "Ruby interpreter (CUI) >>> 1.8.7.334 [i386-mingw32]". This is actually being launched by the >>> puppet-agent service in Windows. >>> >> >> Did you install the service as described in the wiki page with nssm? Or >> did you install the MSI, which installed the service for you? If it''s the >> former, what version of puppet are you using? >> > > I installed Ruby from > http://rubyforge.org/frs/download.php/74293/rubyinstaller-1.8.7-p334.exe using > the instructions in the Wiki and then the two gem install lines listed just > below that to install all the win32-* gems. > > >> >>> The CPU on the host was pegged around 50% all day long. When I shut >>> down the puppet-agent it went down to a reasonable level...hovering in the >>> low single digits most of the time. >>> >>> Start puppet-agent and CPU Usage (on both CPUs in a 2vCPU host) >>> immediately pegs to 40%-60%....stop puppet-agent and it drops to near zero. >>> >>> Is this a known issue? Puppet-agent eating up tons of CPU time (via the >>> ruby interpreter)? >>> >> >> What happens if you stop the puppet-agent service and launch cmd.exe with >> elevated privileges (Run as Administrator), then try running: >> >> 1. facter.bat --timing >> >I can''t tell from the screen shot whether facter exited or not? 2. puppet.bat agent --test --debug>> >When you run `ruby install.rb` it should create the puppet config directory, C:\ProgramData\PuppetLabs\puppet (on 2008). It appears this directory has gone missing. Did you delete the directory due to SSL issues? If the ''C:\ProgramData\PuppetLabs\puppet'' directory does not exist, then puppet will refuse to run. I ran into this issue sometime ago ( http://projects.puppetlabs.com/issues/9460). My understanding is that puppet has had issues in the past creating non-existent parent directories that it isn''t explicitly managing, e.g. /etc/puppet. I don''t know the exact details as it was before my time here. So we decided to keep the windows behavior the same as on unix. However, this means if you run puppet as a service, nssm will keep trying to restart puppet, which is why you keep seeing new ruby processes. Can you stop the service, re-run ruby install.rb, make sure you can run puppet.bat agent --test --debug from the command line (with elevated privileges). If that works, then you should be able to start the service. With that said, we are working on a new puppet MSI package and service code that doesn''t rely on nssm. If you would like to try it out, please contact me directly. Josh -- Josh Cooper Developer, Puppet Labs -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Josh Cooper
2012-Mar-08 19:27 UTC
Re: [Puppet Users] Puppet Agent on Windows - High CPU Usage
On Thu, Mar 8, 2012 at 8:47 AM, Josh Cooper <josh@puppetlabs.com> wrote:> HI Matt, > > On Thu, Mar 8, 2012 at 6:45 AM, Matt Mencel <matt@techminer.net> wrote: > >> Hi Josh, >> >> On Wed, Mar 7, 2012 at 6:12 PM, Josh Cooper <josh@puppetlabs.com> wrote: >> >>> Hi Matt, >>> >>> On Wed, Mar 7, 2012 at 2:58 PM, Matt Mencel <matt@techminer.net> wrote: >>> >>>> I recently noticed that the Windows host where I installed the Puppet >>>> agent for testing was thrashing the CPU. >>>> >>>> The culprit ended up being Ruby.exe *32 - "Ruby interpreter (CUI) >>>> 1.8.7.334 [i386-mingw32]". This is actually being launched by the >>>> puppet-agent service in Windows. >>>> >>> >>> Did you install the service as described in the wiki page with nssm? Or >>> did you install the MSI, which installed the service for you? If it''s the >>> former, what version of puppet are you using? >>> >> >> I installed Ruby from >> http://rubyforge.org/frs/download.php/74293/rubyinstaller-1.8.7-p334.exe using >> the instructions in the Wiki and then the two gem install lines listed just >> below that to install all the win32-* gems. >> >> >>> >>>> The CPU on the host was pegged around 50% all day long. When I shut >>>> down the puppet-agent it went down to a reasonable level...hovering in the >>>> low single digits most of the time. >>>> >>>> Start puppet-agent and CPU Usage (on both CPUs in a 2vCPU host) >>>> immediately pegs to 40%-60%....stop puppet-agent and it drops to near zero. >>>> >>>> Is this a known issue? Puppet-agent eating up tons of CPU time (via >>>> the ruby interpreter)? >>>> >>> >>> What happens if you stop the puppet-agent service and launch cmd.exe >>> with elevated privileges (Run as Administrator), then try running: >>> >>> 1. facter.bat --timing >>> >> > I can''t tell from the screen shot whether facter exited or not? > > 2. puppet.bat agent --test --debug >>> >> > When you run `ruby install.rb` it should create the puppet config > directory, C:\ProgramData\PuppetLabs\puppet (on 2008). It appears this > directory has gone missing. Did you delete the directory due to SSL issues? > > If the ''C:\ProgramData\PuppetLabs\puppet'' directory does not exist, then > puppet will refuse to run. I ran into this issue sometime ago ( > http://projects.puppetlabs.com/issues/9460). My understanding is that > puppet has had issues in the past creating non-existent parent directories > that it isn''t explicitly managing, e.g. /etc/puppet. I don''t know the exact > details as it was before my time here. So we decided to keep the windows > behavior the same as on unix. > >Jeff just pointed out to me that puppet is trying to create a directory C:\ProgramData\PuppetLabs\puppet\run (but failing because its parent directory doesn''t exist). Normally, the run directory is within C:\ProgramData\PuppetLabs\puppet\var\run. Can you include your puppet.conf? However, this means if you run puppet as a service, nssm will keep trying> to restart puppet, which is why you keep seeing new ruby processes. > > Can you stop the service, re-run ruby install.rb, make sure you can run > puppet.bat agent --test --debug from the command line (with elevated > privileges). If that works, then you should be able to start the service. > > With that said, we are working on a new puppet MSI package and service > code that doesn''t rely on nssm. If you would like to try it out, please > contact me directly. > > Josh > > -- > Josh Cooper > Developer, Puppet Labs > >-- Josh Cooper Developer, Puppet Labs -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Matt Mencel
2012-Mar-08 19:29 UTC
Re: [Puppet Users] Puppet Agent on Windows - High CPU Usage
OK... apparently the first time I ran "ruby install.rb" for puppet I did not do it in "Run As Administrator" mode. Even though the account I''m logging in as is a local admin....it seems to work only if I open the command prompt with "Run As Administrator" privileges. Deleting the folder structure under c:\ProgramData\PuppetLabs and then re-running with the correct privileges has solved the CPU issue. Matt On Thu, Mar 8, 2012 at 10:47 AM, Josh Cooper <josh@puppetlabs.com> wrote:> HI Matt, > > On Thu, Mar 8, 2012 at 6:45 AM, Matt Mencel <matt@techminer.net> wrote: > >> Hi Josh, >> >> On Wed, Mar 7, 2012 at 6:12 PM, Josh Cooper <josh@puppetlabs.com> wrote: >> >>> Hi Matt, >>> >>> On Wed, Mar 7, 2012 at 2:58 PM, Matt Mencel <matt@techminer.net> wrote: >>> >>>> I recently noticed that the Windows host where I installed the Puppet >>>> agent for testing was thrashing the CPU. >>>> >>>> The culprit ended up being Ruby.exe *32 - "Ruby interpreter (CUI) >>>> 1.8.7.334 [i386-mingw32]". This is actually being launched by the >>>> puppet-agent service in Windows. >>>> >>> >>> Did you install the service as described in the wiki page with nssm? Or >>> did you install the MSI, which installed the service for you? If it''s the >>> former, what version of puppet are you using? >>> >> >> I installed Ruby from >> http://rubyforge.org/frs/download.php/74293/rubyinstaller-1.8.7-p334.exe using >> the instructions in the Wiki and then the two gem install lines listed just >> below that to install all the win32-* gems. >> >> >>> >>>> The CPU on the host was pegged around 50% all day long. When I shut >>>> down the puppet-agent it went down to a reasonable level...hovering in the >>>> low single digits most of the time. >>>> >>>> Start puppet-agent and CPU Usage (on both CPUs in a 2vCPU host) >>>> immediately pegs to 40%-60%....stop puppet-agent and it drops to near zero. >>>> >>>> Is this a known issue? Puppet-agent eating up tons of CPU time (via >>>> the ruby interpreter)? >>>> >>> >>> What happens if you stop the puppet-agent service and launch cmd.exe >>> with elevated privileges (Run as Administrator), then try running: >>> >>> 1. facter.bat --timing >>> >> > I can''t tell from the screen shot whether facter exited or not? > > 2. puppet.bat agent --test --debug >>> >> > When you run `ruby install.rb` it should create the puppet config > directory, C:\ProgramData\PuppetLabs\puppet (on 2008). It appears this > directory has gone missing. Did you delete the directory due to SSL issues? > > If the ''C:\ProgramData\PuppetLabs\puppet'' directory does not exist, then > puppet will refuse to run. I ran into this issue sometime ago ( > http://projects.puppetlabs.com/issues/9460). My understanding is that > puppet has had issues in the past creating non-existent parent directories > that it isn''t explicitly managing, e.g. /etc/puppet. I don''t know the exact > details as it was before my time here. So we decided to keep the windows > behavior the same as on unix. > > However, this means if you run puppet as a service, nssm will keep trying > to restart puppet, which is why you keep seeing new ruby processes. > > Can you stop the service, re-run ruby install.rb, make sure you can run > puppet.bat agent --test --debug from the command line (with elevated > privileges). If that works, then you should be able to start the service. > > With that said, we are working on a new puppet MSI package and service > code that doesn''t rely on nssm. If you would like to try it out, please > contact me directly. > > Josh > > -- > Josh Cooper > Developer, Puppet Labs > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.