Puppet users - Mar 2012 - Puppet Agent on Windows - High CPU Usage

I recently noticed that the Windows host where I installed the Puppet agent
for testing was thrashing the CPU.

The culprit ended up being Ruby.exe *32 - "Ruby interpreter (CUI) 1.8.7.334
[i386-mingw32]".  This is actually being launched by the puppet-agent
service in Windows.  The CPU on the host was pegged around 50% all day
long.  When I shut down the puppet-agent it went down to a reasonable
level...hovering in the low single digits most of the time.

Start puppet-agent and CPU Usage (on both CPUs in a 2vCPU host) immediately
pegs to 40%-60%....stop puppet-agent and it drops to near zero.

Is this a known issue?  Puppet-agent eating up tons of CPU time (via the
ruby interpreter)?

This host is a Windows 2008 VMware VM (VSphere 5) with 2vCPUs and 4GB of
RAM assigned.  It has no CPU or Memory resource limits (set to unlimited).
 No reservations (set to 0).  CPU and memory resource shares are set to
"High".  VMware Tools are installed, running, and current.

Thanks,
Matt

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Wed, Mar 7, 2012 at 2:58 PM, Matt Mencel <matt@techminer.net> wrote:
> I recently noticed that the Windows host where I installed the Puppet
> agent for testing was thrashing the CPU.
>
> The culprit ended up being Ruby.exe *32 - "Ruby interpreter (CUI)
> 1.8.7.334 [i386-mingw32]".  This is actually being launched by the
> puppet-agent service in Windows.  The CPU on the host was pegged around 50%
> all day long.  When I shut down the puppet-agent it went down to a
> reasonable level...hovering in the low single digits most of the time.
>
> Start puppet-agent and CPU Usage (on both CPUs in a 2vCPU host)
> immediately pegs to 40%-60%....stop puppet-agent and it drops to near zero.
>
> Is this a known issue?  Puppet-agent eating up tons of CPU time (via the
> ruby interpreter)?
>
It''s new information to me, so I don''t think we''ve
started tracking this
specific issue.

I notice you mention it''s ruby.exe that''s spinning.  The
service actually
starts rubyw.exe (Note the w) which invokes a very small ruby program that
runs the windows service event loop [1].

The daemon.rb event loop will then use Process.create to spawn the actual
Puppet agent process using INSTALLDIR\bin\puppet.bat agent --onetime.  The
Puppet agent is run inside of ruby.exe instead of rubyw.exe.

Could you use SysInternals Process Explorer to see if it''s actually the
puppet agent process that''s spinning, of if it''s the service
daemon?

[1]
https://github.com/puppetlabs/puppet_for_the_win/blob/master/conf/windows/service/daemon.rb

Thanks,
-Jeff

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi Matt,

On Wed, Mar 7, 2012 at 2:58 PM, Matt Mencel <matt@techminer.net> wrote:
> I recently noticed that the Windows host where I installed the Puppet
> agent for testing was thrashing the CPU.
>
> The culprit ended up being Ruby.exe *32 - "Ruby interpreter (CUI)
> 1.8.7.334 [i386-mingw32]".  This is actually being launched by the
> puppet-agent service in Windows.
>
Did you install the service as described in the wiki page with nssm? Or did
you install the MSI, which installed the service for you? If it''s the
former, what version of puppet are you using?

>  The CPU on the host was pegged around 50% all day long.  When I shut down
> the puppet-agent it went down to a reasonable level...hovering in the low
> single digits most of the time.
>
> Start puppet-agent and CPU Usage (on both CPUs in a 2vCPU host)
> immediately pegs to 40%-60%....stop puppet-agent and it drops to near zero.
>
> Is this a known issue?  Puppet-agent eating up tons of CPU time (via the
> ruby interpreter)?
>
What happens if you stop the puppet-agent service and launch cmd.exe with
elevated privileges (Run as Administrator), then try running:

1. facter.bat --timing

2. puppet.bat agent --test --debug

This host is a Windows 2008 VMware VM (VSphere 5) with 2vCPUs and 4GB
of
> RAM assigned.  It has no CPU or Memory resource limits (set to unlimited).
>  No reservations (set to 0).  CPU and memory resource shares are set to
> "High".  VMware Tools are installed, running, and current.
>
Josh
-- 
Josh Cooper
Developer, Puppet Labs

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi Josh,

On Wed, Mar 7, 2012 at 6:12 PM, Josh Cooper <josh@puppetlabs.com> wrote:
> Hi Matt,
>
> On Wed, Mar 7, 2012 at 2:58 PM, Matt Mencel <matt@techminer.net>
wrote:
>
>> I recently noticed that the Windows host where I installed the Puppet
>> agent for testing was thrashing the CPU.
>>
>> The culprit ended up being Ruby.exe *32 - "Ruby interpreter (CUI)
>> 1.8.7.334 [i386-mingw32]".  This is actually being launched by the
>> puppet-agent service in Windows.
>>
>
> Did you install the service as described in the wiki page with nssm? Or
> did you install the MSI, which installed the service for you? If
it''s the
> former, what version of puppet are you using?
>
I installed Ruby from
http://rubyforge.org/frs/download.php/74293/rubyinstaller-1.8.7-p334.exe using
the instructions in the Wiki and then the two gem install lines listed just
below that to install all the win32-* gems.

>
>>  The CPU on the host was pegged around 50% all day long.  When I shut
>> down the puppet-agent it went down to a reasonable level...hovering in
the
>> low single digits most of the time.
>>
>> Start puppet-agent and CPU Usage (on both CPUs in a 2vCPU host)
>> immediately pegs to 40%-60%....stop puppet-agent and it drops to near
zero.
>>
>> Is this a known issue?  Puppet-agent eating up tons of CPU time (via
the
>> ruby interpreter)?
>>
>
> What happens if you stop the puppet-agent service and launch cmd.exe with
> elevated privileges (Run as Administrator), then try running:
>
> 1. facter.bat --timing
>
> 2. puppet.bat agent --test --debug
>
Screenshots are attached from these two commands.

>
> This host is a Windows 2008 VMware VM (VSphere 5) with 2vCPUs and 4GB of
>> RAM assigned.  It has no CPU or Memory resource limits (set to
unlimited).
>>  No reservations (set to 0).  CPU and memory resource shares are set to
>> "High".  VMware Tools are installed, running, and current.
>>
>
> Josh
> --
> Josh Cooper
> Developer, Puppet Labs
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi Jeff,

See the attached snapshot from Process Explorer.  It''s one of the ruby
processes under nssm.

Matt


On Wed, Mar 7, 2012 at 5:39 PM, Jeff McCune <jeff@puppetlabs.com> wrote:
> On Wed, Mar 7, 2012 at 2:58 PM, Matt Mencel <matt@techminer.net>
wrote:
>
>> I recently noticed that the Windows host where I installed the Puppet
>> agent for testing was thrashing the CPU.
>>
>> The culprit ended up being Ruby.exe *32 - "Ruby interpreter (CUI)
>> 1.8.7.334 [i386-mingw32]".  This is actually being launched by the
>> puppet-agent service in Windows.  The CPU on the host was pegged around
50%
>> all day long.  When I shut down the puppet-agent it went down to a
>> reasonable level...hovering in the low single digits most of the time.
>>
>> Start puppet-agent and CPU Usage (on both CPUs in a 2vCPU host)
>> immediately pegs to 40%-60%....stop puppet-agent and it drops to near
zero.
>>
>> Is this a known issue?  Puppet-agent eating up tons of CPU time (via
the
>> ruby interpreter)?
>>
>
> It''s new information to me, so I don''t think
we''ve started tracking this
> specific issue.
>
> I notice you mention it''s ruby.exe that''s spinning.  The
service actually
> starts rubyw.exe (Note the w) which invokes a very small ruby program that
> runs the windows service event loop [1].
>
> The daemon.rb event loop will then use Process.create to spawn the actual
> Puppet agent process using INSTALLDIR\bin\puppet.bat agent --onetime.  The
> Puppet agent is run inside of ruby.exe instead of rubyw.exe.
>
> Could you use SysInternals Process Explorer to see if it''s
actually the
> puppet agent process that''s spinning, of if it''s the
service daemon?
>
> [1]
>
https://github.com/puppetlabs/puppet_for_the_win/blob/master/conf/windows/service/daemon.rb
>
> Thanks,
> -Jeff
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Here''s a 10 second movie of Process Explorer.

Matt

On Thu, Mar 8, 2012 at 8:53 AM, Matt Mencel <matt@techminer.net> wrote:
> Hi Jeff,
>
> See the attached snapshot from Process Explorer.  It''s one of the
ruby
> processes under nssm.
>
> Matt
>
>
> On Wed, Mar 7, 2012 at 5:39 PM, Jeff McCune <jeff@puppetlabs.com>
wrote:
>
>> On Wed, Mar 7, 2012 at 2:58 PM, Matt Mencel <matt@techminer.net>
wrote:
>>
>>> I recently noticed that the Windows host where I installed the
Puppet
>>> agent for testing was thrashing the CPU.
>>>
>>> The culprit ended up being Ruby.exe *32 - "Ruby interpreter
(CUI)
>>> 1.8.7.334 [i386-mingw32]".  This is actually being launched by
the
>>> puppet-agent service in Windows.  The CPU on the host was pegged
around 50%
>>> all day long.  When I shut down the puppet-agent it went down to a
>>> reasonable level...hovering in the low single digits most of the
time.
>>>
>>> Start puppet-agent and CPU Usage (on both CPUs in a 2vCPU host)
>>> immediately pegs to 40%-60%....stop puppet-agent and it drops to
near zero.
>>>
>>> Is this a known issue?  Puppet-agent eating up tons of CPU time
(via the
>>> ruby interpreter)?
>>>
>>
>> It''s new information to me, so I don''t think
we''ve started tracking this
>> specific issue.
>>
>> I notice you mention it''s ruby.exe that''s spinning. 
The service actually
>> starts rubyw.exe (Note the w) which invokes a very small ruby program
that
>> runs the windows service event loop [1].
>>
>> The daemon.rb event loop will then use Process.create to spawn the
actual
>> Puppet agent process using INSTALLDIR\bin\puppet.bat agent --onetime. 
The
>> Puppet agent is run inside of ruby.exe instead of rubyw.exe.
>>
>> Could you use SysInternals Process Explorer to see if it''s
actually the
>> puppet agent process that''s spinning, of if it''s the
service daemon?
>>
>> [1]
>>
https://github.com/puppetlabs/puppet_for_the_win/blob/master/conf/windows/service/daemon.rb
>>
>> Thanks,
>> -Jeff
>>
>>  --
>> You received this message because you are subscribed to the Google
Groups
>> "Puppet Users" group.
>> To post to this group, send email to puppet-users@googlegroups.com.
>> To unsubscribe from this group, send email to
>> puppet-users+unsubscribe@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/puppet-users?hl=en.
>>
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

HI Matt,

On Thu, Mar 8, 2012 at 6:45 AM, Matt Mencel <matt@techminer.net> wrote:
> Hi Josh,
>
> On Wed, Mar 7, 2012 at 6:12 PM, Josh Cooper <josh@puppetlabs.com>
wrote:
>
>> Hi Matt,
>>
>> On Wed, Mar 7, 2012 at 2:58 PM, Matt Mencel <matt@techminer.net>
wrote:
>>
>>> I recently noticed that the Windows host where I installed the
Puppet
>>> agent for testing was thrashing the CPU.
>>>
>>> The culprit ended up being Ruby.exe *32 - "Ruby interpreter
(CUI)
>>> 1.8.7.334 [i386-mingw32]".  This is actually being launched by
the
>>> puppet-agent service in Windows.
>>>
>>
>> Did you install the service as described in the wiki page with nssm? Or
>> did you install the MSI, which installed the service for you? If
it''s the
>> former, what version of puppet are you using?
>>
>
> I installed Ruby from
> http://rubyforge.org/frs/download.php/74293/rubyinstaller-1.8.7-p334.exe
using
> the instructions in the Wiki and then the two gem install lines listed just
> below that to install all the win32-* gems.
>
>
>>
>>>  The CPU on the host was pegged around 50% all day long.  When I
shut
>>> down the puppet-agent it went down to a reasonable level...hovering
in the
>>> low single digits most of the time.
>>>
>>> Start puppet-agent and CPU Usage (on both CPUs in a 2vCPU host)
>>> immediately pegs to 40%-60%....stop puppet-agent and it drops to
near zero.
>>>
>>> Is this a known issue?  Puppet-agent eating up tons of CPU time
(via the
>>> ruby interpreter)?
>>>
>>
>> What happens if you stop the puppet-agent service and launch cmd.exe
with
>> elevated privileges (Run as Administrator), then try running:
>>
>> 1. facter.bat --timing
>>
>
I can''t tell from the screen shot whether facter exited or not?

2. puppet.bat agent --test --debug
>>
>
When you run `ruby install.rb` it should create the puppet config
directory, C:\ProgramData\PuppetLabs\puppet (on 2008). It appears this
directory has gone missing. Did you delete the directory due to SSL issues?

If the ''C:\ProgramData\PuppetLabs\puppet'' directory does not
exist, then
puppet will refuse to run. I ran into this issue sometime ago (
http://projects.puppetlabs.com/issues/9460). My understanding is that
puppet has had issues in the past creating non-existent parent directories
that it isn''t explicitly managing, e.g. /etc/puppet. I don''t
know the exact
details as it was before my time here. So we decided to keep the windows
behavior the same as on unix.

However, this means if you run puppet as a service, nssm will keep trying
to restart puppet, which is why you keep seeing new ruby processes.

Can you stop the service, re-run ruby install.rb, make sure you can run
puppet.bat agent --test --debug from the command line (with elevated
privileges). If that works, then you should be able to start the service.

With that said, we are working on a new puppet MSI package and service code
that doesn''t rely on nssm. If you would like to try it out, please
contact
me directly.

Josh

-- 
Josh Cooper
Developer, Puppet Labs

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Thu, Mar 8, 2012 at 8:47 AM, Josh Cooper <josh@puppetlabs.com> wrote:
> HI Matt,
>
> On Thu, Mar 8, 2012 at 6:45 AM, Matt Mencel <matt@techminer.net>
wrote:
>
>> Hi Josh,
>>
>> On Wed, Mar 7, 2012 at 6:12 PM, Josh Cooper <josh@puppetlabs.com>
wrote:
>>
>>> Hi Matt,
>>>
>>> On Wed, Mar 7, 2012 at 2:58 PM, Matt Mencel
<matt@techminer.net> wrote:
>>>
>>>> I recently noticed that the Windows host where I installed the
Puppet
>>>> agent for testing was thrashing the CPU.
>>>>
>>>> The culprit ended up being Ruby.exe *32 - "Ruby
interpreter (CUI)
>>>> 1.8.7.334 [i386-mingw32]".  This is actually being
launched by the
>>>> puppet-agent service in Windows.
>>>>
>>>
>>> Did you install the service as described in the wiki page with
nssm? Or
>>> did you install the MSI, which installed the service for you? If
it''s the
>>> former, what version of puppet are you using?
>>>
>>
>> I installed Ruby from
>>
http://rubyforge.org/frs/download.php/74293/rubyinstaller-1.8.7-p334.exe using
>> the instructions in the Wiki and then the two gem install lines listed
just
>> below that to install all the win32-* gems.
>>
>>
>>>
>>>>  The CPU on the host was pegged around 50% all day long.  When
I shut
>>>> down the puppet-agent it went down to a reasonable
level...hovering in the
>>>> low single digits most of the time.
>>>>
>>>> Start puppet-agent and CPU Usage (on both CPUs in a 2vCPU host)
>>>> immediately pegs to 40%-60%....stop puppet-agent and it drops
to near zero.
>>>>
>>>> Is this a known issue?  Puppet-agent eating up tons of CPU time
(via
>>>> the ruby interpreter)?
>>>>
>>>
>>> What happens if you stop the puppet-agent service and launch
cmd.exe
>>> with elevated privileges (Run as Administrator), then try running:
>>>
>>> 1. facter.bat --timing
>>>
>>
> I can''t tell from the screen shot whether facter exited or not?
>
> 2. puppet.bat agent --test --debug
>>>
>>
> When you run `ruby install.rb` it should create the puppet config
> directory, C:\ProgramData\PuppetLabs\puppet (on 2008). It appears this
> directory has gone missing. Did you delete the directory due to SSL issues?
>
> If the ''C:\ProgramData\PuppetLabs\puppet'' directory does
not exist, then
> puppet will refuse to run. I ran into this issue sometime ago (
> http://projects.puppetlabs.com/issues/9460). My understanding is that
> puppet has had issues in the past creating non-existent parent directories
> that it isn''t explicitly managing, e.g. /etc/puppet. I
don''t know the exact
> details as it was before my time here. So we decided to keep the windows
> behavior the same as on unix.
>
>
Jeff just pointed out to me that puppet is trying to create a directory
C:\ProgramData\PuppetLabs\puppet\run (but failing because its parent
directory doesn''t exist). Normally, the run directory is within
C:\ProgramData\PuppetLabs\puppet\var\run. Can you include your puppet.conf?

However, this means if you run puppet as a service, nssm will keep
trying
> to restart puppet, which is why you keep seeing new ruby processes.
>
> Can you stop the service, re-run ruby install.rb, make sure you can run
> puppet.bat agent --test --debug from the command line (with elevated
> privileges). If that works, then you should be able to start the service.
>
> With that said, we are working on a new puppet MSI package and service
> code that doesn''t rely on nssm. If you would like to try it out,
please
> contact me directly.
>
> Josh
>
> --
> Josh Cooper
> Developer, Puppet Labs
>
>
-- 
Josh Cooper
Developer, Puppet Labs

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

OK... apparently the first time I ran "ruby install.rb" for puppet I
did
not do it in "Run As Administrator" mode.  Even though the account
I''m
logging in as is a local admin....it seems to work only if I open the
command prompt with "Run As Administrator" privileges.

Deleting the folder structure under c:\ProgramData\PuppetLabs and then
re-running with the correct privileges has solved the CPU issue.

Matt



On Thu, Mar 8, 2012 at 10:47 AM, Josh Cooper <josh@puppetlabs.com> wrote:
> HI Matt,
>
> On Thu, Mar 8, 2012 at 6:45 AM, Matt Mencel <matt@techminer.net>
wrote:
>
>> Hi Josh,
>>
>> On Wed, Mar 7, 2012 at 6:12 PM, Josh Cooper <josh@puppetlabs.com>
wrote:
>>
>>> Hi Matt,
>>>
>>> On Wed, Mar 7, 2012 at 2:58 PM, Matt Mencel
<matt@techminer.net> wrote:
>>>
>>>> I recently noticed that the Windows host where I installed the
Puppet
>>>> agent for testing was thrashing the CPU.
>>>>
>>>> The culprit ended up being Ruby.exe *32 - "Ruby
interpreter (CUI)
>>>> 1.8.7.334 [i386-mingw32]".  This is actually being
launched by the
>>>> puppet-agent service in Windows.
>>>>
>>>
>>> Did you install the service as described in the wiki page with
nssm? Or
>>> did you install the MSI, which installed the service for you? If
it''s the
>>> former, what version of puppet are you using?
>>>
>>
>> I installed Ruby from
>>
http://rubyforge.org/frs/download.php/74293/rubyinstaller-1.8.7-p334.exe using
>> the instructions in the Wiki and then the two gem install lines listed
just
>> below that to install all the win32-* gems.
>>
>>
>>>
>>>>  The CPU on the host was pegged around 50% all day long.  When
I shut
>>>> down the puppet-agent it went down to a reasonable
level...hovering in the
>>>> low single digits most of the time.
>>>>
>>>> Start puppet-agent and CPU Usage (on both CPUs in a 2vCPU host)
>>>> immediately pegs to 40%-60%....stop puppet-agent and it drops
to near zero.
>>>>
>>>> Is this a known issue?  Puppet-agent eating up tons of CPU time
(via
>>>> the ruby interpreter)?
>>>>
>>>
>>> What happens if you stop the puppet-agent service and launch
cmd.exe
>>> with elevated privileges (Run as Administrator), then try running:
>>>
>>> 1. facter.bat --timing
>>>
>>
> I can''t tell from the screen shot whether facter exited or not?
>
> 2. puppet.bat agent --test --debug
>>>
>>
> When you run `ruby install.rb` it should create the puppet config
> directory, C:\ProgramData\PuppetLabs\puppet (on 2008). It appears this
> directory has gone missing. Did you delete the directory due to SSL issues?
>
> If the ''C:\ProgramData\PuppetLabs\puppet'' directory does
not exist, then
> puppet will refuse to run. I ran into this issue sometime ago (
> http://projects.puppetlabs.com/issues/9460). My understanding is that
> puppet has had issues in the past creating non-existent parent directories
> that it isn''t explicitly managing, e.g. /etc/puppet. I
don''t know the exact
> details as it was before my time here. So we decided to keep the windows
> behavior the same as on unix.
>
> However, this means if you run puppet as a service, nssm will keep trying
> to restart puppet, which is why you keep seeing new ruby processes.
>
> Can you stop the service, re-run ruby install.rb, make sure you can run
> puppet.bat agent --test --debug from the command line (with elevated
> privileges). If that works, then you should be able to start the service.
>
> With that said, we are working on a new puppet MSI package and service
> code that doesn''t rely on nssm. If you would like to try it out,
please
> contact me directly.
>
> Josh
>
> --
> Josh Cooper
> Developer, Puppet Labs
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.