Puppet users - Oct 2013 - Why does the Puppet-Agent on Windows use a batch file?

Why does the Puppet-Agent on Windows use a batch file?

I posted a question in the Ask PuppetLabs section and was directed to 
create post here as well;
https://ask.puppetlabs.com/question/3506/why-does-the-puppet-agent-on-windows-use-a-batch-file/

------------------------ Original question
The Puppet Enterprise for Windows Agent runs as a windows service, 
basically a deamonised version of Puppet, which is all fine. However the 
Windows Service calls a batch file which seems extremely strange. While it 
does work, i.e. the Service starts and runs, using CMD.EXE as a service 
executable is generally considered a really bad idea.

It does not respond to the usual SCM (Service Control Manager) calls and in 
it''s current state is misconfigured e.g. The service says that it can 
respond to Pause and Continue events but CMD.EXE can''t fulfill those 
requests. Also CMD.EXE does not monitor the ruby process (except for the 
basic operation of is it running) and vice versa. I can kill the cmd.exe 
process and the service manager will report that the Puppet Agent has 
stopped however the ruby process is still quite happily running.

Either I''m missing something and CMD.EXE is an appropriate service 
executable or perhaps the community or puppet labs could create a better 
native wrapper to the ruby based puppet process.

------------------------

So I did a few tests;

   - You can send pause and continue messages to the service but
they''re
   just ignored even though the Services says it''s Paused. 
   - You can kill CMD.EXE service process but the Puppet Agent is still 
   running (It becomes an orphaned process).  You can then start the service 
   again, and you''ll end up with two Puppet Agents running daemonised
at the
   same time.  I''m not sure what will happen if they both try to do a
catalog
   run at the same time, but nothing good can come of it. 
   - CMD.EXE doesn''t respond to power events e.g. going into 
   Standby/Hibernate; but I have no idea how any service wrapper could raise 
   that kind of event in Puppet so that it could deal with it.  Does it even 
   matter if the host goes into Standby in the middle of Puppet run?  
   Admittedly this would be very unlikely scenario as Puppet seems to be more 
   always-on server orientated rather than for laptop configuration management. 
   - If you start Puppet and then quickly attempt to stop it, it get''s 
   stuck in the Stopping state and you have to kill Ruby manually. 
Thtat''s
   because the service control manager raises events in a multithreaded 
   manner, however CMD is running a single thread.  If CMD.EXE is too busy to 
   process the event then things get into a funny state.  This would probably 
   occur during a catalog run too but I haven''t confirmed it. 
   - From a troubleshooting perspective, there are no logs or diagnostic 
   information created by CMD.EXE

I''d be happy to write a service wrapper in C# and give it to the
community,
but before I do I want to make sure that it''s really needed and
I''m not
trying to solve a problem that doesn''t really exist.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

Hi Glenn,


On Sun, Oct 13, 2013 at 6:28 PM, Glenn Sarti <glenn.sarti2@gmail.com>
wrote:
> Why does the Puppet-Agent on Windows use a batch file?
>
Because the service and command line invocations, e.g. puppet apply, both
need to setup their environment to run ruby, e.g. RUBYLIB, etc. The easiest
way to do that was to create daemon.bat that calls environment.bat, and
then invokes ruby.
> I posted a question in the Ask PuppetLabs section and was directed to
> create post here as well;
>
>
https://ask.puppetlabs.com/question/3506/why-does-the-puppet-agent-on-windows-use-a-batch-file/
>
> ------------------------ Original question
> The Puppet Enterprise for Windows Agent runs as a windows service,
> basically a deamonised version of Puppet, which is all fine. However the
> Windows Service calls a batch file which seems extremely strange. While it
> does work, i.e. the Service starts and runs, using CMD.EXE as a service
> executable is generally considered a really bad idea.
>
> It does not respond to the usual SCM (Service Control Manager) calls and
> in it''s current state is misconfigured e.g. The service says that
it can
> respond to Pause and Continue events but CMD.EXE can''t fulfill
those
> requests.
>
The service is hosted in the ruby process that cmd.exe executes, not
cmd.exe itself. Puppet does respond to SCM events like start & stop. The
puppet service does claim to support pause & continue (due to the
win32-service gem[1]), but the service does not implement the necessary
handler logic. Please file a bug on that.
> Also CMD.EXE does not monitor the ruby process (except for the basic
> operation of is it running) and vice versa. I can kill the cmd.exe process
> and the service manager will report that the Puppet Agent has stopped
> however the ruby process is still quite happily running.
>
> Either I''m missing something and CMD.EXE is an appropriate service
> executable or perhaps the community or puppet labs could create a better
> native wrapper to the ruby based puppet process.
>
> ------------------------
>
> So I did a few tests;
>
>    - You can send pause and continue messages to the service but
they''re
>    just ignored even though the Services says it''s Paused.
>
> Puppet''s daemon.rb[2] needs to override the default service_pause
and
service_continue. The pause method should set a flag that the main loop
checks each time.
>
>    - You can kill CMD.EXE service process but the Puppet Agent is still
>    running (It becomes an orphaned process).  You can then start the
service
>    again, and you''ll end up with two Puppet Agents running
daemonised at the
>    same time.
>
> cmd.exe is not essential to the puppet service once it''s running.
>
>    -   I''m not sure what will happen if they both try to do a
catalog run
>    at the same time, but nothing good can come of it.
>
> Puppet uses a lock file to ensure two processes don''t apply a
catalog at
the same time. This scenario can come up in the normal case when puppet is
running as a service, and you run `puppet apply` on the command line.
>
>    - CMD.EXE doesn''t respond to power events e.g. going into
>    Standby/Hibernate; but I have no idea how any service wrapper could
raise
>    that kind of event in Puppet so that it could deal with it.  Does it
even
>    matter if the host goes into Standby in the middle of Puppet run?
>    Admittedly this would be very unlikely scenario as Puppet seems to be
more
>    always-on server orientated rather than for laptop configuration
management.
>    - If you start Puppet and then quickly attempt to stop it, it
get''s
>    stuck in the Stopping state and you have to kill Ruby manually. 
Thtat''s
>    because the service control manager raises events in a multithreaded
>    manner, however CMD is running a single thread.  If CMD.EXE is too busy
to
>    process the event then things get into a funny state.  This would
probably
>    occur during a catalog run too but I haven''t confirmed it.
>
> The service control manager calls back into ruby.exe, not cmd.exe. Our
daemon.rb is what calls StartServiceCtrlDispatcher...
>
>    - From a troubleshooting perspective, there are no logs or diagnostic
>    information created by CMD.EXE
>
> The daemon.rb code creates a log file (with different verbosity levels). I
have been meaning to switch it over to using the application event log, but
haven''t yet, patches welcome!

Josh

[1]
https://github.com/djberg96/win32-service/blob/ffi/lib/win32/daemon.rb#L86-87
[2]
https://github.com/puppetlabs/puppet/blob/master/ext/windows/service/daemon.rb

-- 
Josh Cooper
Developer, Puppet Labs

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

Hi Josh,

After a lot of digging around, I think I have partial solution;

NOTE - This is my first attempt at writing ruby so I expect there are some 
issues with what I''ve written.  I only had a single host (Server 2008
R2
64bit) to test this on, but I believe the changes I''ve made are generic
enough to work on all puppet supported MS Operating Systems, 32 or 64bit. 

While the CMD.EXE is not required by the Agent once running, there Service 
Control Manager is still monitoring that process and if it dies, it will 
say the service is not running, even though the orphaned the RUBY.EXE 
process is still running.

WINDOWS SERVICE CONFIG
The process for the service doesn''t need the entire environment as that
of
puppet, in order run.  The way I see it, the service needs enough 
information to act as a Windows Service and to spawn child processes of 
Puppet.  The Puppet.BAT calls Environment.BAT which does all the work to 
setup the environment variables on a per call basis.

So what I did is change the ImagePath of the pe-puppet service to call ruby 
directly;

HKLM\System\CurrentControlSet\Services\pe-puppet\ImagePath;

FROM:
"C:\Program Files (x86)\Puppet Labs\Puppet
Enterprise\service\daemon.bat"

TO:
"C:\Program Files (x86)\Puppet Labs\Puppet 
Enterprise\sys\ruby\bin\rubyw.exe" -C"C:\Program Files (x86)\Puppet 
Labs\Puppet Enterprise\service" "C:\Program Files (x86)\Puppet
Labs\Puppet
Enterprise\service\daemon.rb"

That is enough information for Ruby to run the service.  Obviously the 
paths in these may differ depending on each host, BUT that can all be 
authored in the puppet MSI easily.

DAEMON.RB
I made some changes to daemon.rb (attached to this post);
* I created a basic function for Windows EventLog logging Puppet Bug 
#21641.  It doesn''t register an application source so it''s a
bit of a hack
and could really do with a more professional cleanup.

* I fixed up the behaiour of Puppet Agent terminating once Paused Bug #22972

* A side effect of not running the daemon from a CMD.EXE was that the call 
to get to runinterval was failing.  I suspect this is due to STDOUT not 
being available anymore.  So I used the well worn method of pipe the output 
to a file and read that instead (Lines 60-79). I still need to try RUBY.EXE 
instead of RUBYW.EXE and see if it makes a difference.

* I put the Puppet Agent run in an IF statement, which will only evaluate 
as true if the service is in a RUNNING or IDLE state (Lines 81-86)

* I think may have found a bug in the Win32 Daemon code which was taking 
the service out of PAUSED and put it into a RUNNING state whenever a 
SERVICE_INTERROGATE event is recieved.  I need to log this with 
the authours. (Lines 108-119).

* I added in a little extra logging in the Resume and Pause events.  I 
changed some of the wording in the main loop to reduce any confusion about 
"Service Resuming"

Glenn.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/f540d5d5-f21b-455f-8ce9-a561cac3ef79%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

>
>  * A side effect of not running the daemon from a CMD.EXE was that the 
> call to get to runinterval was failing.  I suspect this is due to STDOUT 
> not being available anymore.  So I used the well worn method of pipe the 
> output to a file and read that instead (Lines 60-79). I still need to try 
> RUBY.EXE instead of RUBYW.EXE and see if it makes a difference.
>
 
Looks like RUBY.EXE will fix my issue as per Puppet Bug #21043.  I''ll 
update my daemon.rb and repost.
https://projects.puppetlabs.com/issues/21043

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/844378c0-0e97-46d3-aefa-472fb900dc99%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

I also noticed there is a puppetres.dll and puppetres.mc file so I''ll 
investigate that and use it for the EventLog stuff.
 
https://github.com/puppetlabs/puppet/tree/master/ext/windows/eventlog

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/bd3b91a5-6f4d-4d3c-b9dc-d1e2019749bc%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Well that was easier than I expected
 
* daemon.rb now defaults to logging in the Event Log and optionally to the 
windows.log file
 
* The ImagePath string now looks like;
"C:\Program Files (x86)\Puppet Labs\Puppet 
Enterprise\sys\ruby\bin\ruby.exe" -rubygems -C"C:\Program Files 
(x86)\Puppet Labs\Puppet Enterprise\service" "C:\Program Files
(x86)\Puppet
Labs\Puppet Enterprise\service\daemon.rb"
 
I''m not sure if the "-rubygems" is required but it is in the
daemon.bat
file.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/8239aa60-ad19-4b42-9bd7-fed09229f55b%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Wed, Oct 30, 2013 at 8:41 PM, Glenn Sarti <glenn.sarti2@gmail.com>
wrote:
> Well that was easier than I expected
>
> * daemon.rb now defaults to logging in the Event Log and optionally to the
> windows.log file
>
This should resolve https://projects.puppetlabs.com/issues/21641. Can you
submit a PR for this issue?

>
> * The ImagePath string now looks like;
> "C:\Program Files (x86)\Puppet Labs\Puppet
> Enterprise\sys\ruby\bin\ruby.exe" -rubygems -C"C:\Program Files
> (x86)\Puppet Labs\Puppet Enterprise\service" "C:\Program Files
(x86)\Puppet
> Labs\Puppet Enterprise\service\daemon.rb"
>
> I''m not sure if the "-rubygems" is required but it is in
the daemon.bat
> file.
>
> I think this is necessary, at least it was in ruby 1.8, not entirely sure
about 1.9.

Josh

-- 
Josh Cooper
Developer, Puppet Labs

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CA%2Bu97umBtRjJ0%3DCpqBNGvNodgY6PC3SuBvhV4x_Phei3jxTswA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Wed, Oct 30, 2013 at 6:06 PM, Glenn Sarti <glenn.sarti2@gmail.com>
wrote:
>  * A side effect of not running the daemon from a CMD.EXE was that the
>> call to get to runinterval was failing.  I suspect this is due to
STDOUT
>> not being available anymore.  So I used the well worn method of pipe
the
>> output to a file and read that instead (Lines 60-79). I still need to
try
>> RUBY.EXE instead of RUBYW.EXE and see if it makes a difference.
>>
>
> Looks like RUBY.EXE will fix my issue as per Puppet Bug #21043. 
I''ll
> update my daemon.rb and repost.
> https://projects.puppetlabs.com/issues/21043
>
Yes, sadly rubyw.exe is broken in 1.9: http://bugs.ruby-lang.org/issues/7239
.

Josh

-- 
Josh Cooper
Developer, Puppet Labs

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CA%2Bu97u%3DGpiMRZbe2YAhM6oEKiHa-YR6%3D9XM3-EPzL59Wp5Z6fw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Thursday, October 31, 2013 11:54:19 AM UTC+8, Josh Cooper wrote:
>
>  On Wed, Oct 30, 2013 at 8:41 PM, Glenn Sarti
<glenn....@gmail.com<javascript:>
> > wrote:
>  
>>  Well that was easier than I expected
>>  
>> * daemon.rb now defaults to logging in the Event Log and optionally to 
>> the windows.log file
>>
>
> This should resolve https://projects.puppetlabs.com/issues/21641. Can you 
> submit a PR for this issue?
>  
> Josh
>  
> Josh Cooper
>
  Developer, Puppet Labs
>
 
Pull Request #2021 created.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/9fa235eb-1319-454d-afdd-216f9ef07cd8%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.