Puppet users - May 2007 - Ticket 616, service resource takes over puppetd port

Hello Luke,

After our services do now successfully restart, we hit now the already
opened bug where the restarted service takes over puppetd port if it''s
configured to listen.

One solution is to remove the listen option but it''s very nice to
invoke a
puppetrun on the clients from the central server without waiting for its
next run.

Where you already able to take a look? It tried to debug a bit in the
execute
function in util.rb but I''m not very familiar with ruby and it looks
like the handling
differs from a C-fork. Perhaps it''s easier to spawn a thread for the
execute
function instead forking away another process to avoid that puppetd hangs
around while the service is restarting (and maybe blocking).

Cheers,
Andreas

On May 30, 2007, at 2:39 AM, Andreas Unterkircher wrote:
> Hello Luke,
>
> After our services do now successfully restart, we hit now the already
> opened bug where the restarted service takes over puppetd port if
it''s
> configured to listen.
>
> One solution is to remove the listen option but it''s very nice to
> invoke a
> puppetrun on the clients from the central server without waiting  
> for its
> next run.
>
> Where you already able to take a look? It tried to debug a bit in the
> execute
> function in util.rb but I''m not very familiar with ruby and it
looks
> like the handling
> differs from a C-fork. Perhaps it''s easier to spawn a thread for  
> the execute
> function instead forking away another process to avoid that puppetd  
> hangs
> around while the service is restarting (and maybe blocking).
I spent a long time looking at this problem a while ago and was not  
able to figure it out.  I have not yet had a chance to look it again,  
but I hope to do so soon.

Obviously, if anyone has some time to look at it, I would appreciate  
the help.

  --
  To my embarrassment I was born in bed with a lady.
              --Wilson Mizner
  ---------------------------------------------------------------------
  Luke Kanies | http://reductivelabs.com | http://madstop.com

On Sun, Jun 03, 2007 at 08:53:32PM -0500, Luke Kanies
wrote:
> I spent a long time looking at this problem a while ago and was not  
> able to figure it out.  I have not yet had a chance to look it again,  
> but I hope to do so soon.
> 
> Obviously, if anyone has some time to look at it, I would appreciate  
> the help.
It seems that puppet doesn''t set close-on-exec flag for opened files so
they get inherited by the children. For example, lsof lists these open
files for puppetd:

ruby    2383 root    0u   CHR        3,0           1713 /dev/ttyp0
ruby    2383 root    1u   CHR        3,0           1713 /dev/ttyp0
ruby    2383 root    2u   CHR        3,0           1713 /dev/ttyp0
ruby    2383 root    3r  FIFO        0,6           7841 pipe
ruby    2383 root    4w  FIFO        0,6           7841 pipe
ruby    2383 root    5u  unix 0xc76cb6c0           7842 socket
ruby    2383 root    6u   REG        8,1   20762 182340 /var/log/puppet/http.log
ruby    2383 root    7u   REG        8,1   20762 182340 /var/log/puppet/http.log
ruby    2383 root    8u   REG        8,1   20762 182340 /var/log/puppet/http.log
ruby    2383 root    9u  IPv4       7848            TCP *:8139 (LISTEN)
ruby    2383 root   10r   REG        8,1     598  49553
/etc/ssh/ssh_host_dsa_key.pub
ruby    2383 root   11r   REG        8,1     390  49550
/etc/ssh/ssh_host_rsa_key.pub

The same files are open in the ntpd started by puppetd:

ntpd    2436  ntp    0u   CHR        1,3            753 /dev/null
ntpd    2436  ntp    1u   CHR        1,3            753 /dev/null
ntpd    2436  ntp    2u   CHR        1,3            753 /dev/null
ntpd    2436  ntp    3r  FIFO        0,6           7841 pipe
ntpd    2436  ntp    4w  FIFO        0,6           7841 pipe
ntpd    2436  ntp    5u  unix 0xc76cb860           7895 socket
ntpd    2436  ntp    6u   REG        8,1   20762 182340 /var/log/puppet/http.log
ntpd    2436  ntp    7u   REG        8,1   20762 182340 /var/log/puppet/http.log
ntpd    2436  ntp    8u   REG        8,1   20762 182340 /var/log/puppet/http.log
ntpd    2436  ntp    9u  IPv4       7848            TCP *:8139 (LISTEN)
ntpd    2436  ntp   10r   REG        8,1     598  49553
/etc/ssh/ssh_host_dsa_key.pub
ntpd    2436  ntp   11r   REG        8,1     390  49550
/etc/ssh/ssh_host_rsa_key.pub
ntpd    2436  ntp   12w   REG        8,1   21975 182323
/var/log/ntpstats/peerstats.20070604
ntpd    2436  ntp   13w   REG        8,1     596 182384
/var/log/ntpstats/loopstats.20070604
ntpd    2436  ntp   16u  IPv4       7899            UDP *:ntp 
ntpd    2436  ntp   17u  IPv6       7900            UDP *:ntp 
ntpd    2436  ntp   18u  IPv6       7901            UDP [::1]:ntp 
ntpd    2436  ntp   19u  IPv6       7902            UDP
[fe80::216:3eff:fe27:4eea]:ntp
ntpd    2436  ntp   20u  IPv4       7903            UDP front.local:ntp 
ntpd    2436  ntp   21u  IPv4       7904            UDP dhcp132.nat11.irb.lo:ntp

You need to call io.fcntl(Fcntl::F_SETFD, Fcntl::FD_CLOEXEC) on every
opened io object that has a file descriptor associated.

-- 
Valentin

On Mon, Jun 04, 2007 at 12:05:00PM +0200, Valentin Vidic
wrote:
> You need to call io.fcntl(Fcntl::F_SETFD, Fcntl::FD_CLOEXEC) on every
> opened io object that has a file descriptor associated.
Below is a small patch that sets this flag on the socket descriptor.
Similar change should be done for other open file descriptors.

It seems the socket problem is a bug in WEBrick, since it also tries to
set FD_CLOEXEC but calls fcntl with the wrong parameters - 

  io.fcntl(Fcntl::FD_CLOEXEC, 1)

instead of

  io.fcntl(Fcntl::F_SETFD, Fcntl::FD_CLOEXEC)

-- 
Valentin


_______________________________________________
Puppet-users mailing list
Puppet-users@madstop.com
https://mail.madstop.com/mailman/listinfo/puppet-users

Valentin Vidic wrote:
> Below is a small patch that sets this flag on the socket descriptor.
> Similar change should be done for other open file descriptors.
>   
So far I can confirm that this patch worked during a quick test
made on two puppet nodes. The restarted services do not get
inherited the listening ports if I stop puppet afterwards.

Yippi! :)

Cheers,
Andreas

On Jun 4, 2007, at 1:11 PM, Andreas Unterkircher wrote:
> Valentin Vidic wrote:
>> Below is a small patch that sets this flag on the socket descriptor.
>> Similar change should be done for other open file descriptors.
>>
> So far I can confirm that this patch worked during a quick test
> made on two puppet nodes. The restarted services do not get
> inherited the listening ports if I stop puppet afterwards.
Very good news.  I''ll get it applied as soon as I finish the chunk  
I''m working on now.

Once I get this batch of code finished, my goal is to clean up for a  
release ASAP.   I''ll be making an interim milestone (before elmo) and  
move a few of the more critical tickets to that milestone, including  
this one.

  --
  The most likely way for the world to be destroyed, most experts agree,
  is by accident. That''s where we come in; we''re computer
professionals.
  We cause accidents.      --Nathaniel Borenstein
  ---------------------------------------------------------------------
  Luke Kanies | http://reductivelabs.com | http://madstop.com

On Mon, Jun 04, 2007 at 01:40:48PM -0500, Luke Kanies
wrote:
> Very good news.  I''ll get it applied as soon as I finish the chunk
> I''m working on now.
Here is a few more patches to fix the rest of the leaking file
descriptors (listed in the previous posts):

* facter.diff - close the files with ssh host keys (for some reason they
  are not closed automatically)
* event-loop.diff - prevent leaking of the pipe file descriptors
* webrick.diff - prevent leaking of the webrick log file descriptors

-- 
Valentin




_______________________________________________
Puppet-users mailing list
Puppet-users@madstop.com
https://mail.madstop.com/mailman/listinfo/puppet-users

On Jun 4, 2007, at 3:21 PM, Valentin Vidic wrote:
>
> Here is a few more patches to fix the rest of the leaking file
> descriptors (listed in the previous posts):
>
> * facter.diff - close the files with ssh host keys (for some reason  
> they
>   are not closed automatically)
> * event-loop.diff - prevent leaking of the pipe file descriptors
> * webrick.diff - prevent leaking of the webrick log file descriptors
Okay, all four patches have been applied.  Thank you!

  --
  My favorite was a professor at a University I Used To Be Associated
  With who claimed that our requirement of a non-alphabetic character in
  our passwords was an abridgement of his freedom of speech.
                  -- Jacob Haller
  ---------------------------------------------------------------------
  Luke Kanies | http://reductivelabs.com | http://madstop.com