Andreas Unterkircher
2007-May-30 07:39 UTC
Ticket 616, service resource takes over puppetd port
Hello Luke, After our services do now successfully restart, we hit now the already opened bug where the restarted service takes over puppetd port if it''s configured to listen. One solution is to remove the listen option but it''s very nice to invoke a puppetrun on the clients from the central server without waiting for its next run. Where you already able to take a look? It tried to debug a bit in the execute function in util.rb but I''m not very familiar with ruby and it looks like the handling differs from a C-fork. Perhaps it''s easier to spawn a thread for the execute function instead forking away another process to avoid that puppetd hangs around while the service is restarting (and maybe blocking). Cheers, Andreas
Luke Kanies
2007-Jun-04 01:53 UTC
Re: Ticket 616, service resource takes over puppetd port
On May 30, 2007, at 2:39 AM, Andreas Unterkircher wrote:> Hello Luke, > > After our services do now successfully restart, we hit now the already > opened bug where the restarted service takes over puppetd port if it''s > configured to listen. > > One solution is to remove the listen option but it''s very nice to > invoke a > puppetrun on the clients from the central server without waiting > for its > next run. > > Where you already able to take a look? It tried to debug a bit in the > execute > function in util.rb but I''m not very familiar with ruby and it looks > like the handling > differs from a C-fork. Perhaps it''s easier to spawn a thread for > the execute > function instead forking away another process to avoid that puppetd > hangs > around while the service is restarting (and maybe blocking).I spent a long time looking at this problem a while ago and was not able to figure it out. I have not yet had a chance to look it again, but I hope to do so soon. Obviously, if anyone has some time to look at it, I would appreciate the help. -- To my embarrassment I was born in bed with a lady. --Wilson Mizner --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
Valentin Vidic
2007-Jun-04 10:05 UTC
Re: Ticket 616, service resource takes over puppetd port
On Sun, Jun 03, 2007 at 08:53:32PM -0500, Luke Kanies wrote:> I spent a long time looking at this problem a while ago and was not > able to figure it out. I have not yet had a chance to look it again, > but I hope to do so soon. > > Obviously, if anyone has some time to look at it, I would appreciate > the help.It seems that puppet doesn''t set close-on-exec flag for opened files so they get inherited by the children. For example, lsof lists these open files for puppetd: ruby 2383 root 0u CHR 3,0 1713 /dev/ttyp0 ruby 2383 root 1u CHR 3,0 1713 /dev/ttyp0 ruby 2383 root 2u CHR 3,0 1713 /dev/ttyp0 ruby 2383 root 3r FIFO 0,6 7841 pipe ruby 2383 root 4w FIFO 0,6 7841 pipe ruby 2383 root 5u unix 0xc76cb6c0 7842 socket ruby 2383 root 6u REG 8,1 20762 182340 /var/log/puppet/http.log ruby 2383 root 7u REG 8,1 20762 182340 /var/log/puppet/http.log ruby 2383 root 8u REG 8,1 20762 182340 /var/log/puppet/http.log ruby 2383 root 9u IPv4 7848 TCP *:8139 (LISTEN) ruby 2383 root 10r REG 8,1 598 49553 /etc/ssh/ssh_host_dsa_key.pub ruby 2383 root 11r REG 8,1 390 49550 /etc/ssh/ssh_host_rsa_key.pub The same files are open in the ntpd started by puppetd: ntpd 2436 ntp 0u CHR 1,3 753 /dev/null ntpd 2436 ntp 1u CHR 1,3 753 /dev/null ntpd 2436 ntp 2u CHR 1,3 753 /dev/null ntpd 2436 ntp 3r FIFO 0,6 7841 pipe ntpd 2436 ntp 4w FIFO 0,6 7841 pipe ntpd 2436 ntp 5u unix 0xc76cb860 7895 socket ntpd 2436 ntp 6u REG 8,1 20762 182340 /var/log/puppet/http.log ntpd 2436 ntp 7u REG 8,1 20762 182340 /var/log/puppet/http.log ntpd 2436 ntp 8u REG 8,1 20762 182340 /var/log/puppet/http.log ntpd 2436 ntp 9u IPv4 7848 TCP *:8139 (LISTEN) ntpd 2436 ntp 10r REG 8,1 598 49553 /etc/ssh/ssh_host_dsa_key.pub ntpd 2436 ntp 11r REG 8,1 390 49550 /etc/ssh/ssh_host_rsa_key.pub ntpd 2436 ntp 12w REG 8,1 21975 182323 /var/log/ntpstats/peerstats.20070604 ntpd 2436 ntp 13w REG 8,1 596 182384 /var/log/ntpstats/loopstats.20070604 ntpd 2436 ntp 16u IPv4 7899 UDP *:ntp ntpd 2436 ntp 17u IPv6 7900 UDP *:ntp ntpd 2436 ntp 18u IPv6 7901 UDP [::1]:ntp ntpd 2436 ntp 19u IPv6 7902 UDP [fe80::216:3eff:fe27:4eea]:ntp ntpd 2436 ntp 20u IPv4 7903 UDP front.local:ntp ntpd 2436 ntp 21u IPv4 7904 UDP dhcp132.nat11.irb.lo:ntp You need to call io.fcntl(Fcntl::F_SETFD, Fcntl::FD_CLOEXEC) on every opened io object that has a file descriptor associated. -- Valentin
Valentin Vidic
2007-Jun-04 11:30 UTC
Re: Ticket 616, service resource takes over puppetd port
On Mon, Jun 04, 2007 at 12:05:00PM +0200, Valentin Vidic wrote:> You need to call io.fcntl(Fcntl::F_SETFD, Fcntl::FD_CLOEXEC) on every > opened io object that has a file descriptor associated.Below is a small patch that sets this flag on the socket descriptor. Similar change should be done for other open file descriptors. It seems the socket problem is a bug in WEBrick, since it also tries to set FD_CLOEXEC but calls fcntl with the wrong parameters - io.fcntl(Fcntl::FD_CLOEXEC, 1) instead of io.fcntl(Fcntl::F_SETFD, Fcntl::FD_CLOEXEC) -- Valentin _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
Andreas Unterkircher
2007-Jun-04 18:11 UTC
Re: Ticket 616, service resource takes over puppetd port
Valentin Vidic wrote:> Below is a small patch that sets this flag on the socket descriptor. > Similar change should be done for other open file descriptors. >So far I can confirm that this patch worked during a quick test made on two puppet nodes. The restarted services do not get inherited the listening ports if I stop puppet afterwards. Yippi! :) Cheers, Andreas
Luke Kanies
2007-Jun-04 18:40 UTC
Re: Ticket 616, service resource takes over puppetd port
On Jun 4, 2007, at 1:11 PM, Andreas Unterkircher wrote:> Valentin Vidic wrote: >> Below is a small patch that sets this flag on the socket descriptor. >> Similar change should be done for other open file descriptors. >> > So far I can confirm that this patch worked during a quick test > made on two puppet nodes. The restarted services do not get > inherited the listening ports if I stop puppet afterwards.Very good news. I''ll get it applied as soon as I finish the chunk I''m working on now. Once I get this batch of code finished, my goal is to clean up for a release ASAP. I''ll be making an interim milestone (before elmo) and move a few of the more critical tickets to that milestone, including this one. -- The most likely way for the world to be destroyed, most experts agree, is by accident. That''s where we come in; we''re computer professionals. We cause accidents. --Nathaniel Borenstein --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
Valentin Vidic
2007-Jun-04 20:21 UTC
Re: Ticket 616, service resource takes over puppetd port
On Mon, Jun 04, 2007 at 01:40:48PM -0500, Luke Kanies wrote:> Very good news. I''ll get it applied as soon as I finish the chunk > I''m working on now.Here is a few more patches to fix the rest of the leaking file descriptors (listed in the previous posts): * facter.diff - close the files with ssh host keys (for some reason they are not closed automatically) * event-loop.diff - prevent leaking of the pipe file descriptors * webrick.diff - prevent leaking of the webrick log file descriptors -- Valentin _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
Luke Kanies
2007-Jun-05 18:13 UTC
Re: Ticket 616, service resource takes over puppetd port
On Jun 4, 2007, at 3:21 PM, Valentin Vidic wrote:> > Here is a few more patches to fix the rest of the leaking file > descriptors (listed in the previous posts): > > * facter.diff - close the files with ssh host keys (for some reason > they > are not closed automatically) > * event-loop.diff - prevent leaking of the pipe file descriptors > * webrick.diff - prevent leaking of the webrick log file descriptorsOkay, all four patches have been applied. Thank you! -- My favorite was a professor at a University I Used To Be Associated With who claimed that our requirement of a non-alphabetic character in our passwords was an abridgement of his freedom of speech. -- Jacob Haller --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com