win32utils devel - Nov 2006 - fatal flaw in popen4 on windows? [WAS] Re: Nonblocking IO read (fwd)

Hi Ara,

I''ll take a look.  In the meantime, I''ve cc''d the
win32utils-devel
mailing list to see if anyone has any ideas/suggestions.

Regards,

Dan
> -----Original Message-----
> From: ara.t.howard at noaa.gov [mailto:ara.t.howard at noaa.gov] 
> Sent: Wednesday, November 01, 2006 12:36 PM
> To: Berger, Daniel
> Subject: fatal flaw in popen4 on windows? [WAS] Re: 
> Nonblocking IO read (fwd)
> 
> 
> 
> hi dan-
> 
> forgot to cc you on this...  hopefully you''ll have some
suggestions?
> 
> regards.
> 
> -a
> -- 
> my religion is very simple.  my religion is kindness. -- the 
> dalai lama
> 
> ---------- Forwarded message ----------
> Date: Wed, 1 Nov 2006 12:19:12 -0700 (MST)
> From: ara.t.howard at noaa.gov
> To: ruby-talk ML <ruby-talk at ruby-lang.org>
> Subject: fatal flaw in popen4 on windows? [WAS] Re: 
> Nonblocking IO read
> 
> 
> On Thu, 2 Nov 2006, Robert Klemme wrote:
> 
> > Tom Pollard wrote:
> >> Anyway, you would only need nonblocking IO if you wanted 
> to read bits 
> >> of the
> >> stderr stream before the command exited, but that doesn''t
> sound like what 
> >> you''re want.
> > 
> > Actually this is not correct: if there is a lot written to 
> stderr then 
> > you need to read that concurrently.  If you do not do that then the 
> > process will block on some stderr write operation that fills up the 
> > pipe and you get a deadlock because your code waits for process 
> > termination.
> 
> not only is that true but, afaik, it''s why popen4 cannot even 
> work on windows! this program will eventually hang on either 
> windows or unix
> 
> 
>      harp:~ > cat a.rb
>      require ''rubygems''
>      require ''popen4''
> 
>      n = (ARGV.shift || 4242).to_i
>      ruby = ARGV.shift || ''ruby''
> 
>      system "ruby -e 42" or abort "ruby not in your
path!"
> 
>      STDOUT.sync = STDERR.sync = true
> 
>      program = <<-program
>        #{ n }.times do
>          t = Time.now.to_f
>          STDOUT.puts t
>          STDERR.puts t
>        end
>      program
> 
> 
>      POpen4.popen4(ruby) do |stdout, stderr, stdin, pid|
>        STDOUT.puts pid
> 
>        stdin.puts program
>        stdin.close
> 
>        Thread.new{ stdout.each{|line| STDOUT.puts line} }
>        #
>        # uncomment and it won''t hang!!!
>        #
>        #Thread.new{ stderr.each{|line| STDERR.puts line} }
>      end
> 
>      puts ''done''
> 
> 
> 
> however, on windows it will always hang - even if the line 
> above is uncommented.  this is because if one popen4s a 
> process it''s __essential__, as robert correctly points out, 
> to continually consume any stdout or stderr produced - 
> otherwise the program will eventually get stuck in EPIPE and 
> you''ll be waiting for this stuck program.
> 
> and here''s the rub: you cannot reliably consume both stdout 
> and stderr under windows using threads or, afaik, nonblocking 
> io.  perhaps the new nonblock_* methods could help with this? 
>  it''d no doubt be a major reworking...
> 
> *** i''m really hoping someone will chime in here and prove me 
> wrong ***
> 
> for reference i''m including the code for my open4 lib''s
spawn
> method: which illustrates the logical concept of what must be 
> done to avoid a subprocess blocked writing to it''s
parent''s pipes...
> 
> 
> #
> # for complete code see
> # http://codeforpeople.com/lib/ruby/open4/open4-0.9.1/lib/open4.rb
> # http://rubyforge.org/frs/?group_id=1024&release_id=7556
> #
> 
>    def spawn arg, *argv #--{{{
>      argv.unshift(arg)
>      opts = ((argv.size > 1 and Hash === argv.last) ? argv.pop : {})
>      argv.flatten!
>      cmd = argv.join('' '')
> 
>      getopt = getopts opts
> 
>      ignore_exit_failure = getopt[ ''ignore_exit_failure'',
> getopt[''quiet'', false] 
> ]
>      ignore_exec_failure = getopt[ ''ignore_exec_failure'',
> !getopt[''raise'', true] 
> ]
>      exitstatus = getopt[ %w( exitstatus exit_status status ) ]
>      stdin = getopt[ %w( stdin in i 0 ) << 0 ]
>      stdout = getopt[ %w( stdout out o 1 ) << 1 ]
>      stderr = getopt[ %w( stderr err e 2 ) << 2 ]
>      pid = getopt[ ''pid'' ]
>      timeout = getopt[ %w( timeout spawn_timeout ) ]
>      stdin_timeout = getopt[ %w( stdin_timeout ) ]
>      stdout_timeout = getopt[ %w( stdout_timeout io_timeout ) ]
>      stderr_timeout = getopt[ %w( stderr_timeout ) ]
>      status = getopt[ %w( status ) ]
>      cwd = getopt[ %w( cwd dir ), Dir.pwd ]
> 
>      exitstatus >        case exitstatus
>          when TrueClass, FalseClass
>            ignore_exit_failure = true if exitstatus
>            [0]
>          else
>            [*(exitstatus || 0)].map{|i| Integer i}
>        end
> 
>      stdin ||= '''' if stdin_timeout
>      stdout ||= '''' if stdout_timeout
>      stderr ||= '''' if stderr_timeout
> 
>      started = false
> 
>      status >        begin
>          Dir.chdir(cwd) do
>            Timeout::timeout(timeout) do
>              popen4(*argv) do |c, i, o, e|
>                started = true
> 
>                %w( replace pid= << push update ).each do |msg|
>                  break(pid.send(msg, c)) if pid.respond_to? msg
>                end
> 
> #
> # this is the critical bit!!!
> #
> 
>                te = ThreadEnsemble.new c
> 
>                te.add_thread(i, stdin) do |i, stdin|
>                  relay stdin, i, stdin_timeout
>                  i.close rescue nil
>                end
> 
>                te.add_thread(o, stdout) do |o, stdout|
>                  relay o, stdout, stdout_timeout
>                end
> 
>                te.add_thread(e, stderr) do |o, stderr|
>                  relay e, stderr, stderr_timeout
>                end
> 
>                te.run
>              end
>            end
>          end
>        rescue
>          raise unless(not started and ignore_exec_failure)
>        end
> 
>      raise SpawnError.new(cmd, status) unless
>        (ignore_exit_failure or (status.nil? and 
> ignore_exec_failure) or 
> exitstatus.include?(status.exitstatus))
> 
>      status
> #--}}}
>    end
> 
> 
> kind regards.
> 
> -a
> -- 
> my religion is very simple.  my religion is kindness. -- the 
> dalai lama
> 
> -a
> -- 
> my religion is very simple.  my religion is kindness. -- the 
> dalai lama
> 

This communication is the property of Qwest and may contain confidential or
privileged information. Unauthorized use of this communication is strictly 
prohibited and may be unlawful.  If you have received this communication 
in error, please immediately notify the sender by reply e-mail and destroy 
all copies of the communication and any attachments.