Referring to "CLOSED FIXED" Bug 85: https://bugzilla.mindrot.org/show_bug.cgi?id=85 Assuming that you have your machine setup so that the following commands run without prompting: ssh -2 localhost pwd ssh -1 localhost pwd Then this command: ssh -1 localhost yes | true always produces this output: Write failed flushing stdout buffer. write stdout: Broken pipe Yet this command has varying behavior: ssh -2 localhost yes | true Sometimes it returns right away without producing any output at all (which begs the question why the output is so different between -1 and -2, but that's not the primary question of this email). The no-output case happens on Ubuntu where the output of ssh -V is: OpenSSH_5.1p1 Debian-3ubuntu1, OpenSSL 0.9.8g 19 Oct 2007 The same thing happens when running both server AND client from a MacPorts installation where the output of ssh -V is: OpenSSH_5.2p1, OpenSSL 0.9.8k 25 Mar 2009 HOWEVER, the "ssh -2 localhost yes | true" command just hangs indefinitely on Mac OS X 10.5.6 where the output of ssh -V is: OpenSSH_5.1p1, OpenSSL 0.9.7l 28 Sep 2006 It turns out that it's the version of sshd that seems to matter as the older ssh OpenSSL 0.9.7l client talking to a newer sshd daemon doesn't hang (and the logs at the end of this message seem to bear that out). Now back to the "CLOSED FIXED" Bug 85. The last comment is: ---- Comment #9 From Damien Miller 2008-07-22 12:06:22 ---- Mass update RESOLVED->CLOSED after release of openssh-5.1 The problem ssh version on Mac OS X is marked "OpenSSH_5.1p1" and yet it clearly has the bug. The comment on Bug 85 leads me to believe it should have the fix but the date of the comment suggests that it might not (which turns out to be the case). And the bug doesn't mention any dependency on and configure options or on a particular version of OpenSSL for the fix. Can anyone shed some light on how to tell based on the versions reported by ssh -V / sshd -V whether or not the hanging problem will be present? And is there anyway to see the OpenSSL version of the remote sshd via ssh from another host? (The -v option to ssh only reports "remote software version OpenSSH_5.1" and does not include the OpenSSL version.) Thanks, Kyle P.S. No, I don't actually need to run "ssh localhost yes | true" however I often use Subversion with the svn+ssh protocol and the recent Subversion fix for issue 2580 creates a situation very much like "ssh localhost yes | true" where the ssh process never exits. http://subversion.tigris.org/issues/show_bug.cgi?id=2580 P.P.S. The last part of the debug log from ssh client when it hangs: ... debug2: channel 0: write failed debug2: channel 0: close_write debug2: channel 0: send eow debug2: channel 0: output open -> closed connection hangs at this point and sshd log looks like this: ... debug1: server_input_channel_req: channel 0 request eow at openssh.com reply 0 debug2: channel 0: rcvd eow debug2: channel 0: close_read debug2: channel 0: input open -> closed connection hangs at this point whereas when it doesn't hang, ssh client log looks like this: ... debug2: channel 0: write failed debug2: channel 0: close_write debug2: channel 0: send eow debug2: channel 0: output open -> closed debug1: client_input_channel_req: channel 0 rtype exit-signal reply 0 debug1: client_input_channel_req: channel 0 rtype eow at openssh.com reply 0 debug2: channel 0: rcvd eow ... and sshd log looks like this: ... debug1: server_input_channel_req: channel 0 request eow at openssh.com reply 0 debug2: channel 0: rcvd eow debug2: channel 0: close_read debug2: channel 0: input open -> closed debug1: Received SIGCHLD. debug1: session_by_pid: pid 24887 debug1: session_exit_message: session 0 channel 0 pid 24887 debug2: channel 0: request exit-signal confirm 0 debug1: session_exit_message: release channel 0 debug2: channel 0: write failed debug2: channel 0: close_write debug2: channel 0: send eow ...
On Mon, 20 Apr 2009, Kyle McKay wrote:> Referring to "CLOSED FIXED" Bug 85: > > https://bugzilla.mindrot.org/show_bug.cgi?id=85...> HOWEVER, the "ssh -2 localhost yes | true" command just > hangs indefinitely on Mac OS X 10.5.6 where the output of > ssh -V is: > > OpenSSH_5.1p1, OpenSSL 0.9.7l 28 Sep 2006 > > It turns out that it's the version of sshd that seems to matter > as the older ssh OpenSSL 0.9.7l client talking to a newer sshd > daemon doesn't hang (and the logs at the end of this message > seem to bear that out).This doesn't look like bug#85 based on what you sent below. It looks more like a whatever that you were running failing to notice its stdout closing. Either way, the problem is far more likely to be due to modifications that Apple have made to the ssh/sshd they ship in their base OS rather than anything to do with OpenSSL.> Can anyone shed some light on how to tell based on the versions > reported by ssh -V / sshd -V whether or not the hanging problem > will be present?OpenSSH 5.1 has the fix. Since we had to extend the SSH protocol to fix this bug, both ends (ssh and sshd) need to be this version or greater for it to work.> And is there anyway to see the OpenSSL version of the remote > sshd via ssh from another host? (The -v option to ssh only > reports "remote software version OpenSSH_5.1" and does > not include the OpenSSL version.)No, the OpenSSL version is really not relevant since it handles crypto and not much else.> P.S. No, I don't actually need to run "ssh localhost yes | true" > however I often use Subversion with the svn+ssh protocol and the > recent Subversion fix for issue 2580 creates a situation very > much like "ssh localhost yes | true" where the ssh process never > exits. > > http://subversion.tigris.org/issues/show_bug.cgi?id=2580 > > P.P.S. The last part of the debug log from ssh client when it > hangs: > > ... > debug2: channel 0: write failed > debug2: channel 0: close_write > debug2: channel 0: send eow > debug2: channel 0: output open -> closed > connection hangs at this pointThe client is doing the right thing.> and sshd log looks like this: > > ... > debug1: server_input_channel_req: channel 0 request eow at openssh.com > reply 0 > debug2: channel 0: rcvd eow > debug2: channel 0: close_read > debug2: channel 0: input open -> closed > connection hangs at this pointThe server is doing the right thing, but the underlying process is not terminating when its output it closed.> whereas when it doesn't hang, ssh client log looks like this: > > ... > debug2: channel 0: write failed > debug2: channel 0: close_write > debug2: channel 0: send eow > debug2: channel 0: output open -> closed > debug1: client_input_channel_req: channel 0 rtype exit-signal reply 0 > debug1: client_input_channel_req: channel 0 rtype eow at openssh.com > reply 0 > debug2: channel 0: rcvd eow > ...Yes, the client signals that its input fd has closed, the server closed the output fd of its process and the process exited.> and sshd log looks like this: > > ... > debug1: server_input_channel_req: channel 0 request eow at openssh.com > reply 0 > debug2: channel 0: rcvd eow^^^ this is the client signalling that its input has closed> debug2: channel 0: close_read > debug2: channel 0: input open -> closed^^^ server closes output of child process> debug1: Received SIGCHLD.^^^ child process exits> debug1: session_by_pid: pid 24887 > debug1: session_exit_message: session 0 channel 0 pid 24887 > debug2: channel 0: request exit-signal confirm 0 > debug1: session_exit_message: release channel 0^^^ server sends notification that it has exited. -d
On Thu, 23 Apr 2009, Kyle wrote:> I filed bug 6810722 with Apple and sent them a patch that makes the half-close > fix work again without causing problems with shells (like Apple's bash that > attempt to detect when they're being run by a remote shell daemon such as sshd > by apparently checking standard input to see if it's a socket). > > On Apr 21, 2009, at 01:27, Kyle McKay wrote: > > On Apr 20, 2009, at 23:53, Damien Miller wrote: > > > There may be differences depending on whether > > > Apple has modified their sshd to avoid using pipes (thwarting the > > > half-close fix in the process), but you will need to post a full debug > > > log from the server to tell. > > > > > > -d > > > > Yup, you're right. Apple has disabled the USE_PIPES define in their source > > code. > > > FYI, patch is below.I think it violates some assumptions we make in channels.c to mix socketpairs and pipes like this, but I have to check. -d