mathew
2015-Feb-09 21:50 UTC
Connection stalls at debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
On Mon Feb 09 2015 at 1:23:37 PM Petr Lautrbach <plautrba at redhat.com> wrote:> It seems to be the same problem as described and discussed in this > [1] thread. MTU 1400 is not enough for packet sent by > openssh-6.6.1p1-11.1.fc21 with default settings. The size of one > of initial packets could be even 1968. Your VPN probably makes > a fragmentation but doesn't do the correct defragmentation.Connections to other servers across the same VPN, using the same OpenSSH versions, succeed. However, I've located a second server on the same subnet that's running OpenSSH 5.9p1 -- would you expect the same problem with that version? Seems like everything on that particular subnet/at that particular site is affected.> As a workaround you can set shorter lists of MACs used by your client, eg: > > $ ssh -m hmac-sha1 ... >I already checked the FAQ and tried that, but it doesn't seem to help. % ./ssh -vvv -m hmac-sha1 docs.rtp.tecnet OpenSSH_6.7p1, OpenSSL 1.0.1k-fips 8 Jan 2015 debug1: Reading configuration data /home/meta/.ssh/config [...] debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<7680<8192) sent debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP On Mon Feb 09 2015 at 1:29:17 PM Darren Tucker <dtucker at zip.com.au> wrote:> I'd add "if you run netstat on both ends and see "SendQ" non-zero and not > decreasing then this is likely your problem. >With the -m parameter as above, running ss on the client, I see Send-Q go to 1208 and then sit there until I Ctrl-C out the client, when it increases by 1. On the server side, I see nothing. Is that plausible, that the client would proceed all the way through to DH_GEX_GROUP without seeing any data? Without the -m parameter Send-Q goes to 1992. 1208 bytes shouldn't need any fragmentation; I can definitely ping unfragmented packets that large. So I'm thinking firewall problem now, but I'm at a loss as to why OpenSSH is triggering the problem but PuTTY isn't, given that reducing packet size below the MTU limit doesn't seem to help. Any ideas? Thanks, mathew
Darren Tucker
2015-Feb-09 22:11 UTC
Connection stalls at debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
On Mon, Feb 9, 2015 at 4:50 PM, mathew <meta at pobox.com> wrote: [...]> > Connections to other servers across the same VPN, using the same OpenSSH > versions, succeed. >The ciphers offered by a given version of OpenSSH can also vary based on the version of OpenSSL they were compiled against.> However, I've located a second server on the same subnet that's running > OpenSSH 5.9p1 -- would you expect the same problem with that version? >It depends.> With the -m parameter as above, running ss on the client, I see Send-Q go > to 1208 and then sit there until I Ctrl-C out the client, when it increases > by 1. On the server side, I see nothing. Is that plausible, that the client > would proceed all the way through to DH_GEX_GROUP without seeing any data? >No, but the size of the packets sent before that point can be small enough to not trigger MTU problems. Without the -m parameter Send-Q goes to 1992.>What's -m? my netstat doesn't have it.> 1208 bytes shouldn't need any fragmentation; I can definitely ping > unfragmented packets that large. > > So I'm thinking firewall problem now, but I'm at a loss as to why OpenSSH > is triggering the problem but PuTTY isn't, given that reducing packet size > below the MTU limit doesn't seem to help. Any ideas? >There's 3 things that get negotiated: key exchange algorithms (KexAlgorithms), message authentication codes (MACs) and encryption ciphers (Cipers). These vary by SSH implementation and version (and in the case of OpenSSH, the version of libcrypto too). In OpenSSH, the cipher selected also influences the size of the Diffie-Hellman group requested (this is controlled by the client, and was increased in a recent OpenSSH release) Try: ssh -vvv -o KexAlgorithms=diffie-hellman-group14-sha1 yoursever ssh -vvv -o Ciphers=aes128-ctr yoursever In particular, make a note of this line in the debug output: debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<3072<8192) sent This is the lower, preferred and upper sizes requested. Exactly which size gets sent will depend on the content of the servers "moduli" file. debug2: bits set: 1531/3072 The 2nd number is the size the server actually sent (you'll probably need to either run the server in debug mode or kick its loglevel to debug2 to see this for the failure case, though, since it's not making it through). -- Darren Tucker (dtucker at zip.com.au) GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4 37C9 C982 80C7 8FF4 FA69 Good judgement comes with experience. Unfortunately, the experience usually comes from bad judgement.
mathew
2015-Feb-09 22:42 UTC
Connection stalls at debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
More info: We've checked firewall logs, and it seems to be a firewall rule designed to prevent sessions which are subject to the bug detailed at < http://archives.neohapsis.com/archives/bugtraq/2002-06/0294.html>. I've tried explicitly setting PAMAuthenticationViaKBDInt no, KbdInteractiveAuthentication no and UsePrivilegeSeparation yes in sshd_config, but the problem still occurs, so I think the firewall rule is buggy. So, doesn't seem to be an OpenSSH problem per se, but I'll follow up with anything more I discover in case other people encounter the issue -- it's possible that the rule in question is deployed quite widely. mathew
mathew
2015-Feb-13 18:35 UTC
Connection stalls at debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
Root cause established: A firewall appliance was replaced, and an error installing the replacement meant it wasn't receiving rule updates. So, no action at all needed by OpenSSH. Thanks, and sorry for the false alarm. mathew On Mon Feb 09 2015 at 4:42:25 PM mathew <meta at pobox.com> wrote:> More info: We've checked firewall logs, and it seems to be a firewall rule > designed to prevent sessions which are subject to the bug detailed at < > http://archives.neohapsis.com/archives/bugtraq/2002-06/0294.html>. > > I've tried explicitly setting PAMAuthenticationViaKBDInt no, > KbdInteractiveAuthentication no and UsePrivilegeSeparation yes in > sshd_config, but the problem still occurs, so I think the firewall rule is > buggy. > > So, doesn't seem to be an OpenSSH problem per se, but I'll follow up with > anything more I discover in case other people encounter the issue -- it's > possible that the rule in question is deployed quite widely. > > > mathew >