mathew
2015-Feb-09 17:09 UTC
Connection stalls at debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
Trying to connect from Fedora 21 to CentOS 6.6, OpenSSH on both ends.
Connection is via a VPN.
Initially the connection seems good, but OpenSSH stalls at
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP.
Software version on servers:
openssh-server-5.3p1-104.el6_6.1.x86_64
openssh-5.3p1-104.el6_6.1.x86_64
Software version on client:
openssh-6.6.1p1-11.1.fc21.x86_64
also duplicated problem using local build of openssh-6.7p1.tar.gz
Connections to other CentOS 6 servers with identical SSH versions and
configurations are successful. (Configs are managed by Puppet so I'm
confident they really are identical, and I rebooted the server before the
test below.)
Connections to the problem server using Windows PuTTY SSH are successful!
(Using a Windows VM running on the same Fedora 21 client machine, inside
VirtualBox.)
VPN MTU is 1400. Ping check for packet size:
% ping -M do -s 1372 10.77.16.71
PING 10.77.16.71 (10.77.16.71) 1372(1400) bytes of data.
1380 bytes from 10.77.16.71: icmp_seq=1 ttl=61 time=69.4 ms
Server MTU is 1500, and I've confirmed that 1472-byte packets ping
successfully from other servers to the problem server.
Here's a transcript using ssh -vvv and a build of OpenSSH from
openssh-6.7p1 sources:
% ./ssh -vvv docs.rtp.tecnet
OpenSSH_6.7p1, OpenSSL 1.0.1k-fips 8 Jan 2015
debug1: Reading configuration data /home/meta/.ssh/config
/home/meta/.ssh/config line 1: Unsupported option
"gssapiauthentication"
debug2: ssh_connect: needpriv 0
debug1: Connecting to docs.rtp.tecnet [10.77.16.71] port 22.
debug1: Connection established.
debug1: identity file /home/meta/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /home/meta/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/meta/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/meta/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/meta/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/meta/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/meta/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/meta/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.7
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH_5* compat 0x0c000000
debug2: fd 3 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host "docs.rtp.tecnet" from
file
"/home/meta/.ssh/known_hosts"
debug3: load_hostkeys: loaded 0 keys
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: curve25519-sha256 at libssh.org
,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ecdsa-sha2-nistp256-cert-v01 at openssh.com,
ecdsa-sha2-nistp384-cert-v01 at openssh.com,
ecdsa-sha2-nistp521-cert-v01 at openssh.com,ssh-ed25519-cert-v01 at openssh.com,
ssh-rsa-cert-v01 at openssh.com,ssh-dss-cert-v01 at openssh.com,
ssh-rsa-cert-v00 at openssh.com,ssh-dss-cert-v00 at openssh.com
,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,
aes128-gcm at openssh.com,aes256-gcm at openssh.com,chacha20-poly1305 at
openssh.com
,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,
rijndael-cbc at lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,
aes128-gcm at openssh.com,aes256-gcm at openssh.com,chacha20-poly1305 at
openssh.com
,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,
rijndael-cbc at lysator.liu.se
debug2: kex_parse_kexinit: umac-64-etm at openssh.com,umac-128-etm at
openssh.com,
hmac-sha2-256-etm at openssh.com,hmac-sha2-512-etm at openssh.com,
hmac-sha1-etm at openssh.com,umac-64 at openssh.com,umac-128 at openssh.com
,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-md5-etm at openssh.com,
hmac-ripemd160-etm at openssh.com,hmac-sha1-96-etm at openssh.com,
hmac-md5-96-etm at openssh.com,hmac-md5,hmac-ripemd160,
hmac-ripemd160 at openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: umac-64-etm at openssh.com,umac-128-etm at
openssh.com,
hmac-sha2-256-etm at openssh.com,hmac-sha2-512-etm at openssh.com,
hmac-sha1-etm at openssh.com,umac-64 at openssh.com,umac-128 at openssh.com
,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-md5-etm at openssh.com,
hmac-ripemd160-etm at openssh.com,hmac-sha1-96-etm at openssh.com,
hmac-md5-96-etm at openssh.com,hmac-md5,hmac-ripemd160,
hmac-ripemd160 at openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib at openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib at openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit:
diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit:
aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,arcfour
debug2: kex_parse_kexinit:
aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,arcfour
debug2: kex_parse_kexinit: hmac-sha1,hmac-ripemd160
debug2: kex_parse_kexinit: hmac-sha1,hmac-ripemd160
debug2: kex_parse_kexinit: none,zlib at openssh.com
debug2: kex_parse_kexinit: none,zlib at openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: setup hmac-sha1
debug1: kex: server->client aes128-ctr hmac-sha1 none
debug2: mac_setup: setup hmac-sha1
debug1: kex: client->server aes128-ctr hmac-sha1 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<7680<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
[hangs]
Here's the config output from building OpenSSH as used above:
OpenSSH has been configured with the following options:
User binaries: /usr/local/bin
System binaries: /usr/local/sbin
Configuration files: /usr/local/etc
Askpass program: /usr/local/libexec/ssh-askpass
Manual pages: /usr/local/share/man/manX
PID file: /var/run
Privilege separation chroot path: /var/empty
sshd default user PATH:
/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
Manpage format: doc
PAM support: no
OSF SIA support: no
KerberosV support: no
SELinux support: no
Smartcard support:
S/KEY support: no
MD5 password support: no
libedit support: no
Solaris process contract support: no
Solaris project support: no
IP address in $DISPLAY hack: no
Translate v4 in v6 hack: yes
BSD Auth support: no
Random number source: OpenSSL internal ONLY
Privsep sandbox style: seccomp_filter
Host: x86_64-unknown-linux-gnu
Compiler: gcc
Compiler flags: -g -O2 -Wall -Wpointer-arith -Wuninitialized
-Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess
-Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing
-D_FORTIFY_SOURCE=2 -ftrapv -fno-builtin-memset -fstack-protector-strong
-fPIE
Preprocessor flags:
Linker flags: -Wl,-z,relro -Wl,-z,now -Wl,-z,noexecstack
-fstack-protector-strong -pie
Libraries: -lcrypto -ldl -lutil -lz -lnsl -lcrypt -lresolv
So... Any ideas what I can try next to track down the source of the problem?
mathew
Petr Lautrbach
2015-Feb-09 19:23 UTC
Connection stalls at debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
On 02/09/2015 06:09 PM, mathew wrote:> Trying to connect from Fedora 21 to CentOS 6.6, OpenSSH on both ends. > Connection is via a VPN. > > Initially the connection seems good, but OpenSSH stalls at > debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP. > > Software version on servers: > openssh-server-5.3p1-104.el6_6.1.x86_64 > openssh-5.3p1-104.el6_6.1.x86_64 > > Software version on client: > openssh-6.6.1p1-11.1.fc21.x86_64 > also duplicated problem using local build of openssh-6.7p1.tar.gz > > Connections to other CentOS 6 servers with identical SSH versions and > configurations are successful. (Configs are managed by Puppet so I'm > confident they really are identical, and I rebooted the server before the > test below.) > > Connections to the problem server using Windows PuTTY SSH are successful! > (Using a Windows VM running on the same Fedora 21 client machine, inside > VirtualBox.) > > VPN MTU is 1400. Ping check for packet size: > % ping -M do -s 1372 10.77.16.71 > PING 10.77.16.71 (10.77.16.71) 1372(1400) bytes of data. > 1380 bytes from 10.77.16.71: icmp_seq=1 ttl=61 time=69.4 ms > Server MTU is 1500, and I've confirmed that 1472-byte packets ping > successfully from other servers to the problem server.It seems to be the same problem as described and discussed in this [1] thread. MTU 1400 is not enough for packet sent by openssh-6.6.1p1-11.1.fc21 with default settings. The size of one of initial packets could be even 1968. Your VPN probably makes a fragmentation but doesn't do the correct defragmentation. As a workaround you can set shorter lists of MACs used by your client, eg: $ ssh -m hmac-sha1 ... [1] https://lists.mindrot.org/pipermail/openssh-unix-dev/2013-November/031775.html> > Here's a transcript using ssh -vvv and a build of OpenSSH from > openssh-6.7p1 sources: > > % ./ssh -vvv docs.rtp.tecnet > OpenSSH_6.7p1, OpenSSL 1.0.1k-fips 8 Jan 2015 > debug1: Reading configuration data /home/meta/.ssh/config > /home/meta/.ssh/config line 1: Unsupported option "gssapiauthentication" > debug2: ssh_connect: needpriv 0 > debug1: Connecting to docs.rtp.tecnet [10.77.16.71] port 22. > debug1: Connection established. > debug1: identity file /home/meta/.ssh/id_rsa type 1 > debug1: key_load_public: No such file or directory > debug1: identity file /home/meta/.ssh/id_rsa-cert type -1 > debug1: key_load_public: No such file or directory > debug1: identity file /home/meta/.ssh/id_dsa type -1 > debug1: key_load_public: No such file or directory > debug1: identity file /home/meta/.ssh/id_dsa-cert type -1 > debug1: key_load_public: No such file or directory > debug1: identity file /home/meta/.ssh/id_ecdsa type -1 > debug1: key_load_public: No such file or directory > debug1: identity file /home/meta/.ssh/id_ecdsa-cert type -1 > debug1: key_load_public: No such file or directory > debug1: identity file /home/meta/.ssh/id_ed25519 type -1 > debug1: key_load_public: No such file or directory > debug1: identity file /home/meta/.ssh/id_ed25519-cert type -1 > debug1: Enabling compatibility mode for protocol 2.0 > debug1: Local version string SSH-2.0-OpenSSH_6.7 > debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3 > debug1: match: OpenSSH_5.3 pat OpenSSH_5* compat 0x0c000000 > debug2: fd 3 setting O_NONBLOCK > debug3: load_hostkeys: loading entries for host "docs.rtp.tecnet" from file > "/home/meta/.ssh/known_hosts" > debug3: load_hostkeys: loaded 0 keys > debug1: SSH2_MSG_KEXINIT sent > debug1: SSH2_MSG_KEXINIT received > debug2: kex_parse_kexinit: curve25519-sha256 at libssh.org > ,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1 > debug2: kex_parse_kexinit: ecdsa-sha2-nistp256-cert-v01 at openssh.com, > ecdsa-sha2-nistp384-cert-v01 at openssh.com, > ecdsa-sha2-nistp521-cert-v01 at openssh.com,ssh-ed25519-cert-v01 at openssh.com, > ssh-rsa-cert-v01 at openssh.com,ssh-dss-cert-v01 at openssh.com, > ssh-rsa-cert-v00 at openssh.com,ssh-dss-cert-v00 at openssh.com > ,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-rsa,ssh-dss > debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr, > aes128-gcm at openssh.com,aes256-gcm at openssh.com,chacha20-poly1305 at openssh.com > ,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour, > rijndael-cbc at lysator.liu.se > debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr, > aes128-gcm at openssh.com,aes256-gcm at openssh.com,chacha20-poly1305 at openssh.com > ,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour, > rijndael-cbc at lysator.liu.se > debug2: kex_parse_kexinit: umac-64-etm at openssh.com,umac-128-etm at openssh.com, > hmac-sha2-256-etm at openssh.com,hmac-sha2-512-etm at openssh.com, > hmac-sha1-etm at openssh.com,umac-64 at openssh.com,umac-128 at openssh.com > ,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-md5-etm at openssh.com, > hmac-ripemd160-etm at openssh.com,hmac-sha1-96-etm at openssh.com, > hmac-md5-96-etm at openssh.com,hmac-md5,hmac-ripemd160, > hmac-ripemd160 at openssh.com,hmac-sha1-96,hmac-md5-96 > debug2: kex_parse_kexinit: umac-64-etm at openssh.com,umac-128-etm at openssh.com, > hmac-sha2-256-etm at openssh.com,hmac-sha2-512-etm at openssh.com, > hmac-sha1-etm at openssh.com,umac-64 at openssh.com,umac-128 at openssh.com > ,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-md5-etm at openssh.com, > hmac-ripemd160-etm at openssh.com,hmac-sha1-96-etm at openssh.com, > hmac-md5-96-etm at openssh.com,hmac-md5,hmac-ripemd160, > hmac-ripemd160 at openssh.com,hmac-sha1-96,hmac-md5-96 > debug2: kex_parse_kexinit: none,zlib at openssh.com,zlib > debug2: kex_parse_kexinit: none,zlib at openssh.com,zlib > debug2: kex_parse_kexinit: > debug2: kex_parse_kexinit: > debug2: kex_parse_kexinit: first_kex_follows 0 > debug2: kex_parse_kexinit: reserved 0 > debug2: kex_parse_kexinit: > diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 > debug2: kex_parse_kexinit: ssh-rsa,ssh-dss > debug2: kex_parse_kexinit: > aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,arcfour > debug2: kex_parse_kexinit: > aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,arcfour > debug2: kex_parse_kexinit: hmac-sha1,hmac-ripemd160 > debug2: kex_parse_kexinit: hmac-sha1,hmac-ripemd160 > debug2: kex_parse_kexinit: none,zlib at openssh.com > debug2: kex_parse_kexinit: none,zlib at openssh.com > debug2: kex_parse_kexinit: > debug2: kex_parse_kexinit: > debug2: kex_parse_kexinit: first_kex_follows 0 > debug2: kex_parse_kexinit: reserved 0 > debug2: mac_setup: setup hmac-sha1 > debug1: kex: server->client aes128-ctr hmac-sha1 none > debug2: mac_setup: setup hmac-sha1 > debug1: kex: client->server aes128-ctr hmac-sha1 none > debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<7680<8192) sent > debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP > [hangs] > > Here's the config output from building OpenSSH as used above: > > OpenSSH has been configured with the following options: > User binaries: /usr/local/bin > System binaries: /usr/local/sbin > Configuration files: /usr/local/etc > Askpass program: /usr/local/libexec/ssh-askpass > Manual pages: /usr/local/share/man/manX > PID file: /var/run > Privilege separation chroot path: /var/empty > sshd default user PATH: > /usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin > Manpage format: doc > PAM support: no > OSF SIA support: no > KerberosV support: no > SELinux support: no > Smartcard support: > S/KEY support: no > MD5 password support: no > libedit support: no > Solaris process contract support: no > Solaris project support: no > IP address in $DISPLAY hack: no > Translate v4 in v6 hack: yes > BSD Auth support: no > Random number source: OpenSSL internal ONLY > Privsep sandbox style: seccomp_filter > > Host: x86_64-unknown-linux-gnu > Compiler: gcc > Compiler flags: -g -O2 -Wall -Wpointer-arith -Wuninitialized > -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess > -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing > -D_FORTIFY_SOURCE=2 -ftrapv -fno-builtin-memset -fstack-protector-strong > -fPIE > Preprocessor flags: > Linker flags: -Wl,-z,relro -Wl,-z,now -Wl,-z,noexecstack > -fstack-protector-strong -pie > Libraries: -lcrypto -ldl -lutil -lz -lnsl -lcrypt -lresolv > > So... Any ideas what I can try next to track down the source of the problem? > > > mathew > _______________________________________________ > openssh-unix-dev mailing list > openssh-unix-dev at mindrot.org > https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev >-- Petr Lautrbach
Darren Tucker
2015-Feb-09 19:28 UTC
Connection stalls at debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
On Mon, Feb 9, 2015 at 2:23 PM, Petr Lautrbach <plautrba at redhat.com> wrote:> [...] > It seems to be the same problem as described and discussed in this [1] > thread. MTU 1400 is not enough for packet sent by > openssh-6.6.1p1-11.1.fc21 with default settings. The size of one of > initial packets could be even 1968. Your VPN probably makes a > fragmentation but doesn't do the correct defragmentation. As a > workaround you can set shorter lists of MACs used by your client, eg: >I wrote an FAQ entry for this a long time ago: http://www.snailbook.com/faq/mtu-mismatch.auto.html I'd add "if you run netstat on both ends and see "SendQ" non-zero and not decreasing then this is likely your problem. I should add this to the openssh.com faq.... -- Darren Tucker (dtucker at zip.com.au) GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4 37C9 C982 80C7 8FF4 FA69 Good judgement comes with experience. Unfortunately, the experience usually comes from bad judgement.
mathew
2015-Feb-09 21:50 UTC
Connection stalls at debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
On Mon Feb 09 2015 at 1:23:37 PM Petr Lautrbach <plautrba at redhat.com> wrote:> It seems to be the same problem as described and discussed in this > [1] thread. MTU 1400 is not enough for packet sent by > openssh-6.6.1p1-11.1.fc21 with default settings. The size of one > of initial packets could be even 1968. Your VPN probably makes > a fragmentation but doesn't do the correct defragmentation.Connections to other servers across the same VPN, using the same OpenSSH versions, succeed. However, I've located a second server on the same subnet that's running OpenSSH 5.9p1 -- would you expect the same problem with that version? Seems like everything on that particular subnet/at that particular site is affected.> As a workaround you can set shorter lists of MACs used by your client, eg: > > $ ssh -m hmac-sha1 ... >I already checked the FAQ and tried that, but it doesn't seem to help. % ./ssh -vvv -m hmac-sha1 docs.rtp.tecnet OpenSSH_6.7p1, OpenSSL 1.0.1k-fips 8 Jan 2015 debug1: Reading configuration data /home/meta/.ssh/config [...] debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<7680<8192) sent debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP On Mon Feb 09 2015 at 1:29:17 PM Darren Tucker <dtucker at zip.com.au> wrote:> I'd add "if you run netstat on both ends and see "SendQ" non-zero and not > decreasing then this is likely your problem. >With the -m parameter as above, running ss on the client, I see Send-Q go to 1208 and then sit there until I Ctrl-C out the client, when it increases by 1. On the server side, I see nothing. Is that plausible, that the client would proceed all the way through to DH_GEX_GROUP without seeing any data? Without the -m parameter Send-Q goes to 1992. 1208 bytes shouldn't need any fragmentation; I can definitely ping unfragmented packets that large. So I'm thinking firewall problem now, but I'm at a loss as to why OpenSSH is triggering the problem but PuTTY isn't, given that reducing packet size below the MTU limit doesn't seem to help. Any ideas? Thanks, mathew