openssh unix dev - Jan 2018 - issue with openssh-server running in a libvirt based centos virtual machine

Hi,

I have a very strange ssh issue, and I do not know how to solve it.

I have a centos VM (managed by libvirt) running the latest centos 7
and latest openssh server package (debug1: sshd version OpenSSH_7.4,
OpenSSL 1.0.2k-fips  26 Jan 2017).

When I ssh remotely to this centos VM (using latest putty 0.70 and
other ssh clients as well), I do not get a login prompt whatsoever. I
used openssh-server in debug mode, to see where the ssh connection
hangs, and here is what I get:

[root at centos1 ~]# /sbin/sshd -p 22 -D -d  -e
debug1: sshd version OpenSSH_7.4, OpenSSL 1.0.2k-fips  26 Jan 2017
debug1: private host key #0: ssh-rsa
SHA256:pEuFQsodwK+0PoRzbVRba1ahHLEpwp8DG2KGQmxOGJk
debug1: private host key #1: ecdsa-sha2-nistp256
SHA256:F6HrSNWZhYaU7LMweI+RBviqTCHcTYyMBGPDz5OjT4c
debug1: private host key #2: ssh-ed25519
SHA256:aG3V6jjPHXUnNeavbxT/xozqrb5q3yWDkkAmXBCdnGk
debug1: rexec_argv[0]='/sbin/sshd'
debug1: rexec_argv[1]='-p'
debug1: rexec_argv[2]='22'
debug1: rexec_argv[3]='-D'
debug1: rexec_argv[4]='-d'
debug1: rexec_argv[5]='-e'
debug1: Set /proc/self/oom_score_adj from 0 to -1000
debug1: Bind to port 22 on 0.0.0.0.
Server listening on 0.0.0.0 port 22.
debug1: Bind to port 22 on ::.
Server listening on :: port 22.

Here the debug log stops and the openssh server waits for an inbound
ssh client connection. At this stage I am using Putty v0.70 with all
default settings to connect to this openssh sever, and below is the
debug log after Putty starts the ssh connection:

debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: sshd version OpenSSH_7.4, OpenSSL 1.0.2k-fips  26 Jan 2017
debug1: private host key #0: ssh-rsa
SHA256:pEuFQsodwK+0PoRzbVRba1ahHLEpwp8DG2KGQmxOGJk
debug1: private host key #1: ecdsa-sha2-nistp256
SHA256:F6HrSNWZhYaU7LMweI+RBviqTCHcTYyMBGPDz5OjT4c
debug1: private host key #2: ssh-ed25519
SHA256:aG3V6jjPHXUnNeavbxT/xozqrb5q3yWDkkAmXBCdnGk
debug1: inetd sockets after dupping: 3, 3
Connection from x.x.x.181 port 49436 on x.x.x.115 port 22
debug1: Client protocol version 2.0; client software version PuTTY_Release_0.70
debug1: no match: PuTTY_Release_0.70
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Enabling compatibility mode for protocol 2.0
debug1: SELinux support enabled [preauth]
debug1: permanently_set_uid: 74/74 [preauth]
debug1: list_hostkey_types:
ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
[preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]

Here the debug mode stops, and there is no login prompt shown on the
Putty window. Actually nothing happens on the Putty window or in the
openssh-server side, even if I wait for couple of minutes. Finally
when I give up and I close the Putty window, this is what is shown in
the openssh server debug log:

Connection closed by x.x.x.181 port 49436 [preauth]
debug1: do_cleanup [preauth]
debug1: monitor_read_log: child log fd closed
debug1: do_cleanup
debug1: Killing privsep child 1274

I used the log in Putty side to see whats happening, and this is all what I get:

=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2018.01.22 18:51:38
=~=~=~=~=~=~=~=~=~=~=~Event Log: Writing new session log (SSH packets mode) to
file:
C:\Users\adrian\Desktop\putty.log
Event Log: Connecting to x.x.x.115 port 22
Event Log: We claim version: SSH-2.0-PuTTY_Release_0.70

I have tried with other ssh clients as well, like MobaXterm
(OpenSSH_7.1p2, OpenSSL 1.0.1g 7 Apr 2014), and the debug log in this
ssh client side is the following:

[2018-01-25 19:41.04]  ~ # ssh -vvv x.x.x.115
OpenSSH_7.1p2, OpenSSL 1.0.1g 7 Apr 2014
debug1: Reading configuration data /etc/ssh_config
debug2: ssh_connect: needpriv 0
debug1: Connecting to x.x.x.115 [x.x.x.115] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mobaxterm/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.1
debug1: ssh_exchange_identification:
debug1: ssh_exchange_identification:
debug1: ssh_exchange_identification:
\024\324C9\0047\311'\316h\340.\365\213J\243d
debug1: ssh_exchange_identification:
1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug1: ssh_exchange_identification:
@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,blowfish-cbc,cast128-cbc,3des-cbc
debug1: ssh_exchange_identification: cbc
debug1: ssh_exchange_identification:
8-etm at openssh.com,hmac-sha2-256-etm at openssh.com,hmac-sha2-512-etm at
openssh.com,hmac-sha1-etm at openssh.com,umac-64 at openssh.com,umac-128 at
openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1

Here the debug log stops, and I do not get an login prompt whatsoever.

This is the first time I see this kind of issue, and I do not have any
idea how to troubleshoot it further, therefore any suggestion would
really be appreciated.

Many thanks,
Adrian

On 26 January 2018 at 06:44, Adrian Pascalau <adrian27oradea at gmail.com>
wrote:
> [...]
> debug1: SSH2_MSG_KEXINIT sent [preauth]
>
> Here the debug mode stops, and there is no login prompt shown on the
> Putty window.

This behaviour is often caused by a path MTU/fragmentation problem.

If you run "netstat" on both client and server and find the SSH TCP
connection you should see a "Send-Q" column.  This column is the
number of
bytes in the TCP socket buffer that the other end has not acknowledged.  If
it is non-zero number that remains above zero or increases then IP
fragmentation is likely your problem and you need to fix whatever in your
environment is causing that.

-- 
Darren Tucker (dtucker at dtucker.net)
GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860  37F4 9357 ECEF 11EA A6FA (new)
    Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.

On Fri, Jan 26, 2018 at 5:02 AM, Darren Tucker <dtucker at dtucker.net>
wrote:
> On 26 January 2018 at 06:44, Adrian Pascalau <adrian27oradea at
gmail.com>
> wrote:
>>
>> [...]
>> debug1: SSH2_MSG_KEXINIT sent [preauth]
>>
>> Here the debug mode stops, and there is no login prompt shown on the
>> Putty window.
>
>
> This behaviour is often caused by a path MTU/fragmentation problem.
>
> If you run "netstat" on both client and server and find the SSH
TCP
> connection you should see a "Send-Q" column.  This column is the
number of
> bytes in the TCP socket buffer that the other end has not acknowledged.  If
> it is non-zero number that remains above zero or increases then IP
> fragmentation is likely your problem and you need to fix whatever in your
> environment is causing that.
Darren, thanks for your reply.

In the ssh client side, I have a windows host that runs Putty. So I
installed a new centos to be able to see the "Send-Q" column, since
the windows netstat does not have it. Surprise, the ssh connection
works like a charm. I went back to the windows box to try again with
Putty, and I have the same issue. So could this be because of windows?
I cannot suspect Putty, since I tried this with another windows based
ssh client (MobaXterm), and the same issue happens.