libvirt users - Jan 2018 - issue with openssh-server running in a libvirt based centos virtual machine

Hi,

I have a strange issue in a libvirt environment, and I do not know how
to solve it.

I have two centos hosts: first one is a physical server called
server1, that acts as a host for the second one, called centos1. The
centos1 is a virtual machine (VM) running in server1. A linux bridge
in forwarding mode is used to connect the centos1 VM network interface
to the server1 network interface and to the external network. The
centos1 VM and the linux bridge are managed with libvirt (well, the
bridge itself in this case is created manually).

# virsh net-dumpxml br0
<network connections='1'>
  <name>br0</name>
  <uuid>5aaf72a5-023d-4b84-9d7c-d68b0918f620</uuid>
  <forward mode='bridge'/>
  <bridge name='br0'/>
</network>

# brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.fc15b4137688       no              eno1
                                                        vnet0

Both server1 and centos1 have IP addresses in the same subnet, and
both are reachable with ping from every other host in my network. In
both server1 and centos1, the openssh-server configuration in
/etc/ssh/sshd_config is the default one, and has not been changed.

When I ssh with Putty to the physical server server1 IP address,
everything works as expected: I get a login prompt, I enter my
password and I log in.

However, when I use Putty to connect to the centos1 VM, I do not get a
login prompt whatsoever. So I think there might be some issue in
between the server1 physical interface and my centos VM.

I used openssh-server in debug mode, so see where the ssh connection
hangs, and here is what I get:

[...]
debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: sshd version OpenSSH_7.4, OpenSSL 1.0.2k-fips  26 Jan 2017
debug1: private host key #0: ssh-rsa
SHA256:pEuFQsodwK+0PoRzbVRba1ahHLEpwp8DG2KGQmxOGJk
debug1: private host key #1: ecdsa-sha2-nistp256
SHA256:F6HrSNWZhYaU7LMweI+RBviqTCHcTYyMBGPDz5OjT4c
debug1: private host key #2: ssh-ed25519
SHA256:aG3V6jjPHXUnNeavbxT/xozqrb5q3yWDkkAmXBCdnGk
debug1: inetd sockets after dupping: 3, 3
Connection from x.x.x.181 port 49436 on x.x.x.115 port 22
debug1: Client protocol version 2.0; client software version PuTTY_Release_0.70
debug1: no match: PuTTY_Release_0.70
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Enabling compatibility mode for protocol 2.0
debug1: SELinux support enabled [preauth]
debug1: permanently_set_uid: 74/74 [preauth]
debug1: list_hostkey_types:
ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
[preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]

I tried with other windows based ssh clients (MobaXterm) and the same
issue happens. I discussed this with people in the openssh mailing
list, and they said this issue could most probably be caused by a path
MTU/fragmentation problem...

Then I moved my centos1 qcow2 image in another physical server called
server2, with exactly the same hw specs and network connections, where
I have installed an all-in-one OpenStack Pike. The network would be
managed with neutron in this case, however I have configured neutron
exactly so that the centos1 VM interface connects through a linux
bridge (managed by neutron) to the server1 physical network interface,
like in the libvirt case.

# brctl show
bridge name     bridge id               STP enabled     interfaces
brqa13eec69-a4          8000.0e7faabad6d4       no              eno1
                                                        tap8cb53db0-fb
                                                        tapb24a1cc5-20

Above the tapb24a1cc5-20 is the tap interface towards my centos1 VM.

In this case, the Putty issue is gone, and I do not have any issue
anymore. If I go back to the libvirt environment in server1, I get the
same issue again.

So I tend to think that my ssh connection issue is caused by the
libvirt and the way networking is configured, however I do not know
how to troubleshoot this further anymore.

Any help is greatly appreciated.

Adrian