libvirt users - Feb 2016 - Domain cannot talk to itself unless virbr0 is in promiscuous mode

Hi

We have the following situation:

A hypervisor using KVM-Qemu/libvirt with a single VM. The VM has the
internal IP 192.168.122.151 on the default network pool using virbr0 on
the hypervisor.

We've set up netfilter rules to DNAT requests on one IP of the
hypervisor to the VM. So, first of all, forwarding to the internal IP
range is allowed:
> $IPTABLES -I FORWARD -m state -d 192.168.122.0/24 --state
> NEW,RELATED,ESTABLISHED -j ACCEPT

Then we're forwarding a few ports from the hypervisor's public IP to the
VM, imaps being one of them:
> Chain PREROUTING (policy ACCEPT 2606 packets, 169K bytes) pkts bytes
> target prot opt in out source destination 142 8048 DNAT tcp -- * *
> 0.0.0.0/0 a.b.c.145 tcp dpt:993 to:192.168.122.151

Since the VM is also doing SMTP, we need it to send out mail via a
dedicated IPv4, which we achieve through the following SNAT rule:
> Chain POSTROUTING (policy ACCEPT 1121 packets, 77556 bytes) pkts
> bytes target prot opt in out source destination 81 6262 SNAT all -- *
> * 192.168.122.151 0.0.0.0/0 to:a.b.c.145

Now, on this mailserver, we're also running a roundcube instance. For
authentication, roundcube connects to the imaps port of the local
dovecot instance. In order for the SSL certificate check to succeed, it
needs to do so using the actual domain name, and cannot use
"localhost"
or the loopback IP address.

The problem is that, given the above iptables rules, connecting from
inside the VM to itself via the hypervisor's external interface doesn't
work. The connection simply times out, logging reveals they're not
getting forwarded.

We've added logging to iptables and testing in promisc mode vs. not-promisc:

This is what gets logged when virbr0 is not in promisc mode:
> Feb 11 19:12:31 hyper kernel: [199815.004207] prerouting imaps:
> IN=virbr0 OUT= PHYSIN=vnet0
> MAC=fe:54:00:17:c8:2a:52:54:00:17:c8:2a:08:00 SRC=192.168.122.151
> DST=a.b.c.d.145 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=40689 DF
> PROTO=TCP SPT=43460 DPT=993 WINDOW=29200 RES=0x00 SYN URGP=0

And this when the device is in promisc mode:
> Feb 11 19:14:33 hyper kernel: [199937.674189] prerouting imaps:
> IN=virbr0 OUT= PHYSIN=vnet0
> MAC=fe:54:00:17:c8:2a:52:54:00:17:c8:2a:08:00 SRC=192.168.122.151
> DST=a.b.c.d.145 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=2244 DF PROTO=TCP
> SPT=43461 DPT=993 WINDOW=29200 RES=0x00 SYN URGP=0
> Feb 11 19:14:33 hyper kernel: [199937.674221] forwarded imaps: IN=virbr0
> OUT=virbr0 PHYSIN=vnet0 MAC=52:54:00:17:c8:2a:52:54:00:17:c8:2a:08:00
> SRC=192.168.122.151 DST=192.168.122.151 LEN=60 TOS=0x10 PREC=0x00 TTL=63
> ID=2244 DF PROTO=TCP SPT=43461 DPT=993 WINDOW=29200 RES=0x00 SYN URGP=0
> Feb 11 19:14:33 hyper kernel: [199937.674766] forwarded imaps: IN=virbr0
> OUT=virbr0 PHYSIN=vnet0 MAC=52:54:00:17:c8:2a:52:54:00:17:c8:2a:08:00
> SRC=192.168.122.151 DST=192.168.122.151 LEN=52 TOS=0x10 PREC=0x00 TTL=63
> ID=2245 DF PROTO=TCP SPT=43461 DPT=993 WINDOW=229 RES=0x00 ACK URGP=0

Another workaround we found is to disable iptables on the bridge
interface.
> root@hyper:~# echo 0 > /proc/sys/net/bridge/bridge-nf-call-iptables

But I do not consider this a solution, merely a workaround. I
do not understand why the forward rule mentioned above does not apply to
the situation where IN and OUT are both virbr0?


Thank you for any advice,
David

-- 
TenTwentyFour S.à r.l.
W: www.tentwentyfour.lu
T: +352 20 211 1024
F: +352 20 211 1023
9, avenue des Hauts-Fourneaux
L-4362 Esch/Alzette