Anthony Bourguignon
2012-Apr-13 12:33 UTC
[libvirt-users] Guests can't connect to each other
Hi,
I'm using libvirt and qemu on Debian Wheezy. I'm having a strange
behavior. Guests can't connect to each other when they're on the same
host.
On the host I'm using bonding (in active / backup mode) and vlan. It
looks like this :
eth0 \ / macvtap0
bond0 --- vlan222
eth1 / \ macvtap1
So I've got two guests, let's say A and B. When I try to ping B from A,
it works :
# ping -s 3000 -c 5 78.109.95.11
PING 78.109.95.11 (78.109.95.11) 3000(3028) bytes of data.
3008 bytes from 78.109.95.11: icmp_req=1 ttl=64 time=0.065 ms
3008 bytes from 78.109.95.11: icmp_req=2 ttl=64 time=2.19 ms
3008 bytes from 78.109.95.11: icmp_req=3 ttl=64 time=1.43 ms
--- 78.109.95.11 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4001ms
rtt min/avg/max/mdev = 0.065/0.960/2.197/0.760 ms
But nothing happens when I try to ssh it (not even a timeout). You'll
find enclosed the tcpdump captures on the source and the destination.
It's the same when I use netcat in udp.
At the same time, connection from the host to one guest is working
perfectly.
There is no iptables rule on the host, and nothing too on the guests.
Here are the virsh dumpxml of the different components :
# virsh dumpxml vm1
<domain type='kvm' id='11'>
<name>vm1</name>
<uuid>4eaaed00-c610-b468-ad55-600a0b4e244c</uuid>
<memory>2048000</memory>
<currentMemory>2048000</currentMemory>
<memoryBacking>
<hugepages/>
</memoryBacking>
<vcpu cpuset='0,2,4,8,10,12,14'>8</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='0,8'/>
<vcpupin vcpu='1' cpuset='2,10'/>
<vcpupin vcpu='2' cpuset='4,12'/>
<vcpupin vcpu='3' cpuset='6,14'/>
<vcpupin vcpu='4' cpuset='0,8'/>
<vcpupin vcpu='5' cpuset='2,10'/>
<vcpupin vcpu='6' cpuset='4,12'/>
<vcpupin vcpu='7' cpuset='6,14'/>
</cputune>
<os>
<type arch='x86_64' machine='pc-1.0'>hvm</type>
<boot dev='hd'/>
<boot dev='network'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu match='exact'>
<model>Westmere</model>
<vendor>Intel</vendor>
<topology sockets='1' cores='8' threads='1'/>
<feature policy='require' name='tm2'/>
<feature policy='require' name='est'/>
<feature policy='require' name='vmx'/>
<feature policy='require' name='ds'/>
<feature policy='require' name='smx'/>
<feature policy='require' name='ss'/>
<feature policy='require' name='vme'/>
<feature policy='require' name='dtes64'/>
<feature policy='require' name='rdtscp'/>
<feature policy='require' name='ht'/>
<feature policy='require' name='dca'/>
<feature policy='require' name='pbe'/>
<feature policy='require' name='tm'/>
<feature policy='require' name='pdcm'/>
<feature policy='require' name='pdpe1gb'/>
<feature policy='require' name='ds_cpl'/>
<feature policy='require' name='pclmuldq'/>
<feature policy='require' name='xtpr'/>
<feature policy='require' name='acpi'/>
<feature policy='require' name='monitor'/>
</cpu>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/bin/kvm</emulator>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none'
io='native'/>
<source dev='/dev/vps/vm1'/>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x04'
function='0x0'/>
</disk>
<interface type='network'>
<mac address='52:54:00:0e:58:ae'/>
<source network='vlan222'/>
<target dev='macvtap0'/>
<model type='virtio'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x03'
function='0x0'/>
</interface>
<serial type='pty'>
<source path='/dev/pts/0'/>
<target port='0'/>
<alias name='serial0'/>
</serial>
<console type='pty' tty='/dev/pts/0'>
<source path='/dev/pts/0'/>
<target type='serial' port='0'/>
<alias name='serial0'/>
</console>
<input type='mouse' bus='ps2'/>
<graphics type='vnc' port='5900' autoport='yes'
listen='0.0.0.0'
keymap='fr'>
<listen type='address' address='0.0.0.0'/>
</graphics>
<video>
<model type='vga' vram='9216' heads='1'/>
<alias name='video0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x02'
function='0x0'/>
</video>
<memballoon model='virtio'>
<alias name='balloon0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x05'
function='0x0'/>
</memballoon>
</devices>
</domain>
# virsh dumpxml vm2
<domain type='kvm' id='13'>
<name>vm2</name>
<uuid>4f760831-22b1-ff3b-26e7-6b3fec49e918</uuid>
<memory>2048000</memory>
<currentMemory>2048000</currentMemory>
<memoryBacking>
<hugepages/>
</memoryBacking>
<vcpu cpuset='1,3,5,7,9,11,13,15'>8</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='1,3,5,7,9,11,13,15'/>
<vcpupin vcpu='1' cpuset='1,3,5,7,9,11,13,15'/>
<vcpupin vcpu='2' cpuset='1,3,5,7,9,11,13,15'/>
<vcpupin vcpu='3' cpuset='1,3,5,7,9,11,13,15'/>
<vcpupin vcpu='4' cpuset='1,3,5,7,9,11,13,15'/>
<vcpupin vcpu='5' cpuset='1,3,5,7,9,11,13,15'/>
<vcpupin vcpu='6' cpuset='1,3,5,7,9,11,13,15'/>
<vcpupin vcpu='7' cpuset='1,3,5,7,9,11,13,15'/>
</cputune>
<os>
<type arch='x86_64' machine='pc-1.0'>hvm</type>
<boot dev='hd'/>
<boot dev='network'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu match='exact'>
<model>Westmere</model>
<vendor>Intel</vendor>
<topology sockets='1' cores='4' threads='2'/>
<feature policy='require' name='tm2'/>
<feature policy='require' name='est'/>
<feature policy='require' name='vmx'/>
<feature policy='require' name='ds'/>
<feature policy='require' name='smx'/>
<feature policy='require' name='ss'/>
<feature policy='require' name='vme'/>
<feature policy='require' name='dtes64'/>
<feature policy='require' name='rdtscp'/>
<feature policy='require' name='ht'/>
<feature policy='require' name='dca'/>
<feature policy='require' name='pbe'/>
<feature policy='require' name='tm'/>
<feature policy='require' name='pdcm'/>
<feature policy='require' name='pdpe1gb'/>
<feature policy='require' name='ds_cpl'/>
<feature policy='require' name='pclmuldq'/>
<feature policy='require' name='xtpr'/>
<feature policy='require' name='acpi'/>
<feature policy='require' name='monitor'/>
</cpu>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/bin/kvm</emulator>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none'
io='native'/>
<source dev='/dev/vps/vm2'/>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x04'
function='0x0'/>
</disk>
<interface type='network'>
<mac address='52:54:00:cb:ce:41'/>
<source network='vlan222'/>
<target dev='macvtap1'/>
<model type='virtio'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x03'
function='0x0'/>
</interface>
<serial type='pty'>
<source path='/dev/pts/1'/>
<target port='0'/>
<alias name='serial0'/>
</serial>
<console type='pty' tty='/dev/pts/1'>
<source path='/dev/pts/1'/>
<target type='serial' port='0'/>
<alias name='serial0'/>
</console>
<input type='mouse' bus='ps2'/>
<graphics type='vnc' port='5901' autoport='yes'
listen='0.0.0.0'
keymap='fr'>
<listen type='address' address='0.0.0.0'/>
</graphics>
<video>
<model type='vga' vram='9216' heads='1'/>
<alias name='video0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x02'
function='0x0'/>
</video>
<memballoon model='virtio'>
<alias name='balloon0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x05'
function='0x0'/>
</memballoon>
</devices>
</domain>
# virsh net-dumpxml vlan222
<network>
<name>vlan222</name>
<uuid>2b763b5c-4ec1-9b5f-b29d-b7a7ea0f743d</uuid>
<forward dev='vlan222' mode='bridge'>
<interface dev='vlan222'/>
</forward>
</network>
Thanks in advance to help me understand this issue.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: destination.cap
Type: application/octet-stream
Size: 1393 bytes
Desc: not available
URL:
<http://listman.redhat.com/archives/libvirt-users/attachments/20120413/19c5df03/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: source.cap
Type: application/octet-stream
Size: 1393 bytes
Desc: not available
URL:
<http://listman.redhat.com/archives/libvirt-users/attachments/20120413/19c5df03/attachment-0001.obj>
Richard W.M. Jones
2012-Apr-13 19:17 UTC
[libvirt-users] Guests can't connect to each other
On Fri, Apr 13, 2012 at 02:33:24PM +0200, Anthony Bourguignon wrote:> Hi, > > I'm using libvirt and qemu on Debian Wheezy. I'm having a strange > behavior. Guests can't connect to each other when they're on the same > host. > > On the host I'm using bonding (in active / backup mode) and vlan. It > looks like this : > eth0 \ / macvtap0 > bond0 --- vlan222 > eth1 / \ macvtap1I thought it was a known problem that macvtap guests can't talk to each other ...? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming blog: http://rwmj.wordpress.com Fedora now supports 80 OCaml packages (the OPEN alternative to F#) http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora
On 04/13/2012 08:33 AM, Anthony Bourguignon wrote:> Hi, > > I'm using libvirt and qemu on Debian Wheezy. I'm having a strange > behavior. Guests can't connect to each other when they're on the same > host. > > On the host I'm using bonding (in active / backup mode) and vlan. It > looks like this : > eth0 \ / macvtap0 > bond0 --- vlan222 > eth1 / \ macvtap1 > > So I've got two guests, let's say A and B. When I try to ping B from A, > it works : > # ping -s 3000 -c 5 78.109.95.11 > PING 78.109.95.11 (78.109.95.11) 3000(3028) bytes of data. > 3008 bytes from 78.109.95.11: icmp_req=1 ttl=64 time=0.065 ms > 3008 bytes from 78.109.95.11: icmp_req=2 ttl=64 time=2.19 ms > 3008 bytes from 78.109.95.11: icmp_req=3 ttl=64 time=1.43 ms > > --- 78.109.95.11 ping statistics --- > 5 packets transmitted, 5 received, 0% packet loss, time 4001ms > rtt min/avg/max/mdev = 0.065/0.960/2.197/0.760 ms > > But nothing happens when I try to ssh it (not even a timeout). You'll > find enclosed the tcpdump captures on the source and the destination. > It's the same when I use netcat in udp.Your config looks fine (the important part is that you're using bridge mode for macvatap rather than private). I would suspect some sort of bug related to using macvtap on a vlan device (or, even more, a vlan connected to a bond). Try changing your network config to use 1) a vlan connected directly to eth0 or eth1, rather than the bond, 2) bond0 directly, and 3) eth0 or eth1 directly. This will hopefully give you an idea of which part of the equation isn't working.
I just noticed the packet dumps in your original message. They show that
the TCP checksum of all packets past #3 are incorrect.
This is a similar problem with one caused by using the virtio "vhost"
mode with standard tap network devices - if the dhcp server and client
are on the same physical machine, the sender of a packet doesn't fill
in the TCP checksum (assuming that tcp checksum offloading will fix it),
but the tcp checksum offload happens at a level low enough it is never
reached by packets just going to another guest on the same host, so the
checksum is never filled in, and the receiver thus rejects the packet.
(If you want to read a long boring history of this problem, which is
only tangentially related to yours, look here:
https://bugzilla.redhat.com/show_bug.cgi?id=612588 )
It's possible there is a similar bug with certain versions of macvtap
support in the kernel; you'd have to look that up. In the meantime, I
think you should be able to solve your problem by modifying your guests'
interface definitions like this:
<interface type='network'>
<mac address='52:54:00:0e:58:ae'/>
<source network='vlan222'/>
<target dev='macvtap0'/>
<model type='virtio'/>
<driver name='qemu'/> <!-- ADD THIS LINE -->
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x03' function='0x0'/>
</interface>
This forces libvirt to call qemu with vhost turned off.