Jan Mulders
2007-May-03 20:43 UTC
[Xen-users] Xen''s xenbr0 interface eating packets between domUs?
Hello all.
Having a little issue with a Xen setup I''m running. my domU''s
don''t seem to
be able to contact one another...
Can anyone shed some light on where my packets are going? They vanish
between the domU sending the response and the xenbr0 on the dom0.
Here''s some details - IP addresses obfuscated with 11.22.33 for safety.
forest = dom0
squirrel = gameserver domU
tree = openvpn domU
rock = webserver domU (not mentioned here except for ''it has the same
problem'')
here''s some dom0 ip configuration:
[root@forest ~]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:02:B3:E9:A7:EB
inet addr:11.22.33.161 Bcast: 11.22.33.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:172153 errors:0 dropped:0 overruns:0 frame:0
TX packets:28522 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:15218006 (14.5 MiB) TX bytes:11912815 (11.3 MiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:139 errors:0 dropped:0 overruns:0 frame:0
TX packets:139 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:512496 (500.4 KiB) TX bytes:512496 (500.4 KiB)
peth0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
UP BROADCAST RUNNING NOARP MULTICAST MTU:1500 Metric:1
RX packets:17906458 errors:0 dropped:0 overruns:0 frame:0
TX packets:17406759 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3001638846 (2.7 GiB) TX bytes:629520683 (600.3 MiB)
vif0.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:28522 errors:0 dropped:0 overruns:0 frame:0
TX packets:172154 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:11912815 (11.3 MiB) TX bytes:15218084 (14.5 MiB)
vif9.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:12558483 errors:0 dropped:0 overruns:0 frame:0
TX packets:13041996 errors:0 dropped:188 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:65048010 (62.0 MiB) TX bytes:138801861 (132.3 MiB)
vif15.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1357 errors:0 dropped:0 overruns:0 frame:0
TX packets:110982 errors:0 dropped:47 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:752723 (735.0 KiB) TX bytes:10116656 (9.6 MiB)
vif17.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3762543 errors:0 dropped:0 overruns:0 frame:0
TX packets:3559047 errors:0 dropped:2167 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:477432410 (455.3 MiB) TX bytes:1187204888 (1.1 GiB)
xenbr0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:392477 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:144771106 (138.0 MiB) TX bytes:0 (0.0 b)
I''m trying to send udp packets to the gameserver (squirrel) from a
machine
behind the VPN machine (tree). The forward route (me to gameserver) seems to
be fine - but the return trip? Not so good.
11.22.33.163 is me, 11.22.33.138 is the gameserver (squirrel).
Here''s what I get from tcpdump''s on each machine:
First of all, here''s the gameserver.
[root@squirrel ~]# tcpdump -i eth0 src 11.22.33.163 or dst 11.22.33.163 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
16:23:24.275874 IP 11.22.33.163.3724 > 11.22.33.138.27015: UDP, length 25
16:23:24.276129 IP 11.22.33.138.27015 > 11.22.33.163.3724 : UDP, length 109
16:23:31.825710 IP 11.22.33.163.3732 > 11.22.33.138.27015: UDP, length 25
16:23:31.835973 IP 11.22.33.138.27015 > 11.22.33.163.3732: UDP, length 109
16:23:31.839778 IP 11.22.33.163.3733 > 11.22.33.138.27015: UDP, length 25
16:23:31.850976 IP 11.22.33.138.27015 > 11.22.33.163.3733: UDP, length 109
16:23:36.835129 arp who-has 11.22.33.163 tell 11.22.33.138
16:23:36.835800 arp reply 11.22.33.163 is-at 00:16:3e:1d:28:ee
8 packets captured
8 packets received by filter
0 packets dropped by kernel
[root@squirrel ~]#
As you can see, the server responded correctly to all three requests.
Here is the Xen dom0 (forest), sniffing at the bridge:
[root@forest ~]# tcpdump -i xenbr0 src 11.22.33.163 or dst 195.178.107.163-n
tcpdump: WARNING: xenbr0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on xenbr0, link-type EN10MB (Ethernet), capture size 96 bytes
21:23:24.449795 IP 11.22.33.163.3724 > 11.22.33.138.27015: UDP, length 25
21:23:32.001248 IP 11.22.33.163.3732 > 11.22.33.138.27015: UDP, length 25
21:23:32.015363 IP 11.22.33.163.3733 > 11.22.33.138.27015: UDP, length 25
21:23:37.012320 arp reply 11.22.33.163 is-at 00:16:3e:1d:28:ee
28 packets captured
35 packets received by filter
0 packets dropped by kernel
[root@forest ~]#
Now this is interesting. The replies aren''t making it onto the xenbr0
interface! the who-has arp request got gobbled by the filter but it''ll
be
there.
Here is the VPN domU (tree), sniffing at its eth0.
[root@tree ~]# tcpdump -i eth0 src 11.22.33.138 or src 195.178.107.163 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
21:23:24.403375 IP 11.22.33.138.27015 > 11.22.33.163.3724: UDP, length 109
21:23:31.964292 IP 11.22.33.138.27015 > 11.22.33.163.3732 : UDP, length 109
21:23:31.979393 IP 11.22.33.138.27015 > 11.22.33.163.3733: UDP, length 109
21:23:36.943329 arp reply 11.22.33.138 is-at 00:16:3e:39:84:0f
21:23:36.964097 arp who-has 11.22.33.163 tell 11.22.33.138
5 packets captured
27 packets received by filter
0 packets dropped by kernel
[root@tree ~]#
And predictably, the replies don''t make it to the VPN domU (tree).
The only rules in the dom0 which could possibly stop packets systematically
are:
[root@forest ~]# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
ACCEPT all -- anywhere anywhere PHYSDEV match
--physdev-in vif9.0
ACCEPT all -- anywhere anywhere PHYSDEV match
--physdev-in vif15.0
ACCEPT all -- anywhere anywhere PHYSDEV match
--physdev-in vif17.0
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
[root@forest ~]#
People out on the internet can connect to my server no problem, and I can
access the internet fine, too... but accessing this machine within the Xen
bridge doesn''t seem to work.
Some Xen configs:
Tree:
kernel = "/boot/vmlinuz-2.6.16-xen0-100hz"
memory = 64
name = "tree"
vif = [ '''' ]
ip = "11.22.33.139"
gateway = "11.22.33.1"
netmask = "255.255.255.0"
disk =
[''phy:/dev/vg01/vol01,sda1,w'',''phy:/dev/vg01/vol02,sda2,w'']
root = "/dev/sda1 ro"
kernel = "/boot/vmlinuz-2.6.16-xen0-1000hz"
memory = 128
name = "squirrel"
vif = [ '''' ]
ip = "11.22.33.138"
gateway = "11.22.33.1"
netmask = "255.255.255.0"
disk =
[''phy:/dev/vg01/vol03,sda1,w'',''phy:/dev/vg01/vol04,sda2,w'']
root = "/dev/sda1 ro"
I also have another domU called ''rock'', mentioned earlier,
that doesn''t
respond directly from inside the bridge; yet responds fine when the request
comes from outside the Xen bridge (ie, from the internet).
Can anyone shed some light on where my packets are going?
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users