Xen users - Mar 2010 - Spooky networking issue: ping OK on container, stops on VM, restarts with arp -d

I am managing two Xen containers which are exhibiting a spooky behavior.
There are intermittent network failures
that affect only the VM''s, and not dom0.  This exhibits itself as
problems
with nfs locks, snmp, and even icmp ping.

The arp tables look normal on both the dom0 and the guest.  However, it is
definitely arp related.  Causing arp
traffic of any sort results the problem curing itself for a few seconds.

The pings go out, reach the destination, and are replied to.  The missing
replies don''t reach tcpdump on bond0, br0 or eth0 on dom0. Working
replies
show up on bond0 and br0 but oddly enough eth0 only sees the outgoing ping.

Has anyone seen anything like this?
I''d blame the switches, except it works just fine if the ping is
initiated
from dom0.


cacti:/home/bnesbitt# ping 10.103.0.1
...
64 bytes from 10.103.0.1: icmp_seq=13 ttl=64 time=0.961 ms
...
64 bytes from 10.103.0.1: icmp_seq=312 ttl=64 time=0.960 ms
64 bytes from 10.103.0.1: icmp_seq=313 ttl=64 time=0.867 ms
^C
--- 10.103.0.1 ping statistics ---
347 packets transmitted, 136 received, 60% packet loss, time 346891ms
rtt min/avg/max/mdev = 0.716/0.938/5.334/0.533 ms



cacti:/home/bnesbitt# tcpdump host 10.103.0.1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
13:33:14.071645 IP cacti.local > 10.103.0.1: ICMP echo request, id 48646,
seq 266, length 64
13:33:15.073939 IP cacti.local > 10.103.0.1: ICMP echo request, id 48646,
seq 267, length 64
13:33:16.073770 IP cacti.local > 10.103.0.1: ICMP echo request, id 48646,
seq 268, length 64
13:33:17.073848 IP cacti.local > 10.103.0.1: ICMP echo request, id 48646,
seq 269, length 64
13:33:18.077077 IP cacti.local > 10.103.0.1: ICMP echo request, id 48646,
seq 270, length 64
13:33:19.081783 IP cacti.local > 10.103.0.1: ICMP echo request, id 48646,
seq 271, length 64
13:33:20.085792 IP cacti.local > 10.103.0.1: ICMP echo request, id 48646,
seq 272, length 64
13:33:21.089913 IP cacti.local > 10.103.0.1: ICMP echo request, id 48646,
seq 273, length 64
13:33:22.089833 IP cacti.local > 10.103.0.1: ICMP echo request, id 48646,
seq 274, length 64
13:33:23.093808 IP cacti.local > 10.103.0.1: ICMP echo request, id 48646,
seq 275, length 64
13:33:24.101797 arp who-has 10.103.0.1 tell cacti.local
13:33:24.102304 arp reply 10.103.0.1 is-at 00:e0:81:75:d8:1f (oui Unknown)
13:33:24.102311 IP cacti.local > 10.103.0.1: ICMP echo request, id 48646,
seq 276, length 64
13:33:24.103043 IP 10.103.0.1 > cacti.local: ICMP echo reply, id 48646, seq
276, length 64
13:33:25.101795 IP cacti.local > 10.103.0.1: ICMP echo request, id 48646,
seq 277, length 64
13:33:25.102615 IP 10.103.0.1 > cacti.local: ICMP echo reply, id 48646, seq
277, length 64
13:33:26.101831 IP cacti.local > 10.103.0.1: ICMP echo request, id 48646,
seq 278, length 64
13:33:26.102624 IP 10.103.0.1 > cacti.local: ICMP echo reply, id 48646, seq
278, length 64
13:33:27.105810 IP cacti.local > 10.103.0.1: ICMP echo request, id 48646,
seq 279, length 64
13:33:27.106615 IP 10.103.0.1 > cacti.local: ICMP echo reply, id 48646, seq
279, length 64
13:33:28.109806 IP cacti.local > 10.103.0.1: ICMP echo request, id 48646,
seq 280, length 64
13:33:28.110674 IP 10.103.0.1 > cacti.local: ICMP echo reply, id 48646, seq
280, length 64
13:33:29.097217 arp who-has cacti.local tell 10.103.0.1
13:33:29.097230 arp reply cacti.local is-at 00:16:3e:46:ca:2e (oui Unknown)
13:33:29.113758 IP cacti.local > 10.103.0.1: ICMP echo request, id 48646,
seq 281, length 64
13:33:29.114606 IP 10.103.0.1 > cacti.local: ICMP echo reply, id 48646, seq
281, length 64
13:33:30.118016 IP cacti.local > 10.103.0.1: ICMP echo request, id 48646,
seq 282, length 64
13:33:30.118775 IP 10.103.0.1 > cacti.local: ICMP echo reply, id 48646, seq
282, length 64
13:33:31.121856 IP cacti.local > 10.103.0.1: ICMP echo request, id 48646,
seq 283, length 64
13:33:31.122750 IP 10.103.0.1 > cacti.local: ICMP echo reply, id 48646, seq
283, length 64
13:33:32.125809 IP cacti.local > 10.103.0.1: ICMP echo request, id 48646,
seq 284, length 64
13:33:32.126708 IP 10.103.0.1 > cacti.local: ICMP echo reply, id 48646, seq
284, length 64
^C



Xen 3.2-1
Linux x1.xen.bepress.com 2.6.26-2-xen-amd64 #1 SMP Thu Aug 20 00:36:34 UTC
2009 x86_64 GNU/Linux
Xen 3.2-1
Linux x2.xen.bepress.com 2.6.26-2-xen-amd64 #1 SMP Thu Aug 20 00:36:34 UTC
2009 x86_64 GNU/Linux

-- 
Bryce Nesbitt
The Berkeley Electronic Press
bepress: 10 years of accelerating and enhancing the flow of scholarly ideas


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users