I am experiencing strange networking problems between my hypervisors. If I start a ping on hypervisor FOO to hypervisor BAR, the ping will fail, saying that the destination host cannot be reached. However, if I start a ping on FOO to BAR, and, at the same time, start a ping from BAR to FOO both commands will succeed. This is affecting my ability to live migrate my VMs, as one hypervisor will not see the other one unless they both ping one another as described above. Has anyone else had this type of problem? BTW, I can ping any other box on my network from either box, and the ping will succeed. This only happens when I try to ping between hypervisors. And, ping isn''t the only thing that fails. I can SSH into my hypervisors just fine, but when I try to SSH from one to the other, the same thing happens. One cannot find the other --until I ping them from one another. Thank you, -- --------------------------------------------------------------------- Eric Ross Linux Systems Administrator Argonne National Laboratory Computing and Information Systems Phone: 630.252.2600 --------------------------------------------------------------------- _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
2008/8/19 Eric Ross <eross@anl.gov>:> If I start a ping on hypervisor FOO to hypervisor BAR, the ping will > fail, saying that the destination host cannot be reached. However, if I > start a ping on FOO to BAR, and, at the same time, start a ping from BAR > to FOO both commands will succeed.By hypervisor I presume you actually mean domU? Sounds like a MAC address or bridging issue, did you copy BAR from FOO''s xen config file? If so did you change the MAC address? What does a "brctl show" on the dom0 look like? Or an "ifconfig -a" on FOO and BAR and the dom0? _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> By hypervisor I presume you actually mean domU?No, Dom0, sorry about the confusion. I am not experiencing this problem with the VMs. --------------------------------------------------------------------- Eric Ross Linux Systems Administrator Argonne National Laboratory Computing and Information Systems Phone: 630.252.2600 --------------------------------------------------------------------- Andy Burns wrote:> 2008/8/19 Eric Ross <eross@anl.gov>: > >> If I start a ping on hypervisor FOO to hypervisor BAR, the ping will >> fail, saying that the destination host cannot be reached. However, if I >> start a ping on FOO to BAR, and, at the same time, start a ping from BAR >> to FOO both commands will succeed. > > By hypervisor I presume you actually mean domU? > > Sounds like a MAC address or bridging issue, did you copy BAR from > FOO''s xen config file? If so did you change the MAC address? What > does a "brctl show" on the dom0 look like? Or an "ifconfig -a" on FOO > and BAR and the dom0? > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Dom0 brctl output
bridge name bridge id STP enabled interfaces
xenbr0 8000.feffffffffff no vif0.0
peth0
xenbr1 8000.feffffffffff no vif0.1
peth1
NOTE: this is the same for all my Dom0s
ifconfig -a on my Dom0s I had to mask out my IPs so that''s why there
are
XXs there
eth0 Link encap:Ethernet HWaddr 00:14:4F:20:BF:30
inet addr:XXX.XXX.XXX.XXX Bcast:XXX.XXX.XXX.XXX
Mask:255.255.255.0
inet6 addr: fe80::214:4fff:fe20:bf30/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:23612691 errors:0 dropped:0 overruns:0 frame:0
TX packets:10330154 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:5289657592 (4.9 GB) TX bytes:10421270721 (9.7 GB)
eth1 Link encap:Ethernet HWaddr 00:14:4F:20:BF:31
inet addr:XXX.XXX.XXX.XXX Bcast:XXX.XXX.XXX.XXX
Mask:255.255.255.0
inet6 addr: fe80::214:4fff:fe20:bf31/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:27979714 errors:0 dropped:0 overruns:0 frame:0
TX packets:24557864 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:9088421795 (8.4 GB) TX bytes:42414741512 (39.5 GB)
eth2 Link encap:Ethernet HWaddr 00:14:4F:20:BF:32
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Base address:0x8400 Memory:fbda0000-fbdc0000
eth3 Link encap:Ethernet HWaddr 00:14:4F:20:BF:33
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Base address:0x8000 Memory:fbd80000-fbda0000
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:94170 errors:0 dropped:0 overruns:0 frame:0
TX packets:94170 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:14977913 (14.2 MB) TX bytes:14977913 (14.2 MB)
peth0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
UP BROADCAST RUNNING NOARP MTU:1500 Metric:1
RX packets:23738498 errors:0 dropped:0 overruns:0 frame:0
TX packets:13712574 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:5554987958 (5.1 GB) TX bytes:10701176202 (9.9 GB)
Base address:0x8c00 Memory:fbde0000-fbe00000
peth1 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
UP BROADCAST RUNNING NOARP MTU:1500 Metric:1
RX packets:30539818 errors:0 dropped:0 overruns:0 frame:0
TX packets:40785208 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:12639200501 (11.7 GB) TX bytes:43648869440 (40.6 GB)
Base address:0x8800 Memory:fbdc0000-fbde0000
veth2 Link encap:Ethernet HWaddr 00:00:00:00:00:00
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
veth3 Link encap:Ethernet HWaddr 00:00:00:00:00:00
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
vif0.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
UP BROADCAST RUNNING NOARP MTU:1500 Metric:1
RX packets:10330155 errors:0 dropped:0 overruns:0 frame:0
TX packets:23612693 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:10421270763 (9.7 GB) TX bytes:5289657732 (4.9 GB)
vif0.1 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
UP BROADCAST RUNNING NOARP MTU:1500 Metric:1
RX packets:24557864 errors:0 dropped:0 overruns:0 frame:0
TX packets:27979716 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:42414741512 (39.5 GB) TX bytes:9088421935 (8.4 GB)
vif0.2 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
vif0.3 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
xenbr0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:21837296 errors:0 dropped:0 overruns:0 frame:0
TX packets:40 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2926456076 (2.7 GB) TX bytes:1764 (1.7 KB)
xenbr1 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:5143094 errors:0 dropped:0 overruns:0 frame:0
TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:3541364391 (3.2 GB) TX bytes:454 (454.0 b)
Sorry, I forgot to include this in the last email
---------------------------------------------------------------------
Eric Ross
Linux Systems Administrator
Argonne National Laboratory
Computing and Information Systems
Phone: 630.252.2600
---------------------------------------------------------------------
Andy Burns wrote:> 2008/8/19 Eric Ross <eross@anl.gov>:
>
>> If I start a ping on hypervisor FOO to hypervisor BAR, the ping will
>> fail, saying that the destination host cannot be reached. However, if
I
>> start a ping on FOO to BAR, and, at the same time, start a ping from
BAR
>> to FOO both commands will succeed.
>
> By hypervisor I presume you actually mean domU?
>
> Sounds like a MAC address or bridging issue, did you copy BAR from
> FOO''s xen config file? If so did you change the MAC address? What
> does a "brctl show" on the dom0 look like? Or an "ifconfig
-a" on FOO
> and BAR and the dom0?
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
2008/8/20 Eric Ross <eross@anl.gov>:> Dom0 brctl output > > bridge name bridge id STP enabled interfaces > xenbr0 8000.feffffffffff no vif0.0 > peth0 > xenbr1 8000.feffffffffff no vif0.1 > peth1 > NOTE: this is the same for all my Dom0sAre the ports of the switches that your dom0s are connected to set to "portfast" mode? (ok that''s a cisco term but other brands have their equivalent) they should *not* be. If using Cisco, other portsecurity settings can clash such as only allowing single MAC addresses per port etc. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Andy, I talked to the networking guys, they had that feature enabled. Unfortunately disabling it didn''t help. However, I did stumble upon something interesting. route showed something like this localnet * 255.255.255.0 U 0 0 0 eth0 default something 0.0.0.0 UG 100 0 0 eth0 getting rid of the route for localnet and sending everything to the default gateway fixed everything. Unfortunately, I consider this more of a temporary hack then a solution. Does anyone see something I am missing, or perhaps a better solution? Andy - thanks for all the help btw. Thanks, --------------------------------------------------------------------- Eric Ross Linux Systems Administrator Argonne National Laboratory Computing and Information Systems Phone: 630.252.2600 --------------------------------------------------------------------- Andy Burns wrote:> 2008/8/20 Eric Ross <eross@anl.gov>: > >> Dom0 brctl output >> >> bridge name bridge id STP enabled interfaces >> xenbr0 8000.feffffffffff no vif0.0 >> peth0 >> xenbr1 8000.feffffffffff no vif0.1 >> peth1 >> NOTE: this is the same for all my Dom0s > > Are the ports of the switches that your dom0s are connected to set to > "portfast" mode? (ok that''s a cisco term but other brands have their > equivalent) they should *not* be. > > If using Cisco, other portsecurity settings can clash such as only > allowing single MAC addresses per port etc. > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users