Paul Theodoropoulos
2008-Jun-20 15:27 UTC
[Xen-users] lost network connectivity after update to 3.0.3
Centos 5.1, an updated kernel and xen were pushed out RH, i believe in
response to a rash of security bugs found. i updated my guest instances,
rebooted them, they were still reachable. updated domU and rebooted, and
now the guests are no longer reachable. figuring i fubarred the proper
steps i destroyed then created the instances again - still no go.
domU is running:
kernel-xen-2.6.18-53.1.21.el5
xen-3.0.3-41.el5_1.6
xen-libs-3.0.3-41.el5_1.6
and the guests are running:
kernel-xen-2.6.18-53.1.21.el5
right now i have only one guest up (why bother with the others running.
on domU, if have the following IFs:
eth0 Link encap:Ethernet HWaddr 00:30:48:C0:6C:5A
inet addr:10.255.40.100 Bcast:10.255.40.255 Mask:255.255.255.0
inet6 addr: fe80::230:48ff:fec0:6c5a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:39458 errors:0 dropped:0 overruns:0 frame:0
TX packets:1113 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2433216 (2.3 MiB) TX bytes:133979 (130.8 KiB)
Interrupt:19 Base address:0xa000
eth0:1 Link encap:Ethernet HWaddr 00:30:48:C0:6C:5A
inet addr:10.255.40.101 Bcast:10.255.40.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Interrupt:19 Base address:0xa000
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:130 errors:0 dropped:0 overruns:0 frame:0
TX packets:130 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:16056 (15.6 KiB) TX bytes:16056 (15.6 KiB)
vif6.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
UP BROADCAST RUNNING NOARP MTU:1500 Metric:1
RX packets:451 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:32
RX bytes:18682 (18.2 KiB) TX bytes:0 (0.0 b)
xenbr0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
UP BROADCAST RUNNING NOARP MTU:1500 Metric:1
RX packets:4125 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:121554 (118.7 KiB) TX bytes:0 (0.0 b)
and the routing table on domU is:
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt
Iface
10.255.40.0 0.0.0.0 255.255.255.0 U 0 0 0
eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0
eth0
0.0.0.0 10.255.40.1 0.0.0.0 UG 0 0 0
eth0
which is correct. brctl show returns the following:
bridge name bridge id STP enabled interfaces
xenbr0 8000.feffffffffff no vif6.0
on the guest i have:
eth0 Link encap:Ethernet HWaddr 00:16:3E:2C:CF:7E
inet addr:10.255.40.110 Bcast:10.255.40.255 Mask:255.255.255.0
inet6 addr: fe80::216:3eff:fe2c:cf7e/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:479 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:26172 (25.5 KiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:365 errors:0 dropped:0 overruns:0 frame:0
TX packets:365 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:32683 (31.9 KiB) TX bytes:32683 (31.9 KiB)
and the routing table is
Destination Gateway Genmask Flags MSS Window irtt
Iface
10.255.40.0 0.0.0.0 255.255.255.0 U 0 0 0
eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0
eth0
0.0.0.0 10.255.40.1 0.0.0.0 UG 0 0 0
eth0
which again is exactly as expected. interestingly, after xm console''ing
to a guest instance, i can ping, traceroute, and even ssh to *another*
guest instance - but not to the domU or to any other network besides the
other guests.
i''m baffled. i''m unclear what exactly broke, or why/how it
broke - and
most importantly, how to fix it. i''ve been googling for hours.
thoughts, suggestions, brickbats, ?
--
Paul Theodoropoulos
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Paul Theodoropoulos
2008-Jun-20 19:03 UTC
Re: [Xen-users] lost network connectivity after update to 3.0.3
Paul Theodoropoulos wrote:> * PGP Signed: 06/20/08 at 08:27:31 > > Centos 5.1, an updated kernel and xen were pushed out RH, i believe in > response to a rash of security bugs found. i updated my guest > instances, rebooted them, they were still reachable. updated domU and > rebooted, and now the guests are no longer reachable. figuring i > fubarred the proper steps i destroyed then created the instances again > - still no go.my apologies for my broken nomenclature. when i refer to ''domU'' i should have been referring to ''dom0''. i''m fairly new to xen, if that wasn''t already obvious! -- Paul Theodoropoulos _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Joseph L. Casale
2008-Jun-20 21:04 UTC
RE: [Xen-users] lost network connectivity after update to 3.0.3
>my apologies for my broken nomenclature. when i refer to ''domU'' i should >have been referring to ''dom0''. i''m fairly new to xen, if that wasn''t >already obvious!Dom0 is reachable I presume, whats the output of #`brctl show` ? I havent used the distro supplied Xen from CentOS in so long, but is it possible that the network scripts changed? Cat your xend-config.sxp for network-script, see what network script your using: # cat /etc/xen/xend-config.sxp | grep network (network-script multi-network-bridge) Cat that script, see what bridge names it uses: # cat /etc/xen/scripts/multi-network-bridge "$dir/network-bridge" "$@" vifnum=0 netdev=eth0 bridge=eth0 "$dir/network-bridge" "$@" vifnum=1 netdev=dummy0 bridge=dummy0 . . . Cat a vm''s config, see what it wants to use: # cat /etc/xen/builder.pv | grep vif vif = [ ''bridge=eth0, mac=00:16:3E:77:A5:D7'', ] Long shot, but if Dom0 can see the rest of the lan, I know xen changed this somewhere along the line. I doubt RH did but worth a try. jlc _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Paul Theodoropoulos
2008-Jun-20 21:44 UTC
Re: [Xen-users] lost network connectivity after update to 3.0.3
Joseph L. Casale wrote:>> my apologies for my broken nomenclature. when i refer to ''domU'' i should >> have been referring to ''dom0''. i''m fairly new to xen, if that wasn''t >> already obvious! >> > > Dom0 is reachable I presume, whats the output of #`brctl show` ? > > I havent used the distro supplied Xen from CentOS in so long, but is it possible > that the network scripts changed? > > Cat your xend-config.sxp for network-script, see what network script your using: > # cat /etc/xen/xend-config.sxp | grep network > (network-script multi-network-bridge) > > Cat that script, see what bridge names it uses: > # cat /etc/xen/scripts/multi-network-bridge > "$dir/network-bridge" "$@" vifnum=0 netdev=eth0 bridge=eth0 > "$dir/network-bridge" "$@" vifnum=1 netdev=dummy0 bridge=dummy0 > . > . > . > Cat a vm''s config, see what it wants to use: > # cat /etc/xen/builder.pv | grep vif > vif = [ ''bridge=eth0, mac=00:16:3E:77:A5:D7'', ] > > Long shot, but if Dom0 can see the rest of the lan, I know xen changed this somewhere along the line. I doubt RH did but worth a try. > > jlc >thanks for the reply. i did include the ''brctl show'' output, but it was kind of buried in the diagnostics. your reply - plus Fangfei Zhou''s - got me going in the right direction. i ran /etc/xen/scripts/network-bridge status, and it showed that there was no peth0. so i took a leap and ran /etc/xen/scripts/network-bridge start - poof, there''s my networking. the question is, why isn''t this happening when the server boots. it''s not a huge issue to do it by hand, really, since it''s rare that the dom0 gets rebooted. but it''s peculiar that it''s not happening automatically as it did before. my /etc/xen/xend-config.sxp had (network-script network-bridge) in it. i changed it to (network-script ''/etc/xen/scripts/network-bridge netdev=eth0'') but that made no difference- after a reboot, the networking was still down. since i''m not 97% back in business - one of the things i had contemplated as a ''fix'' was to install xen 3.2 from source. are there any risks in upgrading from 3.0.3 to the latest? more broken things, fewer, the same? -- Paul Theodoropoulos _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Joseph L. Casale
2008-Jun-20 21:53 UTC
RE: [Xen-users] lost network connectivity after update to 3.0.3
>since i''m not 97% back in business - one of the things i had >contemplated as a ''fix'' was to install xen 3.2 from source. are there >any risks in upgrading from 3.0.3 to the latest? more broken things, >fewer, the same?So you do have networking for Dom0 when it boots, its just the script doesn''t execute? Is it network-bridge executable (would it need to be if you ran it the way you did?)? Anyway, I have ran 3.2.0 from the srpm for ages and it runs fairly solidly. I had a few issues all my fault, namely related to iptables in Dom0 and memory allocation getting to low for Dom0 but its pretty solid. As far as I know 3.2.1 has some good fixes in it too but I never got it to compile :( jlc _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Paul Theodoropoulos
2008-Jun-20 21:55 UTC
Re: [Xen-users] lost network connectivity after update to 3.0.3
Paul Theodoropoulos wrote:> since i''m not 97% back in business - one of the things i had > contemplated as a ''fix'' was to install xen 3.2 from source. are there > any risks in upgrading from 3.0.3 to the latest? more broken things, > fewer, the same?argh. typos are going to be my downfall. that should be "since i''m NOW 97% back in business. argh. -- Paul Theodoropoulos _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Paul Theodoropoulos
2008-Jun-20 22:03 UTC
Re: [Xen-users] lost network connectivity after update to 3.0.3
Joseph L. Casale wrote:>> since i''m not 97% back in business - one of the things i had >> contemplated as a ''fix'' was to install xen 3.2 from source. are there >> any risks in upgrading from 3.0.3 to the latest? more broken things, >> fewer, the same? >> > > So you do have networking for Dom0 when it boots, its just the script doesn''t execute? > Is it network-bridge executable (would it need to be if you ran it the way you did?)? > > Anyway, I have ran 3.2.0 from the srpm for ages and it runs fairly solidly. I had a few > issues all my fault, namely related to iptables in Dom0 and memory allocation getting to low > for Dom0 but its pretty solid. As far as I know 3.2.1 has some good fixes in it too but I > never got it to compile :( > > jlc >yeah, that was one of the first things i checked - the execute perms on those scripts. some of the execute perms were broken on some of the scripts, but that still didn''t fix it. it''s really strange. i think i''ll give 3.2.1 a shot. can''t hurt any worse than the twenty hours my domU''s were unreachable! -- Paul Theodoropoulos _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users