CentOS virt - Nov 2011 - XEN multiple bridge problem - VM won' start!

Hi,

I've been using CentOS & Xen on a server that has 2 VM's configured.
The
default configuration includes one physical iface that is propagated (by a
default bridge) to the VM's.

Since I wanted to configure additional physical iface, define a new bridge
and propagate it to the viface-s of the VM's, i configured the bridge/phys.
iface and brought it up (here are configurations that I set up):

-> eth3
DEVICE=eth3
BOOTPROTO=static
HWADDR=D4:85:64:4B:76:AB
ONBOOT=yes
#HOTPLUG=no
#DHCP_HOSTNAME=kdr-3k-4r-3o-07
BRIDGE=br0
TYPE=Ethernet

-> br0
DEVICE=br0
TYPE=Bridge
BOOTPROTO=none
ONBOOT=yes

Then I configured new virt. ifaces on VM's (using virsh edit, config is
listed below):

<interface type='bridge'>
<mac address='00:16:36:17:62:5d'/>
<source bridge='xenbr0'/>
<script path='vif-bridge'/>
<target dev='vif5.0'/>
</interface>
<interface type='bridge'>
<mac address='00:16:3e:ca:63:39'/>
<source bridge='br0'/>
<script path='vif-bridge'/>
<target dev='vif5.1'/>
</interface>

You can see the default bridge (xenbr0 -> vif5.0) and my new bridge br0
connected to vif5.1.

So, here I got stuck up - when I try tu start a VM (xm create VIRT_SRV),
the VM starts booting but hangs up (I can ping it form another srv., but
when I try SSH it says "connection refused"). When I remove
configuration
for vif5.1 the VM starts up normally. I actually managed to start VM with
vif5.1 up once, but next time I restarted it, the problem was there :/


Furthermore, I noticed this in the Xen log:

[2011-11-18 09:27:16 xend 8213] DEBUG (DevController:160) Waiting for
devices vif.
[2011-11-18 09:27:16 xend 8213] DEBUG (DevController:166) Waiting for 0.
[2011-11-18 09:27:16 xend 8213] DEBUG (DevController:544)
hotplugStatusCallback /local/domain/0/backend/vif/1/0/hotplug-status.
[2011-11-18 09:27:16 xend 8213] DEBUG (DevController:544)
hotplugStatusCallback /local/domain/0/backend/vif/1/0/hotplug-status.
[2011-11-18 09:27:16 xend 8213] DEBUG (DevController:558)
hotplugStatusCallback 1.
[2011-11-18 09:27:16 xend 8213] DEBUG (DevController:166) Waiting for 1.
[2011-11-18 09:27:16 xend 8213] DEBUG (DevController:544)
hotplugStatusCallback /local/domain/0/backend/vif/1/1/hotplug-status.
[2011-11-18 09:27:16 xend 8213] DEBUG (DevController:558)
hotplugStatusCallback 1.
[2011-11-18 09:27:16 xend 8213] DEBUG (DevController:160) Waiting for
devices usb.
[2011-11-18 09:27:16 xend 8213] DEBUG (DevController:160) Waiting for
devices vbd.
[2011-11-18 09:27:16 xend 8213] DEBUG (DevController:166) Waiting for 768.
[2011-11-18 09:27:16 xend 8213] DEBUG (DevController:544)
hotplugStatusCallback /local/domain/0/backend/vbd/1/768/hotplug-status.
[2011-11-18 09:27:16 xend 8213] DEBUG (DevController:558)
hotplugStatusCallback 1.
[2011-11-18 09:27:16 xend 8213] DEBUG (DevController:166) Waiting for 5632.
[2011-11-18 09:27:16 xend 8213] DEBUG (DevController:544)
hotplugStatusCallback /local/domain/0/backend/vbd/1/5632/hotplug-status.
[2011-11-18 09:27:16 xend 8213] DEBUG (DevController:558)
hotplugStatusCallback 1.
[2011-11-18 09:27:16 xend 8213] DEBUG (DevController:160) Waiting for
devices irq.
[2011-11-18 09:27:16 xend 8213] DEBUG (DevController:160) Waiting for
devices vkbd.
[2011-11-18 09:27:16 xend 8213] DEBUG (DevController:160) Waiting for
devices vfb.
[2011-11-18 09:27:16 xend 8213] DEBUG (DevController:160) Waiting for
devices pci.
[2011-11-18 09:27:16 xend 8213] DEBUG (DevController:160) Waiting for
devices ioports.
[2011-11-18 09:27:16 xend 8213] DEBUG (DevController:160) Waiting for
devices tap.
[2011-11-18 09:27:16 xend 8213] DEBUG (DevController:160) Waiting for
devices vtpm.

So the boot process goes on regulary until the last line ("Waiting for
devices vtpm"). It seems like TPM makes it impossible for VM's to start
regulary. When I open the VM's console (with virt-manager) I see the whole
boot process but it hangs when it tries to start iface eth2 (attached to
the second bridge) ...

Does anybody have a clue why is this happening?

Thanks in advance ..
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.centos.org/pipermail/centos-virt/attachments/20111118/e056a2eb/attachment-0002.html>

Hi,

2011/11/18 Matija Draganovi? <mdra137 at
gmail.com>:
> I've been using CentOS & Xen on a server that has 2 VM's
configured. The
> default configuration includes one physical iface that is propagated (by a
> default bridge) to the VM's.
You do not mention which version of CentOS and Xen you are using?
>
> Since I wanted to configure additional physical iface, define a new bridge
> and propagate it to the viface-s of the VM's, i configured the
bridge/phys.
> iface and brought it up (here are configurations that I set up):
>
> -> eth3
> DEVICE=eth3
> BOOTPROTO=static
> HWADDR=D4:85:64:4B:76:AB
> ONBOOT=yes
> #HOTPLUG=no
> #DHCP_HOSTNAME=kdr-3k-4r-3o-07
> BRIDGE=br0
> TYPE=Ethernet
>
> -> br0
> DEVICE=br0
> TYPE=Bridge
> BOOTPROTO=none
> ONBOOT=yes
If this is a CentOS5 machine with the CentOS provided Xen 3.0
packages, then here is info how i got my bridges to work with that
setup:

* For dom0 I configured eth0 and eth1 as usual in /etc/sysconfig/network-scripts

* I did not configure br0 or br1 in network-scripts, but instead I
created a file called /etc/xen/scripts/my-network-script having this
contents:

#!/bin/sh
dir=$(dirname "$0")
"$dir/network-bridge" "$@" vifnum=0 netdev=eth0
bridge=xenbr0
"$dir/network-bridge" "$@" vifnum=1 netdev=eth1
bridge=xenbr1

* Then I modified in dom0 the file /etc/xen/xend-config.sxp in the
following way:

#(network-script network-bridge)
(network-script my-network-script)

* Now rebooting dom0 made the bridges available

* After that I could configure them in my domU config:

vif = [
"mac=00:16:3E:69:29:25,bridge=xenbr0,script=vif-bridge","mac=00:16:3E:E6:B0:6D,bridge=xenbr1,script=vif-bridge"
]

* After starting the domU I could configure the interfaces in
network-scripts using the hardware addresses specified in the domU
config


It seems that configuring bridging is done a bit differently in
different Xen versions, so this might not work if you are using some
other kind of config. With CentOS6 and 3rd party Xen 4.1 packages this
procedure did not work at all and instead I needed to do the
following:

* in dom0 create the br0 and br1 devices in network-scripts

* in dom0 /etc/xen/xend-config.sxp put instead:

#(network-script network-bridge)
(network-script /bin/true)

* and in dom0 use different configuration for the domU config files like this:

vif = [
"mac=00:16:3E:69:29:25,bridge=br0","mac=00:16:3E:E6:B0:6D,bridge=br1"
]


Hope this helps, unfortuantely I am not familiar with virsh at all.

Best,
Peter

Dear Peter,


Thanks for your reply. I was so occupied with troubleshooting that I forgot
to mention the OS & Xen version :)

My CentOS version is
CentOS release 5.7 (Final)
Linux 2.6.18-274.7.1.el5xen #1 SMP Thu Oct 20 17:06:34 EDT 2011 x86_64
x86_64 x86_64 GNU/Linux

and Xen v3.0.

I tried creating bridges the same way you did and succedded. I also tried
doing the same via ifcfg scripts, and it also worked.

I have no problem with creating bridges and connecting them with physical
ifaces, but I can't start my VM's (most of the time one VM start but the
other one hangs while trying to initalize eth2 that is connected to my
second (non default - xenbr1) bridge)).

The only diference between your config and mine is that I have vifname-s
listed (example below):

vif = [
"mac=00:16:36:72:35:e5,bridge=xenbr0,script=vif-bridge,vifname=vif4.0",
"mac=00:16:3e:57:de:0d,bridge=xenbr1,script=vif-bridge,
vifname=vif4.1" ]

I'm not saying that this could be the problem, but since I'm slowly
running
out of ideas - it's a longshot :)

So, to sum it all up - I can create bridges properly, but my VM's hang when
they try to bring up vifaces connected to the xenbr1. Also, I have no
problem with having only one bridge active - problem appears when I have
multiple bridges.

Br,
Matija
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.centos.org/pipermail/centos-virt/attachments/20111118/2a75ed85/attachment-0002.html>

I tried to remove vifname and vif-script but the problem remains. I think
that the bridges are ok, but for some reason VM's hang when they try to
connect their vifaces to them.

I'll try some different configurations of the physical ifaces (an vifaces
also), maybe I'll be able to find out something. Also, I'll focus on the
vTPM, since the following line shows up in the Xen log when the VM tries to
bring up the iface connected to the second bridge:

[2011-11-18 09:27:16 xend 8213] DEBUG (DevController:160) Waiting for
devices vtpm


Furthermore, it appears that sometimes the VM's manage to start regulary, I
just can't figure out when & why :/

Tnx for your comments ..

Br,
M.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.centos.org/pipermail/centos-virt/attachments/20111119/cc41e78f/attachment.html

Ok, after trying multiple different configurations I decided to play by the
book - I removed ifcfg script for creating the additional bridge and set up
an Xen script. So my new configuration is:

-> eth3 (my physical interface)
# NetXen Incorporated NX3031 Multifunction 1/10-Gigabit Server Adapter
DEVICE=eth3
BOOTPROTO=static
HWADDR= ...
ONBOOT=yes
IPADDR= ...
NETMASK= ...
NETWORK= ...

and the network-xen-multi-bridge script:

#!/bin/sh
# network-xen-multi-bridge
# Exit if anything goes wrong.
set -e
# First arg is the operation.
OP=$1
shift
script=/etc/xen/scripts/network-bridge.xen
case ${OP} in
start)
$script start vifnum=3 bridge=xenbr3 netdev=eth3
$script start vifnum=0 bridge=xenbr0 netdev=eth0
;;
stop)
$script stop vifnum=3 bridge=xenbr3 netdev=eth3
$script stop vifnum=0 bridge=xenbr0 netdev=eth0
;;
status)
$script status vifnum=3 bridge=xenbr3 netdev=eth3
$script status vifnum=0 bridge=xenbr0 netdev=eth0
;;
*)
echo 'Unknown command: ' ${OP}
echo 'Valid commands are: start, stop, status'
exit 1
esac

After I restart the physical server bridges are set up correctly:

# brctl show
bridge name bridge id STP enabled interfaces
virbr0 8000.000000000000 yes
xenbr0 8000.feffffffffff no vif0.0
peth0
xenbr3 8000.feffffffffff no vif0.3
peth3

Unfortunately, this did not solve the problem - I manage to start one VM
(connected to both of the bridges), but when I try to start the other one,
it just hangs when trying to bring up the second interface (connected to
xenbr3) :/

I'm currently completely out of ideas, but I'm not giving up ;)
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.centos.org/pipermail/centos-virt/attachments/20111122/eb760bb6/attachment.html

Finally, I managed to solve the problem. The problem was rather strange -
after configuring bridges I added new ifaces to the VM. For some reason,
during boot process of the VM's, wrong interface was started (VM tried to
bring up eth2 instead eth1?).

After adding the new ifaces through virt-manager, the VM started regulary
and created a new network script for eth2
(/etc/sysconfig/network-scripts/ifcfg-eth2). I think that the problem was
this - when the VM is restarted it tried to bring up eth2 while vif-script
tried to propagate second bridge via eth1 iface. I resolved this simply by
replacing ifcfg-eth2 script with ifcfg-eth1. Now, VM starts/restarts
regulary ;)

So, the process for creating and configuring new bridges that I follow is
(tried on multiple servers):

- create new script that sets up multiple bridges in /etc/xen/scripts/

example: network-xen-multi-bridge

#!/bin/sh
# network-xen-multi-bridge
# Exit if anything goes wrong.
set -e
# First arg is the operation.
OP=$1
shift
script=/etc/xen/scripts/network-bridge.xen
case ${OP} in
start)
        $script start vifnum=3 bridge=xenbr3 netdev=eth3
        $script start vifnum=0 bridge=xenbr0 netdev=eth0
        ;;
stop)
        $script stop vifnum=3 bridge=xenbr3 netdev=eth3
        $script stop vifnum=0 bridge=xenbr0 netdev=eth0
        ;;
status)
        $script status vifnum=3 bridge=xenbr3 netdev=eth3
        $script status vifnum=0 bridge=xenbr0 netdev=eth0
        ;;
*)
        echo 'Unknown command: ' ${OP}
        echo 'Valid commands are: start, stop, status'
        exit 1
esac



- make it executable (chmod 755 network-xen-multi-bridge)

- configure /etc/xen/xend-config.sxp (comment out the current script and
add the new one);

# It is possible to use the network-bridge script in more complicated
# scenarios, such as having two outgoing interfaces, with two bridges, and
# two fake interfaces per guest domain.  To do things like this, write
# yourself a wrapper script, and call network-bridge from it, as appropriate.
#
#(network-script network-bridge)
(network-script  network-xen-multi-bridge)



- restart physical server so the bridges are properly created

- add virt. ifaces to the VM's (via virt-manager or via config. files in
/etc/xen/ ..)

- boot VM

- configure newly added iface in /etc/sysconfig/network-scripts/

- restart VM

Additionaly I check everything by restarting phys. server to make sure that
everything is ok ;)
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.centos.org/pipermail/centos-virt/attachments/20111125/1304c71b/attachment.html