Hello Everyone,
I have set up a xen server with 10 physical network devices using the bridge
mode of XEN. This is because I want to split and separate the network traffic
for security reasons. Maybe the best way is to use pci back hide. But I want to
understand what is going on here.
For all testings I have only activated eth[0-3].
My first tests were successful. I have created a VM using
''xm''. But we got some network problems:
root@bs-xen-p-01:~[0]# netstat -rnv
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
10.1.17.10 10.1.1.1 255.255.255.255 UGH 0 0 0 eth0
10.1.17.0 0.0.0.0 255.255.255.0 U 0 0 0 eth3
10.1.32.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2
10.1.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
10.1.19.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
0.0.0.0 10.1.1.1 0.0.0.0 UG 0 0 0 eth0
root@bs-xen-p-01:~[0]#
The routing for eth3, eth3 and eth1 points back to his source. To get partial
rid of the network problems we first define the first rule (we administrate the
xen server through a portal running in the 10.1.17.0/24 segment).
Then I have had the idea to define the physical interfaces in network config to
some IP''s residing in the same subnet as the xen server itself
(10.1.1.0/24) i. e.:
auto eth1
allow-hotplug eth1
iface eth1 inet static
address 10.1.1.201
netmask 255.255.255.0
network 1.1.1.0
But since I have done this and restarted the network, the VM cannot start any
more.
I have rebooted the machine - nothing
I have set back the network interface - nothing
I have rebooted the machine - nothing
ifconfig shows only eth[0-9] and lo. bridge control:
root@bs-xen-p-01:~[0]# brctl show
bridge name bridge id STP enabled interfaces
root@bs-xen-p-01:~[0]#
All I''ve got is a message:
root@bs-xen-p-01:~[0]# xm create otrs-test.sxp
Using config file "/etc/xen/otrs-test.sxp".
Error: Device 0 (vif) could not be connected. Hotplug scripts not working.
root@bs-xen-p-01:~[0]#
The orts-test.sxp file:
name = "otrs-test"
kernel = "/boot/vmlinuz-2.6.18-4-xen-amd64"
ramdisk = "/boot/initrd.img-2.6.18-4-xen-amd64"
memory = 512
disk = [ ''phy:lvm-vg1/vm-otrs-test,sda1,w'' ]
root = "/dev/sda1 ro"
vif = [ ''ip=10.1.19.20,mac=00:16:de:ad:fa:cd,bridge=xenbr1'' ]
dhcp="off"
netmask="255.255.255.0"
gateway="10.1.19.1"
hostname="otrs-test"
If I set vif to [ '''' ] and try starting the VM:
root@bs-xen-p-01:~[0]# xm create otrs-test.sxp
Using config file "/etc/xen/otrs-test.sxp".
Error: Device 0 (vif) could not be connected. Backend device not found.
root@bs-xen-p-01:~[1]#
So I start the bridge manually:
/etc/xen/scripts/network-bridge start vifnum=0 bridge=xenbr0 netdev=eth0
/etc/xen/scripts/network-bridge start vifnum=1 bridge=xenbr1 netdev=eth1
/etc/xen/scripts/network-bridge start vifnum=2 bridge=xenbr2 netdev=eth2
root@bs-xen-p-01:~[0]# dmesg|grep xenbr0
xenbr0: port 1(vif0.0) entering learning state
xenbr0: topology change detected, propagating
xenbr0: port 1(vif0.0) entering forwarding state
xenbr0: port 2(peth0) entering learning state
xenbr0: topology change detected, propagating
xenbr0: port 2(peth0) entering forwarding state
xenbr0: port 1(vif0.0) entering disabled state
xenbr0: port 2(peth0) entering disabled state
xenbr0: port 2(peth0) entering disabled state
xenbr0: port 1(vif0.0) entering disabled state
root@bs-xen-p-01:~[0]# dmesg|grep xenbr1
xenbr1: port 1(vif0.1) entering learning state
xenbr1: topology change detected, propagating
xenbr1: port 1(vif0.1) entering forwarding state
xenbr1: port 2(peth1) entering learning state
xenbr1: topology change detected, propagating
xenbr1: port 2(peth1) entering forwarding state
root@bs-xen-p-01:~[0]# dmesg|grep xenbr2
xenbr2: port 1(vif0.2) entering learning state
xenbr2: topology change detected, propagating
xenbr2: port 1(vif0.2) entering forwarding state
xenbr2: port 2(peth2) entering learning state
xenbr2: topology change detected, propagating
xenbr2: port 2(peth2) entering forwarding state
root@bs-xen-p-01:~[0]#
ifconfig shows eth[0-3], lo, peth[1-2], vif0.[1-2] and xenbr[1-2]. All
interfaces up and running and have a link status.
I fired up the VM, the VM starts, but ends up in the maintenance mode because
the filesystem of the VM is mounted read only. If I gave the root password to
enter the maintenance mode I can see the networking is not startet (ifconfig
shows nothing).
On the xen side:
root@bs-xen-p-01:~[0]# brctl show
bridge name bridge id STP enabled interfaces
xenbr0 8000.feffffffffff no vif0.0
peth0
xenbr1 8000.feffffffffff no vif0.1
peth1
vif6.0
xenbr2 8000.feffffffffff no vif0.2
peth2
root@bs-xen-p-01:~[0]#
I have viewed /var/log/xen/*, /var/log/messages and dmesg. The only hint I have
found is in xen-hotplug.log:
Nothing to flush.
bridge xenbr1 does not exist!
So I switched on (set -x) the "debug" mode in the
xen-hotplug-common.sh. Then I rebooted the machine. After rebooting I only try
to start the VM. You can see the result in the attached file. I have seen
several ''trap sigerr ERR'' but I don''t understand
where they come from.
Could someone help me, not only explaining the last error.
Are there any suggestions setting up a XEN server with many physical network
interfaces?
Christian Placzek
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
On Thu, 2007-03-15 at 20:06 +0100, Christian Placzek wrote:> auto eth1 > allow-hotplug eth1 > iface eth1 inet static > address 10.1.1.201 > netmask 255.255.255.0 > network 1.1.1.0Stop using network-bridge , its there to construct and deconstruct a bridge so that Xen itself is non destructive (no changing of network init needed to install it). Since you are doing the more production-sensible thing by letting Debian handle your bridges in init, there''s no need to be using network-bridge. Better, more structured and sensible control over all of that is partly achieved by how you construct your guests. First thing to do, get rid of network-bridge In /etc/xen/scripts/ do echo -e "#!/bin/sh\nexit 0" > network-dummy && chmod +x network-dummy Then in /etc/xen/ change xend-config.sxp to call network-dummy instead of network-bridge. Secondly (just suggesting) : Specify vifname= in the vif [] containers in the guests to keep it all nice and neat. If you name the vif after the domname (or something else specific and meaningful), bandwidth accounting becomes *very* easy over an entire farm, which sounds like it would also lend well to your setup. Best, --Tim _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Fri, 2007-03-16 at 03:21 +0800, Tim Post wrote: Sorry, forgot to show you the alternate config :> On Thu, 2007-03-15 at 20:06 +0100, Christian Placzek wrote: > > > auto eth1becomes auto xen-br1> > allow-hotplug eth1you can add allow-hotplug xen-br1 but should not be needed> > iface eth1 inet staticiface xen-br1 inet static> > address 10.1.1.201 > > netmask 255.255.255.0 > > network 1.1.1.0#then add: bridge_fd 0 bridge_maxwait 0 bridge_helo 0 bridge_stp off bridge_ports eth1 Just tell the bridge what eth device to port, or ''none'' if you want to setup a dummy bridge to serve as a switch for some guests. You need to give it an IP even if its a dummy (.1 in any reserved address is fine), just no gatway. Contradictory, every bridge that lets guests talk to the internet will need a gateway, or it will only let guests talk to the same network as the bridge. Tempting as it may seem, leave stp off, you don''t wanna go there. Remember that when you re-start networking, you''ll need to re-attach the vif''s to their respective bridges. you can see scripts/vif-* for how to manage this within your own scripts. Another way to do it is with a vif-detach / attach, but I haven''t played with that way yet. Best, and sorry for the double post. I forgot to paste my version instead of the quoted one. --Tim> > Stop using network-bridge , its there to construct and deconstruct a > bridge so that Xen itself is non destructive (no changing of network > init needed to install it). > > Since you are doing the more production-sensible thing by letting Debian > handle your bridges in init, there''s no need to be using network-bridge. > Better, more structured and sensible control over all of that is partly > achieved by how you construct your guests. > > First thing to do, get rid of network-bridge > > In /etc/xen/scripts/ do > > echo -e "#!/bin/sh\nexit 0" > network-dummy && chmod +x network-dummy > > Then in /etc/xen/ change xend-config.sxp to call network-dummy instead > of network-bridge. > > Secondly (just suggesting) : > > Specify vifname= in the vif [] containers in the guests to keep it all > nice and neat. If you name the vif after the domname (or something else > specific and meaningful), bandwidth accounting becomes *very* easy over > an entire farm, which sounds like it would also lend well to your setup. > > Best, > --Tim > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Tim, thank you for your comments. Tim Post wrote:> On Thu, 2007-03-15 at 20:06 +0100, Christian Placzek wrote: > >> auto eth1 >> allow-hotplug eth1 >> iface eth1 inet static >> address 10.1.1.201 >> netmask 255.255.255.0 >> network 1.1.1.0 > > Stop using network-bridge , its there to construct and deconstruct a > bridge so that Xen itself is non destructive (no changing of network > init needed to install it). > > Since you are doing the more production-sensible thing by letting Debian > handle your bridges in init, there''s no need to be using network-bridge. > Better, more structured and sensible control over all of that is partly > achieved by how you construct your guests. > > First thing to do, get rid of network-bridge > > In /etc/xen/scripts/ do > > echo -e "#!/bin/sh\nexit 0" > network-dummy && chmod +x network-dummyJust for information: this script already exists in the debian package (v. 3.0.3).> > Then in /etc/xen/ change xend-config.sxp to call network-dummy instead > of network-bridge. > > Secondly (just suggesting) : > > Specify vifname= in the vif [] containers in the guests to keep it all > nice and neat. If you name the vif after the domname (or something else > specific and meaningful), bandwidth accounting becomes *very* easy over > an entire farm, which sounds like it would also lend well to your setup.What do you mean? Naming the bridge for easy identifying or are there more options?> > Best, > --Tim > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >-- Mit freundlichen Grüßen / with kind regards Christian Placzek _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Tim, this solution works. I have setup the network as above and in the VM with an IP from the subnet 10.1.19.0. I am little bit confused. Why does we need an IP on the bridge on XEN side? I thought a bridge works on layer 2. Anyway thanks for your help Regards Christian Tim Post wrote:> On Fri, 2007-03-16 at 03:21 +0800, Tim Post wrote: > > Sorry, forgot to show you the alternate config : > >> On Thu, 2007-03-15 at 20:06 +0100, Christian Placzek wrote: >> >>> auto eth1 > becomes auto xen-br1 >>> allow-hotplug eth1 > you can add allow-hotplug xen-br1 but should not be needed >>> iface eth1 inet static > iface xen-br1 inet static >>> address 10.1.1.201 >>> netmask 255.255.255.0 >>> network 1.1.1.0 > #then add: > bridge_fd 0 > bridge_maxwait 0 > bridge_helo 0 > bridge_stp off > bridge_ports eth1 > > Just tell the bridge what eth device to port, or ''none'' if you want to > setup a dummy bridge to serve as a switch for some guests. You need to > give it an IP even if its a dummy (.1 in any reserved address is fine), > just no gatway. > > Contradictory, every bridge that lets guests talk to the internet will > need a gateway, or it will only let guests talk to the same network as > the bridge. Tempting as it may seem, leave stp off, you don''t wanna go > there. > > Remember that when you re-start networking, you''ll need to re-attach the > vif''s to their respective bridges. you can see scripts/vif-* for how to > manage this within your own scripts. > > Another way to do it is with a vif-detach / attach, but I haven''t played > with that way yet. > > Best, and sorry for the double post. I forgot to paste my version > instead of the quoted one. > > --Tim > >> Stop using network-bridge , its there to construct and deconstruct a >> bridge so that Xen itself is non destructive (no changing of network >> init needed to install it). >> >> Since you are doing the more production-sensible thing by letting Debian >> handle your bridges in init, there''s no need to be using network-bridge. >> Better, more structured and sensible control over all of that is partly >> achieved by how you construct your guests. >> >> First thing to do, get rid of network-bridge >> >> In /etc/xen/scripts/ do >> >> echo -e "#!/bin/sh\nexit 0" > network-dummy && chmod +x network-dummy >> >> Then in /etc/xen/ change xend-config.sxp to call network-dummy instead >> of network-bridge. >> >> Secondly (just suggesting) : >> >> Specify vifname= in the vif [] containers in the guests to keep it all >> nice and neat. If you name the vif after the domname (or something else >> specific and meaningful), bandwidth accounting becomes *very* easy over >> an entire farm, which sounds like it would also lend well to your setup. >> >> Best, >> --Tim >> >> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xensource.com >> http://lists.xensource.com/xen-users > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users