Hi everybody! I am currently trying to use VLAN within a domU but unfortunately it does not want to work and I do not understand at all why as I have followed many how-to and read 2 books on the subject... and it has been 2 days I am working on the problem... My configuration is as follow: OS is CentOS 5.2 with Xen3 and all the domU I want to create are running CentOS. All my hosts are connected to a switch with trunk mode on the port. So I need to use tagged-vlan So here is what I have done so far: I managed to have a VLAN running on dom0 and they work well, I can ping other hosts in the same VLAN. But when I am trying to ping other hosts from my domU, it just does not work (From 10.10.2.12 icmp_seq=2 Destination Host Unreachable) Here are my configuration files: [On dom0] /etc/sysconfig/network-scripts/ifcfg-eth6: DEVICE=eth6 ONBOOT=yes BOOTPROTO=static HWADDR=00:1b:21:29:07:60 USERCTL=no TYPE=Ethernet /etc/sysconfig/network-scripts/ifcfg-vlan2: VLAN=yes VLAN_NAME_TYPE=VLAN_PLUS_VID_NO_PAD DEVICE=vlan2 PHYSDEV=eth6 BOOTPROTO=static ONBOOT=yes TYPE=Ethernet IPADDR=10.10.2.10 NETMASK=255.255.255.0 /etc/xen/xend-config.sxp (network-script network-multibridge) /etc/xen/scripts/network-multibridge (This is the interesting bit): script=/etc/xen/scripts/network-bridge.xen $script start vifnum=7 bridge=xenbrVLAN2 netdev=vlan2 /etc/xen/www (this is my domU configuration file and the networking bits): vif = [ "mac=00:16:3e:3b:c1:3b,bridge=xenbr2","mac=00:16:3e:00:00:01,bridge=xenbrVLAN2"] [On domU] /etc/sysconfig/network-scripts/ifcfg-eth1 DEVICE=eth1 ONBOOT=yes BOOTPROTO=static HWADDR=00:16:3e:00:00:01 USERCTL=no TYPE=Ethernet /etc/sysconfig/network-scripts/ifcfg-vlan2 VLAN=yes VLAN_NAME_TYPE=VLAN_PLUS_VID_NO_PAD DEVICE=vlan2 PHYSDEV=eth1 BOOTPROTO=static ONBOOT=yes TYPE=Ethernet IPADDR=10.10.2.12 NETMASK=255.255.255.0 The thing I have noticed as well is that as soon as xen starts the bridge (xenbrVLAN2) the interface on dom0 (vlan2) is down, but I think it is the normal behaviour, am I right? Any help will be much appreciate as this issue starts to drive me crazy and I do not know where to start looking... so if you spot any problem or that I should do something else, just let me know :-) So thanks a million in advance for any of your help! Cheers Gael _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Thu, Nov 6, 2008 at 3:08 PM, Gael Reignier <gael@lonres.com> wrote:> I managed to have a VLAN running on dom0 and they work well, I can ping > other hosts in the same VLAN. > > But when I am trying to ping other hosts from my domU, it just does not > work (From 10.10.2.12 icmp_seq=2 Destination Host Unreachable)it seems you''re adding the the VLAN device to the bridge; therefore the DomU shouldn''t use another layer of VLAN. all the tagging should be done on Dom0 -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Javier Guerra wrote:> On Thu, Nov 6, 2008 at 3:08 PM, Gael Reignier <gael@lonres.com> wrote: >> I managed to have a VLAN running on dom0 and they work well, I can ping >> other hosts in the same VLAN. >> >> But when I am trying to ping other hosts from my domU, it just does not >> work (From 10.10.2.12 icmp_seq=2 Destination Host Unreachable) > > it seems you''re adding the the VLAN device to the bridge; therefore > the DomU shouldn''t use another layer of VLAN. all the tagging should > be done on Dom0 >Well actually, I tried both: - - I tried to do the tagging on Dom0 then not do any tagging on the DomU - - Then after reading more from a book called ''Xen Virtualization'', I did the tagging on Dom0 and on DomU So maybe I should try not to do any tagging on the Dom0 and do all the tagging in the DomU. What do you reckon? Thanks in advance Cheers Gael - -- Gael Reignier - System and Network Engineer Tel: 020 7924 6796 Fax: 020 7924 6633 Lonres.com Limited. First Floor, 37 Battersea Square, London SW11 3RA. Registered in England and Wales. Number 3945780. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkkUPOgACgkQQCnBufpTeWdemACcChXEEn0jaDA6QwFH0BSa/M2h 8skAnj3req4wfGhrkKO3cV5bsOvnWk+1 =lbQV -----END PGP SIGNATURE----- _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Fri, Nov 7, 2008 at 8:04 AM, Gael Reignier <gael@lonres.com> wrote:> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Javier Guerra wrote: >> On Thu, Nov 6, 2008 at 3:08 PM, Gael Reignier <gael@lonres.com> wrote: >>> I managed to have a VLAN running on dom0 and they work well, I can ping >>> other hosts in the same VLAN. >>> >>> But when I am trying to ping other hosts from my domU, it just does not >>> work (From 10.10.2.12 icmp_seq=2 Destination Host Unreachable) >> >> it seems you''re adding the the VLAN device to the bridge; therefore >> the DomU shouldn''t use another layer of VLAN. all the tagging should >> be done on Dom0 >> > Well actually, I tried both: > > - - I tried to do the tagging on Dom0 then not do any tagging on the DomU > > - - Then after reading more from a book called ''Xen Virtualization'', I > did the tagging on Dom0 and on DomU > > > So maybe I should try not to do any tagging on the Dom0 and do all the > tagging in the DomU. > What do you reckon?as always, it depends: - tagging on Dom0: - have to plan and setup for all VLANs - DomUs don''t care about VLANs, they simply get a NIC directly to the VLAN - tagging on DomU: - Dom0 doesn''t care about new VLANs. - DomUs can use watever VLAN they want - bridges can be picky with MTUs of 1504bytes -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> > Javier Guerra wrote: > > On Thu, Nov 6, 2008 at 3:08 PM, Gael Reignier <gael@lonres.com>wrote:> >> I managed to have a VLAN running on dom0 and they work well, I canping> >> other hosts in the same VLAN. > >> > >> But when I am trying to ping other hosts from my domU, it just doesnot> >> work (From 10.10.2.12 icmp_seq=2 Destination Host Unreachable) > > > > it seems you''re adding the the VLAN device to the bridge; therefore > > the DomU shouldn''t use another layer of VLAN. all the taggingshould> > be done on Dom0 > > > Well actually, I tried both: > > - - I tried to do the tagging on Dom0 then not do any tagging on theDomU> > - - Then after reading more from a book called ''Xen Virtualization'', I > did the tagging on Dom0 and on DomU > > So maybe I should try not to do any tagging on the Dom0 and do all the > tagging in the DomU. > What do you reckon? >I haven''t been following this thread so maybe you already know this, but depending on what you want to do you may need to involve ebtables. Also, your hardware adapter may do vlan offloading for you, which could make things not work. When a packet comes in on an Ethernet interface with a VLAN tag on it, Linux has to decide what to do with it - route it onto the bridge or make it appear on a vlan interface (eg eth0.2). ebtables can force the situation in the way you want... I''m not sure what the default is. Last time I tried, you couldn''t ''split'' a packet so that one copy went to Dom0''s local interface (eg eth0.2) and another copy remained tagged and went onto the bridge. I did find a workaround for this via creative use of Dom0''s vif0.X/vethX interfaces, but it always crashed after a few hours. This was a while back though. Using ebtables you can say ''packets with vlan tag 2 go to eth0.2, packets with vlan tag 3 go to eth0.3, all other tagged packets remain tagged and go onto the bridge''. If the hardware adapter supports 802.1q offload though, when you define local interfaces eth0.2 and eth0.3, the kernel tells the adapter ''we are interested in untagged packets, and packets with tags of 2, or 3, but throw the rest away''. If your intention is to route other tagged packets onto your bridge then you won''t get what you expect. I''m not sure if there is a way to turn this off either. James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
2008/11/6 Gael Reignier <gael@lonres.com>> > Any help will be much appreciate as this issue starts to drive me crazy > and I do not know where to start looking... so if you spot any problem > or that I should do something else, just let me know :-)Before you get crazy, try this script, I spent several weeks to get it working. The first part dynamically create the VLAN IF (eth0.x) and the associated bridge (xenbrx). The second one creates config files to make it up at the next reboot. This is the script (sorry for french comments): --->8--- #!/bin/bash # A. Barthe -- jeu nov 22 14:14:18 CET 2007 # Script de fabrication dynamique d''un VLAN pour un dom0 xen # Parametres: <vlan tag> [device] (device par defaut: eth0) # #DEBUG=echo script=$(basename $0) USAGE="Usage: $script <vlan tag> [device] (device par defaut: eth0)" if [ $# -ne 1 -a $# -ne 2 ]; then echo $USAGE >&2 exit 1 fi #-------------------------------------------------------------------- # FABRICATION DYNAMIQUE DU VLAN ET DU BRIDGE ASSOCIE vtag=$1 dev=${2:-eth0} ifconfig | grep -q p$dev || { echo "p$dev: interface non trouvee" >&2 exit 1 } # Creation du VLAN sur pethx $DEBUG modprobe 8021q $DEBUG vconfig add p$dev $vtag || { echo "Echec a la creation du VLAN" >&2 exit 1 } # Renommage de pethx.y en ethx.y $DEBUG ip link set p$dev.$vtag name $dev.$vtag # Recuperation de la MAC de $dev mac=$(ifconfig $dev | head -1 | awk ''{print $NF}'') # Affectation de la MAC a l''interface just created ip link set $dev.$vtag address $mac # Configuration du VLAN $DEBUG ip link set $dev.$vtag promisc on $DEBUG ip link set $dev.$vtag multicast on $DEBUG ip link set $dev.$vtag arp on # Creation du bridge et ajout du VLAN dessus $DEBUG brctl addbr xenbr$vtag $DEBUG brctl addif xenbr$vtag $dev.$vtag # Demarrage du VLAN et du bridge $DEBUG ifconfig $dev.$vtag up $DEBUG ifconfig xenbr$vtag up #-------------------------------------------------------------------- # FABRICATION DES FICHIERS DE DEMARRAGE POUR QUE CA MARCHE AU REBOOT root="/etc/sysconfig/network-scripts" #root="toto" baseif="ifcfg-" iffile="$root/$baseif$dev" test -f $iffile || { echo "$iffile: fichier non trouve" >&2 exit 1 } # Rajout du flag VLAN dans le fichier de demarrage de l''interface grep -q "VLAN=yes" $iffile || echo "VLAN=yes" >> $iffile # Fabrication du fichier de demarrage du vlan vlanfile="$iffile.$vtag" cat > $vlanfile <<EOF DEVICE=$dev.$vtag BOOTPROTO=none ONBOOT=yes TYPE=Ethernet VLAN=yes BRIDGE=xenbr$vtag EOF # Fabrication du fichier de demarrage du bridge associe brfile="$root/${baseif}xenbr$vtag" cat > $brfile <<EOF DEVICE=xenbr$vtag BOOTPROTO=none ONBOOT=yes TYPE=Bridge EOF # Et voila... echo "Le VLAN $dev.$vtag et le bridge xenbr$vtag ont bien ete crees." --->8--- Hope this helps. Alain.> > > So thanks a million in advance for any of your help! > > Cheers > Gael > > > > > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Thanks a lot for you help, I will try this for our new batch of servers as we did not manage to have VLAN on network bridges we finally decided to manage the VLAN on the switch directly without using trunk ports... But for our next bunch of servers we will most probably implement bonding + VLAN on the xen hosts. After reading your script, it seems to me that I did the same but it did not work. I basically create the VLAN at startup then the xen network scripts add the VLAN interface to the bridge... Well thanks anyway! Gael On 12/11/08 12:49, "Alain Barthe" <ab266061@gmail.com> wrote:> 2008/11/6 Gael Reignier <gael@lonres.com> >> >> Any help will be much appreciate as this issue starts to drive me crazy >> and I do not know where to start looking... so if you spot any problem >> or that I should do something else, just let me know :-) > > Before you get crazy, try this script, I spent several weeks to get it > working. > The first part dynamically create the VLAN IF (eth0.x) and the associated > bridge (xenbrx). The second one creates config files to make it up at the next > reboot. > > This is the script (sorry for french comments): > > --->8--- > #!/bin/bash > # A. Barthe -- jeu nov 22 14:14:18 CET 2007 > # Script de fabrication dynamique d''un VLAN pour un dom0 xen > # Parametres: <vlan tag> [device] (device par defaut: eth0) > # > > #DEBUG=echo > > script=$(basename $0) > USAGE="Usage: $script <vlan tag> [device] (device par defaut: eth0)" > > if [ $# -ne 1 -a $# -ne 2 ]; then > echo $USAGE >&2 > exit 1 > fi > > #-------------------------------------------------------------------- > # FABRICATION DYNAMIQUE DU VLAN ET DU BRIDGE ASSOCIE > > vtag=$1 > dev=${2:-eth0} > > ifconfig | grep -q p$dev || { > echo "p$dev: interface non trouvee" >&2 > exit 1 > } > > # Creation du VLAN sur pethx > $DEBUG modprobe 8021q > $DEBUG vconfig add p$dev $vtag || { > echo "Echec a la creation du VLAN" >&2 > exit 1 > } > > # Renommage de pethx.y en ethx.y > $DEBUG ip link set p$dev.$vtag name $dev.$vtag > > # Recuperation de la MAC de $dev > mac=$(ifconfig $dev | head -1 | awk ''{print $NF}'') > > # Affectation de la MAC a l''interface just created > ip link set $dev.$vtag address $mac > > # Configuration du VLAN > $DEBUG ip link set $dev.$vtag promisc on > $DEBUG ip link set $dev.$vtag multicast on > $DEBUG ip link set $dev.$vtag arp on > > # Creation du bridge et ajout du VLAN dessus > $DEBUG brctl addbr xenbr$vtag > $DEBUG brctl addif xenbr$vtag $dev.$vtag > > # Demarrage du VLAN et du bridge > $DEBUG ifconfig $dev.$vtag up > $DEBUG ifconfig xenbr$vtag up > > #-------------------------------------------------------------------- > # FABRICATION DES FICHIERS DE DEMARRAGE POUR QUE CA MARCHE AU REBOOT > > root="/etc/sysconfig/network-scripts" > #root="toto" > baseif="ifcfg-" > > iffile="$root/$baseif$dev" > test -f $iffile || { > echo "$iffile: fichier non trouve" >&2 > exit 1 > } > > # Rajout du flag VLAN dans le fichier de demarrage de l''interface > grep -q "VLAN=yes" $iffile || echo "VLAN=yes" >> $iffile > > # Fabrication du fichier de demarrage du vlan > vlanfile="$iffile.$vtag" > cat > $vlanfile <<EOF > DEVICE=$dev.$vtag > BOOTPROTO=none > ONBOOT=yes > TYPE=Ethernet > VLAN=yes > BRIDGE=xenbr$vtag > EOF > > # Fabrication du fichier de demarrage du bridge associe > brfile="$root/${baseif}xenbr$vtag" > cat > $brfile <<EOF > DEVICE=xenbr$vtag > BOOTPROTO=none > ONBOOT=yes > TYPE=Bridge > EOF > > # Et voila... > echo "Le VLAN $dev.$vtag et le bridge xenbr$vtag ont bien ete crees." > --->8--- > > Hope this helps. > > Alain. > >> >> >> So thanks a million in advance for any of your help! >> >> Cheers >> Gael >> >> >> >> >> >> >> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xensource.com >> http://lists.xensource.com/xen-users >> >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users