Hi List!
This is my first post on this list so let me say hello to all here!
I''m trying to set up HA xen VMs on RHEL 5.2 with Red Hat Cluster Suite.
XEN version used is original Red Hat packaged xen-3.0.3-64.el5_2.1.
Setup is the following:
eth0 + eth1 ==> bond0 (public interface, connected to two core switches with
one vlan)
eth2 (private interface connected to rack mounted 100Mbit switch for cluster
interconnect)
(this will be eth2 + eth3 ==> bond1 in future)
The xen network is working fine after adding "netdev=bond0" to the
(network-script ...) line
in xend-config.sxp - no other changes were made.
But: Yesterday the network administrator complained about error messages in the
switch logs caused by multiple interfaces having MAC address FE:FF:FF:FF:FF:FF
since
XEN was set up.
The only public interface with a configured IP is bond0 with the MAC of enslaved
eth0
so why is this strange MAC (FE:FF..) seen on the switch? We are also wondering
why
eth0 and eth1 do have this MAC. Is there an issue with bonding and xen bridging?
Output of ifconfig is attached, if you need more information, please let me
know.
I would be pleased if anyone could point me to a right direction - google
didn''t help.
regards
Gerhard
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Hi, I assume your switches expect another protocol. Depending on the mode parameter of the bonding module NICs are getting different or same MAC. Did you encounter the problem without a bridge on bond0? cheers Stephan Gerhard Spiegl schrieb:> Hi List! > > This is my first post on this list so let me say hello to all here! > > I''m trying to set up HA xen VMs on RHEL 5.2 with Red Hat Cluster Suite. > XEN version used is original Red Hat packaged xen-3.0.3-64.el5_2.1. > > Setup is the following: > > eth0 + eth1 ==> bond0 (public interface, connected to two core switches with one vlan) > eth2 (private interface connected to rack mounted 100Mbit switch for cluster interconnect) > (this will be eth2 + eth3 ==> bond1 in future) > > The xen network is working fine after adding "netdev=bond0" to the (network-script ...) line > in xend-config.sxp - no other changes were made. > > But: Yesterday the network administrator complained about error messages in the > switch logs caused by multiple interfaces having MAC address FE:FF:FF:FF:FF:FF since > XEN was set up. > The only public interface with a configured IP is bond0 with the MAC of enslaved eth0 > so why is this strange MAC (FE:FF..) seen on the switch? We are also wondering why > eth0 and eth1 do have this MAC. Is there an issue with bonding and xen bridging? > > Output of ifconfig is attached, if you need more information, please let me know. > > I would be pleased if anyone could point me to a right direction - google didn''t help. > > regards > Gerhard > > > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users-- Stephan Seitz Senior System Administrator *netz-haut* e.K. multimediale kommunikation zweierweg 22 97074 würzburg fon: +49 931 2876247 fax: +49 931 2876248 web: www.netz-haut.de <http://www.netz-haut.de/> registriergericht: amtsgericht würzburg, hra 5054 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, thanks for your answers! The bonding mode used is 1 (active-backup) with eth0 as primary interface: alias bond0 bonding options bond0 mode=1 primary=eth0 updelay=40000 miimon=100 max_bonds=2 It''s a Cisco Catalyst without any specific bonding configuration since active-backup should just work (well, at least I thought so ..) Without the xen bridge there are no errors (according to the network admin). Also before the bridge comes up, bond0, eth0 and eth1 have the same MAC inherited from eth0 (the way it should be). Another mess (but not the actual problem) is if xend is not enabled at boot time but invoked manually via "service xend start", the bridge comes up but attaches (p)eth0 instead of (p)bond0. In fact pbond0 isn''t even created as it should cause of the "netdev=bond0" param. Any ideas? Here are the relevant network-scripts: /etc/sysconfig/network-scripts/ifcfg-eth0: ----------------------------- # DEVICE=eth0 BOOTPROTO=none HWADDRONBOOT=yes MASTER=bond0 SLAVE=yes USERCTL=no TYPE=Ethernet /etc/sysconfig/network-scripts/ifcfg-eth1: ----------------------------- # DEVICE=eth1 BOOTPROTO=none HWADDRONBOOT=yes MASTER=bond0 SLAVE=yes USERCTL=no TYPE=Ethernet /etc/sysconfig/network-scripts/ifcfg-bond0: ----------------------------- DEVICE=bond0 BOOTPROTO=none HOSTNAME=ols011 IPADDR=172.27.100.121 NETMASK=255.255.240.0 GATEWAY=172.27.111.254 ONBOOT=yes USERCTL=no PEERDNS=no IPV6INIT=no TYPE=Ethernet Stephan Seitz wrote:> Hi, > > I assume your switches expect another protocol. Depending on the > mode parameter of the bonding module NICs are getting different or > same MAC. > > Did you encounter the problem without a bridge on bond0? > > cheers > > Stephan > > > Gerhard Spiegl schrieb: >> Hi List! >> >> This is my first post on this list so let me say hello to all here! >> >> I''m trying to set up HA xen VMs on RHEL 5.2 with Red Hat Cluster Suite. >> XEN version used is original Red Hat packaged xen-3.0.3-64.el5_2.1. >> >> Setup is the following: >> >> eth0 + eth1 ==> bond0 (public interface, connected to two core >> switches with one vlan) >> eth2 (private interface connected to rack mounted 100Mbit switch for >> cluster interconnect) >> (this will be eth2 + eth3 ==> bond1 in future) >> >> The xen network is working fine after adding "netdev=bond0" to the >> (network-script ...) line >> in xend-config.sxp - no other changes were made. >> >> But: Yesterday the network administrator complained about error >> messages in the >> switch logs caused by multiple interfaces having MAC address >> FE:FF:FF:FF:FF:FF since >> XEN was set up. >> The only public interface with a configured IP is bond0 with the MAC >> of enslaved eth0 >> so why is this strange MAC (FE:FF..) seen on the switch? We are also >> wondering why >> eth0 and eth1 do have this MAC. Is there an issue with bonding and xen >> bridging? >> >> Output of ifconfig is attached, if you need more information, please >> let me know. >> >> I would be pleased if anyone could point me to a right direction - >> google didn''t help. >> >> regards >> Gerhard >> >> >> >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xensource.com >> http://lists.xensource.com/xen-users > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Gerhard; Unfortunately I have never tried to use mode=1 so I would only be speculating to suggest that maybe the bridge is causing the same mac to be broadcast from both of the two interfaces. You could test further by running tcpdump on the bridge and each interface to try to see what's going on in there... As an alternative configuration that I can talk about from experience, I use 802.3ad with bond mode=4. To use this you will need to get your Cisco engineers involved to configure a port group and to put the physical ports into it on the switch. S/he might be happy to help to resolve your mutual issue. I have used this with both Cisco and Dell Power Connect switches with Xen 3.2.1 (Intel and Broadcom adapters) and so far I have not had any issues. This configuration provides both load balancing and failover using (by default) a source mac hash algorithm. As for the xend network-bridge startup script - I have broken that (replaced it with network-dummy) and just use the native networking in Dom0 to create my own bonds, vlans and bridges. Most of the info I used to come up with the configuration is from this list. Sorry I couldn't be of more help with your specific issue. If you find a solution to mode=1 I would be curious to know the answer... G -----Original Message----- From: Gerhard Spiegl [mailto:gspiegl@gmx.at] Sent: 24 June 2008 1:32 PM To: xen-users@lists.xensource.com Cc: Stephan Seitz; Geoff Wiener Subject: Re: [Xen-users] Bonding & xenbr0 Hi, thanks for your answers! The bonding mode used is 1 (active-backup) with eth0 as primary interface: alias bond0 bonding options bond0 mode=1 primary=eth0 updelay=40000 miimon=100 max_bonds=2 It's a Cisco Catalyst without any specific bonding configuration since active-backup should just work (well, at least I thought so ..) Without the xen bridge there are no errors (according to the network admin). Also before the bridge comes up, bond0, eth0 and eth1 have the same MAC inherited from eth0 (the way it should be). Another mess (but not the actual problem) is if xend is not enabled at boot time but invoked manually via "service xend start", the bridge comes up but attaches (p)eth0 instead of (p)bond0. In fact pbond0 isn't even created as it should cause of the "netdev=bond0" param. Any ideas? Here are the relevant network-scripts: /etc/sysconfig/network-scripts/ifcfg-eth0: ----------------------------- # DEVICE=eth0 BOOTPROTO=none HWADDRONBOOT=yes MASTER=bond0 SLAVE=yes USERCTL=no TYPE=Ethernet /etc/sysconfig/network-scripts/ifcfg-eth1: ----------------------------- # DEVICE=eth1 BOOTPROTO=none HWADDRONBOOT=yes MASTER=bond0 SLAVE=yes USERCTL=no TYPE=Ethernet /etc/sysconfig/network-scripts/ifcfg-bond0: ----------------------------- DEVICE=bond0 BOOTPROTO=none HOSTNAME=ols011 IPADDR=172.27.100.121 NETMASK=255.255.240.0 GATEWAY=172.27.111.254 ONBOOT=yes USERCTL=no PEERDNS=no IPV6INIT=no TYPE=Ethernet Stephan Seitz wrote:> Hi, > > I assume your switches expect another protocol. Depending on the > mode parameter of the bonding module NICs are getting different or > same MAC. > > Did you encounter the problem without a bridge on bond0? > > cheers > > Stephan > > > Gerhard Spiegl schrieb: >> Hi List! >> >> This is my first post on this list so let me say hello to all here! >> >> I'm trying to set up HA xen VMs on RHEL 5.2 with Red Hat Cluster Suite. >> XEN version used is original Red Hat packaged xen-3.0.3-64.el5_2.1. >> >> Setup is the following: >> >> eth0 + eth1 ==> bond0 (public interface, connected to two core >> switches with one vlan) >> eth2 (private interface connected to rack mounted 100Mbit switch for >> cluster interconnect) >> (this will be eth2 + eth3 ==> bond1 in future) >> >> The xen network is working fine after adding "netdev=bond0" to the >> (network-script ...) line >> in xend-config.sxp - no other changes were made. >> >> But: Yesterday the network administrator complained about error >> messages in the >> switch logs caused by multiple interfaces having MAC address >> FE:FF:FF:FF:FF:FF since >> XEN was set up. >> The only public interface with a configured IP is bond0 with the MAC >> of enslaved eth0 >> so why is this strange MAC (FE:FF..) seen on the switch? We are also >> wondering why >> eth0 and eth1 do have this MAC. Is there an issue with bonding and xen >> bridging? >> >> Output of ifconfig is attached, if you need more information, please >> let me know. >> >> I would be pleased if anyone could point me to a right direction - >> google didn't help. >> >> regards >> Gerhard >> >> >> >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xensource.com >> http://lists.xensource.com/xen-users > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Geoff, it''s working now! The bonded interfaces are connected to two different switches so only active backup can work. I''ve tried to let the OS do all the bridging stuff but didn''t get it running. Could you drop a post how you create all the stuff? I followed this instructions: http://henning.schmiedehausen.org/wingnut-diaries/tag/xen Here is described what I''ve done, it was as simple as copying the script and adapt it to our environment: http://et.redhat.com/~jmh/docs/Xen_networking.pdf thanks for your support! cheers Gerhard Geoff Wiener wrote:> Hi Gerhard; > > Unfortunately I have never tried to use mode=1 so I would only be speculating to suggest that maybe the bridge is causing the same mac to be broadcast from both of the two interfaces. You could test further by running tcpdump on the bridge and each interface to try to see what''s going on in there... As an alternative configuration that I can talk about from experience, I use 802.3ad with bond mode=4. To use this you will need to get your Cisco engineers involved to configure a port group and to put the physical ports into it on the switch. S/he might be happy to help to resolve your mutual issue. I have used this with both Cisco and Dell Power Connect switches with Xen 3.2.1 (Intel and Broadcom adapters) and so far I have not had any issues. This configuration provides both load balancing and failover using (by default) a source mac hash algorithm. > > As for the xend network-bridge startup script - I have broken that (replaced it with network-dummy) and just use the native networking in Dom0 to create my own bonds, vlans and bridges. Most of the info I used to come up with the configuration is from this list. > > Sorry I couldn''t be of more help with your specific issue. If you find a solution to mode=1 I would be curious to know the answer... > > G > > -----Original Message----- > From: Gerhard Spiegl [mailto:gspiegl@gmx.at] > Sent: 24 June 2008 1:32 PM > To: xen-users@lists.xensource.com > Cc: Stephan Seitz; Geoff Wiener > Subject: Re: [Xen-users] Bonding & xenbr0 > > Hi, > > thanks for your answers! > The bonding mode used is 1 (active-backup) with eth0 as primary interface: > > alias bond0 bonding > options bond0 mode=1 primary=eth0 updelay=40000 miimon=100 max_bonds=2 > > It''s a Cisco Catalyst without any specific bonding configuration since > active-backup should just work (well, at least I thought so ..) > > Without the xen bridge there are no errors (according to the network > admin). Also before the bridge comes up, bond0, eth0 and eth1 have the > same MAC inherited from eth0 (the way it should be). > > Another mess (but not the actual problem) is if xend is not enabled at > boot time but invoked manually via "service xend start", the bridge > comes up but attaches (p)eth0 instead of (p)bond0. In fact pbond0 isn''t > even created as it should cause of the "netdev=bond0" param. > > Any ideas? > > Here are the relevant network-scripts: > > /etc/sysconfig/network-scripts/ifcfg-eth0: > ----------------------------- > # > DEVICE=eth0 > BOOTPROTO=none > HWADDR> ONBOOT=yes > MASTER=bond0 > SLAVE=yes > USERCTL=no > TYPE=Ethernet > > /etc/sysconfig/network-scripts/ifcfg-eth1: > ----------------------------- > # > DEVICE=eth1 > BOOTPROTO=none > HWADDR> ONBOOT=yes > MASTER=bond0 > SLAVE=yes > USERCTL=no > TYPE=Ethernet > > /etc/sysconfig/network-scripts/ifcfg-bond0: > ----------------------------- > DEVICE=bond0 > BOOTPROTO=none > HOSTNAME=ols011 > IPADDR=172.27.100.121 > NETMASK=255.255.240.0 > GATEWAY=172.27.111.254 > ONBOOT=yes > USERCTL=no > PEERDNS=no > IPV6INIT=no > TYPE=Ethernet > > > > > Stephan Seitz wrote: >> Hi, >> >> I assume your switches expect another protocol. Depending on the >> mode parameter of the bonding module NICs are getting different or >> same MAC. >> >> Did you encounter the problem without a bridge on bond0? >> >> cheers >> >> Stephan >> >> >> Gerhard Spiegl schrieb: >>> Hi List! >>> >>> This is my first post on this list so let me say hello to all here! >>> >>> I''m trying to set up HA xen VMs on RHEL 5.2 with Red Hat Cluster Suite. >>> XEN version used is original Red Hat packaged xen-3.0.3-64.el5_2.1. >>> >>> Setup is the following: >>> >>> eth0 + eth1 ==> bond0 (public interface, connected to two core >>> switches with one vlan) >>> eth2 (private interface connected to rack mounted 100Mbit switch for >>> cluster interconnect) >>> (this will be eth2 + eth3 ==> bond1 in future) >>> >>> The xen network is working fine after adding "netdev=bond0" to the >>> (network-script ...) line >>> in xend-config.sxp - no other changes were made. >>> >>> But: Yesterday the network administrator complained about error >>> messages in the >>> switch logs caused by multiple interfaces having MAC address >>> FE:FF:FF:FF:FF:FF since >>> XEN was set up. >>> The only public interface with a configured IP is bond0 with the MAC >>> of enslaved eth0 >>> so why is this strange MAC (FE:FF..) seen on the switch? We are also >>> wondering why >>> eth0 and eth1 do have this MAC. Is there an issue with bonding and xen >>> bridging? >>> >>> Output of ifconfig is attached, if you need more information, please >>> let me know. >>> >>> I would be pleased if anyone could point me to a right direction - >>> google didn''t help. >>> >>> regards >>> Gerhard >>> >>> >>> >>> >>> >>> ------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> Xen-users mailing list >>> Xen-users@lists.xensource.com >>> http://lists.xensource.com/xen-users >> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xensource.com >> http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users