So far this has been the trifecta of evil for me. I have got bond0 to come up when I don''t start xend and it works perfectly. Until I start xend then I get the dreaded: bonding: bond0: Warning: the permanent HWaddr of eth0 is still in use by bond0. Set the HWaddr of eth0 to a different address to avoid conflicts. Nothing to flush. Waiting for pbond0 to negotiate link. can''t add pbond0 to bridge bond0: Invalid argument In my /etc/xen/xend-config.sxp file I have edited the network bridge script to use bond0 instead of eth0. But when xend is started it tries to bridge on eth0, at least that is what I am thinking, instead of bond0. Am I missing something? Is there another file I need to edit so that it comes up on bond0 instead of eth0? Thanks for any help, Jon _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
That would be because both Xen and RHEL/Centos are broken when it comes to bonded bridged interfaces. There are various approaches to making it work. My notes on how I am currently doing it are below. Change interfaces, addresses and BONDING_OPTS for your local requirements. Bonding/Trunking Bonding/Trunking is supported out of the box on RHEL/Centos, but it does not work when combined with a bridge for Xen as of RHEL/Centos 5.3. This has been reported as bug 463014 (https://bugzilla.redhat.com/show_bug.cgi?id=463014) which has a simple patch for /etc/sysconfig/network-scripts/ifup-eth. The standard Xen network-script does not support bonded interfaces at all as it takes the interface down, which breaks the bond. Note that the instructions below use the Red Hat init scripts to setup xenbr0. The Xen script that normally does this is disabled. Apply the patch from https://bugzilla.redhat.com/show_bug.cgi?id=463014 Configure Bond on RHEL/Centos /etc/modprobe.conf alias bond0 bonding /etc/sysconfig/network-srcipt/ifcfg-xenbr0 DEVICE=xenbr0 BOOTPROTO=none ONBOOT=yes TYPE=Bridge NETMASK=255.255.255.0 NETWORK=192.168.42.0 IPADDR=192.168.42.41 /etc/sysconfig/network-srcipt/ifcfg-bond0 DEVICE=bond0 BOOTPROTO=none ONBOOT=yes TYPE=Ethernet BONDING_OPTS=''mode=2 miimon=100 xmit_hash_policy=1'' BRIDGE=xenbr0 /etc/sysconfig/network-srcipt/ifcfg-eth0 DEVICE=eth0 USERCTL=no ONBOOT=yes MASTER=bond0 SLAVE=yes BOOTPROTO=none HWADDR=00:15:17:55:C1:D0 /etc/sysconfig/network-srcipt/ifcfg-eth1 DEVICE=eth1 USERCTL=no ONBOOT=yes MASTER=bond0 SLAVE=yes BOOTPROTO=none HWADDR=00:15:17:55:C1:D1 To configure Xen to use the bond in /etc/xen/xend-config.sxp change the network-script and vif-script lines to the following (network-script ''/bin/true'') (vif-script ''vif-bridge bridge=xenbr0'') Reboot. On Thu, 2009-05-28 at 12:08 -0800, jonr@destar.net wrote:> So far this has been the trifecta of evil for me. I have got bond0 to > come up when I don''t start xend and it works perfectly. Until I start > xend then I get the dreaded: > > > bonding: bond0: Warning: the permanent HWaddr of eth0 is still in use > by bond0. Set the HWaddr of eth0 to a different address to avoid > conflicts. > Nothing to flush. > Waiting for pbond0 to negotiate link. > can''t add pbond0 to bridge bond0: Invalid argument > > > In my /etc/xen/xend-config.sxp file I have edited the network bridge > script to use bond0 instead of eth0. > > But when xend is started it tries to bridge on eth0, at least that is > what I am thinking, instead of bond0. > > Am I missing something? Is there another file I need to edit so that > it comes up on bond0 instead of eth0? > > > Thanks for any help, > > Jon > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> bonding: bond0: Warning: the permanent HWaddr of eth0 is still in use by > bond0. Set the HWaddr of eth0 to a different address to avoid conflicts. > Nothing to flush. > Waiting for pbond0 to negotiate link. > can''t add pbond0W to bridge bond0: Invalid argumentHave you searched on the error? HTH Regards, Serge Fonville _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Fri, May 29, 2009 at 3:55 AM, Mike Brady <mike.brady@devnull.net.nz> wrote:> > That would be because both Xen and RHEL/Centos are broken when it comes > to bonded bridged interfaces.> Apply the patch from https://bugzilla.redhat.com/show_bug.cgi?id=463014Thanks for the info. I have a similar setup, but strangely enough it works without any need for patch. Perhaps its because I''m using bridging on top of vlan on top of bonding. # rpm -q initscripts initscripts-8.45.25-1.el5 # brctl show bridge name bridge id STP enabled interfaces br91 8000.002219899697 no vif3.0 vif1.0 bond0.91 virbr0 8000.000000000000 yes -- Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On May 28, 2009, at 9:16 PM, Fajar A. Nugraha wrote:> On Fri, May 29, 2009 at 3:55 AM, Mike Brady > <mike.brady@devnull.net.nz> wrote: >> >> That would be because both Xen and RHEL/Centos are broken when it >> comes >> to bonded bridged interfaces. > >> Apply the patch from https://bugzilla.redhat.com/show_bug.cgi? >> id=463014 > > Thanks for the info. I have a similar setup, but strangely enough it > works without any need for patch. > Perhaps its because I''m using bridging on top of vlan on top of > bonding. >Mine works as well too, under Red Hat EL5, but I don''t use any config file under sysconfig to create the bridge or any vlan either. This may be why this bug didn''t'' affect me. Instead I have Red Hat only create the bond device from the 2 interfaces, and have Xen create the bridge: /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE=eth0 TYPE=Ethernet USERCTL=no ONBOOT=yes BOOTPROTO=none IPV6INIT=no PEERDNS=no HWADDR=00:1A:64:09:26:F4 MASTER=bond0 SLAVE=yes /etc/sysconfig/network-scripts/ifcfg-eth1 DEVICE=eth1 TYPE=Ethernet USERCTL=no ONBOOT=yes BOOTPROTO=none IPV6INIT=no PEERDNS=no HWADDR=00:1A:64:09:26:F6 MASTER=bond0 SLAVE=yes /etc/sysconfig/network-scripts/ifcfg-bond0 DEVICE=bond0 BOOTPROTO=none ONBOOT=yes USERCTL=no IPV6INIT=no PEERDNS=no IPADDR=10.79.184.5 NETMASK=255.255.255.0 GATEWAY=10.79.184.254 Putting the following options under /etc/modprobe.conf takes care of the rest: alias bond0 bonding options bonding miimon=100 mode=0 Then, Xen creates the bridge to bond0 under /etc/xen/xend.cfg (network-script ''network-bridge netdev=bond0'') and then creates pbond0, and xenbr0 The only problem I''m having with this setup is that I can apparently only attach one guest to xenbr0. If I attach a second, it''s network connectivity is unstable and it''s inaccessible over the network. However, the setup works great for one guest: you can unplug either NIC and the network stays up. There is a network-bridge-bonding script under /etc/xen/scripts that would seem to address everyone''s needs. I tried to use it under xend.cfg: (network-script ''network-bridge-bonding netdev=bond0'') But could never get it to work... I''d like to find out to attach multiple guests to a bonded device, but don''t have an extra multi-NIC system handy to play with. I''d appreciate hearing you all''s views on this. best, ...adam _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Fri, May 29, 2009 at 8:31 PM, Adam Wead <awead@indiana.edu> wrote:> The only problem I''m having with this setup is that I can apparently only > attach one guest to xenbr0. If I attach a second, it''s network connectivity > is unstable and it''s inaccessible over the network. However, the setup > works great for one guest: you can unplug either NIC and the network stays > up.> I''d like to find out to attach multiple guests to a bonded device, but don''t > have an extra multi-NIC system handy to play with. I''d appreciate hearing > you all''s views on this.Using redhat-managed bridge and bonding like above I currently have 5 guests attached to the bridge with no problems. -- Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Fri, 2009-05-29 at 09:31 -0400, Adam Wead wrote:> On May 28, 2009, at 9:16 PM, Fajar A. Nugraha wrote: > > > On Fri, May 29, 2009 at 3:55 AM, Mike Brady > > <mike.brady@devnull.net.nz> wrote: > >> > >> That would be because both Xen and RHEL/Centos are broken when it > >> comes > >> to bonded bridged interfaces. > > > >> Apply the patch from https://bugzilla.redhat.com/show_bug.cgi? > >> id=463014 > > > > Thanks for the info. I have a similar setup, but strangely enough it > > works without any need for patch. > > Perhaps its because I''m using bridging on top of vlan on top of > > bonding. > > > > Mine works as well too, under Red Hat EL5, but I don''t use any config > file under sysconfig to create the bridge or any vlan either. This > may be why this bug didn''t'' affect me. Instead I have Red Hat only > create the bond device from the 2 interfaces, and have Xen create the > bridge: > > /etc/sysconfig/network-scripts/ifcfg-eth0 > DEVICE=eth0 > TYPE=Ethernet > USERCTL=no > ONBOOT=yes > BOOTPROTO=none > IPV6INIT=no > PEERDNS=no > HWADDR=00:1A:64:09:26:F4 > MASTER=bond0 > SLAVE=yes > > /etc/sysconfig/network-scripts/ifcfg-eth1 > DEVICE=eth1 > TYPE=Ethernet > USERCTL=no > ONBOOT=yes > BOOTPROTO=none > IPV6INIT=no > PEERDNS=no > HWADDR=00:1A:64:09:26:F6 > MASTER=bond0 > SLAVE=yes > > /etc/sysconfig/network-scripts/ifcfg-bond0 > DEVICE=bond0 > BOOTPROTO=none > ONBOOT=yes > USERCTL=no > IPV6INIT=no > PEERDNS=no > IPADDR=10.79.184.5 > NETMASK=255.255.255.0 > GATEWAY=10.79.184.254 > > Putting the following options under /etc/modprobe.conf takes care of > the rest: > alias bond0 bonding > options bonding miimon=100 mode=0 > > Then, Xen creates the bridge to bond0 under /etc/xen/xend.cfg > (network-script ''network-bridge netdev=bond0'') > > and then creates pbond0, and xenbr0 > > The only problem I''m having with this setup is that I can apparently > only attach one guest to xenbr0. If I attach a second, it''s network > connectivity is unstable and it''s inaccessible over the network. > However, the setup works great for one guest: you can unplug either > NIC and the network stays up. > > There is a network-bridge-bonding script under /etc/xen/scripts that > would seem to address everyone''s needs. I tried to use it under > xend.cfg: > (network-script ''network-bridge-bonding netdev=bond0'') > > But could never get it to work... > > I''d like to find out to attach multiple guests to a bonded device, > but don''t have an extra multi-NIC system handy to play with. I''d > appreciate hearing you all''s views on this. > > best, > > ...adam > > >Which Xen and therefore network-bridge script are you running? Your configuration will work with Red Hat''s network-bridge script. The default Xen script takes the interface down (in this case bond0) when it is moving the addresses etc around, which breaks the bond. The Red Hat network-bridge script doesn''t take the interface down if is a bond. I was doing things the way you are when I was using the Red Hat Xen packages. When I moved to the gitco packages (which has the standard Xen network-bridge script) I changed to what I have previously indicated. I currently only have 3 DomUs using the bridge, but have had more in the past with no networking issues. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Quoting Mike Brady <mike.brady@devnull.net.nz>:> > That would be because both Xen and RHEL/Centos are broken when it comes > to bonded bridged interfaces. > > There are various approaches to making it work. My notes on how I am > currently doing it are below. Change interfaces, addresses and > BONDING_OPTS for your local requirements. > > Bonding/Trunking > Bonding/Trunking is supported out of the box on RHEL/Centos, but it does > not work when combined with a bridge for Xen as of RHEL/Centos 5.3. This > has been reported as bug 463014 > (https://bugzilla.redhat.com/show_bug.cgi?id=463014) which has a simple > patch for /etc/sysconfig/network-scripts/ifup-eth. > > The standard Xen network-script does not support bonded interfaces at > all as it takes the interface down, which breaks the bond. > > Note that the instructions below use the Red Hat init scripts to setup > xenbr0. The Xen script that normally does this is disabled. > > Apply the patch from https://bugzilla.redhat.com/show_bug.cgi?id=463014 > > Configure Bond on RHEL/Centos > /etc/modprobe.conf > alias bond0 bonding > > /etc/sysconfig/network-srcipt/ifcfg-xenbr0 > DEVICE=xenbr0 > BOOTPROTO=none > ONBOOT=yes > TYPE=Bridge > NETMASK=255.255.255.0 > NETWORK=192.168.42.0 > IPADDR=192.168.42.41 > > /etc/sysconfig/network-srcipt/ifcfg-bond0 > DEVICE=bond0 > BOOTPROTO=none > ONBOOT=yes > TYPE=Ethernet > BONDING_OPTS=''mode=2 miimon=100 xmit_hash_policy=1'' > BRIDGE=xenbr0 > > /etc/sysconfig/network-srcipt/ifcfg-eth0 > DEVICE=eth0 > USERCTL=no > ONBOOT=yes > MASTER=bond0 > SLAVE=yes > BOOTPROTO=none > HWADDR=00:15:17:55:C1:D0 > > /etc/sysconfig/network-srcipt/ifcfg-eth1 > DEVICE=eth1 > USERCTL=no > ONBOOT=yes > MASTER=bond0 > SLAVE=yes > BOOTPROTO=none > HWADDR=00:15:17:55:C1:D1 > > To configure Xen to use the bond in /etc/xen/xend-config.sxp > change the network-script and vif-script lines to the following > > (network-script ''/bin/true'') > (vif-script ''vif-bridge bridge=xenbr0'') > > Reboot. >Thanks Mike! I now have everything coming up nicely. I had looked at that patch before but had been finding different solutions to what seemed like the same problem. Your run down of what needs to happen finally brought everything into focus. I now have one problem left. :) I cannot ping from the dom0 and I am not sure where to look for any problems. I have configured my system as you have shown above, except I put in my values. I have dropped my firewall turned off SELinux Logs show nothing out of the ordinary and am seeing no errors in them, at least none that I notice. I tcpdump on the gateway and I see the arp (who has) coming in and an arp reply coming back out but I only see "Destination Host Unreachable" I have the two ports on the switch bonded as 802.3ad same as my ports on the dom0. Any more help is appreciated, Jon _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Fri, 2009-05-29 at 12:09 -0800, jonr@destar.net wrote:> I now have one problem left. :) > > I cannot ping from the dom0 and I am not sure where to look for any problems. > > I have configured my system as you have shown above, except I put in > my values. > I have dropped my firewall > turned off SELinux > Logs show nothing out of the ordinary and am seeing no errors in them, > at least none that I notice. > > > I tcpdump on the gateway and I see the arp (who has) coming in and an > arp reply coming back out but I only see "Destination Host Unreachable" > > > I have the two ports on the switch bonded as 802.3ad same as my ports > on the dom0. > > Any more help is appreciated, > > Jon > >Maybe the bond isn''t actually functioning? I had problems to begin with figuring out what modes actually worked between Linux and my switch. I suggest taking both the bridge and Xen out of the picture and making sure that the bond is actually working with a normal kernel and no bridge. That is what I did to sort mine out. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Quoting Mike Brady <mike.brady@devnull.net.nz>:> > On Fri, 2009-05-29 at 12:09 -0800, jonr@destar.net wrote: >> I now have one problem left. :) >> >> I cannot ping from the dom0 and I am not sure where to look for any >> problems. >> >> I have configured my system as you have shown above, except I put in >> my values. >> I have dropped my firewall >> turned off SELinux >> Logs show nothing out of the ordinary and am seeing no errors in them, >> at least none that I notice. >> >> >> I tcpdump on the gateway and I see the arp (who has) coming in and an >> arp reply coming back out but I only see "Destination Host Unreachable" >> >> >> I have the two ports on the switch bonded as 802.3ad same as my ports >> on the dom0. >> >> Any more help is appreciated, >> >> Jon >> >> > Maybe the bond isn''t actually functioning? I had problems to begin with > figuring out what modes actually worked between Linux and my switch. > > I suggest taking both the bridge and Xen out of the picture and making > sure that the bond is actually working with a normal kernel and no > bridge. That is what I did to sort mine out. > >Hello Mike, There is a problem when I bond my switch ports and bond my server ports. I am leaning towards the problem being the modes between the two. When I remove the bond on the switch it works when I put it back in place it can''t communicate any longer. At this point I am sure I have everything right and am now dealing with hardware rather than software issue. Thank you for all of the help and for laying it out clearly in your first email to me, that definitely cleared my confusion up. I think my biggest problem came from the bond on the switch and the server. I had built the bond on the server by hand multiple times but could never ping out. So I was chasing the wrong rabbit down the wrong hole. :) Thanks again, Jon _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Sun, 2009-05-31 at 12:13 -0800, jonr@destar.net wrote:> Quoting Mike Brady <mike.brady@devnull.net.nz>: > > > > > On Fri, 2009-05-29 at 12:09 -0800, jonr@destar.net wrote: > >> I now have one problem left. :) > >> > >> I cannot ping from the dom0 and I am not sure where to look for any > >> problems. > >> > >> I have configured my system as you have shown above, except I put in > >> my values. > >> I have dropped my firewall > >> turned off SELinux > >> Logs show nothing out of the ordinary and am seeing no errors in them, > >> at least none that I notice. > >> > >> > >> I tcpdump on the gateway and I see the arp (who has) coming in and an > >> arp reply coming back out but I only see "Destination Host Unreachable" > >> > >> > >> I have the two ports on the switch bonded as 802.3ad same as my ports > >> on the dom0. > >> > >> Any more help is appreciated, > >> > >> Jon > >> > >> > > Maybe the bond isn''t actually functioning? I had problems to begin with > > figuring out what modes actually worked between Linux and my switch. > > > > I suggest taking both the bridge and Xen out of the picture and making > > sure that the bond is actually working with a normal kernel and no > > bridge. That is what I did to sort mine out. > > > > > Hello Mike, > > There is a problem when I bond my switch ports and bond my server > ports. I am leaning towards the problem being the modes between the > two. When I remove the bond on the switch it works when I put it back > in place it can''t communicate any longer. > > At this point I am sure I have everything right and am now dealing > with hardware rather than software issue. > > Thank you for all of the help and for laying it out clearly in your > first email to me, that definitely cleared my confusion up. I think my > biggest problem came from the bond on the switch and the server. I had > built the bond on the server by hand multiple times but could never > ping out. So I was chasing the wrong rabbit down the wrong hole. :) > > > Thanks again, > > Jon >Jon, No problem. If you have the kernel-doc package installed, the Linux side of bonding is detailed in this file file /usr/share/doc/kernel-doc-2.6.26/Documentation/networking/bonding.txt. It is mostly just a mater of matching the mode(s) to what your switch supports. Or there is a copy here http://lxr.linux.no/linux/Documentation/networking/bonding.txt Regards Mike _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users