Tomasz Chmielewski
2007-May-24 13:23 UTC
[Xen-devel] bonding + VLANs -> Oops/panic, no VLAN on 100 Mbit cards [SOLVED]
As the issue concerns both the -user (configuration) and -devel (OOPS), I''m sending this message to both lists. Some time ago I complained about two problems: - VLANs are not working on machines with 100 Mbit cards - I get a kernel Oops (sometimes panic) when I try to use bonding and VLANs with Xen The root of the issue is the same in both cases: VLAN interfaces must *not* be brought up before we start our Xen network scripts. So have four cases with Xen: 1. Bonding + VLAN leads to an Oops 2. Bonding + VLAN works 3. VLANs don''t work with 100 Mbit cards 4. VLANs work with 100 Mbit cards I use network-bridge script shipped with Xen 3.1. 1. Bonding + VLAN leads to an Oops: ifup bond0 ifup bond0.100 ifup bond0.200 ./network-bridge start vifnum=0 netdev=bond0 bridge=xenbr0 ./network-bridge start vifnum=1 netdev=bond0.100 bridge=xenbr100 # Ooops (and/or panic) here! ./network-bridge start vifnum=2 netdev=bond0.200 bridge=xenbr200 2. Bonding + VLAN works - note the "enslave": ifup bond0 ./network-bridge start vifnum=0 netdev=bond0 bridge=xenbr0 ifenslave pbond0 eth0 eth1 ifup bond0.100 ifup bond0.200 ./network-bridge start vifnum=1 netdev=bond0.100 bridge=xenbr100 ./network-bridge start vifnum=2 netdev=bond0.200 bridge=xenbr200 3. VLANs don''t work with 100 Mbit cards (with 1 Gbit will work): ifup eth0 ifup eth0.100 ifup eth0.200 ./network-bridge start vifnum=0 netdev=eth0 bridge=xenbr0 ./network-bridge start vifnum=1 netdev=eth0.100 bridge=xenbr100 ./network-bridge start vifnum=2 netdev=eth0.200 bridge=xenbr200 4. VLANs work with 100 Mbit cards ifup eth0 ./network-bridge start vifnum=0 netdev=eth0 bridge=xenbr0 ifup eth0.100 ifup eth0.200 ./network-bridge start vifnum=1 netdev=eth0.100 bridge=xenbr100 ./network-bridge start vifnum=2 netdev=eth0.200 bridge=xenbr200 One more note: to use it in startup scripts, I had to put a couple of "sleep" commands between each line - otherwise, some peth/pbond interfaces were missing sometimes. -- Tomasz Chmielewski http://wpkg.org _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Fajar A. Nugraha
2007-May-25 02:40 UTC
Re: [Xen-users] bonding + VLANs -> Oops/panic, no VLAN on 100 Mbit cards [SOLVED]
Tomasz Chmielewski wrote:> As the issue concerns both the -user (configuration) and -devel > (OOPS), I''m sending this message to both lists. > > Some time ago I complained about two problems: > > - VLANs are not working on machines with 100 Mbit cards > - I get a kernel Oops (sometimes panic) when I try to use bonding and > VLANs with Xen > > > The root of the issue is the same in both cases: VLAN interfaces must > *not* be brought up before we start our Xen network scripts. >And why not? I''ve been using vlan interfaces with xen since the days of xen-2. Then again, I don''t use xend''s network-bridge to create xenbr0. I use RedHat''s network config to craete the necessary vlans and assign each to their own bridge.> > So have four cases with Xen: > > 1. Bonding + VLAN leads to an Oops > 2. Bonding + VLAN works >I haven''t been able to get bonding to work reliably though, I doubt it would work as expected anyway, since the only way to get bonding and vlan in domU is to detect line status (which would be rather useless if the network problem happens anywhere other than the switch).> 3. VLANs don''t work with 100 Mbit cards > 4. VLANs work with 100 Mbit cards >In my experince the problem comes from network card drivers. Some boradcom network cards on Dell servers wouldn''t work with xen-3.0.2 (or to be more specific, the kernel that comes with xen-3.0.2) but works with xen-3.0.4.> I use network-bridge script shipped with Xen 3.1. >My advice : - Dont''t use network-bridge script. Change the entry on xend-config.sxp from network-bridge to /bin/true - Don''t use bonding in dom0 - It will help A LOT if you have at least two interfaces on dom0: one for dom0 management, one (or more) for domU''s traffic. - Create vlans and bridges in dom0 using your distro''s network config. - If you need bonding and have more than two dom0 interface handling domU''s traffic, do the bonding in domU. Regards, Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Tomasz Chmielewski
2007-May-25 07:16 UTC
Re: [Xen-users] bonding + VLANs -> Oops/panic, no VLAN on 100 Mbit cards [SOLVED]
Fajar A. Nugraha schrieb:> Tomasz Chmielewski wrote: >> As the issue concerns both the -user (configuration) and -devel >> (OOPS), I''m sending this message to both lists. >> >> Some time ago I complained about two problems: >> >> - VLANs are not working on machines with 100 Mbit cards >> - I get a kernel Oops (sometimes panic) when I try to use bonding and >> VLANs with Xen >> >> >> The root of the issue is the same in both cases: VLAN interfaces must >> *not* be brought up before we start our Xen network scripts. >> > And why not?Because then, either bonding doesn''t work with VLANs, or VLANs don''t work with 100 Mbit cards...> I''ve been using vlan interfaces with xen since the days of xen-2. > Then again, I don''t use xend''s network-bridge to create xenbr0. I use > RedHat''s network config to craete the necessary vlans and assign each to > their own bridge. > >> So have four cases with Xen: >> >> 1. Bonding + VLAN leads to an Oops >> 2. Bonding + VLAN works >> > I haven''t been able to get bonding to work reliably though, > I doubt it would work as expected anyway, since the only way to get > bonding and vlan in domU is to detect line status (which would be rather > useless if the network problem happens anywhere other than the switch).No - I''m setting up bonding in dom0. I doubt setting up bonding within domU would make much sense... But in dom0, it works reliably. Certainly, the machine shouldn''t Oops/panic as a result of network setup - and it''s a bug either in Xen, or in the fact that Xen does with networking - which does something that "bonding" driver doesn''t expect.>> 3. VLANs don''t work with 100 Mbit cards >> 4. VLANs work with 100 Mbit cards >> > In my experince the problem comes from network card drivers. Some > boradcom network cards on Dell servers wouldn''t work with xen-3.0.2 (or > to be more specific, the kernel that comes with xen-3.0.2) but works > with xen-3.0.4. >> I use network-bridge script shipped with Xen 3.1. >> > My advice : > - Dont''t use network-bridge script. Change the entry on xend-config.sxp > from network-bridge to /bin/true > - Don''t use bonding in dom0Why not? It''s needed for failover.> - It will help A LOT if you have at least two interfaces on dom0: one > for dom0 management, one (or more) for domU''s traffic.True, but that''s not what bonding was invented for.> - Create vlans and bridges in dom0 using your distro''s network config. > - If you need bonding and have more than two dom0 interface handling > domU''s traffic, do the bonding in domU.Doesn''t make much sense to me to do bonding in domU. What if you want to live migrate the guest to a host which doesn''t have two cards? Also you wrote it earlier ("I doubt it would work as expected anyway, since the only way to get bonding and vlan in domU is to detect line status (which would be rather useless if the network problem happens anywhere other than the switch)") - so I''m not sure what you mean. -- Tomasz Chmielewski http://wpkg.org _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Fajar A. Nugraha
2007-May-25 07:46 UTC
Re: [Xen-users] bonding + VLANs -> Oops/panic, no VLAN on 100 Mbit cards [SOLVED]
Tomasz Chmielewski wrote:> Fajar A. Nugraha schrieb: >> - Create vlans and bridges in dom0 using your distro''s network config. >> - If you need bonding and have more than two dom0 interface handling >> domU''s traffic, do the bonding in domU. > > Doesn''t make much sense to me to do bonding in domU. > What if you want to live migrate the guest to a host which doesn''t > have two cards? > > Also you wrote it earlier ("I doubt it would work as expected anyway, > since the only way to get bonding and vlan in domU is to detect line > status (which would be rather useless if the network problem happens > anywhere other than the switch)") - so I''m not sure what you mean. > >I believe it all comes down to your this line : I doubt it would work as expected anyway, since the only way to get bonding and vlan in *dom0* is to detect line status (which would be rather useless if the network problem happens anywhere other than the switch) I made a typo earlier, it should be dom0, not domU. The reasoning behind that statement is : 1. there are several method that linux ethernet bonding use for link monitoring, ARP monitor and the MII monitor 2. The best HA setup (IMHO) would be to use ARP monitor, and use router IP as arp target. 3. Using vlans means multiple networks and multiple routers are involved. 4. Checking only one router ARP (the router on native-vlan) isn''t really good enough (for me anyway) since it doesn''t check the conditions of routers on other VLANs 5. MII monitor only check whether the link (the switch or hub that your eth is connected to) is up. It doesn''t really check whether the switch connection to the router is working properly or not. Which is why, IMHO, the best way to do bonding is in domU, and use dom0 for VLAN and bridge. If you ever get a solution that would RELIABLY do bonding (as in capable of checking each VLAN''s router ARP) in dom0, I''d love to hear about it :) Regards, Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Tomasz Chmielewski
2007-May-25 12:02 UTC
Re: [Xen-users] bonding + VLANs -> Oops/panic, no VLAN on 100 Mbit cards [SOLVED]
Fajar A. Nugraha schrieb: (...)>> Also you wrote it earlier ("I doubt it would work as expected anyway, >> since the only way to get bonding and vlan in domU is to detect line >> status (which would be rather useless if the network problem happens >> anywhere other than the switch)") - so I''m not sure what you mean. >> >> > > I believe it all comes down to your this line : > I doubt it would work as expected anyway, since the only way to get > bonding and vlan in *dom0* is to detect line status (which would be rather > useless if the network problem happens anywhere other than the switch) > > I made a typo earlier, it should be dom0, not domU. The reasoning behind > that statement is : > 1. there are several method that linux ethernet bonding use for link > monitoring, ARP monitor and the MII monitor > 2. The best HA setup (IMHO) would be to use ARP monitor, and use router > IP as arp target. > 3. Using vlans means multiple networks and multiple routers are involved. > 4. Checking only one router ARP (the router on native-vlan) isn''t really > good enough (for me anyway) since it doesn''t check the conditions of > routers on other VLANs > 5. MII monitor only check whether the link (the switch or hub that your > eth is connected to) is up. It doesn''t really check whether the switch > connection to the router is working properly or not. > > Which is why, IMHO, the best way to do bonding is in domU, and use dom0 > for VLAN and bridge. > If you ever get a solution that would RELIABLY do bonding (as in capable > of checking each VLAN''s router ARP) in dom0, I''d love to hear about it :)Well, with my current setup, bonding works on dom0. As it appears,only "sort of" - I have two issues at least: 1) I have about 1-2% packet losses - without bonding, I don''t have such losses at all 2) when I try to do "arping <some_host>" from a Xen host, that host completely looses network connectivity (domains using VLANs don''t loose the connectivity, though). A workaround is to use "arping -i bond0 <some_host>" - which is pretty strange, as hosts without Xen don''t loose network connectivity in such case (yes, I understand that arping defaults to eth0 - but nevertheless, the host shouldn''t loose network connectivity because of that). Somehow, I don''t like the way Xen messes with networking - as you said, I agree it''s not that reliable :( -- Tomasz Chmielewski http://wpkg.org _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Fajar A. Nugraha
2007-May-28 02:22 UTC
Re: [Xen-users] bonding + VLANs -> Oops/panic, no VLAN on 100 Mbit cards [SOLVED]
Tomasz Chmielewski wrote:> > Well, with my current setup, bonding works on dom0. > As it appears,only "sort of" - I have two issues at least: > > 1) I have about 1-2% packet losses - without bonding, I don''t have > such losses at all > > 2) when I try to do "arping <some_host>" from a Xen host, that host > completely looses network connectivity (domains using VLANs don''t > loose the connectivity, though). A workaround is to use "arping -i > bond0 <some_host>" - which is pretty strange, as hosts without Xen > don''t loose network connectivity in such case (yes, I understand that > arping defaults to eth0 - but nevertheless, the host shouldn''t loose > network connectivity because of that). >Have you tried bonding in domU instead? VLANs on Xen dom0 is reliable enough (provided you create the VLANs on interfaces that xend doesn''t touch, or disable network-bridge script in xend-config.sxp). Regarding "live migrate the guest to a host which doesn''t have two cards", you can simply create an extra bridge on that host which has no real physical interface attached, and bonding''s ARP monitor (not the MII monitor) should properly detect that network connectivity on that interface is bad. Regards, Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Reasonably Related Threads
- bonding + VLANs -> Oops/panic, no VLAN on 100 Mbit cards [SOLVED]
- VLANs with Xen - work with 1 Gbit, doesn''t with 100 Mbit cards?
- Freeze with 2.6.32.19 and xen-4.0.1rc5
- [Bridge] eth2.100: received packet with own address as source address
- Xen not applying custom network script on startup