Xen devel - May 2007 - bonding + VLANs -> Oops/panic, no VLAN on 100 Mbit cards [SOLVED]

As the issue concerns both the -user (configuration) and -devel (OOPS), 
I''m sending this message to both lists.

Some time ago I complained about two problems:

- VLANs are not working on machines with 100 Mbit cards
- I get a kernel Oops (sometimes panic) when I try to use bonding and 
VLANs with Xen


The root of the issue is the same in both cases: VLAN interfaces must 
*not* be brought up before we start our Xen network scripts.


So have four cases with Xen:

1. Bonding + VLAN leads to an Oops
2. Bonding + VLAN works

3. VLANs don''t work with 100 Mbit cards
4. VLANs work with 100 Mbit cards

I use network-bridge script shipped with Xen 3.1.


1. Bonding + VLAN leads to an Oops:

ifup bond0
ifup bond0.100
ifup bond0.200

./network-bridge start vifnum=0 netdev=bond0 bridge=xenbr0

./network-bridge start vifnum=1 netdev=bond0.100 bridge=xenbr100
# Ooops (and/or panic) here!

./network-bridge start vifnum=2 netdev=bond0.200 bridge=xenbr200


2. Bonding + VLAN works - note the "enslave":

ifup bond0
./network-bridge start vifnum=0 netdev=bond0 bridge=xenbr0

ifenslave pbond0 eth0 eth1

ifup bond0.100
ifup bond0.200

./network-bridge start vifnum=1 netdev=bond0.100 bridge=xenbr100

./network-bridge start vifnum=2 netdev=bond0.200 bridge=xenbr200


3. VLANs don''t work with 100 Mbit cards (with 1 Gbit will work):

ifup eth0
ifup eth0.100
ifup eth0.200

./network-bridge start vifnum=0 netdev=eth0 bridge=xenbr0

./network-bridge start vifnum=1 netdev=eth0.100 bridge=xenbr100

./network-bridge start vifnum=2 netdev=eth0.200 bridge=xenbr200


4. VLANs work with 100 Mbit cards

ifup eth0

./network-bridge start vifnum=0 netdev=eth0 bridge=xenbr0

ifup eth0.100
ifup eth0.200

./network-bridge start vifnum=1 netdev=eth0.100 bridge=xenbr100

./network-bridge start vifnum=2 netdev=eth0.200 bridge=xenbr200



One more note: to use it in startup scripts, I had to put a couple of 
"sleep" commands between each line - otherwise, some peth/pbond 
interfaces were missing sometimes.

-- 
Tomasz Chmielewski
http://wpkg.org

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Tomasz Chmielewski wrote:
> As the issue concerns both the -user (configuration) and -devel
> (OOPS), I''m sending this message to both lists.
>
> Some time ago I complained about two problems:
>
> - VLANs are not working on machines with 100 Mbit cards
> - I get a kernel Oops (sometimes panic) when I try to use bonding and
> VLANs with Xen
>
>
> The root of the issue is the same in both cases: VLAN interfaces must
> *not* be brought up before we start our Xen network scripts.
>
And why not?
I''ve been using vlan interfaces with xen since the days of xen-2.
Then again, I don''t use xend''s network-bridge to create
xenbr0. I use
RedHat''s network config to craete the necessary vlans and assign each
to
their own bridge.
>
> So have four cases with Xen:
>
> 1. Bonding + VLAN leads to an Oops
> 2. Bonding + VLAN works
>
I haven''t been able to get bonding to work reliably though,
I doubt it would work as expected anyway, since the only way to get
bonding and vlan in domU is to detect line status (which would be rather
useless if the network problem happens anywhere other than the
switch).
> 3. VLANs don''t work with 100 Mbit cards
> 4. VLANs work with 100 Mbit cards
>
In my experince the problem comes from network card drivers. Some
boradcom network cards on Dell servers wouldn''t work with xen-3.0.2 (or
to be more specific, the kernel that comes with xen-3.0.2) but works
with xen-3.0.4.
> I use network-bridge script shipped with Xen 3.1.
>
My advice :
- Dont''t use network-bridge script. Change the entry on xend-config.sxp
from network-bridge to /bin/true
- Don''t use bonding in dom0
- It will help A LOT if you have at least two interfaces on dom0: one
for dom0 management, one (or more) for domU''s traffic.
- Create vlans and bridges in dom0 using your distro''s network config.
- If you need bonding and have more than two dom0 interface handling
domU''s traffic, do the bonding in domU.

Regards,

Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Fajar A. Nugraha schrieb:
> Tomasz Chmielewski wrote:
>> As the issue concerns both the -user (configuration) and -devel
>> (OOPS), I''m sending this message to both lists.
>>
>> Some time ago I complained about two problems:
>>
>> - VLANs are not working on machines with 100 Mbit cards
>> - I get a kernel Oops (sometimes panic) when I try to use bonding and
>> VLANs with Xen
>>
>>
>> The root of the issue is the same in both cases: VLAN interfaces must
>> *not* be brought up before we start our Xen network scripts.
>>
> And why not?
Because then, either bonding doesn''t work with VLANs, or VLANs
don''t
work with 100 Mbit cards...

> I''ve been using vlan interfaces with xen since the days of xen-2.
> Then again, I don''t use xend''s network-bridge to create
xenbr0. I use
> RedHat''s network config to craete the necessary vlans and assign
each to
> their own bridge.
> 
>> So have four cases with Xen:
>>
>> 1. Bonding + VLAN leads to an Oops
>> 2. Bonding + VLAN works
>>
> I haven''t been able to get bonding to work reliably though,
> I doubt it would work as expected anyway, since the only way to get
> bonding and vlan in domU is to detect line status (which would be rather
> useless if the network problem happens anywhere other than the switch).
No - I''m setting up bonding in dom0. I doubt setting up bonding within 
domU would make much sense...
But in dom0, it works reliably.

Certainly, the machine shouldn''t Oops/panic as a result of network
setup
- and it''s a bug either in Xen, or in the fact that Xen does with 
networking - which does something that "bonding" driver
doesn''t expect.

>> 3. VLANs don''t work with 100 Mbit cards
>> 4. VLANs work with 100 Mbit cards
>>
> In my experince the problem comes from network card drivers. Some
> boradcom network cards on Dell servers wouldn''t work with
xen-3.0.2 (or
> to be more specific, the kernel that comes with xen-3.0.2) but works
> with xen-3.0.4.
>> I use network-bridge script shipped with Xen 3.1.
>>
> My advice :
> - Dont''t use network-bridge script. Change the entry on
xend-config.sxp
> from network-bridge to /bin/true
> - Don''t use bonding in dom0
Why not? It''s needed for failover.

> - It will help A LOT if you have at least two interfaces on dom0: one
> for dom0 management, one (or more) for domU''s traffic.
True, but that''s not what bonding was invented for.

> - Create vlans and bridges in dom0 using your distro''s network
config.
> - If you need bonding and have more than two dom0 interface handling
> domU''s traffic, do the bonding in domU.
Doesn''t make much sense to me to do bonding in domU.
What if you want to live migrate the guest to a host which doesn''t have
two cards?

Also you wrote it earlier ("I doubt it would work as expected anyway, 
since the only way to get bonding and vlan in domU is to detect line 
status (which would be rather useless if the network problem happens 
anywhere other than the switch)") - so I''m not sure what you mean.


-- 
Tomasz Chmielewski
http://wpkg.org

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Tomasz Chmielewski wrote:
> Fajar A. Nugraha schrieb:
>> - Create vlans and bridges in dom0 using your distro''s network
config.
>> - If you need bonding and have more than two dom0 interface handling
>> domU''s traffic, do the bonding in domU.
>
> Doesn''t make much sense to me to do bonding in domU.
> What if you want to live migrate the guest to a host which doesn''t
> have two cards?
>
> Also you wrote it earlier ("I doubt it would work as expected anyway,
> since the only way to get bonding and vlan in domU is to detect line
> status (which would be rather useless if the network problem happens
> anywhere other than the switch)") - so I''m not sure what you
mean.
>
>
I believe it all comes down to your this line :
I doubt it would work as expected anyway, since the only way to get
bonding and vlan in *dom0* is to detect line status (which would be rather
useless if the network problem happens anywhere other than the switch)

I made a typo earlier, it should be dom0, not domU. The reasoning behind
that statement is :
1. there are several method that linux ethernet bonding use for link
monitoring, ARP monitor and the MII monitor
2. The best HA setup (IMHO) would be to use ARP monitor, and use router
IP as arp target.
3. Using vlans means multiple networks and multiple routers are involved.
4. Checking only one router ARP (the router on native-vlan) isn''t
really
good enough (for me anyway) since it doesn''t check the conditions of
routers on other VLANs
5. MII monitor only check whether the link (the switch or hub that your
eth is connected to) is up. It doesn''t really check whether the switch
connection to the router is working properly or not.

Which is why, IMHO, the best way to do bonding is in domU, and use dom0
for VLAN and bridge.
If you ever get a solution that would RELIABLY do bonding (as in capable
of checking each VLAN''s router ARP) in dom0, I''d love to hear
about it :)

Regards,

Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Fajar A. Nugraha schrieb:

(...)
>> Also you wrote it earlier ("I doubt it would work as expected
anyway,
>> since the only way to get bonding and vlan in domU is to detect line
>> status (which would be rather useless if the network problem happens
>> anywhere other than the switch)") - so I''m not sure what
you mean.
>>
>>
> 
> I believe it all comes down to your this line :
> I doubt it would work as expected anyway, since the only way to get
> bonding and vlan in *dom0* is to detect line status (which would be rather
> useless if the network problem happens anywhere other than the switch)
> 
> I made a typo earlier, it should be dom0, not domU. The reasoning behind
> that statement is :
> 1. there are several method that linux ethernet bonding use for link
> monitoring, ARP monitor and the MII monitor
> 2. The best HA setup (IMHO) would be to use ARP monitor, and use router
> IP as arp target.
> 3. Using vlans means multiple networks and multiple routers are involved.
> 4. Checking only one router ARP (the router on native-vlan) isn''t
really
> good enough (for me anyway) since it doesn''t check the conditions
of
> routers on other VLANs
> 5. MII monitor only check whether the link (the switch or hub that your
> eth is connected to) is up. It doesn''t really check whether the
switch
> connection to the router is working properly or not.
> 
> Which is why, IMHO, the best way to do bonding is in domU, and use dom0
> for VLAN and bridge.
> If you ever get a solution that would RELIABLY do bonding (as in capable
> of checking each VLAN''s router ARP) in dom0, I''d love to
hear about it :)
Well, with my current setup, bonding works on dom0.
As it appears,only "sort of" - I have two issues at least:

1) I have about 1-2% packet losses - without bonding, I don''t have such
losses at all

2) when I try to do "arping <some_host>" from a Xen host, that
host
completely looses network connectivity (domains using VLANs don''t loose
the connectivity, though). A workaround is to use "arping -i bond0 
<some_host>" - which is pretty strange, as hosts without Xen
don''t loose
network connectivity in such case (yes, I understand that arping 
defaults to eth0 - but nevertheless, the host shouldn''t loose network 
connectivity because of that).



Somehow, I don''t like the way Xen messes with networking - as you said,
I agree it''s not that reliable :(


-- 
Tomasz Chmielewski
http://wpkg.org

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Tomasz Chmielewski wrote:
>
> Well, with my current setup, bonding works on dom0.
> As it appears,only "sort of" - I have two issues at least:
>
> 1) I have about 1-2% packet losses - without bonding, I don''t have
> such losses at all
>
> 2) when I try to do "arping <some_host>" from a Xen host,
that host
> completely looses network connectivity (domains using VLANs don''t
> loose the connectivity, though). A workaround is to use "arping -i
> bond0 <some_host>" - which is pretty strange, as hosts without
Xen
> don''t loose network connectivity in such case (yes, I understand
that
> arping defaults to eth0 - but nevertheless, the host shouldn''t
loose
> network connectivity because of that).
>
Have you tried bonding in domU instead?
VLANs on Xen dom0 is reliable enough (provided you create the VLANs on
interfaces that xend doesn''t touch, or disable network-bridge script in
xend-config.sxp). Regarding "live migrate the guest to a host which
doesn''t have two cards", you can simply create an extra bridge on
that
host which has no real physical interface attached, and bonding''s ARP
monitor (not the MII monitor) should properly detect that network
connectivity on that interface is bad.

Regards,

Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users