Xen users - May 2010 - quick question about bonding with vlans and Xen

I am looking to move my current Xen setup to one using all 4 available NICs in
my server via bonding and/or VLAN access. Here is the specs:

Dell Poweredge T710 w/ 4 integrate 1000MB NICs
Cisco 4507R with 48 port gigabit switch module
Xen 64-bit 3.4.2 with Centos 5.4 Dom0 64 bit

I will preface with this: I have never delved into the bonding or vlan access
with linux so bear with me here. What I would like to do is bond all 4 nics and
have all my available VLANs accessible across that bond. I have found where to
make the VLANs accessible via a network script similar to:

DEVICE=eth0.30
BOOTPROTO=STATIC
IPADDR=10.111.32.23
NETMASK=255.255.255.240
VLAN=yes
ONBOOT=yes

But do I have to assign an IP to each "interface"? Would I need to
designate one specific NIC to handle Dom0 and it''s static IP or would
Dom0 still be able to have a static IP with all 4 bonded together with vlans on
them?

Thanks in advance for any and all pointers and help!

-- 
 
Donny B.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Sat, May 22, 2010 at 4:57 AM, Donny Brooks <dbrooks@mdah.state.ms.us>
wrote:
> I am looking to move my current Xen setup to one using all 4 available NICs
in my server via bonding and/or VLAN access. Here is the specs:
>
> Dell Poweredge T710 w/ 4 integrate 1000MB NICs
> Cisco 4507R with 48 port gigabit switch module
> Xen 64-bit 3.4.2 with Centos 5.4 Dom0 64 bit
>
> I will preface with this: I have never delved into the bonding or vlan
access with linux so bear with me here. What I would like to do is bond all 4
nics and have all my available VLANs accessible across that bond. I have found
where to make the VLANs accessible via a network script similar to:
>
> DEVICE=eth0.30
> BOOTPROTO=STATIC
> IPADDR=10.111.32.23
> NETMASK=255.255.255.240
> VLAN=yes
> ONBOOT=yes
I''d actually recomend you :
- disable xen''s network-bridge script altogether for your setup. This
will actually make your network setup easier. Comment-out
network-script line from /etc/xen/xend-config.sxp

- create a bond device. config file example,
/etc/sysconfig/network-scripts/ifcfg-bond0 :
DEVICE=bond0
ONBOOT=yes
BOOTPROTO=none

- choose the correct bonding mode. See
/usr/share/doc/iputils-20020927/README.bonding for bonding modes.
Config file example, /etc/modprobe.d/bond0 :
alias bond0 bonding
options bond0 mode=balance-alb miimon=100

- assign some physical NIC to that bond, as many as you need. Config
file example for one device, /etc/sysconfig/network-scripts/ifcfg-eth2
:
DEVICE=eth2
HWADDR=00:22:64:49:ce:0a
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none

- create a bridge. Example, /etc/sysconfig/network-scripts/ifcfg-br9
DEVICE=br9
TYPE=Bridge
BOOTPROTO=static
ONBOOT=yes

- create vlans on top of the bond, and assign it to the bridge. Config
file example, /etc/sysconfig/network-scripts/ifcfg-bond0.9
DEVICE=bond0.9
VLAN=yes
BRIDGE=br9
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet

- tell domUs to use "br9" as the bridge in their config file.
Something like

vif = [ ''mac=00:16:3E:77:E5:D8, bridge=br9'' ]

>
> But do I have to assign an IP to each "interface"? Would I need
to designate one specific NIC to handle Dom0 and it''s static IP or
would Dom0 still be able to have a static IP with all 4 bonded together with
vlans on them?
If you follow my setup above, dom0 will not have an IP address. Yet.
It will be similar to a L2 switch, where it passes lots of vlans, but
the switch itself does not need to have an IP address in those vlans.

For dom0 management, you can either:
- create a management vlan, and put dom0 IP address on that vlan
interface or the bridge. For example, if vlan9 in the example above is
the management vlan, you can put dom0 IP address settings on
/etc/sysconfig/network-scripts/ifcfg-br9. This setup will be similar
to management IP setting on L2 switch, where the management vlan is
usually vlan1. Or,
- assign a dedicated NIC for dom0 management (eth0, or whatever device
you choose), give it it''s own IP address, and do NOT assign it to the
bond.

-- 
Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Tue, May 25, 2010 at 4:25 AM, Donny Brooks <dbrooks@mdah.state.ms.us>
wrote:
> Ok, so far I have followed up till this point with eth2 and eth3 (third and
fourth ports in the server) leaving eth0 and eth1 alone for now.
... which you can add to the bridge as well later, if you like.
> So when I pass the domu "bridge=br1" (changed it to br1 from 9)
and it is a dhcp host, how does it know where to get it''s IP?
I think you''re still missing the basics. Here''s an analogy to
make
things easier. When using the above setup (bonding + vlan + bridge),
Xen dom0 behaves the same way as an L2 switch with vlans enabled and
multiple bonded ports for uplink trunk.

For simplicity sake lets assume the bridge names is the same as vlan
numbers. I wouldn''t recommend using vlan1 (and thus br1), as vlan1 is
usually the default management vlan/default vlan for untagged traffic.
So lets assume we''ll be using vlan9/br9. eth2 and eth3 (which is the
uplink trunk interface) must be connected to a switch as trunk,
possibly requiring special bonding setup on the switch side as well
(depends on which bonding mode you use). The switch must already have
an existing vlan9, which is connected to an existing network with an
existing DHCP server.

So in that sense, a domU connected to br9 behaves just like another
physical machine connected to the switch directly and assigned vlan9.
It can get IP address from an existing DHCP server on that vlan. If no
DHCP server exists, you have to create one first :D

> I have 18 VLAN''s I need to pass to Xen, VLAN 2-19. Is it that br1
needs only setup with one vlan? So if I did a ifcfg-bond0.2 I would make a br2
and point the domu to that? Just trying to clarify since I think that''s
how I would need to do it.
You''d need 18 vlan interface, from bond0.2 to bond0.19, and 18
bridges, from br2 to br19.
>
>
>> >
>> > But do I have to assign an IP to each "interface"? Would
I need to designate one specific NIC to handle Dom0 and it''s static IP
or would Dom0 still be able to have a static IP with all 4 bonded together with
vlans on them?
Again, think L2 switch. Usually it will only have one management IP
address, no matter how many vlans it has. Let''s assume your dom0
management IP address will be on vlan9, so on
/etc/sysconfig/network-scripts/ifcfg-br9 you can put IP address
settings, and leave all other bridges WITHOUT IP address setting. This
setup has the advantage that dom0 will also benefit from the bonding
setup.

Another approach that you can use is to have a dedicated NIC for dom0
management IP address. Let''s assume eth0. You can connect it to a
different switch. This setup has the advantage that you''d get an
out-of-band management network (think HP''s ILO or Sun''s rsc),
but you
don''t have the benefit of bonding.
> For simplicity sake, I might just leave dom0 on eth0 for now and just use
eth1 thru eth3 for the bond. That way if I foobar something I can still get to
dom0 remotely. Thank you for taking the time to explain this. Now I just need to
figure out the switch portion of the bond :)
In my example above, "mode=balance-alb" means you don''t have
to do
much on the switch side. Just make sure the ports eth1-3 uses are set
as trunk, not access. If you use  "mode=802.3ad" or 4, you''d
need to
setup bonding on the switch side as well. This page has a good
explanation:
http://wiki.oracle.com/page/Cisco+Systems+IOS-based+switches-+interface+bonding+and+trunking

-- 
Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 5/24/2010 11:16 PM, Fajar A. Nugraha wrote:
> On Tue, May 25, 2010 at 4:25 AM, Donny
Brooks<dbrooks@mdah.state.ms.us>  wrote:
>    
>> Ok, so far I have followed up till this point with eth2 and eth3 (third
and fourth ports in the server) leaving eth0 and eth1 alone for now.
>>      
> ... which you can add to the bridge as well later, if you like.
>
>    
>> So when I pass the domu "bridge=br1" (changed it to br1 from
9) and it is a dhcp host, how does it know where to get it''s IP?
>>      
> I think you''re still missing the basics. Here''s an
analogy to make
> things easier. When using the above setup (bonding + vlan + bridge),
> Xen dom0 behaves the same way as an L2 switch with vlans enabled and
> multiple bonded ports for uplink trunk.
>
> For simplicity sake lets assume the bridge names is the same as vlan
> numbers. I wouldn''t recommend using vlan1 (and thus br1), as vlan1
is
> usually the default management vlan/default vlan for untagged traffic.
> So lets assume we''ll be using vlan9/br9. eth2 and eth3 (which is
the
> uplink trunk interface) must be connected to a switch as trunk,
> possibly requiring special bonding setup on the switch side as well
> (depends on which bonding mode you use). The switch must already have
> an existing vlan9, which is connected to an existing network with an
> existing DHCP server.
>
> So in that sense, a domU connected to br9 behaves just like another
> physical machine connected to the switch directly and assigned vlan9.
> It can get IP address from an existing DHCP server on that vlan. If no
> DHCP server exists, you have to create one first :D
>
>
>    
>> I have 18 VLAN''s I need to pass to Xen, VLAN 2-19. Is it that
br1 needs only setup with one vlan? So if I did a ifcfg-bond0.2 I would make a
br2 and point the domu to that? Just trying to clarify since I think
that''s how I would need to do it.
>>      
> You''d need 18 vlan interface, from bond0.2 to bond0.19, and 18
> bridges, from br2 to br19.
>
>    
>>
>>      
>>>> But do I have to assign an IP to each "interface"?
Would I need to designate one specific NIC to handle Dom0 and it''s
static IP or would Dom0 still be able to have a static IP with all 4 bonded
together with vlans on them?
>>>>          
> Again, think L2 switch. Usually it will only have one management IP
> address, no matter how many vlans it has. Let''s assume your dom0
> management IP address will be on vlan9, so on
> /etc/sysconfig/network-scripts/ifcfg-br9 you can put IP address
> settings, and leave all other bridges WITHOUT IP address setting. This
> setup has the advantage that dom0 will also benefit from the bonding
> setup.
>
> Another approach that you can use is to have a dedicated NIC for dom0
> management IP address. Let''s assume eth0. You can connect it to a
> different switch. This setup has the advantage that you''d get an
> out-of-band management network (think HP''s ILO or Sun''s
rsc), but you
> don''t have the benefit of bonding.
>
>    
>> For simplicity sake, I might just leave dom0 on eth0 for now and just
use eth1 thru eth3 for the bond. That way if I foobar something I can still get
to dom0 remotely. Thank you for taking the time to explain this. Now I just need
to figure out the switch portion of the bond :)
>>      
> In my example above, "mode=balance-alb" means you don''t
have to do
> much on the switch side. Just make sure the ports eth1-3 uses are set
> as trunk, not access. If you use  "mode=802.3ad" or 4,
you''d need to
> setup bonding on the switch side as well. This page has a good
> explanation:
http://wiki.oracle.com/page/Cisco+Systems+IOS-based+switches-+interface+bonding+and+trunking
>
>    
Just writing back to let everyone know that I was able to get this 
working. It took me a bit as I was mis-reading what Fajar was saying. 
But his examples were spot on. Below is the exact steps I did to 
accomplish this:

1. On the switch (Cisco 4705R in my case) I simply put the 3 interfaces 
into trunk mode with the following commands:

interface GigabitEthernet6/30
  description to xen bond0 interface 1
  switchport trunk encapsulation dot1q
  switchport mode trunk
!
interface GigabitEthernet6/31
  description to xen bond0 interface 2
  switchport trunk encapsulation dot1q
  switchport mode trunk
!
interface GigabitEthernet6/32
  description to xen bond0 interface 3
  switchport trunk encapsulation dot1q
  switchport mode trunk
!

The encapsulation part was needed on my switch but may not on others. It 
griped about it so I added it.

2. On the Xen dom0 I commented out the "(network-script
network-bridge)"
line in /etc/xen/xend-config.sxp
3. I created the bond device on Xen dom0:

cat /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
ONBOOT=yes
BOOTPROTO=none

4. Created the config file:

cat /etc/modprobe.d/bond0
alias bond0 bonding
options bond0 mode=balance-alb miimon=100

5. Added eth1-eth3 to the bond:

cat /etc/sysconfig/network-scripts/ifcfg-eth1
# Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet
DEVICE=eth1
HWADDR=00:00:00:00:00:c5
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none

cat /etc/sysconfig/network-scripts/ifcfg-eth2
# Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet
DEVICE=eth2
HWADDR=00:00:00:00:00:c7
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none

cat /etc/sysconfig/network-scripts/ifcfg-eth3
# Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet
DEVICE=eth3
HWADDR=00:00:00:00:00:c9
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none

6. Created the bridges br2 through br19 (one for each vlan you want 
accesss to) Just first one for example:

cat /etc/sysconfig/network-scripts/ifcfg-br2
DEVICE=br2
TYPE=Bridge
BOOTPROTO=static
ONBOOT=yes

7. Created vlans on top of the bond, and assign it to the bridge. Again 
bond0.2 through bond0.19 one for each vlan. Just first as example:

cat /etc/sysconfig/network-scripts/ifcfg-bond0.2
DEVICE=bond0.2
VLAN=yes
BRIDGE=br2
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet

8. I then "virsh edit domuname" and changed the "bridge"
line to this:

<interface type=''bridge''>
<mac address=''00:00:00:00:00:ee''/>
<source bridge=''br2''/>

where br2 is the bridge you want that machine to have access to.

In a nutshell I was able to bond 3 gigabit interfaces together (soon to 
be 4) and have all of our vlans pass across that bond. What threw me for 
a loop is I misread Fajar''s part about doing the switch side stuff 
according to the 
http://wiki.oracle.com/page/Cisco+Systems+IOS-based+switches-+interface+bonding+and+trunking
site. You don''t need that unless you are doing the 802.3ad bonding 
(which is a PAIN IN THE REAR!!!). So for simplicity just do the above 
steps and you will be rewarded. Now just to get the gigabit drivers for 
my HVM windows 2003 machine.

Donny B.



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 6/14/2010 1:50 PM, Donny Brooks wrote:
> On 5/24/2010 11:16 PM, Fajar A. Nugraha wrote:
>> On Tue, May 25, 2010 at 4:25 AM, Donny 
>> Brooks<dbrooks@mdah.state.ms.us>  wrote:
>>> Ok, so far I have followed up till this point with eth2 and eth3 
>>> (third and fourth ports in the server) leaving eth0 and eth1 alone 
>>> for now.
>> ... which you can add to the bridge as well later, if you like.
>>
>>> So when I pass the domu "bridge=br1" (changed it to br1
from 9) and
>>> it is a dhcp host, how does it know where to get it''s IP?
>> I think you''re still missing the basics. Here''s an
analogy to make
>> things easier. When using the above setup (bonding + vlan + bridge),
>> Xen dom0 behaves the same way as an L2 switch with vlans enabled and
>> multiple bonded ports for uplink trunk.
>>
>> For simplicity sake lets assume the bridge names is the same as vlan
>> numbers. I wouldn''t recommend using vlan1 (and thus br1), as
vlan1 is
>> usually the default management vlan/default vlan for untagged traffic.
>> So lets assume we''ll be using vlan9/br9. eth2 and eth3 (which
is the
>> uplink trunk interface) must be connected to a switch as trunk,
>> possibly requiring special bonding setup on the switch side as well
>> (depends on which bonding mode you use). The switch must already have
>> an existing vlan9, which is connected to an existing network with an
>> existing DHCP server.
>>
>> So in that sense, a domU connected to br9 behaves just like another
>> physical machine connected to the switch directly and assigned vlan9.
>> It can get IP address from an existing DHCP server on that vlan. If no
>> DHCP server exists, you have to create one first :D
>>
>>
>>> I have 18 VLAN''s I need to pass to Xen, VLAN 2-19. Is it
that br1
>>> needs only setup with one vlan? So if I did a ifcfg-bond0.2 I would
>>> make a br2 and point the domu to that? Just trying to clarify since
>>> I think that''s how I would need to do it.
>> You''d need 18 vlan interface, from bond0.2 to bond0.19, and 18
>> bridges, from br2 to br19.
>>
>>>
>>>>> But do I have to assign an IP to each
"interface"? Would I need to
>>>>> designate one specific NIC to handle Dom0 and it''s
static IP or
>>>>> would Dom0 still be able to have a static IP with all 4
bonded
>>>>> together with vlans on them?
>> Again, think L2 switch. Usually it will only have one management IP
>> address, no matter how many vlans it has. Let''s assume your
dom0
>> management IP address will be on vlan9, so on
>> /etc/sysconfig/network-scripts/ifcfg-br9 you can put IP address
>> settings, and leave all other bridges WITHOUT IP address setting. This
>> setup has the advantage that dom0 will also benefit from the bonding
>> setup.
>>
>> Another approach that you can use is to have a dedicated NIC for dom0
>> management IP address. Let''s assume eth0. You can connect it
to a
>> different switch. This setup has the advantage that you''d get
an
>> out-of-band management network (think HP''s ILO or
Sun''s rsc), but you
>> don''t have the benefit of bonding.
>>
>>> For simplicity sake, I might just leave dom0 on eth0 for now and 
>>> just use eth1 thru eth3 for the bond. That way if I foobar
something
>>> I can still get to dom0 remotely. Thank you for taking the time to 
>>> explain this. Now I just need to figure out the switch portion of 
>>> the bond :)
>> In my example above, "mode=balance-alb" means you
don''t have to do
>> much on the switch side. Just make sure the ports eth1-3 uses are set
>> as trunk, not access. If you use  "mode=802.3ad" or 4,
you''d need to
>> setup bonding on the switch side as well. This page has a good
>> explanation: 
>>
http://wiki.oracle.com/page/Cisco+Systems+IOS-based+switches-+interface+bonding+and+trunking
>>
>>
>
> Just writing back to let everyone know that I was able to get this 
> working. It took me a bit as I was mis-reading what Fajar was saying. 
> But his examples were spot on. Below is the exact steps I did to 
> accomplish this:
>
> 1. On the switch (Cisco 4705R in my case) I simply put the 3 
> interfaces into trunk mode with the following commands:
>
> interface GigabitEthernet6/30
>  description to xen bond0 interface 1
>  switchport trunk encapsulation dot1q
>  switchport mode trunk
> !
> interface GigabitEthernet6/31
>  description to xen bond0 interface 2
>  switchport trunk encapsulation dot1q
>  switchport mode trunk
> !
> interface GigabitEthernet6/32
>  description to xen bond0 interface 3
>  switchport trunk encapsulation dot1q
>  switchport mode trunk
> !
>
> The encapsulation part was needed on my switch but may not on others. 
> It griped about it so I added it.
>
> 2. On the Xen dom0 I commented out the "(network-script 
> network-bridge)" line in /etc/xen/xend-config.sxp
> 3. I created the bond device on Xen dom0:
>
> cat /etc/sysconfig/network-scripts/ifcfg-bond0
> DEVICE=bond0
> ONBOOT=yes
> BOOTPROTO=none
>
> 4. Created the config file:
>
> cat /etc/modprobe.d/bond0
> alias bond0 bonding
> options bond0 mode=balance-alb miimon=100
>
> 5. Added eth1-eth3 to the bond:
>
> cat /etc/sysconfig/network-scripts/ifcfg-eth1
> # Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet
> DEVICE=eth1
> HWADDR=00:00:00:00:00:c5
> USERCTL=no
> ONBOOT=yes
> MASTER=bond0
> SLAVE=yes
> BOOTPROTO=none
>
> cat /etc/sysconfig/network-scripts/ifcfg-eth2
> # Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet
> DEVICE=eth2
> HWADDR=00:00:00:00:00:c7
> USERCTL=no
> ONBOOT=yes
> MASTER=bond0
> SLAVE=yes
> BOOTPROTO=none
>
> cat /etc/sysconfig/network-scripts/ifcfg-eth3
> # Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet
> DEVICE=eth3
> HWADDR=00:00:00:00:00:c9
> USERCTL=no
> ONBOOT=yes
> MASTER=bond0
> SLAVE=yes
> BOOTPROTO=none
>
> 6. Created the bridges br2 through br19 (one for each vlan you want 
> accesss to) Just first one for example:
>
> cat /etc/sysconfig/network-scripts/ifcfg-br2
> DEVICE=br2
> TYPE=Bridge
> BOOTPROTO=static
> ONBOOT=yes
>
> 7. Created vlans on top of the bond, and assign it to the bridge. 
> Again bond0.2 through bond0.19 one for each vlan. Just first as example:
>
> cat /etc/sysconfig/network-scripts/ifcfg-bond0.2
> DEVICE=bond0.2
> VLAN=yes
> BRIDGE=br2
> BOOTPROTO=none
> ONBOOT=yes
> TYPE=Ethernet
>
> 8. I then "virsh edit domuname" and changed the
"bridge" line to this:
>
> <interface type=''bridge''>
> <mac address=''00:00:00:00:00:ee''/>
> <source bridge=''br2''/>
>
> where br2 is the bridge you want that machine to have access to.
>
> In a nutshell I was able to bond 3 gigabit interfaces together (soon 
> to be 4) and have all of our vlans pass across that bond. What threw 
> me for a loop is I misread Fajar''s part about doing the switch
side
> stuff according to the 
>
http://wiki.oracle.com/page/Cisco+Systems+IOS-based+switches-+interface+bonding+and+trunking
> site. You don''t need that unless you are doing the 802.3ad bonding
> (which is a PAIN IN THE REAR!!!). So for simplicity just do the above 
> steps and you will be rewarded. Now just to get the gigabit drivers 
> for my HVM windows 2003 machine.
>
> Donny B.
>
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users

And I just THOUGHT I had it working. It seems that no machine on the 
same vlan as my domu can connect to it unless the domu first pings the 
machine trying to connect to it. That and the domu has about 80% packet 
loss! It has to be something I am missing. I did not have this issue 
before on a basic bridge interface. Anyone have any ideas where to even 
begin? Here are the system specs:

Centos 5.5 x86_64
Xen 4.0 from gitco repo

Donny B.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 6/16/2010 8:30 AM, Donny Brooks wrote:
> On 6/14/2010 1:50 PM, Donny Brooks wrote:
>> On 5/24/2010 11:16 PM, Fajar A. Nugraha wrote:
>>> On Tue, May 25, 2010 at 4:25 AM, Donny 
>>> Brooks<dbrooks@mdah.state.ms.us>  wrote:
>>>> Ok, so far I have followed up till this point with eth2 and
eth3
>>>> (third and fourth ports in the server) leaving eth0 and eth1
alone
>>>> for now.
>>> ... which you can add to the bridge as well later, if you like.
>>>
>>>> So when I pass the domu "bridge=br1" (changed it to
br1 from 9) and
>>>> it is a dhcp host, how does it know where to get it''s
IP?
>>> I think you''re still missing the basics. Here''s
an analogy to make
>>> things easier. When using the above setup (bonding + vlan +
bridge),
>>> Xen dom0 behaves the same way as an L2 switch with vlans enabled
and
>>> multiple bonded ports for uplink trunk.
>>>
>>> For simplicity sake lets assume the bridge names is the same as
vlan
>>> numbers. I wouldn''t recommend using vlan1 (and thus br1),
as vlan1 is
>>> usually the default management vlan/default vlan for untagged
traffic.
>>> So lets assume we''ll be using vlan9/br9. eth2 and eth3
(which is the
>>> uplink trunk interface) must be connected to a switch as trunk,
>>> possibly requiring special bonding setup on the switch side as well
>>> (depends on which bonding mode you use). The switch must already
have
>>> an existing vlan9, which is connected to an existing network with
an
>>> existing DHCP server.
>>>
>>> So in that sense, a domU connected to br9 behaves just like another
>>> physical machine connected to the switch directly and assigned
vlan9.
>>> It can get IP address from an existing DHCP server on that vlan. If
no
>>> DHCP server exists, you have to create one first :D
>>>
>>>
>>>> I have 18 VLAN''s I need to pass to Xen, VLAN 2-19. Is
it that br1
>>>> needs only setup with one vlan? So if I did a ifcfg-bond0.2 I
would
>>>> make a br2 and point the domu to that? Just trying to clarify
since
>>>> I think that''s how I would need to do it.
>>> You''d need 18 vlan interface, from bond0.2 to bond0.19,
and 18
>>> bridges, from br2 to br19.
>>>
>>>>
>>>>>> But do I have to assign an IP to each
"interface"? Would I need
>>>>>> to designate one specific NIC to handle Dom0 and
it''s static IP
>>>>>> or would Dom0 still be able to have a static IP with
all 4 bonded
>>>>>> together with vlans on them?
>>> Again, think L2 switch. Usually it will only have one management IP
>>> address, no matter how many vlans it has. Let''s assume
your dom0
>>> management IP address will be on vlan9, so on
>>> /etc/sysconfig/network-scripts/ifcfg-br9 you can put IP address
>>> settings, and leave all other bridges WITHOUT IP address setting.
This
>>> setup has the advantage that dom0 will also benefit from the
bonding
>>> setup.
>>>
>>> Another approach that you can use is to have a dedicated NIC for
dom0
>>> management IP address. Let''s assume eth0. You can connect
it to a
>>> different switch. This setup has the advantage that you''d
get an
>>> out-of-band management network (think HP''s ILO or
Sun''s rsc), but you
>>> don''t have the benefit of bonding.
>>>
>>>> For simplicity sake, I might just leave dom0 on eth0 for now
and
>>>> just use eth1 thru eth3 for the bond. That way if I foobar 
>>>> something I can still get to dom0 remotely. Thank you for
taking
>>>> the time to explain this. Now I just need to figure out the
switch
>>>> portion of the bond :)
>>> In my example above, "mode=balance-alb" means you
don''t have to do
>>> much on the switch side. Just make sure the ports eth1-3 uses are
set
>>> as trunk, not access. If you use  "mode=802.3ad" or 4,
you''d need to
>>> setup bonding on the switch side as well. This page has a good
>>> explanation: 
>>>
http://wiki.oracle.com/page/Cisco+Systems+IOS-based+switches-+interface+bonding+and+trunking
>>>
>>>
>>
>> Just writing back to let everyone know that I was able to get this 
>> working. It took me a bit as I was mis-reading what Fajar was saying. 
>> But his examples were spot on. Below is the exact steps I did to 
>> accomplish this:
>>
>> 1. On the switch (Cisco 4705R in my case) I simply put the 3 
>> interfaces into trunk mode with the following commands:
>>
>> interface GigabitEthernet6/30
>>  description to xen bond0 interface 1
>>  switchport trunk encapsulation dot1q
>>  switchport mode trunk
>> !
>> interface GigabitEthernet6/31
>>  description to xen bond0 interface 2
>>  switchport trunk encapsulation dot1q
>>  switchport mode trunk
>> !
>> interface GigabitEthernet6/32
>>  description to xen bond0 interface 3
>>  switchport trunk encapsulation dot1q
>>  switchport mode trunk
>> !
>>
>> The encapsulation part was needed on my switch but may not on others. 
>> It griped about it so I added it.
>>
>> 2. On the Xen dom0 I commented out the "(network-script 
>> network-bridge)" line in /etc/xen/xend-config.sxp
>> 3. I created the bond device on Xen dom0:
>>
>> cat /etc/sysconfig/network-scripts/ifcfg-bond0
>> DEVICE=bond0
>> ONBOOT=yes
>> BOOTPROTO=none
>>
>> 4. Created the config file:
>>
>> cat /etc/modprobe.d/bond0
>> alias bond0 bonding
>> options bond0 mode=balance-alb miimon=100
>>
>> 5. Added eth1-eth3 to the bond:
>>
>> cat /etc/sysconfig/network-scripts/ifcfg-eth1
>> # Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet
>> DEVICE=eth1
>> HWADDR=00:00:00:00:00:c5
>> USERCTL=no
>> ONBOOT=yes
>> MASTER=bond0
>> SLAVE=yes
>> BOOTPROTO=none
>>
>> cat /etc/sysconfig/network-scripts/ifcfg-eth2
>> # Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet
>> DEVICE=eth2
>> HWADDR=00:00:00:00:00:c7
>> USERCTL=no
>> ONBOOT=yes
>> MASTER=bond0
>> SLAVE=yes
>> BOOTPROTO=none
>>
>> cat /etc/sysconfig/network-scripts/ifcfg-eth3
>> # Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet
>> DEVICE=eth3
>> HWADDR=00:00:00:00:00:c9
>> USERCTL=no
>> ONBOOT=yes
>> MASTER=bond0
>> SLAVE=yes
>> BOOTPROTO=none
>>
>> 6. Created the bridges br2 through br19 (one for each vlan you want 
>> accesss to) Just first one for example:
>>
>> cat /etc/sysconfig/network-scripts/ifcfg-br2
>> DEVICE=br2
>> TYPE=Bridge
>> BOOTPROTO=static
>> ONBOOT=yes
>>
>> 7. Created vlans on top of the bond, and assign it to the bridge. 
>> Again bond0.2 through bond0.19 one for each vlan. Just first as
example:
>>
>> cat /etc/sysconfig/network-scripts/ifcfg-bond0.2
>> DEVICE=bond0.2
>> VLAN=yes
>> BRIDGE=br2
>> BOOTPROTO=none
>> ONBOOT=yes
>> TYPE=Ethernet
>>
>> 8. I then "virsh edit domuname" and changed the
"bridge" line to this:
>>
>> <interface type=''bridge''>
>> <mac address=''00:00:00:00:00:ee''/>
>> <source bridge=''br2''/>
>>
>> where br2 is the bridge you want that machine to have access to.
>>
>> In a nutshell I was able to bond 3 gigabit interfaces together (soon 
>> to be 4) and have all of our vlans pass across that bond. What threw 
>> me for a loop is I misread Fajar''s part about doing the switch
side
>> stuff according to the 
>>
http://wiki.oracle.com/page/Cisco+Systems+IOS-based+switches-+interface+bonding+and+trunking
>> site. You don''t need that unless you are doing the 802.3ad
bonding
>> (which is a PAIN IN THE REAR!!!). So for simplicity just do the above 
>> steps and you will be rewarded. Now just to get the gigabit drivers 
>> for my HVM windows 2003 machine.
>>
>> Donny B.
>>
>>
>>
>> _______________________________________________
>> Xen-users mailing list
>> Xen-users@lists.xensource.com
>> http://lists.xensource.com/xen-users
>
>
> And I just THOUGHT I had it working. It seems that no machine on the 
> same vlan as my domu can connect to it unless the domu first pings the 
> machine trying to connect to it. That and the domu has about 80% 
> packet loss! It has to be something I am missing. I did not have this 
> issue before on a basic bridge interface. Anyone have any ideas where 
> to even begin? Here are the system specs:
>
> Centos 5.5 x86_64
> Xen 4.0 from gitco repo
>
> Donny B.
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
This is most likely not directly related to xen but here goes hoping 
someone has had similar issues that they resolved!

Let me clarify this: Anything NOT on the same vlan has no issues getting 
to the domu. I have tried various domu''s on various vlans and all 
exhibit the same behavior. Anything outside the vlan of the domu can 
connect to it. While anything on the same vlan cannot connect to it 
unless the domu first pings the machine in question. Even with that it 
has about 80% packet loss.

Is anyone else doing a balance-alb style bond0 with various vlans 
passing over it and having success?

Donny B.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Wed, Jun 16, 2010 at 10:14 PM, Donny Brooks <dbrooks@mdah.state.ms.us>
wrote:
>> And I just THOUGHT I had it working. It seems that no machine on the
same
>> vlan as my domu can connect to it unless the domu first pings the
machine
>> trying to connect to it. That and the domu has about 80% packet loss!
It has
>> to be something I am missing.
>> Centos 5.5 x86_64
>> Xen 4.0 from gitco repo
So Xen 4.0 can work with Centos kernel-xen now? Good to know. It
didn''t work the last time I tried.
> Let me clarify this: Anything NOT on the same vlan has no issues getting to
> the domu. I have tried various domu''s on various vlans and all
exhibit the
> same behavior. Anything outside the vlan of the domu can connect to it.
> While anything on the same vlan cannot connect to it unless the domu first
> pings the machine in question. Even with that it has about 80% packet loss.
>
> Is anyone else doing a balance-alb style bond0 with various vlans passing
> over it and having success?
I''d start by troubleshooting the problems one by one.
First of all, the switch side. You MIGHT need spanning-tree portfast there.

Second, bonding mode. You might want to try with simpler modes first
(0,1, or 2), and then simulate links failure (possibly by disabling
the port on switch side) to make sure that at least the basic failover
stuff and switch config is working properly.

If all fails, you might need to actually do 802.3ad.

-- 
Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 6/16/2010 8:39 PM, Fajar A. Nugraha wrote:
> On Wed, Jun 16, 2010 at 10:14 PM, Donny
Brooks<dbrooks@mdah.state.ms.us>  wrote:
>    
>>> And I just THOUGHT I had it working. It seems that no machine on
the same
>>> vlan as my domu can connect to it unless the domu first pings the
machine
>>> trying to connect to it. That and the domu has about 80% packet
loss! It has
>>> to be something I am missing.
>>>        
>    
>>> Centos 5.5 x86_64
>>> Xen 4.0 from gitco repo
>>>        
> So Xen 4.0 can work with Centos kernel-xen now? Good to know. It
> didn''t work the last time I tried.
>
>    
>> Let me clarify this: Anything NOT on the same vlan has no issues
getting to
>> the domu. I have tried various domu''s on various vlans and all
exhibit the
>> same behavior. Anything outside the vlan of the domu can connect to it.
>> While anything on the same vlan cannot connect to it unless the domu
first
>> pings the machine in question. Even with that it has about 80% packet
loss.
>>
>> Is anyone else doing a balance-alb style bond0 with various vlans
passing
>> over it and having success?
>>      
> I''d start by troubleshooting the problems one by one.
> First of all, the switch side. You MIGHT need spanning-tree portfast there.
>
> Second, bonding mode. You might want to try with simpler modes first
> (0,1, or 2), and then simulate links failure (possibly by disabling
> the port on switch side) to make sure that at least the basic failover
> stuff and switch config is working properly.
>
> If all fails, you might need to actually do 802.3ad.
>
>    
I have to be missing something simple. It usually happens that way. I 
have tried the "spanning-tree portfast trunk" to no avail. Also I
tried
mode 0,1,2,4, and 6 with the various configs to no avail. All exhibit 
the same behavior where anything outside the domu''s vlan can access it 
no problem but nothing on the same vlan can.

With all I have tried, I am giving up on the whole "bonding w/ vlans" 
for now. But I would like to at least take advantage of the vlan support 
so I can at least test that. Shouldn''t I be able to do the following to
enable vlan support?:

remove bond0 interface
setup eth1 as vlan master
change any reference to bond0 in the bond0.XX and brXX files to eth1
restart networking
and hopefully it just work?

Currently I have gone back to the xen bridge-script way of doing things 
just to get them working. After much deliberation I have decided it is 
probably best that I only do vlans since my other machines I plan to 
setup as dom0''s only have 2 network cards and I plan to keep one of
them
as a dedicated management interface.

Donny B.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Tue, Jun 22, 2010 at 3:17 AM, Donny Brooks <dbrooks@mdah.state.ms.us>
wrote:
> But I would like to at least take advantage of the vlan support so I
> can at least test that. Shouldn''t I be able to do the following to
enable
> vlan support?:
>
> remove bond0 interface
> setup eth1 as vlan master
> change any reference to bond0 in the bond0.XX and brXX files to eth1
> restart networking
> and hopefully it just work?
You still need the bridge. So you can dump bond0, change all bond0.XX
to eth1.XX, but you still need brXX.

-- 
Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

"Fajar A. Nugraha" <fajar@fajar.net> writes:
> On Tue, Jun 22, 2010 at 3:17 AM, Donny Brooks
<dbrooks@mdah.state.ms.us> wrote:
>> But I would like to at least take advantage of the vlan support so I
>> can at least test that. Shouldn''t I be able to do the
following to enable
>> vlan support?:
>>
>> remove bond0 interface
>> setup eth1 as vlan master
>> change any reference to bond0 in the bond0.XX and brXX files to eth1
>> restart networking
>> and hopefully it just work?
>
> You still need the bridge. So you can dump bond0, change all bond0.XX
> to eth1.XX, but you still need brXX.
Always take care either not to bridge the master interface (bond0 in
this case) or to deny bridging of VLAN tagged packets from it.
Otherwise the bridge with the master interface will eat all your
packets, even the tagged ones, leaving nothing for the bridges with the
tagged interfaces.
-- 
Cheers,
Feri.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 6/21/2010 7:37 PM, Fajar A. Nugraha wrote:
> On Tue, Jun 22, 2010 at 3:17 AM, Donny
Brooks<dbrooks@mdah.state.ms.us>  wrote:
>    
>> But I would like to at least take advantage of the vlan support so I
>> can at least test that. Shouldn''t I be able to do the
following to enable
>> vlan support?:
>>
>> remove bond0 interface
>> setup eth1 as vlan master
>> change any reference to bond0 in the bond0.XX and brXX files to eth1
>> restart networking
>> and hopefully it just work?
>>      
> You still need the bridge. So you can dump bond0, change all bond0.XX
> to eth1.XX, but you still need brXX.
>
>    
Ok, after a little clearing my head and thinking about it I found the 
"simple thing I was missing"... I had to change the MTU setting on my 
servers network cards to allow the vlan header to be passed. I was 
unable to even get the eth2.X interfaces to work so I knew it had to be 
something else. I changed the MTU and it started working on the eth2.X 
devices (with the brXX stuff of course). After changing the MTU to 1496 
on eth1 - eth3 I then commenced testing the various bond modes again. 
Here are the results:

mode=0 No go
mode=1 Works
mode=2 No go
mode=3 Didn''t test
mode=4 Works
mode=5 Didn''t test
mode=6 No go


I tested by doing a clean reboot after changing modes. Any idea why some 
did and some didn''t? All I changed between modes was the mode=X in the 
modules.conf file. And of course the switch information for mode 4.

So I was able to finally settle on mode 4 since it is the only one that 
works and provides fault tolerance and load balancing. I just knew it 
was a simple thing I was missing. Thanks to all that have helped me 
through this. I have backed up all my configs incase anyone else wants 
to try this.

One question though: should mode4 allow an increase in overall 
throughput if I am able to setup another host with bonded interfaces? I 
am mainly looking at my backuppc installation as well as eventually 
doing live failover of domu''s.

Donny B.


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Op dinsdag 22 juni 2010 20:14:18 schreef Donny Brooks:
> On 6/21/2010 7:37 PM, Fajar A. Nugraha wrote:
> > On Tue, Jun 22, 2010 at 3:17 AM, Donny
Brooks<dbrooks@mdah.state.ms.us>
wrote:
> >> But I would like to at least take advantage of the vlan support so
I
> >> can at least test that. Shouldn''t I be able to do the
following to
> >> enable vlan support?:
> >>
> >> remove bond0 interface
> >> setup eth1 as vlan master
> >> change any reference to bond0 in the bond0.XX and brXX files to
eth1
> >> restart networking
> >> and hopefully it just work?
> >
> > You still need the bridge. So you can dump bond0, change all bond0.XX
> > to eth1.XX, but you still need brXX.
> 
> Ok, after a little clearing my head and thinking about it I found the
> "simple thing I was missing"... I had to change the MTU setting
on my
> servers network cards to allow the vlan header to be passed. I was
> unable to even get the eth2.X interfaces to work so I knew it had to be
> something else. I changed the MTU and it started working on the eth2.X
> devices (with the brXX stuff of course). After changing the MTU to 1496
> on eth1 - eth3 I then commenced testing the various bond modes again.
> Here are the results:
> 
> mode=0 No go
> mode=1 Works
> mode=2 No go
> mode=3 Didn''t test
> mode=4 Works
> mode=5 Didn''t test
> mode=6 No go
> 
> 
> I tested by doing a clean reboot after changing modes. Any idea why some
> did and some didn''t? All I changed between modes was the mode=X in
the
> modules.conf file. And of course the switch information for mode 4.
> 
> So I was able to finally settle on mode 4 since it is the only one that
> works and provides fault tolerance and load balancing. I just knew it
> was a simple thing I was missing. Thanks to all that have helped me
> through this. I have backed up all my configs incase anyone else wants
> to try this.
> 
> One question though: should mode4 allow an increase in overall
> throughput if I am able to setup another host with bonded interfaces? I
> am mainly looking at my backuppc installation as well as eventually
> doing live failover of domu''s.
> 
> Donny B.
mode 4 is what i use.

if the switch can handle it, yes.

or if you can get 2 switches that can bond interfaces over different switches, 
that''d be nice too. for switch failover. (but those kind of switches
are
expensive and usually have fibrechannel to connect them together)

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users