Xen users - Feb 2011 - Bonded interfaces with trunk and iscsi SAN

We finally got our Dell MD3200i setup and working. What I now need to do 
is get it talking to my xen dom0''s. One is a T710 with 4 nics and the 
other is a 2900 with 2 nics. Each server has all of it''s interfaces 
bonded together using an 802.3ad type of setup and vlans passing over 
the bonded interface. I have given each dom0 a management IP on the 
bond0 interface. Also each dom0 is running Centos 5.5 with xen4.0 out of 
the gitco repo. I have successfully setup a windows domu with 5 nics (1 
network and 4 iscsi san) to talk to the SAN. That is the easy part. Now 
what I need to do is setup the dom0 to talk directly to it.

This is complicated with the bond+trunk. Basically what I need to happen 
is have a way to assign another IP to the bond0 interface. Being that I 
will have a very limited maintenance window to make this work, I would 
appreciate some feedback. One way I thought about doing it was to create 
a bond0:0 interface and add the SAN ip stuff to it. Should this work?

Here is the basic setup I have in case it helps:

in /etc/sysconfig/network-scripts I have the following:

ifcfg-bond0
ifcfg-bond0.1
ifcfg-bond0.10
ifcfg-bond0.11
ifcfg-bond0.12
ifcfg-bond0.13
ifcfg-bond0.14
ifcfg-bond0.15
ifcfg-bond0.2
ifcfg-bond0.20
ifcfg-bond0.21
ifcfg-bond0.22
ifcfg-bond0.23
ifcfg-bond0.3
ifcfg-bond0.4
ifcfg-bond0.5
ifcfg-bond0.6
ifcfg-bond0.7
ifcfg-bond0.8
ifcfg-bond0.9
ifcfg-br1
ifcfg-br10
ifcfg-br11
ifcfg-br12
ifcfg-br13
ifcfg-br14
ifcfg-br15
ifcfg-br2
ifcfg-br20
ifcfg-br21
ifcfg-br22
ifcfg-br23
ifcfg-br3
ifcfg-br4
ifcfg-br5
ifcfg-br6
ifcfg-br7
ifcfg-br8
ifcfg-br9
ifcfg-eth0
ifcfg-eth1

And the representative contents to help:

ifcfg-bond0:
DEVICE=bond0
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
IPADDR=10.8.1.151
NETMASK=255.255.255.0
NETWORK=10.8.1.0
GATEWAY=10.8.1.1
MTU=1496

ifcfg-bond0.1:
DEVICE=bond0.1
VLAN=yes
BRIDGE=br1
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet

ifcfg-br1:
DEVICE=br1
TYPE=Bridge
BOOTPROTO=static
ONBOOT=yes

ifcfg-eth0:
# Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet
DEVICE=eth0
BOOTPROTO=none
ONBOOT=yes
USECTL=no
MASTER=bond0
SLAVE=yes
MTU=1496


Vlan20 is the SAN vlan and it has 192.168.0.0 on it. In debian I know 
how to accomplish this since you use the vmbrX interface and give it an 
IP. However I am not 100% sure how to do this in centos. I appreciate 
any help on this.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Thu, Feb 17, 2011 at 2:32 AM, Donny Brooks <dbrooks@mdah.state.ms.us>
wrote:
> This is complicated with the bond+trunk. Basically what I need to happen is
> have a way to assign another IP to the bond0 interface. Being that I will
> have a very limited maintenance window to make this work, I would
appreciate
> some feedback. One way I thought about doing it was to create a bond0:0
> interface and add the SAN ip stuff to it. Should this work?
With your setup? No. Since you already carve-out bond0.20, all traffic
on vlan20 will go there.
> ifcfg-bond0
> ifcfg-bond0.20
> ifcfg-br20
> Vlan20 is the SAN vlan and it has 192.168.0.0 on it. In debian I know how
to
> accomplish this since you use the vmbrX interface and give it an IP.
However
> I am not 100% sure how to do this in centos. I appreciate any help on this.
Just put the IP on ifcfg-br20. Add the usual

IPADDRNETMASKGATEWAY
I noticed that you put ip address on bond0. Is that on native vlan? If
not, you should also move it to ifcfg-br*.

-- 
Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 2/16/2011 3:15 PM, Fajar A. Nugraha wrote:
> On Thu, Feb 17, 2011 at 2:32 AM, Donny
Brooks<dbrooks@mdah.state.ms.us>  wrote:
>> This is complicated with the bond+trunk. Basically what I need to
happen is
>> have a way to assign another IP to the bond0 interface. Being that I
will
>> have a very limited maintenance window to make this work, I would
appreciate
>> some feedback. One way I thought about doing it was to create a bond0:0
>> interface and add the SAN ip stuff to it. Should this work?
> With your setup? No. Since you already carve-out bond0.20, all traffic
> on vlan20 will go there.
>
>> ifcfg-bond0
>> ifcfg-bond0.20
>> ifcfg-br20
>> Vlan20 is the SAN vlan and it has 192.168.0.0 on it. In debian I know
how to
>> accomplish this since you use the vmbrX interface and give it an IP.
However
>> I am not 100% sure how to do this in centos. I appreciate any help on
this.
> Just put the IP on ifcfg-br20. Add the usual
>
> IPADDR> NETMASK> GATEWAY>
> I noticed that you put ip address on bond0. Is that on native vlan? If
> not, you should also move it to ifcfg-br*.
>
Thanks for that. That makes sense. Although I won''t need a gateway
since
it does not need to leave that vlan. Also the management IP is on the 
default vlan of 1. But I can move it to the br* interface if it would 
help anything.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 2/16/2011 3:17 PM, Donny Brooks wrote:
> On 2/16/2011 3:15 PM, Fajar A. Nugraha wrote:
>> On Thu, Feb 17, 2011 at 2:32 AM, Donny 
>> Brooks<dbrooks@mdah.state.ms.us>  wrote:
>>> This is complicated with the bond+trunk. Basically what I need to 
>>> happen is
>>> have a way to assign another IP to the bond0 interface. Being that
I
>>> will
>>> have a very limited maintenance window to make this work, I would 
>>> appreciate
>>> some feedback. One way I thought about doing it was to create a
bond0:0
>>> interface and add the SAN ip stuff to it. Should this work?
>> With your setup? No. Since you already carve-out bond0.20, all traffic
>> on vlan20 will go there.
>>
>>> ifcfg-bond0
>>> ifcfg-bond0.20
>>> ifcfg-br20
>>> Vlan20 is the SAN vlan and it has 192.168.0.0 on it. In debian I 
>>> know how to
>>> accomplish this since you use the vmbrX interface and give it an
IP.
>>> However
>>> I am not 100% sure how to do this in centos. I appreciate any help 
>>> on this.
>> Just put the IP on ifcfg-br20. Add the usual
>>
>> IPADDR>> NETMASK>> GATEWAY>>
>> I noticed that you put ip address on bond0. Is that on native vlan? If
>> not, you should also move it to ifcfg-br*.
>>
> Thanks for that. That makes sense. Although I won''t need a gateway
> since it does not need to leave that vlan. Also the management IP is 
> on the default vlan of 1. But I can move it to the br* interface if it 
> would help anything.
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
Ok, I added the IP information to the ifcfg-br20 file and ifdown/ifup 
that interface. I am able to ping the 192.168.130.X range but nothing in 
the 192.168.131.X, 192.168.132.X, or 192.168.133.X ranges. Here is the 
contents of that file:

DEVICE=br20
TYPE=Bridge
BOOTPROTO=static
IPADDR=192.168.130.51
NETMASK=255.255.0.0
NETWORK=192.168.0.0
ONBOOT=yes


I know I am missing something simple. I know it should work as the 
windows 2008 r2 domu I have setup with 4 nics all on br20 is able to 
access all the ranges needed.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Thu, Feb 17, 2011 at 4:35 AM, Donny Brooks <dbrooks@mdah.state.ms.us>
wrote:
> Ok, I added the IP information to the ifcfg-br20 file and ifdown/ifup that
> interface. I am able to ping the 192.168.130.X range but nothing in the
> 192.168.131.X, 192.168.132.X, or 192.168.133.X ranges. Here is the contents
> of that file:
>
> DEVICE=br20
> TYPE=Bridge
> BOOTPROTO=static
> IPADDR=192.168.130.51
> NETMASK=255.255.0.0
> NETWORK=192.168.0.0
> ONBOOT=yes
Does all host in 192.168.X.X use the same /16 (255.255.0.0) netmask?
A common error is when some hosts use /16, while others use /24.

-- 
Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 2/16/2011 3:41 PM, Fajar A. Nugraha wrote:
> On Thu, Feb 17, 2011 at 4:35 AM, Donny
Brooks<dbrooks@mdah.state.ms.us>  wrote:
>> Ok, I added the IP information to the ifcfg-br20 file and ifdown/ifup
that
>> interface. I am able to ping the 192.168.130.X range but nothing in the
>> 192.168.131.X, 192.168.132.X, or 192.168.133.X ranges. Here is the
contents
>> of that file:
>>
>> DEVICE=br20
>> TYPE=Bridge
>> BOOTPROTO=static
>> IPADDR=192.168.130.51
>> NETMASK=255.255.0.0
>> NETWORK=192.168.0.0
>> ONBOOT=yes
> Does all host in 192.168.X.X use the same /16 (255.255.0.0) netmask?
> A common error is when some hosts use /16, while others use /24.
>
The only thing on that vlan is the SAN. Now that I think about it the 
windows machine is set to a /24 on each interface. So it may be that the 
SAN is set for /24 instead of /16. Now just to figure out how to have 4 
"interfaces" each with a /24 subnet...... hmmmm. :)

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Thu, Feb 17, 2011 at 4:45 AM, Donny Brooks <dbrooks@mdah.state.ms.us>
wrote:
> On 2/16/2011 3:41 PM, Fajar A. Nugraha wrote:
>> On Thu, Feb 17, 2011 at 4:35 AM, Donny
Brooks<dbrooks@mdah.state.ms.us>
>>> Ok, I added the IP information to the ifcfg-br20 file and
ifdown/ifup
>>> that
>>> interface. I am able to ping the 192.168.130.X range but nothing in
the
>>> 192.168.131.X, 192.168.132.X, or 192.168.133.X ranges. Here is the
>>> contents
>>> of that file:
>>>
>>> DEVICE=br20
>>> TYPE=Bridge
>>> BOOTPROTO=static
>>> IPADDR=192.168.130.51
>>> NETMASK=255.255.0.0
>>> NETWORK=192.168.0.0
>>> ONBOOT=yes
>>
>> Does all host in 192.168.X.X use the same /16 (255.255.0.0) netmask?
>> A common error is when some hosts use /16, while others use /24.
>>
> The only thing on that vlan is the SAN. Now that I think about it the
> windows machine is set to a /24 on each interface. So it may be that the
SAN
> is set for /24 instead of /16. Now just to figure out how to have 4
> "interfaces" each with a /24 subnet...... hmmmm. :)
That''s easy. ifcfg-br20:1, ifcfg-br20:2, etc. :D

-- 
Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 2/16/2011 4:05 PM, Fajar A. Nugraha wrote:
> On Thu, Feb 17, 2011 at 4:45 AM, Donny
Brooks<dbrooks@mdah.state.ms.us>  wrote:
>> On 2/16/2011 3:41 PM, Fajar A. Nugraha wrote:
>>> On Thu, Feb 17, 2011 at 4:35 AM, Donny
Brooks<dbrooks@mdah.state.ms.us>
>>>> Ok, I added the IP information to the ifcfg-br20 file and
ifdown/ifup
>>>> that
>>>> interface. I am able to ping the 192.168.130.X range but
nothing in the
>>>> 192.168.131.X, 192.168.132.X, or 192.168.133.X ranges. Here is
the
>>>> contents
>>>> of that file:
>>>>
>>>> DEVICE=br20
>>>> TYPE=Bridge
>>>> BOOTPROTO=static
>>>> IPADDR=192.168.130.51
>>>> NETMASK=255.255.0.0
>>>> NETWORK=192.168.0.0
>>>> ONBOOT=yes
>>> Does all host in 192.168.X.X use the same /16 (255.255.0.0)
netmask?
>>> A common error is when some hosts use /16, while others use /24.
>>>
>> The only thing on that vlan is the SAN. Now that I think about it the
>> windows machine is set to a /24 on each interface. So it may be that
the SAN
>> is set for /24 instead of /16. Now just to figure out how to have 4
>> "interfaces" each with a /24 subnet...... hmmmm. :)
> That''s easy. ifcfg-br20:1, ifcfg-br20:2, etc. :D
>
D''OH!!! And I had just mentioned that earlier! Thanks for stating the 
obvious for those oblivious to it. :)

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 2/16/2011 4:05 PM, Fajar A. Nugraha wrote:
> On Thu, Feb 17, 2011 at 4:45 AM, Donny
Brooks<dbrooks@mdah.state.ms.us>  wrote:
>> On 2/16/2011 3:41 PM, Fajar A. Nugraha wrote:
>>> On Thu, Feb 17, 2011 at 4:35 AM, Donny
Brooks<dbrooks@mdah.state.ms.us>
>>>> Ok, I added the IP information to the ifcfg-br20 file and
ifdown/ifup
>>>> that
>>>> interface. I am able to ping the 192.168.130.X range but
nothing in the
>>>> 192.168.131.X, 192.168.132.X, or 192.168.133.X ranges. Here is
the
>>>> contents
>>>> of that file:
>>>>
>>>> DEVICE=br20
>>>> TYPE=Bridge
>>>> BOOTPROTO=static
>>>> IPADDR=192.168.130.51
>>>> NETMASK=255.255.0.0
>>>> NETWORK=192.168.0.0
>>>> ONBOOT=yes
>>> Does all host in 192.168.X.X use the same /16 (255.255.0.0)
netmask?
>>> A common error is when some hosts use /16, while others use /24.
>>>
>> The only thing on that vlan is the SAN. Now that I think about it the
>> windows machine is set to a /24 on each interface. So it may be that
the SAN
>> is set for /24 instead of /16. Now just to figure out how to have 4
>> "interfaces" each with a /24 subnet...... hmmmm. :)
> That''s easy. ifcfg-br20:1, ifcfg-br20:2, etc. :D
>
I think it is getting late in the day.... I added the ifcfg-br20:1/2/3/4 
files with the contents as follows:

[root@xen-test network-scripts]# cat ifcfg-br20\:*
DEVICE=br20:1
TYPE=Bridge
BOOTPROTO=static
IPADDR=192.168.130.53
NETMASK=255.255.255.0
NETWORK=192.168.130.0
ONBOOT=yes

DEVICE=br20:2
TYPE=Bridge
BOOTPROTO=static
IPADDR=192.168.131.53
NETMASK=255.255.255.0
NETWORK=192.168.131.0
ONBOOT=yes

DEVICE=br20:3
TYPE=Bridge
BOOTPROTO=static
IPADDR=192.168.132.53
NETMASK=255.255.255.0
NETWORK=192.168.132.0
ONBOOT=yes

DEVICE=br20:4
TYPE=Bridge
BOOTPROTO=static
IPADDR=192.168.133.53
NETMASK=255.255.255.0
NETWORK=192.168.133.0
ONBOOT=yes

and when I try to ifup the first interface I get this......

[root@xen-test network-scripts]# ifup br20:1
Bridge support not available in this kernel


How is bridge support not available in the kernel when it is bridges I 
am working with already? I think I will take a step away from this for a 
bit and come back to it tomorrow.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Thu, Feb 17, 2011 at 5:19 AM, Donny Brooks <dbrooks@mdah.state.ms.us>
wrote:
>>> The only thing on that vlan is the SAN. Now that I think about it
the
>>> windows machine is set to a /24 on each interface. So it may be
that the
>>> SAN
>>> is set for /24 instead of /16. Now just to figure out how to have 4
>>> "interfaces" each with a /24 subnet...... hmmmm. :)
>>
>> That''s easy. ifcfg-br20:1, ifcfg-br20:2, etc. :D
>>
>
> I think it is getting late in the day.... I added the ifcfg-br20:1/2/3/4
> files with the contents as follows:
>
> [root@xen-test network-scripts]# cat ifcfg-br20\:*
> DEVICE=br20:1
> TYPE=Bridge
> BOOTPROTO=static
> IPADDR=192.168.130.53
> NETMASK=255.255.255.0
> NETWORK=192.168.130.0
> ONBOOT=yes
>
> DEVICE=br20:2
> TYPE=Bridge
> BOOTPROTO=static
> IPADDR=192.168.131.53
> NETMASK=255.255.255.0
> NETWORK=192.168.131.0
> ONBOOT=yes
>
> DEVICE=br20:3
> TYPE=Bridge
> BOOTPROTO=static
> IPADDR=192.168.132.53
> NETMASK=255.255.255.0
> NETWORK=192.168.132.0
> ONBOOT=yes
>
> DEVICE=br20:4
> TYPE=Bridge
> BOOTPROTO=static
> IPADDR=192.168.133.53
> NETMASK=255.255.255.0
> NETWORK=192.168.133.0
> ONBOOT=yes
>
> and when I try to ifup the first interface I get this......
>
> [root@xen-test network-scripts]# ifup br20:1
> Bridge support not available in this kernel
>
>
> How is bridge support not available in the kernel when it is bridges I am
> working with already? I think I will take a step away from this for a bit
> and come back to it tomorrow.

Apparently when you put "TYPE=Bridge", it decides to always create a
bridge with that name (RHEL6 actually ALLOWS you to have a bridge
named "br0:1"). Removing "TYPE=Bridge" should make it work
as
expected.

-- 
Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 2/16/2011 8:03 PM, Fajar A. Nugraha wrote:
> On Thu, Feb 17, 2011 at 5:19 AM, Donny
Brooks<dbrooks@mdah.state.ms.us>  wrote:
>>>> The only thing on that vlan is the SAN. Now that I think about
it the
>>>> windows machine is set to a /24 on each interface. So it may be
that the
>>>> SAN
>>>> is set for /24 instead of /16. Now just to figure out how to
have 4
>>>> "interfaces" each with a /24 subnet...... hmmmm. :)
>>> That''s easy. ifcfg-br20:1, ifcfg-br20:2, etc. :D
>>>
>> I think it is getting late in the day.... I added the
ifcfg-br20:1/2/3/4
>> files with the contents as follows:
>>
>> [root@xen-test network-scripts]# cat ifcfg-br20\:*
>> DEVICE=br20:1
>> TYPE=Bridge
>> BOOTPROTO=static
>> IPADDR=192.168.130.53
>> NETMASK=255.255.255.0
>> NETWORK=192.168.130.0
>> ONBOOT=yes
>>
>> DEVICE=br20:2
>> TYPE=Bridge
>> BOOTPROTO=static
>> IPADDR=192.168.131.53
>> NETMASK=255.255.255.0
>> NETWORK=192.168.131.0
>> ONBOOT=yes
>>
>> DEVICE=br20:3
>> TYPE=Bridge
>> BOOTPROTO=static
>> IPADDR=192.168.132.53
>> NETMASK=255.255.255.0
>> NETWORK=192.168.132.0
>> ONBOOT=yes
>>
>> DEVICE=br20:4
>> TYPE=Bridge
>> BOOTPROTO=static
>> IPADDR=192.168.133.53
>> NETMASK=255.255.255.0
>> NETWORK=192.168.133.0
>> ONBOOT=yes
>>
>> and when I try to ifup the first interface I get this......
>>
>> [root@xen-test network-scripts]# ifup br20:1
>> Bridge support not available in this kernel
>>
>>
>> How is bridge support not available in the kernel when it is bridges I
am
>> working with already? I think I will take a step away from this for a
bit
>> and come back to it tomorrow.
>
> Apparently when you put "TYPE=Bridge", it decides to always
create a
> bridge with that name (RHEL6 actually ALLOWS you to have a bridge
> named "br0:1"). Removing "TYPE=Bridge" should make it
work as
> expected.
>
Fajar you are brilliant! That worked like a charm! Now I can have all 
four vlans and the "redundant failover" that dell recommends. I mean
it
is kind of redundantly redundant.... dual controllers with 4 nics per 
controller in the SAN, with each of the 8 ports split among various 
cards in my core switch, then going to a 802.3ad type bond to the 
servers where all 2 or 4 of the NIC''s are bonded together. And before 
anyone says anything... I know it is better to have the SAN on it''s own
interfaces and setup jumbo frames. That is not an option on the cisco 
switch we have unfortunately.

Thanks for all the help! Now just to get the stuff to work with a nice 
gui frontend like eucalyptus or similar.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users