Denny Schierz
2011-Apr-08 10:30 UTC
FreeBSD-8.2: Channel Bounding: LACP or Roundrobin? With Cisco Catalyst
hi,
I want to bound two e1000 (1Gb/s) channels and use at the moment LCAP,
but the max throughput is slower, than without channel bounding. I've
got round about 70MB/s instead of > 150MB/s - 200MB/s.
I used iperf with standard options:
:~$ iperf -f M -c 1.2.3.4
------------------------------------------------------------
Client connecting to 1.2.3.4, TCP port 5001
TCP window size: 0.02 MByte (default)
------------------------------------------------------------
[ 3] local 1.2.3.5 port 58637 connected with 1.2.3.4 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 705 MBytes 70.5 MBytes/sec
If a second PC do the same, than my 70MB/s splittet into ~30MB/s and
~40MB/s
config:
root@iscsihead-m:~# ifconfig lagg0
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
1500
options=219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC>
ether 00:15:17:f1:5d:5f
inet6 fe80::215:17ff:fef1:5d5f%lagg0 prefixlen 64 scopeid 0x5
inet 1.2.3.4 netmask 0xffffffc0 broadcast 1.2.3.255
nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
media: Ethernet autoselect
status: active
laggproto lacp
laggport: em1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: em0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
Config from the Cisco:
cisco#sh run int po3
Building configuration...
Current configuration : 119 bytes
!
interface Port-channel3
description iscsi-test
switchport
switchport access vlan 111
switchport mode access
end
#sh etherchannel summary
Flags: D - down P - bundled in port-channel
I - stand-alone s - suspended
R - Layer3 S - Layer2
U - in use f - failed to allocate aggregator
M - not in use, minimum links not met
u - unsuitable for bundling
w - waiting to be aggregated
d - default port
Number of channel-groups in use: 3
Number of aggregators: 3
Group Port-channel Protocol Ports
------+-------------+-----------+-----------------------------------------------
1 Po1(SU) - Gi1/1(P) Gi1/2(P)
2 Po2(SU) - Gi6/17(P) Gi6/18(P) Gi6/19(P)
3 Po3(SU) LACP Gi5/41(P) Gi5/44(P)
cisco#sh run int gi5/41
Building configuration...
Current configuration : 182 bytes
!
interface GigabitEthernet5/41
description iscsi-head1
switchport access vlan 111
switchport mode access
no cdp enable
channel-group 3 mode active
spanning-tree portfast
end
#sh run int gi5/44
Building configuration...
Current configuration : 183 bytes
!
interface GigabitEthernet5/44
description GRAU1_iscsi2
switchport access vlan 111
switchport mode access
no cdp enable
channel-group 3 mode active
spanning-tree portfast
end
#sh etherchannel load-balance
EtherChannel Load-Balancing Configuration:
src-dst-mac
EtherChannel Load-Balancing Addresses Used Per-Protocol:
Non-IP: Source XOR Destination MAC address
IPv4: Source XOR Destination MAC address
IPv6: Source XOR Destination MAC address
What I saw with tcpdump: it seems, that only one device is used.
Maybe, Cisco uses mac and FreeBSD IP ?
Any suggestions?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
Url :
http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20110408/2844a6f7/attachment.pgp
Rumen Telbizov
2011-Apr-08 13:52 UTC
FreeBSD-8.2: Channel Bounding: LACP or Roundrobin? With Cisco Catalyst
Denny, Since LACP uses hashing to determine which channel to send the packet to the traffic between two nodes (ip:mac ip:mac) will always get bound to only one of the two channels. I am using HP procurve's here and they do seem to hash by ip too although I can't see/tweak that as in catalyst. If you can add the IP address in the hashing function of the switch then you have a better chance of achieving what you're trying to by bringing an IP alias (might have to try a few until you find one that maps to the opposite channel) and then having two iperfs between different IP addresses. In the real live scenario it will be equivalent to having multiple iscsi targets between the two hosts on top of different IPs if this works for you. As for the load balancing option - my HP 2910al's seemed to get CPU overloaded when I push a gigabit of traffic with this option. My guess is the crazy mac address change might be exhausting the cpu much faster. I'd be interested to see how this affects catalyst - let me know. Good luck, Rumen Telbizov On Fri, Apr 8, 2011 at 3:29 AM, Denny Schierz <linuxmail@4lin.net> wrote:> hi, > > I want to bound two e1000 (1Gb/s) channels and use at the moment LCAP, > but the max throughput is slower, than without channel bounding. I've > got round about 70MB/s instead of > 150MB/s - 200MB/s. > > I used iperf with standard options: > > :~$ iperf -f M -c 1.2.3.4 > ------------------------------------------------------------ > Client connecting to 1.2.3.4, TCP port 5001 > TCP window size: 0.02 MByte (default) > ------------------------------------------------------------ > [ 3] local 1.2.3.5 port 58637 connected with 1.2.3.4 port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-10.0 sec 705 MBytes 70.5 MBytes/sec > > If a second PC do the same, than my 70MB/s splittet into ~30MB/s and > ~40MB/s > > config: > > root@iscsihead-m:~# ifconfig lagg0 > lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu > 1500 > > > options=219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC> > ether 00:15:17:f1:5d:5f > inet6 fe80::215:17ff:fef1:5d5f%lagg0 prefixlen 64 scopeid 0x5 > inet 1.2.3.4 netmask 0xffffffc0 broadcast 1.2.3.255 > nd6 options=3<PERFORMNUD,ACCEPT_RTADV> > media: Ethernet autoselect > status: active > laggproto lacp > laggport: em1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> > laggport: em0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> > > > Config from the Cisco: > > > cisco#sh run int po3 > Building configuration... > > Current configuration : 119 bytes > ! > interface Port-channel3 > description iscsi-test > switchport > switchport access vlan 111 > switchport mode access > end > > > > #sh etherchannel summary > Flags: D - down P - bundled in port-channel > I - stand-alone s - suspended > R - Layer3 S - Layer2 > U - in use f - failed to allocate aggregator > > M - not in use, minimum links not met > u - unsuitable for bundling > w - waiting to be aggregated > d - default port > > > Number of channel-groups in use: 3 > Number of aggregators: 3 > > Group Port-channel Protocol Ports > > ------+-------------+-----------+----------------------------------------------- > 1 Po1(SU) - Gi1/1(P) Gi1/2(P) > 2 Po2(SU) - Gi6/17(P) Gi6/18(P) Gi6/19(P) > 3 Po3(SU) LACP Gi5/41(P) Gi5/44(P) > > cisco#sh run int gi5/41 > Building configuration... > > Current configuration : 182 bytes > ! > interface GigabitEthernet5/41 > description iscsi-head1 > switchport access vlan 111 > switchport mode access > no cdp enable > channel-group 3 mode active > spanning-tree portfast > end > > > > #sh run int gi5/44 > Building configuration... > > Current configuration : 183 bytes > ! > interface GigabitEthernet5/44 > description GRAU1_iscsi2 > switchport access vlan 111 > switchport mode access > no cdp enable > channel-group 3 mode active > spanning-tree portfast > end > > > #sh etherchannel load-balance > EtherChannel Load-Balancing Configuration: > src-dst-mac > > EtherChannel Load-Balancing Addresses Used Per-Protocol: > Non-IP: Source XOR Destination MAC address > IPv4: Source XOR Destination MAC address > IPv6: Source XOR Destination MAC address > > > What I saw with tcpdump: it seems, that only one device is used. > > Maybe, Cisco uses mac and FreeBSD IP ? > > Any suggestions? > >-- Rumen Telbizov http://telbizov.com