CentOS - Mar 2013 - Bonded interfaces - testing

Hi,

I have 2 servers running CentOS 6.3. Each has four 1gb Ethernet ports. I
have bonded all four ports on each server and patched them to the same
switch (following the instructions at
http://wiki.centos.org/TipsAndTricks/BondingInterfaces). I have created
aggregated trunks for the servers respective ports on the switch. The
switch reports that the ports are up and that the Link Aggregation is
enabled.

I was under the impression, perhaps falsely, that I would see an
improvement in the throughput. I am not sure that I am. I used a utility
call nuttcp before the trunking and after to test the throughput. I am
seeing miniscule differences in the Mbps. I am not sure that I am testing
correctly. Perhaps the speed will not change but the available bandwidth
has increased. Is there some way to demonstrate that I am able to transfer
data from these two servers at a rate of 4,000Mbs?

Does anyone have any thoughts? I pasted some details below increase they
have a bearing.
Thanks in advance
Dermot.


./nuttcp otherserver
1125.6573 MB /  10.03 sec =  941.4967 Mbps 10 %TX 47 %RX


cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: slow
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
        Aggregator ID: 2
        Number of ports: 4
        Actor Key: 17
        Partner Key: 420
        Partner Mac Address: 10:0d:7f:4c:16:ca

Slave Interface: eth0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 2c:76:8a:5d:28:1c
Aggregator ID: 2
Slave queue ID: 0

Slave Interface: eth1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 2c:76:8a:5d:28:1d
Aggregator ID: 2
Slave queue ID: 0

Slave Interface: eth2
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 2c:76:8a:5d:28:1e
Aggregator ID: 2
Slave queue ID: 0

Slave Interface: eth3
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 2c:76:8a:5d:28:1f
Aggregator ID: 2
Slave queue ID: 0

> Does anyone have any thoughts? I pasted some details below increase they
> have a bearing.
>

Remember that LACP (802.3ad) uses a hash algorithm (configurable on how
it's carried out and whether you use mac addresses, dst/src IPs and ports
will vary quite often for optimisation) to pick a physical connection for
the TCP flow ... and that will stay over the physical connection.

As such for any one given flow you'll see up to the speed of the physical
interface the data is going over... the speed increases come with multiple
systems communicating with that server and with the right pick of hashing
function having those connections go over differing interfaces.