freebsd stable - Apr 2011 - Network throughput: Never get more than 112MB/s über two NICs

hi,

after testing severals loadbalancing (LACP) types with Cisco, we saw,
that we never get more than 112MB/s with two network cards and iperf. 

So, we tested without loadbalancing, 4 Clients (iperf -f M -c <ip>) and
two target IPs. Every IP has his own 1Gb/s network card.
On the end, two clients had a connection to IP 1 and the second two to
IP 2.

First we used the two onboard NICs and then, one onboard and one
external NIC, but without success. We never get more then 112MB/s

All are connected through a Cisco Catalyst WS-X4515.

The mainboard is a Intel S3420GP.

any suggestion?

cu denny

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
Url :
http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20110411/9d1e73a4/attachment.pgp

Am 11.04.2011 um 16:20 schrieb Michael Loftis:
> Most switches load balance based on MAC addresses, not IP, unless it
> is routing the traffic as a Layer 3 switch then you can enable IP
> based load balancing in some of those.  Also you might simply be
that was the reason, why we disabled the loadbalancer and tested with plain
NICs.
> reaching the limits of your firewall box too you haven't mentioned any
> of it's specs, nor do you seem to have run top while running the iperf
> tests.
The clients (who running iperf -c <ip>) had a load near zero, they are
powerful machines  (Sun sparcs) with 8 cores and more. The machine, with 4 Cores
(Xeon) who is running "iperf -s", had a load round about ~0.8.

No firewall etc. between the hosts, just plain network :-) 

cu denny

On Mon, 11.04.2011 at 12:00:39 +0200, Denny Schierz
wrote:
> hi,
> 
> after testing severals loadbalancing (LACP) types with Cisco, we saw,
> that we never get more than 112MB/s with two network cards and iperf. 
> 
> So, we tested without loadbalancing, 4 Clients (iperf -f M -c <ip>)
and
> two target IPs. Every IP has his own 1Gb/s network card.
> On the end, two clients had a connection to IP 1 and the second two to
> IP 2.
> 
> First we used the two onboard NICs and then, one onboard and one
> external NIC, but without success. We never get more then 112MB/s
> 
> All are connected through a Cisco Catalyst WS-X4515.
> 
> The mainboard is a Intel S3420GP.
Are the NICs PCI or PCIe? If the former, IIRC the PCI bus bandwidth
maxes out at 133MB/s so that might explain your numbers. If your NICs are
PCIe, I have no helpful clues, sorry.

Uli

On 11 April 2011 22:00, Denny Schierz <linuxmail@4lin.net>
wrote:
> hi,
>
> after testing severals loadbalancing (LACP) types with Cisco, we saw,
> that we never get more than 112MB/s with two network cards and iperf.
>
> So, we tested without loadbalancing, 4 Clients (iperf -f M -c <ip>)
and
> two target IPs. Every IP has his own 1Gb/s network card.
> On the end, two clients had a connection to IP 1 and the second two to
> IP 2.
>
> First we used the two onboard NICs and then, one onboard and one
> external NIC, but without success. We never get more then 112MB/s
>
> All are connected through a Cisco Catalyst WS-X4515.
>
> The mainboard is a Intel S3420GP.
>
> any suggestion?
Are you doing LACP from the FreeBSD host? (ie. lagg(4) interface). The
current hash just uses the mac and IP addresses (not tcp/udp ports) so
you need to make sure your multiple streams have different mac/ip
numbers in order to load balance over multiple links.


Andrew