freebsd stable - Aug 2010 - Problem with mxge on RELENG_8_1

Hello.

I recently updated the central file server/router systems for a pair of 
research clusters from RELENG_8_0 to RELENG_8_1.  After following the 
proper procedures, the network throughput when pulling files from both 
machines via mxge0 is 200KB/s or less.  Before the update, 50MB/s was 
the normal rate.

Doing some testing, with the 8.1 kernel booted, I can upload files to 
the server over the 10G link with scp or NFS at the expected rates.  I 
can fetch files from the Internet using this server as the NAT gateway 
also at the expected rates.  Retrieving files from the server over mxge, 
the throughput falls to 200KB/s.  Retrieving files from the server from 
the onboard Broadcom NIC, rates are as to be expected from gigabit. 
With 8.0, everything works as expected.

Hardware Config 1:
Dell Poweredge R610 with 2 Xeon E5530 @ 2.4 GHz, hyperthreading disabled 
and 24GB RAM.  Onboard interface is bce. Disk is attached via a PERC 
6/E. Internal cluster switch is Dell Powerconnect 6248P.  This switch 
sees excessive large packets on the 10G connection on 8.1, but not 8.0.

Hardware Config 2:
HP Proliant DL 320G6 with 1 Xeon  E5540 @ 2.53GHz, hyperthreading 
enabled and 8GB RAM.  Onboard interface is bge.  Disk is attached via a 
HP Smart Array P212.  Internal cluster switch is an HP Procurve 2910al. 
  It does not see any packet errors from the 10G link.

If anyone could point me in the general direction of a solution, I would 
  be very appreciative.  Thank you.

-- 
Bob Healey
Systems Administrator
Biocomputation and Bioinformatics Constellation
and Molecularium
healer@rpi.edu
(518) 276-4407

At 3:21 PM -0400 8/13/10, Robert Healey wrote:
>
>I recently updated the central file server/router systems for a
>pair of research clusters from RELENG_8_0 to RELENG_8_1.  After
>following the proper procedures, the network throughput when
>pulling files from both machines via mxge0 is 200KB/s or less.
>Before the update, 50MB/s was the normal rate.
>
>Doing some testing, with the 8.1 kernel booted, I can upload
>files to the server over the 10G link with scp or NFS at the
>expected rates.  I can fetch files from the Internet using
>this server as the NAT gateway also at the expected rates.
>Retrieving files from the server over mxge, the throughput
>falls to 200KB/s.  Retrieving files from the server from the
>onboard Broadcom NIC, rates are as to be expected from gigabit.
>With 8.0, everything works as expected.
>
>Hardware Config 1:
>Dell Poweredge R610 with 2 Xeon E5530 @ 2.4 GHz, hyperthread
>disabled and 24GB RAM.  Onboard interface is bce. Disk is
>attached via a PERC 6/E. Internal cluster switch is Dell
>Powerconnect 6248P.  This switch sees excessive large packets
>on the 10G connection on 8.1, but not 8.0.
>
>Hardware Config 2:
>HP Proliant DL 320G6 with 1 Xeon  E5540 @ 2.53GHz, hyperthread
>enabled and 8GB RAM.  Onboard interface is bge.  Disk is
>attached via a HP Smart Array P212.  Internal cluster switch
>is an HP Procurve 2910al.  It does not see any packet errors
>from the 10G link.

You're seeing the performance problems with both scp and NFS?

Could you get some tcpdumps of a file transfer with both setups
(8.0 vs 8.1), just to see if something very odd jumps out?

If I'm reading this right, file transfers work at expected speeds
if the file is going *to* the server, but you see the slowdown
when you're pulling files *from* the server.  That sounds similar
to a problem with a duplex-mismatch, except of course that you
shouldn't be getting into duplex issues on a gigabit network.

-- 
Garance Alistair Drosehn            =   gad@gilead.netel.rpi.edu
Senior Systems Programmer           or  gad@freebsd.org
Rensselaer Polytechnic Institute    or  drosih@rpi.edu