Hi all, I am working on the MX support for Lustre. I am also testing Lustre using socklnd (TCP) on our 10 Gb/s NICs in native Ethernet mode (not Ethernet over Myrinet). Without the zero-copy patch, I see read and write performance peak at ~370 MB/s with a single client and server connected back-to--back (switchless). With the zero-copy patch, performance increases to 615 MB/s for read and 600 Mb/s for write. For comparison, netperf gets 1,130 MB/s. In order to test with multiple Ethernet clients, I connected them using a Fujitsu XG700. Going through the switch, performance is 545 MB/s for read and write. With two clients, read gets 1,005 MB/s and write plateaus at 630 MB/s. With three clients, read hits 1,180 MB/s (94% of line rate) and write stays at 630 MB/s. Eric mentioned that even with the ZC patch, one side still does a copy. From the above, I would imagine that the server is copying during the writes and the clients during the reads. Does that sound right? Does anyone have TCP performance numbers for any TCP Offload Engine (TOE) NICs or any standard 10 Gb/s Ethernet NICs? The test machines are dual-core, dual CPU 2.4 GHz Opterons with 8 GB of RAM, Nvidia chipsets, Myri-10G NICs (with CX-4 adapters), and they are running the latest Centos patches and a 2.6.9-34 kernel. I tuned TCP per the README that comes with the Myri-10G Ethernet driver which includes: MSI on Write-combining on TSO on and modify /etc/sysctl.conf with: net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.ipv4.tcp_rmem = 4096 87380 16777216 net.ipv4.tcp_wmem = 4096 65536 16777216 net.core.netdev_max_backlog = 250000 The tests use lctl to send 256 4K pages for read and write. I did not try using less than 256 pages. Scott -- Scott Atchley Myricom Inc. http://www.myri.com