After successfully getting higher MTU to work on my Realtek NICs, I started testing the impact of higher MTU on file transfers using NFS exported ramdisk to ramdisk. The results were unexpected. The higher the MTU on the sending NIC, the lower the file transfer speed. I tested by using time cp to copy a 1GB file (In case compression might affect the results, so I dd the test file from the CentOS DVD ISO) The average of 6 runs per MTU with varying MTU on the receiving NICs MTU 1500 = 116.34 MB/s MTU 3000 = 103.42 MB/s MTU 4000 = 102.52 MB/s I was actually planning to do 1500, 3000, 4500, 6000 and 7200. However, I was surprised to discover although I had two seemingly identical Realtek PCI-E nics, the sending NIC could only go up to 4K. I'm wondering, that since Jumbo Frames was supposed to be better for bulk transfers, why am I seeing these results? Is it the ElRepo drivers I used to enable higher MTUs or possibly some kind of oddity with the realtek NICs I am using? Or am I mistaken about the benefits of jumbo frames and that they are only beneficial in specific configurations, perhaps only with higher network speeds and enough packets at 1500 MTU to overwhelm packet handling hardware?
Emmanuel Noobadmin wrote:> I'm wondering, that since Jumbo Frames was supposed to be better for > bulk transfers, why am I seeing these results? Is it the ElRepo > drivers I used to enable higher MTUs or possibly some kind of oddity > with the realtek NICs I am using? Or am I mistaken about the benefits > of jumbo frames and that they are only beneficial in specific > configurations, perhaps only with higher network speeds and enough > packets at 1500 MTU to overwhelm packet handling hardware? > _______________________________________________ > CentOS mailing list > CentOS at centos.org > http://lists.centos.org/mailman/listinfo/centos > >I googled about Jumbo Frames ("jumbo frames impact") and 2 top hits are very interesting: http://sd.wareonearth.com/~phil/jumbo.html http://www.chelsio.com/jumbo_enet_frames.html I guess if you are not happy with speed provided by Realtek you should definitely buy more expensive NIC's with support of 9000K packets. Ljubomir
On 06/25/2011 01:13 PM, Emmanuel Noobadmin wrote:> The results were unexpected. The higher the MTU on the sending NIC, > the lower the file transfer speed.What kind of switch are you using? If your switches don't support large packets, you'll also see slowdowns like that.
> I'm wondering, that since Jumbo Frames was supposed to be better for > bulk transfers, why am I seeing these results? Is it the ElRepo > drivers I used to enable higher MTUs or possibly some kind of oddity > with the realtek NICs I am using? Or am I mistaken about the benefits > of jumbo frames and that they are only beneficial in specific > configurations, perhaps only with higher network speeds and enough > packets at 1500 MTU to overwhelm packet handling hardware?In short, the entire path from origin to destination must be configured to support jumbo frames. If not, then the devices at various points in the path will attempt to scale and you'll end up getting fragmentation and higher overhead in the devices leading to worse overall performance. Have you verified this is the case? Of course, the quality of the driver for your NIC will also play a role.