Hi list, We've had an interesting NFS performance issue with the following setup. Clients : CentOS 5.5, kernel 2.6.18-194.el5 Server : Solaris 10 kernel Generic_142901-08 Reading a file (not cached) with dd : dd if=file1 of=/dev/null bs=1024k count=100 performance is around.. 700 kB/s ! /proc/mounts shows rsize has been negotiated to 1mB If we force rsize to a smaller value, 32kB, the problem goes away and we're back around 70/80 mB/s read performance. Wiresharks show something interesting in the first (slow) case: It seems that the NFS client is "waiting" between requests: ? 1473 11.756852 ? client.net ?nfs1.net ? ?TCP ? ? ?868 > nfs [ACK] Seq=9345 ? 1474 11.757888 ? nfs1.net ? ?client.net ?RPC ? ? ?Continuation ? 1475 11.797837 ? client.net ?nfs1.net ? ?TCP ? ? ?868 > nfs [ACK] Seq=9345 ? 1476 11.798877 ? nfs1.net ? ?client.net ?RPC ? ? ?Continuation [....] Notice how the server replies within about 0,001s, but the client appears to be waiting 0,04s before asking for the next bit. (we captured the traffic from the client). This pattern repeats for the whole exchange It's likely we missed something, can't figure it out.. has anyone seen this before ? Cheers, Alex
From: Alexandre Lecuyer <alex.ranskis at gmail.com>> We've had an interesting NFS performance issue with the > following setup. > Clients : CentOS 5.5, kernel 2.6.18-194.el5 > Server : Solaris 10 kernel Generic_142901-08 > /proc/mounts shows rsize has been negotiated to 1mB > If we force rsize to a smaller value, 32kB, the problem goes away and > we're back around 70/80 mB/s read performance.Not sure but the nfs faq says to look at /usr/include/linux/nfsd/const.h /* * Maximum blocksize supported by daemon currently at 32K */ #define NFSSVC_MAXBLKSIZE (32*1024) JD
Alexandre Lecuyer wrote:> Hi list, > > We've had an interesting NFS performance issue with the following setup. > Clients : CentOS 5.5, kernel 2.6.18-194.el5 > Server : Solaris 10 kernel Generic_142901-08 > > Reading a file (not cached) with dd : dd if=file1 of=/dev/null > bs=1024k count=100 > performance is around.. 700 kB/s ! > > /proc/mounts shows rsize has been negotiated to 1mB > > If we force rsize to a smaller value, 32kB, the problem goes away and > we're back around 70/80 mB/s read performance. > > Wiresharks show something interesting in the first (slow) case: > It seems that the NFS client is "waiting" between requests: > > > 1473 11.756852 client.net nfs1.net TCP 868 > nfs [ACK] Seq=9345 > 1474 11.757888 nfs1.net client.net RPC Continuation > 1475 11.797837 client.net nfs1.net TCP 868 > nfs [ACK] Seq=9345 > 1476 11.798877 nfs1.net client.net RPC Continuation > [....] > > Notice how the server replies within about 0,001s, but the client > appears to be waiting 0,04s before asking for the next bit. > (we captured the traffic from the client). This pattern repeats for > the whole exchange > > It's likely we missed something, can't figure it out.. has anyone seen > this before ? > > Cheers, > > Alex > _______________________________________________ > CentOS mailing list > CentOS at centos.org > http://lists.centos.org/mailman/listinfo/centos >Well, it's been a long time since I've done troubleshooting on large NFS networks, but here's an idea... Are you seeing any kind of packet loss/retransmissions? Take a look at netstat -s. When I last did this work it was with NFS over udp, but I think retransmitted packets will cause more performance loss with large packet sizes. I used to find machines with broken ethernet interfaces that would cause these kinds of problems. Nataraj
Alexandre Lecuyer wrote:> Hi list, > > We've had an interesting NFS performance issue with the following setup. > Clients : CentOS 5.5, kernel 2.6.18-194.el5 > Server : Solaris 10 kernel Generic_142901-08 > > Reading a file (not cached) with dd : dd if=file1 of=/dev/null > bs=1024k count=100 > performance is around.. 700 kB/s ! > > /proc/mounts shows rsize has been negotiated to 1mBHave you tested the same thing with a Linux NFS server? The CentOS 5.x kernel has a maximum server [rw]size of 32Kb, so you would need to use something with a more recent kernel to get [rw]sizes to be 1Mb. James Pearson