Hi list, despite recent improvements with the nfs code, client performance still seems to be a problem. I am getting < 2 MB/sec where i would expect at least 10 MB/sec. My Setup: Machine | ANTSRV1 | ANTSRV2 =================================================Board | Tyan 2882-D | ASUS A7V8X-X CPU | Opteron 246 | Athlon XP1800+ Mem | 2G | 1G NIC | Broadcom BCM5704C | Intel PRO/1000MT OS:6.2-PRERELEASE #2 as of 01-OCT-2006 64Bit OS on ANTSRV1, 32Bit on ANTSRV2 The 2 machines are connected via a 1GB switch ===========================Trying UDP first ===========================root@antsrv1 [/tmp] # mount_nfs -U -L antsrv2:/export/disk2 /mnt root@antsrv1 [/tmp] # rm /mnt/x/* root@antsrv1 [/tmp] # time dd if=/dev/zero of=/mnt/x/10MB.dat bs=1M count=10 10+0 records in 10+0 records out 10485760 bytes transferred in 4.967248 secs (2110980 bytes/sec) real 0m5.253s user 0m0.005s sys 0m0.022s root@antsrv1 [/tmp] # time dd if=/dev/zero of=/mnt/x/100MB.dat bs=1M count=100 100+0 records in 100+0 records out 104857600 bytes transferred in 69.020366 secs (1519227 bytes/sec) real 1m9.276s user 0m0.028s sys 0m0.211s =============================Now TCP =============================root@antsrv1 [/tmp] # umount /mnt root@antsrv1 [/tmp] # mount_nfs -T -L antsrv2:/export/disk2 /mnt root@antsrv1 [/tmp] # rm /mnt/x/* root@antsrv1 [/tmp] # time dd if=/dev/zero of=/mnt/x/10MB.dat bs=1M count=10 10+0 records in 10+0 records out 10485760 bytes transferred in 5.289492 secs (1982376 bytes/sec) real 0m5.312s user 0m0.001s sys 0m0.023s root@antsrv1 [/tmp] # time dd if=/dev/zero of=/mnt/x/100MB.dat bs=1M count=100 100+0 records in 100+0 records out 104857600 bytes transferred in 58.715595 secs (1785856 bytes/sec) real 0m58.740s user 0m0.000s sys 0m0.204s ==============================SCP ==============================root@antsrv1 [/tmp] # scp 100MB.dat antsrv2:/tmp/x Password: 100MB.dat 100% 100MB 11.1MB/s 00:09 ==============================Iperf ==============================root@antsrv1 [~] # iperf -c antsrv2 ------------------------------------------------------------ Client connecting to antsrv2, TCP port 5001 TCP window size: 32.5 KByte (default) ------------------------------------------------------------ [ 3] local 134.102.176.16 port 59123 connected with 134.102.176.17 port 5001 [ 3] 0.0-10.0 sec 743 MBytes 623 Mbits/sec NFS from a Linux machine (Kernel 2.6.17.8) (connected via 10Mb LAN) yields a constant value of ~10MB/sec (TCP and UDP), so the FreeBSD NFS server seems to be ok. ================================Reversing the setup ================================root@antsrv2 [~] # mount_nfs -L -U antsrv1:/export/huge/x /mnt root@antsrv2 [~] # time dd if=/dev/zero of=/mnt/10MB.dat bs=1M count=10 10+0 records in 10+0 records out 10485760 bytes transferred in 0.533470 secs (19655758 bytes/sec) real 0m0.547s user 0m0.001s sys 0m0.085s root@antsrv2 [~] # time dd if=/dev/zero of=/mnt/100MB.dat bs=1M count=100 100+0 records in 100+0 records out 104857600 bytes transferred in 7.647475 secs (13711401 bytes/sec) real 0m7.673s user 0m0.000s sys 0m0.920s root@antsrv2 [~] # umount /mnt root@antsrv2 [~] # mount_nfs -L -T antsrv1:/export/huge/x /mnt root@antsrv2 [~] # rm /mnt/* root@antsrv2 [~] # time dd if=/dev/zero of=/mnt/10MB.dat bs=1M count=10 10+0 records in 10+0 records out 10485760 bytes transferred in 0.524917 secs (19976030 bytes/sec) real 0m0.557s user 0m0.000s sys 0m0.085s root@antsrv2 [~] # time dd if=/dev/zero of=/mnt/100MB.dat bs=1M count=100 100+0 records in 100+0 records out 104857600 bytes transferred in 5.723110 secs (18321787 bytes/sec) real 0m5.756s user 0m0.000s sys 0m0.704s This yields much better results. Could this be a 64bit-specific problem? Both machines have statd and lockd running. Any ideas where i could tune? -- Heinrich Rebehn University of Bremen Physics / Electrical and Electronics Engineering - Department of Telecommunications - Phone : +49/421/218-4664 Fax : -3341
Heinrich Rebehn wrote:> Hi list, > > despite recent improvements with the nfs code, client performance still > seems to be a problem. I am getting < 2 MB/sec where i would expect at > least 10 MB/sec. > > My Setup: > > Machine | ANTSRV1 | ANTSRV2 > =================================================> Board | Tyan 2882-D | ASUS A7V8X-X > CPU | Opteron 246 | Athlon XP1800+ > Mem | 2G | 1G > NIC | Broadcom BCM5704C | Intel PRO/1000MT > > OS:6.2-PRERELEASE #2 as of 01-OCT-2006 > 64Bit OS on ANTSRV1, 32Bit on ANTSRV2 > > The 2 machines are connected via a 1GB switch > > ===========================> Trying UDP first > ===========================> root@antsrv1 [/tmp] # mount_nfs -U -L antsrv2:/export/disk2 /mnt > root@antsrv1 [/tmp] # rm /mnt/x/* > root@antsrv1 [/tmp] # time dd if=/dev/zero of=/mnt/x/10MB.dat bs=1M > count=10 > 10+0 records in > 10+0 records out > 10485760 bytes transferred in 4.967248 secs (2110980 bytes/sec) > > real 0m5.253s > user 0m0.005s > sys 0m0.022s > root@antsrv1 [/tmp] # time dd if=/dev/zero of=/mnt/x/100MB.dat bs=1M > count=100 > 100+0 records in > 100+0 records out > 104857600 bytes transferred in 69.020366 secs (1519227 bytes/sec) > > real 1m9.276s > user 0m0.028s > sys 0m0.211s > > =============================> Now TCP > =============================> root@antsrv1 [/tmp] # umount /mnt > root@antsrv1 [/tmp] # mount_nfs -T -L antsrv2:/export/disk2 /mnt > root@antsrv1 [/tmp] # rm /mnt/x/* > root@antsrv1 [/tmp] # time dd if=/dev/zero of=/mnt/x/10MB.dat bs=1M > count=10 > 10+0 records in > 10+0 records out > 10485760 bytes transferred in 5.289492 secs (1982376 bytes/sec) > > real 0m5.312s > user 0m0.001s > sys 0m0.023s > root@antsrv1 [/tmp] # time dd if=/dev/zero of=/mnt/x/100MB.dat bs=1M > count=100 > 100+0 records in > 100+0 records out > 104857600 bytes transferred in 58.715595 secs (1785856 bytes/sec) > > real 0m58.740s > user 0m0.000s > sys 0m0.204s > > ==============================> SCP > ==============================> root@antsrv1 [/tmp] # scp 100MB.dat antsrv2:/tmp/x > Password: > 100MB.dat 100% 100MB > 11.1MB/s 00:09 > > ==============================> Iperf > ==============================> root@antsrv1 [~] # iperf -c antsrv2 > ------------------------------------------------------------ > Client connecting to antsrv2, TCP port 5001 > TCP window size: 32.5 KByte (default) > ------------------------------------------------------------ > [ 3] local 134.102.176.16 port 59123 connected with 134.102.176.17 port > 5001 > [ 3] 0.0-10.0 sec 743 MBytes 623 Mbits/sec > > NFS from a Linux machine (Kernel 2.6.17.8) (connected via 10Mb LAN) > yields a constant value of ~10MB/sec (TCP and UDP), so the FreeBSD NFS > server seems to be ok. > > ================================> Reversing the setup > ================================> root@antsrv2 [~] # mount_nfs -L -U antsrv1:/export/huge/x /mnt > root@antsrv2 [~] # time dd if=/dev/zero of=/mnt/10MB.dat bs=1M count=10 > 10+0 records in > 10+0 records out > 10485760 bytes transferred in 0.533470 secs (19655758 bytes/sec) > > real 0m0.547s > user 0m0.001s > sys 0m0.085s > root@antsrv2 [~] # time dd if=/dev/zero of=/mnt/100MB.dat bs=1M count=100 > 100+0 records in > 100+0 records out > 104857600 bytes transferred in 7.647475 secs (13711401 bytes/sec) > > real 0m7.673s > user 0m0.000s > sys 0m0.920s > root@antsrv2 [~] # umount /mnt > root@antsrv2 [~] # mount_nfs -L -T antsrv1:/export/huge/x /mnt > root@antsrv2 [~] # rm /mnt/* > root@antsrv2 [~] # time dd if=/dev/zero of=/mnt/10MB.dat bs=1M count=10 > 10+0 records in > 10+0 records out > 10485760 bytes transferred in 0.524917 secs (19976030 bytes/sec) > > real 0m0.557s > user 0m0.000s > sys 0m0.085s > root@antsrv2 [~] # time dd if=/dev/zero of=/mnt/100MB.dat bs=1M count=100 > 100+0 records in > 100+0 records out > 104857600 bytes transferred in 5.723110 secs (18321787 bytes/sec) > > real 0m5.756s > user 0m0.000s > sys 0m0.704s > > > This yields much better results. Could this be a 64bit-specific problem? > > Both machines have statd and lockd running. > > Any ideas where i could tune? >Hmm, no one? Is 2MB/s on a Gbit link all i can expect on FreeBSD? --Heinrich
Heinrich Rebehn wrote:> Hi list, > > despite recent improvements with the nfs code, client performance still > seems to be a problem. I am getting < 2 MB/sec where i would expect at > least 10 MB/sec. > > My Setup: > > Machine | ANTSRV1 | ANTSRV2 > =================================================> Board | Tyan 2882-D | ASUS A7V8X-X > CPU | Opteron 246 | Athlon XP1800+ > Mem | 2G | 1G > NIC | Broadcom BCM5704C | Intel PRO/1000MT > > OS:6.2-PRERELEASE #2 as of 01-OCT-2006 > 64Bit OS on ANTSRV1, 32Bit on ANTSRV2 > > The 2 machines are connected via a 1GB switch > > ===========================> Trying UDP first > ===========================> root@antsrv1 [/tmp] # mount_nfs -U -L antsrv2:/export/disk2 /mnt > root@antsrv1 [/tmp] # rm /mnt/x/* > root@antsrv1 [/tmp] # time dd if=/dev/zero of=/mnt/x/10MB.dat bs=1M > count=10 > 10+0 records in > 10+0 records out > 10485760 bytes transferred in 4.967248 secs (2110980 bytes/sec) > > real 0m5.253s > user 0m0.005s > sys 0m0.022s > root@antsrv1 [/tmp] # time dd if=/dev/zero of=/mnt/x/100MB.dat bs=1M > count=100 > 100+0 records in > 100+0 records out > 104857600 bytes transferred in 69.020366 secs (1519227 bytes/sec) > > real 1m9.276s > user 0m0.028s > sys 0m0.211s > > =============================> Now TCP > =============================> root@antsrv1 [/tmp] # umount /mnt > root@antsrv1 [/tmp] # mount_nfs -T -L antsrv2:/export/disk2 /mnt > root@antsrv1 [/tmp] # rm /mnt/x/* > root@antsrv1 [/tmp] # time dd if=/dev/zero of=/mnt/x/10MB.dat bs=1M > count=10 > 10+0 records in > 10+0 records out > 10485760 bytes transferred in 5.289492 secs (1982376 bytes/sec) > > real 0m5.312s > user 0m0.001s > sys 0m0.023s > root@antsrv1 [/tmp] # time dd if=/dev/zero of=/mnt/x/100MB.dat bs=1M > count=100 > 100+0 records in > 100+0 records out > 104857600 bytes transferred in 58.715595 secs (1785856 bytes/sec) > > real 0m58.740s > user 0m0.000s > sys 0m0.204s > > ==============================> SCP > ==============================> root@antsrv1 [/tmp] # scp 100MB.dat antsrv2:/tmp/x > Password: > 100MB.dat 100% 100MB > 11.1MB/s 00:09 > > ==============================> Iperf > ==============================> root@antsrv1 [~] # iperf -c antsrv2 > ------------------------------------------------------------ > Client connecting to antsrv2, TCP port 5001 > TCP window size: 32.5 KByte (default) > ------------------------------------------------------------ > [ 3] local 134.102.176.16 port 59123 connected with 134.102.176.17 port > 5001 > [ 3] 0.0-10.0 sec 743 MBytes 623 Mbits/sec > > NFS from a Linux machine (Kernel 2.6.17.8) (connected via 10Mb LAN) > yields a constant value of ~10MB/sec (TCP and UDP), so the FreeBSD NFS > server seems to be ok. > > ================================> Reversing the setup > ================================> root@antsrv2 [~] # mount_nfs -L -U antsrv1:/export/huge/x /mnt > root@antsrv2 [~] # time dd if=/dev/zero of=/mnt/10MB.dat bs=1M count=10 > 10+0 records in > 10+0 records out > 10485760 bytes transferred in 0.533470 secs (19655758 bytes/sec) > > real 0m0.547s > user 0m0.001s > sys 0m0.085s > root@antsrv2 [~] # time dd if=/dev/zero of=/mnt/100MB.dat bs=1M count=100 > 100+0 records in > 100+0 records out > 104857600 bytes transferred in 7.647475 secs (13711401 bytes/sec) > > real 0m7.673s > user 0m0.000s > sys 0m0.920s > root@antsrv2 [~] # umount /mnt > root@antsrv2 [~] # mount_nfs -L -T antsrv1:/export/huge/x /mnt > root@antsrv2 [~] # rm /mnt/* > root@antsrv2 [~] # time dd if=/dev/zero of=/mnt/10MB.dat bs=1M count=10 > 10+0 records in > 10+0 records out > 10485760 bytes transferred in 0.524917 secs (19976030 bytes/sec) > > real 0m0.557s > user 0m0.000s > sys 0m0.085s > root@antsrv2 [~] # time dd if=/dev/zero of=/mnt/100MB.dat bs=1M count=100 > 100+0 records in > 100+0 records out > 104857600 bytes transferred in 5.723110 secs (18321787 bytes/sec) > > real 0m5.756s > user 0m0.000s > sys 0m0.704s > > > This yields much better results. Could this be a 64bit-specific problem? > > Both machines have statd and lockd running. > > Any ideas where i could tune? >Hmm, no one? Is 2MB/s on a Gbit link all i can expect on FreeBSD? --Heinrich