While setting up a test for other purposes, I noticed some really
horrible NFS performance issues.
To explore this, I set up a test environment with two FreeBSD
9.2-RELEASE-p3 virtual machines running under KVM. The NFS server is
configured to serve a 2 gig mfs on /mnt.
The performance of the virtual network is outstanding:
Server:
$ iperf -c 172.20.20.169
------------------------------------------------------------
Client connecting to 172.20.20.169, TCP port 5001
TCP window size: 1.00 MByte (default)
------------------------------------------------------------
[ 3] local 172.20.20.162 port 59717 connected with 172.20.20.169 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 16.1 GBytes 13.8 Gbits/sec
$ iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 1.00 MByte (default)
------------------------------------------------------------
[ 4] local 172.20.20.162 port 5001 connected with 172.20.20.169 port 45655
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 15.8 GBytes 13.6 Gbits/sec
Client:
$ iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 1.00 MByte (default)
------------------------------------------------------------
[ 4] local 172.20.20.169 port 5001 connected with 172.20.20.162 port 59717
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 16.1 GBytes 13.8 Gbits/sec
^C$ iperf -c 172.20.20.162
------------------------------------------------------------
Client connecting to 172.20.20.162, TCP port 5001
TCP window size: 1.00 MByte (default)
------------------------------------------------------------
[ 3] local 172.20.20.169 port 45655 connected with 172.20.20.162 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 15.8 GBytes 13.6 Gbits/sec
The performance of the mfs filesystem on the server is also good.
Server:
$ sudo mdconfig -a -t swap -s 2g
md0
$ sudo newfs -U -b 4k -f 4k /dev/md0
/dev/md0: 2048.0MB (4194304 sectors) block size 4096, fragment size 4096
using 43 cylinder groups of 48.12MB, 12320 blks, 6160 inodes.
with soft updates
super-block backups (for fsck_ffs -b #) at:
144, 98704, 197264, 295824, 394384, 492944, 591504, 690064, 788624, 887184,
985744, 1084304, 1182864, 1281424, 1379984, 1478544, 1577104, 1675664,
1774224, 1872784, 1971344, 2069904, 2168464, 2267024, 2365584, 2464144,
2562704, 2661264, 2759824, 2858384, 2956944, 3055504, 3154064, 3252624,
3351184, 3449744, 3548304, 3646864, 3745424, 3843984, 3942544, 4041104,
4139664
$ sudo mount /dev/md0 /mnt
$ cd /mnt
$ sudo iozone -e -I -s 512m -r 4k -i 0 -i 1 -i 2
Iozone: Performance Test of File I/O
Version $Revision: 3.420 $
[...]
random random
KB reclen write rewrite read reread read write
524288 4 560145 1114593 933699 831902 56347
158904
iozone test complete.
But introduce NFS into the mix and everything falls apart.
Client:
$ sudo mount -o tcp,nfsv3 f12.phxi:/mnt /mnt
$ cd /mnt
$ sudo iozone -e -I -s 512m -r 4k -i 0 -i 1 -i 2
Iozone: Performance Test of File I/O
Version $Revision: 3.420 $
[...]
random random
KB reclen write rewrite read reread read write
524288 4 67246 2923 103295 1272407 172475
196
And the above took 48 minutes to run, compared to 14 seconds for the
local version. So it's 200x slower over NFS. The random write test
is over 800x slower. Of course NFS is slower, that's expected, but it
definitely wasn't this exaggerated in previous releases.
To emphasize that iozone reflects real workloads here, I tried doing
an svn co of the 9-STABLE source tree over NFS but after two hours it
was still in llvm so I gave up.
While all this not-much-of-anything NFS traffic is going on, both
systems are essentially idle. The process on the client sits in
"newnfs" wait state with nearly no CPU. The server is completely idle
except for the occasional 0.10% in an nfsd thread, which otherwise
spend their lives in rpcsvc wait state.
Server iostat:
$ iostat -x -w 10 md0
extended device statistics
device r/s w/s kr/s kw/s qlen svc_t %b
[...]
md0 0.0 36.0 0.0 0.0 0 1.2 0
md0 0.0 38.8 0.0 0.0 0 1.5 0
md0 0.0 73.6 0.0 0.0 0 1.0 0
md0 0.0 53.3 0.0 0.0 0 2.5 0
md0 0.0 33.7 0.0 0.0 0 1.1 0
md0 0.0 45.5 0.0 0.0 0 1.8 0
Server nfsstat:
$ nfsstat -s -w 10
GtAttr Lookup Rdlink Read Write Rename Access Rddir
[...]
0 0 0 471 816 0 0 0
0 0 0 480 751 0 0 0
0 0 0 481 36 0 0 0
0 0 0 469 550 0 0 0
0 0 0 485 814 0 0 0
0 0 0 467 503 0 0 0
0 0 0 473 345 0 0 0
Client nfsstat:
$ nfsstat -c -w 10
GtAttr Lookup Rdlink Read Write Rename Access Rddir
[...]
0 0 0 0 518 0 0 0
0 0 0 0 498 0 0 0
0 0 0 0 503 0 0 0
0 0 0 0 474 0 0 0
0 0 0 0 525 0 0 0
0 0 0 0 497 0 0 0
Server vmstat:
$ vmstat -w 10
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr vt0 vt1 in sy
cs us sy id
[...]
0 4 0 634M 6043M 37 0 0 0 1 0 0 0 1561 46
3431 0 2 98
0 4 0 640M 6042M 62 0 0 0 28 0 0 0 1598 94
3552 0 2 98
0 4 0 648M 6042M 38 0 0 0 0 0 0 0 1609 47
3485 0 1 99
0 4 0 648M 6042M 37 0 0 0 0 0 0 0 1615 46
3667 0 2 98
0 4 0 648M 6042M 37 0 0 0 0 0 0 0 1606 45
3678 0 2 98
0 4 0 648M 6042M 37 0 0 0 0 0 1 0 1561 45
3377 0 2 98
Client vmstat:
$ vmstat -w 10
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr md0 da0 in sy
cs us sy id
[...]
0 0 0 639M 593M 33 0 0 0 1237 0 0 0 281 5575
1043 0 3 97
0 0 0 639M 591M 0 0 0 0 712 0 0 0 235 122
889 0 2 98
0 0 0 639M 583M 0 0 0 0 571 0 0 1 227 120
851 0 2 98
0 0 0 639M 592M 198 0 0 0 1212 0 0 0 251 2497
950 0 3 97
0 0 0 639M 586M 0 0 0 0 614 0 0 0 250 121
924 0 2 98
0 0 0 639M 586M 0 0 0 0 765 0 0 0 250 120
918 0 3 97
Top on the KVM host says it is 93-95% idle and that each VM sits
around 7-10% CPU. So basically nobody is doing anything. There's no
visible bottleneck, and I've no idea where to go from here to figure
out what's going on.
Does anyone have any suggestions for debugging this?
Thanks!