I'm doing some testing with loading up a new storage server. I have couple
ZFS filesystems exported via NFS over UDP.
I have a client machine (8.1 also) that has mounted those filesystems along with
some test PHP scripts that are doing a ton of read/write/fstat operations to
load it up.
If I ctrl-C to kill the scripts on the client I found that I can end up with
nfsd on the server stuck at 100% CPU and in the rc_lo state. /etc/rc.d/nfsd
restart does nothing.
# nfsstat -s -w 1 -W
GtAttr Lookup Rdlink Read Write Rename Access Rddir
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
Client complaining with:
kernel: nfs server 192.168.1.2:/tank/alantest2: not responding
Top on the server:
last pid: 4776; load averages: 1.24, 1.15, 1.15 up
0+23:04:46 18:25:26
53 processes: 2 running, 35 sleeping, 16 lock
CPU: 0.0% user, 0.0% nice, 25.0% system, 0.0% interrupt, 75.0% idle
Mem: 21M Active, 5812K Inact, 2029M Wired, 88K Cache, 8592K Buf, 5857M Free
Swap: 3942M Total, 3942M Free
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
922 root 44 0 5804K 1820K CPU2 2 63:52 100.00% {nfsd: service}
922 root 45 0 5804K 1820K *rc_lo 1 30:23 0.00% {nfsd: service}
922 root 48 0 5804K 1820K rpcsvc 1 30:20 0.00% {nfsd: service}
922 root 46 0 5804K 1820K *rc_lo 1 30:17 0.00% {nfsd: service}
922 root 48 0 5804K 1820K rpcsvc 1 30:17 0.00% {nfsd: service}
922 root 46 0 5804K 1820K rpcsvc 1 30:13 0.00% {nfsd: service}
922 root 45 0 5804K 1820K *rc_lo 2 30:13 0.00% {nfsd: service}
922 root 45 0 5804K 1820K *rc_lo 3 30:12 0.00% {nfsd: service}
922 root 44 0 5804K 1820K *rc_lo 0 30:10 0.00% {nfsd: service}
922 root 46 0 5804K 1820K *rc_lo 2 30:07 0.00% {nfsd: service}
922 root 46 0 5804K 1820K *rc_lo 0 30:03 0.00% {nfsd: service}
922 root 45 0 5804K 1820K *rc_lo 1 30:03 0.00% {nfsd: service}
922 root 46 0 5804K 1820K *rc_lo 2 30:03 0.00% {nfsd: service}
922 root 45 0 5804K 1820K *rc_lo 2 30:02 0.00% {nfsd: service}
922 root 45 0 5804K 1820K *rc_lo 1 29:59 0.00% {nfsd: master}
922 root 45 0 5804K 1820K *rc_lo 2 29:54 0.00% {nfsd: service}
922 root 46 0 5804K 1820K *rc_lo 0 29:54 0.00% {nfsd: service}
922 root 46 0 5804K 1820K *rc_lo 0 29:52 0.00% {nfsd: service}
922 root 47 0 5804K 1820K *rc_lo 2 29:46 0.00% {nfsd: service}
922 root 46 0 5804K 1820K *rc_lo 1 29:44 0.00% {nfsd: service}
> I'm doing some testing with loading up a new storage server. I have > couple ZFS filesystems exported via NFS over UDP. > > I have a client machine (8.1 also) that has mounted those filesystems > along with some test PHP scripts that are doing a ton of > read/write/fstat operations to load it up. > > If I ctrl-C to kill the scripts on the client I found that I can end > up with nfsd on the server stuck at 100% CPU and in the rc_lo state. > /etc/rc.d/nfsd restart does nothing. > > # nfsstat -s -w 1 -W > GtAttr Lookup Rdlink Read Write Rename Access Rddir > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 > > > Client complaining with: > kernel: nfs server 192.168.1.2:/tank/alantest2: not responding > > > > Top on the server: > > last pid: 4776; load averages: 1.24, 1.15, 1.15 up 0+23:04:46 18:25:26 > 53 processes: 2 running, 35 sleeping, 16 lock > CPU: 0.0% user, 0.0% nice, 25.0% system, 0.0% interrupt, 75.0% idle > Mem: 21M Active, 5812K Inact, 2029M Wired, 88K Cache, 8592K Buf, 5857M > Free > Swap: 3942M Total, 3942M Free > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 922 root 44 0 5804K 1820K CPU2 2 63:52 100.00% {nfsd: service} > 922 root 45 0 5804K 1820K *rc_lo 1 30:23 0.00% {nfsd: service} > 922 root 48 0 5804K 1820K rpcsvc 1 30:20 0.00% {nfsd: service} > 922 root 46 0 5804K 1820K *rc_lo 1 30:17 0.00% {nfsd: service} > 922 root 48 0 5804K 1820K rpcsvc 1 30:17 0.00% {nfsd: service} > 922 root 46 0 5804K 1820K rpcsvc 1 30:13 0.00% {nfsd: service} > 922 root 45 0 5804K 1820K *rc_lo 2 30:13 0.00% {nfsd: service} > 922 root 45 0 5804K 1820K *rc_lo 3 30:12 0.00% {nfsd: service} > 922 root 44 0 5804K 1820K *rc_lo 0 30:10 0.00% {nfsd: service} > 922 root 46 0 5804K 1820K *rc_lo 2 30:07 0.00% {nfsd: service} > 922 root 46 0 5804K 1820K *rc_lo 0 30:03 0.00% {nfsd: service} > 922 root 45 0 5804K 1820K *rc_lo 1 30:03 0.00% {nfsd: service} > 922 root 46 0 5804K 1820K *rc_lo 2 30:03 0.00% {nfsd: service} > 922 root 45 0 5804K 1820K *rc_lo 2 30:02 0.00% {nfsd: service} > 922 root 45 0 5804K 1820K *rc_lo 1 29:59 0.00% {nfsd: master} > 922 root 45 0 5804K 1820K *rc_lo 2 29:54 0.00% {nfsd: service} > 922 root 46 0 5804K 1820K *rc_lo 0 29:54 0.00% {nfsd: service} > 922 root 46 0 5804K 1820K *rc_lo 0 29:52 0.00% {nfsd: service} > 922 root 47 0 5804K 1820K *rc_lo 2 29:46 0.00% {nfsd: service} > 922 root 46 0 5804K 1820K *rc_lo 1 29:44 0.00% {nfsd: service} >I have a patch that I think might fix this. (Essentially the same bug was posted on another list recently.) I'll post it to Alan separately, but if anyone else wants to try it, it will be at http://people.freebsd.org/~rmacklem/freebsd8.1-patches/replay.patch in a few minutes. rick