freebsd stable - May 2010 - hung on ufs vnode lock, was Re: NFS trouble on 7.3-STABLE i386

On Tue, 25 May 2010, Mark Morley wrote:
> On Fri, 21 May 2010 11:32:33 -0400 (EDT) Rick Macklem  wrote: On Fri, 21
May 2010, Mark Morley wrote:
>
>> Having an issue with a file server here (7.3-STABLE i386)
>>
>> The nfsd processes are hanging.  Client access to the nfs shares stops
working and the nfsd processes on the server cannot be killed by any means. 
There are no errors showing up anywhere on the server.  The network connection
to the server seems fine (ie: anything other than nfs traffic seems ok). 
Rebooting the server fixes the problem for a while, but it doesn't reboot
easily.  It times out on terminating the nfsd processes.  When it finally does
reboot the file system isn't marked clean, resulting in a long wait for fsck
(although it doesn't find any problems, it's a multi terrabyte share and
it takes a while).
>>
>> This morning it did it again.  This time I tried manually killing nfsd
but nothing I did would make them die.  No errors.
>>
> Next time it happens, do a "ps axlH" to see what the nfsd threads
are
> waiting for. It might give you a hint as to what is happening.
>
> Ok, it did it again.  ps axlH shows all the nfsd processes stuck in the
_ufs_ state.  The server isn't doing anything else, no other processes seem
to be monopolizing resources or disks in any way.
If the nfsd threads are sleeping on WCHAN "ufs", I think that means
that
they are waiting for a ufs vnode lock. I don't know what has changed
between FreeBSD7.1 and FreeBSD7.3 that might have caused this. I changed
the Subject: line in the hopes that someone who might know the answer to
this will take a look.
>
> rpcinfo doesn't show anything amiss as far as I can tell (ie: rpc is
running)
>
> After a reboot, one of the 32 nfsd's almost immediately goes into the
"ufs" state and never leaves it (and never racks up and CPU time
either).  The others are fine.  Slowly over time more and more enter this state.
When I rebooted it today, all but one were in that state.  The clients were
bogging down, presumably because the one and only functioning nfsd was
overworked.
>
> One client is running 8.1-prerelease as a test, and that particular client
only will start getting lots of timeouts accessing the nfs share (even with less
load than the other clients).  Just in case it's tickling something on the
server I've shut it down this time and I'm leaving it off for the time
being.
>
I don't think that the 8.1-prerelease client is an issue. It's just that
the FreeBSD8 krpc likes to generate the "not responding" messages more
agreesively. They are pretty well meaningless, imho.

rick

On Tuesday 25 May 2010 8:24:58 pm Rick Macklem wrote:
> 
> On Tue, 25 May 2010, Mark Morley wrote:
> 
> > On Fri, 21 May 2010 11:32:33 -0400 (EDT) Rick Macklem  wrote: On Fri,
21
May 2010, Mark Morley wrote:
> >
> >> Having an issue with a file server here (7.3-STABLE i386)
> >>
> >> The nfsd processes are hanging.  Client access to the nfs shares
stops
working and the nfsd processes on the server cannot be killed by any means.  
There are no errors showing up anywhere on the server.  The network connection 
to the server seems fine (ie: anything other than nfs traffic seems ok).  
Rebooting the server fixes the problem for a while, but it doesn't reboot 
easily.  It times out on terminating the nfsd processes.  When it finally does 
reboot the file system isn't marked clean, resulting in a long wait for fsck
(although it doesn't find any problems, it's a multi terrabyte share and
it
takes a while).
> >>
> >> This morning it did it again.  This time I tried manually killing
nfsd
but nothing I did would make them die.  No errors.
> >>
> > Next time it happens, do a "ps axlH" to see what the nfsd
threads are
> > waiting for. It might give you a hint as to what is happening.
> >
> > Ok, it did it again.  ps axlH shows all the nfsd processes stuck in
the
_ufs_ state.  The server isn't doing anything else, no other processes seem
to
be monopolizing resources or disks in any way.
> 
> If the nfsd threads are sleeping on WCHAN "ufs", I think that
means that
> they are waiting for a ufs vnode lock. I don't know what has changed
> between FreeBSD7.1 and FreeBSD7.3 that might have caused this. I changed
> the Subject: line in the hopes that someone who might know the answer to
> this will take a look.
If you can break into ddb, you can use 'show sleepchain' to investigate.
If
you built the kernel with debug symbols you can use kgdb to investigate as 
well.  I have something similar to 'show sleepchain' in the gdb scripts
at
www.freebsd.org/~jhb/gdb/.  You can source the gdb6 file and do 'sleepchain 
<pid>' to see what locks a given thread is blocked on.

-- 
John Baldwin