freebsd stable - May 2004 - system command hangs (unkillable); ps shows 'sbwait' state?

Hi,

I'm using fairly recent FreeBSD-stable on intel PC.

Within last few days, I encountered two hangs of a system command, that
I was unable to resolve. I could not kill the command, even a 'kill -9'
did not work.

1. I had a 2.5 Gb disk mounted on /home/software.
    As root, I overloaded the filesystem, with negative percentage left
    on the device (from df command). So as root, I did a 'rm -rf' in
    /home/software, followed by a 'df -h'. But the df command gave no
    response and became unkillable by any means (ctrl-C, kill -9 <pid>).
    Using 'ps', I found the df command in the 'sbwait' state.

2. I had a usb device mounted as /dev/da0s1 on /mnt. Mounting (as root)
    went all well, but when I unmounted it, as root, the umount command
    hanged, again the umount command was in sbwait state.
    In this case it was even worse: when I killed the xterminal
    where the umount command was hanging, the whole system froze.
    Only power off/on helped me out here.

I don't know what happened; don't know how to further investigate this.
Has somebody else similar experiences? Is stability going down for Stable
kernel?


Regards,
Rob.

On Wed, 19 May 2004, Rob wrote:
> I'm using fairly recent FreeBSD-stable on intel PC.
> 
> Within last few days, I encountered two hangs of a system command, that
> I was unable to resolve. I could not kill the command, even a 'kill
-9'
> did not work. 
I don't remember seeing any commits to -STABLE that seem like likely
candidates for a new change causing this problem.  I also haven't seen any
increased instability now that the twe driver problems have been fixed by
the vendor.  Have you made any local configuration or load changes?  In
particular, has the general system load on your system changes recently
(more web traffic, more I/O) in a measurable way?
> 1. I had a 2.5 Gb disk mounted on /home/software.
>     As root, I overloaded the filesystem, with negative percentage left
>     on the device (from df command). So as root, I did a 'rm -rf'
in
>     /home/software, followed by a 'df -h'. But the df command gave
no
>     response and became unkillable by any means (ctrl-C, kill -9
<pid>).
>     Using 'ps', I found the df command in the 'sbwait'
state.
> 
> 2. I had a usb device mounted as /dev/da0s1 on /mnt. Mounting (as root)
>     went all well, but when I unmounted it, as root, the umount command
>     hanged, again the umount command was in sbwait state.
>     In this case it was even worse: when I killed the xterminal
>     where the umount command was hanging, the whole system froze.
>     Only power off/on helped me out here.
> 
> I don't know what happened; don't know how to further investigate
this.
> Has somebody else similar experiences? Is stability going down for
> Stable kernel? 
Could you take a look at the instructions in the Handbook on setting up
for kernel debugging, compile the kernel with DDB, and generate stack
traces for the hung processes + the output of "show lockedvnodes"? 
Also,
if you can get a core dump, it might be interesting to see the output of
netstat -mb on the core.  Finally, are you using any features like NIS or
NFS?  Having umount stuck in sbwait sounds like a fairly unusual failure
mode unless you're using NFS.

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
robert@fledge.watson.org      Senior Research Scientist, McAfee Research