freebsd stable - Dec 2011 - Spinlock panic in FreeBSD 7

(This was originally posted to freebsd-hackers, I'm reposting following 
email suggestions.)

We've observed a panic in FreeBSD 7 (7.2-PRERELEASE FreeBSD) several 
times that we've not been able to track down.  Upgrading is not an 
option at this time.

Does this look at all familiar to anyone?  Here's an example stack trace 
after panic:

spin lock 0xffffffff8086bdc0 (smp rendezvous) held by 0xffffff0006d1f000 
(tid 100060) too long
panic: spin lock held too long
cpuid = 0
KDB: stack backtrace:
db_trace_self_wrapper() at 0xffffffff8019120a = db_trace_self_wrapper+0x2a
panic() at 0xffffffff80308797 = panic+0x187
_mtx_lock_spin_failed() at 0xffffffff802fbda9 = _mtx_lock_spin_failed+0x39
_mtx_lock_spin() at 0xffffffff802fbe4e = _mtx_lock_spin+0x9e
_mtx_lock_spin_flags() at 0xffffffff802fc354 = _mtx_lock_spin_flags+0x104
smp_rendezvous_cpus() at 0xffffffff80340cb3 = smp_rendezvous_cpus+0xd3
xcall() at 0xffffffff80ad755e = xcall+0x3e
cyclic_remove_here() at 0xffffffff80ad7715 = cyclic_remove_here+0x1a5
cyclic_remove() at 0xffffffff80ad7a0f = cyclic_remove+0x5f
profile_disable() at 0xffffffff80acf0e5 = profile_disable+0x15
dtrace_state_destroy() at 0xffffffff80adfabd = dtrace_state_destroy+0x35d
dtrace_close() at 0xffffffff80adffed = dtrace_close+0x8d
devfs_close() at 0xffffffff802a825d = devfs_close+0x2dd
vn_close() at 0xffffffff8039cb06 = vn_close+0xb6
vn_closefile() at 0xffffffff8039cc00 = vn_closefile+0x80
devfs_close_f() at 0xffffffff802a5738 = devfs_close_f+0x28
fdrop() at 0xffffffff802d98bb = fdrop+0xdb
closef() at 0xffffffff802db2f9 = closef+0x29
fdfree() at 0xffffffff802dc061 = fdfree+0x161
exit1() at 0xffffffff802e56b2 = exit1+0x2c2
sigexit() at 0xffffffff8030a86f = sigexit+0x8f
postsig() at 0xffffffff8030b6ce = postsig+0x38e
ast() at 0xffffffff803425f7 = ast+0x337
Xfast_syscall() at 0xffffffff80494efd = Xfast_syscall+0xdd
--- syscall (32, FreeBSD ELF64, getsockname), rip = 0x800df4d5c, rsp = 
0x7fffffffe398, rbp = 0x5 ---
KDB: enter: panic

  The panic always shows up from a syscall, and almost always from 
syscall 32, getsockname, but we've also observed it with syscall 5.
-- 

Charles R. (Charlie) Martin
Senior Software Engineer
SGI logo
1900 Pike Road
Longmont, CO 80501
Phone: 303-532-0209
E-Mail: CRMartin@sgi.com <mailto:CRMartin@sgi.com>
Website: www.sgi.com <http://www.sgi.com>

on 17/12/2011 00:38 Charlie Martin said the following:
> (This was originally posted to freebsd-hackers, I'm reposting following
email
> suggestions.)
> 
> We've observed a panic in FreeBSD 7 (7.2-PRERELEASE FreeBSD) several
times that
> we've not been able to track down.  Upgrading is not an option at this
time.
> 
> Does this look at all familiar to anyone?  Here's an example stack
trace after
> panic:
See r219502.
> spin lock 0xffffffff8086bdc0 (smp rendezvous) held by 0xffffff0006d1f000
(tid
> 100060) too long
> panic: spin lock held too long
> cpuid = 0
> KDB: stack backtrace:
> db_trace_self_wrapper() at 0xffffffff8019120a = db_trace_self_wrapper+0x2a
> panic() at 0xffffffff80308797 = panic+0x187
> _mtx_lock_spin_failed() at 0xffffffff802fbda9 = _mtx_lock_spin_failed+0x39
> _mtx_lock_spin() at 0xffffffff802fbe4e = _mtx_lock_spin+0x9e
> _mtx_lock_spin_flags() at 0xffffffff802fc354 = _mtx_lock_spin_flags+0x104
> smp_rendezvous_cpus() at 0xffffffff80340cb3 = smp_rendezvous_cpus+0xd3
> xcall() at 0xffffffff80ad755e = xcall+0x3e
> cyclic_remove_here() at 0xffffffff80ad7715 = cyclic_remove_here+0x1a5
> cyclic_remove() at 0xffffffff80ad7a0f = cyclic_remove+0x5f
> profile_disable() at 0xffffffff80acf0e5 = profile_disable+0x15
> dtrace_state_destroy() at 0xffffffff80adfabd = dtrace_state_destroy+0x35d
> dtrace_close() at 0xffffffff80adffed = dtrace_close+0x8d
> devfs_close() at 0xffffffff802a825d = devfs_close+0x2dd
> vn_close() at 0xffffffff8039cb06 = vn_close+0xb6
> vn_closefile() at 0xffffffff8039cc00 = vn_closefile+0x80
> devfs_close_f() at 0xffffffff802a5738 = devfs_close_f+0x28
> fdrop() at 0xffffffff802d98bb = fdrop+0xdb
> closef() at 0xffffffff802db2f9 = closef+0x29
> fdfree() at 0xffffffff802dc061 = fdfree+0x161
> exit1() at 0xffffffff802e56b2 = exit1+0x2c2
> sigexit() at 0xffffffff8030a86f = sigexit+0x8f
> postsig() at 0xffffffff8030b6ce = postsig+0x38e
> ast() at 0xffffffff803425f7 = ast+0x337
> Xfast_syscall() at 0xffffffff80494efd = Xfast_syscall+0xdd
> --- syscall (32, FreeBSD ELF64, getsockname), rip = 0x800df4d5c, rsp >
0x7fffffffe398, rbp = 0x5 ---
> KDB: enter: panic
> 
>  The panic always shows up from a syscall, and almost always from syscall
32,
> getsockname, but we've also observed it with syscall 5.

-- 
Andriy Gapon