>Submitter-Id: current-users
>Originator: Eugene Grosbein
>Organization: Svyaz Service
>Confidential: no
>Synopsis: repeatable 6.4-STABLE kernel panic: sleeping thread
>Severity: critical
>Priority: high
>Category: kern
>Class: sw-bug
>Release: FreeBSD 6.4-STABLE i386
>Environment:
System: FreeBSD eg.svzserv.kuzbass.ru 6.4-STABLE FreeBSD 6.4-STABLE #18: Mon Apr
6 12:56:06 KRAST 2009
eugen@eg.svzserv.kuzbass.ru:/usr/local/obj/usr/local/src/sys/EG i386
re(4) network driver
>Description:
1 April I've updated my 6.4-STABLE (running 19 March 2009 sources before)
to lastest RELENG_6 using standard source upgrade path
and now it cannot boot - panices just after inetd start.
It boots with kernel.old just fine. My kernel is monolithic
and there are no kernel modules loaded other than acpi.ko.
Here comes gdb backtrace:
Script started on Mon Apr 6 12:07:44 2009
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "i386-marcel-freebsd"...
Unread portion of the kernel message buffer:
<118> mousechar_start
<118>.
<118>Starting inetd.
Sleeping thread (tid 100084, pid 684) owns a non-sleepable lock
sched_switch(c4e74600,0,1,4c477be9,b39fb614,...) at 0xc053ddcf =
sched_switch+0x158
mi_switch(1,0) at 0xc0531483 = mi_switch+0x1d5
sleepq_switch(c07a7504,4,0,e752cb3c,c04ef432,...) at 0xc054e0f9 =
sleepq_switch+0x93
sleepq_wait_sig(c07a7504,c07a74e0,c07429df,101,0,...) at 0xc054e280 =
sleepq_wait_sig+0x21
cv_wait_sig(c07a7504,c07a74e0,e752cb78,8,e752cb58,...) at 0xc04ef432 =
cv_wait_sig+0x15a
kern_select(c4e74600,8,bfbfe8b0,0,0,...) at 0xc05549ae = kern_select+0x67d
select(c4e74600,e752cd04,14,c4e74600,2817f000,...) at 0xc0554327 = select+0x63
syscall(3b,3b,3b,bfbfedc0,bfbfee40,...) at 0xc070822d = syscall+0x34f
Xint0x80_syscall() at 0xc06f035f = Xint0x80_syscall+0x1f
--- syscall (93, FreeBSD ELF32, select), eip = 0x2816af63, esp = 0xbfbfdb8c, ebp
= 0xbfbfee78 ---
panic: sleeping thread
cpuid = 0
KDB: stack backtrace:
kdb_backtrace(c075ab91,0,c07427ff,e35d1bd0,0,...) at 0xc05470aa =
kdb_backtrace+0x2f
panic(c07427ff,ffffffff,2ac,c4b15a80,e35d1be8,...) at 0xc0528e09 = panic+0x129
propagate_priority(c4b15a80,c4e74600,c05511d8,c4b15a80,e35d1c38,...) at
0xc0550c49 = propagate_priority+0x69
turnstile_wait(c07abfec,c4e74600,0,0,4,...) at 0xc05517b8 = turnstile_wait+0x34b
_mtx_lock_sleep(c07abfec,c4b15a80,0,0,0,...) at 0xc051c240 =
_mtx_lock_sleep+0x10d
tcp_isn_tick(0,0,0,0,1ac3ffac,...) at 0xc0600b86 = tcp_isn_tick+0x4d
softclock(0,e35d1cd4,6,363f5101,c4b15a80,...) at 0xc0538396 = softclock+0x2f6
ithread_execute_handlers(c4b14648,c4b63080,0,0,0,...) at 0xc050a353 =
ithread_execute_handlers+0x162
ithread_loop(c4aee940,e35d1d38,0,0,0,...) at 0xc050a4ae = ithread_loop+0x64
fork_exit(c050a44a,c4aee940,e35d1d38) at 0xc0508d1e = fork_exit+0x7b
fork_trampoline() at 0xc06f036c = fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe35d1d6c, ebp = 0 ---
Uptime: 6s
Dumping 1023 MB (2 chunks)
chunk 0: 1MB (159 pages) ... ok
chunk 1: 1023MB (261872 pages) 1007 991 975 959 943 927 911 895 879 863 847
831 815 799 783 767 751 735 719 703 687 671 655 639 623 607 591 575 559 543 527
511 495 479 463 447 431 415 399 383 367 351 335 319 303 287 271 255 239 223 207
191 175 159 143 127 111 95 79 63 47 31 15
Reading symbols from /boot/modules/snd_hda.ko...done.
Loaded symbols for /boot/modules/snd_hda.ko
Reading symbols from /boot/modules/sound.ko...done.
Loaded symbols for /boot/modules/sound.ko
Reading symbols from /boot/modules/aio.ko...done.
Loaded symbols for /boot/modules/aio.ko
Reading symbols from /boot/modules/acpi.ko...done.
Loaded symbols for /boot/modules/acpi.ko
#0 doadump () at pcpu.h:165
165 __asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb) bt full
#0 doadump () at pcpu.h:165
No locals.
#1 0xc0528ae9 in boot (howto=260)
at /usr/local/src/sys/kern/kern_shutdown.c:410
first_buf_printf = 1
#2 0xc0528ec8 in panic (fmt=0xc07427ff "sleeping thread")
at /usr/local/src/sys/kern/kern_shutdown.c:566
td = (struct thread *) 0xc4b15a80
bootopt = 260
newpanic = 1
ap = 0xc4b15a80 "HF±Äà\215±Ä"
buf = "sleeping thread", '\0' <repeats 240 times>
#3 0xc0550c49 in propagate_priority (td=0xc4e74600)
at /usr/local/src/sys/kern/subr_turnstile.c:209
tc = (struct turnstile_chain *) 0xc4b15a80
ts = (struct turnstile *) 0xc4e73140
pri = 52
#4 0xc05517b8 in turnstile_wait (lock=0xc07abfec, owner=0x0, queue=0)
at /usr/local/src/sys/kern/subr_turnstile.c:715
tc = (struct turnstile_chain *) 0xc07a6a38
ts = (struct turnstile *) 0xc4e73140
td = (struct thread *) 0xc4b15a80
td1 = (struct thread *) 0xc4b15a80
#5 0xc051c240 in _mtx_lock_sleep (m=0xc07abfec, tid=3299957376, opts=0,
---Type <return> to continue, or q <return> to quit---
file=0x0, line=0) at /usr/local/src/sys/kern/kern_mutex.c:579
owner = (volatile struct thread *) 0xc4e74600
v = 0
#6 0xc0600b86 in tcp_isn_tick (xtp=0x0)
at /usr/local/src/sys/netinet/tcp_subr.c:1485
projected_offset = 0
#7 0xc0538396 in softclock (dummy=0x0)
at /usr/local/src/sys/kern/kern_timeout.c:274
c_func = (void (*)(void *)) 0xc0600b39 <tcp_isn_tick>
c_arg = (void *) 0x0
c_mtx = (struct mtx *) 0x0
c_flags = 22
c = (struct callout *) 0x0
bucket = (struct callout_tailq *) 0xd8b21598
curticks = 5545
steps = 0
depth = 3
mpcalls = 1
mtxcalls = 0
gcalls = 2
#8 0xc050a353 in ithread_execute_handlers (p=0xc4b14648, ie=0xc4b63080)
at /usr/local/src/sys/kern/kern_intr.c:682
ih = (struct intr_handler *) 0xc4b62880
ihn = (struct intr_handler *) 0xc4c4ea40
---Type <return> to continue, or q <return> to quit---
#9 0xc050a4ae in ithread_loop (arg=0xc4aee940)
at /usr/local/src/sys/kern/kern_intr.c:766
intr_event = (struct intr_thread *) 0xc4aee940
ie = (struct intr_event *) 0xc4b63080
td = (struct thread *) 0xc4b15a80
p = (struct proc *) 0xc4b14648
#10 0xc0508d1e in fork_exit (callout=0xc050a44a <ithread_loop>, arg=0x0,
frame=0x0) at /usr/local/src/sys/kern/kern_fork.c:788
p = (struct proc *) 0xc4b14648
td = (struct thread *) 0x0
#11 0xc06f036c in fork_trampoline ()
at /usr/local/src/sys/i386/i386/exception.s:208
No locals.
(kgdb) frame 6
#6 0xc0600b86 in tcp_isn_tick (xtp=0x0)
at /usr/local/src/sys/netinet/tcp_subr.c:1485
1485 INP_INFO_WLOCK(&tcbinfo);
(kgdb) l
1480 tcp_isn_tick(xtp)
1481 void *xtp;
1482 {
1483 u_int32_t projected_offset;
1484
1485 INP_INFO_WLOCK(&tcbinfo);
1486 projected_offset = isn_offset_old + ISN_BYTES_PER_SECOND / 100;
1487
1488 if (SEQ_GT(projected_offset, isn_offset))
1489 isn_offset = projected_offset;
(kgdb) quit
Script done on Mon Apr 6 12:08:54 2009
I've investigated the case and found that there was only one
commit to src/sys/netinet, that was ip_output.c,v 1.242.2.20
I've backed it out, rebuilt kernel and it does not panices anymore.
>How-To-Repeat:
Build and run RELENG_6 after 24 March 2009.
>Fix:
Unknown. Workaround is to backout this commit:
http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/netinet/ip_output.c.diff?r1=1.242.2.19;r2=1.242.2.20
Eugene Grosbein