thr3ads.net - freebsd stable - kern/134584: [panic] spin lock held too long [Jul 2009]

If this information is useful, please help other people find it:
Share via:

barbara

2009-Jul-26 06:47 UTC

kern/134584: [panic] spin lock held too long

It happened again, on shutdown.
As the previous time, it happened after a high (for a desktop) uptime and, if it
could matter, after running net-p2p/transmission-gtk2 for several hours.
I don't know if it's related, but often quitting transmission,
doesn't terminate the process. Sometimes it end after several minutes the
gui exited, sometimes it's still running after hours.
I've noticed it as the destination folder is on a manually mounted device
and I can't umount it as fstat reports the device used by a transmission
process.
So I often have to kill it.
This happened both the time I had this kind of panic.

Barbara


Script started on Sun Jul 26 08:22:33 2009
You have mail.
satanasso# kgdb kernel.debug /var/crash/vmcore.17
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for
details.
This GDB was configured as "i386-marcel-freebsd"...

Unread portion of the kernel message buffer:
<118>Stopping devd.
<118>Writing entropy file:
<118>.
<118>Terminated
<118>.
<118>Jul 25 17:45:18 satanasso syslogd: exiting on signal 15
Waiting (max 60 seconds) for system process `vnlru' to stop...done
Waiting (max 60 seconds) for system process `bufdaemon' to stop...done
Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining...7 6 5 3 1 2 2 0 0 done
All buffers synced.
Uptime: 22h41m12s
Rebooting...
cpu_reset: Stopping other CPUs
spin lock 0xc08ae540 (sched lock 1) held by 0xc588f480 (tid 100006) too long
panic: spin lock held too long
cpuid = 0
Uptime: 22h41m14s
Physical memory: 2031 MB
Dumping 171 MB: (CTRL-C to abort)  156 140 124 108 92 76 60 44 28 12

Reading symbols from /boot/kernel/linux.ko...Reading symbols from
/boot/kernel/linux.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/linux.ko
Reading symbols from /boot/modules/nvidia.ko...done.
Loaded symbols for /boot/modules/nvidia.ko
Reading symbols from /boot/kernel/acpi.ko...Reading symbols from
/boot/kernel/acpi.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/acpi.ko
Reading symbols from /boot/kernel/linprocfs.ko...Reading symbols from
/boot/kernel/linprocfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/linprocfs.ko
Reading symbols from /boot/kernel/logo_saver.ko...Reading symbols from
/boot/kernel/logo_saver.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/logo_saver.ko
Reading symbols from /boot/kernel/ntfs.ko...Reading symbols from
/boot/kernel/ntfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ntfs.ko
#0  doadump () at pcpu.h:196
196		__asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb) bt
#0  doadump () at pcpu.h:196
#1  0xc059b097 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418
#2  0xc059b36c in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:574
#3  0xc058bd1f in _mtx_lock_spin_failed (m=0x0) at
/usr/src/sys/kern/kern_mutex.c:449
#4  0xc058c425 in _thread_lock_flags (td=0xc588f480, opts=0, file=0xc08200c8
"/usr/src/sys/kern/kern_intr.c", line=801) at
/usr/src/sys/kern/kern_mutex.c:537
#5  0xc0576787 in intr_event_schedule_thread (ie=Variable "ie" is not
available.
) at /usr/src/sys/kern/kern_intr.c:801
#6  0xc0577aa5 in swi_sched (cookie=0xc5880e00, flags=Variable "flags"
is not available.
) at /usr/src/sys/kern/kern_intr.c:1028
#7  0xc055a6df in hardclock (usermode=0, pc=3229503748) at
/usr/src/sys/kern/kern_clock.c:339
#8  0xc07ce23a in lapic_handle_timer (frame=0xc54aaba0) at
/usr/src/sys/i386/i386/local_apic.c:705
#9  0xc07c6c9f in Xtimerint () at apic_vector.s:108
#10 0xc07e5104 in DELAY (n=1000000) at cpufunc.h:352
#11 0xc07e2c03 in cpu_reset () at /usr/src/sys/i386/i386/vm_machdep.c:594
#12 0xc059a753 in shutdown_reset (junk=0x0, howto=0) at
/usr/src/sys/kern/kern_shutdown.c:493
#13 0xc059b117 in boot (howto=0) at /usr/src/sys/kern/kern_shutdown.c:421
#14 0xc059b247 in reboot (td=0xc588fd80, uap=0xc54aacfc) at
/usr/src/sys/kern/kern_shutdown.c:172
#15 0xc07e07d5 in syscall (frame=0xc54aad38) at
/usr/src/sys/i386/i386/trap.c:1089
#16 0xc07c6720 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:262
#17 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) thread apply all bt

Thread 52 (Thread 100051):
#0  sched_switch (td=0xc5c91d80, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not
available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05d03d7 in sleepq_timedwait (wchan=0xc08bd300) at
/usr/src/sys/kern/subr_sleepqueue.c:615
#4  0xc05a3483 in _sleep (ident=0xc08bd300, lock=0xc08bd2c0, priority=68,
wmesg=0xc0833855 "sdflush", timo=1000) at
/usr/src/sys/kern/kern_synch.c:226
#5  0xc076407e in softdep_flush () at /usr/src/sys/ufs/ffs/ffs_softdep.c:773
#6  0xc0574587 in fork_exit (callout=0xc0763cb0 <softdep_flush>, arg=0x0,
frame=0xc5617d38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 51 (Thread 100050):
#0  sched_switch (td=0xc5a616c0, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not
available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05cfec6 in sleepq_wait (wchan=0xc5abe36c) at
/usr/src/sys/kern/subr_sleepqueue.c:580
#4  0xc05a34a5 in _sleep (ident=0xc5abe36c, lock=0xc5abe350, priority=104,
wmesg=0xc0820618 "ktsusp", timo=0) at
/usr/src/sys/kern/kern_synch.c:230
#5  0xc057e632 in kthread_suspend_check (p=0xc5abe2c0) at
/usr/src/sys/kern/kern_kthread.c:197
#6  0xc0620b1e in vnlru_proc () at /usr/src/sys/kern/vfs_subr.c:724
#7  0xc0574587 in fork_exit (callout=0xc0620ad0 <vnlru_proc>, arg=0x0,
frame=0xc5614d38) at /usr/src/sys/kern/kern_fork.c:811
#8  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 50 (Thread 100049):
#0  sched_switch (td=0xc5a61900, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not
available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05cfec6 in sleepq_wait (wchan=0xc5abe62c) at
/usr/src/sys/kern/subr_sleepqueue.c:580
#4  0xc05a34a5 in _sleep (ident=0xc5abe62c, lock=0xc5abe610, priority=104,
wmesg=0xc0820618 "ktsusp", timo=0) at
/usr/src/sys/kern/kern_synch.c:230
#5  0xc057e632 in kthread_suspend_check (p=0xc5abe580) at
/usr/src/sys/kern/kern_kthread.c:197
#6  0xc061f9c2 in sched_sync () at /usr/src/sys/kern/vfs_subr.c:1719
#7  0xc0574587 in fork_exit (callout=0xc061f8a0 <sched_sync>, arg=0x0,
frame=0xc5611d38) at /usr/src/sys/kern/kern_fork.c:811
#8  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 49 (Thread 100048):
#0  sched_switch (td=0xc5a61b40, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not
available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05cfec6 in sleepq_wait (wchan=0xc5abe8ec) at
/usr/src/sys/kern/subr_sleepqueue.c:580
#4  0xc05a34a5 in _sleep (ident=0xc5abe8ec, lock=0xc5abe8d0, priority=104,
wmesg=0xc0820618 "ktsusp", timo=0) at
/usr/src/sys/kern/kern_synch.c:230
#5  0xc057e632 in kthread_suspend_check (p=0xc5abe840) at
/usr/src/sys/kern/kern_kthread.c:197
#6  0xc0609c4b in buf_daemon () at /usr/src/sys/kern/vfs_bio.c:2155
#7  0xc0574587 in fork_exit (callout=0xc0609b70 <buf_daemon>, arg=0x0,
frame=0xc560ed38) at /usr/src/sys/kern/kern_fork.c:811
#8  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 48 (Thread 100047):
#0  sched_switch (td=0xc5a61d80, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not
available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05d03d7 in sleepq_timedwait (wchan=0xc08bdf14) at
/usr/src/sys/kern/subr_sleepqueue.c:615
#4  0xc05a3483 in _sleep (ident=0xc08bdf14, lock=0xc08bdad0, priority=0,
wmesg=0xc0836ae9 "pgzero", timo=300000) at
/usr/src/sys/kern/kern_synch.c:226
#5  0xc07a155c in vm_pagezero (arg=0x0) at /usr/src/sys/vm/vm_zeroidle.c:136
#6  0xc0574587 in fork_exit (callout=0xc07a1480 <vm_pagezero>, arg=0x0,
frame=0xc560bd38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 47 (Thread 100046):
#0  sched_switch (td=0xc5aa1000, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not
available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
---Type <return> to continue, or q <return> to quit---
#3  0xc05cfec6 in sleepq_wait (wchan=0xc08bdb44) at
/usr/src/sys/kern/subr_sleepqueue.c:580
#4  0xc05a34a5 in _sleep (ident=0xc08bdb44, lock=0xc08bdb48, priority=104,
wmesg=0xc0826d0c "psleep", timo=0) at
/usr/src/sys/kern/kern_synch.c:230
#5  0xc079c4e8 in vm_daemon () at /usr/src/sys/vm/vm_pageout.c:1535
#6  0xc0574587 in fork_exit (callout=0xc079c470 <vm_daemon>, arg=0x0,
frame=0xc5608d38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 46 (Thread 100045):
#0  sched_switch (td=0xc5aa1240, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not
available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05d03d7 in sleepq_timedwait (wchan=0xc08bdb0c) at
/usr/src/sys/kern/subr_sleepqueue.c:615
#4  0xc05a3483 in _sleep (ident=0xc08bdb0c, lock=0xc08bdad0, priority=68,
wmesg=0xc0826d0c "psleep", timo=5000) at
/usr/src/sys/kern/kern_synch.c:226
#5  0xc079ce6b in vm_pageout () at /usr/src/sys/vm/vm_pageout.c:1475
#6  0xc0574587 in fork_exit (callout=0xc079cb80 <vm_pageout>, arg=0x0,
frame=0xc5605d38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 45 (Thread 100044):
#0  sched_switch (td=0xc5aa1480, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not
available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05cfec6 in sleepq_wait (wchan=0xc08b376c) at
/usr/src/sys/kern/subr_sleepqueue.c:580
#4  0xc05a34a5 in _sleep (ident=0xc08b376c, lock=0xc08b3674, priority=0,
wmesg=0xc082a92a "waiting_for_work", timo=0) at
/usr/src/sys/kern/kern_synch.c:230
#5  0xc06693df in sctp_iterator_thread (v=0x0) at
/usr/src/sys/netinet/sctp_bsd_addr.c:95
#6  0xc0574587 in fork_exit (callout=0xc0669360 <sctp_iterator_thread>,
arg=0x0, frame=0xc55d2d38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 44 (Thread 100043):
#0  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:268
Current language:  auto; currently asm

Thread 43 (Thread 100042):
#0  sched_switch (td=0xc5aa1900, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc0577dbb in ithread_loop (arg=0xc5ab2c20) at
/usr/src/sys/kern/kern_intr.c:1225
#3  0xc0574587 in fork_exit (callout=0xc0577ab0 <ithread_loop>,
arg=0xc5ab2c20, frame=0xc55c9d38) at /usr/src/sys/kern/kern_fork.c:811
#4  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271
Current language:  auto; currently c

Thread 42 (Thread 100041):
#0  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:268
Current language:  auto; currently asm

Thread 41 (Thread 100040):
#0  sched_switch (td=0xc5aa1d80, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not
available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05d03d7 in sleepq_timedwait (wchan=0xc5a83c3c) at
/usr/src/sys/kern/subr_sleepqueue.c:615
#4  0xc05a3483 in _sleep (ident=0xc5a83c3c, lock=0xc5a83cf0, priority=76,
wmesg=0xc081c912 "-", timo=1000) at /usr/src/sys/kern/kern_synch.c:226
#5  0xc07b319d in fdc_thread (arg=dwarf2_read_address: Corrupted DWARF
expression.
) at /usr/src/sys/dev/fdc/fdc.c:812
#6  0xc0574587 in fork_exit (callout=0xc07b2db0 <fdc_thread>,
arg=0xc5a83c00, frame=0xc55c3d38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271
Current language:  auto; currently c

Thread 40 (Thread 100039):
#0  sched_switch (td=0xc5aa2000, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc0577dbb in ithread_loop (arg=0xc5a8daa0) at
/usr/src/sys/kern/kern_intr.c:1225
#3  0xc0574587 in fork_exit (callout=0xc0577ab0 <ithread_loop>,
arg=0xc5a8daa0, frame=0xc55b6d38) at /usr/src/sys/kern/kern_fork.c:811
#4  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 39 (Thread 100038):
#0  sched_switch (td=0xc59b2d80, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
---Type <return> to continue, or q <return> to quit---
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc0577dbb in ithread_loop (arg=0xc5a8d1e0) at
/usr/src/sys/kern/kern_intr.c:1225
#3  0xc0574587 in fork_exit (callout=0xc0577ab0 <ithread_loop>,
arg=0xc5a8d1e0, frame=0xc559ed38) at /usr/src/sys/kern/kern_fork.c:811
#4  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 38 (Thread 100037):
#0  sched_switch (td=0xc5a60000, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not
available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05d03d7 in sleepq_timedwait (wchan=0xc5999210) at
/usr/src/sys/kern/subr_sleepqueue.c:615
#4  0xc05a3483 in _sleep (ident=0xc5999210, lock=0x0, priority=92,
wmesg=0xc081bb4e "usbevt", timo=60000) at
/usr/src/sys/kern/kern_synch.c:226
#5  0xc051cad5 in usb_event_thread (arg=0xc5a8f3c0) at
/usr/src/sys/dev/usb/usb.c:445
#6  0xc0574587 in fork_exit (callout=0xc051c9e0 <usb_event_thread>,
arg=0xc5a8f3c0, frame=0xc5597d38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 37 (Thread 100036):
#0  sched_switch (td=0xc5a60240, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not
available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05d03d7 in sleepq_timedwait (wchan=0xc5a7e210) at
/usr/src/sys/kern/subr_sleepqueue.c:615
#4  0xc05a3483 in _sleep (ident=0xc5a7e210, lock=0x0, priority=92,
wmesg=0xc081bb4e "usbevt", timo=60000) at
/usr/src/sys/kern/kern_synch.c:226
#5  0xc051cad5 in usb_event_thread (arg=0xc5a57900) at
/usr/src/sys/dev/usb/usb.c:445
#6  0xc0574587 in fork_exit (callout=0xc051c9e0 <usb_event_thread>,
arg=0xc5a57900, frame=0xc5584d38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 36 (Thread 100035):
#0  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:268
Current language:  auto; currently asm

Thread 35 (Thread 100034):
#0  sched_switch (td=0xc5a606c0, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not
available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05d03d7 in sleepq_timedwait (wchan=0xc5a65210) at
/usr/src/sys/kern/subr_sleepqueue.c:615
#4  0xc05a3483 in _sleep (ident=0xc5a65210, lock=0x0, priority=92,
wmesg=0xc081bb4e "usbevt", timo=60000) at
/usr/src/sys/kern/kern_synch.c:226
#5  0xc051cad5 in usb_event_thread (arg=0xc5a62240) at
/usr/src/sys/dev/usb/usb.c:445
#6  0xc0574587 in fork_exit (callout=0xc051c9e0 <usb_event_thread>,
arg=0xc5a62240, frame=0xc557dd38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271
Current language:  auto; currently c

Thread 34 (Thread 100033):
#0  sched_switch (td=0xc5a60900, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not
available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05d03d7 in sleepq_timedwait (wchan=0xc5a6d210) at
/usr/src/sys/kern/subr_sleepqueue.c:615
#4  0xc05a3483 in _sleep (ident=0xc5a6d210, lock=0x0, priority=92,
wmesg=0xc081bb4e "usbevt", timo=60000) at
/usr/src/sys/kern/kern_synch.c:226
#5  0xc051cad5 in usb_event_thread (arg=0xc5a62a80) at
/usr/src/sys/dev/usb/usb.c:445
#6  0xc0574587 in fork_exit (callout=0xc051c9e0 <usb_event_thread>,
arg=0xc5a62a80, frame=0xc5579d38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 33 (Thread 100032):
#0  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:268
Current language:  auto; currently asm

Thread 32 (Thread 100031):
#0  sched_switch (td=0xc5a60d80, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not
available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05cfec6 in sleepq_wait (wchan=0xc08a1b54) at
/usr/src/sys/kern/subr_sleepqueue.c:580
#4  0xc05a34a5 in _sleep (ident=0xc08a1b54, lock=0x0, priority=92,
wmesg=0xc081bb40 "usbtsk", timo=0) at
/usr/src/sys/kern/kern_synch.c:230
#5  0xc051c97e in usb_task_thread (arg=0xc08a1b54) at
/usr/src/sys/dev/usb/usb.c:475
---Type <return> to continue, or q <return> to quit---
#6  0xc0574587 in fork_exit (callout=0xc051c900 <usb_task_thread>,
arg=0xc08a1b54, frame=0xc5572d38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271
Current language:  auto; currently c

Thread 31 (Thread 100030):
#0  sched_switch (td=0xc5a61000, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not
available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05cfec6 in sleepq_wait (wchan=0xc08a1b40) at
/usr/src/sys/kern/subr_sleepqueue.c:580
#4  0xc05a34a5 in _sleep (ident=0xc08a1b40, lock=0x0, priority=92,
wmesg=0xc081bb40 "usbtsk", timo=0) at
/usr/src/sys/kern/kern_synch.c:230
#5  0xc051c97e in usb_task_thread (arg=0xc08a1b40) at
/usr/src/sys/dev/usb/usb.c:475
#6  0xc0574587 in fork_exit (callout=0xc051c900 <usb_task_thread>,
arg=0xc08a1b40, frame=0xc556fd38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 30 (Thread 100029):
#0  sched_switch (td=0xc5a61240, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not
available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05d03d7 in sleepq_timedwait (wchan=0xc5a4c210) at
/usr/src/sys/kern/subr_sleepqueue.c:615
#4  0xc05a3483 in _sleep (ident=0xc5a4c210, lock=0x0, priority=92,
wmesg=0xc081bb4e "usbevt", timo=60000) at
/usr/src/sys/kern/kern_synch.c:226
#5  0xc051cad5 in usb_event_thread (arg=0xc5a63500) at
/usr/src/sys/dev/usb/usb.c:445
#6  0xc0574587 in fork_exit (callout=0xc051c9e0 <usb_event_thread>,
arg=0xc5a63500, frame=0xc556cd38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 29 (Thread 100028):
#0  sched_switch (td=0xc5a61480, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc0577dbb in ithread_loop (arg=0xc5a5b9d0) at
/usr/src/sys/kern/kern_intr.c:1225
#3  0xc0574587 in fork_exit (callout=0xc0577ab0 <ithread_loop>,
arg=0xc5a5b9d0, frame=0xc5568d38) at /usr/src/sys/kern/kern_fork.c:811
#4  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 28 (Thread 100027):
#0  sched_switch (td=0xc58da6c0, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc0577dbb in ithread_loop (arg=0xc5a5b220) at
/usr/src/sys/kern/kern_intr.c:1225
#3  0xc0574587 in fork_exit (callout=0xc0577ab0 <ithread_loop>,
arg=0xc5a5b220, frame=0xc5565d38) at /usr/src/sys/kern/kern_fork.c:811
#4  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 27 (Thread 100026):
#0  sched_switch (td=0xc58da900, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc0577dbb in ithread_loop (arg=0xc5a5b110) at
/usr/src/sys/kern/kern_intr.c:1225
#3  0xc0574587 in fork_exit (callout=0xc0577ab0 <ithread_loop>,
arg=0xc5a5b110, frame=0xc554fd38) at /usr/src/sys/kern/kern_fork.c:811
#4  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 26 (Thread 100025):
#0  sched_switch (td=0xc58dab40, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc0577dbb in ithread_loop (arg=0xc5a5b020) at
/usr/src/sys/kern/kern_intr.c:1225
#3  0xc0574587 in fork_exit (callout=0xc0577ab0 <ithread_loop>,
arg=0xc5a5b020, frame=0xc5513d38) at /usr/src/sys/kern/kern_fork.c:811
#4  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 25 (Thread 100024):
#0  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:268
Current language:  auto; currently asm

Thread 24 (Thread 100023):
#0  sched_switch (td=0xc59b2000, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc0577dbb in ithread_loop (arg=0xc59fa980) at
/usr/src/sys/kern/kern_intr.c:1225
---Type <return> to continue, or q <return> to quit---
#3  0xc0574587 in fork_exit (callout=0xc0577ab0 <ithread_loop>,
arg=0xc59fa980, frame=0xc54fad38) at /usr/src/sys/kern/kern_fork.c:811
#4  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271
Current language:  auto; currently c

Thread 23 (Thread 100022):
#0  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:268
Current language:  auto; currently asm

Thread 22 (Thread 100021):
#0  sched_switch (td=0xc59b2480, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not
available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05cfec6 in sleepq_wait (wchan=0xc588bd80) at
/usr/src/sys/kern/subr_sleepqueue.c:580
#4  0xc05a34a5 in _sleep (ident=0xc588bd80, lock=0xc588bd9c, priority=0,
wmesg=0xc081c912 "-", timo=0) at /usr/src/sys/kern/kern_synch.c:230
#5  0xc05d1708 in taskqueue_thread_loop (arg=0xc08a279c) at
/usr/src/sys/kern/subr_taskqueue.c:95
#6  0xc0574587 in fork_exit (callout=0xc05d1600 <taskqueue_thread_loop>,
arg=0xc08a279c, frame=0xc54e6d38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271
Current language:  auto; currently c

Thread 21 (Thread 100020):
#0  sched_switch (td=0xc59b26c0, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc0577dbb in ithread_loop (arg=0xc59a3290) at
/usr/src/sys/kern/kern_intr.c:1225
#3  0xc0574587 in fork_exit (callout=0xc0577ab0 <ithread_loop>,
arg=0xc59a3290, frame=0xc54e3d38) at /usr/src/sys/kern/kern_fork.c:811
#4  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 20 (Thread 100019):
#0  sched_switch (td=0xc59b2900, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc0577dbb in ithread_loop (arg=0xc59a32a0) at
/usr/src/sys/kern/kern_intr.c:1225
#3  0xc0574587 in fork_exit (callout=0xc0577ab0 <ithread_loop>,
arg=0xc59a32a0, frame=0xc54e0d38) at /usr/src/sys/kern/kern_fork.c:811
#4  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 19 (Thread 100018):
#0  sched_switch (td=0xc59b2b40, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not
available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05cfec6 in sleepq_wait (wchan=0xc58ef000) at
/usr/src/sys/kern/subr_sleepqueue.c:580
#4  0xc05a34a5 in _sleep (ident=0xc58ef000, lock=0xc58ef01c, priority=0,
wmesg=0xc081c912 "-", timo=0) at /usr/src/sys/kern/kern_synch.c:230
#5  0xc05d1708 in taskqueue_thread_loop (arg=0xc1159f24) at
/usr/src/sys/kern/subr_taskqueue.c:95
#6  0xc0574587 in fork_exit (callout=0xc05d1600 <taskqueue_thread_loop>,
arg=0xc1159f24, frame=0xc54ddd38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 18 (Thread 100017):
#0  sched_switch (td=0xc5891240, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not
available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05cfec6 in sleepq_wait (wchan=0xc58ef000) at
/usr/src/sys/kern/subr_sleepqueue.c:580
#4  0xc05a34a5 in _sleep (ident=0xc58ef000, lock=0xc58ef01c, priority=0,
wmesg=0xc081c912 "-", timo=0) at /usr/src/sys/kern/kern_synch.c:230
#5  0xc05d1708 in taskqueue_thread_loop (arg=0xc1159f24) at
/usr/src/sys/kern/subr_taskqueue.c:95
#6  0xc0574587 in fork_exit (callout=0xc05d1600 <taskqueue_thread_loop>,
arg=0xc1159f24, frame=0xc54dad38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 17 (Thread 100016):
#0  sched_switch (td=0xc5891480, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not
available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05cfec6 in sleepq_wait (wchan=0xc58ef000) at
/usr/src/sys/kern/subr_sleepqueue.c:580
#4  0xc05a34a5 in _sleep (ident=0xc58ef000, lock=0xc58ef01c, priority=0,
wmesg=0xc081c912 "-", timo=0) at /usr/src/sys/kern/kern_synch.c:230
#5  0xc05d1708 in taskqueue_thread_loop (arg=0xc1159f24) at
/usr/src/sys/kern/subr_taskqueue.c:95
#6  0xc0574587 in fork_exit (callout=0xc05d1600 <taskqueue_thread_loop>,
arg=0xc1159f24, frame=0xc54d7d38) at /usr/src/sys/kern/kern_fork.c:811
---Type <return> to continue, or q <return> to quit---
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 16 (Thread 100015):
#0  sched_switch (td=0xc58916c0, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not
available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05cfec6 in sleepq_wait (wchan=0xc58ef080) at
/usr/src/sys/kern/subr_sleepqueue.c:580
#4  0xc05a34a5 in _sleep (ident=0xc58ef080, lock=0xc58ef09c, priority=0,
wmesg=0xc081c912 "-", timo=0) at /usr/src/sys/kern/kern_synch.c:230
#5  0xc05d1708 in taskqueue_thread_loop (arg=0xc08b061c) at
/usr/src/sys/kern/subr_taskqueue.c:95
#6  0xc0574587 in fork_exit (callout=0xc05d1600 <taskqueue_thread_loop>,
arg=0xc08b061c, frame=0xc54d4d38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 15 (Thread 100014):
#0  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:268
Current language:  auto; currently asm

Thread 14 (Thread 100013):
#0  sched_switch (td=0xc5891b40, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc0577dbb in ithread_loop (arg=0xc59a32e0) at
/usr/src/sys/kern/kern_intr.c:1225
#3  0xc0574587 in fork_exit (callout=0xc0577ab0 <ithread_loop>,
arg=0xc59a32e0, frame=0xc54ced38) at /usr/src/sys/kern/kern_fork.c:811
#4  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271
Current language:  auto; currently c

Thread 13 (Thread 100012):
#0  sched_switch (td=0xc5891d80, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not
available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05cfec6 in sleepq_wait (wchan=0xc088d214) at
/usr/src/sys/kern/subr_sleepqueue.c:580
#4  0xc05a34a5 in _sleep (ident=0xc088d214, lock=0xc088d22c, priority=76,
wmesg=0xc0803242 "ccb_scanq", timo=0) at
/usr/src/sys/kern/kern_synch.c:230
#5  0xc0446491 in xpt_scanner_thread (dummy=0x0) at
/usr/src/sys/cam/cam_xpt.c:1399
#6  0xc0574587 in fork_exit (callout=0xc0446450 <xpt_scanner_thread>,
arg=0x0, frame=0xc54cbd38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 12 (Thread 100011):
#0  sched_switch (td=0xc58da000, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not
available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05d03d7 in sleepq_timedwait (wchan=0xc08a44b4) at
/usr/src/sys/kern/subr_sleepqueue.c:615
#4  0xc05a3483 in _sleep (ident=0xc08a44b4, lock=0x0, priority=0,
wmesg=0xc081c912 "-", timo=100) at /usr/src/sys/kern/kern_synch.c:226
#5  0xc05a35a0 in pause (wmesg=0xc081c912 "-", timo=100) at
/usr/src/sys/kern/kern_synch.c:334
#6  0xc04c3e72 in random_kthread (arg=0x0) at
/usr/src/sys/dev/random/randomdev_soft.c:289
#7  0xc0574587 in fork_exit (callout=0xc04c3c40 <random_kthread>, arg=0x0,
frame=0xc54c8d38) at /usr/src/sys/kern/kern_fork.c:811
#8  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 11 (Thread 100010):
#0  sched_switch (td=0xc58da240, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not
available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05d03d7 in sleepq_timedwait (wchan=0xc08a1f0c) at
/usr/src/sys/kern/subr_sleepqueue.c:615
#4  0xc05a3483 in _sleep (ident=0xc08a1f0c, lock=0xc08a1e28, priority=588,
wmesg=0xc081c912 "-", timo=100) at /usr/src/sys/kern/kern_synch.c:226
#5  0xc053f22b in g_io_schedule_down (tp=0xc58da240) at
/usr/src/sys/geom/geom_io.c:487
#6  0xc053f94c in g_down_procbody () at /usr/src/sys/geom/geom_kern.c:118
#7  0xc0574587 in fork_exit (callout=0xc053f8e0 <g_down_procbody>,
arg=0x0, frame=0xc54c5d38) at /usr/src/sys/kern/kern_fork.c:811
#8  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 10 (Thread 100009):
#0  sched_switch (td=0xc58da480, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not
available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
---Type <return> to continue, or q <return> to quit---
#3  0xc05d03d7 in sleepq_timedwait (wchan=0xc08a1f08) at
/usr/src/sys/kern/subr_sleepqueue.c:615
#4  0xc05a3483 in _sleep (ident=0xc08a1f08, lock=0xc08a1e68, priority=588,
wmesg=0xc081c912 "-", timo=100) at /usr/src/sys/kern/kern_synch.c:226
#5  0xc053f532 in g_io_schedule_up (tp=0xc58da480) at
/usr/src/sys/geom/geom_io.c:592
#6  0xc053f82c in g_up_procbody () at /usr/src/sys/geom/geom_kern.c:95
#7  0xc0574587 in fork_exit (callout=0xc053f7c0 <g_up_procbody>, arg=0x0,
frame=0xc54c2d38) at /usr/src/sys/kern/kern_fork.c:811
#8  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 9 (Thread 100008):
#0  sched_switch (td=0xc588f000, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not
available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05d03d7 in sleepq_timedwait (wchan=0xc08a1f00) at
/usr/src/sys/kern/subr_sleepqueue.c:615
#4  0xc05a3483 in _sleep (ident=0xc08a1f00, lock=0x0, priority=76,
wmesg=0xc081c912 "-", timo=100) at /usr/src/sys/kern/kern_synch.c:226
#5  0xc053f8da in g_event_procbody () at /usr/src/sys/geom/geom_kern.c:142
#6  0xc0574587 in fork_exit (callout=0xc053f830 <g_event_procbody>,
arg=0x0, frame=0xc54bfd38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 8 (Thread 100007):
#0  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:268
Current language:  auto; currently asm

Thread 7 (Thread 100006):
#0  sched_switch (td=0xc588f480, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc07d4682 in ipi_bitmap_handler (frame      {tf_fs = 8, tf_es = 40, tf_ds =
40, tf_edi = -644953128, tf_esi = -980880256, tf_ebp = -984900488, tf_isp =
-984900532, tf_ebx = -1064681904, tf_edx = -980877952, tf_ecx = -980877952,
tf_eax = 4, tf_trapno = 0, tf_err = 0, tf_eip = -1067925921, tf_cs = 32,
tf_eflags = 582, tf_esp = -1066191232, tf_ss = 0}) at
/usr/src/sys/i386/i386/mp_machdep.c:1187
#3  0xc07c6dae in Xipi_intr_bitmap_handler () at apic_vector.s:284
#4  0xc058be5f in _mtx_lock_sleep (m=0xc08a3e50, tid=3314087040, opts=0,
file=0x0, line=0) at /usr/src/sys/kern/kern_mutex.c:349
#5  0xc05ad91b in softclock (dummy=0x0) at /usr/src/sys/kern/kern_timeout.c:241
#6  0xc0577c5b in ithread_loop (arg=0xc588c390) at
/usr/src/sys/kern/kern_intr.c:1126
#7  0xc0574587 in fork_exit (callout=0xc0577ab0 <ithread_loop>,
arg=0xc588c390, frame=0xc54b9d38) at /usr/src/sys/kern/kern_fork.c:811
#8  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271
Current language:  auto; currently c

Thread 6 (Thread 100005):
#0  sched_switch (td=0xc588f6c0, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc0577dbb in ithread_loop (arg=0xc588c3a0) at
/usr/src/sys/kern/kern_intr.c:1225
#3  0xc0574587 in fork_exit (callout=0xc0577ab0 <ithread_loop>,
arg=0xc588c3a0, frame=0xc54b6d38) at /usr/src/sys/kern/kern_fork.c:811
#4  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 5 (Thread 100004):
#0  sched_switch (td=0xc588f900, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05bcedd in critical_exit () at kern_switch.c:185
#3  0xc07ce2f5 in lapic_handle_timer (frame=0xc54b1c48) at
/usr/src/sys/i386/i386/local_apic.c:724
#4  0xc07c6c9f in Xtimerint () at apic_vector.s:108
#5  0xc1149b15 in acpi_cpu_c1 () at
/usr/src/sys/modules/acpi/acpi/../../../i386/acpica/acpi_machdep.c:550
#6  0xc1142a1d in acpi_cpu_idle () at
/usr/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi_cpu.c:912
#7  0xc07d0a58 in cpu_idle () at /usr/src/sys/i386/i386/machdep.c:1183
#8  0xc05bdd6b in sched_idletd (dummy=0x0) at /usr/src/sys/kern/sched_ule.c:2681
#9  0xc0574587 in fork_exit (callout=0xc05bdaa0 <sched_idletd>, arg=0x0,
frame=0xc54b1d38) at /usr/src/sys/kern/kern_fork.c:811
#10 0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 4 (Thread 100003):
#0  sched_switch (td=0xc588fb40, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc07d4682 in ipi_bitmap_handler (frame      {tf_fs = 8, tf_es = 40, tf_ds =
40, tf_edi = 1, tf_esi = 15, tf_ebp = -984945472, tf_isp = -984945496, tf_ebx =
-980878528, tf_edx = 256, tf_ecx = 16, tf_eax = 582, tf_trapno = 0, tf_err = 0,
tf_e---Type <return> to continue, or q <return> to quit---
ip = -1065551975, tf_cs = 32, tf_eflags = 582, tf_esp = -1064663296, tf_ss =
-984945420}) at /usr/src/sys/i386/i386/mp_machdep.c:1187
#3  0xc07c6dae in Xipi_intr_bitmap_handler () at apic_vector.s:284
#4  0xc07cf799 in spinlock_exit () at cpufunc.h:365
#5  0xc05bdcd6 in sched_idletd (dummy=0x0) at /usr/src/sys/kern/sched_ule.c:806
#6  0xc0574587 in fork_exit (callout=0xc05bdaa0 <sched_idletd>, arg=0x0,
frame=0xc54aed38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 3 (Thread 100002):
#0  doadump () at pcpu.h:196
#1  0xc059b097 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418
#2  0xc059b36c in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:574
#3  0xc058bd1f in _mtx_lock_spin_failed (m=0x0) at
/usr/src/sys/kern/kern_mutex.c:449
#4  0xc058c425 in _thread_lock_flags (td=0xc588f480, opts=0, file=0xc08200c8
"/usr/src/sys/kern/kern_intr.c", line=801) at
/usr/src/sys/kern/kern_mutex.c:537
#5  0xc0576787 in intr_event_schedule_thread (ie=Variable "ie" is not
available.
) at /usr/src/sys/kern/kern_intr.c:801
#6  0xc0577aa5 in swi_sched (cookie=0xc5880e00, flags=Variable "flags"
is not available.
) at /usr/src/sys/kern/kern_intr.c:1028
#7  0xc055a6df in hardclock (usermode=0, pc=3229503748) at
/usr/src/sys/kern/kern_clock.c:339
#8  0xc07ce23a in lapic_handle_timer (frame=0xc54aaba0) at
/usr/src/sys/i386/i386/local_apic.c:705
#9  0xc07c6c9f in Xtimerint () at apic_vector.s:108
#10 0xc07e5104 in DELAY (n=1000000) at cpufunc.h:352
#11 0xc07e2c03 in cpu_reset () at /usr/src/sys/i386/i386/vm_machdep.c:594
#12 0xc059a753 in shutdown_reset (junk=0x0, howto=0) at
/usr/src/sys/kern/kern_shutdown.c:493
#13 0xc059b117 in boot (howto=0) at /usr/src/sys/kern/kern_shutdown.c:421
#14 0xc059b247 in reboot (td=0xc588fd80, uap=0xc54aacfc) at
/usr/src/sys/kern/kern_shutdown.c:172
#15 0xc07e07d5 in syscall (frame=0xc54aad38) at
/usr/src/sys/i386/i386/trap.c:1089
#16 0xc07c6720 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:262
#17 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)

Thread 2 (Thread 100001):
#0  sched_switch (td=0xc5891000, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not
available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05cfec6 in sleepq_wait (wchan=0xc08bcd48) at
/usr/src/sys/kern/subr_sleepqueue.c:580
#4  0xc055aaaf in _cv_wait (cvp=0xc08bcd48, lock=0xc08bcd28) at
/usr/src/sys/kern/kern_condvar.c:137
#5  0xc0749e2f in audit_worker (arg=0x0) at
/usr/src/sys/security/audit/audit_worker.c:410
#6  0xc0574587 in fork_exit (callout=0xc0749dc0 <audit_worker>, arg=0x0,
frame=0xc54a7d38) at /usr/src/sys/kern/kern_fork.c:811
#7  0xc07c6730 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:271

Thread 1 (Thread 100000):
#0  sched_switch (td=0xc08a2280, newtd=Variable "newtd" is not
available.
) at /usr/src/sys/kern/sched_ule.c:1944
#1  0xc05a3033 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2  0xc05cf859 in sleepq_switch (wchan=Variable "wchan" is not
available.
) at /usr/src/sys/kern/subr_sleepqueue.c:497
#3  0xc05d03d7 in sleepq_timedwait (wchan=0xc08a1fc0) at
/usr/src/sys/kern/subr_sleepqueue.c:615
#4  0xc05a3483 in _sleep (ident=0xc08a1fc0, lock=0x0, priority=68,
wmesg=0xc0823aef "sched", timo=10000) at
/usr/src/sys/kern/kern_synch.c:226
#5  0xc078df39 in scheduler (dummy=0x0) at /usr/src/sys/vm/vm_glue.c:733
#6  0xc05577b6 in mi_startup () at /usr/src/sys/kern/init_main.c:251
#7  0xc0440685 in begin () at /usr/src/sys/i386/i386/locore.s:328
196		__asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb) quit

Attilio Rao

2009-Jul-26 14:36 UTC

head link

kern/134584: [panic] spin lock held too long

2009/7/26 barbara <barbara.xxx1975@libero.it>:> It happened again, on shutdown.
> As the previous time, it happened after a high (for a desktop) uptime and,
if it could matter, after running net-p2p/transmission-gtk2 for several hours.
> I don't know if it's related, but often quitting transmission,
doesn't terminate the process. Sometimes it end after several minutes the
gui exited, sometimes it's still running after hours.
> I've noticed it as the destination folder is on a manually mounted
device and I can't umount it as fstat reports the device used by a
transmission process.
> So I often have to kill it.
> This happened both the time I had this kind of panic.
Can you try to reproduce it with WITNESS and *without*
WITNESS_SKIPSPIN? I would need to look at "show alllocks" and
possibily "ps" because it seems that the lock owner is preempted but
it should not happen while holding a spinlock (unless the acquired
spinlock is the one in the preempting path, in this case thought it
should drop inside sched_switch() and we can try to understand why
that doesn't happen).

Thanks,
Attilio


-- 
Peace can only be achieved by understanding - A. Einstein

Attilio Rao

2009-Jul-27 12:32 UTC

head link

kern/134584: [panic] spin lock held too long

2009/7/26 barbara <barbara.xxx1975@libero.it>:> It happened again, on shutdown.
> As the previous time, it happened after a high (for a desktop) uptime and,
if it could matter, after running net-p2p/transmission-gtk2 for several hours.
> I don't know if it's related, but often quitting transmission,
doesn't terminate the process. Sometimes it end after several minutes the
gui exited, sometimes it's still running after hours.
> I've noticed it as the destination folder is on a manually mounted
device and I can't umount it as fstat reports the device used by a
transmission process.
> So I often have to kill it.
> This happened both the time I had this kind of panic.
What hw is that? How many CPUs does it have?

Attilio


-- 
Peace can only be achieved by understanding - A. Einstein

barbara

2009-Jul-27 21:56 UTC

head link

kern/134584: [panic] spin lock held too long

> 2009/7/26 barbara <barbara.xxx1975@libero.it>:
> > It happened again, on shutdown.
> > As the previous time, it happened after a high (for a desktop) uptime
and, if it could matter, after running net-p2p/transmission-gtk2 for several
hours.
> > I don't know if it's related, but often quitting transmission,
doesn't terminate the process. Sometimes it end after several minutes the
gui exited, sometimes it's still running after hours.
> > I've noticed it as the destination folder is on a manually mounted
device and I can't umount it as fstat reports the device used by a
transmission process.
> > So I often have to kill it.
> > This happened both the time I had this kind of panic.
> 
> What hw is that? How many CPUs does it have?
> 

It's desktop pc, dual Athlon CPU, VIA chipset and...
You can find my dmesg.boot here:
http://pastebin.com/f507125f1
Aks if you need more info!

In the meanwhile I've rebuilt my kernel adding WITNESS, KDB, DDB options. Is
this what you need?
I'm not able to work with the debugger. I suppose I have to type the
commands you asked (ps, show alllocks) when it's panicking and the debbugger
starts and write down the output with pen and paper, am I corrent? Or is there a
way to redirect or dump the output somewhere?
Sorry, I'll try reading the chapter in the dev. handbook and I will try to
do my best, but please, tell me if you have some suggestions.

I'll try to reproduce the problem, even it's not happening regularly.
If you need more info or to test code, etc. I'll be glad to help you.
BTW, can the output of crashinfo be of any help? Here you can find ps -axl from
crashinfo:
http://pastebin.com/f3d54f01b

Thanks
Barbara

barbara

2009-Jul-31 05:20 UTC

head link

kern/134584: [panic] spin lock held too long

> The following reply was made to PR kern/134584; it has been noted by GNATS.
> 
> From: Attilio Rao <attilio@freebsd.org>
> To: barbara <barbara.xxx1975@libero.it>
> Cc: bug-followup <bug-followup@freebsd.org>, FreeBSD-stable
<FreeBSD-stable@freebsd.org>
> Subject: Re: kern/134584: [panic] spin lock held too long
> Date: Sun, 26 Jul 2009 16:36:44 +0200
> 
>  2009/7/26 barbara <barbara.xxx1975@libero.it>:
>  > It happened again, on shutdown.
>  > As the previous time, it happened after a high (for a desktop) uptime
and, if it could matter, after running net-p2p/transmission-gtk2 for several
hours.
>  > I don't know if it's related, but often quitting
transmission, doesn't terminate the process. Sometimes it end after several
minutes the gui exited, sometimes it's still running after hours.
>  > I've noticed it as the destination folder is on a manually
mounted device and I can't umount it as fstat reports the device used by a
transmission process.
>  > So I often have to kill it.
>  > This happened both the time I had this kind of panic.
>  
>  Can you try to reproduce it with WITNESS and *without*
>  WITNESS_SKIPSPIN? I would need to look at "show alllocks" and
>  possibily "ps" because it seems that the lock owner is preempted
but
>  it should not happen while holding a spinlock (unless the acquired
>  spinlock is the one in the preempting path, in this case thought it
>  should drop inside sched_switch() and we can try to understand why
>  that doesn't happen).
>  
Yesterday it happened again but the machines locked on panic command and nothing
was dumped.
It also happened today, and as yesterday after some hours (~10) of uptime, but
this time I've got no ddb.txt, just vmcore.

It stopped after the following message:
cpu reset: Stopping other CPUs

The output of "show alllocks command" is
(http://pastebin.com/f2323ad60):
Process 1 (init) thread 0xc589bd80 (100002)
exclusive sleep mutex Giant r = 0 (0xc0873b10) locked @
/usr/src/sys/kern/kern_module.c:102

That is the "mtx_lock(&Giant);" line in "static void
module_shutdown(void *arg1, int arg2)".


ps -axl from crashinfo (http://pastebin.com/f15e7ff90):
  UID   PID  PPID CPU PRI NI   VSZ   RSS MWCHAN STAT  TT       TIME COMMAND
    0     0     0   0 -16  0     0     0 sched  DLs   ??  10505772:02.00
[swapper]
    0     1     0   0  -8  0  1888     0 -      RLs   ??  8198756:46.00 [init]
    0     2     0   0  -8  0     0     0 -      DL    ??  -31491914:-33.55
[g_event]
    0     3     0   0  -8  0     0     0 -      DL    ??  -18031048:-1.55 [g_up]
    0     4     0   0  -8  0     0     0 -      DL    ??  29902314:39.00
[g_down]
    0     5     0   0   8  0     0     0 -      DL    ??    0:00.00 [kqueue tas
    0     6     0   0  -8  0     0     0 ccb_sc DL    ??    0:00.00 [xpt_thrd]
    0     7     0   0   8  0     0     0 -      DL    ??    0:00.00 [acpi_task_
    0     8     0   0   8  0     0     0 -      DL    ??    0:00.00 [acpi_task_
    0     9     0   0   8  0     0     0 -      DL    ??    0:00.00 [acpi_task_
    0    10     0   0 -16  0     0     0 audit_ DL    ??    0:00.00 [audit]
    0    11     0   0 171  0     0     0 -      RL    ??  30495164:05.00 [idle:
cpu1
    0    12     0   0 171  0     0     0 -      RL    ??  -8752676:-56.55 [idle:
cpu0
    0    13     0   0 -44  0     0     0 -      WL    ??  1297477:00.00 [swi1:
net]
    0    14     0   0 -32  0     0     0 -      RL    ??  25300285:42.00 [swi4:
cloc
    0    15     0   0 -36  0     0     0 -      WL    ??    0:00.00 [swi3: vm]
    0    16     0   0 -16  0     0     0 -      DL    ??  33237141:47.00
[yarrow]
    0    17     0   0 -40  0     0     0 -      WL    ??  33608583:19.00 [swi2:
camb
    0    18     0   0 -28  0     0     0 -      WL    ??    0:00.00 [swi5: +]
    0    19     0   0   8  0     0     0 -      DL    ??  577:12.00 [thread tas
    0    20     0   0 -24  0     0     0 -      WL    ??  15874802:41.00 [swi6:
Gian
    0    21     0   0 -24  0     0     0 -      WL    ??  34388543:22.00 [swi6:
task
    0    22     0   0 -52  0     0     0 -      WL    ??    0:00.00 [irq9: acpi
    0    23     0   0 -80  0     0     0 -      WL    ??  33965005:45.00 [irq24:
vga
    0    24     0   0 -64  0     0     0 -      WL    ??    0:00.00 [irq28: ata
    0    25     0   0 -64  0     0     0 -      WL    ??  15406643:53.00 [irq21:
ata
    0    26     0   0 -64  0     0     0 -      WL    ??  225221:21.00 [irq14:
ata
    0    27     0   0 -64  0     0     0 -      WL    ??  -19343052:-29.55
[irq15: ata
    0    28     0   0 -64  0     0     0 -      WL    ??  -5763326:-52.55
[irq20: uhc
    0    29     0   0   8  0     0     0 usbevt DL    ??  143817:50.00 [usb0]
    0    30     0   0   8  0     0     0 usbtsk DL    ??    0:00.00 [usbtask-hc
    0    31     0   0   8  0     0     0 usbtsk DL    ??    0:00.00 [usbtask-dr
    0    32     0   0 -64  0     0     0 -      WL    ??    0:00.00 [irq22: uhc
    0    33     0   0   8  0     0     0 usbevt DL    ??  139260:44.00 [usb1]
    0    34     0   0   8  0     0     0 usbevt DL    ??  154878:51.00 [usb2]
    0    35     0   0 -64  0     0     0 -      WL    ??    0:00.00 [irq23: uhc
    0    36     0   0   8  0     0     0 usbevt DL    ??  147693:41.00 [usb3]
    0    37     0   0   8  0     0     0 usbevt DL    ??  153607:50.00 [usb4]
    0    38     0   0 -80  0     0     0 -      RL    ??  -23091533:-19.55
[irq256: hd
    0    39     0   0 -68  0     0     0 -      WL    ??  -9517088:-16.55
[irq19: rl0
    0    40     0   0  -8  0     0     0 -      DL    ??  8660889:23.00 [fdc0]
    0    41     0   0 -48  0     0     0 -      WL    ??    0:00.00 [swi0: sio]
    0    42     0   0 -60  0     0     0 -      WL    ??  19344460:25.00 [irq1:
atkb
    0    43     0   0 -60  0     0     0 -      WL    ??    0:00.00 [irq7: ppbu
    0    44     0   0 -16  0     0     0 waitin DL    ??  1395:36.00 [sctp_itera
    0    45     0   0 -16  0     0     0 psleep DL    ??  23513565:41.00
[pagedaemon
    0    46     0   0  20  0     0     0 psleep DL    ??  239:30.00 [vmdaemon]
    0    47     0   0 171  0     0     0 pgzero DL    ??  19812:23.00 [pagezero]
    0    48     0   0  20  0     0     0 ktsusp DL    ??  31705008:58.00
[bufdaemon]
    0    49     0   0  20  0     0     0 ktsusp DL    ??  17747103:57.00
[syncer]
    0    50     0   0  20  0     0     0 ktsusp DL    ??  26534645:02.00 [vnlru]
    0    51     0   0 -16  0     0     0 sdflus DL    ??  4932805:02.00
[softdepflu

freebsd stable - Jul 2009 - kern/134584: [panic] spin lock held too long

kern/134584: [panic] spin lock held too long

kern/134584: [panic] spin lock held too long

kern/134584: [panic] spin lock held too long

kern/134584: [panic] spin lock held too long

kern/134584: [panic] spin lock held too long