thr3ads.net - freebsd stable - panic: spin lock held too long (w/ backtrace) [May 2007]

If this information is useful, please help other people find it:
Share via:

Scott Swanson

2007-May-10 21:23 UTC

panic: spin lock held too long (w/ backtrace)

Hello all,

I have a couple dozen SuperMicro servers in production and have had
reoccurring crashing issues on a couple of them that are under higher
load.

After managing to pry one out of production, I have been able to
re-create the problem fairly consistently just by looping a script that
tars up a large directory and writes the the file to a different
partition on the same raid array.

Can anyone point me towards the next step to debug this issue?  I can
pursue any other debugging info that may be required.

Regards;
Scott Swanson
Inetz Media

> uname -aFreeBSD pyrite.inetz.com 6.2-RELEASE-p4 FreeBSD 6.2-RELEASE-p4 #3: Tue
May  8 13:08:22 MDT 2007
root@biotite.inetz.com:/usr/obj/usr/src/sys/PYRITE  i386


root@pyrite /usr/obj/usr/src/sys/PYRITE > kgdb kernel.debug
/var/crash/vmcore.0
[GDB will not be able to debug user-mode threads:
/usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]

Unread portion of the kernel message buffer:
spin lock smp rendezvous held by 0xc95a5900 for > 5 seconds
panic: spin lock held too long
cpuid = 0
Uptime: 1d3h51m44s
Dumping 3967 MB (3 chunks)
  chunk 0: 1MB (158 pages) ... ok
  chunk 1: 3966MB (1015280 pages) 3950 3934 3918 3902 3886 3870 3854
3838 3822 3806 3790 3774 3758 3742 3726 3710 3694 3678 3662 3646 3630
3614 3598 3582 3566 3550 3534 3518 3502 3486 3470 3454 3438 3422 3406
3390 3374 3358 3342 3326 3310 3294 3278 3262 3246 3230 3214 3198 3182
3166 3150 3134 3118 3102 3086 3070 3054 3038 3022 3006 2990 2974 2958
2942 2926 2910 2894 2878 2862 2846 2830 2814 2798 2782 2766 2750 2734
2718 2702 2686 2670 2654 2638 2622 2606 2590 2574 2558 2542 2526 2510
2494 2478 2462 2446 2430 2414 2398 2382 2366 2350 2334 2318 2302 2286
2270 2254 2238 2222 2206 2190 2174 2158 2142 2126 2110 2094 2078 2062
2046 2030 2014 1998 1982 1966 1950 1934 1918 1902 1886 1870 1854 1838
1822 1806 1790 1774 1758 1742 1726 1710 1694 1678 1662 1646 1630 1614
1598 1582 1566 1550 1534 1518 1502 1486 1470 1454 1438 1422 1406 1390
1374 1358 1342 1326 1310 1294 1278 1262 1246 1230 1214 1198 1182 1166
1150 1134 1118 1102 1086 1070 1054 1038 1022 1006 990 974 958 942 926
910 894 878 862 846 830 814 798 782 766 750 734 718 702 686 670 654 638
622 606 590 574 558 542 526 510 494 478 462 446 430 414 398 382 366 350
334 318 302 286 270 254 238 222 206 190 174 158 142 126 110 94 78 62 46
30 14 ... ok
  chunk 2: 1MB (128 pages)

#0  doadump () at pcpu.h:165
165             __asm __volatile("movl %%fs:0,%0" : "=r"
(td));
(kgdb)
(kgdb) backtrace
#0  doadump () at pcpu.h:165
#1  0xc0632e8a in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2  0xc06331e1 in panic (fmt=0xc087937c "spin lock held too long") at
/usr/src/sys/kern/kern_shutdown.c:565
#3  0xc062a1bd in _mtx_lock_spin (m=0xc094f420, tid=3376316416, opts=0,
file=0x0, line=0) at /usr/src/sys/kern/kern_mutex.c:633
#4  0xc08147ce in pmap_invalidate_range (pmap=0xc0984420,
sva=3820650496, eva=3820666880) at /usr/src/sys/i386/i386/pmap.c:602
#5  0xc0814fad in pmap_qremove (sva=3820650496, count=0) at
/usr/src/sys/i386/i386/pmap.c:974
#6  0xc067ba30 in vfs_vmio_release (bp=0xdd53ae40) at
/usr/src/sys/kern/vfs_bio.c:1498
#7  0xc067c0d8 in getnewbuf (slpflag=0, slptimeo=0, size=16384,
maxsize=16384) at /usr/src/sys/kern/vfs_bio.c:1779
#8  0xc067d7e8 in geteblk (size=16384) at /usr/src/sys/kern/vfs_bio.c:2580
#9  0xc0775df1 in ffs_bufwrite (bp=0xdd4a7778) at
/usr/src/sys/ufs/ffs/ffs_vfsops.c:1700
#10 0xc067a9ab in bawrite (bp=0x0) at buf.h:410
#11 0xc0682f00 in vop_stdfsync (ap=0xeaa0acc0) at
/usr/src/sys/kern/vfs_default.c:431
#12 0xc05e5c8f in devfs_fsync (ap=0xeaa0acc0) at
/usr/src/sys/fs/devfs/devfs_vnops.c:379
#13 0xc082c3cc in VOP_FSYNC_APV (vop=0x0, a=0x0) at vnode_if.c:1020
#14 0xc068bf87 in sync_vnode (bo=0xc97491d0, td=0xc93e8000) at
vnode_if.h:537
#15 0xc068c2a5 in sched_sync () at /usr/src/sys/kern/vfs_subr.c:1698
#16 0xc061c5cd in fork_exit (callout=0xc068c04c <sched_sync>, arg=0x0,
frame=0xeaa0ad38) at /usr/src/sys/kern/kern_fork.c:821
#17 0xc080690c in fork_trampoline () at
/usr/src/sys/i386/i386/exception.s:208

Kris Kennaway

2007-May-10 21:43 UTC

head link

panic: spin lock held too long (w/ backtrace)

On Thu, May 10, 2007 at 02:49:45PM -0600, Scott Swanson
wrote:> Hello all,
> 
> I have a couple dozen SuperMicro servers in production and have had
> reoccurring crashing issues on a couple of them that are under higher
> load.
> 
> After managing to pry one out of production, I have been able to
> re-create the problem fairly consistently just by looping a script that
> tars up a large directory and writes the the file to a different
> partition on the same raid array.
> 
> Can anyone point me towards the next step to debug this issue?  I can
> pursue any other debugging info that may be required.
> 
> Regards;
> Scott Swanson
> Inetz Media
> 
> 
> > uname -a
> FreeBSD pyrite.inetz.com 6.2-RELEASE-p4 FreeBSD 6.2-RELEASE-p4 #3: Tue
> May  8 13:08:22 MDT 2007
> root@biotite.inetz.com:/usr/obj/usr/src/sys/PYRITE  i386
> 
> 
> root@pyrite /usr/obj/usr/src/sys/PYRITE > kgdb kernel.debug
> /var/crash/vmcore.0
> [GDB will not be able to debug user-mode threads:
> /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
> GNU gdb 6.1.1 [FreeBSD]
> 
> Unread portion of the kernel message buffer:
> spin lock smp rendezvous held by 0xc95a5900 for > 5 seconds
What is thread 0xc95a5900 doing?

Kris
> panic: spin lock held too long
> cpuid = 0
> Uptime: 1d3h51m44s
> Dumping 3967 MB (3 chunks)
>   chunk 0: 1MB (158 pages) ... ok
>   chunk 1: 3966MB (1015280 pages) 3950 3934 3918 3902 3886 3870 3854
> 3838 3822 3806 3790 3774 3758 3742 3726 3710 3694 3678 3662 3646 3630
> 3614 3598 3582 3566 3550 3534 3518 3502 3486 3470 3454 3438 3422 3406
> 3390 3374 3358 3342 3326 3310 3294 3278 3262 3246 3230 3214 3198 3182
> 3166 3150 3134 3118 3102 3086 3070 3054 3038 3022 3006 2990 2974 2958
> 2942 2926 2910 2894 2878 2862 2846 2830 2814 2798 2782 2766 2750 2734
> 2718 2702 2686 2670 2654 2638 2622 2606 2590 2574 2558 2542 2526 2510
> 2494 2478 2462 2446 2430 2414 2398 2382 2366 2350 2334 2318 2302 2286
> 2270 2254 2238 2222 2206 2190 2174 2158 2142 2126 2110 2094 2078 2062
> 2046 2030 2014 1998 1982 1966 1950 1934 1918 1902 1886 1870 1854 1838
> 1822 1806 1790 1774 1758 1742 1726 1710 1694 1678 1662 1646 1630 1614
> 1598 1582 1566 1550 1534 1518 1502 1486 1470 1454 1438 1422 1406 1390
> 1374 1358 1342 1326 1310 1294 1278 1262 1246 1230 1214 1198 1182 1166
> 1150 1134 1118 1102 1086 1070 1054 1038 1022 1006 990 974 958 942 926
> 910 894 878 862 846 830 814 798 782 766 750 734 718 702 686 670 654 638
> 622 606 590 574 558 542 526 510 494 478 462 446 430 414 398 382 366 350
> 334 318 302 286 270 254 238 222 206 190 174 158 142 126 110 94 78 62 46
> 30 14 ... ok
>   chunk 2: 1MB (128 pages)
> 
> #0  doadump () at pcpu.h:165
> 165             __asm __volatile("movl %%fs:0,%0" :
"=r" (td));
> (kgdb)
> (kgdb) backtrace
> #0  doadump () at pcpu.h:165
> #1  0xc0632e8a in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
> #2  0xc06331e1 in panic (fmt=0xc087937c "spin lock held too
long") at
> /usr/src/sys/kern/kern_shutdown.c:565
> #3  0xc062a1bd in _mtx_lock_spin (m=0xc094f420, tid=3376316416, opts=0,
> file=0x0, line=0) at /usr/src/sys/kern/kern_mutex.c:633
> #4  0xc08147ce in pmap_invalidate_range (pmap=0xc0984420,
> sva=3820650496, eva=3820666880) at /usr/src/sys/i386/i386/pmap.c:602
> #5  0xc0814fad in pmap_qremove (sva=3820650496, count=0) at
> /usr/src/sys/i386/i386/pmap.c:974
> #6  0xc067ba30 in vfs_vmio_release (bp=0xdd53ae40) at
> /usr/src/sys/kern/vfs_bio.c:1498
> #7  0xc067c0d8 in getnewbuf (slpflag=0, slptimeo=0, size=16384,
> maxsize=16384) at /usr/src/sys/kern/vfs_bio.c:1779
> #8  0xc067d7e8 in geteblk (size=16384) at /usr/src/sys/kern/vfs_bio.c:2580
> #9  0xc0775df1 in ffs_bufwrite (bp=0xdd4a7778) at
> /usr/src/sys/ufs/ffs/ffs_vfsops.c:1700
> #10 0xc067a9ab in bawrite (bp=0x0) at buf.h:410
> #11 0xc0682f00 in vop_stdfsync (ap=0xeaa0acc0) at
> /usr/src/sys/kern/vfs_default.c:431
> #12 0xc05e5c8f in devfs_fsync (ap=0xeaa0acc0) at
> /usr/src/sys/fs/devfs/devfs_vnops.c:379
> #13 0xc082c3cc in VOP_FSYNC_APV (vop=0x0, a=0x0) at vnode_if.c:1020
> #14 0xc068bf87 in sync_vnode (bo=0xc97491d0, td=0xc93e8000) at
> vnode_if.h:537
> #15 0xc068c2a5 in sched_sync () at /usr/src/sys/kern/vfs_subr.c:1698
> #16 0xc061c5cd in fork_exit (callout=0xc068c04c <sched_sync>,
arg=0x0,
> frame=0xeaa0ad38) at /usr/src/sys/kern/kern_fork.c:821
> #17 0xc080690c in fork_trampoline () at
> /usr/src/sys/i386/i386/exception.s:208
> 
> 
> 
> _______________________________________________
> freebsd-stable@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to
"freebsd-stable-unsubscribe@freebsd.org"
>

Scott Swanson

2007-May-11 01:00 UTC

head link

panic: spin lock held too long (w/ backtrace)

Kris Kennaway wrote:> On Thu, May 10, 2007 at 02:49:45PM -0600, Scott Swanson wrote:
>> Hello all,
>>
>> I have a couple dozen SuperMicro servers in production and have had
>> reoccurring crashing issues on a couple of them that are under higher
>> load.
>>
>> After managing to pry one out of production, I have been able to
>> re-create the problem fairly consistently just by looping a script that
>> tars up a large directory and writes the the file to a different
>> partition on the same raid array.
>>
>> Can anyone point me towards the next step to debug this issue?  I can
>> pursue any other debugging info that may be required.
>>
>> Regards;
>> Scott Swanson
>> Inetz Media
>>
>>
>>> uname -a
>> FreeBSD pyrite.inetz.com 6.2-RELEASE-p4 FreeBSD 6.2-RELEASE-p4 #3: Tue
>> May  8 13:08:22 MDT 2007
>> root@biotite.inetz.com:/usr/obj/usr/src/sys/PYRITE  i386
>>
>>
>> root@pyrite /usr/obj/usr/src/sys/PYRITE > kgdb kernel.debug
>> /var/crash/vmcore.0
>> [GDB will not be able to debug user-mode threads:
>> /usr/lib/libthread_db.so: Undefined symbol
"ps_pglobal_lookup"]
>> GNU gdb 6.1.1 [FreeBSD]
>>
>> Unread portion of the kernel message buffer:
>> spin lock smp rendezvous held by 0xc95a5900 for > 5 seconds
> 
> What is thread 0xc95a5900 doing?
> 
> Kris
> 
Is this the best way to determine the action of the thread?

Regards;
Scott

(kgdb) info threads
  90 Thread 100123 (PID=18303: bsdtar)  0xc0644f5b in sched_switch
(td=0xc95a5900, newtd=0xc92aad80, flags=0) at
/usr/src/sys/kern/sched_4bsd.c:973
  89 Thread 100140 (PID=17447: sh)  0xc0644f5b in sched_switch
(td=0xc9a67c00, newtd=0xc95a5900, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  88 Thread 100144 (PID=17441: tcsh)  0xc0644f5b in sched_switch
(td=0xc9a67600, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  87 Thread 100125 (PID=17440: sshd)  0xc0644f5b in sched_switch
(td=0xc95a5600, newtd=0xc95a5900, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  86 Thread 100109 (PID=17437: sshd)  0xc0644f5b in sched_switch
(td=0xc9a41480, newtd=0xc9a67600, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  85 Thread 100039 (PID=898: getty)  0xc0644f5b in sched_switch
(td=0xc93e8900, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  84 Thread 100067 (PID=897: getty)  0xc0644f5b in sched_switch
(td=0xc95a6d80, newtd=0xc93e8900, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  83 Thread 100087 (PID=896: getty)  0xc0644f5b in sched_switch
(td=0xc9604180, newtd=0xc9903180, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  82 Thread 100088 (PID=895: getty)  0xc0644f5b in sched_switch
(td=0xc9903300, newtd=0xc9832780, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  81 Thread 100089 (PID=894: getty)  0xc0644f5b in sched_switch
(td=0xc9903180, newtd=0xc92aa780, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  80 Thread 100082 (PID=893: getty)  0xc0644f5b in sched_switch
(td=0xc9604900, newtd=0xc9903180, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  79 Thread 100073 (PID=892: getty)  0xc0644f5b in sched_switch
(td=0xc9832780, newtd=0xc9832180, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  78 Thread 100077 (PID=891: getty)  0xc0644f5b in sched_switch
(td=0xc9832180, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  77 Thread 100078 (PID=890: getty)  0xc0644f5b in sched_switch
(td=0xc9832000, newtd=0xc9604900, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  76 Thread 100090 (PID=849: cron)  0xc0644f5b in sched_switch
(td=0xc9903000, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  75 Thread 100056 (PID=843: sendmail)  0xc0644f5b in sched_switch
(td=0xc9604000, newtd=0xc92aa780, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  74 Thread 100069 (PID=839: sendmail)  0xc0644f5b in sched_switch
(td=0xc95a6a80, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  73 Thread 100063 (PID=833: sshd)  0xc0644f5b in sched_switch
(td=0xc9603480, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  72 Thread 100107 (PID=832: mysqld)  0xc0644f5b in sched_switch
(td=0xc9a41780, newtd=0xc92aa780, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  71 Thread 100106 (PID=832: mysqld)  0xc0644f5b in sched_switch
(td=0xc9a41900, newtd=0xc92aa780, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  70 Thread 100105 (PID=832: mysqld)  0xc0644f5b in sched_switch
(td=0xc9a41a80, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  69 Thread 100102 (PID=832: mysqld)  0xc0644f5b in sched_switch
(td=0xc95a5c00, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  68 Thread 100104 (PID=832: mysqld)  0xc0644f5b in sched_switch
(td=0xc9a41c00, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  67 Thread 100103 (PID=832: mysqld)  0xc0644f5b in sched_switch
(td=0xc9832a80, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  66 Thread 100101 (PID=832: mysqld)  0xc0644f5b in sched_switch
(td=0xc9832d80, newtd=0xc9a41780, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  65 Thread 100100 (PID=832: mysqld)  0xc0644f5b in sched_switch
(td=0xc9902000, newtd=0xc9a67c00, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  64 Thread 100099 (PID=832: mysqld)  0xc0644f5b in sched_switch
(td=0xc9902180, newtd=0xc9a67c00, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  63 Thread 100098 (PID=832: mysqld)  0xc0644f5b in sched_switch
(td=0xc9902300, newtd=0xc92aa780, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  62 Thread 100060 (PID=832: mysqld)  0xc0644f5b in sched_switch
(td=0xc9603900, newtd=0xc9a41780, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  61 Thread 100059 (PID=796: sh)  0xc0644f5b in sched_switch
(td=0xc9603a80, newtd=0xc9603900, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  60 Thread 100070 (PID=784: usbd)  0xc0644f5b in sched_switch
(td=0xc95a6900, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  59 Thread 100076 (PID=764: ntpd)  0xc0644f5b in sched_switch
(td=0xc9832300, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  58 Thread 100051 (PID=734: nrpe2)  0xc0644f5b in sched_switch
(td=0xc95a6480, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  57 Thread 100074 (PID=720: nfsd)  0xc0644f5b in sched_switch
(td=0xc9832600, newtd=0xc95a5d80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  56 Thread 100061 (PID=718: nfsd)  0xc0644f5b in sched_switch
(td=0xc9603780, newtd=0xc95a6d80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  55 Thread 100057 (PID=717: nfsd)  0xc0644f5b in sched_switch
(td=0xc9603d80, newtd=0xc93e8780, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  54 Thread 100038 (PID=716: nfsd)  0xc0644f5b in sched_switch
(td=0xc93e8a80, newtd=0xc93e8780, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  53 Thread 100040 (PID=715: nfsd)  0xc0644f5b in sched_switch
(td=0xc93e8780, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  52 Thread 100052 (PID=713: mountd)  0xc0644f5b in sched_switch
(td=0xc95a6300, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  51 Thread 100075 (PID=652: rpcbind)  0xc0644f5b in sched_switch
(td=0xc9832480, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  50 Thread 100071 (PID=634: syslogd)  0xc0644f5b in sched_switch
(td=0xc95a6780, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  49 Thread 100050 (PID=568: devd)  0xc0644f5b in sched_switch
(td=0xc95a6600, newtd=0xc92aa780, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  48 Thread 100058 (PID=136: adjkerntz)  0xc0644f5b in sched_switch
(td=0xc9603c00, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  47 Thread 100042 (PID=46: schedcpu)  0xc0644f5b in sched_switch
(td=0xc93e8480, newtd=0xc92aad80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  46 Thread 100043 (PID=45: softdepflush)  0xc0644f5b in sched_switch
(td=0xc93e8300, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
---Type <return> to continue, or q <return> to quit---
  45 Thread 100044 (PID=44: vnlru)  0xc0644f5b in sched_switch
(td=0xc93e8180, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
* 44 Thread 100045 (PID=43: syncer)  doadump () at pcpu.h:165
  43 Thread 100046 (PID=42: bufdaemon)  0xc0644f5b in sched_switch
(td=0xc93abd80, newtd=0xc93ab000, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  42 Thread 100047 (PID=41: pagezero)  0xc0644f5b in sched_switch
(td=0xc93abc00, newtd=0xc92aa780, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  41 Thread 100048 (PID=40: vmdaemon)  0xc0644f5b in sched_switch
(td=0xc93aba80, newtd=0xc93e8480, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  40 Thread 100049 (PID=39: pagedaemon)  0xc0644f5b in sched_switch
(td=0xc93ab900, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  39 Thread 100027 (PID=38: irq7: ppc0)  fork_trampoline () at
/usr/src/sys/i386/i386/exception.s:205
  38 Thread 100028 (PID=37: fdc0)  0xc0644f5b in sched_switch
(td=0xc93ab000, newtd=0xc93e8180, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  37 Thread 100029 (PID=36: swi0: sio)  0xc0644f5b in sched_switch
(td=0xc92afd80, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  36 Thread 100030 (PID=35: irq1: atkbd0)  0xc0644f5b in sched_switch
(td=0xc92afc00, newtd=0xc92af300, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  35 Thread 100031 (PID=34: irq15: ata1)  0xc0644f5b in sched_switch
(td=0xc92afa80, newtd=0xc92af180, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  34 Thread 100032 (PID=33: irq14: ata0)  fork_trampoline () at
/usr/src/sys/i386/i386/exception.s:205
  33 Thread 100033 (PID=32: usb2)  0xc0644f5b in sched_switch
(td=0xc92af780, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  32 Thread 100034 (PID=31: irq18: uhci2)  fork_trampoline () at
/usr/src/sys/i386/i386/exception.s:205
  31 Thread 100035 (PID=30: usb1)  0xc0644f5b in sched_switch
(td=0xc92af480, newtd=0xc92af780, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  30 Thread 100036 (PID=29: irq19: uhci1)  fork_trampoline () at
/usr/src/sys/i386/i386/exception.s:205
  29 Thread 100037 (PID=28: usbtask)  0xc0644f5b in sched_switch
(td=0xc93e8c00, newtd=0xc92af480, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  28 Thread 100017 (PID=27: usb0)  0xc0644f5b in sched_switch
(td=0xc92ab900, newtd=0xc92af480, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  27 Thread 100018 (PID=26: irq16: uhci0)  fork_trampoline () at
/usr/src/sys/i386/i386/exception.s:205
  26 Thread 100019 (PID=25: irq72: asr0)  0xc0644f5b in sched_switch
(td=0xc92ab600, newtd=0xc93ab780, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  25 Thread 100020 (PID=24: irq29: em1)  fork_trampoline () at
/usr/src/sys/i386/i386/exception.s:205
  24 Thread 100021 (PID=23: irq28: em0)  0xc0644f5b in sched_switch
(td=0xc92ab300, newtd=0xc92aa780, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  23 Thread 100022 (PID=22: irq9: acpi0)  fork_trampoline () at
/usr/src/sys/i386/i386/exception.s:205
  22 Thread 100023 (PID=21: swi2: cambio)  0xc0644f5b in sched_switch
(td=0xc93ab780, newtd=0xc92abc00, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  21 Thread 100024 (PID=9: kqueue taskq)  0xc0644f5b in sched_switch
(td=0xc93ab600, newtd=0xc09475c0, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  20 Thread 100025 (PID=8: acpi_task_2)  0xc0644f5b in sched_switch
(td=0xc93ab480, newtd=0xc93ab600, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  19 Thread 100026 (PID=7: acpi_task_1)  0xc0644f5b in sched_switch
(td=0xc93ab300, newtd=0xc93ab480, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  18 Thread 100008 (PID=6: acpi_task_0)  0xc0644f5b in sched_switch
(td=0xc92aa300, newtd=0xc93ab300, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  17 Thread 100009 (PID=20: swi5: +)  fork_trampoline () at
/usr/src/sys/i386/i386/exception.s:205
  16 Thread 100010 (PID=5: thread taskq)  0xc0644f5b in sched_switch
(td=0xc92aa000, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  15 Thread 100011 (PID=19: swi6: Giant taskq)  0xc0644f5b in
sched_switch (td=0xc92af300, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  14 Thread 100012 (PID=18: swi6: task queue)  0xc0644f5b in
sched_switch (td=0xc92af180, newtd=0xc92aa780, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  13 Thread 100013 (PID=17: yarrow)  0xc0644f5b in sched_switch
(td=0xc92af000, newtd=0xc92aba80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  12 Thread 100014 (PID=4: g_down)  0xc0644f5b in sched_switch
(td=0xc92abd80, newtd=0xc93e8000, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  11 Thread 100015 (PID=3: g_up)  0xc0644f5b in sched_switch
(td=0xc92abc00, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  10 Thread 100016 (PID=2: g_event)  0xc0644f5b in sched_switch
(td=0xc92aba80, newtd=0xc92aaa80, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  9 Thread 100000 (PID=16: swi3: vm)  fork_trampoline () at
/usr/src/sys/i386/i386/exception.s:205
  8 Thread 100001 (PID=15: swi4: clock sio)  0xc0644f5b in sched_switch
(td=0xc92aad80, newtd=0xc93e8000, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  7 Thread 100002 (PID=14: swi1: net)  0xc0644f5b in sched_switch
(td=0xc92aac00, newtd=0xc92aa780, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  6 Thread 100003 (PID=13: idle: cpu0)  0xc0644f5b in sched_switch
(td=0xc92aaa80, newtd=0xc92aad80, flags=2) at
/usr/src/sys/kern/sched_4bsd.c:973
  5 Thread 100004 (PID=12: idle: cpu1)  fork_trampoline () at
/usr/src/sys/i386/i386/exception.s:205
  4 Thread 100005 (PID=11: idle: cpu2)  0xc0644f5b in sched_switch
(td=0xc92aa780, newtd=0xc95a5900, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  3 Thread 100006 (PID=10: idle: cpu3)  fork_trampoline () at
/usr/src/sys/i386/i386/exception.s:205
  2 Thread 100007 (PID=1: init)  0xc0644f5b in sched_switch
(td=0xc92aa480, newtd=0xc9603480, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973
  1 Thread 0 (PID=0: swapper)  0xc0644f5b in sched_switch
(td=0xc09475c0, newtd=0xc93ab780, flags=1) at
/usr/src/sys/kern/sched_4bsd.c:973

Scott Swanson

2007-May-11 16:44 UTC

head link

panic: spin lock held too long (w/ backtrace)

>> root@pyrite /usr/src/sys/i386/conf > kldstat
>> Id Refs Address    Size     Name
>>  1    3 0xc0400000 65e308   kernel
>>  2    1 0xc0a5f000 59f20    acpi.ko
> 
> So, yes then :) Can you follow the steps for debugging modules and see
> if it gives a better trace?
> 
> Kris
> 
Unfortunately, after prepping and adding the symbol file for acpi.ko, I
got the exact same backtrace.  Any other thoughts?

Regards;
Scott

Kris Kennaway

2007-May-11 18:47 UTC

head link

panic: spin lock held too long (w/ backtrace)

On Fri, May 11, 2007 at 10:43:54AM -0600, Scott Swanson
wrote:> >> root@pyrite /usr/src/sys/i386/conf > kldstat
> >> Id Refs Address    Size     Name
> >>  1    3 0xc0400000 65e308   kernel
> >>  2    1 0xc0a5f000 59f20    acpi.ko
> > 
> > So, yes then :) Can you follow the steps for debugging modules and see
> > if it gives a better trace?
> > 
> > Kris
> > 
> 
> Unfortunately, after prepping and adding the symbol file for acpi.ko, I
> got the exact same backtrace.  Any other thoughts?
Sometimes -O2 can confuse gdb...unfortunately there is no way to
repair it after the fact.  Maybe someone else has ideas.

Kris

freebsd stable - May 2007 - panic: spin lock held too long (w/ backtrace)

panic: spin lock held too long (w/ backtrace)

panic: spin lock held too long (w/ backtrace)

panic: spin lock held too long (w/ backtrace)

panic: spin lock held too long (w/ backtrace)

panic: spin lock held too long (w/ backtrace)