All -
I've been getting panics every couple of days (sometimes daily) with
6.2-STABLE (as of Feb 12th). These have been happening ever since
sometime early in 6.1, and I believe they are related to the Quagga
routing daemon, specifically when the zebra(8) process adds or removes
routes.
I have two PR's open regarding them:
http://www.freebsd.org/cgi/query-pr.cgi?pr=104569
http://www.freebsd.org/cgi/query-pr.cgi?pr=105966
105966 was resolved back in Dec. 2006, but resurfaced again in Jan.
2007. I have not heard anything on 104569.
Has anyone else experienced similar problems? Any suggestions?
Here's an example of the panic that's described in kern/104569:
Unread portion of the kernel message buffer:
kernel trap 12 with interrupts disabled
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0x78
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc0554bcb
stack pointer = 0x28:0xdea8ea64
frame pointer = 0x28:0xdea8ea68
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = resume, IOPL = 0
current process = 1548 (zebra)
trap number = 12
panic: page fault
Uptime: 2d5h52m33s
Dumping 510 MB (2 chunks)
chunk 0: 1MB (159 pages) ... ok
chunk 1: 510MB (130544 pages) 494 478 462 446 430 414 398 382 366 350 334 318
302 286 270 254 238 222 206 190 174 158 142 126 110 94 78 62 46 30 14
#0 doadump () at pcpu.h:165
165 __asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb) bt
#0 doadump () at pcpu.h:165
#1 0xc052f46e in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2 0xc052f778 in panic (fmt=0xc0709d51 "%s") at
/usr/src/sys/kern/kern_shutdown.c:565
#3 0xc06e5d2d in trap_fatal (frame=0xdea8ea24, eva=0) at
/usr/src/sys/i386/i386/trap.c:837
#4 0xc06e5445 in trap (frame= {tf_fs = -629538808, tf_es = -1066139608, tf_ds =
40, tf_edi = -1015486796, tf_esi = -1014488704, tf_ebp = -559355288, tf_isp =
-559355312, tf_ebx = -1015492032, tf_edx = -1014488704, tf_ecx = 4, tf_eax = 4,
tf_trapno = 12, tf_err = 0, tf_eip = -1068151861, tf_cs = 32, tf_eflags = 65543,
tf_esp = -1014488704, tf_ss = -559355252}) at /usr/src/sys/i386/i386/trap.c:270
#5 0xc06d27ca in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#6 0xc0554bcb in turnstile_setowner (ts=0xc378d240, owner=0x4) at
/usr/src/sys/kern/subr_turnstile.c:432
#7 0xc0554ef7 in turnstile_wait (lock=0xc38c6504, owner=0x4) at
/usr/src/sys/kern/subr_turnstile.c:591
#8 0xc0524ddb in _mtx_lock_sleep (m=0xc38c6504, tid=3280478592, opts=0,
file=0x0, line=0) at /usr/src/sys/kern/kern_mutex.c:579
#9 0xc05bcb44 in rtrequest1 (req=2, info=0xdea8eb24, ret_nrt=0xdea8eb10) at
/usr/src/sys/net/route.c:703
#10 0xc05be7e5 in route_output (m=0xc55fa800, so=0xc3553164) at
/usr/src/sys/net/rtsock.c:391
#11 0xc05bbb12 in raw_usend (so=0x4, flags=0, m=0xc3882180, nam=0x0,
control=0x4, td=0xc3882180) at /usr/src/sys/net/raw_usrreq.c:263
#12 0xc05be457 in rts_send (so=0x4, flags=4, m=0x4, nam=0x4, control=0x4,
td=0x4) at /usr/src/sys/net/rtsock.c:269
#13 0xc057136c in sosend (so=0xc3553164, addr=0x0, uio=0xdea8ecb0,
top=0xc55fa800, control=0x0, flags=0, td=0xc3882180) at
/usr/src/sys/kern/uipc_socket.c:836
#14 0xc055d2b8 in soo_write (fp=0x4, uio=0xdea8ecb0, active_cred=0xc33d2c00,
flags=0, td=0xc3882180) at /usr/src/sys/kern/sys_socket.c:118
#15 0xc05569e0 in dofilewrite (td=0xc3882180, fd=4, fp=0xc37b0c18,
auio=0xdea8ecb0, offset=Unhandled dwarf expression opcode 0x93) at file.h:252
#16 0xc0556817 in kern_writev (td=0xc3882180, fd=6, auio=0x4) at
/usr/src/sys/kern/sys_generic.c:402
---Type <return> to continue, or q <return> to quit---
#17 0xc05566e9 in write (td=0x4, uap=0x4) at /usr/src/sys/kern/sys_generic.c:326
#18 0xc06e60e3 in syscall (frame= {tf_fs = 672006203, tf_es = 672006203, tf_ds =
-1078001605, tf_edi = -1077941792, tf_esi = -1077942328, tf_ebp = -1077941864,
tf_isp = -559354524, tf_ebx = 20, tf_edx = -1077942496, tf_ecx = 0, tf_eax = 4,
tf_trapno = 0, tf_err = 2, tf_eip = 673045383, tf_cs = 51, tf_eflags = 514,
tf_esp = -1077942516, tf_ss = 59}) at /usr/src/sys/i386/i386/trap.c:983
#19 0xc06d281f in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:200
#20 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb)
Here's one from kern/105966:
Unread portion of the kernel message buffer:
kernel trap 12 with interrupts disabled
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0x78
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc0555579
stack pointer = 0x28:0xd43f2b28
frame pointer = 0x28:0xd43f2b2c
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = resume, IOPL = 0
current process = 11 (swi1: net)
trap number = 12
panic: page fault
Uptime: 9d19h23m7s
Dumping 510 MB (2 chunks)
chunk 0: 1MB (159 pages) ... ok
chunk 1: 510MB (130544 pages) 494 478 462 446 430 414 398 382 366 350 334 318
302 286 270 254 238 222 206 190 174 158 142 126 110 94 78 62 46 30 14
#0 doadump () at pcpu.h:165
165 __asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb) bt
#0 doadump () at pcpu.h:165
#1 0xc052fe16 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2 0xc0530120 in panic (fmt=0xc070b714 "%s") at
/usr/src/sys/kern/kern_shutdown.c:565
#3 0xc06e75c5 in trap_fatal (frame=0xd43f2ae8, eva=0) at
/usr/src/sys/i386/i386/trap.c:837
#4 0xc06e6cdd in trap (frame= {tf_fs = -1067450360, tf_es = -734068696, tf_ds =
40, tf_edi = -1019703296, tf_esi = -1020561536, tf_ebp = -734057684, tf_isp =
-734057708, tf_ebx = -1020603584, tf_edx = -1020561536, tf_ecx = 4, tf_eax = 4,
tf_trapno = 12, tf_err = 0, tf_eip = -1068149383, tf_cs = 32, tf_eflags = 65543,
tf_esp = -1020561536, tf_ss = -734057648}) at /usr/src/sys/i386/i386/trap.c:270
#5 0xc06d408a in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#6 0xc0555579 in turnstile_setowner (ts=0xc32ad340, owner=0x4)
at /usr/src/sys/kern/subr_turnstile.c:434
#7 0xc05558a5 in turnstile_wait (lock=0xc38a9504, owner=0x4)
at /usr/src/sys/kern/subr_turnstile.c:593
#8 0xc0525783 in _mtx_lock_sleep (m=0xc38a9504, tid=3274405760, opts=0,
file=0x0, line=0) at /usr/src/sys/kern/kern_mutex.c:579
#9 0xc06016ae in nd6_output (ifp=0xc3389000, origifp=0x4, m0=0xc60f5500,
dst=0xc38a831c, rt0=0xc3764630) at /usr/src/sys/netinet6/nd6.c:2010
#10 0xc05f5218 in ip6_forward (m=0xc60f5500, srcrt=0) at
/usr/src/sys/netinet6/ip6_forward.c:626
#11 0xc05f64ad in ip6_input (m=0xc60f5500) at
/usr/src/sys/netinet6/ip6_input.c:732
#12 0xc05b8a67 in netisr_processqueue (ni=0xc0779d44) at
/usr/src/sys/net/netisr.c:236
#13 0xc05b8c5d in swi_net (dummy=0x0) at /usr/src/sys/net/netisr.c:343
#14 0xc0516cca in ithread_execute_handlers (p=0xc32b6a78, ie=0xc32f8300)
at /usr/src/sys/kern/kern_intr.c:682
#15 0xc0516e0b in ithread_loop (arg=0xc3283700) at
/usr/src/sys/kern/kern_intr.c:765
#16 0xc0515901 in fork_exit (callout=0xc0516da8 <ithread_loop>, arg=0x4,
frame=0x4) at /usr/src/sys/kern/kern_fork.c:821
#17 0xc06d40ec in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:208
(kgdb)
Thanks!
- Mark
--
Mark Kamichoff
prox@prolixium.com
http://prolixium.com/
Rensselaer Polytechnic Institute, Class of 2004