The box has a fairly heavy UDP load. Its RELENG_7 as of today and
took 3hrs for it to dump core.
Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address = 0x68
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc0637146
stack pointer = 0x28:0xe766eaac
frame pointer = 0x28:0xe766eb54
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 761 (bsnmpd)
trap number = 12
panic: page fault
cpuid = 1
Uptime: 3h47m43s
Physical memory: 2036 MB
Dumping 83 MB: 68 52 36 20 4
(kgdb) bt
#0 doadump () at pcpu.h:196
#1 0xc05964d7 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418
#2 0xc05967a9 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:574
#3 0xc07f64ac in trap_fatal (frame=0xe766ea6c, eva=104) at
/usr/src/sys/i386/i386/trap.c:939
#4 0xc07f6730 in trap_pfault (frame=0xe766ea6c, usermode=0, eva=104)
at /usr/src/sys/i386/i386/trap.c:852
#5 0xc07f70dc in trap (frame=0xe766ea6c) at /usr/src/sys/i386/i386/trap.c:530
#6 0xc07db7eb in calltrap () at /usr/src/sys/i386/i386/exception.s:159
#7 0xc0637146 in sysctl_ifdata (oidp=0xc08816a0, arg1=0xe766ec24,
arg2=2, req=0xe766eba4) at /usr/src/sys/net/if_mib.c:127
#8 0xc059fd77 in sysctl_root (oidp=Variable "oidp" is not available.
) at /usr/src/sys/kern/kern_sysctl.c:1413
#9 0xc059ff14 in userland_sysctl (td=0xc5374460, name=0xe766ec14,
namelen=6, old=0x0, oldlenp=0xbfbf8478, inkernel=0, new=0x0,
newlen=0, retval=0xe766ec10, flags=0) at
/usr/src/sys/kern/kern_sysctl.c:1506
#10 0xc05a0064 in __sysctl (td=0xc5374460, uap=0xe766ecfc) at
/usr/src/sys/kern/kern_sysctl.c:1443
#11 0xc07f6a85 in syscall (frame=0xe766ed38) at
/usr/src/sys/i386/i386/trap.c:1090
#12 0xc07db850 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:255
#13 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb)
--------------------------------------------------------------------
Mike Tancsa, tel +1 519 651 3400
Sentex Communications, mike@sentex.net
Providing Internet since 1994 www.sentex.net
Cambridge, Ontario Canada www.sentex.net/mike
On Tuesday 21 April 2009 1:25:06 am Mike Tancsa wrote:> The box has a fairly heavy UDP load. Its RELENG_7 as of today and > took 3hrs for it to dump core. > > Fatal trap 12: page fault while in kernel mode > cpuid = 1; apic id = 01 > fault virtual address = 0x68 > fault code = supervisor read, page not present > instruction pointer = 0x20:0xc0637146 > stack pointer = 0x28:0xe766eaac > frame pointer = 0x28:0xe766eb54 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, def32 1, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 761 (bsnmpd) > trap number = 12 > panic: page fault > cpuid = 1 > Uptime: 3h47m43s > Physical memory: 2036 MB > Dumping 83 MB: 68 52 36 20 4 > > (kgdb) bt > #0 doadump () at pcpu.h:196 > #1 0xc05964d7 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418 > #2 0xc05967a9 in panic (fmt=Variable "fmt" is not available. > ) at /usr/src/sys/kern/kern_shutdown.c:574 > #3 0xc07f64ac in trap_fatal (frame=0xe766ea6c, eva=104) at > /usr/src/sys/i386/i386/trap.c:939 > #4 0xc07f6730 in trap_pfault (frame=0xe766ea6c, usermode=0, eva=104) > at /usr/src/sys/i386/i386/trap.c:852 > #5 0xc07f70dc in trap (frame=0xe766ea6c)at /usr/src/sys/i386/i386/trap.c:530> #6 0xc07db7eb in calltrap () at /usr/src/sys/i386/i386/exception.s:159 > #7 0xc0637146 in sysctl_ifdata (oidp=0xc08816a0, arg1=0xe766ec24, > arg2=2, req=0xe766eba4) at /usr/src/sys/net/if_mib.c:127 > #8 0xc059fd77 in sysctl_root (oidp=Variable "oidp" is not available. > ) at /usr/src/sys/kern/kern_sysctl.c:1413 > #9 0xc059ff14 in userland_sysctl (td=0xc5374460, name=0xe766ec14, > namelen=6, old=0x0, oldlenp=0xbfbf8478, inkernel=0, new=0x0, > newlen=0, retval=0xe766ec10, flags=0) at > /usr/src/sys/kern/kern_sysctl.c:1506 > #10 0xc05a0064 in __sysctl (td=0xc5374460, uap=0xe766ecfc) at > /usr/src/sys/kern/kern_sysctl.c:1443 > #11 0xc07f6a85 in syscall (frame=0xe766ed38) at > /usr/src/sys/i386/i386/trap.c:1090 > #12 0xc07db850 in Xint0x80_syscall ()at /usr/src/sys/i386/i386/exception.s:255> #13 0x00000033 in ?? () > Previous frame inner to this frame (corrupt stack?) > (kgdb)Can you do 'frame 7' followed by 'l', 'p ifp', and 'p ifp->if_snd'? -- John Baldwin
On Tue, 21 Apr 2009 01:25:06 -0400 Mike Tancsa wrote: MT> The box has a fairly heavy UDP load. Its RELENG_7 as of today and MT> took 3hrs for it to dump core. MT> Fatal trap 12: page fault while in kernel mode MT> cpuid = 1; apic id = 01 MT> fault virtual address = 0x68 MT> fault code = supervisor read, page not present MT> instruction pointer = 0x20:0xc0637146 MT> stack pointer = 0x28:0xe766eaac MT> frame pointer = 0x28:0xe766eb54 MT> code segment = base 0x0, limit 0xfffff, type 0x1b MT> = DPL 0, pres 1, def32 1, gran 1 MT> processor eflags = interrupt enabled, resume, IOPL = 0 MT> current process = 761 (bsnmpd) MT> trap number = 12 MT> panic: page fault MT> cpuid = 1 MT> Uptime: 3h47m43s MT> Physical memory: 2036 MB MT> Dumping 83 MB: 68 52 36 20 4 MT> (kgdb) bt MT> #0 doadump () at pcpu.h:196 MT> #1 0xc05964d7 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418 MT> #2 0xc05967a9 in panic (fmt=Variable "fmt" is not available. MT> ) at /usr/src/sys/kern/kern_shutdown.c:574 MT> #3 0xc07f64ac in trap_fatal (frame=0xe766ea6c, eva=104) at MT> /usr/src/sys/i386/i386/trap.c:939 MT> #4 0xc07f6730 in trap_pfault (frame=0xe766ea6c, usermode=0, eva=104) MT> at /usr/src/sys/i386/i386/trap.c:852 MT> #5 0xc07f70dc in trap (frame=0xe766ea6c) at /usr/src/sys/i386/i386/trap.c:530 MT> #6 0xc07db7eb in calltrap () at /usr/src/sys/i386/i386/exception.s:159 MT> #7 0xc0637146 in sysctl_ifdata (oidp=0xc08816a0, arg1=0xe766ec24, MT> arg2=2, req=0xe766eba4) at /usr/src/sys/net/if_mib.c:127 MT> #8 0xc059fd77 in sysctl_root (oidp=Variable "oidp" is not available. MT> ) at /usr/src/sys/kern/kern_sysctl.c:1413 MT> #9 0xc059ff14 in userland_sysctl (td=0xc5374460, name=0xe766ec14, MT> namelen=6, old=0x0, oldlenp=0xbfbf8478, inkernel=0, new=0x0, MT> newlen=0, retval=0xe766ec10, flags=0) at MT> /usr/src/sys/kern/kern_sysctl.c:1506 MT> #10 0xc05a0064 in __sysctl (td=0xc5374460, uap=0xe766ecfc) at MT> /usr/src/sys/kern/kern_sysctl.c:1443 MT> #11 0xc07f6a85 in syscall (frame=0xe766ed38) at MT> /usr/src/sys/i386/i386/trap.c:1090 MT> #12 0xc07db850 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:255 MT> #13 0x00000033 in ?? () MT> Previous frame inner to this frame (corrupt stack?) MT> (kgdb) Just FYI, the same problem has already been registered in pr database as kern/132734. -- Mikolaj Golub
At 04:53 PM 4/21/2009, Mikolaj Golub wrote:>Just FYI, the same problem has already been registered in pr >database as kern/132734.Thanks, http://www.freebsd.org/cgi/query-pr.cgi?pr=132734 does look familiar :) If you disable the snmpwalk, is the box stable ? ---Mike
On Tue, 21 Apr 2009, Mike Tancsa wrote:> At 04:53 PM 4/21/2009, Mikolaj Golub wrote: > >> Just FYI, the same problem has already been registered in pr database as >> kern/132734. > > Thanks, > http://www.freebsd.org/cgi/query-pr.cgi?pr=132734 does look familiar > :) If you disable the snmpwalk, is the box stable ?It would be interesting to know if this tweak at least eliminates the instant panic: Index: if_mib.c ==================================================================--- if_mib.c (revision 191424) +++ if_mib.c (working copy) @@ -82,11 +82,9 @@ return EINVAL; if (name[0] <= 0 || name[0] > if_index || - ifnet_byindex(name[0]) == NULL) + (ifp = ifnet_byindex(name[0])) == NULL) return ENOENT; - ifp = ifnet_byindex(name[0]); - switch(name[1]) { default: return ENOENT; Robert N M Watson Computer Laboratory University of Cambridge