I finally(!) got around to enabling crash dumps on the primary machine
here at the house ... and managed to make use of it (unfortunately).
I've copied the relevant files (both those from /var/crash and
dmesg.boot) so they should be visibale at
<http://www.catwhisker.org/~david/FreeBSD/panic_24Dec2012/> (though
only the dmesg.boot, core.text.0, & info.0 files should be fetchable
for now). [I'll make the vmcore.0 available to individuals who
wish to work on the problem; please contact me to arrange this.]
Here's a bit of information excerpted from core.text.0:
Mon Dec 24 11:16:04 PST 2012
FreeBSD albert.catwhisker.org 9.1-PRERELEASE FreeBSD 9.1-PRERELEASE #434
244582M: Sat Dec 22 05:06:29 PST 2012 root at
freebeast.catwhisker.org:/usr/obj/usr/src/sys/ALBERT i386
Note that while the version string says "244582M":
* Userland was at r244608.
* The "Modification" was merely a change to src/sys/newvers.sh to
re-factor
the extraction of the version string.
panic: page fault
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x34
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc0ad475c
stack pointer = 0x28:0xc6fba9d8
frame pointer = 0x28:0xc6fbaa18
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = resume, IOPL = 0
current process = 11 (idle: cpu0)
trap number = 12
panic: page fault
cpuid = 0
KDB: stack backtrace:
db_trace_self_wrapper(c0ffbab8,46,1,ca931e80,0,...) at 0xc051ef76 =
db_trace_self_wrapper+0x36/frame 0xc6fba740
kdb_backtrace(c1033ff1,0,c0e75cc4,c6fba7ec,c71f08d0,...) at 0xc0afc400 =
kdb_backtrace+0x30/frame 0xc6fba7a0
panic(c0e75cc4,c1034ddb,c71f0a84,1,1,...) at 0xc0ac763c = panic+0x1bc/frame
0xc6fba7e0
trap_fatal(28,7fffffff,3,0,28,...) at 0xc0e35560 = trap_fatal+0x340/frame
0xc6fba828
trap_pfault(34,c,1,c11a68b0,c6fba940,...) at 0xc0e358cb =
trap_pfault+0x35b/frame 0xc6fba8a0
trap(c6fba998) at 0xc0e34e13 = trap+0x443/frame 0xc6fba98c
calltrap() at 0xc0e1e86c = calltrap+0x6/frame 0xc6fba98c
--- trap 0xc, eip = 0xc0ad475c, esp = 0xc6fba9d8, ebp = 0xc6fbaa18 ---
tc_windup(1,0,c0ff3ba6,21c,0,...) at 0xc0ad475c = tc_windup+0x1c/frame
0xc6fbaa18
hardclock_cnt(1,0,0,3,0,...) at 0xc0a77e39 = hardclock_cnt+0x2e9/frame
0xc6fbaa68
handleevents(c6fbaaf8,2,46,c71f08d0,c6fbaae4,...) at 0xc0e3c534 =
handleevents+0x184/frame 0xc6fbaac0
timercb(c7564064,0,c76a82f0,c6fbab58,c0a99a0e,...) at 0xc0e3d1a1 =
timercb+0x281/frame 0xc6fbab14
hpet_intr_single(c7564064,c7569780,0,c6fbabbc,c6fbab78,...) at 0xc053a345 =
hpet_intr_single+0x195/frame 0xc6fbab40
hpet_intr(c7564000,0,c71f08d0,14,c723b710,...) at 0xc053a3cf =
hpet_intr+0x6f/frame 0xc6fbab58
intr_event_handle(c723c280,c6fbabbc,c6fbab94,0,c7182600,...) at 0xc0a99c5c =
intr_event_handle+0x7c/frame 0xc6fbab78
intr_execute_handlers(c723b710,c6fbabbc,0) at 0xc0e4c552 =
intr_execute_handlers+0x42/frame 0xc6fbab98
lapic_handle_intr(33,c6fbabbc) at 0xc0e4f50d = lapic_handle_intr+0x3d/frame
0xc6fbabac
Xapic_isr1() at 0xc0e1ec35 = Xapic_isr1+0x35/frame 0xc6fbabac
--- interrupt, eip = 0xc0e1a202, esp = 0xc6fbabfc, ebp = 0xc6fbac3c ---
acpi_cpu_c1(0,c6fbac58,c0e250a6,0,c1198018,...) at 0xc0e1a202 =
acpi_cpu_c1+0x2/frame 0xc6fbac3c
cpu_idle_acpi(0,c1198018,c6fbacd0,c0aee519,0,...) at 0xc0e24fff =
cpu_idle_acpi+0x2f/frame 0xc6fbac48
cpu_idle(0,2,c0ffa49a,a36,c71f08d0,...) at 0xc0e250a6 = cpu_idle+0x96/frame
0xc6fbac58
sched_idletd(0,c6fbad08,0,0,c0aee250,...) at 0xc0aee519 =
sched_idletd+0x2c9/frame 0xc6fbacd0
fork_exit(c0aee250,0,c6fbad08) at 0xc0a977c7 = fork_exit+0x67/frame 0xc6fbacf4
fork_trampoline() at 0xc0e1e8e4 = fork_trampoline+0x8/frame 0xc6fbacf4
--- trap 0, eip = 0, esp = 0xc6fbad40, ebp = 0 ---
Uptime: 7h11m46s
Physical memory: 3045 MB
#0 doadump (textdump=<value optimized out>) at pcpu.h:249
249 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) #0 doadump (textdump=<value optimized out>) at pcpu.h:249
#1 0xc0ac71fa in kern_reboot (howto=Unhandled dwarf expression opcode 0xc0
)
at /usr/src/sys/kern/kern_shutdown.c:448
#2 0xc0ac7688 in panic (fmt=Unhandled dwarf expression opcode 0xc0
) at /usr/src/sys/kern/kern_shutdown.c:636
#3 0xc0e35560 in trap_fatal (frame=<value optimized out>,
eva=<value optimized out>) at /usr/src/sys/i386/i386/trap.c:1043
#4 0xc0e358cb in trap_pfault (frame=<value optimized out>,
usermode=Unhandled dwarf expression opcode 0xc3
)
at /usr/src/sys/i386/i386/trap.c:858
#5 0xc0e34e13 in trap (frame=<value optimized out>)
at /usr/src/sys/i386/i386/trap.c:555
#6 0xc0e1e86c in calltrap () at /tmp/exception-SmXQMs.s:94
#7 0xc0ad475c in tc_windup () at /usr/src/sys/kern/kern_tc.c:450
#8 0xc0a77e39 in hardclock_cnt (usermode=<value optimized out>)
at /usr/src/sys/kern/kern_clock.c:556
#9 0xc0e3c534 in handleevents (now=<value optimized out>,
fake=<value optimized out>) at
/usr/src/sys/kern/kern_clocksource.c:215
#10 0xc0e3d1a1 in timercb (et=Unhandled dwarf expression opcode 0xc0
) at /usr/src/sys/kern/kern_clocksource.c:390
#11 0xc053a345 in hpet_intr_single (arg=<value optimized out>)
at /usr/src/sys/dev/acpica/acpi_hpet.c:260
#12 0xc053a3cf in hpet_intr (arg=<value optimized out>)
at /usr/src/sys/dev/acpica/acpi_hpet.c:278
#13 0xc0a99c5c in intr_event_handle (ie=Unhandled dwarf expression opcode 0xa1
)
at /usr/src/sys/kern/kern_intr.c:1435
#14 0xc0e4c552 in intr_execute_handlers (isrc=<value optimized out>,
frame=<value optimized out>) at
/usr/src/sys/x86/x86/intr_machdep.c:269
#15 0xc0e4f50d in lapic_handle_intr (vector=-956585056, frame=0xc6fbaba0)
at /usr/src/sys/x86/x86/local_apic.c:780
#16 0xc0e1ec35 in Xapic_isr1 () at /tmp/exception-SmXQMs.s:214
#17 0xc0e1a202 in acpi_cpu_c1 ()
at /usr/src/sys/i386/acpica/acpi_machdep.c:114
Previous frame inner to this frame (corrupt stack?)
Any suggestions? At this point, I don't really know enough to file a PR.
Peace,
david
--
David H. Wolfskill david at catwhisker.org
Taliban: Evil men with guns afraid of truth from a 14-year old girl.
See http://www.catwhisker.org/~david/publickey.gpg for my public key.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
URL:
<http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20121224/f373d952/attachment.sig>
on 24/12/2012 21:58 David Wolfskill said the following:> I finally(!) got around to enabling crash dumps on the primary machine > here at the house ... and managed to make use of it (unfortunately). > > I've copied the relevant files (both those from /var/crash and > dmesg.boot) so they should be visibale at > <http://www.catwhisker.org/~david/FreeBSD/panic_24Dec2012/> (though > only the dmesg.boot, core.text.0, & info.0 files should be fetchable > for now). [I'll make the vmcore.0 available to individuals who > wish to work on the problem; please contact me to arrange this.] > > Here's a bit of information excerpted from core.text.0: > > Mon Dec 24 11:16:04 PST 2012 > > FreeBSD albert.catwhisker.org 9.1-PRERELEASE FreeBSD 9.1-PRERELEASE #434 244582M: Sat Dec 22 05:06:29 PST 2012 root at freebeast.catwhisker.org:/usr/obj/usr/src/sys/ALBERT i386 > > Note that while the version string says "244582M": > > * Userland was at r244608. > > * The "Modification" was merely a change to src/sys/newvers.sh to re-factor > the extraction of the version string. > > > > panic: page fault > > Fatal trap 12: page fault while in kernel mode > cpuid = 0; apic id = 00 > fault virtual address = 0x34 > fault code = supervisor read, page not present > instruction pointer = 0x20:0xc0ad475c > stack pointer = 0x28:0xc6fba9d8 > frame pointer = 0x28:0xc6fbaa18 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, def32 1, gran 1 > processor eflags = resume, IOPL = 0 > current process = 11 (idle: cpu0) > trap number = 12 > panic: page fault > cpuid = 0 > KDB: stack backtrace: > db_trace_self_wrapper(c0ffbab8,46,1,ca931e80,0,...) at 0xc051ef76 = db_trace_self_wrapper+0x36/frame 0xc6fba740 > kdb_backtrace(c1033ff1,0,c0e75cc4,c6fba7ec,c71f08d0,...) at 0xc0afc400 = kdb_backtrace+0x30/frame 0xc6fba7a0 > panic(c0e75cc4,c1034ddb,c71f0a84,1,1,...) at 0xc0ac763c = panic+0x1bc/frame 0xc6fba7e0 > trap_fatal(28,7fffffff,3,0,28,...) at 0xc0e35560 = trap_fatal+0x340/frame 0xc6fba828 > trap_pfault(34,c,1,c11a68b0,c6fba940,...) at 0xc0e358cb = trap_pfault+0x35b/frame 0xc6fba8a0 > trap(c6fba998) at 0xc0e34e13 = trap+0x443/frame 0xc6fba98c > calltrap() at 0xc0e1e86c = calltrap+0x6/frame 0xc6fba98c > --- trap 0xc, eip = 0xc0ad475c, esp = 0xc6fba9d8, ebp = 0xc6fbaa18 --- > tc_windup(1,0,c0ff3ba6,21c,0,...) at 0xc0ad475c = tc_windup+0x1c/frame 0xc6fbaa18 > hardclock_cnt(1,0,0,3,0,...) at 0xc0a77e39 = hardclock_cnt+0x2e9/frame 0xc6fbaa68 > handleevents(c6fbaaf8,2,46,c71f08d0,c6fbaae4,...) at 0xc0e3c534 = handleevents+0x184/frame 0xc6fbaac0 > timercb(c7564064,0,c76a82f0,c6fbab58,c0a99a0e,...) at 0xc0e3d1a1 = timercb+0x281/frame 0xc6fbab14 > hpet_intr_single(c7564064,c7569780,0,c6fbabbc,c6fbab78,...) at 0xc053a345 = hpet_intr_single+0x195/frame 0xc6fbab40 > hpet_intr(c7564000,0,c71f08d0,14,c723b710,...) at 0xc053a3cf = hpet_intr+0x6f/frame 0xc6fbab58 > intr_event_handle(c723c280,c6fbabbc,c6fbab94,0,c7182600,...) at 0xc0a99c5c = intr_event_handle+0x7c/frame 0xc6fbab78 > intr_execute_handlers(c723b710,c6fbabbc,0) at 0xc0e4c552 = intr_execute_handlers+0x42/frame 0xc6fbab98 > lapic_handle_intr(33,c6fbabbc) at 0xc0e4f50d = lapic_handle_intr+0x3d/frame 0xc6fbabac > Xapic_isr1() at 0xc0e1ec35 = Xapic_isr1+0x35/frame 0xc6fbabac > --- interrupt, eip = 0xc0e1a202, esp = 0xc6fbabfc, ebp = 0xc6fbac3c --- > acpi_cpu_c1(0,c6fbac58,c0e250a6,0,c1198018,...) at 0xc0e1a202 = acpi_cpu_c1+0x2/frame 0xc6fbac3c > cpu_idle_acpi(0,c1198018,c6fbacd0,c0aee519,0,...) at 0xc0e24fff = cpu_idle_acpi+0x2f/frame 0xc6fbac48 > cpu_idle(0,2,c0ffa49a,a36,c71f08d0,...) at 0xc0e250a6 = cpu_idle+0x96/frame 0xc6fbac58 > sched_idletd(0,c6fbad08,0,0,c0aee250,...) at 0xc0aee519 = sched_idletd+0x2c9/frame 0xc6fbacd0 > fork_exit(c0aee250,0,c6fbad08) at 0xc0a977c7 = fork_exit+0x67/frame 0xc6fbacf4 > fork_trampoline() at 0xc0e1e8e4 = fork_trampoline+0x8/frame 0xc6fbacf4 > --- trap 0, eip = 0, esp = 0xc6fbad40, ebp = 0 --- > Uptime: 7h11m46s > Physical memory: 3045 MB > > > #0 doadump (textdump=<value optimized out>) at pcpu.h:249 > 249 pcpu.h: No such file or directory. > in pcpu.h > (kgdb) #0 doadump (textdump=<value optimized out>) at pcpu.h:249 > #1 0xc0ac71fa in kern_reboot (howto=Unhandled dwarf expression opcode 0xc0 > ) > at /usr/src/sys/kern/kern_shutdown.c:448 > #2 0xc0ac7688 in panic (fmt=Unhandled dwarf expression opcode 0xc0 > ) at /usr/src/sys/kern/kern_shutdown.c:636 > #3 0xc0e35560 in trap_fatal (frame=<value optimized out>, > eva=<value optimized out>) at /usr/src/sys/i386/i386/trap.c:1043 > #4 0xc0e358cb in trap_pfault (frame=<value optimized out>, usermode=Unhandled dwarf expression opcode 0xc3 > ) > at /usr/src/sys/i386/i386/trap.c:858 > #5 0xc0e34e13 in trap (frame=<value optimized out>) > at /usr/src/sys/i386/i386/trap.c:555 > #6 0xc0e1e86c in calltrap () at /tmp/exception-SmXQMs.s:94 > #7 0xc0ad475c in tc_windup () at /usr/src/sys/kern/kern_tc.c:450I'd say that what you see is impossible... Could you please provide the following info from kgdb? p timehands p th0 ... p th9 disassemble tc_windup> #8 0xc0a77e39 in hardclock_cnt (usermode=<value optimized out>) > at /usr/src/sys/kern/kern_clock.c:556 > #9 0xc0e3c534 in handleevents (now=<value optimized out>, > fake=<value optimized out>) at /usr/src/sys/kern/kern_clocksource.c:215 > #10 0xc0e3d1a1 in timercb (et=Unhandled dwarf expression opcode 0xc0 > ) at /usr/src/sys/kern/kern_clocksource.c:390 > #11 0xc053a345 in hpet_intr_single (arg=<value optimized out>) > at /usr/src/sys/dev/acpica/acpi_hpet.c:260 > #12 0xc053a3cf in hpet_intr (arg=<value optimized out>) > at /usr/src/sys/dev/acpica/acpi_hpet.c:278 > #13 0xc0a99c5c in intr_event_handle (ie=Unhandled dwarf expression opcode 0xa1 > ) > at /usr/src/sys/kern/kern_intr.c:1435 > #14 0xc0e4c552 in intr_execute_handlers (isrc=<value optimized out>, > frame=<value optimized out>) at /usr/src/sys/x86/x86/intr_machdep.c:269 > #15 0xc0e4f50d in lapic_handle_intr (vector=-956585056, frame=0xc6fbaba0) > at /usr/src/sys/x86/x86/local_apic.c:780 > #16 0xc0e1ec35 in Xapic_isr1 () at /tmp/exception-SmXQMs.s:214 > #17 0xc0e1a202 in acpi_cpu_c1 () > at /usr/src/sys/i386/acpica/acpi_machdep.c:114 > Previous frame inner to this frame (corrupt stack?) > > Any suggestions? At this point, I don't really know enough to file a PR. > > Peace, > david >-- Andriy Gapon