Hello List,
I am trying to debug a 6.1 panic. When I run kgdb kernel.debug
/var/crash/vmcore.7 all I get is:
kgdb: kvm_read: invalid address (0x24)
kgdb: kvm_read: invalid address (0x24)
kgdb: kvm_read: invalid address (0x24)
kgdb: kvm_read: invalid address (0x24)
kgdb: kvm_read: invalid address (0x24)
kgdb: kvm_read: invalid address (0x24)
kgdb: kvm_read: invalid address (0x24)
...
the info file shows:
Dump header from device /dev/ad0s1b
Architecture: i386
Architecture Version: 2
Dump Length: 116981760B (111 MB)
Blocksize: 512
Dumptime: Mon Feb 4 04:13:09 2008
Hostname: G301482.netws.com
Magic: FreeBSD Kernel Dump
Version String: FreeBSD 6.1-STABLE #25: Wed Nov 14 10:30:01 EST 2007
root@J301002.nwv01.com:/mnt/src/sys/i386/compile/WOLFPAC6SMP
Panic String: page fault
Dump Parity: 1156397610
Bounds: 7
Dump Status: good
Does my kernel.debug have to match exactly the crash file kernel. I have
made the following change
to my kernel that the kernel.debug is based on.
--- route.h.orig Tue Apr 4 22:07:23 2006
+++ route.h Mon Dec 17 13:11:44 2007
@@ -289,6 +289,7 @@
#define RT_LOCK_INIT(_rt) \
mtx_init(&(_rt)->rt_mtx, "rtentry", NULL, MTX_DEF |
MTX_DUPOK)
#define RT_LOCK(_rt) mtx_lock(&(_rt)->rt_mtx)
+#define RT_TRYLOCK(_rt) mtx_trylock(&(_rt)->rt_mtx)
#define RT_UNLOCK(_rt) mtx_unlock(&(_rt)->rt_mtx)
#define RT_LOCK_DESTROY(_rt) mtx_destroy(&(_rt)->rt_mtx)
#define RT_LOCK_ASSERT(_rt) mtx_assert(&(_rt)->rt_mtx,
MA_OWNED)
--- route.c.orig Tue Oct 30 19:07:54 2007
+++ route.c Mon Dec 17 15:13:20 2007
@@ -996,6 +996,7 @@
struct radix_node_head *rnh = rt_tables[dst->sa_family];
int dlen = SA_SIZE(dst), glen = SA_SIZE(gate);
+again:
RT_LOCK_ASSERT(rt);
/*
@@ -1029,7 +1030,15 @@
RT_REMREF(rt);
return (EADDRINUSE); /* failure */
}
- RT_LOCK(rt);
+ /*
+ * Try to reacquire the lock on rt, and if it fails,
+ * clean state and restart from scratch.
+ */
+ if (!RT_TRYLOCK(rt)) {
+ RTFREE_LOCKED(gwrt);
+ RT_LOCK(rt);
+ goto again;
+ }
/*
* If there is already a gwroute, then drop it. If we
* are asked to replace route with itself, then do
Thanks,
Steve
--
"They that give up essential liberty to obtain temporary safety,
deserve neither liberty nor safety." (Ben Franklin)
"The course of history shows that as a government grows, liberty
decreases." (Thomas Jefferson)
Stephen Clark wrote:> Hello List, > > I am trying to debug a 6.1 panic. When I run kgdb kernel.debug > /var/crash/vmcore.7 all I get is: > > kgdb: kvm_read: invalid address (0x24) > kgdb: kvm_read: invalid address (0x24) > kgdb: kvm_read: invalid address (0x24) > kgdb: kvm_read: invalid address (0x24) > kgdb: kvm_read: invalid address (0x24) > kgdb: kvm_read: invalid address (0x24) > kgdb: kvm_read: invalid address (0x24) > ... > > the info file shows: > Dump header from device /dev/ad0s1b > Architecture: i386 > Architecture Version: 2 > Dump Length: 116981760B (111 MB) > Blocksize: 512 > Dumptime: Mon Feb 4 04:13:09 2008 > Hostname: G301482.netws.com > Magic: FreeBSD Kernel Dump > Version String: FreeBSD 6.1-STABLE #25: Wed Nov 14 10:30:01 EST 2007 > root@J301002.nwv01.com:/mnt/src/sys/i386/compile/WOLFPAC6SMP > Panic String: page fault > Dump Parity: 1156397610 > Bounds: 7 > Dump Status: good > > > Does my kernel.debug have to match exactly the crash file kernel. I > have made the following change > to my kernel that the kernel.debug is based on. > --- route.h.orig Tue Apr 4 22:07:23 2006 > +++ route.h Mon Dec 17 13:11:44 2007 > @@ -289,6 +289,7 @@ > #define RT_LOCK_INIT(_rt) \ > mtx_init(&(_rt)->rt_mtx, "rtentry", NULL, MTX_DEF | MTX_DUPOK) > #define RT_LOCK(_rt) mtx_lock(&(_rt)->rt_mtx) > +#define RT_TRYLOCK(_rt) mtx_trylock(&(_rt)->rt_mtx) > #define RT_UNLOCK(_rt) mtx_unlock(&(_rt)->rt_mtx) > #define RT_LOCK_DESTROY(_rt) mtx_destroy(&(_rt)->rt_mtx) > #define RT_LOCK_ASSERT(_rt) mtx_assert(&(_rt)->rt_mtx, > MA_OWNED) > --- route.c.orig Tue Oct 30 19:07:54 2007 > +++ route.c Mon Dec 17 15:13:20 2007 > @@ -996,6 +996,7 @@ > struct radix_node_head *rnh = rt_tables[dst->sa_family]; > int dlen = SA_SIZE(dst), glen = SA_SIZE(gate); > > +again: > RT_LOCK_ASSERT(rt); > > /* > @@ -1029,7 +1030,15 @@ > RT_REMREF(rt); > return (EADDRINUSE); /* failure */ > } > - RT_LOCK(rt); > + /* > + * Try to reacquire the lock on rt, and if it fails, > + * clean state and restart from scratch. > + */ > + if (!RT_TRYLOCK(rt)) { > + RTFREE_LOCKED(gwrt); > + RT_LOCK(rt); > + goto again; > + } > /* > * If there is already a gwroute, then drop it. If we > * are asked to replace route with itself, then do > > Thanks, > Steve >Well I recompiled the kernel without the above changes and I am now kgdb comes up. It looks like the panic instruction pointer is in a loadable kernel module. Is there some way to have kgdb look at the kernel module? Below is what i get now: GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x4 fault code = supervisor read, page not present instruction pointer = 0x20:0xc0b4536c stack pointer = 0x28:0xc7516a30 frame pointer = 0x28:0xc7516a48 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 13 (swi1: net) trap number = 12 panic: page fault cpuid = 0 Uptime: 8h31m18s Dumping 111 MB (2 chunks) chunk 0: 1MB (159 pages) ... ok chunk 1: 111MB (28400 pages) 95 79 63 47 31 15 #0 doadump () at pcpu.h:165 165 __asm __volatile("movl %%fs:0,%0" : "=r" (td)); (kgdb) bt #0 doadump () at pcpu.h:165 #1 0xc06492b2 in boot (howto=260) at ../../../kern/kern_shutdown.c:409 #2 0xc06495d9 in panic (fmt=0xc0910386 "%s") at ../../../kern/kern_shutdown.c:565 #3 0xc082d99c in trap_fatal (frame=0xc75169f0, eva=4) at ../../../i386/i386/trap.c:837 #4 0xc082d6db in trap_pfault (frame=0xc75169f0, usermode=0, eva=4) at ../../../i386/i386/trap.c:745 #5 0xc082d335 in trap (frame {tf_fs = 8, tf_es = 40, tf_ds = 40, tf_edi = -950965440, tf_esi = -1026657792, tf_ebp = -950965688, tf_isp = -950965732, tf_ebx = -1045086208, tf_edx = -1047438316, tf_ecx = 0, tf_eax = 0, tf_trapno = 12, tf_err = 0, tf_eip = -1061923988, tf_cs = 32, tf_eflags = 590406, tf_esp = -929974260, tf_ss = 0}) at ../../../i386/i386/trap.c:435 #6 0xc08198fa in calltrap () at ../../../i386/i386/exception.s:139 #7 0xc0b4536c in ?? () Cannot access memory at address 0xc891b80c (kgdb) list *0xc0b4536c No source file for address 0xc0b4536c. (kgdb) looking at the loadable kernel modules on the system I get: sudo kldstat Id Refs Address Size Name 1 13 0xc0400000 72862c kernel 2 1 0xc0b29000 2340 accf_http.ko 3 1 0xc0b2c000 3b180 ipf.ko 4 1 0xc0b68000 5c2f8 acpi.ko 5 1 0xc1dcc000 3000 ng_iface.ko 6 1 0xc1dcf000 6000 ng_ppp.ko 7 1 0xc1dd6000 4000 ng_bpf.ko 8 1 0xc1ddd000 4000 ng_vjc.ko Which makes think the panic instruction pointer is in ipf.ko? This is ipf 4.1.26 compiled out of the kernel tree. Steve -- "They that give up essential liberty to obtain temporary safety, deserve neither liberty nor safety." (Ben Franklin) "The course of history shows that as a government grows, liberty decreases." (Thomas Jefferson)
On Monday 04 February 2008 11:38:06 am Stephen Clark wrote:> Stephen Clark wrote: > > Hello List, > > > > I am trying to debug a 6.1 panic. When I run kgdb kernel.debug > > /var/crash/vmcore.7 all I get is: > > > > kgdb: kvm_read: invalid address (0x24) > > kgdb: kvm_read: invalid address (0x24) > > kgdb: kvm_read: invalid address (0x24) > > kgdb: kvm_read: invalid address (0x24) > > kgdb: kvm_read: invalid address (0x24) > > kgdb: kvm_read: invalid address (0x24) > > kgdb: kvm_read: invalid address (0x24) > > ... > > > > the info file shows: > > Dump header from device /dev/ad0s1b > > Architecture: i386 > > Architecture Version: 2 > > Dump Length: 116981760B (111 MB) > > Blocksize: 512 > > Dumptime: Mon Feb 4 04:13:09 2008 > > Hostname: G301482.netws.com > > Magic: FreeBSD Kernel Dump > > Version String: FreeBSD 6.1-STABLE #25: Wed Nov 14 10:30:01 EST 2007 > > root@J301002.nwv01.com:/mnt/src/sys/i386/compile/WOLFPAC6SMP > > Panic String: page fault > > Dump Parity: 1156397610 > > Bounds: 7 > > Dump Status: good > > > > > > Does my kernel.debug have to match exactly the crash file kernel. I > > have made the following change > > to my kernel that the kernel.debug is based on. > > --- route.h.orig Tue Apr 4 22:07:23 2006 > > +++ route.h Mon Dec 17 13:11:44 2007 > > @@ -289,6 +289,7 @@ > > #define RT_LOCK_INIT(_rt) \ > > mtx_init(&(_rt)->rt_mtx, "rtentry", NULL, MTX_DEF | MTX_DUPOK) > > #define RT_LOCK(_rt) mtx_lock(&(_rt)->rt_mtx) > > +#define RT_TRYLOCK(_rt) mtx_trylock(&(_rt)->rt_mtx) > > #define RT_UNLOCK(_rt) mtx_unlock(&(_rt)->rt_mtx) > > #define RT_LOCK_DESTROY(_rt) mtx_destroy(&(_rt)->rt_mtx) > > #define RT_LOCK_ASSERT(_rt) mtx_assert(&(_rt)->rt_mtx, > > MA_OWNED) > > --- route.c.orig Tue Oct 30 19:07:54 2007 > > +++ route.c Mon Dec 17 15:13:20 2007 > > @@ -996,6 +996,7 @@ > > struct radix_node_head *rnh = rt_tables[dst->sa_family]; > > int dlen = SA_SIZE(dst), glen = SA_SIZE(gate); > > > > +again: > > RT_LOCK_ASSERT(rt); > > > > /* > > @@ -1029,7 +1030,15 @@ > > RT_REMREF(rt); > > return (EADDRINUSE); /* failure */ > > } > > - RT_LOCK(rt); > > + /* > > + * Try to reacquire the lock on rt, and if it fails, > > + * clean state and restart from scratch. > > + */ > > + if (!RT_TRYLOCK(rt)) { > > + RTFREE_LOCKED(gwrt); > > + RT_LOCK(rt); > > + goto again; > > + } > > /* > > * If there is already a gwroute, then drop it. If we > > * are asked to replace route with itself, then do > > > > Thanks, > > Steve > > > Well I recompiled the kernel without the above changes and I am now kgdb > comes up. It looks like > the panic instruction pointer is in a loadable kernel module. Is there > some way to have kgdb look at the kernel > module? > Below is what i get now: > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain > conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "i386-marcel-freebsd". > > Unread portion of the kernel message buffer: > > > Fatal trap 12: page fault while in kernel mode > cpuid = 0; apic id = 00 > fault virtual address = 0x4 > fault code = supervisor read, page not present > instruction pointer = 0x20:0xc0b4536c > stack pointer = 0x28:0xc7516a30 > frame pointer = 0x28:0xc7516a48 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, def32 1, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 13 (swi1: net) > trap number = 12 > panic: page fault > cpuid = 0 > Uptime: 8h31m18s > Dumping 111 MB (2 chunks) > chunk 0: 1MB (159 pages) ... ok > chunk 1: 111MB (28400 pages) 95 79 63 47 31 15 > > #0 doadump () at pcpu.h:165 > 165 __asm __volatile("movl %%fs:0,%0" : "=r" (td)); > (kgdb) bt > #0 doadump () at pcpu.h:165 > #1 0xc06492b2 in boot (howto=260) at ../../../kern/kern_shutdown.c:409 > #2 0xc06495d9 in panic (fmt=0xc0910386 "%s") at > ../../../kern/kern_shutdown.c:565 > #3 0xc082d99c in trap_fatal (frame=0xc75169f0, eva=4) at > ../../../i386/i386/trap.c:837 > #4 0xc082d6db in trap_pfault (frame=0xc75169f0, usermode=0, eva=4) > at ../../../i386/i386/trap.c:745 > #5 0xc082d335 in trap (frame> {tf_fs = 8, tf_es = 40, tf_ds = 40, tf_edi = -950965440, tf_esi = > -1026657792, tf_ebp = -950965688, tf_isp = -950965732, tf_ebx = > -1045086208, tf_edx = -1047438316, tf_ecx = 0, tf_eax = 0, tf_trapno = > 12, tf_err = 0, tf_eip = -1061923988, tf_cs = 32, tf_eflags = 590406, > tf_esp = -929974260, tf_ss = 0}) at ../../../i386/i386/trap.c:435 > #6 0xc08198fa in calltrap () at ../../../i386/i386/exception.s:139 > #7 0xc0b4536c in ?? () > Cannot access memory at address 0xc891b80c > (kgdb) list *0xc0b4536c > No source file for address 0xc0b4536c. > (kgdb) > > looking at the loadable kernel modules on the system I get: > sudo kldstat > Id Refs Address Size Name > 1 13 0xc0400000 72862c kernel > 2 1 0xc0b29000 2340 accf_http.ko > 3 1 0xc0b2c000 3b180 ipf.ko > 4 1 0xc0b68000 5c2f8 acpi.ko > 5 1 0xc1dcc000 3000 ng_iface.ko > 6 1 0xc1dcf000 6000 ng_ppp.ko > 7 1 0xc1dd6000 4000 ng_bpf.ko > 8 1 0xc1ddd000 4000 ng_vjc.ko > > Which makes think the panic instruction pointer is in ipf.ko? > This is ipf 4.1.26 compiled out of the kernel tree. > > SteveYou can use asf(8) with the -c option to build a .asf file you can source into kgdb to load symbols for the kernel modules to give you a decent backtrace. -- John Baldwin