Hi,
had recently this panic during some I/O stress tests:
> $<msgbuf
[...]
panic[cpu1]/thread=fffffe80005c3c80:
BAD TRAP: type=e (#pf Page fault) rp=fffffe80005c3980 addr=30 occurred in
module "zfs" due to a NULL pointer dereference
sched:
#pf Page fault
Bad kernel fault at addr=0x30
pid=0, pc=0xfffffffff3ee322e, sp=0xfffffe80005c3a70, eflags=0x10206
cr0: 8005003b<pg,wp,ne,et,ts,mp,pe> cr4:
6f0<xmme,fxsr,pge,mce,pae,pse>
cr2: 30 cr3: a49a000 cr8: c
rdi: fffffe80f0aa2b40 rsi: ffffffff89c3a050 rdx: 6352
rcx: 2f r8: 0 r9: 30
rax: 64f2 rbx: 20000 rbp: fffffe80005c3aa0
r10: fffffe80f0c979 r11: bd7189449a7087 r12: ffffffff89c3a040
r13: ffffffff89c3a040 r14: 32790 r15: 0
fsb: ffffffff80000000 gsb: ffffffff8149d800 ds: 43
es: 43 fs: 0 gs: 1c3
trp: e err: 0 rip: fffffffff3ee322e
cs: 28 rfl: 10206 rsp: fffffe80005c3a70
ss: 30
fffffe80005c3870 unix:die+eb ()
fffffe80005c3970 unix:trap+14f9 ()
fffffe80005c3980 unix:cmntrap+140 ()
fffffe80005c3aa0 zfs:buf_hash_remove+54 ()
fffffe80005c3b00 zfs:arc_change_state+1bd ()
fffffe80005c3b70 zfs:arc_evict_ghost+d1 ()
fffffe80005c3b90 zfs:arc_adjust+10f ()
fffffe80005c3bb0 zfs:arc_kmem_reclaim+d0 ()
fffffe80005c3bf0 zfs:arc_kmem_reap_now+30 ()
fffffe80005c3c60 zfs:arc_reclaim_thread+108 ()
fffffe80005c3c70 unix:thread_start+8 ()
syncing file systems...
done
dumping to /dev/md/dsk/swap, offset 644874240, content: kernel
> $c
buf_hash_remove+0x54(ffffffff89c3a040)
arc_change_state+0x1bd(ffffffffc0099370, ffffffff89c3a040, ffffffffc0098f30)
arc_evict_ghost+0xd1(ffffffffc0099470, 14b5c0c4)
arc_adjust+0x10f()
arc_kmem_reclaim+0xd0()
arc_kmem_reap_now+0x30(0)
arc_reclaim_thread+0x108()
thread_start+8()
> ::status
debugging crash dump vmcore.0 (64-bit) from server
operating system: 5.11 snv_39 (i86pc)
panic message:
BAD TRAP: type=e (#pf Page fault) rp=fffffe80005c3980 addr=30 occurred in
module "zfs" due to a NULL pointer dereference
dump content: kernel pages only
Daniel
Out of curiosity, is this panic reproducible? A bug should be filed on this for more investigation. Feel free to open one or I''ll open it if you forward me info on where the crash dump is and information on the I/O stress test you were running. thanks, Noel :-) ************************************************************************ ** "Question all the answers" On Jun 12, 2006, at 3:45 PM, Daniel Rock wrote:> Hi, > > had recently this panic during some I/O stress tests: > > > $<msgbuf > [...] > panic[cpu1]/thread=fffffe80005c3c80: > BAD TRAP: type=e (#pf Page fault) rp=fffffe80005c3980 addr=30 occurred > in module "zfs" due to a NULL pointer dereference > > > sched: > #pf Page fault > Bad kernel fault at addr=0x30 > pid=0, pc=0xfffffffff3ee322e, sp=0xfffffe80005c3a70, eflags=0x10206 > cr0: 8005003b<pg,wp,ne,et,ts,mp,pe> cr4: 6f0<xmme,fxsr,pge,mce,pae,pse> > cr2: 30 cr3: a49a000 cr8: c > rdi: fffffe80f0aa2b40 rsi: ffffffff89c3a050 rdx: > 6352 > rcx: 2f r8: 0 r9: > 30 > rax: 64f2 rbx: 20000 rbp: > fffffe80005c3aa0 > r10: fffffe80f0c979 r11: bd7189449a7087 r12: > ffffffff89c3a040 > r13: ffffffff89c3a040 r14: 32790 r15: > 0 > fsb: ffffffff80000000 gsb: ffffffff8149d800 ds: > 43 > es: 43 fs: 0 gs: > 1c3 > trp: e err: 0 rip: > fffffffff3ee322e > cs: 28 rfl: 10206 rsp: > fffffe80005c3a70 > ss: 30 > > fffffe80005c3870 unix:die+eb () > fffffe80005c3970 unix:trap+14f9 () > fffffe80005c3980 unix:cmntrap+140 () > fffffe80005c3aa0 zfs:buf_hash_remove+54 () > fffffe80005c3b00 zfs:arc_change_state+1bd () > fffffe80005c3b70 zfs:arc_evict_ghost+d1 () > fffffe80005c3b90 zfs:arc_adjust+10f () > fffffe80005c3bb0 zfs:arc_kmem_reclaim+d0 () > fffffe80005c3bf0 zfs:arc_kmem_reap_now+30 () > fffffe80005c3c60 zfs:arc_reclaim_thread+108 () > fffffe80005c3c70 unix:thread_start+8 () > > syncing file systems... > done > dumping to /dev/md/dsk/swap, offset 644874240, content: kernel > > $c > buf_hash_remove+0x54(ffffffff89c3a040) > arc_change_state+0x1bd(ffffffffc0099370, ffffffff89c3a040, > ffffffffc0098f30) > arc_evict_ghost+0xd1(ffffffffc0099470, 14b5c0c4) > arc_adjust+0x10f() > arc_kmem_reclaim+0xd0() > arc_kmem_reap_now+0x30(0) > arc_reclaim_thread+0x108() > thread_start+8() > > ::status > debugging crash dump vmcore.0 (64-bit) from server > operating system: 5.11 snv_39 (i86pc) > panic message: > BAD TRAP: type=e (#pf Page fault) rp=fffffe80005c3980 addr=30 occurred > in module "zfs" due to a NULL pointer dereference > dump content: kernel pages only > > > > Daniel > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Noel Dellofano schrieb:> Out of curiosity, is this panic reproducible?Hmm, not directly. The panic happened during a long running I/O stress test in the middle of the night. The tests had already run for ~6 hours at that time. > A bug should be filed on> this for more investigation. Feel free to open one or I''ll open it if > you forward me info on where the crash dump is and information on the > I/O stress test you were running.The core dump is very large. Even compressed with bzip2 it is still ~300MB in size. I will upload it to my external server this night and post details where the crash dump can be found. The tests I ran were Oracle database tests with many concurrent connections to the database. During the time of crash system load and I/O was just average though. Daniel
I have filed 6439484 panic in buf_hash_remove due to a NULL pointer dereference to track this issue. thanks, Noel :-) ************************************************************************ ** "Question all the answers" On Jun 12, 2006, at 3:45 PM, Daniel Rock wrote:> Hi, > > had recently this panic during some I/O stress tests: > > > $<msgbuf > [...] > panic[cpu1]/thread=fffffe80005c3c80: > BAD TRAP: type=e (#pf Page fault) rp=fffffe80005c3980 addr=30 occurred > in module "zfs" due to a NULL pointer dereference > > > sched: > #pf Page fault > Bad kernel fault at addr=0x30 > pid=0, pc=0xfffffffff3ee322e, sp=0xfffffe80005c3a70, eflags=0x10206 > cr0: 8005003b<pg,wp,ne,et,ts,mp,pe> cr4: 6f0<xmme,fxsr,pge,mce,pae,pse> > cr2: 30 cr3: a49a000 cr8: c > rdi: fffffe80f0aa2b40 rsi: ffffffff89c3a050 rdx: > 6352 > rcx: 2f r8: 0 r9: > 30 > rax: 64f2 rbx: 20000 rbp: > fffffe80005c3aa0 > r10: fffffe80f0c979 r11: bd7189449a7087 r12: > ffffffff89c3a040 > r13: ffffffff89c3a040 r14: 32790 r15: > 0 > fsb: ffffffff80000000 gsb: ffffffff8149d800 ds: > 43 > es: 43 fs: 0 gs: > 1c3 > trp: e err: 0 rip: > fffffffff3ee322e > cs: 28 rfl: 10206 rsp: > fffffe80005c3a70 > ss: 30 > > fffffe80005c3870 unix:die+eb () > fffffe80005c3970 unix:trap+14f9 () > fffffe80005c3980 unix:cmntrap+140 () > fffffe80005c3aa0 zfs:buf_hash_remove+54 () > fffffe80005c3b00 zfs:arc_change_state+1bd () > fffffe80005c3b70 zfs:arc_evict_ghost+d1 () > fffffe80005c3b90 zfs:arc_adjust+10f () > fffffe80005c3bb0 zfs:arc_kmem_reclaim+d0 () > fffffe80005c3bf0 zfs:arc_kmem_reap_now+30 () > fffffe80005c3c60 zfs:arc_reclaim_thread+108 () > fffffe80005c3c70 unix:thread_start+8 () > > syncing file systems... > done > dumping to /dev/md/dsk/swap, offset 644874240, content: kernel > > $c > buf_hash_remove+0x54(ffffffff89c3a040) > arc_change_state+0x1bd(ffffffffc0099370, ffffffff89c3a040, > ffffffffc0098f30) > arc_evict_ghost+0xd1(ffffffffc0099470, 14b5c0c4) > arc_adjust+0x10f() > arc_kmem_reclaim+0xd0() > arc_kmem_reap_now+0x30(0) > arc_reclaim_thread+0x108() > thread_start+8() > > ::status > debugging crash dump vmcore.0 (64-bit) from server > operating system: 5.11 snv_39 (i86pc) > panic message: > BAD TRAP: type=e (#pf Page fault) rp=fffffe80005c3980 addr=30 occurred > in module "zfs" due to a NULL pointer dereference > dump content: kernel pages only > > > > Daniel > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss