I'm getting panics like this every so often (couple weeks, sometimes just a
few days.) A second machine that has identical hardware and is running the same
source has no such problems.
FreeBSD XXX.hanse.de 8.0-STABLE FreeBSD 8.0-STABLE #16: Tue Dec 1 14:30:54 UTC
2009 root@XXX.hanse.de:/usr/obj/usr/src/sys/EISENBOOT amd64
# zpool status
pool: tank
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
ad4s1d ONLINE 0 0 0
# cat /boot/loader.conf
vfs.zfs.arc_max="512M"
vfs.zfs.prefetch_disable="1"
vfs.zfs.zil_disable="1"
Fatal trap 9: general protection fault while in kernel mode
cpuid = 0; apic id = 00
instruction pointer = 0x20:0xffffffff80a39900
stack pointer = 0x28:0xffffff80622ddae0
frame pointer = 0x28:0xffffff80622ddb10
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 0 (spa_zio)
trap number = 9
panic: general protection fault
cpuid = 0
Uptime: 17h44m5s
Physical memory: 3313 MB
Dumping 1843 MB: 1828 1812 1796 1780 1764 1748 1732 1716 1700 1684 1668 1652
1636 1620 1604 1588 1572 1556 1540 1524 1508 1492 1476 1460 1444 1428 1412 1396
1380 1364 1348 1332 1316 1300 1284 1268 1252 1236 1220 1204 1188 1172 1156 1140
1124 1108 1092 1076 1060 1044 1028 1012 996 980 964 948 932 916 900 884 868 852
836 820 804 788 772 756 740 724 708 692 676 660 644 628 612 596 580 564 548 532
516 500 484 468 452 436 420 404 388 372 356 340 324 308 292 276 260 244 228 212
196 180 164 148 132 116 100 84 68 52 36 20 4
#0 doadump () at pcpu.h:223
223 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) #0 doadump () at pcpu.h:223
#1 0xffffffff803374b9 in boot (howto=260)
at /usr/src/sys/kern/kern_shutdown.c:416
#2 0xffffffff8033790c in panic (fmt=Variable "fmt" is not available.
)
at /usr/src/sys/kern/kern_shutdown.c:579
#3 0xffffffff805cbb8d in trap_fatal (frame=0x9, eva=Variable "eva" is
not available.
)
at /usr/src/sys/amd64/amd64/trap.c:857
#4 0xffffffff805cc6f2 in trap (frame=0xffffff80622dda30)
at /usr/src/sys/amd64/amd64/trap.c:644
#5 0xffffffff805b2223 in calltrap ()
at /usr/src/sys/amd64/amd64/exception.S:224
#6 0xffffffff80a39900 in vdev_queue_agg_io_done (aio=0xffffff00374562d0)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:174
#7 0xffffffff80a4be6f in zio_done (zio=0xffffff00374562d0)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2243
#8 0xffffffff80a49e87 in zio_execute (zio=0xffffff00374562d0)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:996
#9 0xffffffff809ed603 in taskq_run (arg=0xffffff008d8d0420, pending=Variable
"pending" is not available.
)
at
/usr/src/sys/modules/zfs/../../cddl/compat/opensolaris/kern/opensolaris_taskq.c:108
#10 0xffffffff80373533 in taskqueue_run (queue=0xffffff00017e1400)
at /usr/src/sys/kern/subr_taskqueue.c:239
#11 0xffffffff803737b6 in taskqueue_thread_loop (arg=Variable "arg" is
not available.
)
at /usr/src/sys/kern/subr_taskqueue.c:360
#12 0xffffffff8030e0b8 in fork_exit (
callout=0xffffffff80373770 <taskqueue_thread_loop>,
arg=0xffffff00016434e0, frame=0xffffff80622ddc80)
at /usr/src/sys/kern/kern_fork.c:843
#13 0xffffffff805b26fe in fork_trampoline ()
at /usr/src/sys/amd64/amd64/exception.S:561
#14 0x0000000000000000 in ?? ()
#15 0x0000000000000000 in ?? ()
#16 0x0000000000000000 in ?? ()
#17 0x0000000000000000 in ?? ()
#18 0x0000000000000000 in ?? ()
#19 0x0000000000000000 in ?? ()
#20 0x0000000000000000 in ?? ()
#21 0x0000000000000000 in ?? ()
#22 0x0000000000000000 in ?? ()
#23 0x0000000000000000 in ?? ()
#24 0x0000000000000000 in ?? ()
#25 0x0000000000000000 in ?? ()
#26 0x0000000000000000 in ?? ()
#27 0x0000000000000000 in ?? ()
#28 0x0000000000000000 in ?? ()
#29 0x0000000000000000 in ?? ()
#30 0x0000000000000000 in ?? ()
#31 0x0000000000000000 in ?? ()
#32 0x0000000000000000 in ?? ()
#33 0x0000000000000000 in ?? ()
#34 0x0000000000000000 in ?? ()
#35 0x0000000000000000 in ?? ()
#36 0x0000000000000000 in ?? ()
#37 0x0000000000000000 in ?? ()
#38 0x0000000000c6c000 in ?? ()
#39 0x0000000000000000 in ?? ()
#40 0x000000000000000b in ?? ()
#41 0xffffffff80832500 in affinity ()
#42 0xffffff000173c390 in ?? ()
#43 0xffffff80622dd240 in ?? ()
#44 0xffffff80622dd1f8 in ?? ()
#45 0xffffff00015ecab0 in ?? ()
#46 0xffffffff8035aa48 in sched_switch (td=0xffffffff80373770,
newtd=0xffffff00016434e0, flags=Variable "flags" is not available.
) at /usr/src/sys/kern/sched_ule.c:1858
Previous frame inner to this frame (corrupt stack?)
(kgdb)
--
Stefan Bethke <stb@lassitu.de> Fon +49 151 14070811
Am 04.12.2009 um 17:52 schrieb Stefan Bethke:> I'm getting panics like this every so often (couple weeks, sometimes just a few days.) A second machine that has identical hardware and is running the same source has no such problems. > > FreeBSD XXX.hanse.de 8.0-STABLE FreeBSD 8.0-STABLE #16: Tue Dec 1 14:30:54 UTC 2009 root@XXX.hanse.de:/usr/obj/usr/src/sys/EISENBOOT amd64 > > # zpool status > pool: tank > state: ONLINE > scrub: none requested > config: > > NAME STATE READ WRITE CKSUM > tank ONLINE 0 0 0 > ad4s1d ONLINE 0 0 0 > # cat /boot/loader.conf > vfs.zfs.arc_max="512M" > vfs.zfs.prefetch_disable="1" > vfs.zfs.zil_disable="1"Got another, different one. Any tuning suggestions or similar? #0 doadump () at pcpu.h:223 223 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump () at pcpu.h:223 #1 0xffffffff80337bd9 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:416 #2 0xffffffff8033802c in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:579 #3 0xffffffff805cc2ad in trap_fatal (frame=0x9, eva=Variable "eva" is not available. ) at /usr/src/sys/amd64/amd64/trap.c:857 #4 0xffffffff805cce12 in trap (frame=0xffffff80625db030) at /usr/src/sys/amd64/amd64/trap.c:644 #5 0xffffffff805b2943 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:224 #6 0xffffffff80586c7a in vm_map_entry_splay (addr=Variable "addr" is not available. ) at /usr/src/sys/vm/vm_map.c:771 #7 0xffffffff80587f37 in vm_map_lookup_entry (map=0xffffff00010000e8, address=18446743523979624448, entry=0xffffff80625db170) at /usr/src/sys/vm/vm_map.c:1021 #8 0xffffffff80588aa3 in vm_map_delete (map=0xffffff00010000e8, start=18446743523979624448, end=18446743523979689984) at /usr/src/sys/vm/vm_map.c:2685 #9 0xffffffff80588e61 in vm_map_remove (map=0xffffff00010000e8, start=18446743523979624448, end=18446743523979689984) at /usr/src/sys/vm/vm_map.c:2774 #10 0xffffffff8057db85 in uma_large_free (slab=0xffffff005fcc7000) at /usr/src/sys/vm/uma_core.c:3021 #11 0xffffffff80325987 in free (addr=0xffffff80018b0000, mtp=0xffffffff80ac61e0) at /usr/src/sys/kern/kern_malloc.c:471 #12 0xffffffff80a36d03 in vdev_cache_evict (vc=0xffffff0001723ce0, ve=0xffffff003dd52200) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_cache.c:151 #13 0xffffffff80a372ad in vdev_cache_read (zio=0xffffff005f5ca2d0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_cache.c:182 #14 0xffffffff80a4a954 in zio_vdev_io_start (zio=0xffffff005f5ca2d0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1814 #15 0xffffffff80a4ae87 in zio_execute (zio=0xffffff005f5ca2d0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:996 #16 0xffffffff80a3a080 in vdev_mirror_io_start (zio=0xffffff005f811b40) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_mirror.c:303 #17 0xffffffff80a4ae87 in zio_execute (zio=0xffffff005f811b40) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:996 #18 0xffffffff809ff45a in arc_read_nolock (pio=0xffffff005f66d5a0, spa=0xffffff000150a000, bp=0xffffff800a91c440, done=0xffffffff80a02630 <dbuf_read_done>, private=Variable "private" is not available. ) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:2763 #19 0xffffffff809ff8ec in arc_read (pio=0xffffff005f66d5a0, spa=0xffffff000150a000, bp=0xffffff800a91c440, pbuf=0xffffff0042a3ca20, done=0xffffffff80a02630 <dbuf_read_done>, private=0xffffff005fbfc620, priority=0, zio_flags=1, arc_flags=0xffffff80625db5ec, zb=0xffffff80625db5c0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:2508 #20 0xffffffff80a02aba in dbuf_read (db=0xffffff005fbfc620, zio=0xffffff005f66d5a0, flags=2) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:521 #21 0xffffffff80a0602c in dmu_buf_hold (os=Variable "os" is not available. ) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c:106 #22 0xffffffff80a40db5 in zap_lockdir (os=0xffffff005f937610, obj=247890, tx=0x0, lti=RW_READER, fatreader=1, adding=0, zapp=0xffffff80625db888) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:388 #23 0xffffffff80a41724 in zap_cursor_retrieve (zc=0xffffff80625db880, za=0xffffff80625db8c0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:1004 #24 0xffffffff80a61b66 in zfs_freebsd_readdir (ap=Variable "ap" is not available. ) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:2157 #25 0xffffffff803cfde9 in kern_getdirentries (td=0xffffff0057bfe000, fd=Variable "fd" is not available. ) at vnode_if.h:758 #26 0xffffffff803d0093 in getdirentries (td=Variable "td" is not available. ) at /usr/src/sys/kern/vfs_syscalls.c:4051 #27 0xffffffff805cc906 in syscall (frame=0xffffff80625dbc80) at /usr/src/sys/amd64/amd64/trap.c:989 #28 0xffffffff805b2c21 in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:373 #29 0x0000000800724cdc in ?? () Previous frame inner to this frame (corrupt stack?) -- Stefan Bethke <stb@lassitu.de> Fon +49 151 14070811
Am 04.12.2009 um 17:52 schrieb Stefan Bethke:> I'm getting panics like this every so often (couple weeks, sometimes just a few days.) A second machine that has identical hardware and is running the same source has no such problems.Thanks to all who suggested bad hardware: it turned out to be a case of capacitor plague. With a new mainboard, everything appears to be working solidly again. Stefan -- Stefan Bethke <stb@lassitu.de> Fon +49 151 14070811