Hi.
After some time using zfs on a 10.x server (and a couple of panics) I'm
now getting the reproducible and repeatable panic on any operations
involving particular pool. I managed to boot from USB stick, import only
healthy pools and substitite the server's zpool.cache file with one
referencing only healthy pools. Now I'm able to boot, but when I try to
import the remaining pool I'm getting the panic (attached below). Now
questions:
- do I understand correctly that "#7 0xffffffff8188e076 in
vdev_readable (vd=0x0)" means vd is NULL, and this is triggering the panic
?
- I saw a similar (but not identical) panic in
http://lists.freebsd.org/pipermail/freebsd-fs/2012-January/013513.html ,
- are there any possible tricks that could help me getting my data ?
the target pool itself shows as healthy too:
# zpool import
pool: xtank
id: 16620000996171732653
state: ONLINE
status: Some supported features are not enabled on the pool.
action: The pool can be imported using its name or numeric identifier,
though
some features will not be available without an explicit 'zpool
upgrade'.
config:
xtank ONLINE
mirror-0 ONLINE
gpt/xtank0 ONLINE
gpt/xtank1 ONLINE
logs
mirror-1 ONLINE
gpt/xlog0 ONLINE
gpt/xlog1 ONLINE
Backtrace and stuff:
===Cut==# less core.txt.4
witchdoctor.hq.norma.perm.ru dumped core - see /var/crash/vmcore.4
Fri Mar 21 13:00:30 YEKT 2014
FreeBSD witchdoctor.hq.norma.perm.ru 10.0-STABLE FreeBSD 10.0-STABLE #0
r263266: Mon Mar 17 23:17:32 YEKT 2014
emz at witchdoctor.hq.norma.perm.ru:/usr/obj/usr/src/sys/GENERIC amd64
panic: page fault
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "amd64-marcel-freebsd"...
Unread portion of the kernel message buffer:
Fatal trap 12: page fault while in kernel mode
Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 03
fault virtual address = 0x50
cpuid = 1; apic id = 01
fault code = supervisor read data, page not present
fault virtual address = 0xa0
instruction pointer = 0x20:0xffffffff8188e076
fault code = supervisor read data, page not present
stack pointer = 0x28:0xfffffe01208e5b00
instruction pointer = 0x20:0xffffffff818ab666
frame pointer = 0x28:0xfffffe01208e5b10
stack pointer = 0x28:0xfffffe0120fc85b0
code segment = base 0x0, limit 0xfffff, type 0x1b
frame pointer = 0x28:0xfffffe0120fc8640
= DPL 0, pres 1, long 1, def32 0, gran 1
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, processor eflags resume, IOPL = 0
interrupt enabled, resume, IOPL = 0
current process = 0 (system_taskq_2)
trap number = 12
current process = 1363 (zpool)
panic: page fault
cpuid = 3
KDB: stack backtrace:
#0 0xffffffff808f01d0 at kdb_backtrace+0x60
#1 0xffffffff808b7ba5 at panic+0x155
#2 0xffffffff80c98f32 at trap_fatal+0x3a2
#3 0xffffffff80c99209 at trap_pfault+0x2c9
#4 0xffffffff80c9899b at trap+0x5bb
#5 0xffffffff80c7fc52 at calltrap+0x8
#6 0xffffffff81894740 at vdev_mirror_child_select+0x70
#7 0xffffffff81894284 at vdev_mirror_io_start+0x234
#8 0xffffffff818ae754 at zio_vdev_io_start+0x184
#9 0xffffffff818aba8a at zio_execute+0x15a
#10 0xffffffff8183a158 at arc_read+0x958
#11 0xffffffff81852fde at traverse_prefetcher+0x13e
#12 0xffffffff8185243d at traverse_visitbp+0x20d
#13 0xffffffff81852e6f at traverse_dnode+0xef
#14 0xffffffff81852bb7 at traverse_visitbp+0x987
#15 0xffffffff81852613 at traverse_visitbp+0x3e3
#16 0xffffffff81852613 at traverse_visitbp+0x3e3
#17 0xffffffff81852613 at traverse_visitbp+0x3e3
Uptime: 6m45s
Dumping 326 out of 4043 MB:..5%..15%..25%..35%..45%..54%..64%..74%..84%..94%
Reading symbols from /boot/kernel/zfs.ko.symbols...done.
Loaded symbols for /boot/kernel/zfs.ko.symbols
Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
Loaded symbols for /boot/kernel/opensolaris.ko.symbols
Reading symbols from /boot/kernel/geom_mirror.ko.symbols...done.
Loaded symbols for /boot/kernel/geom_mirror.ko.symbols
Reading symbols from /boot/kernel/ums.ko.symbols...done.
Loaded symbols for /boot/kernel/ums.ko.symbols
Reading symbols from /boot/kernel/uhid.ko.symbols...done.
Loaded symbols for /boot/kernel/uhid.ko.symbols
#0 doadump (textdump=<value optimized out>) at pcpu.h:219
219 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) #0 doadump (textdump=<value optimized out>) at pcpu.h:219
#1 0xffffffff808b7820 in kern_reboot (howto=260)
at /usr/src/sys/kern/kern_shutdown.c:452
#2 0xffffffff808b7be4 in panic (fmt=<value optimized out>)
at /usr/src/sys/kern/kern_shutdown.c:759
#3 0xffffffff80c98f32 in trap_fatal (frame=<value optimized out>,
eva=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:875
#4 0xffffffff80c99209 in trap_pfault (frame=0xfffffe01208e5a50, usermode=0)
at /usr/src/sys/amd64/amd64/trap.c:692
#5 0xffffffff80c9899b in trap (frame=0xfffffe01208e5a50)
at /usr/src/sys/amd64/amd64/trap.c:456
#6 0xffffffff80c7fc52 in calltrap ()
at /usr/src/sys/amd64/amd64/exception.S:232
#7 0xffffffff8188e076 in vdev_readable (vd=0x0)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c:2632
#8 0xffffffff81894740 in vdev_mirror_child_select (zio=<value optimized
out>)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_mirror.c:238
#9 0xffffffff81894284 in vdev_mirror_io_start (zio=0xfffff8004f228398)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_mirror.c:295
#10 0xffffffff818ae754 in zio_vdev_io_start (zio=0xfffff8004f228398)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2514
#11 0xffffffff818aba8a in zio_execute (zio=0xfffff8004f228398)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1346
#12 0xffffffff8183a158 in arc_read (pio=0x0, spa=0xfffff800799b8000,
bp=<value optimized out>, done=0x2, private=0x0,
priority=ZIO_PRIORITY_ASYNC_READ, zio_flags=512,
arc_flags=<value optimized out>, zb=0x0)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:3417
#13 0xffffffff81852fde in traverse_prefetcher (spa=0xfffff800799b8000,
zilog=0xf01ff, bp=<value optimized out>, zb=<value optimized
out>,
dnp=0xfffff8004f1d3240, arg=<value optimized out>)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:451
#14 0xffffffff8185243d in traverse_visitbp (td=0xfffffe01208e6980,
dnp=0xfffffe00276fe800, bp=0xfffffe00276fe980, zb=0xfffffe01208e5f08)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:250
#15 0xffffffff81852e6f in traverse_dnode (td=0xfffffe01208e6980,
dnp=0xfffffe00276fe800, objset=41, object=57388)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:417
#16 0xffffffff81852bb7 in traverse_visitbp (td=0xfffffe01208e6980,
dnp=0xfffffe00276fd000, bp=0xfffffe00276f1080, zb=0xfffffe01208e6128)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:309
#17 0xffffffff81852613 in traverse_visitbp (td=0xfffffe01208e6980,
dnp=0xfffff80079bb9800, bp=0xfffffe0027709700, zb=0xfffffe01208e6258)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:284
#18 0xffffffff81852613 in traverse_visitbp (td=0xfffffe01208e6980,
dnp=0xfffff80079bb9800, bp=0xfffffe00275dd000, zb=0xfffffe01208e6388)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:284
#19 0xffffffff81852613 in traverse_visitbp (td=0xfffffe01208e6980,
dnp=0xfffff80079bb9800, bp=0xfffffe0027651000, zb=0xfffffe01208e64b8)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:284
#20 0xffffffff81852613 in traverse_visitbp (td=0xfffffe01208e6980,
dnp=0xfffff80079bb9800, bp=0xfffffe0027584000, zb=0xfffffe01208e65e8)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:284
#21 0xffffffff81852613 in traverse_visitbp (td=0xfffffe01208e6980,
dnp=0xfffff80079bb9800, bp=0xfffffe002755c000, zb=0xfffffe01208e6718)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:284
#22 0xffffffff81852613 in traverse_visitbp (td=0xfffffe01208e6980,
dnp=0xfffff80079bb9800, bp=0xfffff80079bb9840, zb=0xfffffe01208e67d8)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:284
#23 0xffffffff81852e04 in traverse_dnode (td=0xfffffe01208e6980,
dnp=0xfffff80079bb9800, objset=41, object=0)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:407
#24 0xffffffff818528c0 in traverse_visitbp (td=0xfffffe01208e6980, dnp=0x0,
bp=0xfffff800794c6280, zb=0xfffffe01208e6960)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:338
#25 0xffffffff818521d6 in traverse_prefetch_thread (arg=0xfffffe0120fc9420)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:470
#26 0xffffffff81828c00 in taskq_run (arg=0xfffff80018b01ea0, pending=983551)
at
/usr/src/sys/modules/zfs/../../cddl/compat/opensolaris/kern/opensolaris_taskq.c:109
#27 0xffffffff808fe1b6 in taskqueue_run_locked (queue=0xfffff8001614c300)
at /usr/src/sys/kern/subr_taskqueue.c:342
#28 0xffffffff808fec28 in taskqueue_thread_loop (arg=<value optimized
out>)
at /usr/src/sys/kern/subr_taskqueue.c:563
#29 0xffffffff80889cba in fork_exit (
callout=0xffffffff808feb80 <taskqueue_thread_loop>,
arg=0xfffff8001616f720, frame=0xfffffe01208e6ac0)
at /usr/src/sys/kern/kern_fork.c:995
#30 0xffffffff80c8018e in fork_trampoline ()
at /usr/src/sys/amd64/amd64/exception.S:606
#31 0x0000000000000000 in ?? ()
Current language: auto; currently minimal
(kgdb)
===Cut==
Thanks.
Eugene.