thr3ads.net - freebsd stable - Panic on BETA1 in the ZFS subsystem [Jul 2016]

If this information is useful, please help other people find it:
Share via:

Karl Denninger

2016-Jul-20 14:26 UTC

Panic on BETA1 in the ZFS subsystem

Came out of the ZFS system; this machine is running *unpatched* (none of
my changes are in the kernel); I have a good core dump if further trace
information will help.

Dump header from device: /dev/gpt/dump
  Architecture: amd64
  Architecture Version: 2
  Dump Length: 9582022656
  Blocksize: 512
  Dumptime: Wed Jul 20 09:10:31 2016
  Hostname: NewFS.denninger.net
  Magic: FreeBSD Kernel Dump
  Version String: FreeBSD 11.0-BETA1 #0 r302489: Sat Jul  9 10:15:24 CDT
2016
    karl at NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP
  Panic String: double fault
  Dump Parity: 445173880
  Bounds: 4
  Dump Status: good

NewFS.denninger.net dumped core - see /var/crash/vmcore.4

Wed Jul 20 09:18:15 CDT 2016

FreeBSD NewFS.denninger.net 11.0-BETA1 FreeBSD 11.0-BETA1 #0 r302489:
Sat Jul  9 10:15:24 CDT 2016    
karl at NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP  amd64

panic: double fault

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for
details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:

Fatal double fault
rip = 0xffffffff8220bc92
rsp = 0xfffffe0667f0ef30
rbp = 0xfffffe0667f0f3c0
cpuid = 14; apic id = 34
panic: double fault
cpuid = 14
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
0xfffffe0649db8e30
vpanic() at vpanic+0x182/frame 0xfffffe0649db8eb0
panic() at panic+0x43/frame 0xfffffe0649db8f10
dblfault_handler() at dblfault_handler+0xa2/frame 0xfffffe0649db8f30
Xdblfault() at Xdblfault+0xac/frame 0xfffffe0649db8f30
--- trap 0x17, rip = 0xffffffff8220bc92, rsp = 0xfffffe0667f0ef30, rbp
0xfffffe0667f0f3c0 ---
vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x22/frame
0xfffffe0667f0f3c0
vdev_queue_io_done() at vdev_queue_io_done+0x83/frame 0xfffffe0667f0f400
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f430
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f480
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0f4e0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f530
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0f570
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f5a0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f5f0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0f650
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f6a0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0f6e0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f710
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f760
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0f7c0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f810
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0f850
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f880
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f8d0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0f930
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f980
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0f9c0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f9f0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fa40
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0faa0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0faf0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0fb30
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0fb60
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fbb0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0fc10
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fc60
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0fca0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0fcd0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fd20
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0fd80
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fdd0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0fe10
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0fe40
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fe90
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0fef0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0ff40
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0ff80
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0ffb0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10000
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10060
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f100b0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f100f0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10120
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10170
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f101d0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10220
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10260
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10290
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f102e0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10340
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10390
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f103d0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10400
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10450
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f104b0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10500
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10540
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10570
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f105c0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10620
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10670
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f106b0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f106e0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10730
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10790
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f107e0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10820
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10850
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f108a0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10900
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10950
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10990
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f109c0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10a10
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10a70
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10ac0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10b00
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10b30
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10b80
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10be0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10c30
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10c70
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10ca0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10cf0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10d50
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10da0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10de0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10e10
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10e60
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10ec0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10f10
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10f50
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10f80
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10fd0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11030
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11080
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f110c0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f110f0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11140
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f111a0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f111f0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11230
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11260
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f112b0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11310
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11360
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f113a0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f113d0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11420
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11480
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f114d0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11510
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11540
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11590
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f115f0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11640
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11680
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f116b0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11700
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11760
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f117b0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f117f0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11820
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11870
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f118d0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11920
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11960
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11990
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f119e0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11a40
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11a90
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11ad0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11b00
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11b50
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11bb0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11c00
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11c40
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11c70
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11cc0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11d20
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11d70
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11db0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11de0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11e30
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11e90
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11ee0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11f20
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11f50
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11fa0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f12000
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12050
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f12090
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f120c0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12110
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f12170
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f121c0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f12200
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12230
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12280
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f122e0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12330
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f12370
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f123a0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f123f0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f12450
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f124a0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f124e0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12510
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12560
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f125c0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12610
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f12650
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12680
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f126d0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f12730
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12780
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f127c0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f127f0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12840
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f128a0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f128f0
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f12930
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12960
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f129b0
zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f12a10
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12a60
vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f12aa0
zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12ad0
zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12b20
taskqueue_run_locked() at taskqueue_run_locked+0x14a/frame
0xfffffe0667f12b80
taskqueue_thread_loop() at taskqueue_thread_loop+0xe8/frame
0xfffffe0667f12bb0
fork_exit() at fork_exit+0x85/frame 0xfffffe0667f12bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0667f12bf0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Uptime: 10d21h36m8s
Dumping 9138 out of 24531
MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

Reading symbols from /boot/kernel/zfs.ko...Reading symbols from
/usr/lib/debug//boot/kernel/zfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from
/usr/lib/debug//boot/kernel/opensolaris.ko.debug...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
Reading symbols from /boot/kernel/geom_eli.ko...Reading symbols from
/usr/lib/debug//boot/kernel/geom_eli.ko.debug...done.
done.
Loaded symbols for /boot/kernel/geom_eli.ko
Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from
/usr/lib/debug//boot/kernel/geom_mirror.ko.debug...done.
done.
Loaded symbols for /boot/kernel/geom_mirror.ko
Reading symbols from /boot/kernel/aesni.ko...Reading symbols from
/usr/lib/debug//boot/kernel/aesni.ko.debug...done.
done.
Loaded symbols for /boot/kernel/aesni.ko
Reading symbols from /boot/kernel/uhid.ko...Reading symbols from
/usr/lib/debug//boot/kernel/uhid.ko.debug...done.
done.
Loaded symbols for /boot/kernel/uhid.ko
Reading symbols from /boot/kernel/ums.ko...Reading symbols from
/usr/lib/debug//boot/kernel/ums.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ums.ko
Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from
/usr/lib/debug//boot/kernel/ipfw.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ipfw.ko
Reading symbols from /boot/kernel/ipfw_nat.ko...Reading symbols from
/usr/lib/debug//boot/kernel/ipfw_nat.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ipfw_nat.ko
Reading symbols from /boot/kernel/libalias.ko...Reading symbols from
/usr/lib/debug//boot/kernel/libalias.ko.debug...done.
done.
Loaded symbols for /boot/kernel/libalias.ko
Reading symbols from /boot/kernel/daemon_saver.ko...Reading symbols from
/usr/lib/debug//boot/kernel/daemon_saver.ko.debug...done.
done.
Loaded symbols for /boot/kernel/daemon_saver.ko
#0  doadump (textdump=1) at pcpu.h:221
221    pcpu.h: No such file or directory.
    in pcpu.h
(kgdb) #0  doadump (textdump=1) at pcpu.h:221
#1  0xffffffff80ae548e in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:366
#2  0xffffffff80ae5a5b in vpanic (fmt=<value optimized out>,
    ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759
#3  0xffffffff80ae5893 in panic (fmt=0x0)
    at /usr/src/sys/kern/kern_shutdown.c:690
#4  0xffffffff80fc9692 in dblfault_handler (frame=<value optimized out>)
    at /usr/src/sys/amd64/amd64/trap.c:868
#5  0xffffffff80fac15c in Xdblfault ()
    at /usr/src/sys/amd64/amd64/exception.S:292
#6  0xffffffff8220bc92 in vdev_queue_io_to_issue (vq=0xfffff804d86833e0)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:585
#7  0xffffffff8220c6e3 in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:881
#8  0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8018859c000)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#9  0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#10 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8018859c000)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#11 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#12 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#13 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff801df1f43b8)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#14 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#15 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff801df1f43b8)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#16 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#17 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#18 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8024583e770)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#19 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#20 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8024583e770)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#21 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#22 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#23 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff802e12df770)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#24 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#25 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff802e12df770)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#26 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#27 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#28 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8004f81f3b8)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#29 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#30 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8004f81f3b8)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#31 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#32 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#33 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8021bdf7b28)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#34 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#35 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8021bdf7b28)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#36 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#37 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#38 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80165851770)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#39 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#40 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80165851770)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#41 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#42 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#43 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8060de643b8)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#44 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#45 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8060de643b8)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#46 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#47 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#48 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff800793c0770)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#49 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#50 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff800793c0770)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#51 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#52 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#53 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8026917cb28)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#54 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#55 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8026917cb28)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#56 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#57 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#58 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8012408d3b8)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#59 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#60 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8012408d3b8)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#61 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#62 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#63 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff802bfc9ab28)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#64 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#65 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff802bfc9ab28)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#66 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#67 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#68 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff802dda5db28)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#69 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#70 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff802dda5db28)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#71 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#72 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#73 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff800ae13d000)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#74 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#75 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff800ae13d000)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#76 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#77 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#78 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff804a70efb28)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#79 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#80 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff804a70efb28)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#81 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#82 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#83 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80485f5f000)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#84 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#85 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80485f5f000)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#86 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#87 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#88 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff800666fc3b8)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#89 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#90 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff800666fc3b8)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#91 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#92 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#93 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8010da3ab28)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#94 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#95 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8010da3ab28)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#96 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#97 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#98 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80315f73770)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#99 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#100 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80315f73770)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#101 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#102 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#103 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff806011703b8)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#104 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#105 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff806011703b8)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#106 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#107 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#108 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff805964f1770)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#109 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#110 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff805964f1770)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#111 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#112 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#113 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8023902f770)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#114 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#115 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8023902f770)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#116 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#117 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#118 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff803bc7e2b28)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#119 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#120 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff803bc7e2b28)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#121 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#122 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#123 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff804dfbf43b8)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#124 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#125 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff804dfbf43b8)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#126 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#127 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#128 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8005be4a3b8)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#129 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#130 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8005be4a3b8)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#131 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#132 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#133 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8004ff79000)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#134 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#135 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8004ff79000)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#136 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#137 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#138 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff801aa2a2000)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#139 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#140 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff801aa2a2000)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#141 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#142 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#143 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff800592003b8)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#144 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#145 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff800592003b8)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#146 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#147 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#148 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80202d11770)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#149 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#150 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80202d11770)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#151 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#152 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#153 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80392d3c3b8)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#154 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#155 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80392d3c3b8)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#156 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#157 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#158 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80550fdd000)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#159 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#160 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80550fdd000)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#161 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#162 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#163 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8055fd15770)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#164 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#165 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8055fd15770)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#166 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#167 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#168 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80576f99770)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#169 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#170 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80576f99770)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#171 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#172 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#173 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff802269bf000)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#174 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#175 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff802269bf000)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#176 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#177 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#178 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8003d7973b8)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#179 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#180 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8003d7973b8)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#181 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#182 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#183 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80108754770)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#184 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#185 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80108754770)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#186 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#187 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#188 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff804cfdf7770)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#189 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#190 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff804cfdf7770)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#191 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#192 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#193 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff801469a13b8)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#194 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#195 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff801469a13b8)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
#196 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#197 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
#198 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff805dcd9d3b8)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
#199 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
#200 0xffffffff80b415aa in taskqueue_run_locked (queue=<value optimized
out>)
    at /usr/src/sys/kern/subr_taskqueue.c:465
#201 0xffffffff80b424b8 in taskqueue_thread_loop (arg=<value optimized
out>)
    at /usr/src/sys/kern/subr_taskqueue.c:719
#202 0xffffffff80a9ad85 in fork_exit (
    callout=0xffffffff80b423d0 <taskqueue_thread_loop>,
    arg=0xfffff8037db02580, frame=0xfffffe0667f12c00)
    at /usr/src/sys/kern/kern_fork.c:1038
#203 0xffffffff80fac59e in fork_trampoline ()
    at /usr/src/sys/amd64/amd64/exception.S:611
#204 0x0000000000000000 in ?? ()
Current language:  auto; currently minimal
(kgdb)

-- 
Karl Denninger
karl at denninger.net <mailto:karl at denninger.net>
/The Market Ticker/
/[S/MIME encrypted email preferred]/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2996 bytes
Desc: S/MIME Cryptographic Signature
URL:
<http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20160720/05bdadfb/attachment.bin>

Karl Denninger

2016-Jul-20 14:32 UTC

head link

Panic on BETA1 in the ZFS subsystem

The panic occurred during a zfs send/receive operation for system
backup..... I've seen this one before, unfortunately, and it appears
that it's still there -- may be related to
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=207464

On 7/20/2016 09:26, Karl Denninger wrote:> Came out of the ZFS system; this machine is running *unpatched* (none of
> my changes are in the kernel); I have a good core dump if further trace
> information will help.
>
> Dump header from device: /dev/gpt/dump
>   Architecture: amd64
>   Architecture Version: 2
>   Dump Length: 9582022656
>   Blocksize: 512
>   Dumptime: Wed Jul 20 09:10:31 2016
>   Hostname: NewFS.denninger.net
>   Magic: FreeBSD Kernel Dump
>   Version String: FreeBSD 11.0-BETA1 #0 r302489: Sat Jul  9 10:15:24 CDT
> 2016
>     karl at NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP
>   Panic String: double fault
>   Dump Parity: 445173880
>   Bounds: 4
>   Dump Status: good
>
> NewFS.denninger.net dumped core - see /var/crash/vmcore.4
>
> Wed Jul 20 09:18:15 CDT 2016
>
> FreeBSD NewFS.denninger.net 11.0-BETA1 FreeBSD 11.0-BETA1 #0 r302489:
> Sat Jul  9 10:15:24 CDT 2016    
> karl at NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP  amd64
>
> panic: double fault
>
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you
are
> welcome to change it and/or distribute copies of it under certain
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty"
for details.
> This GDB was configured as "amd64-marcel-freebsd"...
>
> Unread portion of the kernel message buffer:
>
> Fatal double fault
> rip = 0xffffffff8220bc92
> rsp = 0xfffffe0667f0ef30
> rbp = 0xfffffe0667f0f3c0
> cpuid = 14; apic id = 34
> panic: double fault
> cpuid = 14
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
> 0xfffffe0649db8e30
> vpanic() at vpanic+0x182/frame 0xfffffe0649db8eb0
> panic() at panic+0x43/frame 0xfffffe0649db8f10
> dblfault_handler() at dblfault_handler+0xa2/frame 0xfffffe0649db8f30
> Xdblfault() at Xdblfault+0xac/frame 0xfffffe0649db8f30
> --- trap 0x17, rip = 0xffffffff8220bc92, rsp = 0xfffffe0667f0ef30, rbp >
0xfffffe0667f0f3c0 ---
> vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x22/frame
> 0xfffffe0667f0f3c0
> vdev_queue_io_done() at vdev_queue_io_done+0x83/frame 0xfffffe0667f0f400
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f430
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f480
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0f4e0
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f530
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0f570
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f5a0
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f5f0
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0f650
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f6a0
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0f6e0
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f710
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f760
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0f7c0
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f810
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0f850
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f880
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f8d0
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0f930
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f980
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0f9c0
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f9f0
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fa40
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0faa0
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0faf0
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0fb30
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0fb60
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fbb0
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0fc10
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fc60
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0fca0
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0fcd0
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fd20
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0fd80
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fdd0
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0fe10
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0fe40
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fe90
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0fef0
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0ff40
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0ff80
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0ffb0
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10000
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10060
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f100b0
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f100f0
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10120
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10170
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f101d0
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10220
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10260
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10290
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f102e0
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10340
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10390
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f103d0
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10400
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10450
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f104b0
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10500
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10540
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10570
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f105c0
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10620
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10670
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f106b0
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f106e0
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10730
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10790
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f107e0
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10820
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10850
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f108a0
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10900
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10950
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10990
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f109c0
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10a10
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10a70
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10ac0
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10b00
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10b30
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10b80
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10be0
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10c30
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10c70
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10ca0
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10cf0
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10d50
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10da0
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10de0
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10e10
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10e60
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10ec0
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10f10
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10f50
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10f80
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10fd0
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11030
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11080
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f110c0
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f110f0
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11140
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f111a0
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f111f0
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11230
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11260
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f112b0
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11310
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11360
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f113a0
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f113d0
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11420
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11480
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f114d0
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11510
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11540
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11590
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f115f0
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11640
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11680
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f116b0
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11700
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11760
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f117b0
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f117f0
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11820
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11870
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f118d0
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11920
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11960
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11990
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f119e0
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11a40
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11a90
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11ad0
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11b00
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11b50
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11bb0
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11c00
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11c40
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11c70
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11cc0
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11d20
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11d70
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11db0
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11de0
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11e30
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11e90
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11ee0
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11f20
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11f50
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11fa0
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f12000
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12050
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f12090
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f120c0
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12110
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f12170
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f121c0
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f12200
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12230
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12280
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f122e0
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12330
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f12370
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f123a0
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f123f0
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f12450
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f124a0
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f124e0
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12510
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12560
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f125c0
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12610
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f12650
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12680
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f126d0
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f12730
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12780
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f127c0
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f127f0
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12840
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f128a0
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f128f0
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f12930
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12960
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f129b0
> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f12a10
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12a60
> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f12aa0
> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12ad0
> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12b20
> taskqueue_run_locked() at taskqueue_run_locked+0x14a/frame
> 0xfffffe0667f12b80
> taskqueue_thread_loop() at taskqueue_thread_loop+0xe8/frame
> 0xfffffe0667f12bb0
> fork_exit() at fork_exit+0x85/frame 0xfffffe0667f12bf0
> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0667f12bf0
> --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
> Uptime: 10d21h36m8s
> Dumping 9138 out of 24531
> MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%
>
> Reading symbols from /boot/kernel/zfs.ko...Reading symbols from
> /usr/lib/debug//boot/kernel/zfs.ko.debug...done.
> done.
> Loaded symbols for /boot/kernel/zfs.ko
> Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from
> /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done.
> done.
> Loaded symbols for /boot/kernel/opensolaris.ko
> Reading symbols from /boot/kernel/geom_eli.ko...Reading symbols from
> /usr/lib/debug//boot/kernel/geom_eli.ko.debug...done.
> done.
> Loaded symbols for /boot/kernel/geom_eli.ko
> Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from
> /usr/lib/debug//boot/kernel/geom_mirror.ko.debug...done.
> done.
> Loaded symbols for /boot/kernel/geom_mirror.ko
> Reading symbols from /boot/kernel/aesni.ko...Reading symbols from
> /usr/lib/debug//boot/kernel/aesni.ko.debug...done.
> done.
> Loaded symbols for /boot/kernel/aesni.ko
> Reading symbols from /boot/kernel/uhid.ko...Reading symbols from
> /usr/lib/debug//boot/kernel/uhid.ko.debug...done.
> done.
> Loaded symbols for /boot/kernel/uhid.ko
> Reading symbols from /boot/kernel/ums.ko...Reading symbols from
> /usr/lib/debug//boot/kernel/ums.ko.debug...done.
> done.
> Loaded symbols for /boot/kernel/ums.ko
> Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from
> /usr/lib/debug//boot/kernel/ipfw.ko.debug...done.
> done.
> Loaded symbols for /boot/kernel/ipfw.ko
> Reading symbols from /boot/kernel/ipfw_nat.ko...Reading symbols from
> /usr/lib/debug//boot/kernel/ipfw_nat.ko.debug...done.
> done.
> Loaded symbols for /boot/kernel/ipfw_nat.ko
> Reading symbols from /boot/kernel/libalias.ko...Reading symbols from
> /usr/lib/debug//boot/kernel/libalias.ko.debug...done.
> done.
> Loaded symbols for /boot/kernel/libalias.ko
> Reading symbols from /boot/kernel/daemon_saver.ko...Reading symbols from
> /usr/lib/debug//boot/kernel/daemon_saver.ko.debug...done.
> done.
> Loaded symbols for /boot/kernel/daemon_saver.ko
> #0  doadump (textdump=1) at pcpu.h:221
> 221    pcpu.h: No such file or directory.
>     in pcpu.h
> (kgdb) #0  doadump (textdump=1) at pcpu.h:221
> #1  0xffffffff80ae548e in kern_reboot (howto=260)
>     at /usr/src/sys/kern/kern_shutdown.c:366
> #2  0xffffffff80ae5a5b in vpanic (fmt=<value optimized out>,
>     ap=<value optimized out>) at
/usr/src/sys/kern/kern_shutdown.c:759
> #3  0xffffffff80ae5893 in panic (fmt=0x0)
>     at /usr/src/sys/kern/kern_shutdown.c:690
> #4  0xffffffff80fc9692 in dblfault_handler (frame=<value optimized
out>)
>     at /usr/src/sys/amd64/amd64/trap.c:868
> #5  0xffffffff80fac15c in Xdblfault ()
>     at /usr/src/sys/amd64/amd64/exception.S:292
> #6  0xffffffff8220bc92 in vdev_queue_io_to_issue (vq=0xfffff804d86833e0)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:585
> #7  0xffffffff8220c6e3 in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:881
> #8  0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8018859c000)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #9  0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #10 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8018859c000)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #11 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #12 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #13 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff801df1f43b8)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #14 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #15 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff801df1f43b8)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #16 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #17 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #18 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8024583e770)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #19 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #20 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8024583e770)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #21 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #22 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #23 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff802e12df770)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #24 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #25 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff802e12df770)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #26 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #27 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #28 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8004f81f3b8)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #29 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #30 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8004f81f3b8)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #31 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #32 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #33 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8021bdf7b28)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #34 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #35 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8021bdf7b28)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #36 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #37 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #38 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80165851770)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #39 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #40 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80165851770)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #41 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #42 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #43 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8060de643b8)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #44 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #45 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8060de643b8)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #46 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #47 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #48 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff800793c0770)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #49 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #50 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff800793c0770)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #51 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #52 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #53 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8026917cb28)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #54 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #55 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8026917cb28)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #56 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #57 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #58 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8012408d3b8)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #59 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #60 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8012408d3b8)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #61 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #62 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #63 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff802bfc9ab28)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #64 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #65 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff802bfc9ab28)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #66 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #67 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #68 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff802dda5db28)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #69 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #70 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff802dda5db28)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #71 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #72 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #73 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff800ae13d000)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #74 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #75 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff800ae13d000)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #76 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #77 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #78 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff804a70efb28)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #79 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #80 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff804a70efb28)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #81 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #82 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #83 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80485f5f000)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #84 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #85 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80485f5f000)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #86 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #87 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #88 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff800666fc3b8)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #89 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #90 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff800666fc3b8)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #91 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #92 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #93 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8010da3ab28)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #94 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #95 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8010da3ab28)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #96 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #97 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #98 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80315f73770)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #99 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #100 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80315f73770)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #101 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #102 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #103 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff806011703b8)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #104 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #105 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff806011703b8)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #106 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #107 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #108 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff805964f1770)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #109 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #110 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff805964f1770)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #111 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #112 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #113 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8023902f770)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #114 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #115 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8023902f770)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #116 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #117 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #118 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff803bc7e2b28)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #119 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #120 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff803bc7e2b28)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #121 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #122 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #123 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff804dfbf43b8)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #124 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #125 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff804dfbf43b8)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #126 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #127 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #128 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8005be4a3b8)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #129 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #130 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8005be4a3b8)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #131 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #132 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #133 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8004ff79000)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #134 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #135 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8004ff79000)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #136 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #137 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #138 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff801aa2a2000)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #139 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #140 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff801aa2a2000)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #141 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #142 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #143 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff800592003b8)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #144 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #145 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff800592003b8)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #146 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #147 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #148 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80202d11770)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #149 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #150 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80202d11770)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #151 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #152 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #153 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80392d3c3b8)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #154 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #155 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80392d3c3b8)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #156 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #157 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #158 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80550fdd000)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #159 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #160 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80550fdd000)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #161 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #162 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #163 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8055fd15770)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #164 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #165 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8055fd15770)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #166 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #167 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #168 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80576f99770)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #169 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #170 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80576f99770)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #171 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #172 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #173 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff802269bf000)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #174 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #175 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff802269bf000)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #176 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #177 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #178 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8003d7973b8)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #179 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #180 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8003d7973b8)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #181 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #182 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #183 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80108754770)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #184 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #185 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80108754770)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #186 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #187 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #188 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff804cfdf7770)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #189 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #190 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff804cfdf7770)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #191 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #192 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #193 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff801469a13b8)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #194 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #195 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff801469a13b8)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #196 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #197 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #198 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff805dcd9d3b8)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #199 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #200 0xffffffff80b415aa in taskqueue_run_locked (queue=<value optimized
> out>)
>     at /usr/src/sys/kern/subr_taskqueue.c:465
> #201 0xffffffff80b424b8 in taskqueue_thread_loop (arg=<value optimized
out>)
>     at /usr/src/sys/kern/subr_taskqueue.c:719
> #202 0xffffffff80a9ad85 in fork_exit (
>     callout=0xffffffff80b423d0 <taskqueue_thread_loop>,
>     arg=0xfffff8037db02580, frame=0xfffffe0667f12c00)
>     at /usr/src/sys/kern/kern_fork.c:1038
> #203 0xffffffff80fac59e in fork_trampoline ()
>     at /usr/src/sys/amd64/amd64/exception.S:611
> #204 0x0000000000000000 in ?? ()
> Current language:  auto; currently minimal
> (kgdb)
>
-- 
Karl Denninger
karl at denninger.net <mailto:karl at denninger.net>
/The Market Ticker/
/[S/MIME encrypted email preferred]/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2996 bytes
Desc: S/MIME Cryptographic Signature
URL:
<http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20160720/3dfd8777/attachment.bin>

Andriy Gapon

2016-Jul-20 21:32 UTC

head link

Panic on BETA1 in the ZFS subsystem

On 20/07/2016 17:26, Karl Denninger wrote:> #185 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80108754770)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #186 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #187 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #188 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff804cfdf7770)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #189 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #190 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff804cfdf7770)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #191 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #192 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #193 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff801469a13b8)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
> #194 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #195 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff801469a13b8)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
> #196 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>     at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
> #197 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>     at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
> #198 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff805dcd9d3b8)
I see how this pseudo-recursion can happen in theory, but not why it has
happened.  It would interesting to see 3 or 4 of zio-s printed,
including the top-most one at 0xfffff805dcd9d3b8.

Something like:

set print pretty
p *zio
p *zio->io_vd
p *zio->io_vd->vdev_ops

in several frames.

-- 
Andriy Gapon

freebsd stable - Jul 2016 - Panic on BETA1 in the ZFS subsystem

Panic on BETA1 in the ZFS subsystem

Panic on BETA1 in the ZFS subsystem

Panic on BETA1 in the ZFS subsystem