freebsd stable - Oct 2016 - Repeatable panic on ZFS filesystem (used for backups); 11.0-STABLE

I can certainly attempt setting that higher.... but is that not just
hiding the problem rather than addressing it?


On 10/17/2016 13:54, Steven Hartland wrote:
> You're hitting stack exhaustion, have you tried increasing the kernel
> stack pages?
> It can be changed from /boot/loader.conf
> kern.kstack_pages="6"
>
> Default on amd64 is 4 IIRC
>
> On 17/10/2016 19:08, Karl Denninger wrote:
>> The target (and devices that trigger this) are a pair of 4Gb 7200RPM
>> SATA rotating rust drives (zmirror) with each provider geli-encrypted
>> (that is, the actual devices used for the pool create are the
.eli's)
>>
>> The machine generating the problem has both rotating rust devices *and*
>> SSDs, so I can't simply shut TRIM off system-wide and call it a day
as
>> TRIM itself is heavily-used; both the boot/root pools and a Postgresql
>> database pool are on SSDs, while several terabytes of lesser-used data
>> is on a pool of Raidz2 that is made up of spinning rust.
> snip...
>>
>> NewFS.denninger.net dumped core - see /var/crash/vmcore.1
>>
>> Mon Oct 17 09:02:33 CDT 2016
>>
>> FreeBSD NewFS.denninger.net 11.0-STABLE FreeBSD 11.0-STABLE #13
>> r307318M: Fri Oct 14 09:23:46 CDT 2016
>> karl at NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP  amd64
>>
>> panic: double fault
>>
>> GNU gdb 6.1.1 [FreeBSD]
>> Copyright 2004 Free Software Foundation, Inc.
>> GDB is free software, covered by the GNU General Public License, and
>> you are
>> welcome to change it and/or distribute copies of it under certain
>> conditions.
>> Type "show copying" to see the conditions.
>> There is absolutely no warranty for GDB.  Type "show
warranty" for
>> details.
>> This GDB was configured as "amd64-marcel-freebsd"...
>>
>> Unread portion of the kernel message buffer:
>>
>> Fatal double fault
>> rip = 0xffffffff8220d9ec
>> rsp = 0xfffffe066821f000
>> rbp = 0xfffffe066821f020
>> cpuid = 6; apic id = 14
>> panic: double fault
>> cpuid = 6
>> KDB: stack backtrace:
>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
>> 0xfffffe0649d78e30
>> vpanic() at vpanic+0x182/frame 0xfffffe0649d78eb0
>> panic() at panic+0x43/frame 0xfffffe0649d78f10
>> dblfault_handler() at dblfault_handler+0xa2/frame 0xfffffe0649d78f30
>> Xdblfault() at Xdblfault+0xac/frame 0xfffffe0649d78f30
>> --- trap 0x17, rip = 0xffffffff8220d9ec, rsp = 0xfffffe066821f000, rbp
>> 0xfffffe066821f020 ---
>> avl_rotation() at avl_rotation+0xc/frame 0xfffffe066821f020
>> avl_remove() at avl_remove+0x1c8/frame 0xfffffe066821f070
>> vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x87f/frame
>> 0xfffffe066821f530
>> vdev_queue_io_done() at vdev_queue_io_done+0x83/frame
0xfffffe066821f570
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f5a0
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f5f0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821f650
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f6a0
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe066821f6e0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f710
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f760
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821f7c0
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f810
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe066821f850
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f880
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f8d0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821f930
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f980
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe066821f9c0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f9f0
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fa40
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821faa0
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821faf0
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe066821fb30
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fb60
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fbb0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821fc10
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fc60
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe066821fca0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fcd0
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fd20
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821fd80
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fdd0
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe066821fe10
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fe40
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fe90
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821fef0
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821ff40
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe066821ff80
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821ffb0
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220000
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220060
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682200b0
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe06682200f0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220120
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220170
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682201d0
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220220
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668220260
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220290
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682202e0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220340
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220390
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe06682203d0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220400
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220450
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682204b0
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220500
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668220540
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220570
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682205c0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220620
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220670
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe06682206b0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682206e0
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220730
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220790
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682207e0
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668220820
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220850
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682208a0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220900
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220950
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668220990
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682209c0
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220a10
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220a70
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220ac0
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668220b00
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220b30
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220b80
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220be0
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220c30
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668220c70
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220ca0
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220cf0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220d50
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220da0
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668220de0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220e10
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220e60
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220ec0
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220f10
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668220f50
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220f80
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220fd0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221030
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221080
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe06682210c0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682210f0
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221140
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682211a0
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682211f0
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668221230
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221260
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682212b0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221310
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221360
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe06682213a0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682213d0
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221420
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221480
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682214d0
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668221510
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221540
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221590
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682215f0
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221640
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668221680
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682216b0
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221700
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221760
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682217b0
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe06682217f0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221820
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221870
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682218d0
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221920
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668221960
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221990
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682219e0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221a40
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221a90
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668221ad0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221b00
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221b50
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221bb0
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221c00
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668221c40
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221c70
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221cc0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221d20
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221d70
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668221db0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221de0
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221e30
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221e90
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221ee0
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668221f20
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221f50
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221fa0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222000
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222050
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668222090
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682220c0
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222110
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222170
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682221c0
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668222200
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222230
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222280
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682222e0
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222330
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668222370
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682223a0
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682223f0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222450
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682224a0
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe06682224e0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222510
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222560
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682225c0
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222610
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668222650
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222680
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682226d0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222730
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222780
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe06682227c0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682227f0
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222840
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682228a0
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682228f0
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668222930
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222960
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682229b0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222a10
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222a60
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668222aa0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222ad0
>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222b20
>> taskqueue_run_locked() at taskqueue_run_locked+0x14a/frame
>> 0xfffffe0668222b80
>> taskqueue_thread_loop() at taskqueue_thread_loop+0xe8/frame
>> 0xfffffe0668222bb0
>> fork_exit() at fork_exit+0x85/frame 0xfffffe0668222bf0
>> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0668222bf0
>> --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
>> Uptime: 4m29s
>> Dumping 1497 out of 24531
>> MB:..2%..11%..21%..31%..41%..51%..61%..71%..81%..91%
>>
>> Reading symbols from /boot/kernel/zfs.ko...Reading symbols from
>> /usr/lib/debug//boot/kernel/zfs.ko.debug...done.
>> done.
>> Loaded symbols for /boot/kernel/zfs.ko
>> Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from
>> /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done.
>> done.
>> Loaded symbols for /boot/kernel/opensolaris.ko
>> Reading symbols from /boot/kernel/geom_eli.ko...Reading symbols from
>> /usr/lib/debug//boot/kernel/geom_eli.ko.debug...done.
>> done.
>> Loaded symbols for /boot/kernel/geom_eli.ko
>> Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from
>> /usr/lib/debug//boot/kernel/geom_mirror.ko.debug...done.
>> done.
>> Loaded symbols for /boot/kernel/geom_mirror.ko
>> Reading symbols from /boot/kernel/aesni.ko...Reading symbols from
>> /usr/lib/debug//boot/kernel/aesni.ko.debug...done.
>> done.
>> Loaded symbols for /boot/kernel/aesni.ko
>> Reading symbols from /boot/kernel/uhid.ko...Reading symbols from
>> /usr/lib/debug//boot/kernel/uhid.ko.debug...done.
>> done.
>> Loaded symbols for /boot/kernel/uhid.ko
>> Reading symbols from /boot/kernel/ums.ko...Reading symbols from
>> /usr/lib/debug//boot/kernel/ums.ko.debug...done.
>> done.
>> Loaded symbols for /boot/kernel/ums.ko
>> Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from
>> /usr/lib/debug//boot/kernel/ipfw.ko.debug...done.
>> done.
>> Loaded symbols for /boot/kernel/ipfw.ko
>> Reading symbols from /boot/kernel/ipfw_nat.ko...Reading symbols from
>> /usr/lib/debug//boot/kernel/ipfw_nat.ko.debug...done.
>> done.
>> Loaded symbols for /boot/kernel/ipfw_nat.ko
>> Reading symbols from /boot/kernel/libalias.ko...Reading symbols from
>> /usr/lib/debug//boot/kernel/libalias.ko.debug...done.
>> done.
>> Loaded symbols for /boot/kernel/libalias.ko
>> Reading symbols from /boot/kernel/daemon_saver.ko...Reading symbols
from
>> /usr/lib/debug//boot/kernel/daemon_saver.ko.debug...done.
>> done.
>> Loaded symbols for /boot/kernel/daemon_saver.ko
>> #0  doadump (textdump=1) at pcpu.h:221
>> 221     pcpu.h: No such file or directory.
>>          in pcpu.h
>> (kgdb) #0  doadump (textdump=1) at pcpu.h:221
>> #1  0xffffffff80ae9b6e in kern_reboot (howto=260)
>>      at /usr/src/sys/kern/kern_shutdown.c:366
>> #2  0xffffffff80aea13b in vpanic (fmt=<value optimized out>,
>>      ap=<value optimized out>) at
/usr/src/sys/kern/kern_shutdown.c:759
>> #3  0xffffffff80ae9f73 in panic (fmt=0x0)
>>      at /usr/src/sys/kern/kern_shutdown.c:690
>> #4  0xffffffff80fd18f2 in dblfault_handler (frame=<value optimized
out>)
>>      at /usr/src/sys/amd64/amd64/trap.c:868
>> #5  0xffffffff80fb406c in Xdblfault ()
>>      at /usr/src/sys/amd64/amd64/exception.S:292
>> #6  0xffffffff8220d9ec in avl_rotation (tree=0xfffff8004dc54510,
>>      node=0xfffff8056e45cdd0, balance=2)
>>      at /usr/src/sys/cddl/contrib/opensolaris/common/avl/avl.c:304
>> #7  0xffffffff8220de58 in avl_remove (tree=<value optimized out>,
>>      data=<value optimized out>)
>>      at /usr/src/sys/cddl/contrib/opensolaris/common/avl/avl.c:818
>> #8  0xffffffff822a1eff in vdev_queue_io_to_issue (vq=<value
optimized
>> out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:439
>> #9  0xffffffff822a2123 in vdev_queue_io_done (zio=<value optimized
out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:907
>> #10 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e3f33d8)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #11 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #12 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e3f33d8)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>> #13 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #14 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>> #15 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e578000)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #16 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #17 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e578000)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>> #18 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #19 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>> #20 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5ec3d8)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #21 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #22 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5ec3d8)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>> #23 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #24 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>> #25 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e48a000)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #26 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #27 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e48a000)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>> #28 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #29 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>> #30 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e57ab88)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #31 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #32 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e57ab88)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>> #33 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #34 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>> #35 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800a69a3000)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #36 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #37 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800a69a3000)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>> #38 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #39 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>> #40 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5c8000)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #41 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #42 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5c8000)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>> #43 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #44 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>> #45 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a23f9000)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #46 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #47 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a23f9000)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>> #48 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #49 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>> #50 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5147b0)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #51 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #52 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5147b0)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>> #53 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #54 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>> #55 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e977000)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #56 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #57 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e977000)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>> #58 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #59 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>> #60 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e90b3d8)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #61 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #62 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e90b3d8)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>> #63 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #64 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>> #65 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800b1160b88)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #66 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #67 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800b1160b88)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>> #68 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #69 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>> #70 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e4f53d8)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #71 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #72 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e4f53d8)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>> #73 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #74 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>> #75 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e8ebb88)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #76 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #77 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e8ebb88)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>> #78 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #79 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>> #80 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e8d9b88)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #81 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #82 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e8d9b88)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>> #83 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #84 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>> #85 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a21a03d8)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #86 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #87 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a21a03d8)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>> #88 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #89 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>> #90 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e6903d8)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #91 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #92 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e6903d8)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>> #93 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #94 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>> #95 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a22dc7b0)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #96 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #97 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a22dc7b0)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>> #98 0xffffffff822bbefd in zio_execute (zio=<value optimized out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #99 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>> #100 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5fb7b0)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #101 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #102 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5fb7b0)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>> #103 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #104 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>> out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>> #105 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5ae3d8)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #106 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #107 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5ae3d8)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>> #108 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #109 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>> out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>> #110 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a2131b88)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #111 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #112 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a2131b88)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>> #113 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #114 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>> out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>> #115 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e80b7b0)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #116 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #117 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e80b7b0)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>> #118 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #119 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>> out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>> #120 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800b14b9000)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #121 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #122 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800b14b9000)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>> #123 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #124 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>> out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>> #125 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a21fa3d8)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #126 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #127 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a21fa3d8)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>> #128 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #129 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>> out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>> #130 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5333d8)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #131 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #132 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5333d8)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>> #133 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #134 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>> out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>> #135 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e90ab88)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #136 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #137 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e90ab88)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>> #138 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #139 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>> out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>> #140 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e4047b0)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #141 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #142 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e4047b0)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>> #143 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #144 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>> out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>> #145 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e7083d8)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #146 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #147 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e7083d8)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>> #148 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #149 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>> out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>> #150 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a23917b0)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #151 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #152 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a23917b0)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>> #153 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #154 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>> out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>> #155 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e636000)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #156 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #157 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e636000)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>> #158 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #159 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>> out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>> #160 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5afb88)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #161 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #162 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5afb88)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>> #163 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #164 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>> out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>> #165 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800a69523d8)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #166 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #167 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800a69523d8)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>> #168 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #169 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>> out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>> #170 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e47c000)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #171 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #172 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e47c000)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>> #173 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #174 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>> out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>> #175 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800b17b23d8)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #176 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #177 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800b17b23d8)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>> #178 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #179 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>> out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>> #180 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800a6d367b0)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #181 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #182 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800a6d367b0)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>> #183 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #184 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>> out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>> #185 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e99fb88)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #186 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #187 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e99fb88)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>> #188 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #189 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>> out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>> #190 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5227b0)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #191 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #192 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5227b0)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>> #193 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #194 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized
>> out>)
>>      at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>> #195 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800a6d3c000)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>> #196 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>      at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>> #197 0xffffffff80b4895a in taskqueue_run_locked (queue=<value
optimized
>> out>)
>>      at /usr/src/sys/kern/subr_taskqueue.c:454
>> #198 0xffffffff80b49b58 in taskqueue_thread_loop (arg=<value
>> optimized out>)
>>      at /usr/src/sys/kern/subr_taskqueue.c:724
>> #199 0xffffffff80a9f255 in fork_exit (
>>      callout=0xffffffff80b49a70 <taskqueue_thread_loop>,
>>      arg=0xfffff8057caa72a0, frame=0xfffffe0668222c00)
>>      at /usr/src/sys/kern/kern_fork.c:1040
>> #200 0xffffffff80fb44ae in fork_trampoline ()
>>      at /usr/src/sys/amd64/amd64/exception.S:611
>> #201 0x0000000000000000 in ?? ()
>> Current language:  auto; currently minimal
>> (kgdb)
>>
>>
>>>> -- 
>>>> Karl Denninger
>>>> karl at denninger.net <mailto:karl at denninger.net>
>>>> /The Market Ticker/
>>>> /[S/MIME encrypted email preferred]/
>
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at
freebsd.org"
-- 
Karl Denninger
karl at denninger.net <mailto:karl at denninger.net>
/The Market Ticker/
/[S/MIME encrypted email preferred]/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2996 bytes
Desc: S/MIME Cryptographic Signature
URL:
<http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20161017/e2f500b2/attachment-0001.bin>

Be good to confirm its not an infinite loop by giving it a good bump first.

On 17/10/2016 19:58, Karl Denninger wrote:
> I can certainly attempt setting that higher.... but is that not just
> hiding the problem rather than addressing it?
>
>
> On 10/17/2016 13:54, Steven Hartland wrote:
>> You're hitting stack exhaustion, have you tried increasing the
kernel
>> stack pages?
>> It can be changed from /boot/loader.conf
>> kern.kstack_pages="6"
>>
>> Default on amd64 is 4 IIRC
>>
>> On 17/10/2016 19:08, Karl Denninger wrote:
>>> The target (and devices that trigger this) are a pair of 4Gb
7200RPM
>>> SATA rotating rust drives (zmirror) with each provider
geli-encrypted
>>> (that is, the actual devices used for the pool create are the
.eli's)
>>>
>>> The machine generating the problem has both rotating rust devices
*and*
>>> SSDs, so I can't simply shut TRIM off system-wide and call it a
day as
>>> TRIM itself is heavily-used; both the boot/root pools and a
Postgresql
>>> database pool are on SSDs, while several terabytes of lesser-used
data
>>> is on a pool of Raidz2 that is made up of spinning rust.
>> snip...
>>> NewFS.denninger.net dumped core - see /var/crash/vmcore.1
>>>
>>> Mon Oct 17 09:02:33 CDT 2016
>>>
>>> FreeBSD NewFS.denninger.net 11.0-STABLE FreeBSD 11.0-STABLE #13
>>> r307318M: Fri Oct 14 09:23:46 CDT 2016
>>> karl at NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP  amd64
>>>
>>> panic: double fault
>>>
>>> GNU gdb 6.1.1 [FreeBSD]
>>> Copyright 2004 Free Software Foundation, Inc.
>>> GDB is free software, covered by the GNU General Public License,
and
>>> you are
>>> welcome to change it and/or distribute copies of it under certain
>>> conditions.
>>> Type "show copying" to see the conditions.
>>> There is absolutely no warranty for GDB.  Type "show
warranty" for
>>> details.
>>> This GDB was configured as "amd64-marcel-freebsd"...
>>>
>>> Unread portion of the kernel message buffer:
>>>
>>> Fatal double fault
>>> rip = 0xffffffff8220d9ec
>>> rsp = 0xfffffe066821f000
>>> rbp = 0xfffffe066821f020
>>> cpuid = 6; apic id = 14
>>> panic: double fault
>>> cpuid = 6
>>> KDB: stack backtrace:
>>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
>>> 0xfffffe0649d78e30
>>> vpanic() at vpanic+0x182/frame 0xfffffe0649d78eb0
>>> panic() at panic+0x43/frame 0xfffffe0649d78f10
>>> dblfault_handler() at dblfault_handler+0xa2/frame
0xfffffe0649d78f30
>>> Xdblfault() at Xdblfault+0xac/frame 0xfffffe0649d78f30
>>> --- trap 0x17, rip = 0xffffffff8220d9ec, rsp = 0xfffffe066821f000,
rbp >>> 0xfffffe066821f020 ---
>>> avl_rotation() at avl_rotation+0xc/frame 0xfffffe066821f020
>>> avl_remove() at avl_remove+0x1c8/frame 0xfffffe066821f070
>>> vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x87f/frame
>>> 0xfffffe066821f530
>>> vdev_queue_io_done() at vdev_queue_io_done+0x83/frame
0xfffffe066821f570
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe066821f5a0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f5f0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe066821f650
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f6a0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe066821f6e0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe066821f710
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f760
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe066821f7c0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f810
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe066821f850
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe066821f880
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f8d0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe066821f930
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f980
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe066821f9c0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe066821f9f0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fa40
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe066821faa0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821faf0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe066821fb30
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe066821fb60
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fbb0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe066821fc10
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fc60
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe066821fca0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe066821fcd0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fd20
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe066821fd80
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fdd0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe066821fe10
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe066821fe40
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fe90
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe066821fef0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821ff40
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe066821ff80
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe066821ffb0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220000
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668220060
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682200b0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe06682200f0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668220120
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220170
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe06682201d0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220220
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668220260
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668220290
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682202e0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668220340
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220390
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe06682203d0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668220400
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220450
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe06682204b0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220500
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668220540
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668220570
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682205c0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668220620
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220670
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe06682206b0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe06682206e0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220730
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668220790
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682207e0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668220820
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668220850
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682208a0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668220900
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220950
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668220990
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe06682209c0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220a10
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668220a70
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220ac0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668220b00
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668220b30
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220b80
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668220be0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220c30
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668220c70
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668220ca0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220cf0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668220d50
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220da0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668220de0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668220e10
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220e60
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668220ec0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220f10
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668220f50
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668220f80
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220fd0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668221030
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221080
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe06682210c0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe06682210f0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221140
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe06682211a0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682211f0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668221230
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668221260
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682212b0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668221310
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221360
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe06682213a0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe06682213d0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221420
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668221480
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682214d0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668221510
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668221540
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221590
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe06682215f0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221640
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668221680
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe06682216b0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221700
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668221760
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682217b0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe06682217f0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668221820
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221870
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe06682218d0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221920
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668221960
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668221990
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682219e0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668221a40
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221a90
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668221ad0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668221b00
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221b50
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668221bb0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221c00
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668221c40
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668221c70
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221cc0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668221d20
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221d70
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668221db0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668221de0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221e30
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668221e90
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221ee0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668221f20
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668221f50
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221fa0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668222000
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222050
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668222090
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe06682220c0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222110
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668222170
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682221c0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668222200
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668222230
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222280
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe06682222e0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222330
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668222370
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe06682223a0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682223f0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668222450
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682224a0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe06682224e0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668222510
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222560
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe06682225c0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222610
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668222650
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668222680
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682226d0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668222730
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222780
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe06682227c0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe06682227f0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222840
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe06682228a0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682228f0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668222930
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668222960
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682229b0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668222a10
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222a60
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668222aa0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668222ad0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222b20
>>> taskqueue_run_locked() at taskqueue_run_locked+0x14a/frame
>>> 0xfffffe0668222b80
>>> taskqueue_thread_loop() at taskqueue_thread_loop+0xe8/frame
>>> 0xfffffe0668222bb0
>>> fork_exit() at fork_exit+0x85/frame 0xfffffe0668222bf0
>>> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0668222bf0
>>> --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
>>> Uptime: 4m29s
>>> Dumping 1497 out of 24531
>>> MB:..2%..11%..21%..31%..41%..51%..61%..71%..81%..91%
>>>
>>> Reading symbols from /boot/kernel/zfs.ko...Reading symbols from
>>> /usr/lib/debug//boot/kernel/zfs.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/zfs.ko
>>> Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols
from
>>> /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/opensolaris.ko
>>> Reading symbols from /boot/kernel/geom_eli.ko...Reading symbols
from
>>> /usr/lib/debug//boot/kernel/geom_eli.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/geom_eli.ko
>>> Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols
from
>>> /usr/lib/debug//boot/kernel/geom_mirror.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/geom_mirror.ko
>>> Reading symbols from /boot/kernel/aesni.ko...Reading symbols from
>>> /usr/lib/debug//boot/kernel/aesni.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/aesni.ko
>>> Reading symbols from /boot/kernel/uhid.ko...Reading symbols from
>>> /usr/lib/debug//boot/kernel/uhid.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/uhid.ko
>>> Reading symbols from /boot/kernel/ums.ko...Reading symbols from
>>> /usr/lib/debug//boot/kernel/ums.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/ums.ko
>>> Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from
>>> /usr/lib/debug//boot/kernel/ipfw.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/ipfw.ko
>>> Reading symbols from /boot/kernel/ipfw_nat.ko...Reading symbols
from
>>> /usr/lib/debug//boot/kernel/ipfw_nat.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/ipfw_nat.ko
>>> Reading symbols from /boot/kernel/libalias.ko...Reading symbols
from
>>> /usr/lib/debug//boot/kernel/libalias.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/libalias.ko
>>> Reading symbols from /boot/kernel/daemon_saver.ko...Reading symbols
from
>>> /usr/lib/debug//boot/kernel/daemon_saver.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/daemon_saver.ko
>>> #0  doadump (textdump=1) at pcpu.h:221
>>> 221     pcpu.h: No such file or directory.
>>>           in pcpu.h
>>> (kgdb) #0  doadump (textdump=1) at pcpu.h:221
>>> #1  0xffffffff80ae9b6e in kern_reboot (howto=260)
>>>       at /usr/src/sys/kern/kern_shutdown.c:366
>>> #2  0xffffffff80aea13b in vpanic (fmt=<value optimized out>,
>>>       ap=<value optimized out>) at
/usr/src/sys/kern/kern_shutdown.c:759
>>> #3  0xffffffff80ae9f73 in panic (fmt=0x0)
>>>       at /usr/src/sys/kern/kern_shutdown.c:690
>>> #4  0xffffffff80fd18f2 in dblfault_handler (frame=<value
optimized out>)
>>>       at /usr/src/sys/amd64/amd64/trap.c:868
>>> #5  0xffffffff80fb406c in Xdblfault ()
>>>       at /usr/src/sys/amd64/amd64/exception.S:292
>>> #6  0xffffffff8220d9ec in avl_rotation (tree=0xfffff8004dc54510,
>>>       node=0xfffff8056e45cdd0, balance=2)
>>>       at /usr/src/sys/cddl/contrib/opensolaris/common/avl/avl.c:304
>>> #7  0xffffffff8220de58 in avl_remove (tree=<value optimized
out>,
>>>       data=<value optimized out>)
>>>       at /usr/src/sys/cddl/contrib/opensolaris/common/avl/avl.c:818
>>> #8  0xffffffff822a1eff in vdev_queue_io_to_issue (vq=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:439
>>> #9  0xffffffff822a2123 in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:907
>>> #10 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e3f33d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #11 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #12 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e3f33d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #13 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #14 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #15 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e578000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #16 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #17 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e578000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #18 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #19 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #20 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5ec3d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #21 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #22 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e5ec3d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #23 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #24 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #25 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e48a000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #26 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #27 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e48a000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #28 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #29 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #30 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e57ab88)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #31 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #32 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e57ab88)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #33 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #34 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #35 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800a69a3000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #36 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #37 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff800a69a3000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #38 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #39 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #40 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5c8000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #41 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #42 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e5c8000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #43 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #44 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #45 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a23f9000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #46 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #47 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff802a23f9000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #48 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #49 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #50 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5147b0)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #51 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #52 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e5147b0)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #53 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #54 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #55 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e977000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #56 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #57 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e977000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #58 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #59 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #60 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e90b3d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #61 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #62 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e90b3d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #63 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #64 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #65 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800b1160b88)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #66 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #67 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff800b1160b88)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #68 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #69 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #70 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e4f53d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #71 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #72 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e4f53d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #73 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #74 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #75 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e8ebb88)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #76 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #77 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e8ebb88)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #78 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #79 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #80 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e8d9b88)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #81 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #82 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e8d9b88)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #83 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #84 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #85 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a21a03d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #86 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #87 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff802a21a03d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #88 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #89 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #90 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e6903d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #91 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #92 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e6903d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #93 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #94 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #95 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a22dc7b0)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #96 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #97 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff802a22dc7b0)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #98 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #99 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #100 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e5fb7b0)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #101 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #102 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e5fb7b0)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #103 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #104 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #105 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e5ae3d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #106 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #107 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e5ae3d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #108 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #109 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #110 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff802a2131b88)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #111 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #112 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff802a2131b88)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #113 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #114 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #115 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e80b7b0)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #116 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #117 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e80b7b0)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #118 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #119 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #120 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff800b14b9000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #121 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #122 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff800b14b9000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #123 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #124 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #125 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff802a21fa3d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #126 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #127 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff802a21fa3d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #128 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #129 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #130 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e5333d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #131 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #132 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e5333d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #133 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #134 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #135 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e90ab88)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #136 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #137 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e90ab88)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #138 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #139 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #140 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e4047b0)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #141 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #142 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e4047b0)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #143 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #144 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #145 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e7083d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #146 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #147 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e7083d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #148 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #149 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #150 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff802a23917b0)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #151 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #152 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff802a23917b0)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #153 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #154 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #155 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e636000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #156 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #157 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e636000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #158 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #159 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #160 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e5afb88)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #161 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #162 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e5afb88)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #163 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #164 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #165 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff800a69523d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #166 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #167 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff800a69523d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #168 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #169 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #170 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e47c000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #171 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #172 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e47c000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #173 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #174 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #175 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff800b17b23d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #176 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #177 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff800b17b23d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #178 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #179 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #180 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff800a6d367b0)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #181 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #182 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff800a6d367b0)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #183 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #184 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #185 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e99fb88)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #186 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #187 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e99fb88)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #188 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #189 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #190 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e5227b0)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #191 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #192 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e5227b0)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #193 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #194 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #195 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff800a6d3c000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #196 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #197 0xffffffff80b4895a in taskqueue_run_locked (queue=<value
optimized
>>> out>)
>>>       at /usr/src/sys/kern/subr_taskqueue.c:454
>>> #198 0xffffffff80b49b58 in taskqueue_thread_loop (arg=<value
>>> optimized out>)
>>>       at /usr/src/sys/kern/subr_taskqueue.c:724
>>> #199 0xffffffff80a9f255 in fork_exit (
>>>       callout=0xffffffff80b49a70 <taskqueue_thread_loop>,
>>>       arg=0xfffff8057caa72a0, frame=0xfffffe0668222c00)
>>>       at /usr/src/sys/kern/kern_fork.c:1040
>>> #200 0xffffffff80fb44ae in fork_trampoline ()
>>>       at /usr/src/sys/amd64/amd64/exception.S:611
>>> #201 0x0000000000000000 in ?? ()
>>> Current language:  auto; currently minimal
>>> (kgdb)
>>>
>>>
>>>>> -- 
>>>>> Karl Denninger
>>>>> karl at denninger.net <mailto:karl at denninger.net>
>>>>> /The Market Ticker/
>>>>> /[S/MIME encrypted email preferred]/
>> _______________________________________________
>> freebsd-stable at freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
>> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at
freebsd.org"