freebsd stable - Oct 2016 - Repeatable panic on ZFS filesystem (used for backups); 11.0-STABLE

Be good to confirm its not an infinite loop by giving it a good bump first.

On 17/10/2016 19:58, Karl Denninger wrote:
> I can certainly attempt setting that higher.... but is that not just
> hiding the problem rather than addressing it?
>
>
> On 10/17/2016 13:54, Steven Hartland wrote:
>> You're hitting stack exhaustion, have you tried increasing the
kernel
>> stack pages?
>> It can be changed from /boot/loader.conf
>> kern.kstack_pages="6"
>>
>> Default on amd64 is 4 IIRC
>>
>> On 17/10/2016 19:08, Karl Denninger wrote:
>>> The target (and devices that trigger this) are a pair of 4Gb
7200RPM
>>> SATA rotating rust drives (zmirror) with each provider
geli-encrypted
>>> (that is, the actual devices used for the pool create are the
.eli's)
>>>
>>> The machine generating the problem has both rotating rust devices
*and*
>>> SSDs, so I can't simply shut TRIM off system-wide and call it a
day as
>>> TRIM itself is heavily-used; both the boot/root pools and a
Postgresql
>>> database pool are on SSDs, while several terabytes of lesser-used
data
>>> is on a pool of Raidz2 that is made up of spinning rust.
>> snip...
>>> NewFS.denninger.net dumped core - see /var/crash/vmcore.1
>>>
>>> Mon Oct 17 09:02:33 CDT 2016
>>>
>>> FreeBSD NewFS.denninger.net 11.0-STABLE FreeBSD 11.0-STABLE #13
>>> r307318M: Fri Oct 14 09:23:46 CDT 2016
>>> karl at NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP  amd64
>>>
>>> panic: double fault
>>>
>>> GNU gdb 6.1.1 [FreeBSD]
>>> Copyright 2004 Free Software Foundation, Inc.
>>> GDB is free software, covered by the GNU General Public License,
and
>>> you are
>>> welcome to change it and/or distribute copies of it under certain
>>> conditions.
>>> Type "show copying" to see the conditions.
>>> There is absolutely no warranty for GDB.  Type "show
warranty" for
>>> details.
>>> This GDB was configured as "amd64-marcel-freebsd"...
>>>
>>> Unread portion of the kernel message buffer:
>>>
>>> Fatal double fault
>>> rip = 0xffffffff8220d9ec
>>> rsp = 0xfffffe066821f000
>>> rbp = 0xfffffe066821f020
>>> cpuid = 6; apic id = 14
>>> panic: double fault
>>> cpuid = 6
>>> KDB: stack backtrace:
>>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
>>> 0xfffffe0649d78e30
>>> vpanic() at vpanic+0x182/frame 0xfffffe0649d78eb0
>>> panic() at panic+0x43/frame 0xfffffe0649d78f10
>>> dblfault_handler() at dblfault_handler+0xa2/frame
0xfffffe0649d78f30
>>> Xdblfault() at Xdblfault+0xac/frame 0xfffffe0649d78f30
>>> --- trap 0x17, rip = 0xffffffff8220d9ec, rsp = 0xfffffe066821f000,
rbp >>> 0xfffffe066821f020 ---
>>> avl_rotation() at avl_rotation+0xc/frame 0xfffffe066821f020
>>> avl_remove() at avl_remove+0x1c8/frame 0xfffffe066821f070
>>> vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x87f/frame
>>> 0xfffffe066821f530
>>> vdev_queue_io_done() at vdev_queue_io_done+0x83/frame
0xfffffe066821f570
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe066821f5a0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f5f0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe066821f650
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f6a0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe066821f6e0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe066821f710
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f760
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe066821f7c0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f810
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe066821f850
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe066821f880
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f8d0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe066821f930
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f980
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe066821f9c0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe066821f9f0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fa40
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe066821faa0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821faf0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe066821fb30
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe066821fb60
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fbb0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe066821fc10
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fc60
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe066821fca0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe066821fcd0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fd20
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe066821fd80
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fdd0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe066821fe10
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe066821fe40
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fe90
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe066821fef0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821ff40
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe066821ff80
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe066821ffb0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220000
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668220060
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682200b0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe06682200f0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668220120
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220170
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe06682201d0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220220
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668220260
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668220290
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682202e0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668220340
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220390
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe06682203d0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668220400
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220450
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe06682204b0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220500
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668220540
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668220570
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682205c0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668220620
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220670
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe06682206b0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe06682206e0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220730
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668220790
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682207e0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668220820
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668220850
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682208a0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668220900
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220950
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668220990
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe06682209c0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220a10
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668220a70
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220ac0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668220b00
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668220b30
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220b80
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668220be0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220c30
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668220c70
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668220ca0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220cf0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668220d50
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220da0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668220de0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668220e10
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220e60
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668220ec0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220f10
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668220f50
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668220f80
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220fd0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668221030
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221080
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe06682210c0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe06682210f0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221140
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe06682211a0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682211f0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668221230
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668221260
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682212b0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668221310
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221360
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe06682213a0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe06682213d0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221420
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668221480
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682214d0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668221510
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668221540
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221590
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe06682215f0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221640
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668221680
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe06682216b0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221700
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668221760
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682217b0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe06682217f0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668221820
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221870
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe06682218d0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221920
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668221960
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668221990
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682219e0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668221a40
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221a90
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668221ad0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668221b00
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221b50
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668221bb0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221c00
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668221c40
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668221c70
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221cc0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668221d20
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221d70
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668221db0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668221de0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221e30
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668221e90
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221ee0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668221f20
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668221f50
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221fa0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668222000
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222050
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668222090
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe06682220c0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222110
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668222170
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682221c0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668222200
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668222230
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222280
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe06682222e0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222330
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668222370
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe06682223a0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682223f0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668222450
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682224a0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe06682224e0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668222510
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222560
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe06682225c0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222610
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668222650
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668222680
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682226d0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668222730
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222780
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe06682227c0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe06682227f0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222840
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe06682228a0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682228f0
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668222930
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668222960
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682229b0
>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
0xfffffe0668222a10
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222a60
>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0668222aa0
>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668222ad0
>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222b20
>>> taskqueue_run_locked() at taskqueue_run_locked+0x14a/frame
>>> 0xfffffe0668222b80
>>> taskqueue_thread_loop() at taskqueue_thread_loop+0xe8/frame
>>> 0xfffffe0668222bb0
>>> fork_exit() at fork_exit+0x85/frame 0xfffffe0668222bf0
>>> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0668222bf0
>>> --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
>>> Uptime: 4m29s
>>> Dumping 1497 out of 24531
>>> MB:..2%..11%..21%..31%..41%..51%..61%..71%..81%..91%
>>>
>>> Reading symbols from /boot/kernel/zfs.ko...Reading symbols from
>>> /usr/lib/debug//boot/kernel/zfs.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/zfs.ko
>>> Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols
from
>>> /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/opensolaris.ko
>>> Reading symbols from /boot/kernel/geom_eli.ko...Reading symbols
from
>>> /usr/lib/debug//boot/kernel/geom_eli.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/geom_eli.ko
>>> Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols
from
>>> /usr/lib/debug//boot/kernel/geom_mirror.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/geom_mirror.ko
>>> Reading symbols from /boot/kernel/aesni.ko...Reading symbols from
>>> /usr/lib/debug//boot/kernel/aesni.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/aesni.ko
>>> Reading symbols from /boot/kernel/uhid.ko...Reading symbols from
>>> /usr/lib/debug//boot/kernel/uhid.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/uhid.ko
>>> Reading symbols from /boot/kernel/ums.ko...Reading symbols from
>>> /usr/lib/debug//boot/kernel/ums.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/ums.ko
>>> Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from
>>> /usr/lib/debug//boot/kernel/ipfw.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/ipfw.ko
>>> Reading symbols from /boot/kernel/ipfw_nat.ko...Reading symbols
from
>>> /usr/lib/debug//boot/kernel/ipfw_nat.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/ipfw_nat.ko
>>> Reading symbols from /boot/kernel/libalias.ko...Reading symbols
from
>>> /usr/lib/debug//boot/kernel/libalias.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/libalias.ko
>>> Reading symbols from /boot/kernel/daemon_saver.ko...Reading symbols
from
>>> /usr/lib/debug//boot/kernel/daemon_saver.ko.debug...done.
>>> done.
>>> Loaded symbols for /boot/kernel/daemon_saver.ko
>>> #0  doadump (textdump=1) at pcpu.h:221
>>> 221     pcpu.h: No such file or directory.
>>>           in pcpu.h
>>> (kgdb) #0  doadump (textdump=1) at pcpu.h:221
>>> #1  0xffffffff80ae9b6e in kern_reboot (howto=260)
>>>       at /usr/src/sys/kern/kern_shutdown.c:366
>>> #2  0xffffffff80aea13b in vpanic (fmt=<value optimized out>,
>>>       ap=<value optimized out>) at
/usr/src/sys/kern/kern_shutdown.c:759
>>> #3  0xffffffff80ae9f73 in panic (fmt=0x0)
>>>       at /usr/src/sys/kern/kern_shutdown.c:690
>>> #4  0xffffffff80fd18f2 in dblfault_handler (frame=<value
optimized out>)
>>>       at /usr/src/sys/amd64/amd64/trap.c:868
>>> #5  0xffffffff80fb406c in Xdblfault ()
>>>       at /usr/src/sys/amd64/amd64/exception.S:292
>>> #6  0xffffffff8220d9ec in avl_rotation (tree=0xfffff8004dc54510,
>>>       node=0xfffff8056e45cdd0, balance=2)
>>>       at /usr/src/sys/cddl/contrib/opensolaris/common/avl/avl.c:304
>>> #7  0xffffffff8220de58 in avl_remove (tree=<value optimized
out>,
>>>       data=<value optimized out>)
>>>       at /usr/src/sys/cddl/contrib/opensolaris/common/avl/avl.c:818
>>> #8  0xffffffff822a1eff in vdev_queue_io_to_issue (vq=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:439
>>> #9  0xffffffff822a2123 in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:907
>>> #10 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e3f33d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #11 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #12 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e3f33d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #13 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #14 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #15 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e578000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #16 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #17 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e578000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #18 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #19 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #20 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5ec3d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #21 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #22 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e5ec3d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #23 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #24 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #25 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e48a000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #26 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #27 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e48a000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #28 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #29 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #30 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e57ab88)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #31 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #32 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e57ab88)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #33 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #34 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #35 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800a69a3000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #36 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #37 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff800a69a3000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #38 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #39 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #40 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5c8000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #41 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #42 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e5c8000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #43 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #44 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #45 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a23f9000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #46 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #47 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff802a23f9000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #48 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #49 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #50 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5147b0)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #51 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #52 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e5147b0)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #53 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #54 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #55 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e977000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #56 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #57 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e977000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #58 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #59 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #60 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e90b3d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #61 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #62 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e90b3d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #63 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #64 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #65 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800b1160b88)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #66 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #67 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff800b1160b88)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #68 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #69 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #70 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e4f53d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #71 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #72 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e4f53d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #73 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #74 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #75 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e8ebb88)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #76 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #77 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e8ebb88)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #78 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #79 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #80 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e8d9b88)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #81 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #82 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e8d9b88)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #83 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #84 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #85 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a21a03d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #86 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #87 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff802a21a03d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #88 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #89 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #90 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e6903d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #91 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #92 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e6903d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #93 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #94 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #95 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a22dc7b0)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #96 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #97 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff802a22dc7b0)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #98 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #99 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #100 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e5fb7b0)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #101 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #102 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e5fb7b0)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #103 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #104 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #105 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e5ae3d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #106 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #107 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e5ae3d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #108 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #109 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #110 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff802a2131b88)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #111 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #112 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff802a2131b88)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #113 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #114 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #115 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e80b7b0)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #116 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #117 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e80b7b0)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #118 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #119 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #120 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff800b14b9000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #121 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #122 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff800b14b9000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #123 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #124 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #125 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff802a21fa3d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #126 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #127 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff802a21fa3d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #128 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #129 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #130 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e5333d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #131 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #132 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e5333d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #133 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #134 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #135 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e90ab88)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #136 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #137 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e90ab88)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #138 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #139 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #140 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e4047b0)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #141 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #142 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e4047b0)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #143 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #144 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #145 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e7083d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #146 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #147 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e7083d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #148 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #149 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #150 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff802a23917b0)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #151 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #152 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff802a23917b0)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #153 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #154 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #155 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e636000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #156 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #157 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e636000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #158 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #159 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #160 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e5afb88)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #161 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #162 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e5afb88)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #163 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #164 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #165 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff800a69523d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #166 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #167 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff800a69523d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #168 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #169 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #170 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e47c000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #171 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #172 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e47c000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #173 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #174 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #175 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff800b17b23d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #176 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #177 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff800b17b23d8)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #178 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #179 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #180 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff800a6d367b0)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #181 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #182 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff800a6d367b0)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #183 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #184 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #185 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e99fb88)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #186 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #187 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e99fb88)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #188 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #189 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #190 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e5227b0)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #191 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #192 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e5227b0)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>> #193 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #194 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>> out>)
>>>       at
>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>> #195 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff800a6d3c000)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>> #196 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>       at
>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>> #197 0xffffffff80b4895a in taskqueue_run_locked (queue=<value
optimized
>>> out>)
>>>       at /usr/src/sys/kern/subr_taskqueue.c:454
>>> #198 0xffffffff80b49b58 in taskqueue_thread_loop (arg=<value
>>> optimized out>)
>>>       at /usr/src/sys/kern/subr_taskqueue.c:724
>>> #199 0xffffffff80a9f255 in fork_exit (
>>>       callout=0xffffffff80b49a70 <taskqueue_thread_loop>,
>>>       arg=0xfffff8057caa72a0, frame=0xfffffe0668222c00)
>>>       at /usr/src/sys/kern/kern_fork.c:1040
>>> #200 0xffffffff80fb44ae in fork_trampoline ()
>>>       at /usr/src/sys/amd64/amd64/exception.S:611
>>> #201 0x0000000000000000 in ?? ()
>>> Current language:  auto; currently minimal
>>> (kgdb)
>>>
>>>
>>>>> -- 
>>>>> Karl Denninger
>>>>> karl at denninger.net <mailto:karl at denninger.net>
>>>>> /The Market Ticker/
>>>>> /[S/MIME encrypted email preferred]/
>> _______________________________________________
>> freebsd-stable at freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
>> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at
freebsd.org"

Since I cleared it (by setting TRIM off on the test machine, rebooting,
importing the pool and noting that it did not panic -- pulled drives,
re-inserted into the production machine and ran backup routine -- all
was normal) it may be a while before I see it again (a week or so is usual.)

It appears to be related to entropy in the filesystem that comes up as
"eligible" to be removed from the backup volume, which (not
surprisingly) tends to happen a few days after I do a new world build or
something similar (the daily and/or periodic snapshots roll off at about
that point.)

I don't happen to have a spare pair of high-performance SSDs I can stick
in the sandbox machine in an attempt to force the condition to assert
itself in test, unfortunately.

I *am* concerned that it's not "simple" stack exhaustion because
setting
the max outstanding TRIMs on a per-vdev basis down quite-dramatically
did *not* prevent it from happening -- and if it was simply stack depth
related I would have expected that to put a stop to it.

On 10/17/2016 15:16, Steven Hartland wrote:
> Be good to confirm its not an infinite loop by giving it a good bump
> first.
>
> On 17/10/2016 19:58, Karl Denninger wrote:
>> I can certainly attempt setting that higher.... but is that not just
>> hiding the problem rather than addressing it?
>>
>>
>> On 10/17/2016 13:54, Steven Hartland wrote:
>>> You're hitting stack exhaustion, have you tried increasing the
kernel
>>> stack pages?
>>> It can be changed from /boot/loader.conf
>>> kern.kstack_pages="6"
>>>
>>> Default on amd64 is 4 IIRC
>>>
>>> On 17/10/2016 19:08, Karl Denninger wrote:
>>>> The target (and devices that trigger this) are a pair of 4Gb
7200RPM
>>>> SATA rotating rust drives (zmirror) with each provider
geli-encrypted
>>>> (that is, the actual devices used for the pool create are the
.eli's)
>>>>
>>>> The machine generating the problem has both rotating rust
devices
>>>> *and*
>>>> SSDs, so I can't simply shut TRIM off system-wide and call
it a day as
>>>> TRIM itself is heavily-used; both the boot/root pools and a
Postgresql
>>>> database pool are on SSDs, while several terabytes of
lesser-used data
>>>> is on a pool of Raidz2 that is made up of spinning rust.
>>> snip...
>>>> NewFS.denninger.net dumped core - see /var/crash/vmcore.1
>>>>
>>>> Mon Oct 17 09:02:33 CDT 2016
>>>>
>>>> FreeBSD NewFS.denninger.net 11.0-STABLE FreeBSD 11.0-STABLE #13
>>>> r307318M: Fri Oct 14 09:23:46 CDT 2016
>>>> karl at NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP  amd64
>>>>
>>>> panic: double fault
>>>>
>>>> GNU gdb 6.1.1 [FreeBSD]
>>>> Copyright 2004 Free Software Foundation, Inc.
>>>> GDB is free software, covered by the GNU General Public
License, and
>>>> you are
>>>> welcome to change it and/or distribute copies of it under
certain
>>>> conditions.
>>>> Type "show copying" to see the conditions.
>>>> There is absolutely no warranty for GDB.  Type "show
warranty" for
>>>> details.
>>>> This GDB was configured as "amd64-marcel-freebsd"...
>>>>
>>>> Unread portion of the kernel message buffer:
>>>>
>>>> Fatal double fault
>>>> rip = 0xffffffff8220d9ec
>>>> rsp = 0xfffffe066821f000
>>>> rbp = 0xfffffe066821f020
>>>> cpuid = 6; apic id = 14
>>>> panic: double fault
>>>> cpuid = 6
>>>> KDB: stack backtrace:
>>>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
>>>> 0xfffffe0649d78e30
>>>> vpanic() at vpanic+0x182/frame 0xfffffe0649d78eb0
>>>> panic() at panic+0x43/frame 0xfffffe0649d78f10
>>>> dblfault_handler() at dblfault_handler+0xa2/frame
0xfffffe0649d78f30
>>>> Xdblfault() at Xdblfault+0xac/frame 0xfffffe0649d78f30
>>>> --- trap 0x17, rip = 0xffffffff8220d9ec, rsp =
0xfffffe066821f000,
>>>> rbp >>>> 0xfffffe066821f020 ---
>>>> avl_rotation() at avl_rotation+0xc/frame 0xfffffe066821f020
>>>> avl_remove() at avl_remove+0x1c8/frame 0xfffffe066821f070
>>>> vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x87f/frame
>>>> 0xfffffe066821f530
>>>> vdev_queue_io_done() at vdev_queue_io_done+0x83/frame
>>>> 0xfffffe066821f570
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe066821f5a0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f5f0
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe066821f650
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f6a0
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe066821f6e0
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe066821f710
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f760
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe066821f7c0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f810
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe066821f850
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe066821f880
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f8d0
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe066821f930
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f980
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe066821f9c0
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe066821f9f0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fa40
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe066821faa0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821faf0
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe066821fb30
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe066821fb60
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fbb0
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe066821fc10
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fc60
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe066821fca0
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe066821fcd0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fd20
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe066821fd80
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fdd0
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe066821fe10
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe066821fe40
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fe90
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe066821fef0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821ff40
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe066821ff80
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe066821ffb0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220000
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668220060
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682200b0
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe06682200f0
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668220120
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220170
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe06682201d0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220220
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668220260
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668220290
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682202e0
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668220340
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220390
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe06682203d0
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668220400
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220450
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe06682204b0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220500
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668220540
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668220570
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682205c0
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668220620
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220670
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe06682206b0
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe06682206e0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220730
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668220790
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682207e0
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668220820
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668220850
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682208a0
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668220900
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220950
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668220990
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe06682209c0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220a10
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668220a70
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220ac0
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668220b00
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668220b30
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220b80
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668220be0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220c30
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668220c70
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668220ca0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220cf0
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668220d50
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220da0
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668220de0
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668220e10
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220e60
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668220ec0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220f10
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668220f50
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668220f80
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220fd0
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668221030
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221080
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe06682210c0
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe06682210f0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221140
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe06682211a0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682211f0
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668221230
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668221260
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682212b0
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668221310
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221360
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe06682213a0
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe06682213d0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221420
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668221480
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682214d0
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668221510
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668221540
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221590
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe06682215f0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221640
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668221680
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe06682216b0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221700
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668221760
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682217b0
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe06682217f0
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668221820
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221870
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe06682218d0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221920
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668221960
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668221990
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682219e0
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668221a40
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221a90
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668221ad0
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668221b00
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221b50
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668221bb0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221c00
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668221c40
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668221c70
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221cc0
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668221d20
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221d70
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668221db0
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668221de0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221e30
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668221e90
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221ee0
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668221f20
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668221f50
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221fa0
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668222000
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222050
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668222090
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe06682220c0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222110
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668222170
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682221c0
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668222200
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668222230
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222280
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe06682222e0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222330
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668222370
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe06682223a0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682223f0
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668222450
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682224a0
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe06682224e0
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668222510
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222560
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe06682225c0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222610
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668222650
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668222680
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682226d0
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668222730
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222780
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe06682227c0
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe06682227f0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222840
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe06682228a0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682228f0
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668222930
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668222960
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682229b0
>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>> 0xfffffe0668222a10
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222a60
>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>> 0xfffffe0668222aa0
>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668222ad0
>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222b20
>>>> taskqueue_run_locked() at taskqueue_run_locked+0x14a/frame
>>>> 0xfffffe0668222b80
>>>> taskqueue_thread_loop() at taskqueue_thread_loop+0xe8/frame
>>>> 0xfffffe0668222bb0
>>>> fork_exit() at fork_exit+0x85/frame 0xfffffe0668222bf0
>>>> fork_trampoline() at fork_trampoline+0xe/frame
0xfffffe0668222bf0
>>>> --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
>>>> Uptime: 4m29s
>>>> Dumping 1497 out of 24531
>>>> MB:..2%..11%..21%..31%..41%..51%..61%..71%..81%..91%
>>>>
>>>> Reading symbols from /boot/kernel/zfs.ko...Reading symbols from
>>>> /usr/lib/debug//boot/kernel/zfs.ko.debug...done.
>>>> done.
>>>> Loaded symbols for /boot/kernel/zfs.ko
>>>> Reading symbols from /boot/kernel/opensolaris.ko...Reading
symbols
>>>> from
>>>> /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done.
>>>> done.
>>>> Loaded symbols for /boot/kernel/opensolaris.ko
>>>> Reading symbols from /boot/kernel/geom_eli.ko...Reading symbols
from
>>>> /usr/lib/debug//boot/kernel/geom_eli.ko.debug...done.
>>>> done.
>>>> Loaded symbols for /boot/kernel/geom_eli.ko
>>>> Reading symbols from /boot/kernel/geom_mirror.ko...Reading
symbols
>>>> from
>>>> /usr/lib/debug//boot/kernel/geom_mirror.ko.debug...done.
>>>> done.
>>>> Loaded symbols for /boot/kernel/geom_mirror.ko
>>>> Reading symbols from /boot/kernel/aesni.ko...Reading symbols
from
>>>> /usr/lib/debug//boot/kernel/aesni.ko.debug...done.
>>>> done.
>>>> Loaded symbols for /boot/kernel/aesni.ko
>>>> Reading symbols from /boot/kernel/uhid.ko...Reading symbols
from
>>>> /usr/lib/debug//boot/kernel/uhid.ko.debug...done.
>>>> done.
>>>> Loaded symbols for /boot/kernel/uhid.ko
>>>> Reading symbols from /boot/kernel/ums.ko...Reading symbols from
>>>> /usr/lib/debug//boot/kernel/ums.ko.debug...done.
>>>> done.
>>>> Loaded symbols for /boot/kernel/ums.ko
>>>> Reading symbols from /boot/kernel/ipfw.ko...Reading symbols
from
>>>> /usr/lib/debug//boot/kernel/ipfw.ko.debug...done.
>>>> done.
>>>> Loaded symbols for /boot/kernel/ipfw.ko
>>>> Reading symbols from /boot/kernel/ipfw_nat.ko...Reading symbols
from
>>>> /usr/lib/debug//boot/kernel/ipfw_nat.ko.debug...done.
>>>> done.
>>>> Loaded symbols for /boot/kernel/ipfw_nat.ko
>>>> Reading symbols from /boot/kernel/libalias.ko...Reading symbols
from
>>>> /usr/lib/debug//boot/kernel/libalias.ko.debug...done.
>>>> done.
>>>> Loaded symbols for /boot/kernel/libalias.ko
>>>> Reading symbols from /boot/kernel/daemon_saver.ko...Reading
symbols
>>>> from
>>>> /usr/lib/debug//boot/kernel/daemon_saver.ko.debug...done.
>>>> done.
>>>> Loaded symbols for /boot/kernel/daemon_saver.ko
>>>> #0  doadump (textdump=1) at pcpu.h:221
>>>> 221     pcpu.h: No such file or directory.
>>>>           in pcpu.h
>>>> (kgdb) #0  doadump (textdump=1) at pcpu.h:221
>>>> #1  0xffffffff80ae9b6e in kern_reboot (howto=260)
>>>>       at /usr/src/sys/kern/kern_shutdown.c:366
>>>> #2  0xffffffff80aea13b in vpanic (fmt=<value optimized
out>,
>>>>       ap=<value optimized out>) at
>>>> /usr/src/sys/kern/kern_shutdown.c:759
>>>> #3  0xffffffff80ae9f73 in panic (fmt=0x0)
>>>>       at /usr/src/sys/kern/kern_shutdown.c:690
>>>> #4  0xffffffff80fd18f2 in dblfault_handler (frame=<value
optimized
>>>> out>)
>>>>       at /usr/src/sys/amd64/amd64/trap.c:868
>>>> #5  0xffffffff80fb406c in Xdblfault ()
>>>>       at /usr/src/sys/amd64/amd64/exception.S:292
>>>> #6  0xffffffff8220d9ec in avl_rotation
(tree=0xfffff8004dc54510,
>>>>       node=0xfffff8056e45cdd0, balance=2)
>>>>       at
/usr/src/sys/cddl/contrib/opensolaris/common/avl/avl.c:304
>>>> #7  0xffffffff8220de58 in avl_remove (tree=<value optimized
out>,
>>>>       data=<value optimized out>)
>>>>       at
/usr/src/sys/cddl/contrib/opensolaris/common/avl/avl.c:818
>>>> #8  0xffffffff822a1eff in vdev_queue_io_to_issue (vq=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:439
>>>>
>>>> #9  0xffffffff822a2123 in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:907
>>>>
>>>> #10 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e3f33d8)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #11 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #12 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e3f33d8)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #13 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #14 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>
>>>> #15 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e578000)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #16 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #17 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e578000)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #18 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #19 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>
>>>> #20 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e5ec3d8)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #21 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #22 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e5ec3d8)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #23 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #24 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>
>>>> #25 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e48a000)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #26 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #27 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e48a000)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #28 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #29 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>
>>>> #30 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e57ab88)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #31 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #32 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e57ab88)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #33 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #34 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>
>>>> #35 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff800a69a3000)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #36 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #37 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff800a69a3000)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #38 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #39 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>
>>>> #40 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e5c8000)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #41 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #42 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e5c8000)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #43 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #44 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>
>>>> #45 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff802a23f9000)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #46 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #47 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff802a23f9000)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #48 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #49 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>
>>>> #50 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e5147b0)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #51 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #52 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e5147b0)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #53 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #54 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>
>>>> #55 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e977000)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #56 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #57 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e977000)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #58 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #59 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>
>>>> #60 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e90b3d8)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #61 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #62 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e90b3d8)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #63 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #64 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>
>>>> #65 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff800b1160b88)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #66 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #67 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff800b1160b88)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #68 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #69 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>
>>>> #70 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e4f53d8)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #71 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #72 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e4f53d8)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #73 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #74 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>
>>>> #75 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e8ebb88)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #76 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #77 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e8ebb88)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #78 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #79 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>
>>>> #80 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e8d9b88)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #81 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #82 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e8d9b88)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #83 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #84 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>
>>>> #85 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff802a21a03d8)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #86 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #87 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff802a21a03d8)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #88 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #89 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>
>>>> #90 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e6903d8)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #91 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #92 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e6903d8)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #93 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #94 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>
>>>> #95 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff802a22dc7b0)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #96 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #97 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff802a22dc7b0)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #98 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #99 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>
>>>> #100 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e5fb7b0)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #101 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #102 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e5fb7b0)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #103 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #104 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>
>>>> #105 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e5ae3d8)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #106 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #107 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e5ae3d8)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #108 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #109 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>
>>>> #110 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff802a2131b88)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #111 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #112 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff802a2131b88)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #113 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #114 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>
>>>> #115 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e80b7b0)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #116 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #117 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e80b7b0)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #118 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #119 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>
>>>> #120 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff800b14b9000)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #121 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #122 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff800b14b9000)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #123 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #124 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>
>>>> #125 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff802a21fa3d8)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #126 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #127 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff802a21fa3d8)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #128 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #129 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>
>>>> #130 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e5333d8)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #131 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #132 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e5333d8)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #133 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #134 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>
>>>> #135 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e90ab88)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #136 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #137 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e90ab88)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #138 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #139 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>
>>>> #140 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e4047b0)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #141 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #142 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e4047b0)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #143 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #144 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>
>>>> #145 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e7083d8)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #146 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #147 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e7083d8)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #148 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #149 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>
>>>> #150 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff802a23917b0)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #151 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #152 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff802a23917b0)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #153 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #154 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>
>>>> #155 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e636000)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #156 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #157 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e636000)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #158 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #159 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>
>>>> #160 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e5afb88)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #161 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #162 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e5afb88)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #163 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #164 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>
>>>> #165 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff800a69523d8)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #166 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #167 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff800a69523d8)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #168 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #169 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>
>>>> #170 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e47c000)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #171 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #172 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e47c000)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #173 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #174 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>
>>>> #175 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff800b17b23d8)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #176 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #177 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff800b17b23d8)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #178 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #179 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>
>>>> #180 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff800a6d367b0)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #181 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #182 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff800a6d367b0)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #183 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #184 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>
>>>> #185 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e99fb88)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #186 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #187 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e99fb88)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #188 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #189 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>
>>>> #190 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e5227b0)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #191 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #192 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e5227b0)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>> #193 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #194 0xffffffff822a216d in vdev_queue_io_done (zio=<value
optimized
>>>> out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>
>>>> #195 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff800a6d3c000)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>> #196 0xffffffff822bbefd in zio_execute (zio=<value optimized
out>)
>>>>       at
>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>> #197 0xffffffff80b4895a in taskqueue_run_locked
(queue=<value
>>>> optimized
>>>> out>)
>>>>       at /usr/src/sys/kern/subr_taskqueue.c:454
>>>> #198 0xffffffff80b49b58 in taskqueue_thread_loop (arg=<value
>>>> optimized out>)
>>>>       at /usr/src/sys/kern/subr_taskqueue.c:724
>>>> #199 0xffffffff80a9f255 in fork_exit (
>>>>       callout=0xffffffff80b49a70 <taskqueue_thread_loop>,
>>>>       arg=0xfffff8057caa72a0, frame=0xfffffe0668222c00)
>>>>       at /usr/src/sys/kern/kern_fork.c:1040
>>>> #200 0xffffffff80fb44ae in fork_trampoline ()
>>>>       at /usr/src/sys/amd64/amd64/exception.S:611
>>>> #201 0x0000000000000000 in ?? ()
>>>> Current language:  auto; currently minimal
>>>> (kgdb)
>>>>
>>>>
>>>>>> -- 
>>>>>> Karl Denninger
>>>>>> karl at denninger.net <mailto:karl at
denninger.net>
>>>>>> /The Market Ticker/
>>>>>> /[S/MIME encrypted email preferred]/
>>> _______________________________________________
>>> freebsd-stable at freebsd.org mailing list
>>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
>>> To unsubscribe, send any mail to
>>> "freebsd-stable-unsubscribe at freebsd.org"
>
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at
freebsd.org"
-- 
Karl Denninger
karl at denninger.net <mailto:karl at denninger.net>
/The Market Ticker/
/[S/MIME encrypted email preferred]/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2996 bytes
Desc: S/MIME Cryptographic Signature
URL:
<http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20161017/2099ab18/attachment-0001.bin>