Steven Hartland
2016-Oct-17 18:54 UTC
Repeatable panic on ZFS filesystem (used for backups); 11.0-STABLE
You're hitting stack exhaustion, have you tried increasing the kernel stack pages? It can be changed from /boot/loader.conf kern.kstack_pages="6" Default on amd64 is 4 IIRC On 17/10/2016 19:08, Karl Denninger wrote:> The target (and devices that trigger this) are a pair of 4Gb 7200RPM > SATA rotating rust drives (zmirror) with each provider geli-encrypted > (that is, the actual devices used for the pool create are the .eli's) > > The machine generating the problem has both rotating rust devices *and* > SSDs, so I can't simply shut TRIM off system-wide and call it a day as > TRIM itself is heavily-used; both the boot/root pools and a Postgresql > database pool are on SSDs, while several terabytes of lesser-used data > is on a pool of Raidz2 that is made up of spinning rust.snip...> > NewFS.denninger.net dumped core - see /var/crash/vmcore.1 > > Mon Oct 17 09:02:33 CDT 2016 > > FreeBSD NewFS.denninger.net 11.0-STABLE FreeBSD 11.0-STABLE #13 > r307318M: Fri Oct 14 09:23:46 CDT 2016 > karl at NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP amd64 > > panic: double fault > > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain > conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "amd64-marcel-freebsd"... > > Unread portion of the kernel message buffer: > > Fatal double fault > rip = 0xffffffff8220d9ec > rsp = 0xfffffe066821f000 > rbp = 0xfffffe066821f020 > cpuid = 6; apic id = 14 > panic: double fault > cpuid = 6 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame > 0xfffffe0649d78e30 > vpanic() at vpanic+0x182/frame 0xfffffe0649d78eb0 > panic() at panic+0x43/frame 0xfffffe0649d78f10 > dblfault_handler() at dblfault_handler+0xa2/frame 0xfffffe0649d78f30 > Xdblfault() at Xdblfault+0xac/frame 0xfffffe0649d78f30 > --- trap 0x17, rip = 0xffffffff8220d9ec, rsp = 0xfffffe066821f000, rbp > 0xfffffe066821f020 --- > avl_rotation() at avl_rotation+0xc/frame 0xfffffe066821f020 > avl_remove() at avl_remove+0x1c8/frame 0xfffffe066821f070 > vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x87f/frame > 0xfffffe066821f530 > vdev_queue_io_done() at vdev_queue_io_done+0x83/frame 0xfffffe066821f570 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f5a0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f5f0 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821f650 > zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f6a0 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821f6e0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f710 > zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f760 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821f7c0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f810 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821f850 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f880 > zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f8d0 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821f930 > zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f980 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821f9c0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f9f0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fa40 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821faa0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe066821faf0 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821fb30 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fb60 > zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fbb0 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821fc10 > zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fc60 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821fca0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fcd0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fd20 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821fd80 > zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fdd0 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821fe10 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fe40 > zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fe90 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821fef0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe066821ff40 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821ff80 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821ffb0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220000 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220060 > zio_execute() at zio_execute+0x23d/frame 0xfffffe06682200b0 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682200f0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220120 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220170 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682201d0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220220 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220260 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220290 > zio_execute() at zio_execute+0x23d/frame 0xfffffe06682202e0 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220340 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220390 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682203d0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220400 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220450 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682204b0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220500 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220540 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220570 > zio_execute() at zio_execute+0x23d/frame 0xfffffe06682205c0 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220620 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220670 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682206b0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682206e0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220730 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220790 > zio_execute() at zio_execute+0x23d/frame 0xfffffe06682207e0 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220820 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220850 > zio_execute() at zio_execute+0x23d/frame 0xfffffe06682208a0 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220900 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220950 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220990 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682209c0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220a10 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220a70 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220ac0 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220b00 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220b30 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220b80 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220be0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220c30 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220c70 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220ca0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220cf0 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220d50 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220da0 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220de0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220e10 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220e60 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220ec0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220f10 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220f50 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220f80 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220fd0 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221030 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221080 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682210c0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682210f0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221140 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682211a0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe06682211f0 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221230 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221260 > zio_execute() at zio_execute+0x23d/frame 0xfffffe06682212b0 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221310 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221360 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682213a0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682213d0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221420 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221480 > zio_execute() at zio_execute+0x23d/frame 0xfffffe06682214d0 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221510 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221540 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221590 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682215f0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221640 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221680 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682216b0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221700 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221760 > zio_execute() at zio_execute+0x23d/frame 0xfffffe06682217b0 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682217f0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221820 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221870 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682218d0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221920 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221960 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221990 > zio_execute() at zio_execute+0x23d/frame 0xfffffe06682219e0 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221a40 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221a90 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221ad0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221b00 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221b50 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221bb0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221c00 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221c40 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221c70 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221cc0 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221d20 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221d70 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221db0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221de0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221e30 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221e90 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221ee0 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221f20 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221f50 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221fa0 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222000 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222050 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222090 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682220c0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222110 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222170 > zio_execute() at zio_execute+0x23d/frame 0xfffffe06682221c0 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222200 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222230 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222280 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682222e0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222330 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222370 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682223a0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe06682223f0 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222450 > zio_execute() at zio_execute+0x23d/frame 0xfffffe06682224a0 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682224e0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222510 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222560 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682225c0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222610 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222650 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222680 > zio_execute() at zio_execute+0x23d/frame 0xfffffe06682226d0 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222730 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222780 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682227c0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682227f0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222840 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682228a0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe06682228f0 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222930 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222960 > zio_execute() at zio_execute+0x23d/frame 0xfffffe06682229b0 > zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222a10 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222a60 > vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222aa0 > zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222ad0 > zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222b20 > taskqueue_run_locked() at taskqueue_run_locked+0x14a/frame > 0xfffffe0668222b80 > taskqueue_thread_loop() at taskqueue_thread_loop+0xe8/frame > 0xfffffe0668222bb0 > fork_exit() at fork_exit+0x85/frame 0xfffffe0668222bf0 > fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0668222bf0 > --- trap 0, rip = 0, rsp = 0, rbp = 0 --- > Uptime: 4m29s > Dumping 1497 out of 24531 > MB:..2%..11%..21%..31%..41%..51%..61%..71%..81%..91% > > Reading symbols from /boot/kernel/zfs.ko...Reading symbols from > /usr/lib/debug//boot/kernel/zfs.ko.debug...done. > done. > Loaded symbols for /boot/kernel/zfs.ko > Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from > /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done. > done. > Loaded symbols for /boot/kernel/opensolaris.ko > Reading symbols from /boot/kernel/geom_eli.ko...Reading symbols from > /usr/lib/debug//boot/kernel/geom_eli.ko.debug...done. > done. > Loaded symbols for /boot/kernel/geom_eli.ko > Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from > /usr/lib/debug//boot/kernel/geom_mirror.ko.debug...done. > done. > Loaded symbols for /boot/kernel/geom_mirror.ko > Reading symbols from /boot/kernel/aesni.ko...Reading symbols from > /usr/lib/debug//boot/kernel/aesni.ko.debug...done. > done. > Loaded symbols for /boot/kernel/aesni.ko > Reading symbols from /boot/kernel/uhid.ko...Reading symbols from > /usr/lib/debug//boot/kernel/uhid.ko.debug...done. > done. > Loaded symbols for /boot/kernel/uhid.ko > Reading symbols from /boot/kernel/ums.ko...Reading symbols from > /usr/lib/debug//boot/kernel/ums.ko.debug...done. > done. > Loaded symbols for /boot/kernel/ums.ko > Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from > /usr/lib/debug//boot/kernel/ipfw.ko.debug...done. > done. > Loaded symbols for /boot/kernel/ipfw.ko > Reading symbols from /boot/kernel/ipfw_nat.ko...Reading symbols from > /usr/lib/debug//boot/kernel/ipfw_nat.ko.debug...done. > done. > Loaded symbols for /boot/kernel/ipfw_nat.ko > Reading symbols from /boot/kernel/libalias.ko...Reading symbols from > /usr/lib/debug//boot/kernel/libalias.ko.debug...done. > done. > Loaded symbols for /boot/kernel/libalias.ko > Reading symbols from /boot/kernel/daemon_saver.ko...Reading symbols from > /usr/lib/debug//boot/kernel/daemon_saver.ko.debug...done. > done. > Loaded symbols for /boot/kernel/daemon_saver.ko > #0 doadump (textdump=1) at pcpu.h:221 > 221 pcpu.h: No such file or directory. > in pcpu.h > (kgdb) #0 doadump (textdump=1) at pcpu.h:221 > #1 0xffffffff80ae9b6e in kern_reboot (howto=260) > at /usr/src/sys/kern/kern_shutdown.c:366 > #2 0xffffffff80aea13b in vpanic (fmt=<value optimized out>, > ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759 > #3 0xffffffff80ae9f73 in panic (fmt=0x0) > at /usr/src/sys/kern/kern_shutdown.c:690 > #4 0xffffffff80fd18f2 in dblfault_handler (frame=<value optimized out>) > at /usr/src/sys/amd64/amd64/trap.c:868 > #5 0xffffffff80fb406c in Xdblfault () > at /usr/src/sys/amd64/amd64/exception.S:292 > #6 0xffffffff8220d9ec in avl_rotation (tree=0xfffff8004dc54510, > node=0xfffff8056e45cdd0, balance=2) > at /usr/src/sys/cddl/contrib/opensolaris/common/avl/avl.c:304 > #7 0xffffffff8220de58 in avl_remove (tree=<value optimized out>, > data=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/common/avl/avl.c:818 > #8 0xffffffff822a1eff in vdev_queue_io_to_issue (vq=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:439 > #9 0xffffffff822a2123 in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:907 > #10 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e3f33d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #11 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #12 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e3f33d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 > #13 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #14 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 > #15 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e578000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #16 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #17 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e578000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 > #18 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #19 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 > #20 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5ec3d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #21 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #22 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5ec3d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 > #23 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #24 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 > #25 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e48a000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #26 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #27 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e48a000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 > #28 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #29 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 > #30 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e57ab88) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #31 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #32 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e57ab88) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 > #33 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #34 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 > #35 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800a69a3000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #36 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #37 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800a69a3000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 > #38 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #39 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 > #40 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5c8000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #41 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #42 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5c8000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 > #43 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #44 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 > #45 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a23f9000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #46 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #47 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a23f9000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 > #48 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #49 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 > #50 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5147b0) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #51 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #52 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5147b0) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 > #53 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #54 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 > #55 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e977000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #56 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #57 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e977000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 > #58 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #59 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 > #60 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e90b3d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #61 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #62 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e90b3d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 > #63 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #64 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 > #65 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800b1160b88) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #66 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #67 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800b1160b88) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 > #68 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #69 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 > #70 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e4f53d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #71 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #72 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e4f53d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 > #73 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #74 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 > #75 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e8ebb88) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #76 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #77 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e8ebb88) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 > #78 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #79 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 > #80 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e8d9b88) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #81 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #82 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e8d9b88) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 > #83 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #84 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 > #85 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a21a03d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #86 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #87 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a21a03d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 > #88 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #89 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 > #90 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e6903d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #91 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #92 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e6903d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 > #93 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #94 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 > #95 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a22dc7b0) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #96 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #97 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a22dc7b0) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 > #98 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #99 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 > #100 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5fb7b0) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #101 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #102 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5fb7b0) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 > #103 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #104 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 > #105 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5ae3d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #106 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #107 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5ae3d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 > #108 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #109 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 > #110 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a2131b88) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #111 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #112 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a2131b88) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 > #113 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #114 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 > #115 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e80b7b0) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #116 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #117 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e80b7b0) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 > #118 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #119 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 > #120 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800b14b9000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #121 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #122 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800b14b9000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 > #123 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #124 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 > #125 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a21fa3d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #126 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #127 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a21fa3d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 > #128 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #129 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 > #130 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5333d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #131 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #132 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5333d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 > #133 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #134 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 > #135 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e90ab88) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #136 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #137 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e90ab88) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 > #138 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #139 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 > #140 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e4047b0) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #141 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #142 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e4047b0) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 > #143 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #144 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 > #145 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e7083d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #146 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #147 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e7083d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 > #148 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #149 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 > #150 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a23917b0) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #151 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #152 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a23917b0) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 > #153 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #154 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 > #155 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e636000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #156 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #157 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e636000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 > #158 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #159 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 > #160 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5afb88) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #161 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #162 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5afb88) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 > #163 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #164 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 > #165 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800a69523d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #166 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #167 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800a69523d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 > #168 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #169 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 > #170 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e47c000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #171 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #172 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e47c000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 > #173 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #174 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 > #175 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800b17b23d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #176 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #177 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800b17b23d8) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 > #178 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #179 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 > #180 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800a6d367b0) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #181 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #182 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800a6d367b0) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 > #183 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #184 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 > #185 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e99fb88) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #186 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #187 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e99fb88) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 > #188 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #189 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 > #190 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5227b0) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #191 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #192 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5227b0) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 > #193 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #194 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) > at > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 > #195 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800a6d3c000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 > #196 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 > #197 0xffffffff80b4895a in taskqueue_run_locked (queue=<value optimized > out>) > at /usr/src/sys/kern/subr_taskqueue.c:454 > #198 0xffffffff80b49b58 in taskqueue_thread_loop (arg=<value optimized out>) > at /usr/src/sys/kern/subr_taskqueue.c:724 > #199 0xffffffff80a9f255 in fork_exit ( > callout=0xffffffff80b49a70 <taskqueue_thread_loop>, > arg=0xfffff8057caa72a0, frame=0xfffffe0668222c00) > at /usr/src/sys/kern/kern_fork.c:1040 > #200 0xffffffff80fb44ae in fork_trampoline () > at /usr/src/sys/amd64/amd64/exception.S:611 > #201 0x0000000000000000 in ?? () > Current language: auto; currently minimal > (kgdb) > > >>> -- >>> Karl Denninger >>> karl at denninger.net <mailto:karl at denninger.net> >>> /The Market Ticker/ >>> /[S/MIME encrypted email preferred]/
Karl Denninger
2016-Oct-17 18:58 UTC
Repeatable panic on ZFS filesystem (used for backups); 11.0-STABLE
I can certainly attempt setting that higher.... but is that not just hiding the problem rather than addressing it? On 10/17/2016 13:54, Steven Hartland wrote:> You're hitting stack exhaustion, have you tried increasing the kernel > stack pages? > It can be changed from /boot/loader.conf > kern.kstack_pages="6" > > Default on amd64 is 4 IIRC > > On 17/10/2016 19:08, Karl Denninger wrote: >> The target (and devices that trigger this) are a pair of 4Gb 7200RPM >> SATA rotating rust drives (zmirror) with each provider geli-encrypted >> (that is, the actual devices used for the pool create are the .eli's) >> >> The machine generating the problem has both rotating rust devices *and* >> SSDs, so I can't simply shut TRIM off system-wide and call it a day as >> TRIM itself is heavily-used; both the boot/root pools and a Postgresql >> database pool are on SSDs, while several terabytes of lesser-used data >> is on a pool of Raidz2 that is made up of spinning rust. > snip... >> >> NewFS.denninger.net dumped core - see /var/crash/vmcore.1 >> >> Mon Oct 17 09:02:33 CDT 2016 >> >> FreeBSD NewFS.denninger.net 11.0-STABLE FreeBSD 11.0-STABLE #13 >> r307318M: Fri Oct 14 09:23:46 CDT 2016 >> karl at NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP amd64 >> >> panic: double fault >> >> GNU gdb 6.1.1 [FreeBSD] >> Copyright 2004 Free Software Foundation, Inc. >> GDB is free software, covered by the GNU General Public License, and >> you are >> welcome to change it and/or distribute copies of it under certain >> conditions. >> Type "show copying" to see the conditions. >> There is absolutely no warranty for GDB. Type "show warranty" for >> details. >> This GDB was configured as "amd64-marcel-freebsd"... >> >> Unread portion of the kernel message buffer: >> >> Fatal double fault >> rip = 0xffffffff8220d9ec >> rsp = 0xfffffe066821f000 >> rbp = 0xfffffe066821f020 >> cpuid = 6; apic id = 14 >> panic: double fault >> cpuid = 6 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame >> 0xfffffe0649d78e30 >> vpanic() at vpanic+0x182/frame 0xfffffe0649d78eb0 >> panic() at panic+0x43/frame 0xfffffe0649d78f10 >> dblfault_handler() at dblfault_handler+0xa2/frame 0xfffffe0649d78f30 >> Xdblfault() at Xdblfault+0xac/frame 0xfffffe0649d78f30 >> --- trap 0x17, rip = 0xffffffff8220d9ec, rsp = 0xfffffe066821f000, rbp >> 0xfffffe066821f020 --- >> avl_rotation() at avl_rotation+0xc/frame 0xfffffe066821f020 >> avl_remove() at avl_remove+0x1c8/frame 0xfffffe066821f070 >> vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x87f/frame >> 0xfffffe066821f530 >> vdev_queue_io_done() at vdev_queue_io_done+0x83/frame 0xfffffe066821f570 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f5a0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f5f0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821f650 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f6a0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821f6e0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f710 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f760 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821f7c0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f810 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821f850 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f880 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f8d0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821f930 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f980 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821f9c0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f9f0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fa40 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821faa0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821faf0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821fb30 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fb60 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fbb0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821fc10 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fc60 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821fca0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fcd0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fd20 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821fd80 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fdd0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821fe10 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fe40 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fe90 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821fef0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821ff40 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821ff80 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821ffb0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220000 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220060 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682200b0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682200f0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220120 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220170 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682201d0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220220 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220260 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220290 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682202e0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220340 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220390 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682203d0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220400 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220450 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682204b0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220500 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220540 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220570 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682205c0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220620 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220670 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682206b0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682206e0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220730 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220790 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682207e0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220820 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220850 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682208a0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220900 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220950 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220990 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682209c0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220a10 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220a70 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220ac0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220b00 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220b30 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220b80 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220be0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220c30 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220c70 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220ca0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220cf0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220d50 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220da0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220de0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220e10 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220e60 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220ec0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220f10 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220f50 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220f80 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220fd0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221030 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221080 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682210c0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682210f0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221140 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682211a0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682211f0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221230 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221260 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682212b0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221310 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221360 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682213a0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682213d0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221420 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221480 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682214d0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221510 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221540 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221590 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682215f0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221640 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221680 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682216b0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221700 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221760 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682217b0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682217f0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221820 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221870 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682218d0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221920 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221960 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221990 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682219e0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221a40 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221a90 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221ad0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221b00 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221b50 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221bb0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221c00 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221c40 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221c70 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221cc0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221d20 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221d70 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221db0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221de0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221e30 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221e90 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221ee0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221f20 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221f50 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221fa0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222000 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222050 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222090 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682220c0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222110 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222170 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682221c0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222200 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222230 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222280 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682222e0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222330 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222370 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682223a0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682223f0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222450 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682224a0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682224e0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222510 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222560 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682225c0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222610 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222650 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222680 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682226d0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222730 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222780 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682227c0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682227f0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222840 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682228a0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682228f0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222930 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222960 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682229b0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222a10 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222a60 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222aa0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222ad0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222b20 >> taskqueue_run_locked() at taskqueue_run_locked+0x14a/frame >> 0xfffffe0668222b80 >> taskqueue_thread_loop() at taskqueue_thread_loop+0xe8/frame >> 0xfffffe0668222bb0 >> fork_exit() at fork_exit+0x85/frame 0xfffffe0668222bf0 >> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0668222bf0 >> --- trap 0, rip = 0, rsp = 0, rbp = 0 --- >> Uptime: 4m29s >> Dumping 1497 out of 24531 >> MB:..2%..11%..21%..31%..41%..51%..61%..71%..81%..91% >> >> Reading symbols from /boot/kernel/zfs.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/zfs.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/zfs.ko >> Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/opensolaris.ko >> Reading symbols from /boot/kernel/geom_eli.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/geom_eli.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/geom_eli.ko >> Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/geom_mirror.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/geom_mirror.ko >> Reading symbols from /boot/kernel/aesni.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/aesni.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/aesni.ko >> Reading symbols from /boot/kernel/uhid.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/uhid.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/uhid.ko >> Reading symbols from /boot/kernel/ums.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/ums.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/ums.ko >> Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/ipfw.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/ipfw.ko >> Reading symbols from /boot/kernel/ipfw_nat.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/ipfw_nat.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/ipfw_nat.ko >> Reading symbols from /boot/kernel/libalias.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/libalias.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/libalias.ko >> Reading symbols from /boot/kernel/daemon_saver.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/daemon_saver.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/daemon_saver.ko >> #0 doadump (textdump=1) at pcpu.h:221 >> 221 pcpu.h: No such file or directory. >> in pcpu.h >> (kgdb) #0 doadump (textdump=1) at pcpu.h:221 >> #1 0xffffffff80ae9b6e in kern_reboot (howto=260) >> at /usr/src/sys/kern/kern_shutdown.c:366 >> #2 0xffffffff80aea13b in vpanic (fmt=<value optimized out>, >> ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759 >> #3 0xffffffff80ae9f73 in panic (fmt=0x0) >> at /usr/src/sys/kern/kern_shutdown.c:690 >> #4 0xffffffff80fd18f2 in dblfault_handler (frame=<value optimized out>) >> at /usr/src/sys/amd64/amd64/trap.c:868 >> #5 0xffffffff80fb406c in Xdblfault () >> at /usr/src/sys/amd64/amd64/exception.S:292 >> #6 0xffffffff8220d9ec in avl_rotation (tree=0xfffff8004dc54510, >> node=0xfffff8056e45cdd0, balance=2) >> at /usr/src/sys/cddl/contrib/opensolaris/common/avl/avl.c:304 >> #7 0xffffffff8220de58 in avl_remove (tree=<value optimized out>, >> data=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/common/avl/avl.c:818 >> #8 0xffffffff822a1eff in vdev_queue_io_to_issue (vq=<value optimized >> out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:439 >> #9 0xffffffff822a2123 in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:907 >> #10 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e3f33d8) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #11 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #12 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e3f33d8) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #13 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #14 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #15 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e578000) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #16 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #17 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e578000) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #18 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #19 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #20 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5ec3d8) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #21 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #22 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5ec3d8) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #23 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #24 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #25 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e48a000) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #26 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #27 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e48a000) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #28 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #29 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #30 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e57ab88) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #31 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #32 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e57ab88) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #33 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #34 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #35 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800a69a3000) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #36 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #37 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800a69a3000) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #38 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #39 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #40 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5c8000) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #41 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #42 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5c8000) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #43 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #44 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #45 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a23f9000) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #46 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #47 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a23f9000) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #48 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #49 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #50 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5147b0) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #51 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #52 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5147b0) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #53 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #54 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #55 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e977000) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #56 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #57 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e977000) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #58 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #59 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #60 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e90b3d8) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #61 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #62 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e90b3d8) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #63 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #64 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #65 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800b1160b88) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #66 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #67 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800b1160b88) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #68 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #69 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #70 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e4f53d8) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #71 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #72 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e4f53d8) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #73 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #74 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #75 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e8ebb88) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #76 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #77 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e8ebb88) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #78 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #79 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #80 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e8d9b88) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #81 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #82 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e8d9b88) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #83 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #84 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #85 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a21a03d8) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #86 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #87 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a21a03d8) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #88 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #89 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #90 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e6903d8) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #91 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #92 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e6903d8) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #93 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #94 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #95 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a22dc7b0) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #96 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #97 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a22dc7b0) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #98 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #99 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #100 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5fb7b0) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #101 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #102 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5fb7b0) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #103 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #104 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >> out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #105 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5ae3d8) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #106 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #107 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5ae3d8) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #108 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #109 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >> out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #110 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a2131b88) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #111 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #112 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a2131b88) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #113 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #114 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >> out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #115 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e80b7b0) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #116 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #117 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e80b7b0) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #118 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #119 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >> out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #120 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800b14b9000) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #121 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #122 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800b14b9000) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #123 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #124 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >> out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #125 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a21fa3d8) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #126 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #127 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a21fa3d8) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #128 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #129 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >> out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #130 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5333d8) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #131 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #132 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5333d8) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #133 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #134 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >> out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #135 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e90ab88) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #136 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #137 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e90ab88) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #138 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #139 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >> out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #140 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e4047b0) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #141 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #142 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e4047b0) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #143 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #144 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >> out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #145 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e7083d8) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #146 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #147 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e7083d8) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #148 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #149 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >> out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #150 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a23917b0) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #151 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #152 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a23917b0) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #153 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #154 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >> out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #155 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e636000) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #156 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #157 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e636000) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #158 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #159 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >> out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #160 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5afb88) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #161 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #162 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5afb88) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #163 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #164 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >> out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #165 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800a69523d8) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #166 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #167 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800a69523d8) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #168 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #169 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >> out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #170 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e47c000) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #171 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #172 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e47c000) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #173 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #174 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >> out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #175 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800b17b23d8) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #176 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #177 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800b17b23d8) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #178 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #179 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >> out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #180 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800a6d367b0) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #181 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #182 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800a6d367b0) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #183 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #184 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >> out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #185 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e99fb88) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #186 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #187 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e99fb88) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #188 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #189 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >> out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #190 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5227b0) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #191 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #192 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5227b0) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #193 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #194 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >> out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #195 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800a6d3c000) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #196 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #197 0xffffffff80b4895a in taskqueue_run_locked (queue=<value optimized >> out>) >> at /usr/src/sys/kern/subr_taskqueue.c:454 >> #198 0xffffffff80b49b58 in taskqueue_thread_loop (arg=<value >> optimized out>) >> at /usr/src/sys/kern/subr_taskqueue.c:724 >> #199 0xffffffff80a9f255 in fork_exit ( >> callout=0xffffffff80b49a70 <taskqueue_thread_loop>, >> arg=0xfffff8057caa72a0, frame=0xfffffe0668222c00) >> at /usr/src/sys/kern/kern_fork.c:1040 >> #200 0xffffffff80fb44ae in fork_trampoline () >> at /usr/src/sys/amd64/amd64/exception.S:611 >> #201 0x0000000000000000 in ?? () >> Current language: auto; currently minimal >> (kgdb) >> >> >>>> -- >>>> Karl Denninger >>>> karl at denninger.net <mailto:karl at denninger.net> >>>> /The Market Ticker/ >>>> /[S/MIME encrypted email preferred]/ > > _______________________________________________ > freebsd-stable at freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"-- Karl Denninger karl at denninger.net <mailto:karl at denninger.net> /The Market Ticker/ /[S/MIME encrypted email preferred]/ -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2996 bytes Desc: S/MIME Cryptographic Signature URL: <http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20161017/e2f500b2/attachment-0001.bin>
Andriy Gapon
2016-Oct-17 19:52 UTC
Repeatable panic on ZFS filesystem (used for backups); 11.0-STABLE
On 17/10/2016 21:54, Steven Hartland wrote:> You're hitting stack exhaustion, have you tried increasing the kernel stack pages? > It can be changed from /boot/loader.conf > kern.kstack_pages="6" > > Default on amd64 is 4 IIRCSteve, perhaps you can think of a more proper fix? :-) https://lists.freebsd.org/pipermail/freebsd-stable/2016-July/085047.html> On 17/10/2016 19:08, Karl Denninger wrote: >> The target (and devices that trigger this) are a pair of 4Gb 7200RPM >> SATA rotating rust drives (zmirror) with each provider geli-encrypted >> (that is, the actual devices used for the pool create are the .eli's) >> >> The machine generating the problem has both rotating rust devices *and* >> SSDs, so I can't simply shut TRIM off system-wide and call it a day as >> TRIM itself is heavily-used; both the boot/root pools and a Postgresql >> database pool are on SSDs, while several terabytes of lesser-used data >> is on a pool of Raidz2 that is made up of spinning rust. > snip... >> >> NewFS.denninger.net dumped core - see /var/crash/vmcore.1 >> >> Mon Oct 17 09:02:33 CDT 2016 >> >> FreeBSD NewFS.denninger.net 11.0-STABLE FreeBSD 11.0-STABLE #13 >> r307318M: Fri Oct 14 09:23:46 CDT 2016 >> karl at NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP amd64 >> >> panic: double fault >> >> GNU gdb 6.1.1 [FreeBSD] >> Copyright 2004 Free Software Foundation, Inc. >> GDB is free software, covered by the GNU General Public License, and you are >> welcome to change it and/or distribute copies of it under certain >> conditions. >> Type "show copying" to see the conditions. >> There is absolutely no warranty for GDB. Type "show warranty" for details. >> This GDB was configured as "amd64-marcel-freebsd"... >> >> Unread portion of the kernel message buffer: >> >> Fatal double fault >> rip = 0xffffffff8220d9ec >> rsp = 0xfffffe066821f000 >> rbp = 0xfffffe066821f020 >> cpuid = 6; apic id = 14 >> panic: double fault >> cpuid = 6 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame >> 0xfffffe0649d78e30 >> vpanic() at vpanic+0x182/frame 0xfffffe0649d78eb0 >> panic() at panic+0x43/frame 0xfffffe0649d78f10 >> dblfault_handler() at dblfault_handler+0xa2/frame 0xfffffe0649d78f30 >> Xdblfault() at Xdblfault+0xac/frame 0xfffffe0649d78f30 >> --- trap 0x17, rip = 0xffffffff8220d9ec, rsp = 0xfffffe066821f000, rbp >> 0xfffffe066821f020 --- >> avl_rotation() at avl_rotation+0xc/frame 0xfffffe066821f020 >> avl_remove() at avl_remove+0x1c8/frame 0xfffffe066821f070 >> vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x87f/frame >> 0xfffffe066821f530 >> vdev_queue_io_done() at vdev_queue_io_done+0x83/frame 0xfffffe066821f570 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f5a0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f5f0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821f650 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f6a0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821f6e0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f710 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f760 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821f7c0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f810 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821f850 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f880 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f8d0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821f930 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f980 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821f9c0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f9f0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fa40 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821faa0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821faf0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821fb30 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fb60 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fbb0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821fc10 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fc60 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821fca0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fcd0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fd20 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821fd80 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fdd0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821fe10 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fe40 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fe90 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821fef0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821ff40 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821ff80 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821ffb0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220000 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220060 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682200b0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682200f0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220120 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220170 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682201d0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220220 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220260 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220290 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682202e0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220340 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220390 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682203d0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220400 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220450 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682204b0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220500 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220540 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220570 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682205c0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220620 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220670 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682206b0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682206e0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220730 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220790 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682207e0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220820 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220850 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682208a0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220900 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220950 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220990 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682209c0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220a10 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220a70 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220ac0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220b00 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220b30 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220b80 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220be0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220c30 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220c70 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220ca0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220cf0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220d50 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220da0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220de0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220e10 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220e60 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220ec0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220f10 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220f50 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220f80 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220fd0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221030 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221080 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682210c0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682210f0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221140 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682211a0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682211f0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221230 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221260 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682212b0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221310 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221360 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682213a0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682213d0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221420 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221480 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682214d0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221510 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221540 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221590 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682215f0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221640 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221680 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682216b0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221700 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221760 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682217b0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682217f0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221820 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221870 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682218d0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221920 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221960 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221990 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682219e0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221a40 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221a90 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221ad0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221b00 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221b50 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221bb0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221c00 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221c40 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221c70 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221cc0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221d20 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221d70 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221db0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221de0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221e30 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221e90 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221ee0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221f20 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221f50 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221fa0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222000 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222050 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222090 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682220c0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222110 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222170 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682221c0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222200 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222230 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222280 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682222e0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222330 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222370 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682223a0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682223f0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222450 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682224a0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682224e0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222510 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222560 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682225c0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222610 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222650 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222680 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682226d0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222730 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222780 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe06682227c0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682227f0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222840 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682228a0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682228f0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222930 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222960 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682229b0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222a10 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222a60 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222aa0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222ad0 >> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222b20 >> taskqueue_run_locked() at taskqueue_run_locked+0x14a/frame >> 0xfffffe0668222b80 >> taskqueue_thread_loop() at taskqueue_thread_loop+0xe8/frame >> 0xfffffe0668222bb0 >> fork_exit() at fork_exit+0x85/frame 0xfffffe0668222bf0 >> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0668222bf0 >> --- trap 0, rip = 0, rsp = 0, rbp = 0 --- >> Uptime: 4m29s >> Dumping 1497 out of 24531 >> MB:..2%..11%..21%..31%..41%..51%..61%..71%..81%..91% >> >> Reading symbols from /boot/kernel/zfs.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/zfs.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/zfs.ko >> Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/opensolaris.ko >> Reading symbols from /boot/kernel/geom_eli.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/geom_eli.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/geom_eli.ko >> Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/geom_mirror.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/geom_mirror.ko >> Reading symbols from /boot/kernel/aesni.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/aesni.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/aesni.ko >> Reading symbols from /boot/kernel/uhid.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/uhid.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/uhid.ko >> Reading symbols from /boot/kernel/ums.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/ums.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/ums.ko >> Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/ipfw.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/ipfw.ko >> Reading symbols from /boot/kernel/ipfw_nat.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/ipfw_nat.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/ipfw_nat.ko >> Reading symbols from /boot/kernel/libalias.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/libalias.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/libalias.ko >> Reading symbols from /boot/kernel/daemon_saver.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/daemon_saver.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/daemon_saver.ko >> #0 doadump (textdump=1) at pcpu.h:221 >> 221 pcpu.h: No such file or directory. >> in pcpu.h >> (kgdb) #0 doadump (textdump=1) at pcpu.h:221 >> #1 0xffffffff80ae9b6e in kern_reboot (howto=260) >> at /usr/src/sys/kern/kern_shutdown.c:366 >> #2 0xffffffff80aea13b in vpanic (fmt=<value optimized out>, >> ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759 >> #3 0xffffffff80ae9f73 in panic (fmt=0x0) >> at /usr/src/sys/kern/kern_shutdown.c:690 >> #4 0xffffffff80fd18f2 in dblfault_handler (frame=<value optimized out>) >> at /usr/src/sys/amd64/amd64/trap.c:868 >> #5 0xffffffff80fb406c in Xdblfault () >> at /usr/src/sys/amd64/amd64/exception.S:292 >> #6 0xffffffff8220d9ec in avl_rotation (tree=0xfffff8004dc54510, >> node=0xfffff8056e45cdd0, balance=2) >> at /usr/src/sys/cddl/contrib/opensolaris/common/avl/avl.c:304 >> #7 0xffffffff8220de58 in avl_remove (tree=<value optimized out>, >> data=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/common/avl/avl.c:818 >> #8 0xffffffff822a1eff in vdev_queue_io_to_issue (vq=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:439 >> #9 0xffffffff822a2123 in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:907 >> #10 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e3f33d8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #11 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #12 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e3f33d8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #13 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #14 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #15 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e578000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #16 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #17 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e578000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #18 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #19 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #20 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5ec3d8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #21 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #22 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5ec3d8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #23 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #24 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #25 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e48a000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #26 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #27 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e48a000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #28 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #29 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #30 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e57ab88) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #31 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #32 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e57ab88) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #33 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #34 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #35 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800a69a3000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #36 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #37 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800a69a3000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #38 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #39 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #40 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5c8000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #41 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #42 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5c8000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #43 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #44 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #45 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a23f9000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #46 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #47 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a23f9000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #48 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #49 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #50 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5147b0) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #51 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #52 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5147b0) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #53 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #54 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #55 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e977000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #56 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #57 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e977000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #58 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #59 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #60 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e90b3d8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #61 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #62 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e90b3d8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #63 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #64 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #65 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800b1160b88) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #66 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #67 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800b1160b88) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #68 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #69 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #70 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e4f53d8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #71 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #72 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e4f53d8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #73 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #74 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #75 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e8ebb88) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #76 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #77 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e8ebb88) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #78 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #79 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #80 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e8d9b88) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #81 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #82 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e8d9b88) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #83 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #84 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #85 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a21a03d8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #86 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #87 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a21a03d8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #88 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #89 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #90 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e6903d8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #91 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #92 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e6903d8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #93 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #94 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #95 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a22dc7b0) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #96 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #97 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a22dc7b0) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #98 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #99 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #100 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5fb7b0) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #101 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #102 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5fb7b0) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #103 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #104 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #105 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5ae3d8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #106 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #107 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5ae3d8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #108 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #109 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #110 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a2131b88) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #111 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #112 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a2131b88) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #113 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #114 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #115 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e80b7b0) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #116 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #117 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e80b7b0) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #118 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #119 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #120 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800b14b9000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #121 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #122 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800b14b9000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #123 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #124 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #125 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a21fa3d8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #126 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #127 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a21fa3d8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #128 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #129 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #130 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5333d8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #131 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #132 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5333d8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #133 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #134 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #135 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e90ab88) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #136 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #137 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e90ab88) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #138 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #139 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #140 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e4047b0) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #141 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #142 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e4047b0) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #143 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #144 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #145 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e7083d8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #146 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #147 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e7083d8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #148 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #149 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #150 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a23917b0) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #151 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #152 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a23917b0) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #153 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #154 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #155 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e636000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #156 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #157 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e636000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #158 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #159 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #160 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5afb88) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #161 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #162 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5afb88) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #163 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #164 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #165 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800a69523d8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #166 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #167 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800a69523d8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #168 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #169 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #170 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e47c000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #171 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #172 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e47c000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #173 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #174 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #175 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800b17b23d8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #176 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #177 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800b17b23d8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #178 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #179 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #180 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800a6d367b0) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #181 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #182 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800a6d367b0) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #183 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #184 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #185 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e99fb88) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #186 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #187 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e99fb88) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #188 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #189 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #190 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5227b0) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #191 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #192 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5227b0) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >> #193 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #194 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >> #195 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800a6d3c000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >> #196 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >> #197 0xffffffff80b4895a in taskqueue_run_locked (queue=<value optimized >> out>) >> at /usr/src/sys/kern/subr_taskqueue.c:454 >> #198 0xffffffff80b49b58 in taskqueue_thread_loop (arg=<value optimized out>) >> at /usr/src/sys/kern/subr_taskqueue.c:724 >> #199 0xffffffff80a9f255 in fork_exit ( >> callout=0xffffffff80b49a70 <taskqueue_thread_loop>, >> arg=0xfffff8057caa72a0, frame=0xfffffe0668222c00) >> at /usr/src/sys/kern/kern_fork.c:1040 >> #200 0xffffffff80fb44ae in fork_trampoline () >> at /usr/src/sys/amd64/amd64/exception.S:611 >> #201 0x0000000000000000 in ?? () >> Current language: auto; currently minimal >> (kgdb) >> >> >>>> -- >>>> Karl Denninger >>>> karl at denninger.net <mailto:karl at denninger.net> >>>> /The Market Ticker/ >>>> /[S/MIME encrypted email preferred]/ > > _______________________________________________ > freebsd-stable at freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org" >-- Andriy Gapon