Steven Hartland
2016-Oct-17 21:40 UTC
Repeatable panic on ZFS filesystem (used for backups); 11.0-STABLE
Setting those values will only effect what's queued to the device not what's actually outstanding. On 17/10/2016 21:22, Karl Denninger wrote:> Since I cleared it (by setting TRIM off on the test machine, rebooting, > importing the pool and noting that it did not panic -- pulled drives, > re-inserted into the production machine and ran backup routine -- all > was normal) it may be a while before I see it again (a week or so is usual.) > > It appears to be related to entropy in the filesystem that comes up as > "eligible" to be removed from the backup volume, which (not > surprisingly) tends to happen a few days after I do a new world build or > something similar (the daily and/or periodic snapshots roll off at about > that point.) > > I don't happen to have a spare pair of high-performance SSDs I can stick > in the sandbox machine in an attempt to force the condition to assert > itself in test, unfortunately. > > I *am* concerned that it's not "simple" stack exhaustion because setting > the max outstanding TRIMs on a per-vdev basis down quite-dramatically > did *not* prevent it from happening -- and if it was simply stack depth > related I would have expected that to put a stop to it. > > On 10/17/2016 15:16, Steven Hartland wrote: >> Be good to confirm its not an infinite loop by giving it a good bump >> first. >> >> On 17/10/2016 19:58, Karl Denninger wrote: >>> I can certainly attempt setting that higher.... but is that not just >>> hiding the problem rather than addressing it? >>> >>> >>> On 10/17/2016 13:54, Steven Hartland wrote: >>>> You're hitting stack exhaustion, have you tried increasing the kernel >>>> stack pages? >>>> It can be changed from /boot/loader.conf >>>> kern.kstack_pages="6" >>>> >>>> Default on amd64 is 4 IIRC >>>> >>>> On 17/10/2016 19:08, Karl Denninger wrote: >>>>> The target (and devices that trigger this) are a pair of 4Gb 7200RPM >>>>> SATA rotating rust drives (zmirror) with each provider geli-encrypted >>>>> (that is, the actual devices used for the pool create are the .eli's) >>>>> >>>>> The machine generating the problem has both rotating rust devices >>>>> *and* >>>>> SSDs, so I can't simply shut TRIM off system-wide and call it a day as >>>>> TRIM itself is heavily-used; both the boot/root pools and a Postgresql >>>>> database pool are on SSDs, while several terabytes of lesser-used data >>>>> is on a pool of Raidz2 that is made up of spinning rust. >>>> snip... >>>>> NewFS.denninger.net dumped core - see /var/crash/vmcore.1 >>>>> >>>>> Mon Oct 17 09:02:33 CDT 2016 >>>>> >>>>> FreeBSD NewFS.denninger.net 11.0-STABLE FreeBSD 11.0-STABLE #13 >>>>> r307318M: Fri Oct 14 09:23:46 CDT 2016 >>>>> karl at NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP amd64 >>>>> >>>>> panic: double fault >>>>> >>>>> GNU gdb 6.1.1 [FreeBSD] >>>>> Copyright 2004 Free Software Foundation, Inc. >>>>> GDB is free software, covered by the GNU General Public License, and >>>>> you are >>>>> welcome to change it and/or distribute copies of it under certain >>>>> conditions. >>>>> Type "show copying" to see the conditions. >>>>> There is absolutely no warranty for GDB. Type "show warranty" for >>>>> details. >>>>> This GDB was configured as "amd64-marcel-freebsd"... >>>>> >>>>> Unread portion of the kernel message buffer: >>>>> >>>>> Fatal double fault >>>>> rip = 0xffffffff8220d9ec >>>>> rsp = 0xfffffe066821f000 >>>>> rbp = 0xfffffe066821f020 >>>>> cpuid = 6; apic id = 14 >>>>> panic: double fault >>>>> cpuid = 6 >>>>> KDB: stack backtrace: >>>>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame >>>>> 0xfffffe0649d78e30 >>>>> vpanic() at vpanic+0x182/frame 0xfffffe0649d78eb0 >>>>> panic() at panic+0x43/frame 0xfffffe0649d78f10 >>>>> dblfault_handler() at dblfault_handler+0xa2/frame 0xfffffe0649d78f30 >>>>> Xdblfault() at Xdblfault+0xac/frame 0xfffffe0649d78f30 >>>>> --- trap 0x17, rip = 0xffffffff8220d9ec, rsp = 0xfffffe066821f000, >>>>> rbp >>>>> 0xfffffe066821f020 --- >>>>> avl_rotation() at avl_rotation+0xc/frame 0xfffffe066821f020 >>>>> avl_remove() at avl_remove+0x1c8/frame 0xfffffe066821f070 >>>>> vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x87f/frame >>>>> 0xfffffe066821f530 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0x83/frame >>>>> 0xfffffe066821f570 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f5a0 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f5f0 >>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>> 0xfffffe066821f650 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f6a0 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>> 0xfffffe066821f6e0 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f710 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f760 >>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>> 0xfffffe066821f7c0 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f810 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>> 0xfffffe066821f850 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f880 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f8d0 >>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>> 0xfffffe066821f930 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f980 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>> 0xfffffe066821f9c0 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f9f0 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fa40 >>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>> 0xfffffe066821faa0 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821faf0 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>> 0xfffffe066821fb30 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fb60 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fbb0 >>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>> 0xfffffe066821fc10 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fc60 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>> 0xfffffe066821fca0 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fcd0 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fd20 >>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>> 0xfffffe066821fd80 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fdd0 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>> 0xfffffe066821fe10 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fe40 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fe90 >>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>> 0xfffffe066821fef0 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821ff40 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>> 0xfffffe066821ff80 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821ffb0 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220000 >>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>> 0xfffffe0668220060 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682200b0 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>> 0xfffffe06682200f0 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220120 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220170 >>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>> 0xfffffe06682201d0 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220220 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>> 0xfffffe0668220260 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220290 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682202e0 >>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>> 0xfffffe0668220340 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220390 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>> 0xfffffe06682203d0 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220400 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220450 >>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>> 0xfffffe06682204b0 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220500 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>> 0xfffffe0668220540 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220570 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682205c0 >>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>> 0xfffffe0668220620 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220670 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>> 0xfffffe06682206b0 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682206e0 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220730 >>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>> 0xfffffe0668220790 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682207e0 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>> 0xfffffe0668220820 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220850 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682208a0 >>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>> 0xfffffe0668220900 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220950 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>> 0xfffffe0668220990 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682209c0 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220a10 >>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>> 0xfffffe0668220a70 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220ac0 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>> 0xfffffe0668220b00 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220b30 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220b80 >>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>> 0xfffffe0668220be0 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220c30 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>> 0xfffffe0668220c70 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220ca0 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220cf0 >>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>> 0xfffffe0668220d50 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220da0 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>> 0xfffffe0668220de0 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220e10 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220e60 >>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>> 0xfffffe0668220ec0 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220f10 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>> 0xfffffe0668220f50 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220f80 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220fd0 >>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>> 0xfffffe0668221030 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221080 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>> 0xfffffe06682210c0 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682210f0 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221140 >>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>> 0xfffffe06682211a0 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682211f0 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>> 0xfffffe0668221230 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221260 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682212b0 >>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>> 0xfffffe0668221310 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221360 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>> 0xfffffe06682213a0 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682213d0 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221420 >>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>> 0xfffffe0668221480 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682214d0 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>> 0xfffffe0668221510 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221540 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221590 >>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>> 0xfffffe06682215f0 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221640 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>> 0xfffffe0668221680 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682216b0 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221700 >>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>> 0xfffffe0668221760 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682217b0 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>> 0xfffffe06682217f0 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221820 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221870 >>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>> 0xfffffe06682218d0 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221920 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>> 0xfffffe0668221960 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221990 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682219e0 >>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>> 0xfffffe0668221a40 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221a90 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>> 0xfffffe0668221ad0 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221b00 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221b50 >>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>> 0xfffffe0668221bb0 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221c00 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>> 0xfffffe0668221c40 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221c70 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221cc0 >>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>> 0xfffffe0668221d20 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221d70 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>> 0xfffffe0668221db0 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221de0 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221e30 >>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>> 0xfffffe0668221e90 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221ee0 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>> 0xfffffe0668221f20 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221f50 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221fa0 >>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>> 0xfffffe0668222000 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222050 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>> 0xfffffe0668222090 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682220c0 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222110 >>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>> 0xfffffe0668222170 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682221c0 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>> 0xfffffe0668222200 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222230 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222280 >>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>> 0xfffffe06682222e0 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222330 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>> 0xfffffe0668222370 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682223a0 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682223f0 >>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>> 0xfffffe0668222450 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682224a0 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>> 0xfffffe06682224e0 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222510 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222560 >>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>> 0xfffffe06682225c0 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222610 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>> 0xfffffe0668222650 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222680 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682226d0 >>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>> 0xfffffe0668222730 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222780 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>> 0xfffffe06682227c0 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682227f0 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222840 >>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>> 0xfffffe06682228a0 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682228f0 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>> 0xfffffe0668222930 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222960 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682229b0 >>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>> 0xfffffe0668222a10 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222a60 >>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>> 0xfffffe0668222aa0 >>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222ad0 >>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222b20 >>>>> taskqueue_run_locked() at taskqueue_run_locked+0x14a/frame >>>>> 0xfffffe0668222b80 >>>>> taskqueue_thread_loop() at taskqueue_thread_loop+0xe8/frame >>>>> 0xfffffe0668222bb0 >>>>> fork_exit() at fork_exit+0x85/frame 0xfffffe0668222bf0 >>>>> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0668222bf0 >>>>> --- trap 0, rip = 0, rsp = 0, rbp = 0 --- >>>>> Uptime: 4m29s >>>>> Dumping 1497 out of 24531 >>>>> MB:..2%..11%..21%..31%..41%..51%..61%..71%..81%..91% >>>>> >>>>> Reading symbols from /boot/kernel/zfs.ko...Reading symbols from >>>>> /usr/lib/debug//boot/kernel/zfs.ko.debug...done. >>>>> done. >>>>> Loaded symbols for /boot/kernel/zfs.ko >>>>> Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols >>>>> from >>>>> /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done. >>>>> done. >>>>> Loaded symbols for /boot/kernel/opensolaris.ko >>>>> Reading symbols from /boot/kernel/geom_eli.ko...Reading symbols from >>>>> /usr/lib/debug//boot/kernel/geom_eli.ko.debug...done. >>>>> done. >>>>> Loaded symbols for /boot/kernel/geom_eli.ko >>>>> Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols >>>>> from >>>>> /usr/lib/debug//boot/kernel/geom_mirror.ko.debug...done. >>>>> done. >>>>> Loaded symbols for /boot/kernel/geom_mirror.ko >>>>> Reading symbols from /boot/kernel/aesni.ko...Reading symbols from >>>>> /usr/lib/debug//boot/kernel/aesni.ko.debug...done. >>>>> done. >>>>> Loaded symbols for /boot/kernel/aesni.ko >>>>> Reading symbols from /boot/kernel/uhid.ko...Reading symbols from >>>>> /usr/lib/debug//boot/kernel/uhid.ko.debug...done. >>>>> done. >>>>> Loaded symbols for /boot/kernel/uhid.ko >>>>> Reading symbols from /boot/kernel/ums.ko...Reading symbols from >>>>> /usr/lib/debug//boot/kernel/ums.ko.debug...done. >>>>> done. >>>>> Loaded symbols for /boot/kernel/ums.ko >>>>> Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from >>>>> /usr/lib/debug//boot/kernel/ipfw.ko.debug...done. >>>>> done. >>>>> Loaded symbols for /boot/kernel/ipfw.ko >>>>> Reading symbols from /boot/kernel/ipfw_nat.ko...Reading symbols from >>>>> /usr/lib/debug//boot/kernel/ipfw_nat.ko.debug...done. >>>>> done. >>>>> Loaded symbols for /boot/kernel/ipfw_nat.ko >>>>> Reading symbols from /boot/kernel/libalias.ko...Reading symbols from >>>>> /usr/lib/debug//boot/kernel/libalias.ko.debug...done. >>>>> done. >>>>> Loaded symbols for /boot/kernel/libalias.ko >>>>> Reading symbols from /boot/kernel/daemon_saver.ko...Reading symbols >>>>> from >>>>> /usr/lib/debug//boot/kernel/daemon_saver.ko.debug...done. >>>>> done. >>>>> Loaded symbols for /boot/kernel/daemon_saver.ko >>>>> #0 doadump (textdump=1) at pcpu.h:221 >>>>> 221 pcpu.h: No such file or directory. >>>>> in pcpu.h >>>>> (kgdb) #0 doadump (textdump=1) at pcpu.h:221 >>>>> #1 0xffffffff80ae9b6e in kern_reboot (howto=260) >>>>> at /usr/src/sys/kern/kern_shutdown.c:366 >>>>> #2 0xffffffff80aea13b in vpanic (fmt=<value optimized out>, >>>>> ap=<value optimized out>) at >>>>> /usr/src/sys/kern/kern_shutdown.c:759 >>>>> #3 0xffffffff80ae9f73 in panic (fmt=0x0) >>>>> at /usr/src/sys/kern/kern_shutdown.c:690 >>>>> #4 0xffffffff80fd18f2 in dblfault_handler (frame=<value optimized >>>>> out>) >>>>> at /usr/src/sys/amd64/amd64/trap.c:868 >>>>> #5 0xffffffff80fb406c in Xdblfault () >>>>> at /usr/src/sys/amd64/amd64/exception.S:292 >>>>> #6 0xffffffff8220d9ec in avl_rotation (tree=0xfffff8004dc54510, >>>>> node=0xfffff8056e45cdd0, balance=2) >>>>> at /usr/src/sys/cddl/contrib/opensolaris/common/avl/avl.c:304 >>>>> #7 0xffffffff8220de58 in avl_remove (tree=<value optimized out>, >>>>> data=<value optimized out>) >>>>> at /usr/src/sys/cddl/contrib/opensolaris/common/avl/avl.c:818 >>>>> #8 0xffffffff822a1eff in vdev_queue_io_to_issue (vq=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:439 >>>>> >>>>> #9 0xffffffff822a2123 in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:907 >>>>> >>>>> #10 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e3f33d8) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #11 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #12 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e3f33d8) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>>> #13 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #14 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>>>> >>>>> #15 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e578000) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #16 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #17 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e578000) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>>> #18 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #19 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>>>> >>>>> #20 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5ec3d8) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #21 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #22 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5ec3d8) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>>> #23 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #24 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>>>> >>>>> #25 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e48a000) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #26 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #27 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e48a000) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>>> #28 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #29 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>>>> >>>>> #30 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e57ab88) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #31 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #32 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e57ab88) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>>> #33 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #34 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>>>> >>>>> #35 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800a69a3000) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #36 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #37 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800a69a3000) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>>> #38 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #39 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>>>> >>>>> #40 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5c8000) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #41 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #42 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5c8000) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>>> #43 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #44 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>>>> >>>>> #45 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a23f9000) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #46 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #47 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a23f9000) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>>> #48 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #49 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>>>> >>>>> #50 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5147b0) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #51 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #52 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5147b0) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>>> #53 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #54 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>>>> >>>>> #55 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e977000) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #56 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #57 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e977000) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>>> #58 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #59 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>>>> >>>>> #60 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e90b3d8) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #61 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #62 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e90b3d8) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>>> #63 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #64 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>>>> >>>>> #65 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800b1160b88) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #66 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #67 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800b1160b88) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>>> #68 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #69 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>>>> >>>>> #70 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e4f53d8) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #71 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #72 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e4f53d8) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>>> #73 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #74 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>>>> >>>>> #75 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e8ebb88) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #76 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #77 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e8ebb88) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>>> #78 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #79 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>>>> >>>>> #80 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e8d9b88) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #81 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #82 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e8d9b88) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>>> #83 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #84 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>>>> >>>>> #85 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a21a03d8) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #86 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #87 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a21a03d8) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>>> #88 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #89 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>>>> >>>>> #90 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e6903d8) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #91 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #92 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e6903d8) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>>> #93 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #94 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>>>> >>>>> #95 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a22dc7b0) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #96 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #97 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a22dc7b0) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>>> #98 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #99 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>>>> >>>>> #100 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5fb7b0) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #101 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #102 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5fb7b0) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>>> #103 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #104 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>>>> >>>>> #105 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5ae3d8) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #106 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #107 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5ae3d8) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>>> #108 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #109 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>>>> >>>>> #110 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a2131b88) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #111 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #112 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a2131b88) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>>> #113 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #114 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>>>> >>>>> #115 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e80b7b0) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #116 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #117 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e80b7b0) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>>> #118 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #119 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>>>> >>>>> #120 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800b14b9000) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #121 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #122 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800b14b9000) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>>> #123 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #124 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>>>> >>>>> #125 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a21fa3d8) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #126 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #127 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a21fa3d8) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>>> #128 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #129 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>>>> >>>>> #130 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5333d8) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #131 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #132 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5333d8) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>>> #133 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #134 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>>>> >>>>> #135 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e90ab88) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #136 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #137 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e90ab88) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>>> #138 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #139 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>>>> >>>>> #140 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e4047b0) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #141 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #142 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e4047b0) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>>> #143 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #144 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>>>> >>>>> #145 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e7083d8) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #146 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #147 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e7083d8) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>>> #148 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #149 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>>>> >>>>> #150 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff802a23917b0) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #151 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #152 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff802a23917b0) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>>> #153 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #154 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>>>> >>>>> #155 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e636000) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #156 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #157 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e636000) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>>> #158 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #159 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>>>> >>>>> #160 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5afb88) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #161 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #162 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5afb88) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>>> #163 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #164 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>>>> >>>>> #165 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800a69523d8) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #166 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #167 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800a69523d8) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>>> #168 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #169 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>>>> >>>>> #170 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e47c000) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #171 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #172 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e47c000) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>>> #173 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #174 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>>>> >>>>> #175 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800b17b23d8) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #176 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #177 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800b17b23d8) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>>> #178 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #179 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>>>> >>>>> #180 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800a6d367b0) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #181 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #182 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff800a6d367b0) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>>> #183 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #184 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>>>> >>>>> #185 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e99fb88) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #186 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #187 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e99fb88) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>>> #188 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #189 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>>>> >>>>> #190 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff8056e5227b0) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #191 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #192 0xffffffff822bf75d in zio_vdev_io_start (zio=0xfffff8056e5227b0) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>>> #193 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #194 0xffffffff822a216d in vdev_queue_io_done (zio=<value optimized >>>>> out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913 >>>>> >>>>> #195 0xffffffff822bf919 in zio_vdev_io_done (zio=0xfffff800a6d3c000) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>>> #196 0xffffffff822bbefd in zio_execute (zio=<value optimized out>) >>>>> at >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>>> #197 0xffffffff80b4895a in taskqueue_run_locked (queue=<value >>>>> optimized >>>>> out>) >>>>> at /usr/src/sys/kern/subr_taskqueue.c:454 >>>>> #198 0xffffffff80b49b58 in taskqueue_thread_loop (arg=<value >>>>> optimized out>) >>>>> at /usr/src/sys/kern/subr_taskqueue.c:724 >>>>> #199 0xffffffff80a9f255 in fork_exit ( >>>>> callout=0xffffffff80b49a70 <taskqueue_thread_loop>, >>>>> arg=0xfffff8057caa72a0, frame=0xfffffe0668222c00) >>>>> at /usr/src/sys/kern/kern_fork.c:1040 >>>>> #200 0xffffffff80fb44ae in fork_trampoline () >>>>> at /usr/src/sys/amd64/amd64/exception.S:611 >>>>> #201 0x0000000000000000 in ?? () >>>>> Current language: auto; currently minimal >>>>> (kgdb) >>>>> >>>>> >>>>>>> -- >>>>>>> Karl Denninger >>>>>>> karl at denninger.net <mailto:karl at denninger.net> >>>>>>> /The Market Ticker/ >>>>>>> /[S/MIME encrypted email preferred]/ >>>> _______________________________________________ >>>> freebsd-stable at freebsd.org mailing list >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >>>> To unsubscribe, send any mail to >>>> "freebsd-stable-unsubscribe at freebsd.org" >> _______________________________________________ >> freebsd-stable at freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"
Karl Denninger
2016-Oct-17 21:50 UTC
Repeatable panic on ZFS filesystem (used for backups); 11.0-STABLE
I will make some effort on the sandbox machine to see if I can come up
with a way to replicate this. I do have plenty of spare larger drives
laying around that used to be in service and were obsolesced due to
capacity -- but what I don't know if whether the system will misbehave
if the source is all spinning rust.
In other words:
1. Root filesystem is mirrored spinning rust (production is mirrored SSDs)
2. Backup is mirrored spinning rust (of approx the same size)
3. Set up auto-snapshot exactly as the production system has now (which
the sandbox is NOT since I don't care about incremental recovery on that
machine; it's a sandbox!)
4. Run a bunch of build-somethings (e.g. buildworlds, cross-build for
the Pi2s I have here, etc) to generate a LOT of filesystem entropy
across lots of snapshots.
5. Back that up.
6. Export the backup pool.
7. Re-import it and "zfs destroy -r" the backup filesystem.
That is what got me in a reboot loop after the *first* panic; I was
simply going to destroy the backup filesystem and re-run the backup, but
as soon as I issued that zfs destroy the machine panic'd and as soon as
I re-attached it after a reboot it panic'd again. Repeat until I set
trim=0.
But... if I CAN replicate it that still shouldn't be happening, and the
system should *certainly* survive attempting to TRIM on a vdev that
doesn't support TRIMs, even if the removal is for a large amount of
space and/or files on the target, without blowing up.
BTW I bet it isn't that rare -- if you're taking timed snapshots on an
active filesystem (with lots of entropy) and then make the mistake of
trying to remove those snapshots (as is the case with a zfs destroy -r
or a zfs recv of an incremental copy that attempts to sync against a
source) on a pool that has been imported before the system realizes that
TRIM is unavailable on those vdevs.
Noting this:
Yes need to find some time to have a look at it, but given how rare
this is and with TRIM being re-implemented upstream in a totally
different manor I'm reticent to spend any real time on it.
What's in-process in this regard, if you happen to have a reference?
On 10/17/2016 16:40, Steven Hartland wrote:> Setting those values will only effect what's queued to the device not
> what's actually outstanding.
>
> On 17/10/2016 21:22, Karl Denninger wrote:
>> Since I cleared it (by setting TRIM off on the test machine, rebooting,
>> importing the pool and noting that it did not panic -- pulled drives,
>> re-inserted into the production machine and ran backup routine -- all
>> was normal) it may be a while before I see it again (a week or so is
>> usual.)
>>
>> It appears to be related to entropy in the filesystem that comes up as
>> "eligible" to be removed from the backup volume, which (not
>> surprisingly) tends to happen a few days after I do a new world build
or
>> something similar (the daily and/or periodic snapshots roll off at
about
>> that point.)
>>
>> I don't happen to have a spare pair of high-performance SSDs I can
stick
>> in the sandbox machine in an attempt to force the condition to assert
>> itself in test, unfortunately.
>>
>> I *am* concerned that it's not "simple" stack exhaustion
because setting
>> the max outstanding TRIMs on a per-vdev basis down quite-dramatically
>> did *not* prevent it from happening -- and if it was simply stack depth
>> related I would have expected that to put a stop to it.
>>
>> On 10/17/2016 15:16, Steven Hartland wrote:
>>> Be good to confirm its not an infinite loop by giving it a good
bump
>>> first.
>>>
>>> On 17/10/2016 19:58, Karl Denninger wrote:
>>>> I can certainly attempt setting that higher.... but is that not
just
>>>> hiding the problem rather than addressing it?
>>>>
>>>>
>>>> On 10/17/2016 13:54, Steven Hartland wrote:
>>>>> You're hitting stack exhaustion, have you tried
increasing the kernel
>>>>> stack pages?
>>>>> It can be changed from /boot/loader.conf
>>>>> kern.kstack_pages="6"
>>>>>
>>>>> Default on amd64 is 4 IIRC
>>>>>
>>>>> On 17/10/2016 19:08, Karl Denninger wrote:
>>>>>> The target (and devices that trigger this) are a pair
of 4Gb 7200RPM
>>>>>> SATA rotating rust drives (zmirror) with each provider
>>>>>> geli-encrypted
>>>>>> (that is, the actual devices used for the pool create
are the
>>>>>> .eli's)
>>>>>>
>>>>>> The machine generating the problem has both rotating
rust devices
>>>>>> *and*
>>>>>> SSDs, so I can't simply shut TRIM off system-wide
and call it a
>>>>>> day as
>>>>>> TRIM itself is heavily-used; both the boot/root pools
and a
>>>>>> Postgresql
>>>>>> database pool are on SSDs, while several terabytes of
lesser-used
>>>>>> data
>>>>>> is on a pool of Raidz2 that is made up of spinning
rust.
>>>>> snip...
>>>>>> NewFS.denninger.net dumped core - see
/var/crash/vmcore.1
>>>>>>
>>>>>> Mon Oct 17 09:02:33 CDT 2016
>>>>>>
>>>>>> FreeBSD NewFS.denninger.net 11.0-STABLE FreeBSD
11.0-STABLE #13
>>>>>> r307318M: Fri Oct 14 09:23:46 CDT 2016
>>>>>> karl at
NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP amd64
>>>>>>
>>>>>> panic: double fault
>>>>>>
>>>>>> GNU gdb 6.1.1 [FreeBSD]
>>>>>> Copyright 2004 Free Software Foundation, Inc.
>>>>>> GDB is free software, covered by the GNU General Public
License, and
>>>>>> you are
>>>>>> welcome to change it and/or distribute copies of it
under certain
>>>>>> conditions.
>>>>>> Type "show copying" to see the conditions.
>>>>>> There is absolutely no warranty for GDB. Type
"show warranty" for
>>>>>> details.
>>>>>> This GDB was configured as
"amd64-marcel-freebsd"...
>>>>>>
>>>>>> Unread portion of the kernel message buffer:
>>>>>>
>>>>>> Fatal double fault
>>>>>> rip = 0xffffffff8220d9ec
>>>>>> rsp = 0xfffffe066821f000
>>>>>> rbp = 0xfffffe066821f020
>>>>>> cpuid = 6; apic id = 14
>>>>>> panic: double fault
>>>>>> cpuid = 6
>>>>>> KDB: stack backtrace:
>>>>>> db_trace_self_wrapper() at
db_trace_self_wrapper+0x2b/frame
>>>>>> 0xfffffe0649d78e30
>>>>>> vpanic() at vpanic+0x182/frame 0xfffffe0649d78eb0
>>>>>> panic() at panic+0x43/frame 0xfffffe0649d78f10
>>>>>> dblfault_handler() at dblfault_handler+0xa2/frame
0xfffffe0649d78f30
>>>>>> Xdblfault() at Xdblfault+0xac/frame 0xfffffe0649d78f30
>>>>>> --- trap 0x17, rip = 0xffffffff8220d9ec, rsp =
0xfffffe066821f000,
>>>>>> rbp >>>>>> 0xfffffe066821f020 ---
>>>>>> avl_rotation() at avl_rotation+0xc/frame
0xfffffe066821f020
>>>>>> avl_remove() at avl_remove+0x1c8/frame
0xfffffe066821f070
>>>>>> vdev_queue_io_to_issue() at
vdev_queue_io_to_issue+0x87f/frame
>>>>>> 0xfffffe066821f530
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0x83/frame
>>>>>> 0xfffffe066821f570
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe066821f5a0
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe066821f5f0
>>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>>>> 0xfffffe066821f650
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe066821f6a0
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>>>> 0xfffffe066821f6e0
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe066821f710
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe066821f760
>>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>>>> 0xfffffe066821f7c0
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe066821f810
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>>>> 0xfffffe066821f850
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe066821f880
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe066821f8d0
>>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>>>> 0xfffffe066821f930
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe066821f980
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>>>> 0xfffffe066821f9c0
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe066821f9f0
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe066821fa40
>>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>>>> 0xfffffe066821faa0
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe066821faf0
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>>>> 0xfffffe066821fb30
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe066821fb60
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe066821fbb0
>>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>>>> 0xfffffe066821fc10
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe066821fc60
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>>>> 0xfffffe066821fca0
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe066821fcd0
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe066821fd20
>>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>>>> 0xfffffe066821fd80
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe066821fdd0
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>>>> 0xfffffe066821fe10
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe066821fe40
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe066821fe90
>>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>>>> 0xfffffe066821fef0
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe066821ff40
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>>>> 0xfffffe066821ff80
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe066821ffb0
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668220000
>>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>>>> 0xfffffe0668220060
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe06682200b0
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>>>> 0xfffffe06682200f0
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668220120
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668220170
>>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>>>> 0xfffffe06682201d0
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668220220
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>>>> 0xfffffe0668220260
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668220290
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe06682202e0
>>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>>>> 0xfffffe0668220340
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668220390
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>>>> 0xfffffe06682203d0
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668220400
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668220450
>>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>>>> 0xfffffe06682204b0
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668220500
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>>>> 0xfffffe0668220540
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668220570
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe06682205c0
>>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>>>> 0xfffffe0668220620
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668220670
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>>>> 0xfffffe06682206b0
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe06682206e0
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668220730
>>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>>>> 0xfffffe0668220790
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe06682207e0
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>>>> 0xfffffe0668220820
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668220850
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe06682208a0
>>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>>>> 0xfffffe0668220900
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668220950
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>>>> 0xfffffe0668220990
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe06682209c0
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668220a10
>>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>>>> 0xfffffe0668220a70
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668220ac0
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>>>> 0xfffffe0668220b00
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668220b30
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668220b80
>>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>>>> 0xfffffe0668220be0
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668220c30
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>>>> 0xfffffe0668220c70
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668220ca0
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668220cf0
>>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>>>> 0xfffffe0668220d50
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668220da0
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>>>> 0xfffffe0668220de0
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668220e10
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668220e60
>>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>>>> 0xfffffe0668220ec0
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668220f10
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>>>> 0xfffffe0668220f50
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668220f80
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668220fd0
>>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>>>> 0xfffffe0668221030
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668221080
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>>>> 0xfffffe06682210c0
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe06682210f0
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668221140
>>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>>>> 0xfffffe06682211a0
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe06682211f0
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>>>> 0xfffffe0668221230
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668221260
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe06682212b0
>>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>>>> 0xfffffe0668221310
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668221360
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>>>> 0xfffffe06682213a0
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe06682213d0
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668221420
>>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>>>> 0xfffffe0668221480
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe06682214d0
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>>>> 0xfffffe0668221510
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668221540
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668221590
>>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>>>> 0xfffffe06682215f0
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668221640
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>>>> 0xfffffe0668221680
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe06682216b0
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668221700
>>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>>>> 0xfffffe0668221760
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe06682217b0
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>>>> 0xfffffe06682217f0
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668221820
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668221870
>>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>>>> 0xfffffe06682218d0
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668221920
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>>>> 0xfffffe0668221960
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668221990
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe06682219e0
>>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>>>> 0xfffffe0668221a40
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668221a90
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>>>> 0xfffffe0668221ad0
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668221b00
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668221b50
>>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>>>> 0xfffffe0668221bb0
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668221c00
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>>>> 0xfffffe0668221c40
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668221c70
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668221cc0
>>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>>>> 0xfffffe0668221d20
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668221d70
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>>>> 0xfffffe0668221db0
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668221de0
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668221e30
>>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>>>> 0xfffffe0668221e90
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668221ee0
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>>>> 0xfffffe0668221f20
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668221f50
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668221fa0
>>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>>>> 0xfffffe0668222000
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668222050
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>>>> 0xfffffe0668222090
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe06682220c0
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668222110
>>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>>>> 0xfffffe0668222170
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe06682221c0
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>>>> 0xfffffe0668222200
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668222230
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668222280
>>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>>>> 0xfffffe06682222e0
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668222330
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>>>> 0xfffffe0668222370
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe06682223a0
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe06682223f0
>>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>>>> 0xfffffe0668222450
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe06682224a0
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>>>> 0xfffffe06682224e0
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668222510
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668222560
>>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>>>> 0xfffffe06682225c0
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668222610
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>>>> 0xfffffe0668222650
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668222680
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe06682226d0
>>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>>>> 0xfffffe0668222730
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668222780
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>>>> 0xfffffe06682227c0
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe06682227f0
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668222840
>>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>>>> 0xfffffe06682228a0
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe06682228f0
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>>>> 0xfffffe0668222930
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668222960
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe06682229b0
>>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame
>>>>>> 0xfffffe0668222a10
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668222a60
>>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
>>>>>> 0xfffffe0668222aa0
>>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame
0xfffffe0668222ad0
>>>>>> zio_execute() at zio_execute+0x23d/frame
0xfffffe0668222b20
>>>>>> taskqueue_run_locked() at
taskqueue_run_locked+0x14a/frame
>>>>>> 0xfffffe0668222b80
>>>>>> taskqueue_thread_loop() at
taskqueue_thread_loop+0xe8/frame
>>>>>> 0xfffffe0668222bb0
>>>>>> fork_exit() at fork_exit+0x85/frame 0xfffffe0668222bf0
>>>>>> fork_trampoline() at fork_trampoline+0xe/frame
0xfffffe0668222bf0
>>>>>> --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
>>>>>> Uptime: 4m29s
>>>>>> Dumping 1497 out of 24531
>>>>>> MB:..2%..11%..21%..31%..41%..51%..61%..71%..81%..91%
>>>>>>
>>>>>> Reading symbols from /boot/kernel/zfs.ko...Reading
symbols from
>>>>>> /usr/lib/debug//boot/kernel/zfs.ko.debug...done.
>>>>>> done.
>>>>>> Loaded symbols for /boot/kernel/zfs.ko
>>>>>> Reading symbols from
/boot/kernel/opensolaris.ko...Reading symbols
>>>>>> from
>>>>>>
/usr/lib/debug//boot/kernel/opensolaris.ko.debug...done.
>>>>>> done.
>>>>>> Loaded symbols for /boot/kernel/opensolaris.ko
>>>>>> Reading symbols from /boot/kernel/geom_eli.ko...Reading
symbols from
>>>>>> /usr/lib/debug//boot/kernel/geom_eli.ko.debug...done.
>>>>>> done.
>>>>>> Loaded symbols for /boot/kernel/geom_eli.ko
>>>>>> Reading symbols from
/boot/kernel/geom_mirror.ko...Reading symbols
>>>>>> from
>>>>>>
/usr/lib/debug//boot/kernel/geom_mirror.ko.debug...done.
>>>>>> done.
>>>>>> Loaded symbols for /boot/kernel/geom_mirror.ko
>>>>>> Reading symbols from /boot/kernel/aesni.ko...Reading
symbols from
>>>>>> /usr/lib/debug//boot/kernel/aesni.ko.debug...done.
>>>>>> done.
>>>>>> Loaded symbols for /boot/kernel/aesni.ko
>>>>>> Reading symbols from /boot/kernel/uhid.ko...Reading
symbols from
>>>>>> /usr/lib/debug//boot/kernel/uhid.ko.debug...done.
>>>>>> done.
>>>>>> Loaded symbols for /boot/kernel/uhid.ko
>>>>>> Reading symbols from /boot/kernel/ums.ko...Reading
symbols from
>>>>>> /usr/lib/debug//boot/kernel/ums.ko.debug...done.
>>>>>> done.
>>>>>> Loaded symbols for /boot/kernel/ums.ko
>>>>>> Reading symbols from /boot/kernel/ipfw.ko...Reading
symbols from
>>>>>> /usr/lib/debug//boot/kernel/ipfw.ko.debug...done.
>>>>>> done.
>>>>>> Loaded symbols for /boot/kernel/ipfw.ko
>>>>>> Reading symbols from /boot/kernel/ipfw_nat.ko...Reading
symbols from
>>>>>> /usr/lib/debug//boot/kernel/ipfw_nat.ko.debug...done.
>>>>>> done.
>>>>>> Loaded symbols for /boot/kernel/ipfw_nat.ko
>>>>>> Reading symbols from /boot/kernel/libalias.ko...Reading
symbols from
>>>>>> /usr/lib/debug//boot/kernel/libalias.ko.debug...done.
>>>>>> done.
>>>>>> Loaded symbols for /boot/kernel/libalias.ko
>>>>>> Reading symbols from
/boot/kernel/daemon_saver.ko...Reading symbols
>>>>>> from
>>>>>>
/usr/lib/debug//boot/kernel/daemon_saver.ko.debug...done.
>>>>>> done.
>>>>>> Loaded symbols for /boot/kernel/daemon_saver.ko
>>>>>> #0 doadump (textdump=1) at pcpu.h:221
>>>>>> 221 pcpu.h: No such file or directory.
>>>>>> in pcpu.h
>>>>>> (kgdb) #0 doadump (textdump=1) at pcpu.h:221
>>>>>> #1 0xffffffff80ae9b6e in kern_reboot (howto=260)
>>>>>> at /usr/src/sys/kern/kern_shutdown.c:366
>>>>>> #2 0xffffffff80aea13b in vpanic (fmt=<value
optimized out>,
>>>>>> ap=<value optimized out>) at
>>>>>> /usr/src/sys/kern/kern_shutdown.c:759
>>>>>> #3 0xffffffff80ae9f73 in panic (fmt=0x0)
>>>>>> at /usr/src/sys/kern/kern_shutdown.c:690
>>>>>> #4 0xffffffff80fd18f2 in dblfault_handler
(frame=<value optimized
>>>>>> out>)
>>>>>> at /usr/src/sys/amd64/amd64/trap.c:868
>>>>>> #5 0xffffffff80fb406c in Xdblfault ()
>>>>>> at /usr/src/sys/amd64/amd64/exception.S:292
>>>>>> #6 0xffffffff8220d9ec in avl_rotation
(tree=0xfffff8004dc54510,
>>>>>> node=0xfffff8056e45cdd0, balance=2)
>>>>>> at
/usr/src/sys/cddl/contrib/opensolaris/common/avl/avl.c:304
>>>>>> #7 0xffffffff8220de58 in avl_remove (tree=<value
optimized out>,
>>>>>> data=<value optimized out>)
>>>>>> at
/usr/src/sys/cddl/contrib/opensolaris/common/avl/avl.c:818
>>>>>> #8 0xffffffff822a1eff in vdev_queue_io_to_issue
(vq=<value
>>>>>> optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:439
>>>>>>
>>>>>>
>>>>>> #9 0xffffffff822a2123 in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:907
>>>>>>
>>>>>>
>>>>>> #10 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e3f33d8)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #11 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #12 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e3f33d8)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>>>> #13 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #14 0xffffffff822a216d in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>>>
>>>>>>
>>>>>> #15 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e578000)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #16 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #17 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e578000)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>>>> #18 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #19 0xffffffff822a216d in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>>>
>>>>>>
>>>>>> #20 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e5ec3d8)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #21 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #22 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e5ec3d8)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>>>> #23 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #24 0xffffffff822a216d in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>>>
>>>>>>
>>>>>> #25 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e48a000)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #26 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #27 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e48a000)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>>>> #28 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #29 0xffffffff822a216d in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>>>
>>>>>>
>>>>>> #30 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e57ab88)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #31 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #32 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e57ab88)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>>>> #33 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #34 0xffffffff822a216d in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>>>
>>>>>>
>>>>>> #35 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff800a69a3000)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #36 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #37 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff800a69a3000)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>>>> #38 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #39 0xffffffff822a216d in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>>>
>>>>>>
>>>>>> #40 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e5c8000)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #41 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #42 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e5c8000)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>>>> #43 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #44 0xffffffff822a216d in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>>>
>>>>>>
>>>>>> #45 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff802a23f9000)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #46 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #47 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff802a23f9000)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>>>> #48 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #49 0xffffffff822a216d in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>>>
>>>>>>
>>>>>> #50 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e5147b0)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #51 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #52 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e5147b0)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>>>> #53 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #54 0xffffffff822a216d in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>>>
>>>>>>
>>>>>> #55 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e977000)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #56 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #57 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e977000)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>>>> #58 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #59 0xffffffff822a216d in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>>>
>>>>>>
>>>>>> #60 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e90b3d8)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #61 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #62 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e90b3d8)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>>>> #63 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #64 0xffffffff822a216d in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>>>
>>>>>>
>>>>>> #65 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff800b1160b88)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #66 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #67 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff800b1160b88)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>>>> #68 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #69 0xffffffff822a216d in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>>>
>>>>>>
>>>>>> #70 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e4f53d8)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #71 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #72 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e4f53d8)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>>>> #73 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #74 0xffffffff822a216d in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>>>
>>>>>>
>>>>>> #75 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e8ebb88)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #76 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #77 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e8ebb88)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>>>> #78 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #79 0xffffffff822a216d in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>>>
>>>>>>
>>>>>> #80 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e8d9b88)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #81 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #82 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e8d9b88)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>>>> #83 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #84 0xffffffff822a216d in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>>>
>>>>>>
>>>>>> #85 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff802a21a03d8)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #86 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #87 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff802a21a03d8)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>>>> #88 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #89 0xffffffff822a216d in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>>>
>>>>>>
>>>>>> #90 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e6903d8)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #91 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #92 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff8056e6903d8)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>>>> #93 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #94 0xffffffff822a216d in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>>>
>>>>>>
>>>>>> #95 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff802a22dc7b0)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #96 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #97 0xffffffff822bf75d in zio_vdev_io_start
(zio=0xfffff802a22dc7b0)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>>>> #98 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #99 0xffffffff822a216d in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>>>
>>>>>>
>>>>>> #100 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e5fb7b0)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #101 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #102 0xffffffff822bf75d in zio_vdev_io_start
>>>>>> (zio=0xfffff8056e5fb7b0)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>>>> #103 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #104 0xffffffff822a216d in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>>>
>>>>>>
>>>>>> #105 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e5ae3d8)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #106 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #107 0xffffffff822bf75d in zio_vdev_io_start
>>>>>> (zio=0xfffff8056e5ae3d8)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>>>> #108 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #109 0xffffffff822a216d in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>>>
>>>>>>
>>>>>> #110 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff802a2131b88)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #111 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #112 0xffffffff822bf75d in zio_vdev_io_start
>>>>>> (zio=0xfffff802a2131b88)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>>>> #113 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #114 0xffffffff822a216d in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>>>
>>>>>>
>>>>>> #115 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e80b7b0)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #116 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #117 0xffffffff822bf75d in zio_vdev_io_start
>>>>>> (zio=0xfffff8056e80b7b0)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>>>> #118 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #119 0xffffffff822a216d in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>>>
>>>>>>
>>>>>> #120 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff800b14b9000)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #121 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #122 0xffffffff822bf75d in zio_vdev_io_start
>>>>>> (zio=0xfffff800b14b9000)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>>>> #123 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #124 0xffffffff822a216d in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>>>
>>>>>>
>>>>>> #125 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff802a21fa3d8)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #126 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #127 0xffffffff822bf75d in zio_vdev_io_start
>>>>>> (zio=0xfffff802a21fa3d8)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>>>> #128 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #129 0xffffffff822a216d in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>>>
>>>>>>
>>>>>> #130 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e5333d8)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #131 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #132 0xffffffff822bf75d in zio_vdev_io_start
>>>>>> (zio=0xfffff8056e5333d8)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>>>> #133 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #134 0xffffffff822a216d in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>>>
>>>>>>
>>>>>> #135 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e90ab88)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #136 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #137 0xffffffff822bf75d in zio_vdev_io_start
>>>>>> (zio=0xfffff8056e90ab88)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>>>> #138 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #139 0xffffffff822a216d in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>>>
>>>>>>
>>>>>> #140 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e4047b0)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #141 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #142 0xffffffff822bf75d in zio_vdev_io_start
>>>>>> (zio=0xfffff8056e4047b0)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>>>> #143 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #144 0xffffffff822a216d in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>>>
>>>>>>
>>>>>> #145 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e7083d8)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #146 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #147 0xffffffff822bf75d in zio_vdev_io_start
>>>>>> (zio=0xfffff8056e7083d8)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>>>> #148 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #149 0xffffffff822a216d in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>>>
>>>>>>
>>>>>> #150 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff802a23917b0)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #151 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #152 0xffffffff822bf75d in zio_vdev_io_start
>>>>>> (zio=0xfffff802a23917b0)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>>>> #153 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #154 0xffffffff822a216d in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>>>
>>>>>>
>>>>>> #155 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e636000)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #156 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #157 0xffffffff822bf75d in zio_vdev_io_start
>>>>>> (zio=0xfffff8056e636000)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>>>> #158 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #159 0xffffffff822a216d in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>>>
>>>>>>
>>>>>> #160 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e5afb88)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #161 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #162 0xffffffff822bf75d in zio_vdev_io_start
>>>>>> (zio=0xfffff8056e5afb88)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>>>> #163 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #164 0xffffffff822a216d in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>>>
>>>>>>
>>>>>> #165 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff800a69523d8)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #166 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #167 0xffffffff822bf75d in zio_vdev_io_start
>>>>>> (zio=0xfffff800a69523d8)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>>>> #168 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #169 0xffffffff822a216d in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>>>
>>>>>>
>>>>>> #170 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e47c000)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #171 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #172 0xffffffff822bf75d in zio_vdev_io_start
>>>>>> (zio=0xfffff8056e47c000)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>>>> #173 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #174 0xffffffff822a216d in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>>>
>>>>>>
>>>>>> #175 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff800b17b23d8)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #176 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #177 0xffffffff822bf75d in zio_vdev_io_start
>>>>>> (zio=0xfffff800b17b23d8)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>>>> #178 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #179 0xffffffff822a216d in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>>>
>>>>>>
>>>>>> #180 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff800a6d367b0)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #181 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #182 0xffffffff822bf75d in zio_vdev_io_start
>>>>>> (zio=0xfffff800a6d367b0)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>>>> #183 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #184 0xffffffff822a216d in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>>>
>>>>>>
>>>>>> #185 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e99fb88)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #186 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #187 0xffffffff822bf75d in zio_vdev_io_start
>>>>>> (zio=0xfffff8056e99fb88)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>>>> #188 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #189 0xffffffff822a216d in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>>>
>>>>>>
>>>>>> #190 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff8056e5227b0)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #191 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #192 0xffffffff822bf75d in zio_vdev_io_start
>>>>>> (zio=0xfffff8056e5227b0)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112
>>>>>> #193 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #194 0xffffffff822a216d in vdev_queue_io_done
(zio=<value optimized
>>>>>> out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:913
>>>>>>
>>>>>>
>>>>>> #195 0xffffffff822bf919 in zio_vdev_io_done
(zio=0xfffff800a6d3c000)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137
>>>>>> #196 0xffffffff822bbefd in zio_execute (zio=<value
optimized out>)
>>>>>> at
>>>>>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651
>>>>>> #197 0xffffffff80b4895a in taskqueue_run_locked
(queue=<value
>>>>>> optimized
>>>>>> out>)
>>>>>> at /usr/src/sys/kern/subr_taskqueue.c:454
>>>>>> #198 0xffffffff80b49b58 in taskqueue_thread_loop
(arg=<value
>>>>>> optimized out>)
>>>>>> at /usr/src/sys/kern/subr_taskqueue.c:724
>>>>>> #199 0xffffffff80a9f255 in fork_exit (
>>>>>> callout=0xffffffff80b49a70
<taskqueue_thread_loop>,
>>>>>> arg=0xfffff8057caa72a0,
frame=0xfffffe0668222c00)
>>>>>> at /usr/src/sys/kern/kern_fork.c:1040
>>>>>> #200 0xffffffff80fb44ae in fork_trampoline ()
>>>>>> at /usr/src/sys/amd64/amd64/exception.S:611
>>>>>> #201 0x0000000000000000 in ?? ()
>>>>>> Current language: auto; currently minimal
>>>>>> (kgdb)
>>>>>>
>>>>>>
>>>>>>>> --
>>>>>>>> Karl Denninger
>>>>>>>> karl at denninger.net <mailto:karl at
denninger.net>
>>>>>>>> /The Market Ticker/
>>>>>>>> /[S/MIME encrypted email preferred]/
>>>>> _______________________________________________
>>>>> freebsd-stable at freebsd.org mailing list
>>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
>>>>> To unsubscribe, send any mail to
>>>>> "freebsd-stable-unsubscribe at freebsd.org"
>>> _______________________________________________
>>> freebsd-stable at freebsd.org mailing list
>>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
>>> To unsubscribe, send any mail to
>>> "freebsd-stable-unsubscribe at freebsd.org"
>
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at
freebsd.org"
--
Karl Denninger
karl at denninger.net <mailto:karl at denninger.net>
/The Market Ticker/
/[S/MIME encrypted email preferred]/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2996 bytes
Desc: S/MIME Cryptographic Signature
URL:
<http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20161017/36747b47/attachment-0001.bin>