The panic was due to stack exhaustion, why it was so deep not looked.
On 20/07/2016 15:32, Karl Denninger wrote:> The panic occurred during a zfs send/receive operation for system
> backup..... I've seen this one before, unfortunately, and it appears
> that it's still there -- may be related to
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=207464
>
> On 7/20/2016 09:26, Karl Denninger wrote:
>> Came out of the ZFS system; this machine is running *unpatched* (none
of
>> my changes are in the kernel); I have a good core dump if further trace
>> information will help.
>>
>> Dump header from device: /dev/gpt/dump
>> Architecture: amd64
>> Architecture Version: 2
>> Dump Length: 9582022656
>> Blocksize: 512
>> Dumptime: Wed Jul 20 09:10:31 2016
>> Hostname: NewFS.denninger.net
>> Magic: FreeBSD Kernel Dump
>> Version String: FreeBSD 11.0-BETA1 #0 r302489: Sat Jul 9 10:15:24
CDT
>> 2016
>> karl at NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP
>> Panic String: double fault
>> Dump Parity: 445173880
>> Bounds: 4
>> Dump Status: good
>>
>> NewFS.denninger.net dumped core - see /var/crash/vmcore.4
>>
>> Wed Jul 20 09:18:15 CDT 2016
>>
>> FreeBSD NewFS.denninger.net 11.0-BETA1 FreeBSD 11.0-BETA1 #0 r302489:
>> Sat Jul 9 10:15:24 CDT 2016
>> karl at NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP amd64
>>
>> panic: double fault
>>
>> GNU gdb 6.1.1 [FreeBSD]
>> Copyright 2004 Free Software Foundation, Inc.
>> GDB is free software, covered by the GNU General Public License, and
you are
>> welcome to change it and/or distribute copies of it under certain
>> conditions.
>> Type "show copying" to see the conditions.
>> There is absolutely no warranty for GDB. Type "show
warranty" for details.
>> This GDB was configured as "amd64-marcel-freebsd"...
>>
>> Unread portion of the kernel message buffer:
>>
>> Fatal double fault
>> rip = 0xffffffff8220bc92
>> rsp = 0xfffffe0667f0ef30
>> rbp = 0xfffffe0667f0f3c0
>> cpuid = 14; apic id = 34
>> panic: double fault
>> cpuid = 14
>> KDB: stack backtrace:
>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
>> 0xfffffe0649db8e30
>> vpanic() at vpanic+0x182/frame 0xfffffe0649db8eb0
>> panic() at panic+0x43/frame 0xfffffe0649db8f10
>> dblfault_handler() at dblfault_handler+0xa2/frame 0xfffffe0649db8f30
>> Xdblfault() at Xdblfault+0xac/frame 0xfffffe0649db8f30
>> --- trap 0x17, rip = 0xffffffff8220bc92, rsp = 0xfffffe0667f0ef30, rbp
>> 0xfffffe0667f0f3c0 ---
>> vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x22/frame
>> 0xfffffe0667f0f3c0
>> vdev_queue_io_done() at vdev_queue_io_done+0x83/frame
0xfffffe0667f0f400
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f430
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f480
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0f4e0
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f530
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f0f570
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f5a0
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f5f0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0f650
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f6a0
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f0f6e0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f710
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f760
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0f7c0
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f810
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f0f850
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f880
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f8d0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0f930
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f980
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f0f9c0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f9f0
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fa40
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0faa0
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0faf0
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f0fb30
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0fb60
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fbb0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0fc10
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fc60
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f0fca0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0fcd0
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fd20
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0fd80
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fdd0
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f0fe10
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0fe40
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fe90
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0fef0
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0ff40
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f0ff80
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0ffb0
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10000
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10060
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f100b0
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f100f0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10120
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10170
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f101d0
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10220
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f10260
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10290
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f102e0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10340
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10390
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f103d0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10400
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10450
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f104b0
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10500
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f10540
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10570
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f105c0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10620
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10670
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f106b0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f106e0
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10730
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10790
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f107e0
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f10820
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10850
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f108a0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10900
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10950
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f10990
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f109c0
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10a10
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10a70
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10ac0
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f10b00
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10b30
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10b80
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10be0
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10c30
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f10c70
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10ca0
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10cf0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10d50
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10da0
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f10de0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10e10
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10e60
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10ec0
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10f10
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f10f50
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10f80
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10fd0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11030
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11080
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f110c0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f110f0
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11140
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f111a0
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f111f0
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f11230
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11260
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f112b0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11310
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11360
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f113a0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f113d0
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11420
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11480
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f114d0
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f11510
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11540
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11590
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f115f0
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11640
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f11680
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f116b0
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11700
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11760
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f117b0
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f117f0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11820
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11870
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f118d0
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11920
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f11960
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11990
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f119e0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11a40
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11a90
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f11ad0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11b00
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11b50
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11bb0
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11c00
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f11c40
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11c70
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11cc0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11d20
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11d70
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f11db0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11de0
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11e30
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11e90
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11ee0
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f11f20
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11f50
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11fa0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f12000
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12050
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f12090
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f120c0
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12110
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f12170
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f121c0
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f12200
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12230
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12280
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f122e0
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12330
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f12370
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f123a0
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f123f0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f12450
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f124a0
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f124e0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12510
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12560
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f125c0
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12610
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f12650
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12680
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f126d0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f12730
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12780
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f127c0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f127f0
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12840
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f128a0
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f128f0
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f12930
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12960
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f129b0
>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f12a10
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12a60
>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame
0xfffffe0667f12aa0
>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12ad0
>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12b20
>> taskqueue_run_locked() at taskqueue_run_locked+0x14a/frame
>> 0xfffffe0667f12b80
>> taskqueue_thread_loop() at taskqueue_thread_loop+0xe8/frame
>> 0xfffffe0667f12bb0
>> fork_exit() at fork_exit+0x85/frame 0xfffffe0667f12bf0
>> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0667f12bf0
>> --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
>> Uptime: 10d21h36m8s
>> Dumping 9138 out of 24531
>> MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%
>>
>> Reading symbols from /boot/kernel/zfs.ko...Reading symbols from
>> /usr/lib/debug//boot/kernel/zfs.ko.debug...done.
>> done.
>> Loaded symbols for /boot/kernel/zfs.ko
>> Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from
>> /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done.
>> done.
>> Loaded symbols for /boot/kernel/opensolaris.ko
>> Reading symbols from /boot/kernel/geom_eli.ko...Reading symbols from
>> /usr/lib/debug//boot/kernel/geom_eli.ko.debug...done.
>> done.
>> Loaded symbols for /boot/kernel/geom_eli.ko
>> Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from
>> /usr/lib/debug//boot/kernel/geom_mirror.ko.debug...done.
>> done.
>> Loaded symbols for /boot/kernel/geom_mirror.ko
>> Reading symbols from /boot/kernel/aesni.ko...Reading symbols from
>> /usr/lib/debug//boot/kernel/aesni.ko.debug...done.
>> done.
>> Loaded symbols for /boot/kernel/aesni.ko
>> Reading symbols from /boot/kernel/uhid.ko...Reading symbols from
>> /usr/lib/debug//boot/kernel/uhid.ko.debug...done.
>> done.
>> Loaded symbols for /boot/kernel/uhid.ko
>> Reading symbols from /boot/kernel/ums.ko...Reading symbols from
>> /usr/lib/debug//boot/kernel/ums.ko.debug...done.
>> done.
>> Loaded symbols for /boot/kernel/ums.ko
>> Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from
>> /usr/lib/debug//boot/kernel/ipfw.ko.debug...done.
>> done.
>> Loaded symbols for /boot/kernel/ipfw.ko
>> Reading symbols from /boot/kernel/ipfw_nat.ko...Reading symbols from
>> /usr/lib/debug//boot/kernel/ipfw_nat.ko.debug...done.
>> done.
>> Loaded symbols for /boot/kernel/ipfw_nat.ko
>> Reading symbols from /boot/kernel/libalias.ko...Reading symbols from
>> /usr/lib/debug//boot/kernel/libalias.ko.debug...done.
>> done.
>> Loaded symbols for /boot/kernel/libalias.ko
>> Reading symbols from /boot/kernel/daemon_saver.ko...Reading symbols
from
>> /usr/lib/debug//boot/kernel/daemon_saver.ko.debug...done.
>> done.
>> Loaded symbols for /boot/kernel/daemon_saver.ko
>> #0 doadump (textdump=1) at pcpu.h:221
>> 221 pcpu.h: No such file or directory.
>> in pcpu.h
>> (kgdb) #0 doadump (textdump=1) at pcpu.h:221
>> #1 0xffffffff80ae548e in kern_reboot (howto=260)
>> at /usr/src/sys/kern/kern_shutdown.c:366
>> #2 0xffffffff80ae5a5b in vpanic (fmt=<value optimized out>,
>> ap=<value optimized out>) at
/usr/src/sys/kern/kern_shutdown.c:759
>> #3 0xffffffff80ae5893 in panic (fmt=0x0)
>> at /usr/src/sys/kern/kern_shutdown.c:690
>> #4 0xffffffff80fc9692 in dblfault_handler (frame=<value optimized
out>)
>> at /usr/src/sys/amd64/amd64/trap.c:868
>> #5 0xffffffff80fac15c in Xdblfault ()
>> at /usr/src/sys/amd64/amd64/exception.S:292
>> #6 0xffffffff8220bc92 in vdev_queue_io_to_issue
(vq=0xfffff804d86833e0)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:585
>> #7 0xffffffff8220c6e3 in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:881
>> #8 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8018859c000)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #9 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #10 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8018859c000)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #11 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #12 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #13 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff801df1f43b8)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #14 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #15 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff801df1f43b8)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #16 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #17 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #18 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8024583e770)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #19 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #20 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8024583e770)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #21 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #22 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #23 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff802e12df770)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #24 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #25 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff802e12df770)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #26 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #27 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #28 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8004f81f3b8)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #29 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #30 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8004f81f3b8)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #31 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #32 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #33 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8021bdf7b28)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #34 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #35 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8021bdf7b28)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #36 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #37 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #38 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80165851770)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #39 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #40 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80165851770)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #41 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #42 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #43 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8060de643b8)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #44 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #45 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8060de643b8)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #46 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #47 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #48 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff800793c0770)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #49 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #50 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff800793c0770)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #51 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #52 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #53 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8026917cb28)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #54 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #55 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8026917cb28)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #56 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #57 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #58 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8012408d3b8)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #59 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #60 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8012408d3b8)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #61 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #62 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #63 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff802bfc9ab28)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #64 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #65 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff802bfc9ab28)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #66 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #67 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #68 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff802dda5db28)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #69 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #70 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff802dda5db28)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #71 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #72 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #73 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff800ae13d000)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #74 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #75 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff800ae13d000)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #76 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #77 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #78 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff804a70efb28)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #79 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #80 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff804a70efb28)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #81 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #82 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #83 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80485f5f000)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #84 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #85 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80485f5f000)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #86 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #87 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #88 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff800666fc3b8)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #89 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #90 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff800666fc3b8)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #91 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #92 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #93 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8010da3ab28)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #94 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #95 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8010da3ab28)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #96 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #97 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #98 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80315f73770)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #99 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #100 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80315f73770)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #101 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #102 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #103 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff806011703b8)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #104 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #105 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff806011703b8)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #106 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #107 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #108 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff805964f1770)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #109 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #110 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff805964f1770)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #111 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #112 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #113 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8023902f770)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #114 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #115 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8023902f770)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #116 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #117 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #118 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff803bc7e2b28)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #119 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #120 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff803bc7e2b28)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #121 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #122 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #123 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff804dfbf43b8)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #124 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #125 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff804dfbf43b8)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #126 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #127 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #128 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8005be4a3b8)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #129 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #130 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8005be4a3b8)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #131 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #132 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #133 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8004ff79000)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #134 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #135 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8004ff79000)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #136 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #137 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #138 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff801aa2a2000)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #139 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #140 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff801aa2a2000)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #141 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #142 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #143 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff800592003b8)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #144 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #145 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff800592003b8)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #146 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #147 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #148 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80202d11770)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #149 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #150 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80202d11770)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #151 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #152 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #153 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80392d3c3b8)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #154 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #155 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80392d3c3b8)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #156 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #157 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #158 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80550fdd000)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #159 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #160 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80550fdd000)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #161 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #162 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #163 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8055fd15770)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #164 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #165 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8055fd15770)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #166 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #167 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #168 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80576f99770)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #169 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #170 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80576f99770)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #171 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #172 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #173 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff802269bf000)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #174 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #175 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff802269bf000)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #176 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #177 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #178 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8003d7973b8)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #179 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #180 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8003d7973b8)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #181 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #182 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #183 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80108754770)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #184 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #185 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80108754770)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #186 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #187 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #188 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff804cfdf7770)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #189 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #190 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff804cfdf7770)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #191 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #192 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #193 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff801469a13b8)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #194 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #195 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff801469a13b8)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #196 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #197 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized
out>)
>> at
>>
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #198 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff805dcd9d3b8)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #199 0xffffffff82225d96 in zio_execute (zio=<value optimized
out>)
>> at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #200 0xffffffff80b415aa in taskqueue_run_locked (queue=<value
optimized
>> out>)
>> at /usr/src/sys/kern/subr_taskqueue.c:465
>> #201 0xffffffff80b424b8 in taskqueue_thread_loop (arg=<value
optimized out>)
>> at /usr/src/sys/kern/subr_taskqueue.c:719
>> #202 0xffffffff80a9ad85 in fork_exit (
>> callout=0xffffffff80b423d0 <taskqueue_thread_loop>,
>> arg=0xfffff8037db02580, frame=0xfffffe0667f12c00)
>> at /usr/src/sys/kern/kern_fork.c:1038
>> #203 0xffffffff80fac59e in fork_trampoline ()
>> at /usr/src/sys/amd64/amd64/exception.S:611
>> #204 0x0000000000000000 in ?? ()
>> Current language: auto; currently minimal
>> (kgdb)
>>