Nils Goroll
2009-Sep-12 10:03 UTC
[zfs-discuss] zpool hanging after I/O error (usb) on all mirror components
Hi, yesterday, my backup zpool on two usb drives failed for USB errors (I don''t know if connecting my iPhone plays a role) while scrubbing the pool. This lead to all I/O on the zpool hanging, including df, zpool and zfs commands. init 6 would also hang due to bootadm hanging: process id 16329 is /sbin/bootadm -ea update_all, parent process is 1 [...] stack trace is: unix: _resume_from_idle+0xfa () unix: resume (?) unix: swtch+0x147 () genunix: cv_wait+0x61 (0xffffff0206e199e0,0xffffff0206e199d8) zfs: zio_wait+0x5d (0xffffff0206e19720) zfs: dbuf_read+0x1e8 (0xffffff01f62d3070,0,2) zfs: dmu_buf_hold+0x96 (0xffffff02302e51e8,3,0,0,0xffffff0008a66538) zfs: zap_lockdir+0x67 (0xffffff02302e51e8,3,0,1,1,B_FALSE,0xffffff0008a665c8) zfs: zap_lookup_norm+0x55 (0xffffff02302e51e8,3,0xffffff0008a66a70,8,1, 0xffffff0008a66778,MT_EXACT,0,0,0) [...] At this point, I didn''t know of a better way than to hard reboot the machine. This happened on snv_111 running as an xvm dom0. My question is if anyone is interested in analyzing this. I''ll provide some detail here and in case anyone is in interested, I could provide crash and core-dumps. Thank you, Nils -- Here''s an excerpt from /var/adm/messages: Sep 11 22:01:48 haggis usba: [ID 912658 kern.info] USB 2.0 device (usb5ac,1292) operating at hi speed (USB 2.x) on USB 2.0 root hub: device at 5, usb_mid8 at bus add ress 2 Sep 11 22:01:48 haggis usba: [ID 349649 kern.info] Apple Inc. iPhone 05a9686f1c32714410c9c14f94f800dbeaa830dd Sep 11 22:01:48 haggis genunix: [ID 936769 kern.info] usb_mid8 is /pci at 0,0/pci103c,30c5 at 1d,7/device at 5 Sep 11 22:01:48 haggis genunix: [ID 408114 kern.info] /pci at 0,0/pci103c,30c5 at 1d,7/device at 5 (usb_mid8) online Sep 11 22:01:48 haggis usba: [ID 349649 kern.info] usba: no driver found for interface 0 (nodename: ''image'') of Apple Inc. iPhone 05a9686f1c32714410c9c14f9 4f800dbeaa830dd Sep 11 22:01:48 haggis ata: [ID 776753 kern.info] ata: simplex controller. DMA on channel 0 PIO on channel 1 Sep 11 22:01:48 haggis unix: [ID 954099 kern.info] NOTICE: IRQ18 is being shared by drivers with different interrupt levels. Sep 11 22:01:48 haggis This may result in reduced system performance. Sep 11 22:01:48 haggis xpv_psm: [ID 805372 kern.info] xVM_psm: ide (ata) instance 3 irq 0xf vector 0x98 ioapic 0x1 intin 0xf is bound to cpu 0 Sep 11 22:01:48 haggis unix: [ID 954099 kern.info] NOTICE: IRQ18 is being shared by drivers with different interrupt levels. Sep 11 22:01:48 haggis This may result in reduced system performance. Sep 11 22:01:49 haggis ata: [ID 776753 kern.info] ata: simplex controller. DMA on channel 0 PIO on channel 1 Sep 11 22:01:49 haggis unix: [ID 954099 kern.info] NOTICE: IRQ18 is being shared by drivers with different interrupt levels. Sep 11 22:01:49 haggis This may result in reduced system performance. Sep 11 22:01:49 haggis unix: [ID 954099 kern.info] NOTICE: IRQ18 is being shared by drivers with different interrupt levels. Sep 11 22:01:49 haggis This may result in reduced system performance. Sep 11 22:01:49 haggis xpv_psm: [ID 805372 kern.info] xVM_psm: ide (ata) instance 3 irq 0xf vector 0x98 ioapic 0x1 intin 0xf is bound to cpu 1 Sep 11 22:02:00 haggis xntpd[8808]: [ID 898919 daemon.error] server returns a permission denied error Sep 11 22:07:00 haggis last message repeated 23 times Sep 11 22:08:00 haggis xntpd[8808]: [ID 898919 daemon.error] server returns a permission denied error Sep 11 22:09:00 haggis last message repeated 7 times Sep 11 22:09:29 haggis scsi: [ID 107833 kern.warning] WARNING: /pci at 0,0/pci103c,30c5 at 1d,7/storage at 3/disk at 0,0 (sd2): Sep 11 22:09:29 haggis Command failed to complete...Device is gone Sep 11 22:09:29 haggis genunix: [ID 408114 kern.info] /pci at 0,0/pci103c,30c5 at 1d,7/device at 5 (usb_mid8) offline Sep 11 22:09:29 haggis ata: [ID 776753 kern.info] ata: simplex controller. DMA on channel 0 PIO on channel 1 Sep 11 22:09:29 haggis unix: [ID 954099 kern.info] NOTICE: IRQ18 is being shared by drivers with different interrupt levels. Sep 11 22:09:29 haggis This may result in reduced system performance. Sep 11 22:09:29 haggis xpv_psm: [ID 805372 kern.info] xVM_psm: ide (ata) instance 3 irq 0xf vector 0x98 ioapic 0x1 intin 0xf is bound to cpu 0 Sep 11 22:09:29 haggis unix: [ID 954099 kern.info] NOTICE: IRQ18 is being shared by drivers with different interrupt levels. Sep 11 22:09:29 haggis This may result in reduced system performance. Sep 11 22:09:30 haggis ata: [ID 776753 kern.info] ata: simplex controller. DMA on channel 0 PIO on channel 1 Sep 11 22:09:30 haggis unix: [ID 954099 kern.info] NOTICE: IRQ18 is being shared by drivers with different interrupt levels. Sep 11 22:09:30 haggis This may result in reduced system performance. Sep 11 22:09:30 haggis unix: [ID 954099 kern.info] NOTICE: IRQ18 is being shared by drivers with different interrupt levels. Sep 11 22:09:30 haggis This may result in reduced system performance. Sep 11 22:09:30 haggis xpv_psm: [ID 805372 kern.info] xVM_psm: ide (ata) instance 3 irq 0xf vector 0x98 ioapic 0x1 intin 0xf is bound to cpu 1 Sep 11 22:09:32 haggis scsi: [ID 107833 kern.warning] WARNING: /pci at 0,0/pci103c,30c5 at 1d,7/storage at 1/disk at 0,0 (sd3): Sep 11 22:09:32 haggis Command failed to complete...Device is gone Sep 11 22:09:56 haggis fmd: [ID 441519 daemon.error] SUNW-MSG-ID: ZFS-8000-FD, TYPE: Fault, VER: 1, SEVERITY: Major Sep 11 22:09:56 haggis EVENT-TIME: Fri Sep 11 22:09:55 CEST 2009 Sep 11 22:09:56 haggis PLATFORM: HP Compaq 8510w , CSN: CNU9054HB3, HOSTNAME: haggis Sep 11 22:09:56 haggis SOURCE: zfs-diagnosis, REV: 1.0 Sep 11 22:09:56 haggis EVENT-ID: 2769ff11-472f-ef45-9bc4-b13f35af6d67 Sep 11 22:09:56 haggis DESC: The number of I/O errors associated with a ZFS device exceeded Sep 11 22:09:56 haggis acceptable levels. Refer to http://sun.com/msg/ZFS-8000-FD for more information. Sep 11 22:09:56 haggis AUTO-RESPONSE: The device has been offlined and marked as faulted. An attempt Sep 11 22:09:56 haggis will be made to activate a hot spare if available. Sep 11 22:09:56 haggis IMPACT: Fault tolerance of the pool may be compromised. Sep 11 22:09:56 haggis REC-ACTION: Run ''zpool status -x'' and replace the bad device. Sep 11 22:10:00 haggis xntpd[8808]: [ID 898919 daemon.error] server returns a permission denied error Sep 11 22:14:00 haggis last message repeated 19 times Sep 11 22:15:00 haggis xntpd[8808]: [ID 898919 daemon.error] server returns a permission denied error Sep 11 22:15:00 haggis last message repeated 3 times Sep 11 22:15:52 haggis fmd: [ID 441519 daemon.error] SUNW-MSG-ID: ZFS-8000-FD, TYPE: Fault, VER: 1, SEVERITY: Major Sep 11 22:15:52 haggis EVENT-TIME: Fri Sep 11 22:15:51 CEST 2009 Sep 11 22:15:52 haggis PLATFORM: HP Compaq 8510w , CSN: CNU9054HB3, HOSTNAME: haggis Sep 11 22:15:52 haggis SOURCE: zfs-diagnosis, REV: 1.0 Sep 11 22:15:52 haggis EVENT-ID: 18fe6343-f480-e092-9240-85d12fab0d14 Sep 11 22:15:52 haggis DESC: The number of I/O errors associated with a ZFS device exceeded Sep 11 22:15:52 haggis acceptable levels. Refer to http://sun.com/msg/ZFS-8000-FD for more information. Sep 11 22:15:52 haggis AUTO-RESPONSE: The device has been offlined and marked as faulted. An attempt Sep 11 22:15:52 haggis will be made to activate a hot spare if available. Sep 11 22:15:52 haggis IMPACT: Fault tolerance of the pool may be compromised. Sep 11 22:15:52 haggis REC-ACTION: Run ''zpool status -x'' and replace the bad device. Sep 11 22:15:53 haggis fmd: [ID 441519 daemon.error] SUNW-MSG-ID: ZFS-8000-HC, TYPE: Error, VER: 1, SEVERITY: Major Sep 11 22:15:53 haggis EVENT-TIME: Fri Sep 11 22:15:53 CEST 2009 Sep 11 22:15:53 haggis PLATFORM: HP Compaq 8510w , CSN: CNU9054HB3, HOSTNAME: haggis Sep 11 22:15:53 haggis SOURCE: zfs-diagnosis, REV: 1.0 Sep 11 22:15:53 haggis EVENT-ID: d1ce414b-8e17-cc26-b9b8-a785ae8a2fe2 Sep 11 22:15:53 haggis DESC: The ZFS pool has experienced currently unrecoverable I/O Sep 11 22:15:53 haggis failures. Refer to http://sun.com/msg/ZFS-8000-HC for more information. Sep 11 22:15:53 haggis AUTO-RESPONSE: No automated response will be taken. Sep 11 22:15:53 haggis IMPACT: Read and write I/Os cannot be serviced. Sep 11 22:15:53 haggis REC-ACTION: Make sure the affected devices are connected, then run Sep 11 22:15:53 haggis ''zpool clear''. So it looks like the two USB devices went offline at 23:04:16 and 23:04:21. This morning, I found the system in this state and tried zpool clear rbackup2 c3t0d0 which would hang. "zfs list" and "df -h" would also hang. I got a live crashdump and coredumps of the df, zfs and zpool processes. Importing the pool was fine, and scrubbing restarted: root at haggis:~# zpool status pool: rbackup2 state: ONLINE scrub: scrub in progress for 0h1m, 0.83% done, 2h46m to go config: NAME STATE READ WRITE CKSUM rbackup2 ONLINE 0 0 0 mirror ONLINE 0 0 0 c3t0d0 ONLINE 0 0 0 c2t0d0 ONLINE 0 0 0 errors: No known data errors act output of the crash dump shows a couple of threads waiting for an event which obviously didn''t arrive (excerpt): *** Kernel thread: thread addr ffffff000740fc60, proc addr fffffffffbc5f8f0, lwp addr 0 t_state is 0x1 - TS_SLEEP Scheduling info: t_pri is 0x3c, t_epri is 0, t_cid is 0 scheduling class is: SYS t_disp_time: is 0x721bb5, 0t7478197 last ran: 7 secs ago on cpu 0 t_stk ffffff000740fc60 stack trace is: unix: _resume_from_idle+0xfa () unix: resume (?) unix: swtch+0x147 () genunix: cv_timedwait+0xba (0xffffff01ba98d258,0xffffff01ba98d218,0x72274c) zfs: txg_thread_wait+0x7b (0xffffff01ba98d210,0xffffff000740fbf0, 0xffffff01ba98d258,0xb97) zfs: txg_sync_thread+0x114 (0xffffff01ba98d080) unix: thread_start+8 () *** process id 16549 is df -h, parent process is 16540 uid is 0, gid is 0 thread addr ffffff01bd99ae40, proc addr ffffff01d685b1c0, lwp addr ffffff01e0474210 t_state is 0x1 - TS_SLEEP Scheduling info: t_pri is 0x3c, t_epri is 0, t_cid is 0x4 scheduling class is: IA t_disp_time: is 0x71e7e9, 0t7464937 last ran: 2 mins 19 secs ago on cpu 0 t_stk ffffff000934df10 stack trace is: unix: _resume_from_idle+0xfa () unix: resume (?) unix: swtch+0x147 () genunix: cv_wait+0x61 (0xffffff01f6242468,0xffffff01f6242448) zfs: dbuf_read+0x237 (0xffffff01f62423f0,0,2) zfs: dmu_buf_hold+0x96 (0xffffff0239dfa870,3,0,0,0xffffff000934d7e8) zfs: zap_lockdir+0x67 (0xffffff0239dfa870,3,0,1,1,B_FALSE,0xffffff000934da78) zfs: zap_cursor_retrieve+0x74 (0xffffff000934da70,0xffffff000934daa0) zfs: dsl_prop_get_all+0x12a (0xffffff01df0d13a0,0xffffff000934dc18,B_FALSE) zfs: zfs_ioc_objset_stats+0x68 (?) zfs: zfsdev_ioctl+0x10b (0xb600000000,0x5a11,0x8046510,0x100003, 0xffffff01d6fc9028,0xffffff000934dde4) genunix: cdev_ioctl+0x45 (0xb600000000,0x5a11,0x8046510,0x100003, 0xffffff01d6fc9028,0xffffff000934dde4) specfs: spec_ioctl+0x83 (0xffffff01cd7f9400,0x5a11,0x8046510,0x100003, 0xffffff01d6fc9028,0xffffff000934dde4,0) genunix: fop_ioctl+0x7b (0xffffff01cd7f9400,0x5a11,0x8046510,0x100003, 0xffffff01d6fc9028,0xffffff000934dde4,0) genunix: ioctl+0x18e (3,0x5a11,0x8046510) unix: sys_syscall32+0x141 () *** process id 16210 is zpool clear rbackup2 c3t0d0, parent process is 1 uid is 0, gid is 0 thread addr ffffff01c17d8580, proc addr ffffff01d6859900, lwp addr ffffff01e5f0d950 t_state is 0x1 - TS_SLEEP Scheduling info: t_pri is 0x31, t_epri is 0, t_cid is 0x4 scheduling class is: IA t_disp_time: is 0x3357de, 0t3364830 last ran: 11 hours 25 mins 40 secs ago on cpu 0 t_stk ffffff00096aef10 stack trace is: unix: _resume_from_idle+0xfa () unix: resume (?) unix: swtch+0x147 () genunix: cv_wait+0x61 (0xffffff01e678959a,0xffffff01e6789558) zfs: txg_wait_synced+0x7f (0xffffff01e67893c0,0x27078) zfs: dsl_sync_task_group_wait+0xee (0xffffff01e0420b18) zfs: dsl_sync_task_do+0x65 (0xffffff01e67893c0,0,spa_history_log_sync, 0xffffff01e7e18000,0xffffff00096ae7d0,0) zfs: spa_history_log+0x6f (0xffffff01e7e18000,0xffffff01eeb05540,1) zfs: zfs_log_history+0x5b (0xffffff01e03b3000) zfs: zfsdev_ioctl+0x138 (0xb600000000,0x5a22,0x8042520,0x100003, 0xffffff01e79a64c8,0xffffff00096aede4) genunix: cdev_ioctl+0x45 (0xb600000000,0x5a22,0x8042520,0x100003, 0xffffff01e79a64c8,0xffffff00096aede4) specfs: spec_ioctl+0x83 (0xffffff01cd7f9400,0x5a22,0x8042520,0x100003, 0xffffff01e79a64c8,0xffffff00096aede4,0) genunix: fop_ioctl+0x7b (0xffffff01cd7f9400,0x5a22,0x8042520,0x100003, 0xffffff01e79a64c8,0xffffff00096aede4,0) genunix: ioctl+0x18e (3,0x5a22,0x8042520) unix: sys_syscall32+0x141 () *** process id 1443 is /usr/lib/fm/fmd/fmd, parent process is 1 uid is 0, gid is 0 thread addr ffffff01c17c7060, proc addr ffffff01ba5aa8d8, lwp addr ffffff01c7f012d0 t_state is 0x1 - TS_SLEEP Scheduling info: t_pri is 0x3b, t_epri is 0, t_cid is 0x1 scheduling class is: TS t_disp_time: is 0x2ef4a8, 0t3077288 last ran: 12 hours 13 mins 36 secs ago on cpu 0 t_stk ffffff0007c0ff10 stack trace is: unix: _resume_from_idle+0xfa () unix: resume (?) unix: swtch+0x147 () genunix: cv_wait+0x61 (0xffffff01e678959a,0xffffff01e6789558) zfs: txg_wait_synced+0x7f (0xffffff01e67893c0,0) zfs: spa_vdev_state_exit+0x4e (0xffffff01e7e18000,0xffffff01dfd83700,0) zfs: vdev_fault+0xde (0xffffff01e7e18000,0x594655ece244786e) zfs: zfs_ioc_vdev_set_state+0xa6 (?) zfs: zfsdev_ioctl+0x10b (0xb600000000,0x5a0d,0xfc60da74,0x100003, 0xffffff01cd825e50,0xffffff0007c0fde4) genunix: cdev_ioctl+0x45 (0xb600000000,0x5a0d,0xfc60da74,0x100003, 0xffffff01cd825e50,0xffffff0007c0fde4) specfs: spec_ioctl+0x83 (0xffffff01cd7f9400,0x5a0d,0xfc60da74,0x100003, 0xffffff01cd825e50,0xffffff0007c0fde4,0) genunix: fop_ioctl+0x7b (0xffffff01cd7f9400,0x5a0d,0xfc60da74,0x100003, 0xffffff01cd825e50,0xffffff0007c0fde4,0) genunix: ioctl+0x18e (0x2a,0x5a0d,0xfc60da74) unix: sys_syscall32+0x141 () *** Kernel thread: thread addr ffffff0009365c60, proc addr fffffffffbc5f8f0, lwp addr 0 t_state is 0x1 - TS_SLEEP Scheduling info: t_pri is 0x3c, t_epri is 0, t_cid is 0 scheduling class is: SYS t_disp_time: is 0x2ef4a8, 0t3077288 last ran: 12 hours 13 mins 36 secs ago on cpu 0 t_stk ffffff0009365c60 stack trace is: unix: _resume_from_idle+0xfa () unix: resume (?) unix: swtch+0x147 () genunix: cv_wait+0x61 (0xffffff01f6fd0fc8,0xffffff01f6fd0fc0) zfs: zio_wait+0x5d (0xffffff01f6fd0d08) zfs: dbuf_read+0x1e8 (0xffffff0211c90748,0,2) zfs: dmu_buf_hold+0x96 (0xffffff0239dfa870,1,0,0,0xffffff00093657e8) zfs: zap_lockdir+0x67 (0xffffff0239dfa870,1,0xffffff01fc4b4080,RW_WRITER,1,1, 0xffffff0009365878) zfs: zap_update+0x5b (0xffffff0239dfa870,1,"scrub_bookmark",8,4, 0xffffff01e6789510,0xffffff01fc4b4080) zfs: dsl_pool_scrub_sync+0x28e (0xffffff01e67893c0,0xffffff01fc4b4080) zfs: dsl_pool_sync+0x18c (0xffffff01e67893c0,0x27077) zfs: spa_sync+0x32a (0xffffff01e7e18000,0x27077) zfs: txg_sync_thread+0x265 (0xffffff01e67893c0) unix: thread_start+8 () *** Kernel thread: thread addr ffffff0007cc0c60, proc addr fffffffffbc5f8f0, lwp addr 0 t_state is 0x1 - TS_SLEEP Scheduling info: t_pri is 0x3c, t_epri is 0, t_cid is 0 scheduling class is: SYS t_disp_time: is 0x2e5479, 0t3036281 last ran: 12 hours 20 mins 26 secs ago on cpu 0 t_stk ffffff0007cc0c60 stack trace is: unix: _resume_from_idle+0xfa () unix: resume (?) unix: swtch+0x147 () genunix: cv_wait+0x61 (0xffffff01e678959c,0xffffff01e6789558) zfs: txg_thread_wait+0x5f (0xffffff01e6789550,0xffffff0007cc0bf0, 0xffffff01e678959c,0) zfs: txg_quiesce_thread+0x94 (0xffffff01e67893c0) unix: thread_start+8 () *** process id 16581 is zfs list, parent process is 16552 uid is 0, gid is 0 thread addr ffffff01d44f6e60, proc addr ffffff01d6ef76f8, lwp addr ffffff01e5eff830 t_state is 0x1 - TS_SLEEP Scheduling info: t_pri is 0x3c, t_epri is 0, t_cid is 0x4 scheduling class is: IA t_disp_time: is 0x72096a, 0t7473514 last ran: 54 secs ago on cpu 0x1 t_stk ffffff000922ef10 stack trace is: unix: _resume_from_idle+0xfa () unix: resume (?) unix: swtch+0x147 () genunix: cv_wait+0x61 (0xffffff01f6242468,0xffffff01f6242448) zfs: dbuf_read+0x237 (0xffffff01f62423f0,0,2) zfs: dmu_buf_hold+0x96 (0xffffff0239dfa870,3,0,0,0xffffff000922e7e8) zfs: zap_lockdir+0x67 (0xffffff0239dfa870,3,0,1,1,B_FALSE,0xffffff000922ea78) zfs: zap_cursor_retrieve+0x74 (0xffffff000922ea70,0xffffff000922eaa0) zfs: dsl_prop_get_all+0x12a (0xffffff01df0d1710,0xffffff000922ec18,B_FALSE) zfs: zfs_ioc_objset_stats+0x68 (?) zfs: zfsdev_ioctl+0x10b (0xb600000000,0x5a11,0x8046af0,0x100003, 0xffffff01dcbabcf0,0xffffff000922ede4) genunix: cdev_ioctl+0x45 (0xb600000000,0x5a11,0x8046af0,0x100003, 0xffffff01dcbabcf0,0xffffff000922ede4) specfs: spec_ioctl+0x83 (0xffffff01cd7f9400,0x5a11,0x8046af0,0x100003, 0xffffff01dcbabcf0,0xffffff000922ede4,0) genunix: fop_ioctl+0x7b (0xffffff01cd7f9400,0x5a11,0x8046af0,0x100003, 0xffffff01dcbabcf0,0xffffff000922ede4,0) genunix: ioctl+0x18e (3,0x5a11,0x8046af0) unix: sys_syscall32+0x141 () *** process id 16354 is df -h, parent process is 16348 uid is 0, gid is 0 thread addr ffffff01c17d5700, proc addr ffffff0208b2b070, lwp addr ffffff01e5f06330 t_state is 0x1 - TS_SLEEP Scheduling info: t_pri is 0x3c, t_epri is 0, t_cid is 0x4 scheduling class is: IA t_disp_time: is 0x713060, 0t7417952 last ran: 10 mins 9 secs ago on cpu 0x1 t_stk ffffff000964ef10 stack trace is: unix: _resume_from_idle+0xfa () unix: resume (?) unix: swtch+0x147 () genunix: cv_wait+0x61 (0xffffff02076c30f0,0xffffff02076c30e8) zfs: zio_wait+0x5d (0xffffff02076c2e30) zfs: dbuf_read+0x1e8 (0xffffff01f62423f0,0,2) zfs: dmu_buf_hold+0x96 (0xffffff0239dfa870,3,0,0,0xffffff000964e7e8) zfs: zap_lockdir+0x67 (0xffffff0239dfa870,3,0,1,1,B_FALSE,0xffffff000964ea78) zfs: zap_cursor_retrieve+0x74 (0xffffff000964ea70,0xffffff000964eaa0) zfs: dsl_prop_get_all+0x12a (0xffffff01e9b28240,0xffffff000964ec18,B_FALSE) zfs: zfs_ioc_objset_stats+0x68 (?) zfs: zfsdev_ioctl+0x10b (0xb600000000,0x5a11,0x8046510,0x100003, 0xffffff01dcbab030,0xffffff000964ede4) genunix: cdev_ioctl+0x45 (0xb600000000,0x5a11,0x8046510,0x100003, 0xffffff01dcbab030,0xffffff000964ede4) specfs: spec_ioctl+0x83 (0xffffff01cd7f9400,0x5a11,0x8046510,0x100003, 0xffffff01dcbab030,0xffffff000964ede4,0) genunix: fop_ioctl+0x7b (0xffffff01cd7f9400,0x5a11,0x8046510,0x100003, 0xffffff01dcbab030,0xffffff000964ede4,0) genunix: ioctl+0x18e (3,0x5a11,0x8046510) unix: sys_syscall32+0x141 () *** process id 195 is /usr/lib/sysevent/syseventd, parent process is 1 uid is 0, gid is 0 thread addr ffffff01c1b83a80, proc addr ffffff01c1616190, lwp addr ffffff01c1be5780 t_state is 0x1 - TS_SLEEP Scheduling info: t_pri is 0x3b, t_epri is 0, t_cid is 0x1 scheduling class is: TS t_disp_time: is 0x334b76, 0t3361654 last ran: 11 hours 26 mins 12 secs ago on cpu 0x1 t_stk ffffff0008d3bf10 stack trace is: unix: _resume_from_idle+0xfa () unix: resume (?) unix: swtch+0x147 () genunix: cv_wait+0x61 (0xffffff01e678959a,0xffffff01e6789558) zfs: txg_wait_synced+0x7f (0xffffff01e67893c0,0) zfs: spa_vdev_state_exit+0x4e (0xffffff01e7e18000,0xffffff01dfd83700,0) zfs: vdev_online+0x110 (0xffffff01e7e18000,0x594655ece244786e,3, 0xffffff0008d3bc14) zfs: zfs_ioc_vdev_set_state+0x7c (?) zfs: zfsdev_ioctl+0x10b (0xb600000000,0x5a0d,0xfdbdc820,0x100003, 0xffffff01cd8253d0,0xffffff0008d3bde4) genunix: cdev_ioctl+0x45 (0xb600000000,0x5a0d,0xfdbdc820,0x100003, 0xffffff01cd8253d0,0xffffff0008d3bde4) specfs: spec_ioctl+0x83 (0xffffff01cd7f9400,0x5a0d,0xfdbdc820,0x100003, 0xffffff01cd8253d0,0xffffff0008d3bde4,0) genunix: fop_ioctl+0x7b (0xffffff01cd7f9400,0x5a0d,0xfdbdc820,0x100003, 0xffffff01cd8253d0,0xffffff0008d3bde4,0) genunix: ioctl+0x18e (5,0x5a0d,0xfdbdc820) unix: sys_syscall32+0x141 ()