thr3ads.net - zfs discuss - [zfs-discuss] ZFS Locking Up periodically [Apr 2009]

If this information is useful, please help other people find it:
Share via:

Andrew Robert Nicols

2009-Apr-01 08:50 UTC

[zfs-discuss] ZFS Locking Up periodically

I''ve recently re-installed an X4500 running Nevada b109 and have been
experiencing ZFS lock ups regularly (perhaps once every 2-3 days).

The machine is a backup server and receives hourly ZFS snapshots from
another thumper - as such, the amount of zfs activity tends to be
reasonably high. After about 48 - 72 hours, the file system seems to lock
up and I''m unable to do anything with the zfs system - e.g. zfs list,
df on
the file system in question, zfs receive, etc. zpool does still list
information and targetted zfs list does work. NFS also locks up. I''m
unable
to test whether I can write as the file system is read only.

The structure of the pool is something like:
/thumperpool <-- Does not lock up
/thumperpool/mnt <-- Does lock up and is the file system receiving lots of
snapshots

The server is currently in it''s locked state so if anyone can suggest
useful diagnostics to run on it while it''s like this, please get back
to me
asap. I will need to restart the box so that our backups aren''t too out
of
sync this afternoon.

Thanks in advance,

Andrew Nicols

--
Systems Developer

e: andrew.nicols at luns.net.uk
im: a.nicols at jabber.lancs.ac.uk
t: +44 (0)1524 5 10147

Lancaster University Network Services is a limited company registered in
England and Wales. Registered number: 4311892. Registered office:
University House, Lancaster University, Lancaster, LA1 4YW
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: Digital signature
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090401/b97f7f10/attachment.bin>

Andrew Robert Nicols

2009-Apr-02 08:08 UTC

head link

[zfs-discuss] ZFS Locking Up periodically

Hi Victor,

I had to reboot yesterday in the end before I got your message but it''s
just locked up again.

On Wed, Apr 01, 2009 at 06:49:14PM +0600, Victor Latushkin
wrote:> At the very least you can try to collect threadlist with
>
> echo "::threadlist -v" | mdb -k
Attached. Is there anything else which would help to diagnose the issue?
> If you need to reboot your box anyway, use "reboot -d" -
it''ll generate
> kernel crash dump which can be analyses later with mdb and/or  
> SolarisCAT.
Thanks - useful to know. Will that force a reboot even if everything is
locking on ZFS? In the past I''ve had to perform a hard reset from the
SP.
> I suspect that this behaviour may be caused by bug in the ohci driver.  
> If you do not use rKVM for console and instead redirect it to serial  
> port, you can easily exclude ohci driver from loading by adding to the 
> /etc/system
>
> exclude: drv/ohci
I''ll add this since we only use the rKVM at install time to mount ISOs.
> I do not have bug number handy but if it helps, I''ll look it up
for you
> next week
If you do  happen to come across it, that would be handy. I''ve had a
quick
search of defect.opensolaris.org.

Andrew

-- 
Systems Developer

e: andrew.nicols at luns.net.uk
im: a.nicols at jabber.lancs.ac.uk
t: +44 (0)1524 5 10147

Lancaster University Network Services is a limited company registered in
England and Wales. Registered number: 4311892. Registered office:
University House, Lancaster University, Lancaster, LA1 4YW
-------------- next part --------------
            ADDR             PROC              LWP CLS PRI            WCHAN
fffffffffbc2c6e0 fffffffffbc2ba70 fffffffffbc2e2e0   0  96                0
  PC: _resume_from_idle+0xf1    CMD: sched
  stack pointer for thread fffffffffbc2c6e0: fffffffffbc4d380
  [ fffffffffbc4d380 _resume_from_idle+0xf1() ]
    swtch+0x160()
    sched+0x3fd()
    main+0x437()
    _locore_start+0x92()

ffffff001e405c60 fffffffffbc2ba70                0   0  -1                0
  PC: _resume_from_idle+0xf1    THREAD: idle()
  stack pointer for thread ffffff001e405c60: ffffff001e405bf0
  [ ffffff001e405bf0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    idle+0xc4()
    thread_start+8()

ffffff001e40bc60 fffffffffbc2ba70                0   0  60 fffffffffbcb60e8
  PC: _resume_from_idle+0xf1    THREAD: thread_reaper()
  stack pointer for thread ffffff001e40bc60: ffffff001e40bb80
  [ ffffff001e40bb80 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    thread_reaper+0xb1()
    thread_start+8()

ffffff001e411c60 fffffffffbc2ba70                0   0  60 ffffff04da3e0e90
  PC: _resume_from_idle+0xf1    TASKQ: kmem_move_taskq
  stack pointer for thread ffffff001e411c60: ffffff001e411b60
  [ ffffff001e411b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e417c60 fffffffffbc2ba70                0   0  60 ffffff04da3e0db8
  PC: _resume_from_idle+0xf1    TASKQ: kmem_taskq
  stack pointer for thread ffffff001e417c60: ffffff001e417b60
  [ ffffff001e417b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e41dc60 fffffffffbc2ba70                0   0  60 ffffff04da3e0ce0
  PC: _resume_from_idle+0xf1    TASKQ: pseudo_nexus_enum_tq
  stack pointer for thread ffffff001e41dc60: ffffff001e41db60
  [ ffffff001e41db60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e423c60 fffffffffbc2ba70                0   0  60 ffffff04da3e0c08
  PC: _resume_from_idle+0xf1    TASKQ: scsi_vhci_nexus_enum_tq
  stack pointer for thread ffffff001e423c60: ffffff001e423b60
  [ ffffff001e423b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e429c60 fffffffffbc2ba70                0   0  60 ffffff04da3e0b30
  PC: _resume_from_idle+0xf1    TASKQ: mdi_taskq
  stack pointer for thread ffffff001e429c60: ffffff001e429b60
  [ ffffff001e429b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x143()
    thread_start+8()

ffffff001e42fc60 fffffffffbc2ba70                0   0  60 ffffff04da3e0b30
  PC: _resume_from_idle+0xf1    TASKQ: mdi_taskq
  stack pointer for thread ffffff001e42fc60: ffffff001e42fb60
  [ ffffff001e42fb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x143()
    thread_start+8()

ffffff001e435c60 fffffffffbc2ba70                0   0  60 ffffff04da3e0b30
  PC: _resume_from_idle+0xf1    TASKQ: mdi_taskq
  stack pointer for thread ffffff001e435c60: ffffff001e435b60
  [ ffffff001e435b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x143()
    thread_start+8()

ffffff001e43bc60 fffffffffbc2ba70                0   0  60 ffffff04da3e0b30
  PC: _resume_from_idle+0xf1    TASKQ: mdi_taskq
  stack pointer for thread ffffff001e43bc60: ffffff001e43bb60
  [ ffffff001e43bb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x143()
    thread_start+8()

ffffff001e441c60 fffffffffbc2ba70                0   0  60 ffffff04da3e0b30
  PC: _resume_from_idle+0xf1    TASKQ: mdi_taskq
  stack pointer for thread ffffff001e441c60: ffffff001e441b60
  [ ffffff001e441b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x143()
    thread_start+8()

ffffff001e447c60 fffffffffbc2ba70                0   0  60 ffffff04da3e0b30
  PC: _resume_from_idle+0xf1    TASKQ: mdi_taskq
  stack pointer for thread ffffff001e447c60: ffffff001e447b60
  [ ffffff001e447b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x143()
    thread_start+8()

ffffff001e44dc60 fffffffffbc2ba70                0   0  60 ffffff04da3e0b30
  PC: _resume_from_idle+0xf1    TASKQ: mdi_taskq
  stack pointer for thread ffffff001e44dc60: ffffff001e44db60
  [ ffffff001e44db60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x143()
    thread_start+8()

ffffff001e453c60 fffffffffbc2ba70                0   0  60 ffffff04da3e0b30
  PC: _resume_from_idle+0xf1    TASKQ: mdi_taskq
  stack pointer for thread ffffff001e453c60: ffffff001e453b60
  [ ffffff001e453b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x143()
    thread_start+8()

ffffff001e459c60 fffffffffbc2ba70                0   0  60 ffffff04da3e0a58
  PC: _resume_from_idle+0xf1    TASKQ: vhci_taskq
  stack pointer for thread ffffff001e459c60: ffffff001e459b60
  [ ffffff001e459b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e45fc60 fffffffffbc2ba70                0   0  60 ffffff04da3e0980
  PC: _resume_from_idle+0xf1    TASKQ: vhci_update_pathstates
  stack pointer for thread ffffff001e45fc60: ffffff001e45fb60
  [ ffffff001e45fb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e465c60 fffffffffbc2ba70                0   0  60 ffffff04da3e0980
  PC: _resume_from_idle+0xf1    TASKQ: vhci_update_pathstates
  stack pointer for thread ffffff001e465c60: ffffff001e465b60
  [ ffffff001e465b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e46bc60 fffffffffbc2ba70                0   0  60 ffffff04da3e0980
  PC: _resume_from_idle+0xf1    TASKQ: vhci_update_pathstates
  stack pointer for thread ffffff001e46bc60: ffffff001e46bb60
  [ ffffff001e46bb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e471c60 fffffffffbc2ba70                0   0  60 ffffff04da3e0980
  PC: _resume_from_idle+0xf1    TASKQ: vhci_update_pathstates
  stack pointer for thread ffffff001e471c60: ffffff001e471b60
  [ ffffff001e471b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e477c60 fffffffffbc2ba70                0   0  60 ffffff04da3e0980
  PC: _resume_from_idle+0xf1    TASKQ: vhci_update_pathstates
  stack pointer for thread ffffff001e477c60: ffffff001e477b60
  [ ffffff001e477b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e47dc60 fffffffffbc2ba70                0   0  60 ffffff04da3e0980
  PC: _resume_from_idle+0xf1    TASKQ: vhci_update_pathstates
  stack pointer for thread ffffff001e47dc60: ffffff001e47db60
  [ ffffff001e47db60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e483c60 fffffffffbc2ba70                0   0  60 ffffff04da3e0980
  PC: _resume_from_idle+0xf1    TASKQ: vhci_update_pathstates
  stack pointer for thread ffffff001e483c60: ffffff001e483b60
  [ ffffff001e483b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e489c60 fffffffffbc2ba70                0   0  60 ffffff04da3e0980
  PC: _resume_from_idle+0xf1    TASKQ: vhci_update_pathstates
  stack pointer for thread ffffff001e489c60: ffffff001e489b60
  [ ffffff001e489b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e48fc60 fffffffffbc2ba70                0   0  60 ffffff04da3e08a8
  PC: _resume_from_idle+0xf1    TASKQ: isa_nexus_enum_tq
  stack pointer for thread ffffff001e48fc60: ffffff001e48fb60
  [ ffffff001e48fb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e4d1c60 fffffffffbc2ba70                0   0  99 ffffff04da3e07d0
  PC: _resume_from_idle+0xf1    TASKQ: timeout_taskq
  stack pointer for thread ffffff001e4d1c60: ffffff001e4d1b60
  [ ffffff001e4d1b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x143()
    thread_start+8()

ffffff001e4d7c60 fffffffffbc2ba70                0   0  99 ffffff04da3e07d0
  PC: _resume_from_idle+0xf1    TASKQ: timeout_taskq
  stack pointer for thread ffffff001e4d7c60: ffffff001e4d7b60
  [ ffffff001e4d7b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x143()
    thread_start+8()

ffffff001e4ddc60 fffffffffbc2ba70                0   0  99 ffffff04da3e07d0
  PC: _resume_from_idle+0xf1    TASKQ: timeout_taskq
  stack pointer for thread ffffff001e4ddc60: ffffff001e4ddb60
  [ ffffff001e4ddb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x143()
    thread_start+8()

ffffff001e4e3c60 fffffffffbc2ba70                0   0  99 ffffff04da3e07d0
  PC: _resume_from_idle+0xf1    TASKQ: timeout_taskq
  stack pointer for thread ffffff001e4e3c60: ffffff001e4e3b60
  [ ffffff001e4e3b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x143()
    thread_start+8()

ffffff001e4e9c60 fffffffffbc2ba70                0   0  99 fffffffffbc8b710
  PC: _resume_from_idle+0xf1    THREAD: timeout_taskq_thread()
  stack pointer for thread ffffff001e4e9c60: ffffff001e4e9b80
  [ ffffff001e4e9b80 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    timeout_taskq_thread+0xa7()
    thread_start+8()

ffffff001e4efc60 fffffffffbc2ba70                0   0  99 ffffff04da3e06f8
  PC: _resume_from_idle+0xf1    TASKQ: callout_taskq
  stack pointer for thread ffffff001e4efc60: ffffff001e4efb60
  [ ffffff001e4efb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x143()
    thread_start+8()

ffffff001e4f5c60 fffffffffbc2ba70                0   0  99 ffffff04da3e06f8
  PC: _resume_from_idle+0xf1    TASKQ: callout_taskq
  stack pointer for thread ffffff001e4f5c60: ffffff001e4f5b60
  [ ffffff001e4f5b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x143()
    thread_start+8()

ffffff001eb7dc60 fffffffffbc2ba70                0   0  60 ffffff05b28cc0f0
  PC: _resume_from_idle+0xf1    TASKQ: system_taskq
  stack pointer for thread ffffff001eb7dc60: ffffff001eb7db30
  [ ffffff001eb7db30 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait+0xba()
    taskq_d_thread+0x15e()
    thread_start+8()

ffffff001fbdbc60 fffffffffbc2ba70                0   0  60 ffffff05f7451388
  PC: _resume_from_idle+0xf1    TASKQ: system_taskq
  stack pointer for thread ffffff001fbdbc60: ffffff001fbdbb30
  [ ffffff001fbdbb30 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait+0xba()
    taskq_d_thread+0x15e()
    thread_start+8()

ffffff001fc0ec60 fffffffffbc2ba70                0   0  60 ffffff052d705650
  PC: _resume_from_idle+0xf1    TASKQ: system_taskq
  stack pointer for thread ffffff001fc0ec60: ffffff001fc0eb30
  [ ffffff001fc0eb30 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait+0xba()
    taskq_d_thread+0x15e()
    thread_start+8()

ffffff001e59dc60 fffffffffbc2ba70                0   0  60 ffffff057bcd1298
  PC: _resume_from_idle+0xf1    TASKQ: system_taskq
  stack pointer for thread ffffff001e59dc60: ffffff001e59db30
  [ ffffff001e59db30 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait+0xba()
    taskq_d_thread+0x15e()
    thread_start+8()

ffffff001e4fbc60 fffffffffbc2ba70                0   0  60 ffffff04da3e0620
  PC: _resume_from_idle+0xf1    TASKQ: system_taskq
  stack pointer for thread ffffff001e4fbc60: ffffff001e4fbb60
  [ ffffff001e4fbb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001eab3c60 fffffffffbc2ba70                0   0  60 ffffff04df1af710
  PC: _resume_from_idle+0xf1    TASKQ: USB_device_0_pipehndl_tq_2_
  stack pointer for thread ffffff001eab3c60: ffffff001eab3b60
  [ ffffff001eab3b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001eab9c60 fffffffffbc2ba70                0   0  60 ffffff04df1af710
  PC: _resume_from_idle+0xf1    TASKQ: USB_device_0_pipehndl_tq_2_
  stack pointer for thread ffffff001eab9c60: ffffff001eab9b60
  [ ffffff001eab9b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001eabfc60 fffffffffbc2ba70                0   0  60 ffffff04df1af710
  PC: _resume_from_idle+0xf1    TASKQ: USB_device_0_pipehndl_tq_2_
  stack pointer for thread ffffff001eabfc60: ffffff001eabfb60
  [ ffffff001eabfb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001eac5c60 fffffffffbc2ba70                0   0  60 ffffff04df1af710
  PC: _resume_from_idle+0xf1    TASKQ: USB_device_0_pipehndl_tq_2_
  stack pointer for thread ffffff001eac5c60: ffffff001eac5b60
  [ ffffff001eac5b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001ecc9c60 fffffffffbc2ba70                0   0  60 ffffff04df1af560
  PC: _resume_from_idle+0xf1    TASKQ: usb_mid_nexus_enum_tq
  stack pointer for thread ffffff001ecc9c60: ffffff001ecc9b60
  [ ffffff001ecc9b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001ede3c60 fffffffffbc2ba70                0   0  60 ffffff04df1af488
  PC: _resume_from_idle+0xf1    TASKQ: USB_hid_81_pipehndl_tq_0
  stack pointer for thread ffffff001ede3c60: ffffff001ede3b60
  [ ffffff001ede3b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001ede9c60 fffffffffbc2ba70                0   0  60 ffffff04df1af488
  PC: _resume_from_idle+0xf1    TASKQ: USB_hid_81_pipehndl_tq_0
  stack pointer for thread ffffff001ede9c60: ffffff001ede9b60
  [ ffffff001ede9b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001ee32c60 fffffffffbc2ba70                0   0  60 ffffff04df1af3b0
  PC: _resume_from_idle+0xf1    TASKQ: USB_hid_82_pipehndl_tq_1
  stack pointer for thread ffffff001ee32c60: ffffff001ee32b60
  [ ffffff001ee32b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001ee38c60 fffffffffbc2ba70                0   0  60 ffffff04df1af3b0
  PC: _resume_from_idle+0xf1    TASKQ: USB_hid_82_pipehndl_tq_1
  stack pointer for thread ffffff001ee38c60: ffffff001ee38b60
  [ ffffff001ee38b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001ecf6c60 fffffffffbc2ba70                0   0  99 ffffff04e04fea28
  PC: _resume_from_idle+0xf1    THREAD: mac_srs_worker()
  stack pointer for thread ffffff001ecf6c60: ffffff001ecf6b80
  [ ffffff001ecf6b80 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    mac_srs_worker+0x1cb()
    thread_start+8()

ffffff001e7cbc60 fffffffffbc2ba70                0   0  99 ffffff04e04fe428
  PC: _resume_from_idle+0xf1    THREAD: mac_srs_worker()
  stack pointer for thread ffffff001e7cbc60: ffffff001e7cbb80
  [ ffffff001e7cbb80 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    mac_srs_worker+0x1cb()
    thread_start+8()

ffffff001eccfc60 fffffffffbc2ba70                0   0  99 ffffff04e04fde28
  PC: _resume_from_idle+0xf1    THREAD: mac_srs_worker()
  stack pointer for thread ffffff001eccfc60: ffffff001eccfb80
  [ ffffff001eccfb80 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    mac_srs_worker+0x1cb()
    thread_start+8()

ffffff001f0d9c60 fffffffffbc2ba70                0   0  99 ffffff04e04fde2a
  PC: _resume_from_idle+0xf1    THREAD: mac_rx_srs_poll_ring()
  stack pointer for thread ffffff001f0d9c60: ffffff001f0d9b60
  [ ffffff001f0d9b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    mac_rx_srs_poll_ring+0xa0()
    thread_start+8()

ffffff001ec31c60 fffffffffbc2ba70                0   0  99 ffffff04e518de68
  PC: _resume_from_idle+0xf1    THREAD: mac_soft_ring_worker()
  stack pointer for thread ffffff001ec31c60: ffffff001ec31b80
  [ ffffff001ec31b80 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    mac_soft_ring_worker+0xb0()
    thread_start+8()

ffffff0020244c60 fffffffffbc2ba70                0   0  99 ffffff04e518dce8
  PC: _resume_from_idle+0xf1    THREAD: mac_soft_ring_worker()
  stack pointer for thread ffffff0020244c60: ffffff0020244b80
  [ ffffff0020244b80 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    mac_soft_ring_worker+0xb0()
    thread_start+8()

ffffff002024ac60 fffffffffbc2ba70                0   0  99 ffffff04e518db68
  PC: _resume_from_idle+0xf1    THREAD: mac_soft_ring_worker()
  stack pointer for thread ffffff002024ac60: ffffff002024ab80
  [ ffffff002024ab80 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    mac_soft_ring_worker+0xb0()
    thread_start+8()

ffffff0020250c60 fffffffffbc2ba70                0   0  99 ffffff04e518d9e8
  PC: _resume_from_idle+0xf1    THREAD: mac_soft_ring_worker()
  stack pointer for thread ffffff0020250c60: ffffff0020250b80
  [ ffffff0020250b80 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    mac_soft_ring_worker+0xb0()
    thread_start+8()

ffffff001fc2ac60 fffffffffbc2ba70                0   0  99 ffffff04e518d868
  PC: _resume_from_idle+0xf1    THREAD: mac_soft_ring_worker()
  stack pointer for thread ffffff001fc2ac60: ffffff001fc2ab80
  [ ffffff001fc2ab80 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    mac_soft_ring_worker+0xb0()
    thread_start+8()

ffffff001fc30c60 fffffffffbc2ba70                0   0  99 ffffff04e518d6e8
  PC: _resume_from_idle+0xf1    THREAD: mac_soft_ring_worker()
  stack pointer for thread ffffff001fc30c60: ffffff001fc30b80
  [ ffffff001fc30b80 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    mac_soft_ring_worker+0xb0()
    thread_start+8()

ffffff001ec7ac60 fffffffffbc2ba70                0   0  60 ffffff04df1af638
  PC: _resume_from_idle+0xf1    TASKQ: USB_device_0_pipehndl_tq_4_
  stack pointer for thread ffffff001ec7ac60: ffffff001ec7ab60
  [ ffffff001ec7ab60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001ec80c60 fffffffffbc2ba70                0   0  60 ffffff04df1af638
  PC: _resume_from_idle+0xf1    TASKQ: USB_device_0_pipehndl_tq_4_
  stack pointer for thread ffffff001ec80c60: ffffff001ec80b60
  [ ffffff001ec80b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001ec86c60 fffffffffbc2ba70                0   0  60 ffffff04df1af638
  PC: _resume_from_idle+0xf1    TASKQ: USB_device_0_pipehndl_tq_4_
  stack pointer for thread ffffff001ec86c60: ffffff001ec86b60
  [ ffffff001ec86b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001ec8cc60 fffffffffbc2ba70                0   0  60 ffffff04df1af638
  PC: _resume_from_idle+0xf1    TASKQ: USB_device_0_pipehndl_tq_4_
  stack pointer for thread ffffff001ec8cc60: ffffff001ec8cb60
  [ ffffff001ec8cb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001eeacc60 fffffffffbc2ba70                0   0  60 ffffff04df1af2d8
  PC: _resume_from_idle+0xf1    TASKQ: hubd_nexus_enum_tq
  stack pointer for thread ffffff001eeacc60: ffffff001eeacb60
  [ ffffff001eeacb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001f0cdc60 fffffffffbc2ba70                0   0  60 ffffff04df1af200
  PC: _resume_from_idle+0xf1    TASKQ: USB_hubd_81_pipehndl_tq_0
  stack pointer for thread ffffff001f0cdc60: ffffff001f0cdb60
  [ ffffff001f0cdb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001f0d3c60 fffffffffbc2ba70                0   0  60 ffffff04df1af200
  PC: _resume_from_idle+0xf1    TASKQ: USB_hubd_81_pipehndl_tq_0
  stack pointer for thread ffffff001f0d3c60: ffffff001f0d3b60
  [ ffffff001f0d3b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001fc3cc60 fffffffffbc2ba70                0   0  99 ffffff04deb46bd0
  PC: _resume_from_idle+0xf1    THREAD: squeue_worker()
  stack pointer for thread ffffff001fc3cc60: ffffff001fc3cb80
  [ ffffff001fc3cb80 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    squeue_worker+0xc0()
    thread_start+8()

ffffff001fc42c60 fffffffffbc2ba70                0   0  99 ffffff04deb46bd2
  PC: _resume_from_idle+0xf1    THREAD: squeue_polling_thread()
  stack pointer for thread ffffff001fc42c60: ffffff001fc42b50
  [ ffffff001fc42b50 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    squeue_polling_thread+0x90()
    thread_start+8()

ffffff001fc56c60 fffffffffbc2ba70                0   0  99 ffffff04deb46b10
  PC: _resume_from_idle+0xf1    THREAD: squeue_worker()
  stack pointer for thread ffffff001fc56c60: ffffff001fc56b80
  [ ffffff001fc56b80 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    squeue_worker+0xc0()
    thread_start+8()

ffffff001fc5cc60 fffffffffbc2ba70                0   0  99 ffffff04deb46b12
  PC: _resume_from_idle+0xf1    THREAD: squeue_polling_thread()
  stack pointer for thread ffffff001fc5cc60: ffffff001fc5cb50
  [ ffffff001fc5cb50 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    squeue_polling_thread+0x90()
    thread_start+8()

ffffff001e519c60 fffffffffbc2ba70                0   0  60 fffffffffbce810c
  PC: _resume_from_idle+0xf1    THREAD: streams_bufcall_service()
  stack pointer for thread ffffff001e519c60: ffffff001e519b90
  [ ffffff001e519b90 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    streams_bufcall_service+0x152()
    thread_start+8()

ffffff001e51fc60 fffffffffbc2ba70                0   0  60 fffffffffbcb59c0
  PC: _resume_from_idle+0xf1    THREAD: streams_qbkgrnd_service()
  stack pointer for thread ffffff001e51fc60: ffffff001e51fb90
  [ ffffff001e51fb90 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    streams_qbkgrnd_service+0xaa()
    thread_start+8()

ffffff001e525c60 fffffffffbc2ba70                0   0  60 fffffffffbcb59c2
  PC: _resume_from_idle+0xf1    THREAD: streams_sqbkgrnd_service()
  stack pointer for thread ffffff001e525c60: ffffff001e525b80
  [ ffffff001e525b80 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    streams_sqbkgrnd_service+0x99()
    thread_start+8()

ffffff001e52bc60 fffffffffbc2ba70                0   0  60 fffffffffbc2ea20
  PC: _resume_from_idle+0xf1    THREAD: page_capture_thread()
  stack pointer for thread ffffff001e52bc60: ffffff001e52bb80
  [ ffffff001e52bb80 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait+0xba()
    page_capture_thread+0xe2()
    thread_start+8()

ffffff001e531c60 fffffffffbc2ba70                0   0  60 fffffffffbd07bf0
  PC: _resume_from_idle+0xf1    THREAD: arc_reclaim_thread()
  stack pointer for thread ffffff001e531c60: ffffff001e531b50
  [ ffffff001e531b50 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait+0xba()
    arc_reclaim_thread+0x17b()
    thread_start+8()

ffffff001e537c60 fffffffffbc2ba70                0   0  60 fffffffffbd07ef2
  PC: _resume_from_idle+0xf1    THREAD: l2arc_feed_thread()
  stack pointer for thread ffffff001e537c60: ffffff001e537b50
  [ ffffff001e537b50 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait+0xba()
    l2arc_feed_thread+0xa5()
    thread_start+8()

ffffff001e53dc60 fffffffffbc2ba70                0   0  60 fffffffffbcdb8d0
  PC: _resume_from_idle+0xf1    THREAD: pm_dep_thread()
  stack pointer for thread ffffff001e53dc60: ffffff001e53db80
  [ ffffff001e53db80 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    pm_dep_thread+0x95()
    thread_start+8()

ffffff001e543c60 fffffffffbc2ba70                0   0  60 ffffff04da3e0548
  PC: _resume_from_idle+0xf1    TASKQ: ppm_nexus_enum_tq
  stack pointer for thread ffffff001e543c60: ffffff001e543b60
  [ ffffff001e543b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e549c60 fffffffffbc2ba70                0   0  60 ffffff04da3e0470
  PC: _resume_from_idle+0xf1    TASKQ: pci_nexus_enum_tq
  stack pointer for thread ffffff001e549c60: ffffff001e549b60
  [ ffffff001e549b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e54fc60 fffffffffbc2ba70                0   0  60 ffffff04da3e0398
  PC: _resume_from_idle+0xf1    TASKQ: pci_pci_nexus_enum_tq
  stack pointer for thread ffffff001e54fc60: ffffff001e54fb60
  [ ffffff001e54fb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e555c60 fffffffffbc2ba70                0   0  60 ffffff04da3e02c0
  PC: _resume_from_idle+0xf1    TASKQ: marvell88sx_nexus_enum_tq
  stack pointer for thread ffffff001e555c60: ffffff001e555b60
  [ ffffff001e555b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e55bc60 fffffffffbc2ba70                0   0  60 ffffff04da3e01e8
  PC: _resume_from_idle+0xf1    TASKQ: pci11ab_11ab_3
  stack pointer for thread ffffff001e55bc60: ffffff001e55bb60
  [ ffffff001e55bb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e561c60 fffffffffbc2ba70                0   0  60 fffffffffbd1b910
  PC: _resume_from_idle+0xf1    THREAD: sata_event_daemon()
  stack pointer for thread ffffff001e561c60: ffffff001e561b90
  [ ffffff001e561b90 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait+0xba()
    sata_event_daemon+0x101()
    thread_start+8()

ffffff001e567c60 fffffffffbc2ba70                0   0  97 ffffff04da3e0110
  PC: _resume_from_idle+0xf1    TASKQ: sd_drv_taskq
  stack pointer for thread ffffff001e567c60: ffffff001e567b60
  [ ffffff001e567b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e56dc60 fffffffffbc2ba70                0   0  97 ffffff04da3e0110
  PC: _resume_from_idle+0xf1    TASKQ: sd_drv_taskq
  stack pointer for thread ffffff001e56dc60: ffffff001e56db60
  [ ffffff001e56db60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e573c60 fffffffffbc2ba70                0   0  97 ffffff04da3e0110
  PC: _resume_from_idle+0xf1    TASKQ: sd_drv_taskq
  stack pointer for thread ffffff001e573c60: ffffff001e573b60
  [ ffffff001e573b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e579c60 fffffffffbc2ba70                0   0  97 ffffff04da3e0110
  PC: _resume_from_idle+0xf1    TASKQ: sd_drv_taskq
  stack pointer for thread ffffff001e579c60: ffffff001e579b60
  [ ffffff001e579b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e57fc60 fffffffffbc2ba70                0   0  97 ffffff04da3e0110
  PC: _resume_from_idle+0xf1    TASKQ: sd_drv_taskq
  stack pointer for thread ffffff001e57fc60: ffffff001e57fb60
  [ ffffff001e57fb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e585c60 fffffffffbc2ba70                0   0  97 ffffff04da3e0110
  PC: _resume_from_idle+0xf1    TASKQ: sd_drv_taskq
  stack pointer for thread ffffff001e585c60: ffffff001e585b60
  [ ffffff001e585b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e58bc60 fffffffffbc2ba70                0   0  97 ffffff04da3e0110
  PC: _resume_from_idle+0xf1    TASKQ: sd_drv_taskq
  stack pointer for thread ffffff001e58bc60: ffffff001e58bb60
  [ ffffff001e58bb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e591c60 fffffffffbc2ba70                0   0  97 ffffff04da3e0110
  PC: _resume_from_idle+0xf1    TASKQ: sd_drv_taskq
  stack pointer for thread ffffff001e591c60: ffffff001e591b60
  [ ffffff001e591b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e597c60 fffffffffbc2ba70                0   0  97 ffffff04da3e0038
  PC: _resume_from_idle+0xf1    TASKQ: sd_rmw_taskq
  stack pointer for thread ffffff001e597c60: ffffff001e597b60
  [ ffffff001e597b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e5a3c60 fffffffffbc2ba70                0   0  97 ffffff04ddd49e98
  PC: _resume_from_idle+0xf1    TASKQ: xbuf_taskq
  stack pointer for thread ffffff001e5a3c60: ffffff001e5a3b60
  [ ffffff001e5a3b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e645c60 fffffffffbc2ba70                0   0  99 ffffff04ddd49478
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001e645c60: ffffff001e645b60
  [ ffffff001e645b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e64bc60 fffffffffbc2ba70                0   0  99 ffffff04ddd49550
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001e64bc60: ffffff001e64bb60
  [ ffffff001e64bb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e651c60 fffffffffbc2ba70                0   0  99 ffffff04ddd49628
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001e651c60: ffffff001e651b60
  [ ffffff001e651b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e657c60 fffffffffbc2ba70                0   0  99 ffffff04ddd49700
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001e657c60: ffffff001e657b60
  [ ffffff001e657b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e65dc60 fffffffffbc2ba70                0   0  99 ffffff04ddd49700
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001e65dc60: ffffff001e65db60
  [ ffffff001e65db60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e663c60 fffffffffbc2ba70                0   0  99 ffffff04ddd49700
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001e663c60: ffffff001e663b60
  [ ffffff001e663b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e669c60 fffffffffbc2ba70                0   0  99 ffffff04ddd49700
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001e669c60: ffffff001e669b60
  [ ffffff001e669b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e66fc60 fffffffffbc2ba70                0   0  99 ffffff04ddd49700
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001e66fc60: ffffff001e66fb60
  [ ffffff001e66fb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e675c60 fffffffffbc2ba70                0   0  99 ffffff04ddd49700
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001e675c60: ffffff001e675b60
  [ ffffff001e675b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e67bc60 fffffffffbc2ba70                0   0  99 ffffff04ddd49700
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001e67bc60: ffffff001e67bb60
  [ ffffff001e67bb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e681c60 fffffffffbc2ba70                0   0  99 ffffff04ddd49700
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001e681c60: ffffff001e681b60
  [ ffffff001e681b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e687c60 fffffffffbc2ba70                0   0  99 ffffff04ddd497d8
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001e687c60: ffffff001e687b60
  [ ffffff001e687b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e68dc60 fffffffffbc2ba70                0   0  99 ffffff04ddd497d8
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001e68dc60: ffffff001e68db60
  [ ffffff001e68db60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e693c60 fffffffffbc2ba70                0   0  99 ffffff04ddd497d8
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001e693c60: ffffff001e693b60
  [ ffffff001e693b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e699c60 fffffffffbc2ba70                0   0  99 ffffff04ddd497d8
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001e699c60: ffffff001e699b60
  [ ffffff001e699b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e69fc60 fffffffffbc2ba70                0   0  99 ffffff04ddd497d8
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001e69fc60: ffffff001e69fb60
  [ ffffff001e69fb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e6a5c60 fffffffffbc2ba70                0   0  99 ffffff04ddd497d8
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001e6a5c60: ffffff001e6a5b60
  [ ffffff001e6a5b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e6abc60 fffffffffbc2ba70                0   0  99 ffffff04ddd497d8
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001e6abc60: ffffff001e6abb60
  [ ffffff001e6abb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e6b1c60 fffffffffbc2ba70                0   0  99 ffffff04ddd497d8
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001e6b1c60: ffffff001e6b1b60
  [ ffffff001e6b1b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e6b7c60 fffffffffbc2ba70                0   0  99 ffffff04ddd498b0
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001e6b7c60: ffffff001e6b7b60
  [ ffffff001e6b7b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e6bdc60 fffffffffbc2ba70                0   0  99 ffffff04ddd49988
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001e6bdc60: ffffff001e6bdb60
  [ ffffff001e6bdb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e6c3c60 fffffffffbc2ba70                0   0  99 ffffff04ddd49a60
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001e6c3c60: ffffff001e6c3b60
  [ ffffff001e6c3b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e6c9c60 fffffffffbc2ba70                0   0  99 ffffff04ddd49b38
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001e6c9c60: ffffff001e6c9b60
  [ ffffff001e6c9b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e6cfc60 fffffffffbc2ba70                0   0  99 ffffff04ddd49c10
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001e6cfc60: ffffff001e6cfb60
  [ ffffff001e6cfb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e6d5c60 fffffffffbc2ba70                0   0  99 ffffff04ddd49ce8
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001e6d5c60: ffffff001e6d5b60
  [ ffffff001e6d5b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e6dbc60 fffffffffbc2ba70                0   0  99 ffffff04ddd49dc0
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001e6dbc60: ffffff001e6dbb60
  [ ffffff001e6dbb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e61bc60 fffffffffbc2ba70                0   0  60 ffffff04d826855c
  PC: _resume_from_idle+0xf1    THREAD: txg_quiesce_thread()
  stack pointer for thread ffffff001e61bc60: ffffff001e61bb10
  [ ffffff001e61bb10 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    txg_thread_wait+0x5f()
    txg_quiesce_thread+0x94()
    thread_start+8()

ffffff001e621c60 fffffffffbc2ba70                0   0  60 ffffff04d8268558
  PC: _resume_from_idle+0xf1    THREAD: txg_sync_thread()
  stack pointer for thread ffffff001e621c60: ffffff001e621ad0
  [ ffffff001e621ad0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait+0xba()
    txg_thread_wait+0x7b()
    txg_sync_thread+0x114()
    thread_start+8()

ffffff001e627c60 fffffffffbc2ba70                0   0  60 ffffff04ddd493a0
  PC: _resume_from_idle+0xf1    TASKQ: zil_clean
  stack pointer for thread ffffff001e627c60: ffffff001e627b60
  [ ffffff001e627b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e633c60 fffffffffbc2ba70                0   0  60 ffffff04ddd492c8
  PC: _resume_from_idle+0xf1    TASKQ: cpunex_nexus_enum_tq
  stack pointer for thread ffffff001e633c60: ffffff001e633b60
  [ ffffff001e633b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e639c60 fffffffffbc2ba70                0   0  60 ffffff04ddd491f0
  PC: _resume_from_idle+0xf1    TASKQ: IP_INJECT_QUEUE_OUT
  stack pointer for thread ffffff001e639c60: ffffff001e639b60
  [ ffffff001e639b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e63fc60 fffffffffbc2ba70                0   0  60 ffffff04ddd49118
  PC: _resume_from_idle+0xf1    TASKQ: IP_INJECT_QUEUE_IN
  stack pointer for thread ffffff001e63fc60: ffffff001e63fb60
  [ ffffff001e63fb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e5a9c60 fffffffffbc2ba70                0   0  60 ffffff04ddd49040
  PC: _resume_from_idle+0xf1    TASKQ: IP_NIC_EVENT_QUEUE
  stack pointer for thread ffffff001e5a9c60: ffffff001e5a9b60
  [ ffffff001e5a9b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e5afc60 fffffffffbc2ba70                0   0  60 ffffff04db5f5070
  PC: _resume_from_idle+0xf1    THREAD: ill_taskq_dispatch()
  stack pointer for thread ffffff001e5afc60: ffffff001e5afb40
  [ ffffff001e5afb40 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    ill_taskq_dispatch+0x11f()
    thread_start+8()

ffffff001e5b5c60 fffffffffbc2ba70                0   0  60 ffffff04db5f597c
  PC: _resume_from_idle+0xf1    THREAD: mcast_restart_timers_thread()
  stack pointer for thread ffffff001e5b5c60: ffffff001e5b5b30
  [ ffffff001e5b5b30 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    mcast_restart_timers_thread+0xd8()
    thread_start+8()

ffffff001e5bbc60 fffffffffbc2ba70                0   0  99 ffffff04db5ef818
  PC: _resume_from_idle+0xf1    THREAD: ipsec_loader()
  stack pointer for thread ffffff001e5bbc60: ffffff001e5bbb80
  [ ffffff001e5bbb80 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    ipsec_loader+0x9d()
    thread_start+8()

ffffff001e5c1c60 fffffffffbc2ba70                0   0  99 ffffff04deb46ed0
  PC: _resume_from_idle+0xf1    THREAD: squeue_worker()
  stack pointer for thread ffffff001e5c1c60: ffffff001e5c1b80
  [ ffffff001e5c1b80 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    squeue_worker+0xc0()
    thread_start+8()

ffffff001e5c7c60 fffffffffbc2ba70                0   0  99 ffffff04deb46ed2
  PC: _resume_from_idle+0xf1    THREAD: squeue_polling_thread()
  stack pointer for thread ffffff001e5c7c60: ffffff001e5c7b50
  [ ffffff001e5c7b50 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    squeue_polling_thread+0x90()
    thread_start+8()

ffffff001e5cdc60 fffffffffbc2ba70                0   0  60 ffffff04deb45ea0
  PC: _resume_from_idle+0xf1    TASKQ: tcp_taskq
  stack pointer for thread ffffff001e5cdc60: ffffff001e5cdb60
  [ ffffff001e5cdb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e5d3c60 fffffffffbc2ba70                0   0  60 ffffff04deb45dc8
  PC: _resume_from_idle+0xf1    TASKQ: sctp_taskq
  stack pointer for thread ffffff001e5d3c60: ffffff001e5d3b60
  [ ffffff001e5d3b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e5d9c60 fffffffffbc2ba70                0   0  60 ffffff04deb45cf0
  PC: _resume_from_idle+0xf1    TASKQ: sctp_def_recvq_taskq
  stack pointer for thread ffffff001e5d9c60: ffffff001e5d9b60
  [ ffffff001e5d9b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e5dfc60 fffffffffbc2ba70                0   0  60 ffffff04deb45cf0
  PC: _resume_from_idle+0xf1    TASKQ: sctp_def_recvq_taskq
  stack pointer for thread ffffff001e5dfc60: ffffff001e5dfb60
  [ ffffff001e5dfb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e5e5c60 fffffffffbc2ba70                0   0  60 ffffff04deb45cf0
  PC: _resume_from_idle+0xf1    TASKQ: sctp_def_recvq_taskq
  stack pointer for thread ffffff001e5e5c60: ffffff001e5e5b60
  [ ffffff001e5e5b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e5ebc60 fffffffffbc2ba70                0   0  60 ffffff04deb45cf0
  PC: _resume_from_idle+0xf1    TASKQ: sctp_def_recvq_taskq
  stack pointer for thread ffffff001e5ebc60: ffffff001e5ebb60
  [ ffffff001e5ebb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e5f1c60 fffffffffbc2ba70                0   0  60 fffffffffbd30e50
  PC: _resume_from_idle+0xf1    THREAD: dld_taskq_dispatch()
  stack pointer for thread ffffff001e5f1c60: ffffff001e5f1b80
  [ ffffff001e5f1b80 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    dld_taskq_dispatch+0xfc()
    thread_start+8()

ffffff001e5f7c60 fffffffffbc2ba70                0   0  60 ffffff04deb45c18
  PC: _resume_from_idle+0xf1    TASKQ: pci_nexus_enum_tq
  stack pointer for thread ffffff001e5f7c60: ffffff001e5f7b60
  [ ffffff001e5f7b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e5fdc60 fffffffffbc2ba70                0   0  60 ffffff04deb45b40
  PC: _resume_from_idle+0xf1    TASKQ: pci_pci_nexus_enum_tq
  stack pointer for thread ffffff001e5fdc60: ffffff001e5fdb60
  [ ffffff001e5fdb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e603c60 fffffffffbc2ba70                0   0  60 ffffff04deb45a68
  PC: _resume_from_idle+0xf1    TASKQ: ehci_nexus_enum_tq
  stack pointer for thread ffffff001e603c60: ffffff001e603b60
  [ ffffff001e603b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e609c60 fffffffffbc2ba70                0   0  60 ffffff04deb45990
  PC: _resume_from_idle+0xf1    TASKQ: USB_ehci_0_pipehndl_tq_0
  stack pointer for thread ffffff001e609c60: ffffff001e609b60
  [ ffffff001e609b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e60fc60 fffffffffbc2ba70                0   0  60 ffffff04deb45990
  PC: _resume_from_idle+0xf1    TASKQ: USB_ehci_0_pipehndl_tq_0
  stack pointer for thread ffffff001e60fc60: ffffff001e60fb60
  [ ffffff001e60fb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e6e1c60 fffffffffbc2ba70                0   0  60 ffffff04deb45990
  PC: _resume_from_idle+0xf1    TASKQ: USB_ehci_0_pipehndl_tq_0
  stack pointer for thread ffffff001e6e1c60: ffffff001e6e1b60
  [ ffffff001e6e1b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e6e7c60 fffffffffbc2ba70                0   0  60 ffffff04deb45990
  PC: _resume_from_idle+0xf1    TASKQ: USB_ehci_0_pipehndl_tq_0
  stack pointer for thread ffffff001e6e7c60: ffffff001e6e7b60
  [ ffffff001e6e7b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e6edc60 fffffffffbc2ba70                0   0  60 ffffff04deb458b8
  PC: _resume_from_idle+0xf1    TASKQ: USB_ehci_81_pipehndl_tq_0
  stack pointer for thread ffffff001e6edc60: ffffff001e6edb60
  [ ffffff001e6edb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e6f3c60 fffffffffbc2ba70                0   0  60 ffffff04deb458b8
  PC: _resume_from_idle+0xf1    TASKQ: USB_ehci_81_pipehndl_tq_0
  stack pointer for thread ffffff001e6f3c60: ffffff001e6f3b60
  [ ffffff001e6f3b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e6f9c60 fffffffffbc2ba70                0   0  60 ffffff04deb457e0
  PC: _resume_from_idle+0xf1    TASKQ: ohci_nexus_enum_tq
  stack pointer for thread ffffff001e6f9c60: ffffff001e6f9b60
  [ ffffff001e6f9b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e6ffc60 fffffffffbc2ba70                0   0  60 ffffff04deb45708
  PC: _resume_from_idle+0xf1    TASKQ: USB_ohci_0_pipehndl_tq_0
  stack pointer for thread ffffff001e6ffc60: ffffff001e6ffb60
  [ ffffff001e6ffb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e705c60 fffffffffbc2ba70                0   0  60 ffffff04deb45708
  PC: _resume_from_idle+0xf1    TASKQ: USB_ohci_0_pipehndl_tq_0
  stack pointer for thread ffffff001e705c60: ffffff001e705b60
  [ ffffff001e705b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e70bc60 fffffffffbc2ba70                0   0  60 ffffff04deb45708
  PC: _resume_from_idle+0xf1    TASKQ: USB_ohci_0_pipehndl_tq_0
  stack pointer for thread ffffff001e70bc60: ffffff001e70bb60
  [ ffffff001e70bb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e711c60 fffffffffbc2ba70                0   0  60 ffffff04deb45708
  PC: _resume_from_idle+0xf1    TASKQ: USB_ohci_0_pipehndl_tq_0
  stack pointer for thread ffffff001e711c60: ffffff001e711b60
  [ ffffff001e711b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e717c60 fffffffffbc2ba70                0   0  60 ffffff04deb45630
  PC: _resume_from_idle+0xf1    TASKQ: USB_ohci_81_pipehndl_tq_0
  stack pointer for thread ffffff001e717c60: ffffff001e717b60
  [ ffffff001e717b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e71dc60 fffffffffbc2ba70                0   0  60 ffffff04deb45630
  PC: _resume_from_idle+0xf1    TASKQ: USB_ohci_81_pipehndl_tq_0
  stack pointer for thread ffffff001e71dc60: ffffff001e71db60
  [ ffffff001e71db60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e723c60 fffffffffbc2ba70                0   0  60 ffffff04deb45558
  PC: _resume_from_idle+0xf1    TASKQ: ohci_nexus_enum_tq
  stack pointer for thread ffffff001e723c60: ffffff001e723b60
  [ ffffff001e723b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e729c60 fffffffffbc2ba70                0   0  60 ffffff04deb45480
  PC: _resume_from_idle+0xf1    TASKQ: USB_ohci_0_pipehndl_tq_1
  stack pointer for thread ffffff001e729c60: ffffff001e729b60
  [ ffffff001e729b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e72fc60 fffffffffbc2ba70                0   0  60 ffffff04deb45480
  PC: _resume_from_idle+0xf1    TASKQ: USB_ohci_0_pipehndl_tq_1
  stack pointer for thread ffffff001e72fc60: ffffff001e72fb60
  [ ffffff001e72fb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e735c60 fffffffffbc2ba70                0   0  60 ffffff04deb45480
  PC: _resume_from_idle+0xf1    TASKQ: USB_ohci_0_pipehndl_tq_1
  stack pointer for thread ffffff001e735c60: ffffff001e735b60
  [ ffffff001e735b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e73bc60 fffffffffbc2ba70                0   0  60 ffffff04deb45480
  PC: _resume_from_idle+0xf1    TASKQ: USB_ohci_0_pipehndl_tq_1
  stack pointer for thread ffffff001e73bc60: ffffff001e73bb60
  [ ffffff001e73bb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e741c60 fffffffffbc2ba70                0   0  60 ffffff04deb453a8
  PC: _resume_from_idle+0xf1    TASKQ: USB_ohci_81_pipehndl_tq_1
  stack pointer for thread ffffff001e741c60: ffffff001e741b60
  [ ffffff001e741b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e747c60 fffffffffbc2ba70                0   0  60 ffffff04deb453a8
  PC: _resume_from_idle+0xf1    TASKQ: USB_ohci_81_pipehndl_tq_1
  stack pointer for thread ffffff001e747c60: ffffff001e747b60
  [ ffffff001e747b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e74dc60 fffffffffbc2ba70                0   0  60 ffffff04deb452d0
  PC: _resume_from_idle+0xf1    TASKQ: ohci_nexus_enum_tq
  stack pointer for thread ffffff001e74dc60: ffffff001e74db60
  [ ffffff001e74db60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e753c60 fffffffffbc2ba70                0   0  60 ffffff04deb451f8
  PC: _resume_from_idle+0xf1    TASKQ: USB_ohci_0_pipehndl_tq_2
  stack pointer for thread ffffff001e753c60: ffffff001e753b60
  [ ffffff001e753b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e759c60 fffffffffbc2ba70                0   0  60 ffffff04deb451f8
  PC: _resume_from_idle+0xf1    TASKQ: USB_ohci_0_pipehndl_tq_2
  stack pointer for thread ffffff001e759c60: ffffff001e759b60
  [ ffffff001e759b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e75fc60 fffffffffbc2ba70                0   0  60 ffffff04deb451f8
  PC: _resume_from_idle+0xf1    TASKQ: USB_ohci_0_pipehndl_tq_2
  stack pointer for thread ffffff001e75fc60: ffffff001e75fb60
  [ ffffff001e75fb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e765c60 fffffffffbc2ba70                0   0  60 ffffff04deb451f8
  PC: _resume_from_idle+0xf1    TASKQ: USB_ohci_0_pipehndl_tq_2
  stack pointer for thread ffffff001e765c60: ffffff001e765b60
  [ ffffff001e765b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e76bc60 fffffffffbc2ba70                0   0  60 ffffff04deb45120
  PC: _resume_from_idle+0xf1    TASKQ: USB_ohci_81_pipehndl_tq_2
  stack pointer for thread ffffff001e76bc60: ffffff001e76bb60
  [ ffffff001e76bb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e771c60 fffffffffbc2ba70                0   0  60 ffffff04deb45120
  PC: _resume_from_idle+0xf1    TASKQ: USB_ohci_81_pipehndl_tq_2
  stack pointer for thread ffffff001e771c60: ffffff001e771b60
  [ ffffff001e771b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e777c60 fffffffffbc2ba70                0   0  60 ffffff04deb45048
  PC: _resume_from_idle+0xf1    TASKQ: ohci_nexus_enum_tq
  stack pointer for thread ffffff001e777c60: ffffff001e777b60
  [ ffffff001e777b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e77dc60 fffffffffbc2ba70                0   0  60 ffffff04df1afea8
  PC: _resume_from_idle+0xf1    TASKQ: USB_ohci_0_pipehndl_tq_3
  stack pointer for thread ffffff001e77dc60: ffffff001e77db60
  [ ffffff001e77db60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e783c60 fffffffffbc2ba70                0   0  60 ffffff04df1afea8
  PC: _resume_from_idle+0xf1    TASKQ: USB_ohci_0_pipehndl_tq_3
  stack pointer for thread ffffff001e783c60: ffffff001e783b60
  [ ffffff001e783b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e789c60 fffffffffbc2ba70                0   0  60 ffffff04df1afea8
  PC: _resume_from_idle+0xf1    TASKQ: USB_ohci_0_pipehndl_tq_3
  stack pointer for thread ffffff001e789c60: ffffff001e789b60
  [ ffffff001e789b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e78fc60 fffffffffbc2ba70                0   0  60 ffffff04df1afea8
  PC: _resume_from_idle+0xf1    TASKQ: USB_ohci_0_pipehndl_tq_3
  stack pointer for thread ffffff001e78fc60: ffffff001e78fb60
  [ ffffff001e78fb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e795c60 fffffffffbc2ba70                0   0  60 ffffff04df1afdd0
  PC: _resume_from_idle+0xf1    TASKQ: USB_ohci_81_pipehndl_tq_3
  stack pointer for thread ffffff001e795c60: ffffff001e795b60
  [ ffffff001e795b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e79bc60 fffffffffbc2ba70                0   0  60 ffffff04df1afdd0
  PC: _resume_from_idle+0xf1    TASKQ: USB_ohci_81_pipehndl_tq_3
  stack pointer for thread ffffff001e79bc60: ffffff001e79bb60
  [ ffffff001e79bb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e7d1c60 fffffffffbc2ba70                0   0  60 ffffff04d5c91c00
  PC: _resume_from_idle+0xf1    THREAD: evch_delivery_thr()
  stack pointer for thread ffffff001e7d1c60: ffffff001e7d1ae0
  [ ffffff001e7d1ae0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    evch_delivery_hold+0x85()
    evch_delivery_thr+0x23e()
    thread_start+8()

ffffff001e7d7c60 fffffffffbc2ba70                0   0  60 ffffff04da3a2818
  PC: _resume_from_idle+0xf1    THREAD: evch_delivery_thr()
  stack pointer for thread ffffff001e7d7c60: ffffff001e7d7ae0
  [ ffffff001e7d7ae0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    evch_delivery_hold+0x85()
    evch_delivery_thr+0x23e()
    thread_start+8()

ffffff001e7ddc60 fffffffffbc2ba70                0   0 109                0
  PC: _resume_from_idle+0xf1    THREAD: cpu_pause()
  stack pointer for thread ffffff001e7ddc60: ffffff001e7ddbd0
  [ ffffff001e7ddbd0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cpu_pause+0x8c()
    thread_start+8()

ffffff001e847c60 fffffffffbc2ba70                0   0  99 ffffff04deb46e10
  PC: _resume_from_idle+0xf1    THREAD: squeue_worker()
  stack pointer for thread ffffff001e847c60: ffffff001e847b80
  [ ffffff001e847b80 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    squeue_worker+0xc0()
    thread_start+8()

ffffff001e84dc60 fffffffffbc2ba70                0   0  99 ffffff04deb46e12
  PC: _resume_from_idle+0xf1    THREAD: squeue_polling_thread()
  stack pointer for thread ffffff001e84dc60: ffffff001e84db50
  [ ffffff001e84db50 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    squeue_polling_thread+0x90()
    thread_start+8()

ffffff001e7e9c60 fffffffffbc2ba70                0   0  -1                0
  PC: _resume_from_idle+0xf1    THREAD: idle()
  stack pointer for thread ffffff001e7e9c60: ffffff001e7e9b00
  [ ffffff001e7e9b00 _resume_from_idle+0xf1() ]
    mach_cpu_idle+0xb()
    cpu_idle+0xc2()
    idle+0x114()
    thread_start+8()

ffffff001e82bc60 fffffffffbc2ba70                0   0 109                0
  PC: _resume_from_idle+0xf1    THREAD: cpu_pause()
  stack pointer for thread ffffff001e82bc60: ffffff001e82bbd0
  [ ffffff001e82bbd0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cpu_pause+0x8c()
    thread_start+8()

ffffff001e8a1c60 fffffffffbc2ba70                0   0  99 ffffff04deb46d50
  PC: _resume_from_idle+0xf1    THREAD: squeue_worker()
  stack pointer for thread ffffff001e8a1c60: ffffff001e8a1b80
  [ ffffff001e8a1b80 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    squeue_worker+0xc0()
    thread_start+8()

ffffff001e8a7c60 fffffffffbc2ba70                0   0  99 ffffff04deb46d52
  PC: _resume_from_idle+0xf1    THREAD: squeue_polling_thread()
  stack pointer for thread ffffff001e8a7c60: ffffff001e8a7b50
  [ ffffff001e8a7b50 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    squeue_polling_thread+0x90()
    thread_start+8()

ffffff001e859c60 fffffffffbc2ba70                0   0  -1                0
  PC: _resume_from_idle+0xf1    THREAD: idle()
  stack pointer for thread ffffff001e859c60: ffffff001e859bf0
  [ ffffff001e859bf0 _resume_from_idle+0xf1() ]
    cpu_idle+0xc2()
    idle+0x114()
    thread_start+8()

ffffff001e89bc60 fffffffffbc2ba70                0   0 109                0
  PC: _resume_from_idle+0xf1    THREAD: cpu_pause()
  stack pointer for thread ffffff001e89bc60: ffffff001e89bbd0
  [ ffffff001e89bbd0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cpu_pause+0x8c()
    thread_start+8()

ffffff001e8fbc60 fffffffffbc2ba70                0   0  99 ffffff04deb46c90
  PC: _resume_from_idle+0xf1    THREAD: squeue_worker()
  stack pointer for thread ffffff001e8fbc60: ffffff001e8fbb80
  [ ffffff001e8fbb80 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    squeue_worker+0xc0()
    thread_start+8()

ffffff001e901c60 fffffffffbc2ba70                0   0  99 ffffff04deb46c92
  PC: _resume_from_idle+0xf1    THREAD: squeue_polling_thread()
  stack pointer for thread ffffff001e901c60: ffffff001e901b50
  [ ffffff001e901b50 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    squeue_polling_thread+0x90()
    thread_start+8()

ffffff001e8b3c60 fffffffffbc2ba70                0   0  -1                0
  PC: _resume_from_idle+0xf1    THREAD: idle()
  stack pointer for thread ffffff001e8b3c60: ffffff001e8b3bf0
  [ ffffff001e8b3bf0 _resume_from_idle+0xf1() ]
    cpu_idle+0xc2()
    idle+0x114()
    thread_start+8()

ffffff001e8f5c60 fffffffffbc2ba70                0   0 109                0
  PC: thread_start    THREAD: cpu_pause()
  Possible stack pointers for thread ffffff001e8f5c60:

ffffff001e98cc60 fffffffffbc2ba70                0   0  60 ffffff04df1afb48
  PC: _resume_from_idle+0xf1    TASKQ: pci_nexus_enum_tq
  stack pointer for thread ffffff001e98cc60: ffffff001e98cb60
  [ ffffff001e98cb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e980c60 fffffffffbc2ba70                0   0  60 ffffff04df1afa70
  PC: _resume_from_idle+0xf1    TASKQ: pci_nexus_enum_tq
  stack pointer for thread ffffff001e980c60: ffffff001e980b60
  [ ffffff001e980b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e912c60 fffffffffbc2ba70                0   0  99 ffffff04df1af998
  PC: _resume_from_idle+0xf1    TASKQ: callout_taskq
  stack pointer for thread ffffff001e912c60: ffffff001e912b60
  [ ffffff001e912b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x143()
    thread_start+8()

ffffff001e918c60 fffffffffbc2ba70                0   0  99 ffffff04df1af998
  PC: _resume_from_idle+0xf1    TASKQ: callout_taskq
  stack pointer for thread ffffff001e918c60: ffffff001e918b60
  [ ffffff001e918b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x143()
    thread_start+8()

ffffff001e91ec60 fffffffffbc2ba70                0   0  99 ffffff04df1af8c0
  PC: _resume_from_idle+0xf1    TASKQ: callout_taskq
  stack pointer for thread ffffff001e91ec60: ffffff001e91eb60
  [ ffffff001e91eb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x143()
    thread_start+8()

ffffff001e924c60 fffffffffbc2ba70                0   0  99 ffffff04df1af8c0
  PC: _resume_from_idle+0xf1    TASKQ: callout_taskq
  stack pointer for thread ffffff001e924c60: ffffff001e924b60
  [ ffffff001e924b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x143()
    thread_start+8()

ffffff001e92ac60 fffffffffbc2ba70                0   0  99 ffffff04df1af7e8
  PC: _resume_from_idle+0xf1    TASKQ: callout_taskq
  stack pointer for thread ffffff001e92ac60: ffffff001e92ab60
  [ ffffff001e92ab60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x143()
    thread_start+8()

ffffff001e930c60 fffffffffbc2ba70                0   0  99 ffffff04df1af7e8
  PC: _resume_from_idle+0xf1    TASKQ: callout_taskq
  stack pointer for thread ffffff001e930c60: ffffff001e930b60
  [ ffffff001e930b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x143()
    thread_start+8()

ffffff001f240c60 fffffffffbc2ba70                0   0  98 ffffff04df775eb0
  PC: _resume_from_idle+0xf1    TASKQ: console_taskq
  stack pointer for thread ffffff001f240c60: ffffff001f240b60
  [ ffffff001f240b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff04df5ee8c0 ffffff04de62fde8 ffffff04de249010   1  59 ffffff04de630860
  PC: _resume_from_idle+0xf1    CMD: /lib/svc/bin/svc.configd
  stack pointer for thread ffffff04df5ee8c0: ffffff001e8add80
  [ ffffff001e8add80 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig+0x14b()
    door_unref+0x89()
    doorfs32+0x82()
    sys_syscall32+0x101()

ffffff04dfc2d080 ffffff04de62fde8 ffffff04df9d6130   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /lib/svc/bin/svc.configd
  stack pointer for thread ffffff04dfc2d080: ffffff002022cd20
  [ ffffff002022cd20 _resume_from_idle+0xf1() ]
    swtch_to+0xe1()
    shuttle_resume+0x328()
    door_return+0x21a()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff04df979400 ffffff04de62fde8 ffffff04df9e01b0   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /lib/svc/bin/svc.configd
  stack pointer for thread ffffff04df979400: ffffff001ee13d20
  [ ffffff001ee13d20 _resume_from_idle+0xf1() ]
    swtch_to+0xe1()
    shuttle_resume+0x328()
    door_return+0x21a()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff04df38a1a0 ffffff04de62fde8 ffffff04de240020   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /lib/svc/bin/svc.configd
  stack pointer for thread ffffff04df38a1a0: ffffff001e96cd20
  [ ffffff001e96cd20 _resume_from_idle+0xf1() ]
    swtch_to+0xe1()
    shuttle_resume+0x328()
    door_return+0x21a()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff04df97be60 ffffff04de62fde8 ffffff04de23db50   1  59 ffffff04df97c036
  PC: _resume_from_idle+0xf1    CMD: /lib/svc/bin/svc.configd
  stack pointer for thread ffffff04df97be60: ffffff001eca4cc0
  [ ffffff001eca4cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04df979080 ffffff04de62fde8 ffffff04df9dfc20   1  59 ffffff04df979256
  PC: _resume_from_idle+0xf1    CMD: /lib/svc/bin/svc.configd
  stack pointer for thread ffffff04df979080: ffffff001ee19cc0
  [ ffffff001ee19cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04df5ee1c0 ffffff04de62fde8 ffffff04de249b30   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /lib/svc/bin/svc.configd
  stack pointer for thread ffffff04df5ee1c0: ffffff001f283d20
  [ ffffff001f283d20 _resume_from_idle+0xf1() ]
    swtch_to+0xe1()
    shuttle_resume+0x328()
    door_return+0x21a()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff04e0d7be40 ffffff04de62fde8 ffffff04df9de5e0   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /lib/svc/bin/svc.configd
  stack pointer for thread ffffff04e0d7be40: ffffff0020260d20
  [ ffffff0020260d20 _resume_from_idle+0xf1() ]
    swtch_to+0xe1()
    shuttle_resume+0x328()
    door_return+0x21a()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff04df96fe20 ffffff04de62fde8 ffffff04de23ec00   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /lib/svc/bin/svc.configd
  stack pointer for thread ffffff04df96fe20: ffffff0020232d20
  [ ffffff0020232d20 _resume_from_idle+0xf1() ]
    swtch_to+0xe1()
    shuttle_resume+0x328()
    door_return+0x21a()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff04e91f6060 ffffff04de62fde8 ffffff04e050b100   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /lib/svc/bin/svc.configd
  stack pointer for thread ffffff04e91f6060: ffffff0020185d20
  [ ffffff0020185d20 _resume_from_idle+0xf1() ]
    swtch_to+0xe1()
    shuttle_resume+0x328()
    door_return+0x21a()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff04df97bae0 ffffff04de62fde8 ffffff04de23d5c0   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /lib/svc/bin/svc.configd
  stack pointer for thread ffffff04df97bae0: ffffff001ecaad20
  [ ffffff001ecaad20 _resume_from_idle+0xf1() ]
    swtch_to+0xe1()
    shuttle_resume+0x328()
    door_return+0x21a()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff04dfc29e00 ffffff04de62fde8 ffffff04df9cf630   1  59 ffffff04dfc29fd6
  PC: _resume_from_idle+0xf1    CMD: /lib/svc/bin/svc.configd
  stack pointer for thread ffffff04dfc29e00: ffffff001ebb4cc0
  [ ffffff001ebb4cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04df97cc60 ffffff04de62fde8 ffffff04de2405b0   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /lib/svc/bin/svc.configd
  stack pointer for thread ffffff04df97cc60: ffffff001e966d20
  [ ffffff001e966d20 _resume_from_idle+0xf1() ]
    swtch_to+0xe1()
    shuttle_resume+0x328()
    door_return+0x21a()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff04df5ee540 ffffff04de62fde8 ffffff04de242710   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /lib/svc/bin/svc.configd
  stack pointer for thread ffffff04df5ee540: ffffff001e986d20
  [ ffffff001e986d20 _resume_from_idle+0xf1() ]
    swtch_to+0xe1()
    shuttle_resume+0x328()
    door_return+0x21a()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff04df5eec40 ffffff04de62fde8 ffffff04de2495a0   1  59 ffffff04df5eee16
  PC: _resume_from_idle+0xf1    CMD: /lib/svc/bin/svc.configd
  stack pointer for thread ffffff04df5eec40: ffffff001f2c0ca0
  [ ffffff001f2c0ca0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    sigtimedwait+0x1b5()
    sys_syscall32+0x101()

ffffff04df3893a0 ffffff04de62f188 ffffff04de24a650   1  59 ffffff04df7864e4
  PC: _resume_from_idle+0xf1    CMD: /lib/svc/bin/svc.startd
  stack pointer for thread ffffff04df3893a0: ffffff001ec5eb20
  [ ffffff001ec5eb20 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig+0x14b()
    cte_get_event+0xd7()
    ctfs_endpoint_ioctl+0xa7()
    ctfs_bu_ioctl+0x51()
    fop_ioctl+0x7b()
    ioctl+0x18e()
    sys_syscall32+0x101()

ffffff04df5ede40 ffffff04de62f188 ffffff04de242180   1  59 ffffff04df5ee016
  PC: _resume_from_idle+0xf1    CMD: /lib/svc/bin/svc.startd
  stack pointer for thread ffffff04df5ede40: ffffff001e7e3cc0
  [ ffffff001e7e3cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04ddefec00 ffffff04de62e528 ffffff050157e880   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /sbin/dlmgmtd
  stack pointer for thread ffffff04ddefec00: ffffff001f2d8d20
  [ ffffff001f2d8d20 _resume_from_idle+0xf1() ]
    swtch_to+0xe1()
    shuttle_resume+0x328()
    door_return+0x21a()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff04dfc153e0 ffffff04de62e528 ffffff04e0503d00   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /sbin/dlmgmtd
  stack pointer for thread ffffff04dfc153e0: ffffff001f7f9d20
  [ ffffff001f7f9d20 _resume_from_idle+0xf1() ]
    swtch_to+0xe1()
    shuttle_resume+0x328()
    door_return+0x21a()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff04e058f520 ffffff04de62e528 ffffff04e050c1b0   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /sbin/dlmgmtd
  stack pointer for thread ffffff04e058f520: ffffff0020136d20
  [ ffffff0020136d20 _resume_from_idle+0xf1() ]
    swtch_to+0xe1()
    shuttle_resume+0x328()
    door_return+0x21a()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff04dfc268a0 ffffff04de62e528 ffffff04df9cd160   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /sbin/dlmgmtd
  stack pointer for thread ffffff04dfc268a0: ffffff001ebd8d20
  [ ffffff001ebd8d20 _resume_from_idle+0xf1() ]
    swtch_to+0xe1()
    shuttle_resume+0x328()
    door_return+0x21a()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff04df96c8c0 ffffff04de62e528 ffffff04df9d8b90   1  59 ffffff04df96ca96
  PC: _resume_from_idle+0xf1    CMD: /sbin/dlmgmtd
  stack pointer for thread ffffff04df96c8c0: ffffff001f114df0
  [ ffffff001f114df0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    pause+0x48()
    sys_syscall32+0x101()

ffffff002013cc60 fffffffffbc2ba70                0   0  60 ffffff04df775dd8
  PC: _resume_from_idle+0xf1    TASKQ: pci_pci_nexus_enum_tq
  stack pointer for thread ffffff002013cc60: ffffff002013cb60
  [ ffffff002013cb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff0020287c60 fffffffffbc2ba70                0   0  60 ffffff04e08d4878
  PC: _resume_from_idle+0xf1    THREAD: i_mac_notify_thread()
  stack pointer for thread ffffff0020287c60: ffffff0020287b70
  [ ffffff0020287b70 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    i_mac_notify_thread+0xbb()
    thread_start+8()

ffffff001fb9cc60 fffffffffbc2ba70                0   0  60 ffffff04df775d00
  PC: _resume_from_idle+0xf1    TASKQ: zil_clean
  stack pointer for thread ffffff001fb9cc60: ffffff001fb9cb60
  [ ffffff001fb9cb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff04e0d7d720 ffffff04dfdcca58 ffffff04e050ec10   1  59 ffffff04e0d7d8f6
  PC: _resume_from_idle+0xf1    CMD: devfsadmd
  stack pointer for thread ffffff04e0d7d720: ffffff002010ccc0
  [ ffffff002010ccc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04e0d7d3a0 ffffff04dfdcca58 ffffff04e050d7f0   1  59 ffffff04e0d7d576
  PC: _resume_from_idle+0xf1    CMD: devfsadmd
  stack pointer for thread ffffff04e0d7d3a0: ffffff002011ecc0
  [ ffffff002011ecc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04e7a963c0 ffffff04dfdcca58 ffffff04df9df690   1  59                0
  PC: _resume_from_idle+0xf1    CMD: devfsadmd
  stack pointer for thread ffffff04e7a963c0: ffffff001ed44d30
  [ ffffff001ed44d30 _resume_from_idle+0xf1() ]
    swtch+0x160()
    shuttle_swtch+0x259()
    door_return+0x242()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff04dfc2db00 ffffff04dfdcca58 ffffff04e04fad10   1  59                0
  PC: _resume_from_idle+0xf1    CMD: devfsadmd
  stack pointer for thread ffffff04dfc2db00: ffffff002016cd20
  [ ffffff002016cd20 _resume_from_idle+0xf1() ]
    swtch_to+0xe1()
    shuttle_resume+0x328()
    door_return+0x21a()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff001fb3bc60 fffffffffbc2ba70                0   0  60 ffffff04df775c28
  PC: _resume_from_idle+0xf1    TASKQ: iscsi_nexus_enum_tq
  stack pointer for thread ffffff001fb3bc60: ffffff001fb3bb60
  [ ffffff001fb3bb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001fb41c60 fffffffffbc2ba70                0   0  60 ffffff04df775b50
  PC: _resume_from_idle+0xf1    TASKQ: isns_reg_query_taskq
  stack pointer for thread ffffff001fb41c60: ffffff001fb41b60
  [ ffffff001fb41b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001fb47c60 fffffffffbc2ba70                0   0  60 ffffff04df775a78
  PC: _resume_from_idle+0xf1    TASKQ: isns_scn_taskq
  stack pointer for thread ffffff001fb47c60: ffffff001fb47b60
  [ ffffff001fb47b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001fb4dc60 fffffffffbc2ba70                0   0  60 ffffff04df7759a0
  PC: _resume_from_idle+0xf1    TASKQ: iscsi_Static
  stack pointer for thread ffffff001fb4dc60: ffffff001fb4db60
  [ ffffff001fb4db60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001fb53c60 fffffffffbc2ba70                0   0  60 ffffff04df7758c8
  PC: _resume_from_idle+0xf1    TASKQ: iscsi_SendTarget
  stack pointer for thread ffffff001fb53c60: ffffff001fb53b60
  [ ffffff001fb53b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001fb59c60 fffffffffbc2ba70                0   0  60 ffffff04df7757f0
  PC: _resume_from_idle+0xf1    TASKQ: iscsi_SLP
  stack pointer for thread ffffff001fb59c60: ffffff001fb59b60
  [ ffffff001fb59b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001fb5fc60 fffffffffbc2ba70                0   0  60 ffffff04df775718
  PC: _resume_from_idle+0xf1    TASKQ: iscsi_iSNS
  stack pointer for thread ffffff001fb5fc60: ffffff001fb5fb60
  [ ffffff001fb5fb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff00200ecc60 fffffffffbc2ba70                0   0  60 ffffff04df775640
  PC: _resume_from_idle+0xf1    TASKQ: pci_pci_nexus_enum_tq
  stack pointer for thread ffffff00200ecc60: ffffff00200ecb60
  [ ffffff00200ecb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff00201a3c60 fffffffffbc2ba70                0   0  60 ffffff04df7753b8
  PC: _resume_from_idle+0xf1    TASKQ: pci_pci_nexus_enum_tq
  stack pointer for thread ffffff00201a3c60: ffffff00201a3b60
  [ ffffff00201a3b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001ed8cc60 fffffffffbc2ba70                0   0  60 ffffff04e5584b58
  PC: _resume_from_idle+0xf1    TASKQ: pci_ide_nexus_enum_tq
  stack pointer for thread ffffff001ed8cc60: ffffff001ed8cb60
  [ ffffff001ed8cb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001edaac60 fffffffffbc2ba70                0   0  60 ffffff04e5584a80
  PC: _resume_from_idle+0xf1    TASKQ: marvell88sx_nexus_enum_tq
  stack pointer for thread ffffff001edaac60: ffffff001edaab60
  [ ffffff001edaab60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff00200cfc60 fffffffffbc2ba70                0   0  60 ffffff04e5584648
  PC: _resume_from_idle+0xf1    TASKQ: pci11ab_11ab_0
  stack pointer for thread ffffff00200cfc60: ffffff00200cfb60
  [ ffffff00200cfb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001edcec60 fffffffffbc2ba70                0   0  60 ffffff04e55849a8
  PC: _resume_from_idle+0xf1    TASKQ: marvell88sx_nexus_enum_tq
  stack pointer for thread ffffff001edcec60: ffffff001edceb60
  [ ffffff001edceb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001ed80c60 fffffffffbc2ba70                0   0  60 ffffff04e5584498
  PC: _resume_from_idle+0xf1    TASKQ: pci11ab_11ab_1
  stack pointer for thread ffffff001ed80c60: ffffff001ed80b60
  [ ffffff001ed80b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff00201a9c60 fffffffffbc2ba70                0   0  60 ffffff04df7752e0
  PC: _resume_from_idle+0xf1    TASKQ: pci_pci_nexus_enum_tq
  stack pointer for thread ffffff00201a9c60: ffffff00201a9b60
  [ ffffff00201a9b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001fb65c60 fffffffffbc2ba70                0   0  60 ffffff04e5584eb8
  PC: _resume_from_idle+0xf1    TASKQ: pci_pci_nexus_enum_tq
  stack pointer for thread ffffff001fb65c60: ffffff001fb65b60
  [ ffffff001fb65b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001ec07c60 fffffffffbc2ba70                0   0  60 ffffff04e5584c30
  PC: _resume_from_idle+0xf1    TASKQ: marvell88sx_nexus_enum_tq
  stack pointer for thread ffffff001ec07c60: ffffff001ec07b60
  [ ffffff001ec07b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001ebdec60 fffffffffbc2ba70                0   0  60 ffffff04e5584720
  PC: _resume_from_idle+0xf1    TASKQ: pci11ab_11ab_2
  stack pointer for thread ffffff001ebdec60: ffffff001ebdeb60
  [ ffffff001ebdeb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff0020197c60 fffffffffbc2ba70                0   0  60 ffffff04df775568
  PC: _resume_from_idle+0xf1    TASKQ: pci_pci_nexus_enum_tq
  stack pointer for thread ffffff0020197c60: ffffff0020197b60
  [ ffffff0020197b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff00201afc60 fffffffffbc2ba70                0   0  60 ffffff04df775208
  PC: _resume_from_idle+0xf1    TASKQ: pci_pci_nexus_enum_tq
  stack pointer for thread ffffff00201afc60: ffffff00201afb60
  [ ffffff00201afb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001fb8ec60 fffffffffbc2ba70                0   0  60 ffffff04e5584de0
  PC: _resume_from_idle+0xf1    TASKQ: marvell88sx_nexus_enum_tq
  stack pointer for thread ffffff001fb8ec60: ffffff001fb8eb60
  [ ffffff001fb8eb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001fbfcc60 fffffffffbc2ba70                0   0  60 ffffff04e55847f8
  PC: _resume_from_idle+0xf1    TASKQ: pci11ab_11ab_4
  stack pointer for thread ffffff001fbfcc60: ffffff001fbfcb60
  [ ffffff001fbfcb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001ebfbc60 fffffffffbc2ba70                0   0  60 ffffff04e5584d08
  PC: _resume_from_idle+0xf1    TASKQ: marvell88sx_nexus_enum_tq
  stack pointer for thread ffffff001ebfbc60: ffffff001ebfbb60
  [ ffffff001ebfbb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001fbd5c60 fffffffffbc2ba70                0   0  60 ffffff04e55848d0
  PC: _resume_from_idle+0xf1    TASKQ: pci11ab_11ab_5
  stack pointer for thread ffffff001fbd5c60: ffffff001fbd5b60
  [ ffffff001fbd5b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff002019dc60 fffffffffbc2ba70                0   0  60 ffffff04df775490
  PC: _resume_from_idle+0xf1    TASKQ: pci_pci_nexus_enum_tq
  stack pointer for thread ffffff002019dc60: ffffff002019db60
  [ ffffff002019db60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff00201c6c60 fffffffffbc2ba70                0   0  60 ffffff04df775058
  PC: _resume_from_idle+0xf1    TASKQ: pci_pci_nexus_enum_tq
  stack pointer for thread ffffff00201c6c60: ffffff00201c6b60
  [ ffffff00201c6b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff00201c0c60 fffffffffbc2ba70                0   0  60 ffffff04df775130
  PC: _resume_from_idle+0xf1    TASKQ: pseudo_nexus_enum_tq
  stack pointer for thread ffffff00201c0c60: ffffff00201c0b60
  [ ffffff00201c0b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001fb07c60 fffffffffbc2ba70                0   0  60 ffffffffc00a28a6
  PC: _resume_from_idle+0xf1    THREAD: ufs_thread_idle()
  stack pointer for thread ffffff001fb07c60: ffffff001fb07b90
  [ ffffff001fb07b90 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    ufs_thread_idle+0x152()
    thread_start+8()

ffffff001fe57c60 fffffffffbc2ba70                0   0  60 ffffffffc00a2e56
  PC: _resume_from_idle+0xf1    THREAD: ufs_thread_hlock()
  stack pointer for thread ffffff001fe57c60: ffffff001fe57b20
  [ ffffff001fe57b20 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    ufs_thread_run+0x12d()
    ufs_thread_hlock+0x6f()
    thread_start+8()

ffffff04e0d7d020 ffffff04dfdcca58 ffffff04e04fb830   1  59                0
  PC: _resume_from_idle+0xf1    CMD: devfsadmd
  stack pointer for thread ffffff04e0d7d020: ffffff001fc62d20
  [ ffffff001fc62d20 _resume_from_idle+0xf1() ]
    swtch_to+0xe1()
    shuttle_resume+0x328()
    door_return+0x21a()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff04e0d7cc40 ffffff04dfdcca58 ffffff04e04fb2a0   1  59 ffffff04e0d7ce16
  PC: _resume_from_idle+0xf1    CMD: devfsadmd
  stack pointer for thread ffffff04e0d7cc40: ffffff001fba2cc0
  [ ffffff001fba2cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04e0d7daa0 ffffff04dfdcca58 ffffff04e050e680   1  59 ffffff04e0d7dc76
  PC: _resume_from_idle+0xf1    CMD: devfsadmd
  stack pointer for thread ffffff04e0d7daa0: ffffff0020112df0
  [ ffffff0020112df0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    pause+0x75()
    sys_syscall32+0x101()

ffffff001f32ac60 fffffffffbc2ba70                0   0  60 ffffff04e5584210
  PC: _resume_from_idle+0xf1    TASKQ: zil_clean
  stack pointer for thread ffffff001f32ac60: ffffff001f32ab60
  [ ffffff001f32ab60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001f3c0c60 fffffffffbc2ba70                0   0  99 ffffff04e75ea728
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001f3c0c60: ffffff001f3c0b60
  [ ffffff001f3c0b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001f414c60 fffffffffbc2ba70                0   0  99 ffffff04e75ea800
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001f414c60: ffffff001f414b60
  [ ffffff001f414b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001f408c60 fffffffffbc2ba70                0   0  99 ffffff04e75ea8d8
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001f408c60: ffffff001f408b60
  [ ffffff001f408b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001f38ac60 fffffffffbc2ba70                0   0  99 ffffff04e75ea9b0
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001f38ac60: ffffff001f38ab60
  [ ffffff001f38ab60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001f342c60 fffffffffbc2ba70                0   0  99 ffffff04e75ea9b0
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001f342c60: ffffff001f342b60
  [ ffffff001f342b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001f40ec60 fffffffffbc2ba70                0   0  99 ffffff04e75ea9b0
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001f40ec60: ffffff001f40eb60
  [ ffffff001f40eb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001f360c60 fffffffffbc2ba70                0   0  99 ffffff04e75ea9b0
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001f360c60: ffffff001f360b60
  [ ffffff001f360b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff002023ec60 fffffffffbc2ba70                0   0  99 ffffff04e75ea9b0
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff002023ec60: ffffff002023eb60
  [ ffffff002023eb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001ecb6c60 fffffffffbc2ba70                0   0  99 ffffff04e75ea9b0
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001ecb6c60: ffffff001ecb6b60
  [ ffffff001ecb6b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001ecfcc60 fffffffffbc2ba70                0   0  99 ffffff04e75ea9b0
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001ecfcc60: ffffff001ecfcb60
  [ ffffff001ecfcb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001ed86c60 fffffffffbc2ba70                0   0  99 ffffff04e75ea9b0
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001ed86c60: ffffff001ed86b60
  [ ffffff001ed86b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001edc8c60 fffffffffbc2ba70                0   0  99 ffffff04e75eaa88
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001edc8c60: ffffff001edc8b60
  [ ffffff001edc8b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001ec01c60 fffffffffbc2ba70                0   0  99 ffffff04e75eaa88
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001ec01c60: ffffff001ec01b60
  [ ffffff001ec01b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001f384c60 fffffffffbc2ba70                0   0  99 ffffff04e75eaa88
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001f384c60: ffffff001f384b60
  [ ffffff001f384b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001ee1fc60 fffffffffbc2ba70                0   0  99 ffffff04e75eaa88
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001ee1fc60: ffffff001ee1fb60
  [ ffffff001ee1fb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001f11ac60 fffffffffbc2ba70                0   0  99 ffffff04e75eaa88
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001f11ac60: ffffff001f11ab60
  [ ffffff001f11ab60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff0020172c60 fffffffffbc2ba70                0   0  99 ffffff04e75eaa88
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff0020172c60: ffffff0020172b60
  [ ffffff0020172b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001fb94c60 fffffffffbc2ba70                0   0  99 ffffff04e75eaa88
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001fb94c60: ffffff001fb94b60
  [ ffffff001fb94b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff0020256c60 fffffffffbc2ba70                0   0  99 ffffff04e75eaa88
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff0020256c60: ffffff0020256b60
  [ ffffff0020256b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001f2b6c60 fffffffffbc2ba70                0   0  99 ffffff04e75eab60
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001f2b6c60: ffffff001f2b6b60
  [ ffffff001f2b6b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e907c60 fffffffffbc2ba70                0   0  99 ffffff04e75eac38
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001e907c60: ffffff001e907b60
  [ ffffff001e907b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff00201b5c60 fffffffffbc2ba70                0   0  99 ffffff04e75ead10
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff00201b5c60: ffffff00201b5b60
  [ ffffff00201b5b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001edf5c60 fffffffffbc2ba70                0   0  99 ffffff04e75ea650
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001edf5c60: ffffff001edf5b60
  [ ffffff001edf5b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001edfbc60 fffffffffbc2ba70                0   0  99 ffffff04e75ea578
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001edfbc60: ffffff001edfbb60
  [ ffffff001edfbb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001f108c60 fffffffffbc2ba70                0   0  99 ffffff04e75ea4a0
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001f108c60: ffffff001f108b60
  [ ffffff001f108b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001f10ec60 fffffffffbc2ba70                0   0  99 ffffff04e75ea3c8
  PC: _resume_from_idle+0xf1    TASKQ: spa_zio
  stack pointer for thread ffffff001f10ec60: ffffff001f10eb60
  [ ffffff001f10eb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001f48cc60 fffffffffbc2ba70                0   0  60 ffffff04e769cb1c
  PC: _resume_from_idle+0xf1    THREAD: txg_quiesce_thread()
  stack pointer for thread ffffff001f48cc60: ffffff001f48cb10
  [ ffffff001f48cb10 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    txg_thread_wait+0x5f()
    txg_quiesce_thread+0x94()
    thread_start+8()

ffffff001f492c60 fffffffffbc2ba70                0   0  60 ffffff04e769cb18
  PC: _resume_from_idle+0xf1    THREAD: txg_sync_thread()
  stack pointer for thread ffffff001f492c60: ffffff001f492ad0
  [ ffffff001f492ad0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait+0xba()
    txg_thread_wait+0x7b()
    txg_sync_thread+0x114()
    thread_start+8()

ffffff00201cec60 fffffffffbc2ba70                0   0  60 ffffff04db5b9ba8
  PC: _resume_from_idle+0xf1    THREAD: kcf_failover_thread()
  stack pointer for thread ffffff00201cec60: ffffff00201ceb70
  [ ffffff00201ceb70 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    kcf_failover_thread+0x185()
    thread_start+8()

ffffff00201d4c60 fffffffffbc2ba70                0   0  60 fffffffffbd0c6c0
  PC: _resume_from_idle+0xf1    THREAD: crypto_bufcall_service()
  stack pointer for thread ffffff00201d4c60: ffffff00201d4ba0
  [ ffffff00201d4ba0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    crypto_bufcall_service+0x152()
    thread_start+8()

ffffff04ea7bf900 ffffff04de6268d0 ffffff04e050fcc0   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /lib/crypto/kcfd
  stack pointer for thread ffffff04ea7bf900: ffffff002021ad30
  [ ffffff002021ad30 _resume_from_idle+0xf1() ]
    swtch+0x160()
    shuttle_swtch+0x259()
    door_return+0x242()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff04e0587560 ffffff04de6268d0 ffffff04e05091c0   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /lib/crypto/kcfd
  stack pointer for thread ffffff04e0587560: ffffff00201e6d20
  [ ffffff00201e6d20 _resume_from_idle+0xf1() ]
    swtch_to+0xe1()
    shuttle_resume+0x328()
    door_return+0x21a()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff04e05878e0 ffffff04de6268d0 ffffff04e0509750   1  59 ffffff04e0587ab6
  PC: _resume_from_idle+0xf1    CMD: /lib/crypto/kcfd
  stack pointer for thread ffffff04e05878e0: ffffff00201e0df0
  [ ffffff00201e0df0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    pause+0x48()
    sys_syscall32+0x101()

ffffff04df963060 ffffff04de62f188 ffffff04df9d4780   1  59 ffffff04df963236
  PC: _resume_from_idle+0xf1    CMD: /lib/svc/bin/svc.startd
  stack pointer for thread ffffff04df963060: ffffff001ee74cc0
  [ ffffff001ee74cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff001f498c60 fffffffffbc2ba70                0   0  60 ffffff04e5584570
  PC: _resume_from_idle+0xf1    TASKQ: zil_clean
  stack pointer for thread ffffff001f498c60: ffffff001f498b60
  [ ffffff001f498b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff04dfc20540 ffffff04de629a50 ffffff04df9da760   1  59 ffffff04dfc20716
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/sysevent/syseventd
  stack pointer for thread ffffff04dfc20540: ffffff001edefcc0
  [ ffffff001edefcc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04dfc1fe40 ffffff04de629a50 ffffff04df9d96b0   1  59 ffffff04dfc20016
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/sysevent/syseventd
  stack pointer for thread ffffff04dfc1fe40: ffffff001ee9ecc0
  [ ffffff001ee9ecc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04dfc1fac0 ffffff04de629a50 ffffff04df9d3140   1  59 ffffff04dfc1fc96
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/sysevent/syseventd
  stack pointer for thread ffffff04dfc1fac0: ffffff001eea4ca0
  [ ffffff001eea4ca0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    sigtimedwait+0x1ce()
    sys_syscall32+0x101()

ffffff04dfc11e80 ffffff04de629a50 ffffff04df9d6c50   1  59 ffffff04dfc12056
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/sysevent/syseventd
  stack pointer for thread ffffff04dfc11e80: ffffff002020ecc0
  [ ffffff002020ecc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04dfc11b00 ffffff04de629a50 ffffff04e0504820   1  59 ffffff04dfc11cd6
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/sysevent/syseventd
  stack pointer for thread ffffff04dfc11b00: ffffff0020118cc0
  [ ffffff0020118cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04e0586e60 ffffff04de629a50 ffffff04df9d3c60   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/sysevent/syseventd
  stack pointer for thread ffffff04e0586e60: ffffff001e83dd20
  [ ffffff001e83dd20 _resume_from_idle+0xf1() ]
    swtch_to+0xe1()
    shuttle_resume+0x328()
    door_return+0x21a()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff04dfc11080 ffffff04de629a50 ffffff04df9d2bb0   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/sysevent/syseventd
  stack pointer for thread ffffff04dfc11080: ffffff001ecbcd30
  [ ffffff001ecbcd30 _resume_from_idle+0xf1() ]
    swtch+0x160()
    shuttle_swtch+0x259()
    door_return+0x242()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff04e0595180 ffffff04de629a50 ffffff04df9d41f0   1  59 ffffff04e0595356
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/sysevent/syseventd
  stack pointer for thread ffffff04e0595180: ffffff00200f2cc0
  [ ffffff00200f2cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04e0586ae0 ffffff04de629a50 ffffff04df9d8600   1  59 ffffff04e0586cb6
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/sysevent/syseventd
  stack pointer for thread ffffff04e0586ae0: ffffff00201f2cc0
  [ ffffff00201f2cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04e0586760 ffffff04de629a50 ffffff04df9d7770   1  59 ffffff04e0586936
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/sysevent/syseventd
  stack pointer for thread ffffff04e0586760: ffffff0020214cc0
  [ ffffff0020214cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04e0581900 ffffff04de629a50 ffffff04df9e1b60   1  59 ffffff04e0581ad6
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/sysevent/syseventd
  stack pointer for thread ffffff04e0581900: ffffff00201eccc0
  [ ffffff00201eccc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04e0581580 ffffff04de629a50 ffffff04df9d36d0   1  59 ffffff04e0581756
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/sysevent/syseventd
  stack pointer for thread ffffff04e0581580: ffffff001f330cc0
  [ ffffff001f330cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04e0581200 ffffff04de629a50 ffffff04df9d2090   1  59 ffffff04e05813d6
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/sysevent/syseventd
  stack pointer for thread ffffff04e0581200: ffffff001f336cc0
  [ ffffff001f336cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04e0580e80 ffffff04de629a50 ffffff04df9d1200   1  59 ffffff04e0581056
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/sysevent/syseventd
  stack pointer for thread ffffff04e0580e80: ffffff001f33ccc0
  [ ffffff001f33ccc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04e0580b00 ffffff04de629a50 ffffff04df9d0150   1  59 ffffff04e0580cd6
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/sysevent/syseventd
  stack pointer for thread ffffff04e0580b00: ffffff001e831cc0
  [ ffffff001e831cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04e0580080 ffffff04de629a50 ffffff04df9e20f0   1  59 ffffff04e0580256
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/sysevent/syseventd
  stack pointer for thread ffffff04e0580080: ffffff001e837cc0
  [ ffffff001e837cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff001f4a4c60 fffffffffbc2ba70                0   0  60 fffffffffbc950ac
  PC: _resume_from_idle+0xf1    THREAD: log_event_deliver()
  stack pointer for thread ffffff001f4a4c60: ffffff001f4a4b90
  [ ffffff001f4a4b90 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    log_event_deliver+0x1f5()
    thread_start+8()

ffffff04dfc253a0 ffffff04de629a50 ffffff04df9db060   1  59 ffffff04dfc25576
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/sysevent/syseventd
  stack pointer for thread ffffff04dfc253a0: ffffff001ee0ddf0
  [ ffffff001ee0ddf0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    pause+0x75()
    sys_syscall32+0x101()

ffffff04e0581c80 ffffff04dfdcd6b8 ffffff04df9d5080   1  59 ffffff04e0581e56
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/nscd
  stack pointer for thread ffffff04e0581c80: ffffff00200f8d00
  [ ffffff00200f8d00 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    nanosleep+0x120()
    sys_syscall32+0x101()

ffffff04e0580780 ffffff04dfdcd6b8 ffffff04df9ce7a0   1  59 ffffff04dfdce130
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/nscd
  stack pointer for thread ffffff04e0580780: ffffff001ece1d80
  [ ffffff001ece1d80 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig+0x14b()
    door_unref+0x89()
    doorfs32+0x82()
    sys_syscall32+0x101()

ffffff04e058e020 ffffff04dfdcd6b8 ffffff04e0505120   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/nscd
  stack pointer for thread ffffff04e058e020: ffffff0020130d30
  [ ffffff0020130d30 _resume_from_idle+0xf1() ]
    swtch+0x160()
    shuttle_swtch+0x259()
    door_return+0x242()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff04e058eaa0 ffffff04dfdcd6b8 ffffff04df9d9120   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/nscd
  stack pointer for thread ffffff04e058eaa0: ffffff00200e1d20
  [ ffffff00200e1d20 _resume_from_idle+0xf1() ]
    swtch_to+0xe1()
    shuttle_resume+0x328()
    door_return+0x21a()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff04e058e720 ffffff04dfdcd6b8 ffffff04df9cb7b0   1  59 ffffff04e058e8f6
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/nscd
  stack pointer for thread ffffff04e058e720: ffffff001ec13d00
  [ ffffff001ec13d00 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    nanosleep+0x120()
    sys_syscall32+0x101()

ffffff04e058ee20 ffffff04dfdcd6b8 ffffff04df9dbb80   1  59 ffffff04e058eff6
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/nscd
  stack pointer for thread ffffff04e058ee20: ffffff001ec19d00
  [ ffffff001ec19d00 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    nanosleep+0x120()
    sys_syscall32+0x101()

ffffff04df97c1e0 ffffff04dfdcd6b8 ffffff04df9cb220   1  59 ffffff04df97c3b6
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/nscd
  stack pointer for thread ffffff04df97c1e0: ffffff001ec1fd00
  [ ffffff001ec1fd00 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    nanosleep+0x120()
    sys_syscall32+0x101()

ffffff04e0578000 ffffff04dfdcd6b8 ffffff04df9ca700   1  59 ffffff04e05781d6
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/nscd
  stack pointer for thread ffffff04e0578000: ffffff001ebe4d00
  [ ffffff001ebe4d00 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    nanosleep+0x120()
    sys_syscall32+0x101()

ffffff04df96b740 ffffff04dfdcd6b8 ffffff04df9cac90   1  59 ffffff04df96b916
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/nscd
  stack pointer for thread ffffff04df96b740: ffffff001ed92d00
  [ ffffff001ed92d00 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    nanosleep+0x120()
    sys_syscall32+0x101()

ffffff04dfc15060 ffffff04dfdcd6b8 ffffff04df9c90c0   1  59 ffffff04dfc15236
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/nscd
  stack pointer for thread ffffff04dfc15060: ffffff001eda4d00
  [ ffffff001eda4d00 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    nanosleep+0x120()
    sys_syscall32+0x101()

ffffff04dfc15760 ffffff04dfdcd6b8 ffffff04df9c8230   1  59 ffffff04dfc15936
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/nscd
  stack pointer for thread ffffff04dfc15760: ffffff001f36ccc0
  [ ffffff001f36ccc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04dfc25aa0 ffffff04dfdcd6b8 ffffff04df9c7710   1  59 ffffff04dfc25c76
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/nscd
  stack pointer for thread ffffff04dfc25aa0: ffffff001f372d00
  [ ffffff001f372d00 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    nanosleep+0x120()
    sys_syscall32+0x101()

ffffff04df96b040 ffffff04dfdcd6b8 ffffff04df9c6660   1  59 ffffff04df96b216
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/nscd
  stack pointer for thread ffffff04df96b040: ffffff001f378d00
  [ ffffff001f378d00 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    nanosleep+0x120()
    sys_syscall32+0x101()

ffffff04e05863e0 ffffff04dfdcd6b8 ffffff04df9ce210   1  59 ffffff04e05865b6
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/nscd
  stack pointer for thread ffffff04e05863e0: ffffff001ec0dd00
  [ ffffff001ec0dd00 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    nanosleep+0x120()
    sys_syscall32+0x101()

ffffff04dfc168e0 ffffff04dfdcd6b8 ffffff04df9e31a0   1  59 ffffff04dfc16ab6
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/nscd
  stack pointer for thread ffffff04dfc168e0: ffffff001f35ad00
  [ ffffff001f35ad00 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    nanosleep+0x120()
    sys_syscall32+0x101()

ffffff04df975a80 ffffff04dfdcd6b8 ffffff04e0512cb0   1  59 ffffff04df975c56
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/nscd
  stack pointer for thread ffffff04df975a80: ffffff001f0e5d00
  [ ffffff001f0e5d00 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    nanosleep+0x120()
    sys_syscall32+0x101()

ffffff04e0d7bac0 ffffff04dfdcd6b8 ffffff04e050a800   1  59 ffffff04e0d7bc96
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/nscd
  stack pointer for thread ffffff04e0d7bac0: ffffff002018bd00
  [ ffffff002018bd00 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    nanosleep+0x120()
    sys_syscall32+0x101()

ffffff04e058c040 ffffff04dfdcd6b8 ffffff04e0507810   1  59 ffffff04e058c216
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/nscd
  stack pointer for thread ffffff04e058c040: ffffff0020106d00
  [ ffffff0020106d00 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    nanosleep+0x120()
    sys_syscall32+0x101()

ffffff04ea7be400 ffffff04dfdcd6b8 ffffff04e050bc20   1  59 ffffff04ea7be5d6
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/nscd
  stack pointer for thread ffffff04ea7be400: ffffff001ebccd00
  [ ffffff001ebccd00 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    nanosleep+0x120()
    sys_syscall32+0x101()

ffffff04ddefd000 ffffff04dfdcd6b8 ffffff04f1f51150   1  59 ffffff04ddefd1d6
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/nscd
  stack pointer for thread ffffff04ddefd000: ffffff001ed56d00
  [ ffffff001ed56d00 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    nanosleep+0x120()
    sys_syscall32+0x101()

ffffff04e7df6760 ffffff04dfdcd6b8 ffffff04f1f53840   1  59 ffffff04e7df6936
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/nscd
  stack pointer for thread ffffff04e7df6760: ffffff001ee5cd00
  [ ffffff001ee5cd00 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    nanosleep+0x120()
    sys_syscall32+0x101()

ffffff04e7df63e0 ffffff04dfdcd6b8 ffffff04f1f516e0   1  59 ffffff04e7df65b6
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/nscd
  stack pointer for thread ffffff04e7df63e0: ffffff001ed50d00
  [ ffffff001ed50d00 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    nanosleep+0x120()
    sys_syscall32+0x101()

ffffff04ddf8be20 ffffff04dfdcd6b8 ffffff04f1f52200   1  59 ffffff04ddf8bff6
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/nscd
  stack pointer for thread ffffff04ddf8be20: ffffff001f348d00
  [ ffffff001f348d00 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    nanosleep+0x120()
    sys_syscall32+0x101()

ffffff04e7a96e40 ffffff04dfdcd6b8 ffffff04e0512720   1  59 ffffff04e7a97016
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/nscd
  stack pointer for thread ffffff04e7a96e40: ffffff002012ad00
  [ ffffff002012ad00 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    nanosleep+0x120()
    sys_syscall32+0x101()

ffffff04df979e80 ffffff04dfdcd6b8 ffffff04e05031e0   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/nscd
  stack pointer for thread ffffff04df979e80: ffffff0020179d20
  [ ffffff0020179d20 _resume_from_idle+0xf1() ]
    swtch_to+0xe1()
    shuttle_resume+0x328()
    door_return+0x21a()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff04e8b27c80 ffffff04dfdcd6b8 ffffff04e050f730   1  59 ffffff04e8b27e56
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/nscd
  stack pointer for thread ffffff04e8b27c80: ffffff001eba8d00
  [ ffffff001eba8d00 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    nanosleep+0x120()
    sys_syscall32+0x101()

ffffff04e8b27900 ffffff04dfdcd6b8 ffffff04e050e0f0   1  59 ffffff04e8b27ad6
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/nscd
  stack pointer for thread ffffff04e8b27900: ffffff001ebaed00
  [ ffffff001ebaed00 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    nanosleep+0x120()
    sys_syscall32+0x101()

ffffff04e7e0cae0 ffffff04dfdcd6b8 ffffff04df9c9650   1  59 ffffff04e7e0ccb6
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/nscd
  stack pointer for thread ffffff04e7e0cae0: ffffff001ee07d00
  [ ffffff001ee07d00 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    nanosleep+0x120()
    sys_syscall32+0x101()

ffffff04e9f97c60 ffffff04dfdcd6b8 ffffff04e0513240   1  59 ffffff04e9f97e36
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/nscd
  stack pointer for thread ffffff04e9f97c60: ffffff001f0dfd00
  [ ffffff001f0dfd00 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    nanosleep+0x120()
    sys_syscall32+0x101()

ffffff04e0580400 ffffff04dfdcd6b8 ffffff04df9ccbd0   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/nscd
  stack pointer for thread ffffff04e0580400: ffffff00200d5d20
  [ ffffff00200d5d20 _resume_from_idle+0xf1() ]
    swtch_to+0xe1()
    shuttle_resume+0x328()
    door_return+0x21a()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff04df975700 ffffff04dfdcd6b8 ffffff04df9cf0a0   1  60 ffffff04df968df8
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/nscd
  stack pointer for thread ffffff04df975700: ffffff00200db9c0
  [ ffffff00200db9c0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig+0x14b()
    so_dequeue_msg+0x2a0()
    so_recvmsg+0x181()
    socket_recvmsg+0x3d()
    socket_vop_read+0x52()
    fop_read+0x6b()
    read+0x2b8()
    read32+0x22()
    sys_syscall32+0x101()

ffffff04dfc15ae0 ffffff04dfdcd6b8 ffffff04df9c87c0   1  59 ffffff04dfc15cb6
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/nscd
  stack pointer for thread ffffff04dfc15ae0: ffffff001fc4edf0
  [ ffffff001fc4edf0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    pause+0x48()
    sys_syscall32+0x101()

ffffff04dfc12c80 ffffff04dfdce318 ffffff04df9df100   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /lib/svc/bin/svc.ipfd
  stack pointer for thread ffffff04dfc12c80: ffffff001ee01cd0
  [ ffffff001ee01cd0 _resume_from_idle+0xf1() ]
    swtch_to+0xe1()
    shuttle_resume+0x328()
    door_call+0x2b1()
    doorfs32+0x141()
    sys_syscall32+0x101()

ffffff04e0579500 ffffff04fed2f540 ffffff04df9d0c70   1  49 ffffff04e05796d6
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/ipmon -Ds
  stack pointer for thread ffffff04e0579500: ffffff001ec64d40
  [ ffffff001ec64d40 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    nanosleep+0x120()
    sys_syscall+0x17b()

ffffff04dfc2de80 ffffff04fed301a0 ffffff04df9cc640   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/picl/picld
  stack pointer for thread ffffff04dfc2de80: ffffff001ecdbd30
  [ ffffff001ecdbd30 _resume_from_idle+0xf1() ]
    swtch+0x160()
    shuttle_swtch+0x259()
    door_return+0x242()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff04df9641e0 ffffff04fed301a0 ffffff04df9cd6f0   1  59 ffffff04df9643b6
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/picl/picld
  stack pointer for thread ffffff04df9641e0: ffffff001ed98cc0
  [ ffffff001ed98cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04dfc1f040 ffffff04fed301a0 ffffff04df9ca170   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/picl/picld
  stack pointer for thread ffffff04dfc1f040: ffffff001ed9ed30
  [ ffffff001ed9ed30 _resume_from_idle+0xf1() ]
    swtch+0x160()
    shuttle_swtch+0x259()
    door_return+0x242()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff04e058d8c0 ffffff04fed301a0 ffffff04df9d8070   1  59 ffffff04e058da96
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/picl/picld
  stack pointer for thread ffffff04e058d8c0: ffffff001ecb0df0
  [ ffffff001ecb0df0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    pause+0x48()
    sys_syscall32+0x101()

ffffff001f49ec60 fffffffffbc2ba70                0   0  60 ffffff04e75ea218
  PC: _resume_from_idle+0xf1    TASKQ: zil_clean
  stack pointer for thread ffffff001f49ec60: ffffff001f49eb60
  [ ffffff001f49eb60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001f4aac60 fffffffffbc2ba70                0   0  60 ffffff04e75ea2f0
  PC: _resume_from_idle+0xf1    TASKQ: zil_clean
  stack pointer for thread ffffff001f4aac60: ffffff001f4aab60
  [ ffffff001f4aab60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001f4b0c60 fffffffffbc2ba70                0   0  60 ffffff04e75ea140
  PC: _resume_from_idle+0xf1    TASKQ: zil_clean
  stack pointer for thread ffffff001f4b0c60: ffffff001f4b0b60
  [ ffffff001f4b0b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001e62dc60 fffffffffbc2ba70                0   0  60 ffffff04e75ea068
  PC: _resume_from_idle+0xf1    TASKQ: zil_clean
  stack pointer for thread ffffff001e62dc60: ffffff001e62db60
  [ ffffff001e62db60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001ec3dc60 fffffffffbc2ba70                0   0  60 ffffff04e9b21ec8
  PC: _resume_from_idle+0xf1    TASKQ: zil_clean
  stack pointer for thread ffffff001ec3dc60: ffffff001ec3db60
  [ ffffff001ec3db60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001ec43c60 fffffffffbc2ba70                0   0  60 ffffff04e9b21df0
  PC: _resume_from_idle+0xf1    TASKQ: zil_clean
  stack pointer for thread ffffff001ec43c60: ffffff001ec43b60
  [ ffffff001ec43b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001ec49c60 fffffffffbc2ba70                0   0  60 ffffff04e9b21d18
  PC: _resume_from_idle+0xf1    TASKQ: zil_clean
  stack pointer for thread ffffff001ec49c60: ffffff001ec49b60
  [ ffffff001ec49b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001ec55c60 fffffffffbc2ba70                0   0  60 ffffff04e9b21b68
  PC: _resume_from_idle+0xf1    TASKQ: zil_clean
  stack pointer for thread ffffff001ec55c60: ffffff001ec55b60
  [ ffffff001ec55b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff04e05871e0 ffffff04de62a6b0 ffffff04df9d2620   2 100 ffffff04e05873b6
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/inet/xntpd
  stack pointer for thread ffffff04e05871e0: ffffff001ee8cdc0
  [ ffffff001ee8cdc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    sigsuspend+0xeb()
    sys_syscall32+0x101()

ffffff04dfc208c0 ffffff04de628df0 ffffff04df9d71e0   2 100 ffffff04dfc20a96
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/inet/xntpd
  stack pointer for thread ffffff04dfc208c0: ffffff00201f8dc0
  [ ffffff00201f8dc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    sigsuspend+0xeb()
    sys_syscall32+0x101()

ffffff04ddefda80 ffffff04ffad06c8 ffffff04df9e2c10   1  59 ffffff04e6d69b8a
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/inet/in.ndpd
  stack pointer for thread ffffff04ddefda80: ffffff001ee98c40
  [ ffffff001ee98c40 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    poll_common+0x461()
    pollsys+0xe4()
    sys_syscall32+0x101()

ffffff04e7a85c00 ffffff04dfdc8018 ffffff04f1f4ec80   1  59 ffffff04ea88a5bc
  PC: _resume_from_idle+0xf1    CMD: /bin/mail root
  stack pointer for thread ffffff04e7a85c00: ffffff001ec6aba0
  [ ffffff001ec6aba0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    fifo_read+0xca()
    fop_read+0x6b()
    read+0x2b8()
    read32+0x22()
    sys_syscall32+0x101()

ffffff04e91f7560 ffffff04de62d8c8 ffffff04df9d5ba0   1  59 ffffff04ef3ce818
  PC: _resume_from_idle+0xf1    CMD: df -k
  stack pointer for thread ffffff04e91f7560: ffffff001ed748e0
  [ ffffff001ed748e0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    rrw_enter_read+0x57()
    rrw_enter+0x2c()
    zfs_root+0x3d()
    fsop_root+0x2e()
    traverse+0x61()
    lookuppnvp+0x423()
    lookuppnat+0x12c()
    lookupnameat+0x91()
    lookupname+0x28()
    statvfs64_32+0x39()
    sys_syscall32+0x101()

ffffff04dfc201c0 ffffff04fed2e8e0 ffffff04df9c6bf0   1  59 ffffff04fed2e9a0
  PC: _resume_from_idle+0xf1    CMD: 
  /bin/bash /opt/csw/etc/log.d/scripts/services/zz-disk_space
  stack pointer for thread ffffff04dfc201c0: ffffff001f102c70
  [ ffffff001f102c70 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    waitid+0x2a0()
    waitsys32+0x30()
    sys_syscall32+0x101()

ffffff04e0586060 ffffff04fed30e00 ffffff04e050ccd0   1  59 ffffff04df88957c
  PC: _resume_from_idle+0xf1    CMD: /opt/csw/bin/perl -w /opt/csw/sbin/logwatch
  stack pointer for thread ffffff04e0586060: ffffff001ee86ba0
  [ ffffff001ee86ba0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    fifo_read+0xca()
    fop_read+0x6b()
    read+0x2b8()
    read32+0x22()
    sys_syscall32+0x101()

ffffff04df976880 ffffff04ffabaa70 ffffff04df9c9be0   1  59 ffffff04ffabab30
  PC: _resume_from_idle+0xf1    CMD: sh -c /opt/csw/sbin/logwatch #CSWlogwatch
  stack pointer for thread ffffff04df976880: ffffff001f34ec70
  [ ffffff001f34ec70 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    waitid+0x2a0()
    waitsys32+0x30()
    sys_syscall32+0x101()

ffffff04ea7cc760 ffffff04ffad1328 ffffff04e05086a0   1  59 ffffff04e6d554ea
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/cron
  stack pointer for thread ffffff04ea7cc760: ffffff001ebd2c40
  [ ffffff001ebd2c40 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    poll_common+0x461()
    pollsys+0xe4()
    sys_syscall32+0x101()

ffffff04df97a200 ffffff0501522e18 ffffff04e0512190   1  59 ffffff04ea64144a
  PC: _resume_from_idle+0xf1    CMD: /opt/csw/sbin/exim -bd -q30m
  stack pointer for thread ffffff04df97a200: ffffff001f0f1c40
  [ ffffff001f0f1c40 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    poll_common+0x461()
    pollsys+0xe4()
    sys_syscall32+0x101()

ffffff04e0594380 ffffff04fface1a8 ffffff04e050f1a0   1  59 ffffff04e6d6968a
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/rpcbind
  stack pointer for thread ffffff04e0594380: ffffff001fc02c40
  [ ffffff001fc02c40 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    poll_common+0x461()
    pollsys+0xe4()
    sys_syscall32+0x101()

ffffff04dfc2d780 ffffff04dfdca538 ffffff04e0508c30   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/autofs/automountd
  stack pointer for thread ffffff04dfc2d780: ffffff002027cd30
  [ ffffff002027cd30 _resume_from_idle+0xf1() ]
    swtch+0x160()
    shuttle_swtch+0x259()
    door_return+0x242()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff04e0d7c1c0 ffffff04de62b310 ffffff04e050c740   1  59 ffffff04e0d7c396
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/autofs/automountd
  stack pointer for thread ffffff04e0d7c1c0: ffffff001fb83cc0
  [ ffffff001fb83cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04e7a14ac0 ffffff04de62b310 ffffff04df9e15d0   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/autofs/automountd
  stack pointer for thread ffffff04e7a14ac0: ffffff001fbecd30
  [ ffffff001fbecd30 _resume_from_idle+0xf1() ]
    swtch+0x160()
    shuttle_swtch+0x259()
    door_return+0x242()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff04dfc25720 ffffff04de62b310 ffffff04e0505c40   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/autofs/automountd
  stack pointer for thread ffffff04dfc25720: ffffff001f289d20
  [ ffffff001f289d20 _resume_from_idle+0xf1() ]
    swtch_to+0xe1()
    shuttle_resume+0x328()
    door_return+0x21a()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff001f268c60 fffffffffbc2ba70                0   0  60 fffffffffbcdae78
  PC: _resume_from_idle+0xf1    THREAD: auto_do_unmount()
  stack pointer for thread ffffff001f268c60: ffffff001f268b10
  [ ffffff001f268b10 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait+0xba()
    zone_status_timedwait+0x53()
    auto_do_unmount+0xbc()
    thread_start+8()

ffffff04e0587c60 ffffff04de62b310 ffffff04e05061d0   1  59 ffffff04e0587e36
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/autofs/automountd
  stack pointer for thread ffffff04e0587c60: ffffff001fb7ddf0
  [ ffffff001fb7ddf0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    pause+0x48()
    sys_syscall32+0x101()

ffffff04e0d7e520 ffffff04dfdca538 ffffff04e0511c00   1  59 ffffff04dfdca5f8
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/autofs/automountd
  stack pointer for thread ffffff04e0d7e520: ffffff001fb71c70
  [ ffffff001fb71c70 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    waitid+0x2a0()
    waitsys32+0x30()
    sys_syscall32+0x101()

ffffff04e7a84000 ffffff04ffacbc88 ffffff04e04f96d0   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/inet/inetd start
  stack pointer for thread ffffff04e7a84000: ffffff001ee50d30
  [ ffffff001ee50d30 _resume_from_idle+0xf1() ]
    swtch+0x160()
    shuttle_swtch+0x259()
    door_return+0x242()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff04de0a9e20 ffffff04ffacbc88 ffffff04e0502130   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/inet/inetd start
  stack pointer for thread ffffff04de0a9e20: ffffff001ee44d30
  [ ffffff001ee44d30 _resume_from_idle+0xf1() ]
    swtch+0x160()
    shuttle_swtch+0x259()
    door_return+0x242()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff001ee4ac60 fffffffffbc2ba70                0   0  60 ffffff04e9034d50
  PC: _resume_from_idle+0xf1    THREAD: evch_delivery_thr()
  stack pointer for thread ffffff001ee4ac60: ffffff001ee4aae0
  [ ffffff001ee4aae0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    evch_delivery_hold+0x85()
    evch_delivery_thr+0x23e()
    thread_start+8()

ffffff04e7a14740 ffffff04ffab6c90 ffffff04e050b690   1  60 ffffff04fee1e4c8
  PC: _resume_from_idle+0xf1    CMD: zfs receive -vF thumperpool/mnt
  stack pointer for thread ffffff04e7a14740: ffffff001ecc2880
  [ ffffff001ecc2880 _resume_from_idle+0xf1() ]
    swtch+0x160()
    turnstile_block+0x764()
    rw_enter_sleep+0x1a3()
    dsl_dataset_clone_swap+0x61()
    dmu_recv_end+0x57()
    zfs_ioc_recv+0x31e()
    zfsdev_ioctl+0x10b()
    cdev_ioctl+0x45()
    spec_ioctl+0x83()
    fop_ioctl+0x7b()
    ioctl+0x18e()
    sys_syscall32+0x101()

ffffff04e96f5900 ffffff04ffab91b0 ffffff050157ad70   1  59 ffffff04ffab9270
  PC: _resume_from_idle+0xf1    CMD: 
  /opt/csw/bin/perl -w /opt/csw/sbin/snapshotrcv
  stack pointer for thread ffffff04e96f5900: ffffff002026cc70
  [ ffffff002026cc70 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    waitid+0x2a0()
    waitsys32+0x30()
    sys_syscall32+0x101()

ffffff04dfc2a180 ffffff04ffacbc88 ffffff04e0503770   1  59 ffffff04e6a8f53a
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/inet/inetd start
  stack pointer for thread ffffff04dfc2a180: ffffff001f39cc40
  [ ffffff001f39cc40 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    poll_common+0x461()
    pollsys+0xe4()
    sys_syscall32+0x101()

ffffff04e7a14040 ffffff04ffabc330 ffffff04e0504290   1  59 ffffff04e6cf2a92
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/utmpd
  stack pointer for thread ffffff04e7a14040: ffffff001ee6ec40
  [ ffffff001ee6ec40 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    poll_common+0x461()
    pollsys+0xe4()
    sys_syscall32+0x101()

ffffff04e9b8b200 ffffff04dfdc98d8 ffffff04f1f502c0   1  59 ffffff04e9b8b3d6
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/saf/ttymon
  stack pointer for thread ffffff04e9b8b200: ffffff001ed62df0
  [ ffffff001ed62df0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    pause+0x48()
    sys_syscall32+0x101()

ffffff04e91f71e0 ffffff04fed2dc80 ffffff04e0506760   1  59 ffffff04fee1dbbc
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/saf/sac -t 300
  stack pointer for thread ffffff04e91f71e0: ffffff001fb6bba0
  [ ffffff001fb6bba0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    fifo_read+0xca()
    fop_read+0x6b()
    read+0x2b8()
    read32+0x22()
    sys_syscall32+0x101()

ffffff04dddd2e80 ffffff04ffacfa68 ffffff04e05056b0   1  59 ffffff04dddd3056
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/syslogd
  stack pointer for thread ffffff04dddd2e80: ffffff001fb23cc0
  [ ffffff001fb23cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04dddd2b00 ffffff04ffacfa68 ffffff05015872d0   1  59 ffffff04dddd2cd6
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/syslogd
  stack pointer for thread ffffff04dddd2b00: ffffff001eae3cc0
  [ ffffff001eae3cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04dddd2780 ffffff04ffacfa68 ffffff0501586d40   1  59 ffffff04dddd2956
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/syslogd
  stack pointer for thread ffffff04dddd2780: ffffff001eae9cc0
  [ ffffff001eae9cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04dddd2400 ffffff04ffacfa68 ffffff05015867b0   1  59 ffffff04dddd25d6
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/syslogd
  stack pointer for thread ffffff04dddd2400: ffffff001eaefcc0
  [ ffffff001eaefcc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04dddd2080 ffffff04ffacfa68 ffffff0501586220   1  59 ffffff04dddd2256
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/syslogd
  stack pointer for thread ffffff04dddd2080: ffffff001eaf5cc0
  [ ffffff001eaf5cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04dde7cc00 ffffff04ffacfa68 ffffff0501585c90   1  59 ffffff04dde7cdd6
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/syslogd
  stack pointer for thread ffffff04dde7cc00: ffffff001eafbcc0
  [ ffffff001eafbcc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04dde7c880 ffffff04ffacfa68 ffffff0501585700   1  59 ffffff04dde7ca56
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/syslogd
  stack pointer for thread ffffff04dde7c880: ffffff001eb01cc0
  [ ffffff001eb01cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04dde7c180 ffffff04ffacfa68 ffffff0501584870   1  59 ffffff04e8cf013a
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/syslogd
  stack pointer for thread ffffff04dde7c180: ffffff001eb65c40
  [ ffffff001eb65c40 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    poll_common+0x461()
    pollsys+0xe4()
    sys_syscall32+0x101()

ffffff04e7f158c0 ffffff04ffacfa68 ffffff04f1f4f7a0   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/syslogd
  stack pointer for thread ffffff04e7f158c0: ffffff001ed6ed20
  [ ffffff001ed6ed20 _resume_from_idle+0xf1() ]
    swtch_to+0xe1()
    shuttle_resume+0x328()
    door_return+0x21a()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff04dde7be00 ffffff04ffacfa68 ffffff05015842e0   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/syslogd
  stack pointer for thread ffffff04dde7be00: ffffff001eb6bd20
  [ ffffff001eb6bd20 _resume_from_idle+0xf1() ]
    swtch_to+0xe1()
    shuttle_resume+0x328()
    door_return+0x21a()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff04e8715520 ffffff04ffacfa68 ffffff04f1f4fd30   1  59 ffffff04e87156f6
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/syslogd
  stack pointer for thread ffffff04e8715520: ffffff001ed68ca0
  [ ffffff001ed68ca0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    sigtimedwait+0x1b5()
    sys_syscall32+0x101()

ffffff04e796e020 ffffff04fed2d020 ffffff04e04f9140   1  59 ffffff04e6e32282
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/nfs/statd
  stack pointer for thread ffffff04e796e020: ffffff001ee56c40
  [ ffffff001ee56c40 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    poll_common+0x461()
    pollsys+0xe4()
    sys_syscall32+0x101()

ffffff04e796f520 ffffff05015246d8 ffffff04df9d66c0   1  59 ffffff05214a8900
  PC: _resume_from_idle+0xf1    CMD: 
  ssh -i /etc/snapshotmgr/keys/thumper2.cleo.infra.local_rsa zfstrans at
172.16.24.
  1
  stack pointer for thread ffffff04e796f520: ffffff001e7bfc20
  [ ffffff001e7bfc20 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    so_wait_connected_locked+0x69()
    so_wait_connected+0x40()
    so_connect+0xd2()
    socket_connect+0x47()
    connect+0xec()
    sys_syscall32+0x101()

ffffff04df389e20 ffffff05015208f8 ffffff04df9d5610   1  59 ffffff04ea88a6fc
  PC: _resume_from_idle+0xf1    CMD: 
  /opt/csw/bin/perl -w /opt/csw/sbin/snapshotmgr --nocreate --verbose
  stack pointer for thread ffffff04df389e20: ffffff0020272ba0
  [ ffffff0020272ba0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    fifo_read+0xca()
    fop_read+0x6b()
    read+0x2b8()
    read32+0x22()
    sys_syscall32+0x101()

ffffff04de0aa520 ffffff060325b1c0 ffffff050157a7e0   1  59 ffffff04fee1e4d0
  PC: _resume_from_idle+0xf1    CMD: zfs list
  stack pointer for thread ffffff04de0aa520: ffffff001eddaa10
  [ ffffff001eddaa10 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    dsl_dataset_hold_ref+0x86()
    dsl_dataset_hold+0xa6()
    dmu_objset_open+0x99()
    zfs_ioc_objset_stats+0x33()
    zfs_ioc_dataset_list_next+0x160()
    zfsdev_ioctl+0x10b()
    cdev_ioctl+0x45()
    spec_ioctl+0x83()
    fop_ioctl+0x7b()
    ioctl+0x18e()
    sys_syscall32+0x101()

ffffff04e0579180 ffffff04fed31a60 ffffff04df9cc0b0   1  59 ffffff04fed31b20
  PC: _resume_from_idle+0xf1    CMD: bash
  stack pointer for thread ffffff04e0579180: ffffff001f0fcc70
  [ ffffff001f0fcc70 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    waitid+0x2a0()
    waitsys32+0x30()
    sys_syscall32+0x101()

ffffff04e8b26400 ffffff04fed33320 ffffff04e050d260   1  59 ffffff04fed333e0
  PC: _resume_from_idle+0xf1    CMD: -bash
  stack pointer for thread ffffff04e8b26400: ffffff001ee7ac70
  [ ffffff001ee7ac70 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    waitid+0x2a0()
    waitsys32+0x30()
    sys_syscall32+0x101()

ffffff04e7d95500 ffffff04de625010 ffffff04df9e1040   1  59 ffffff04e6a76f92
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/ssh/sshd
  stack pointer for thread ffffff04e7d95500: ffffff001eb96c40
  [ ffffff001eb96c40 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    poll_common+0x461()
    pollsys+0xe4()
    sys_syscall32+0x101()

ffffff04e9f963e0 ffffff04dfdc8c78 ffffff04df9c60d0   1  59 ffffff0502c4b502
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/ssh/sshd
  stack pointer for thread ffffff04e9f963e0: ffffff001ee80c40
  [ ffffff001ee80c40 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    poll_common+0x461()
    pollsys+0xe4()
    sys_syscall32+0x101()

ffffff04e9b8a780 ffffff0501523a78 ffffff04df9de050   1  60 ffffff04fee1e4d0
  PC: _resume_from_idle+0xf1    CMD: 
  /usr/sbin/zfs list -Ho name,type,creation,mountpoint -t snapshot
  stack pointer for thread ffffff04e9b8a780: ffffff001ec70940
  [ ffffff001ec70940 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    dsl_dataset_hold_ref+0x86()
    dsl_dataset_hold+0xa6()
    dmu_objset_prefetch+0x24()
    findfunc+0x23()
    dmu_objset_find_spa+0x30d()
    dmu_objset_find+0x40()
    zfs_ioc_snapshot_list_next+0x59()
    zfsdev_ioctl+0x10b()
    cdev_ioctl+0x45()
    spec_ioctl+0x83()
    fop_ioctl+0x7b()
    ioctl+0x18e()
    sys_syscall32+0x101()

ffffff04df976180 ffffff04ffab6030 ffffff04df9deb70   1  59 ffffff04ffab60f0
  PC: _resume_from_idle+0xf1    CMD: 
  sh -c /usr/sbin/zfs list -Ho name,type,creation,mountpoint -t snapshot
  stack pointer for thread ffffff04df976180: ffffff0020208c70
  [ ffffff0020208c70 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    waitid+0x2a0()
    waitsys32+0x30()
    sys_syscall32+0x101()

ffffff04e96f5580 ffffff04fed326c0 ffffff0501579730   1  59 ffffff04e6a8f3aa
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/ssh/sshd
  stack pointer for thread ffffff04e96f5580: ffffff001eba2c40
  [ ffffff001eba2c40 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    poll_common+0x461()
    pollsys+0xe4()
    sys_syscall32+0x101()

ffffff04e7df6060 ffffff04de625c70 ffffff04e05107e0   1  59 ffffff04e6ce42d2
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/ssh/sshd
  stack pointer for thread ffffff04e7df6060: ffffff001f366c40
  [ ffffff001f366c40 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    poll_common+0x461()
    pollsys+0xe4()
    sys_syscall32+0x101()

ffffff04e7d903e0 ffffff04de628190 ffffff04df9db5f0   1  10                0
  PC: _resume_from_idle+0xf1    CMD: mdb -k
  stack pointer for thread ffffff04e7d903e0: ffffff001fbf2840
    x86pte_mapin+0x3f()
    x86pte_release_pagetable+0x20()
    x86pte_set+0x136()
    hati_pte_map+0x39c()
    hati_load_common+0x167()
    hat_devload+0x154()
    mmio+0x165()
    mmrw+0x19d()
    mmread+0x22()
    cdev_read+0x3d()
    spec_read+0x270()
    fop_read+0x6b()
    pread+0x22c()
    sys_syscall+0x17b()

ffffff04ddefde00 ffffff04de62cc68 ffffff0501579cc0   1  54 ffffff04de62cd28
  PC: _resume_from_idle+0xf1    CMD: bash
  stack pointer for thread ffffff04ddefde00: ffffff001f3f0c70
  [ ffffff001f3f0c70 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    waitid+0x2a0()
    waitsys32+0x30()
    sys_syscall32+0x101()

ffffff04e058f8a0 ffffff04ffab78f0 ffffff050157b300   1  59 ffffff04ffab79b0
  PC: _resume_from_idle+0xf1    CMD: -bash
  stack pointer for thread ffffff04e058f8a0: ffffff001f3c6c70
  [ ffffff001f3c6c70 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    waitid+0x2a0()
    waitsys32+0x30()
    sys_syscall32+0x101()

ffffff04df9633e0 ffffff04ffab9e10 ffffff04df9c7180   1  59 ffffff0502d193fa
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/ssh/sshd
  stack pointer for thread ffffff04df9633e0: ffffff001eb9cc40
  [ ffffff001eb9cc40 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    poll_common+0x461()
    pollsys+0xe4()
    sys_syscall32+0x101()

ffffff04e7a15540 ffffff0501525338 ffffff0501583230   1  59 ffffff0502c7f272
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/ssh/sshd
  stack pointer for thread ffffff04e7a15540: ffffff001ee92c40
  [ ffffff001ee92c40 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    poll_common+0x461()
    pollsys+0xe4()
    sys_syscall32+0x101()

ffffff04e058fc20 ffffff04dfdcb198 ffffff0501582ca0   1  59 ffffff04e6e3968a
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/ssh/sshd
  stack pointer for thread ffffff04e058fc20: ffffff001eb77c40
  [ ffffff001eb77c40 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    poll_common+0x461()
    pollsys+0xe4()
    sys_syscall32+0x101()

ffffff04e8b26b00 ffffff04ffabb6d0 ffffff04f1f532b0   1  59 ffffff04ea64126a
  PC: _resume_from_idle+0xf1    CMD: 
  /usr/lib/saf/ttymon -g -d /dev/console -l console -m ldterm,ttcompat -h -p thu
  m
  stack pointer for thread ffffff04e8b26b00: ffffff001ee62c40
  [ ffffff001ee62c40 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    poll_common+0x461()
    pollsys+0xe4()
    sys_syscall32+0x101()

ffffff001eb71c60 fffffffffbc2ba70                0   0  60 ffffff04e7d2ca1a
  PC: _resume_from_idle+0xf1    THREAD: svc_thread_creator()
  stack pointer for thread ffffff001eb71c60: ffffff001eb71b70
  [ ffffff001eb71b70 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    svc_thread_creator+0xbf()
    thread_start+8()

ffffff04e92cf520 ffffff04de62c008 ffffff04df9d1790   3  60 ffffff04e7d2ca72
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/nfs/lockd
  stack pointer for thread ffffff04e92cf520: ffffff001ecd5b30
  [ ffffff001ecd5b30 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig+0x14b()
    svc_wait+0xb9()
    nfssys+0x792()
    sys_syscall32+0x101()

ffffff04e7d911e0 ffffff04de62c008 ffffff0501587860   3  60 ffffff04ea9f5a02
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/nfs/lockd
  stack pointer for thread ffffff04e7d911e0: ffffff001f3dec40
  [ ffffff001f3dec40 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    poll_common+0x461()
    pollsys+0xe4()
    sys_syscall32+0x101()

ffffff04e7df6e60 ffffff04ffacd548 ffffff050157dd60   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/nfs/mountd
  stack pointer for thread ffffff04e7df6e60: ffffff001eacbd30
  [ ffffff001eacbd30 _resume_from_idle+0xf1() ]
    swtch+0x160()
    shuttle_swtch+0x259()
    door_return+0x242()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff04ea7be780 ffffff04ffacd548 ffffff0501582180   1  59 ffffff04ea7be956
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/nfs/mountd
  stack pointer for thread ffffff04ea7be780: ffffff001f42cdf0
  [ ffffff001f42cdf0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    pause+0x48()
    sys_syscall32+0x101()

ffffff04ea7cd1e0 ffffff04ffacd548 ffffff050157e2f0   1  59 ffffff04ea7cd3b6
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/nfs/mountd
  stack pointer for thread ffffff04ea7cd1e0: ffffff001ead7df0
  [ ffffff001ead7df0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    pause+0x48()
    sys_syscall32+0x101()

ffffff04e7c85540 ffffff04ffacd548 ffffff0501582710   1  59 ffffff04e0481052
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/nfs/mountd
  stack pointer for thread ffffff04e7c85540: ffffff001f2ccc40
  [ ffffff001f2ccc40 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    poll_common+0x461()
    pollsys+0xe4()
    sys_syscall32+0x101()

ffffff001f426c60 fffffffffbc2ba70                0   0  60 ffffff04e88a8962
  PC: _resume_from_idle+0xf1    THREAD: svc_thread_creator()
  stack pointer for thread ffffff001f426c60: ffffff001f426b70
  [ ffffff001f426b70 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    svc_thread_creator+0xbf()
    thread_start+8()

ffffff04e8715c20 ffffff04ffab8550 ffffff050157ccb0   3  60 ffffff04e88a89ba
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/nfs/nfsd
  stack pointer for thread ffffff04e8715c20: ffffff001fb35b30
  [ ffffff001fb35b30 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig+0x14b()
    svc_wait+0xb9()
    nfssys+0x792()
    sys_syscall32+0x101()

ffffff001f2d2c60 fffffffffbc2ba70                0   0  60 ffffff04e04196ce
  PC: _resume_from_idle+0xf1    THREAD: reaper_thread()
  stack pointer for thread ffffff001f2d2c60: ffffff001f2d2ac0
  [ ffffff001f2d2ac0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait+0xba()
    cv_timedwait_sig_internal+0x31e()
    cv_timedwait_sig+0x1f()
    reaper_thread+0xe1()
    thread_start+8()

ffffff001f432c60 fffffffffbc2ba70                0   0  60 ffffff04da4d6c4e
  PC: _resume_from_idle+0xf1    THREAD: reaper_thread()
  stack pointer for thread ffffff001f432c60: ffffff001f432ac0
  [ ffffff001f432ac0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait+0xba()
    cv_timedwait_sig_internal+0x31e()
    cv_timedwait_sig+0x1f()
    reaper_thread+0xe1()
    thread_start+8()

ffffff001f314c60 fffffffffbc2ba70                0   0  60 ffffff04da4d538e
  PC: _resume_from_idle+0xf1    THREAD: reaper_thread()
  stack pointer for thread ffffff001f314c60: ffffff001f314ac0
  [ ffffff001f314ac0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait+0xba()
    cv_timedwait_sig_internal+0x31e()
    cv_timedwait_sig+0x1f()
    reaper_thread+0xe1()
    thread_start+8()

ffffff001f2c6c60 fffffffffbc2ba70                0   0  60 ffffff04dfb5558e
  PC: _resume_from_idle+0xf1    THREAD: reaper_thread()
  stack pointer for thread ffffff001f2c6c60: ffffff001f2c6ac0
  [ ffffff001f2c6ac0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait+0xba()
    cv_timedwait_sig_internal+0x31e()
    cv_timedwait_sig+0x1f()
    reaper_thread+0xe1()
    thread_start+8()

ffffff001f2e4c60 fffffffffbc2ba70                0   0  60 ffffff04df9315ce
  PC: _resume_from_idle+0xf1    THREAD: reaper_thread()
  stack pointer for thread ffffff001f2e4c60: ffffff001f2e4ac0
  [ ffffff001f2e4ac0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait+0xba()
    cv_timedwait_sig_internal+0x31e()
    cv_timedwait_sig+0x1f()
    reaper_thread+0xe1()
    thread_start+8()

ffffff001f420c60 fffffffffbc2ba70                0   0  60 ffffff04d827818e
  PC: _resume_from_idle+0xf1    THREAD: reaper_thread()
  stack pointer for thread ffffff001f420c60: ffffff001f420ac0
  [ ffffff001f420ac0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait+0xba()
    cv_timedwait_sig_internal+0x31e()
    cv_timedwait_sig+0x1f()
    reaper_thread+0xe1()
    thread_start+8()

ffffff0020266c60 fffffffffbc2ba70                0   0  60 ffffff04df5a570e
  PC: _resume_from_idle+0xf1    THREAD: reaper_thread()
  stack pointer for thread ffffff0020266c60: ffffff0020266ac0
  [ ffffff0020266ac0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait+0xba()
    cv_timedwait_sig_internal+0x31e()
    cv_timedwait_sig+0x1f()
    reaper_thread+0xe1()
    thread_start+8()

ffffff04e8714720 ffffff04ffab8550 ffffff050157d240   3  60 ffffff04ea5d477a
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/nfs/nfsd
  stack pointer for thread ffffff04e8714720: ffffff001fb2fc40
  [ ffffff001fb2fc40 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    poll_common+0x461()
    pollsys+0xe4()
    sys_syscall32+0x101()

ffffff001f3d8c60 fffffffffbc2ba70                0   0  99 ffffff04e75eade8
  PC: _resume_from_idle+0xf1    TASKQ: dtrace_taskq
  stack pointer for thread ffffff001f3d8c60: ffffff001f3d8b60
  [ ffffff001f3d8b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff04e9618e20 ffffff04ffacb028 ffffff04df9d9c40   1  59 ffffff04e9618ff6
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/nfs/nfsmapid
  stack pointer for thread ffffff04e9618e20: ffffff001f262ca0
  [ ffffff001f262ca0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    sigtimedwait+0x1b5()
    sys_syscall32+0x101()

ffffff04e7d91560 ffffff04ffacb028 ffffff04e050a270   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/nfs/nfsmapid
  stack pointer for thread ffffff04e7d91560: ffffff001fc19d30
  [ ffffff001fc19d30 _resume_from_idle+0xf1() ]
    swtch+0x160()
    shuttle_swtch+0x259()
    door_return+0x242()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff04e96191a0 ffffff04ffacb028 ffffff04f1f4e6f0   1  59 ffffff04e9619376
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/nfs/nfsmapid
  stack pointer for thread ffffff04e96191a0: ffffff001f3ccdf0
  [ ffffff001f3ccdf0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    pause+0x48()
    sys_syscall32+0x101()

ffffff04e87158a0 ffffff04dfdcbdf8 ffffff050157c190   1  59 ffffff04e8715a76
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/fm/fmd/fmd
  stack pointer for thread ffffff04e87158a0: ffffff001ebead00
  [ ffffff001ebead00 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    nanosleep+0x120()
    sys_syscall32+0x101()

ffffff04e0594000 ffffff04dfdcbdf8 ffffff04e04fa780   1  59 ffffff04e05941d6
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/fm/fmd/fmd
  stack pointer for thread ffffff04e0594000: ffffff001f354cc0
  [ ffffff001f354cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04e7ce4020 ffffff04dfdcbdf8 ffffff04df9dc6a0   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/fm/fmd/fmd
  stack pointer for thread ffffff04e7ce4020: ffffff001eeb2d30
  [ ffffff001eeb2d30 _resume_from_idle+0xf1() ]
    swtch+0x160()
    shuttle_swtch+0x259()
    door_return+0x242()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff04e0d7c8c0 ffffff04dfdcbdf8 ffffff04df9dd750   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/fm/fmd/fmd
  stack pointer for thread ffffff04e0d7c8c0: ffffff002017fd20
  [ ffffff002017fd20 _resume_from_idle+0xf1() ]
    swtch_to+0xe1()
    shuttle_resume+0x328()
    door_return+0x21a()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff001f444c60 fffffffffbc2ba70                0   0  60 ffffff05019d6da0
  PC: _resume_from_idle+0xf1    THREAD: evch_delivery_thr()
  stack pointer for thread ffffff001f444c60: ffffff001f444ae0
  [ ffffff001f444ae0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    evch_delivery_hold+0x85()
    evch_delivery_thr+0x23e()
    thread_start+8()

ffffff04e7a96ac0 ffffff04dfdcbdf8 ffffff0501583d50   1  59 ffffff04e7a96c96
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/fm/fmd/fmd
  stack pointer for thread ffffff04e7a96ac0: ffffff001fb29cc0
  [ ffffff001fb29cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff001fa33c60 fffffffffbc2ba70                0   0  60 ffffff04e08d2cb8
  PC: _resume_from_idle+0xf1    THREAD: i_mac_notify_thread()
  stack pointer for thread ffffff001fa33c60: ffffff001fa33b70
  [ ffffff001fa33b70 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    i_mac_notify_thread+0xbb()
    thread_start+8()

ffffff001fa70c60 fffffffffbc2ba70                0   0  60 ffffff04e08d25c8
  PC: _resume_from_idle+0xf1    THREAD: i_mac_notify_thread()
  stack pointer for thread ffffff001fa70c60: ffffff001fa70b70
  [ ffffff001fa70b70 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    i_mac_notify_thread+0xbb()
    thread_start+8()

ffffff001fa76c60 fffffffffbc2ba70                0   0  60 ffffff04e08d1ed8
  PC: _resume_from_idle+0xf1    THREAD: i_mac_notify_thread()
  stack pointer for thread ffffff001fa76c60: ffffff001fa76b70
  [ ffffff001fa76b70 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    i_mac_notify_thread+0xbb()
    thread_start+8()

ffffff001f1fbc60 fffffffffbc2ba70                0   0  60 ffffffffc005a8a8
  PC: _resume_from_idle+0xf1    THREAD: start_daemon()
  stack pointer for thread ffffff001f1fbc60: ffffff001f1fbb40
  [ ffffff001f1fbb40 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    md_daemon+0x13c()
    start_daemon+0x1a()
    thread_start+8()

ffffff001f201c60 fffffffffbc2ba70                0   0  60 ffffffffc005a868
  PC: _resume_from_idle+0xf1    THREAD: start_daemon()
  stack pointer for thread ffffff001f201c60: ffffff001f201b40
  [ ffffff001f201b40 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    md_daemon+0x13c()
    start_daemon+0x1a()
    thread_start+8()

ffffff001f207c60 fffffffffbc2ba70                0   0  60 ffffffffc00590c8
  PC: _resume_from_idle+0xf1    THREAD: start_daemon()
  stack pointer for thread ffffff001f207c60: ffffff001f207b40
  [ ffffff001f207b40 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    md_daemon+0x13c()
    start_daemon+0x1a()
    thread_start+8()

ffffff001f20dc60 fffffffffbc2ba70                0   0  60 ffffffffc005a7e8
  PC: _resume_from_idle+0xf1    THREAD: start_daemon()
  stack pointer for thread ffffff001f20dc60: ffffff001f20db40
  [ ffffff001f20db40 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    md_daemon+0x13c()
    start_daemon+0x1a()
    thread_start+8()

ffffff001f213c60 fffffffffbc2ba70                0   0  60 ffffffffc005a7e8
  PC: _resume_from_idle+0xf1    THREAD: start_daemon()
  stack pointer for thread ffffff001f213c60: ffffff001f213b40
  [ ffffff001f213b40 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    md_daemon+0x13c()
    start_daemon+0x1a()
    thread_start+8()

ffffff001f219c60 fffffffffbc2ba70                0   0  60 ffffffffc005a7e8
  PC: _resume_from_idle+0xf1    THREAD: start_daemon()
  stack pointer for thread ffffff001f219c60: ffffff001f219b40
  [ ffffff001f219b40 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    md_daemon+0x13c()
    start_daemon+0x1a()
    thread_start+8()

ffffff001f21fc60 fffffffffbc2ba70                0   0  60 ffffffffc0059048
  PC: _resume_from_idle+0xf1    THREAD: start_daemon()
  stack pointer for thread ffffff001f21fc60: ffffff001f21fb40
  [ ffffff001f21fb40 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    md_daemon+0x13c()
    start_daemon+0x1a()
    thread_start+8()

ffffff001f225c60 fffffffffbc2ba70                0   0  60 ffffffffc005a768
  PC: _resume_from_idle+0xf1    THREAD: start_daemon()
  stack pointer for thread ffffff001f225c60: ffffff001f225b40
  [ ffffff001f225b40 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    md_daemon+0x13c()
    start_daemon+0x1a()
    thread_start+8()

ffffff001f22bc60 fffffffffbc2ba70                0   0  60 ffffffffc005a7a8
  PC: _resume_from_idle+0xf1    THREAD: start_daemon()
  stack pointer for thread ffffff001f22bc60: ffffff001f22bb40
  [ ffffff001f22bb40 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    md_daemon+0x13c()
    start_daemon+0x1a()
    thread_start+8()

ffffff001f231c60 fffffffffbc2ba70                0   0  60 ffffffffc0059088
  PC: _resume_from_idle+0xf1    THREAD: start_daemon()
  stack pointer for thread ffffff001f231c60: ffffff001f231b40
  [ ffffff001f231b40 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    md_daemon+0x13c()
    start_daemon+0x1a()
    thread_start+8()

ffffff001f237c60 fffffffffbc2ba70                0   0  60 ffffffffc00593e8
  PC: _resume_from_idle+0xf1    THREAD: start_daemon()
  stack pointer for thread ffffff001f237c60: ffffff001f237b40
  [ ffffff001f237b40 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    md_daemon+0x13c()
    start_daemon+0x1a()
    thread_start+8()

ffffff001e992c60 fffffffffbc2ba70                0   0  60 ffffffffc0059428
  PC: _resume_from_idle+0xf1    THREAD: start_daemon()
  stack pointer for thread ffffff001e992c60: ffffff001e992b40
  [ ffffff001e992b40 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    md_daemon+0x13c()
    start_daemon+0x1a()
    thread_start+8()

ffffff001ea74c60 fffffffffbc2ba70                0   0  60 ffffff04e55842e8
  PC: _resume_from_idle+0xf1    TASKQ: ah_taskq
  stack pointer for thread ffffff001ea74c60: ffffff001ea74b60
  [ ffffff001ea74b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001ef9ac60 fffffffffbc2ba70                0   0  60 ffffff04df1af128
  PC: _resume_from_idle+0xf1    TASKQ: esp_taskq
  stack pointer for thread ffffff001ef9ac60: ffffff001ef9ab60
  [ ffffff001ef9ab60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001ff44c60 fffffffffbc2ba70                0   0  60 ffffff04df1af050
  PC: _resume_from_idle+0xf1    TASKQ: ipnet
  stack pointer for thread ffffff001ff44c60: ffffff001ff44b60
  [ ffffff001ff44b60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff001ff4ac60 fffffffffbc2ba70                0   0  60 ffffff04df1afc20
  PC: _resume_from_idle+0xf1    TASKQ: ipnet_nic_event_queue
  stack pointer for thread ffffff001ff4ac60: ffffff001ff4ab60
  [ ffffff001ff4ab60 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    taskq_thread+0x10b()
    thread_start+8()

ffffff04e0d7de20 ffffff04dfdcbdf8 ffffff04df9e0740   1  59 ffffff04e0d7dff6
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/fm/fmd/fmd
  stack pointer for thread ffffff04e0d7de20: ffffff0020226cc0
  [ ffffff0020226cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04e91f78e0 ffffff04dfdcbdf8 ffffff04e04fa1f0   1  59 ffffff04e91f7ab6
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/fm/fmd/fmd
  stack pointer for thread ffffff04e91f78e0: ffffff001f3b8cc0
  [ ffffff001f3b8cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04dfc2a880 ffffff04dfdcbdf8 ffffff04df9dd1c0   1  59 ffffff04e6c7cdc2
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/fm/fmd/fmd
  stack pointer for thread ffffff04dfc2a880: ffffff001f0ebc40
  [ ffffff001f0ebc40 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    poll_common+0x461()
    pollsys+0xe4()
    sys_syscall32+0x101()

ffffff04ddefe180 ffffff04dfdcbdf8 ffffff04de23e670   1  59 ffffff04ddefe356
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/fm/fmd/fmd
  stack pointer for thread ffffff04ddefe180: ffffff0020220cc0
  [ ffffff0020220cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04ea7cd560 ffffff04dfdcbdf8 ffffff04f1f52790   1  59 ffffff04ea7cd736
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/fm/fmd/fmd
  stack pointer for thread ffffff04ea7cd560: ffffff001ed3ecc0
  [ ffffff001ed3ecc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04e7a84e00 ffffff04dfdcbdf8 ffffff04e0507280   1  59 ffffff04e7a84fd6
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/fm/fmd/fmd
  stack pointer for thread ffffff04e7a84e00: ffffff001f26ecc0
  [ ffffff001f26ecc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04df979b00 ffffff04dfdcbdf8 ffffff04df9cfbc0   1  59 ffffff04df979cd6
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/fm/fmd/fmd
  stack pointer for thread ffffff04df979b00: ffffff00201dacc0
  [ ffffff00201dacc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04ea7cd8e0 ffffff04dfdcbdf8 ffffff04de23d030   1  59 ffffff04ea7cdab6
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/fm/fmd/fmd
  stack pointer for thread ffffff04ea7cd8e0: ffffff001ebf0cc0
  [ ffffff001ebf0cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04e9b8a400 ffffff04dfdcbdf8 ffffff05015837c0   1  59 ffffff04e9b8a5d6
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/fm/fmd/fmd
  stack pointer for thread ffffff04e9b8a400: ffffff001fc08cc0
  [ ffffff001fc08cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04e86123c0 ffffff04dfdcbdf8 ffffff050157c720   1  59 ffffff04e8612596
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/fm/fmd/fmd
  stack pointer for thread ffffff04e86123c0: ffffff0020124cc0
  [ ffffff0020124cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04e0579c00 ffffff04dfdcbdf8 ffffff04e0509ce0   1  59 ffffff04e0579dd6
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/fm/fmd/fmd
  stack pointer for thread ffffff04e0579c00: ffffff001f396cc0
  [ ffffff001f396cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04dfc11780 ffffff04dfdcbdf8 ffffff04e0510250   1  59 ffffff04dfc11956
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/fm/fmd/fmd
  stack pointer for thread ffffff04dfc11780: ffffff001fb77cc0
  [ ffffff001fb77cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04df96c540 ffffff04dfdcbdf8 ffffff050157d7d0   1  59 ffffff04df96c716
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/fm/fmd/fmd
  stack pointer for thread ffffff04df96c540: ffffff0020191cc0
  [ ffffff0020191cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04de0a93a0 ffffff04dfdcbdf8 ffffff04df9cdc80   1  59 ffffff04de0a9576
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/fm/fmd/fmd
  stack pointer for thread ffffff04de0a93a0: ffffff001ebbacc0
  [ ffffff001ebbacc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04e7a15c40 ffffff04dfdcbdf8 ffffff04f1f4f210   1  59 ffffff04e7a15e16
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/fm/fmd/fmd
  stack pointer for thread ffffff04e7a15c40: ffffff001ebc6cc0
  [ ffffff001ebc6cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04e7a151c0 ffffff04dfdcbdf8 ffffff04df9dcc30   1  59 ffffff04e7a15396
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/fm/fmd/fmd
  stack pointer for thread ffffff04e7a151c0: ffffff001ebc0cc0
  [ ffffff001ebc0cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04ea7bf580 ffffff04dfdcbdf8 ffffff04df9da1d0   1  59 ffffff04ea7bf756
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/fm/fmd/fmd
  stack pointer for thread ffffff04ea7bf580: ffffff001fc48cc0
  [ ffffff001fc48cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04dfc2a500 ffffff04dfdcbdf8 ffffff050157b890   1  59 ffffff04dfc2a6d6
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/fm/fmd/fmd
  stack pointer for thread ffffff04dfc2a500: ffffff0020202cc0
  [ ffffff0020202cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04e92cf1a0 ffffff04dfdcbdf8 ffffff04f1f4e160   1  59 ffffff04e92cf376
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/fm/fmd/fmd
  stack pointer for thread ffffff04e92cf1a0: ffffff001f3d2dc0
  [ ffffff001f3d2dc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    sigsuspend+0xeb()
    sys_syscall32+0x101()

ffffff04e058c3c0 ffffff05015221b8 ffffff0501585170   1  59 ffffff04f23cc902
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/nsrexecd
  stack pointer for thread ffffff04e058c3c0: ffffff001eb5fc80
  [ ffffff001eb5fc80 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    poll_common+0x461()
    pollsys+0xe4()
    sys_syscall+0x17b()

ffffff04e058d540 ffffff05015221b8 ffffff04f1f51c70   1  59 ffffff04e6ce96da
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/nsrexecd
  stack pointer for thread ffffff04e058d540: ffffff001ed4ac80
  [ ffffff001ed4ac80 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    poll_common+0x461()
    pollsys+0xe4()
    sys_syscall+0x17b()

ffffff04e7c84e40 ffffff05015221b8 ffffff04e0506cf0   1  59 ffffff04f23cc7c2
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/nsrexecd
  stack pointer for thread ffffff04e7c84e40: ffffff0020158c80
  [ ffffff0020158c80 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    poll_common+0x461()
    pollsys+0xe4()
    sys_syscall+0x17b()

ffffff04e0595c00 ffffff05015221b8 ffffff04e0502c50   1  59 ffffff04f23cc722
  PC: _resume_from_idle+0xf1    CMD: /usr/sbin/nsrexecd
  stack pointer for thread ffffff04e0595c00: ffffff001f390c80
  [ ffffff001f390c80 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    poll_common+0x461()
    pollsys+0xe4()
    sys_syscall+0x17b()

ffffff04e7a96740 ffffff04ffacc8e8 ffffff04e04f9c60   1  59 ffffff04e7a96916
  PC: _resume_from_idle+0xf1    CMD: /usr/perl5/bin/perl /usr/lib/intrd
  stack pointer for thread ffffff04e7a96740: ffffff001ee3ed00
  [ ffffff001ee3ed00 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    nanosleep+0x120()
    sys_syscall32+0x101()

ffffff04df5edac0 ffffff04de62f188 ffffff04de241bf0   1  59 ffffff04df5edc96
  PC: _resume_from_idle+0xf1    CMD: /lib/svc/bin/svc.startd
  stack pointer for thread ffffff04df5edac0: ffffff001e853cc0
  [ ffffff001e853cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04df5ed740 ffffff04de62f188 ffffff04de241660   1  59 ffffff04e06edc54
  PC: _resume_from_idle+0xf1    CMD: /lib/svc/bin/svc.startd
  stack pointer for thread ffffff04df5ed740: ffffff001ecebb20
  [ ffffff001ecebb20 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig+0x14b()
    cte_get_event+0xd7()
    ctfs_endpoint_ioctl+0xa7()
    ctfs_bu_ioctl+0x51()
    fop_ioctl+0x7b()
    ioctl+0x18e()
    sys_syscall32+0x101()

ffffff04df5ed3c0 ffffff04de62f188 ffffff04de2410d0   1  59 ffffff04e7f4d284
  PC: _resume_from_idle+0xf1    CMD: /lib/svc/bin/svc.startd
  stack pointer for thread ffffff04df5ed3c0: ffffff001e95ab90
  [ ffffff001e95ab90 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    port_getn+0x23b()
    portfs+0x282()
    portfs32+0x57()
    sys_syscall32+0x101()

ffffff001ec37c60 fffffffffbc2ba70                0   0  60 ffffff04dfd0a6d8
  PC: _resume_from_idle+0xf1    THREAD: evch_delivery_thr()
  stack pointer for thread ffffff001ec37c60: ffffff001ec37ae0
  [ ffffff001ec37ae0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    evch_delivery_hold+0x85()
    evch_delivery_thr+0x23e()
    thread_start+8()

ffffff001ec25c60 fffffffffbc2ba70                0   0  60 ffffff04dfd0a668
  PC: _resume_from_idle+0xf1    THREAD: evch_delivery_thr()
  stack pointer for thread ffffff001ec25c60: ffffff001ec25ae0
  [ ffffff001ec25ae0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    evch_delivery_hold+0x85()
    evch_delivery_thr+0x23e()
    thread_start+8()

ffffff04e7d95880 ffffff04de62f188 ffffff04f1f50850   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /lib/svc/bin/svc.startd
  stack pointer for thread ffffff04e7d95880: ffffff001ed5cd30
  [ ffffff001ed5cd30 _resume_from_idle+0xf1() ]
    swtch+0x160()
    shuttle_swtch+0x259()
    door_return+0x242()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff04e8b27200 ffffff04de62f188 ffffff04f1f52d20   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /lib/svc/bin/svc.startd
  stack pointer for thread ffffff04e8b27200: ffffff001ee68d30
  [ ffffff001ee68d30 _resume_from_idle+0xf1() ]
    swtch+0x160()
    shuttle_swtch+0x259()
    door_return+0x242()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff04dfc12580 ffffff04de62f188 ffffff04e0508110   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /lib/svc/bin/svc.startd
  stack pointer for thread ffffff04dfc12580: ffffff001f274d30
  [ ffffff001f274d30 _resume_from_idle+0xf1() ]
    swtch+0x160()
    shuttle_swtch+0x259()
    door_return+0x242()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff04dfc26520 ffffff04de62f188 ffffff04df9dc110   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /lib/svc/bin/svc.startd
  stack pointer for thread ffffff04dfc26520: ffffff001f27cd30
  [ ffffff001f27cd30 _resume_from_idle+0xf1() ]
    swtch+0x160()
    shuttle_swtch+0x259()
    door_return+0x242()
    doorfs32+0x134()
    sys_syscall32+0x101()

ffffff001ec2bc60 fffffffffbc2ba70                0   0  60 ffffff04dfd0a588
  PC: _resume_from_idle+0xf1    THREAD: evch_delivery_thr()
  stack pointer for thread ffffff001ec2bc60: ffffff001ec2bae0
  [ ffffff001ec2bae0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    evch_delivery_hold+0x85()
    evch_delivery_thr+0x23e()
    thread_start+8()

ffffff04df5ed040 ffffff04de62f188 ffffff04de240b40   1  59 ffffff04df5ed216
  PC: _resume_from_idle+0xf1    CMD: /lib/svc/bin/svc.startd
  stack pointer for thread ffffff04df5ed040: ffffff001e960cc0
  [ ffffff001e960cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04df97c8e0 ffffff04de62f188 ffffff04de23f720   1  59                0
  PC: _resume_from_idle+0xf1    CMD: /lib/svc/bin/svc.startd
  stack pointer for thread ffffff04df97c8e0: ffffff001e972cd0
  [ ffffff001e972cd0 _resume_from_idle+0xf1() ]
    swtch_to+0xe1()
    shuttle_resume+0x328()
    door_call+0x2b1()
    doorfs32+0x141()
    sys_syscall32+0x101()

ffffff04df97c560 ffffff04de62f188 ffffff04de23f190   1  59 ffffff04df97c736
  PC: _resume_from_idle+0xf1    CMD: /lib/svc/bin/svc.startd
  stack pointer for thread ffffff04df97c560: ffffff001e978cc0
  [ ffffff001e978cc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    cv_waituntil_sig+0x135()
    lwp_park+0x157()
    syslwp_park+0x31()
    sys_syscall32+0x101()

ffffff04df389720 ffffff04de62f188 ffffff04de24a0c0   1  59 ffffff04df3898f6
  PC: _resume_from_idle+0xf1    CMD: /lib/svc/bin/svc.startd
  stack pointer for thread ffffff04df389720: ffffff001f246dc0
  [ ffffff001f246dc0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait_sig_swap_core+0x170()
    cv_wait_sig_swap+0x18()
    sigsuspend+0x109()
    sys_syscall32+0x101()

ffffff04df38ac20 ffffff04de632308 ffffff04de24b700   1  59 ffffff04df789e52
  PC: _resume_from_idle+0xf1    CMD: /sbin/init
  stack pointer for thread ffffff04df38ac20: ffffff001e936c40
  [ ffffff001e936c40 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait_sig_internal+0x1d6()
    cv_waituntil_sig+0xba()
    poll_common+0x461()
    pollsys+0xe4()
    sys_syscall32+0x101()

ffffff001e94ec60 ffffff04de6316a8                0   0  97 ffffff04de631768
  PC: _resume_from_idle+0xf1    THREAD: pageout_scanner()
  stack pointer for thread ffffff001e94ec60: ffffff001e94eb20
  [ ffffff001e94eb20 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    pageout_scanner+0x131()
    thread_start+8()

ffffff04df38a8a0 ffffff04de6316a8 ffffff04de24b170   0  98 fffffffffbcca8c8
  PC: _resume_from_idle+0xf1    CMD: pageout
  stack pointer for thread ffffff04df38a8a0: ffffff001e93ce30
  [ ffffff001e93ce30 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    pageout+0x261()
    thread_start+8()

ffffff04df38a520 ffffff04de630a48 ffffff04de24abe0   0  60 fffffffffbcdbe84
  PC: _resume_from_idle+0xf1    CMD: fsflush
  stack pointer for thread ffffff04df38a520: ffffff001e942e00
  [ ffffff001e942e00 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    fsflush+0x1fd()
    thread_start+8()

ffffff001e948c60 fffffffffbc2ba70                0   0  60 fffffffffbce8b98
  PC: _resume_from_idle+0xf1    THREAD: mod_uninstall_daemon()
  stack pointer for thread ffffff001e948c60: ffffff001e948ba0
  [ ffffff001e948ba0 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_wait+0x61()
    mod_uninstall_daemon+0x97()
    thread_start+8()

ffffff001e954c60 fffffffffbc2ba70                0   0  60 fffffffffbccaa00
  PC: _resume_from_idle+0xf1    THREAD: seg_pasync_thread()
  stack pointer for thread ffffff001e954c60: ffffff001e954b80
  [ ffffff001e954b80 _resume_from_idle+0xf1() ]
    swtch+0x160()
    cv_timedwait+0xba()
    seg_pasync_thread+0xcd()
    thread_start+8()

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: Digital signature
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090402/4e673878/attachment.bin>

Andrew Robert Nicols

2009-Apr-03 12:38 UTC

head link

[zfs-discuss] ZFS Locking Up periodically

On Wed, Apr 01, 2009 at 06:49:14PM +0600, Victor Latushkin
wrote:> I suspect that this behaviour may be caused by bug in the ohci driver.  
> If you do not use rKVM for console and instead redirect it to serial  
> port, you can easily exclude ohci driver from loading by adding to the 
> /etc/system
>
> exclude: drv/ohci
Sadly, no joy with this one. It seems to be hanging particularly on ZFS
receives and a reboot -d does not do the job to reboot. The only way I can
get the box to reboot is to force it not to sync disks as this hangs (not
helpful).

I''ve generated a core dump but not sure where the best place to start
analysis of this is - any tips would be appreciated as I''ve not been
into
the nitty gritty of the Solaris kernel yet.

Thanks in advance,

Andrew

-- 
Systems Developer

e: andrew.nicols at luns.net.uk
im: a.nicols at jabber.lancs.ac.uk
t: +44 (0)1524 5 10147

Lancaster University Network Services is a limited company registered in
England and Wales. Registered number: 4311892. Registered office:
University House, Lancaster University, Lancaster, LA1 4YW
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: Digital signature
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090403/c175000f/attachment.bin>

Andrew Robert Nicols

2009-Apr-17 11:29 UTC

head link

[zfs-discuss] ZFS Locking Up periodically

I''m still seeing this problem frequently and the suggestions Viktor
made
below haven''t helped (exclude: drv/ohci in /etc/system).

I''ve got a selection of core dumps for analysis if anyone can suggest
what
analysis to do with them.

I''ve also replicated this on a second identical X4500 which was running
Nevada release 110 though I''ve only seen it fail once so far.
I''ve just
upgraded this box to Nevada 112. I''m not sure whether it''s
related to the
sending/receiving of snapshots but that''s the only FS activity on these
two
boxes.

TIA,

Andrew

On Wed, Apr 01, 2009 at 06:49:14PM +0600, Victor Latushkin
wrote:> At the very least you can try to collect threadlist with
>
> echo "::threadlist -v" | mdb -k
>
> If you need to reboot your box anyway, use "reboot -d" -
it''ll generate
> kernel crash dump which can be analyses later with mdb and/or  
> SolarisCAT.
>
> I suspect that this behaviour may be caused by bug in the ohci driver.  
> If you do not use rKVM for console and instead redirect it to serial  
> port, you can easily exclude ohci driver from loading by adding to the 
> /etc/system
>
> exclude: drv/ohci
>
> I do not have bug number handy but if it helps, I''ll look it up
for you
> next week
>
> Victor
>
> Sent from my iPhone
>
> On Apr 1, 2009, at 14:50, Andrew Robert Nicols <andrew.nicols at
luns.net.uk
> > wrote:
>
>> I''ve recently re-installed an X4500 running Nevada b109 and
have been
>> experiencing ZFS lock ups regularly (perhaps once every 2-3 days).
>>
>> The machine is a backup server and receives hourly ZFS snapshots from
>> another thumper - as such, the amount of zfs activity tends to be
>> reasonably high. After about 48 - 72 hours, the file system seems to  
>> lock
>> up and I''m unable to do anything with the zfs system - e.g.
zfs list,
>> df on
>> the file system in question, zfs receive, etc. zpool does still list
>> information and targetted zfs list does work. NFS also locks up.
I''m
>> unable
>> to test whether I can write as the file system is read only.
>>
>> The structure of the pool is something like:
>> /thumperpool      <-- Does not lock up
>> /thumperpool/mnt  <-- Does lock up and is the file system receiving
>> lots of
>>                      snapshots
>>
>> The server is currently in it''s locked state so if anyone can
suggest
>> useful diagnostics to run on it while it''s like this, please
get back
>> to me
>> asap. I will need to restart the box so that our backups
aren''t too
>> out of
>> sync this afternoon.
>>
>> Thanks in advance,
>>
>> Andrew Nicols
>>
>> -- 
>> Systems Developer
>>
>> e: andrew.nicols at luns.net.uk
>> im: a.nicols at jabber.lancs.ac.uk
>> t: +44 (0)1524 5 10147
>>
>> Lancaster University Network Services is a limited company registered 
>> in
>> England and Wales. Registered number: 4311892. Registered office:
>> University House, Lancaster University, Lancaster, LA1 4YW
>> _______________________________________________
>> zfs-discuss mailing list
>> zfs-discuss at opensolaris.org
>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
-- 
Systems Developer

e: andrew.nicols at luns.net.uk
im: a.nicols at jabber.lancs.ac.uk
t: +44 (0)1524 5 10147

Lancaster University Network Services is a limited company registered in
England and Wales. Registered number: 4311892. Registered office:
University House, Lancaster University, Lancaster, LA1 4YW
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: Digital signature
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090417/cc83322c/attachment.bin>

Andrew Robert Nicols

2009-Apr-17 14:45 UTC

head link

[zfs-discuss] ZFS Locking Up periodically

On Fri, Apr 17, 2009 at 12:29:23PM +0100, Andrew Robert Nicols
wrote:> I''m still seeing this problem frequently and the suggestions
Viktor made
> below haven''t helped (exclude: drv/ohci in /etc/system).
> 
> I''ve got a selection of core dumps for analysis if anyone can
suggest what
> analysis to do with them.
> 
> I''ve also replicated this on a second identical X4500 which was
running
> Nevada release 110 though I''ve only seen it fail once so far.
I''ve just
> upgraded this box to Nevada 112. I''m not sure whether
it''s related to the
> sending/receiving of snapshots but that''s the only FS activity on
these two
Looks like I''m also seeing this on Nevada 112. Since the only way I can
get
the box to reboot is to reset /SYS or reboot with -n to prevent it syncing
to disk, I haven''t on this occasion managed to get a core dump:

panic[cpu3]/thread=ffffff04ebb88e20: forced crash dump initiated at user
request

ffffff001f386e60 genunix:kadmin+18e ()
ffffff001f386ec0 genunix:uadmin+10a ()
ffffff001f386f10 unix:brand_sys_syscall32+197 ()

dumping to /dev/zvol/dsk/rpool/dump, offset 65536, content: kernel
79% done: 757698 pages dumped, compression ratio 3.00, dump failed: error 28
rebooting...

Andrew

-- 
Systems Developer

e: andrew.nicols at luns.net.uk
im: a.nicols at jabber.lancs.ac.uk
t: +44 (0)1524 5 10147

Lancaster University Network Services is a limited company registered in
England and Wales. Registered number: 4311892. Registered office:
University House, Lancaster University, Lancaster, LA1 4YW
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: Digital signature
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090417/1cefc576/attachment.bin>

Apparently Analagous Threads

Search for more reasonably related threads

zfs discuss - Apr 2009 - ZFS Locking Up periodically

[zfs-discuss] ZFS Locking Up periodically

[zfs-discuss] ZFS Locking Up periodically

[zfs-discuss] ZFS Locking Up periodically

[zfs-discuss] ZFS Locking Up periodically

[zfs-discuss] ZFS Locking Up periodically

Apparently Analagous Threads