Recently a number of my production 12.0 systems have experienced what I can
only gather is a ZFS deadlock related to vnodes. It seems similar to the
relatively recent FreeBSD-EN-18:18.zfs (ZFS vnode reclaim deadlock)
problem. Previously the same systems were running 11.1-RELEASE without
problem.
Threads are always stuck with the stack around
vn_lock->zfs_root->lookup->namei. When the system is in this state, a
simple `ls /` or `ls /tmp` always hangs, but other datasets seem
unaffected. I have a fairly straightforward ZFS root setup on a single pool
with one SSD. The workload is a ruby/rails/nginx/postgresql backed web
application combined with some data warehousing and other periodic tasks.
Sometimes I can remote SSH in, other times that fails because the user
shell fails to load, and I can run commands via `ssh ... command`.
Sometimes the system is not accessible remotely at all, or it eventually
becomes inaccessible if left long enough in this state. I always have to
physically reboot the device because the shutdown procedure fails. The
network stack (e.g. ping) seems to work completely fine whilst this is
going on, until you try to interact with or spawn a process that hits the
deadlock.
Like previous similar ZFS deadlock issues, increasing kern.vnodes seems to
make the system last longer by up to a few weeks, but is still a bandaid.
However, I have yet to witness vnodes usage actually getting close to the
maximum.
I haven't had any luck reproducing this reliably, but eventually it happens
after a few days or a few weeks... I managed to connect to a system in this
state and grab a procstat and get (hopefully) something useful out of kgdb.
I will note that although I was able to install debug symbols, I couldn't
manage to get the source files onto it for kgdb purposes before I lost SSH
access.
I am hoping someone can help me figure out if this is a legitimate bug, or
something already fixed in 12-STABLE. I wish I could reproduce it reliably
to try against STABLE, but there doesn't appear to be any related ZFS fixes
not in RELEASE. Thanks.
Below is an abbreviated procstat and what I was able to get out of kgdb for
an affected thread. Note that the thread backtrace is from a simple `ls`
command. The procstat dump below is truncated because my last attempt to
send this was rejected by this list for being too long - so a number of
sh/cron processes and some zfs threads in a hung state were removed.
ld# kgdb
GNU gdb (GDB) 8.2.1 [GDB v8.2.1 for FreeBSD]
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd12.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to
"word"...
Reading symbols from /boot/kernel/kernel...Reading symbols from
/usr/lib/debug//boot/kernel/kernel.debug...done.
done.
sched_switch (td=0xfffff8002452a000, newtd=0xfffff80003625580,
flags=<optimized out>)
at /usr/src/sys/kern/sched_ule.c:2112
2112 /usr/src/sys/kern/sched_ule.c: No such file or directory.
(kgdb) tid 102023
(kgdb) bt
#0 sched_switch (td=0xfffff801a83dc580, newtd=0xfffff80003550580,
flags=<optimized out>)
at /usr/src/sys/kern/sched_ule.c:2112
#1 0xffffffff80d0e0a1 in mi_switch (flags=<optimized out>, newtd=0x0) at
/usr/src/sys/kern/kern_synch.c:439
#2 0xffffffff80d5c80c in sleepq_wait (wchan=<unavailable>,
pri=<unavailable>)
at /usr/src/sys/kern/subr_sleepqueue.c:692
#3 0xffffffff80cd9105 in sleeplk (lk=0xfffff800247307e8, flags=<optimized
out>, ilk=<optimized out>,
wmesg=<optimized out>, pri=<optimized out>, timo=51, queue=1) at
/usr/src/sys/kern/kern_lock.c:300
#4 0xffffffff80cd7f85 in lockmgr_slock_hard (lk=<optimized out>,
flags=<optimized out>, ilk=<optimized out>,
file=<optimized out>, line=0, lwa=<optimized out>) at
/usr/src/sys/kern/kern_lock.c:646
#5 0xffffffff813acc5e in VOP_LOCK1_APV (vop=<optimized out>,
a=0xfffffe00f89dd450) at vnode_if.c:2087
#6 0xffffffff80de2820 in VOP_LOCK1 (vp=0xfffff80024730780, flags=2105344,
file=0xffffffff814d4f74
"/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c",
line=2074)
at ./vnode_if.h:859
#7 _vn_lock (vp=0xfffff80024730780, flags=2105344,
file=0xffffffff814d4f74
"/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c",
line=2074)
at /usr/src/sys/kern/vfs_vnops.c:1533
#8 0xffffffff8049f68d in zfs_root (vfsp=<optimized out>, flags=2105344,
vpp=0xfffffe00f89dd558)
at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c:2074
#9 0xffffffff80dc5d43 in lookup (ndp=0xfffffe00f89dd780) at
/usr/src/sys/kern/vfs_lookup.c:961
#10 0xffffffff80dc4f9b in namei (ndp=0xfffffe00f89dd780) at
/usr/src/sys/kern/vfs_lookup.c:444
#11 0xffffffff80ddc637 in kern_statat (td=0xfffff801a83dc580,
flag=<optimized out>, fd=<optimized out>,
path=0x800729c78 <error: Cannot access memory at address 0x800729c78>,
pathseg=UIO_USERSPACE,
sbp=0xfffffe00f89dd898, hook=0x0) at
/usr/src/sys/kern/vfs_syscalls.c:2283
#12 0xffffffff80ddcdcf in sys_fstatat (td=<unavailable>,
uap=0xfffff801a83dc940)
at /usr/src/sys/kern/vfs_syscalls.c:2260
#13 0xffffffff81222449 in syscallenter (td=<optimized out>)
at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:135
#14 amd64_syscall (td=0xfffff801a83dc580, traced=0) at
/usr/src/sys/amd64/amd64/trap.c:1076
#15 <signal handler called>
#16 0x000000080047941a in ?? ()
Backtrace stopped: Cannot access memory at address 0x7fffffffe178
(kgdb) fr 7
#7 _vn_lock (vp=0xfffff80024730780, flags=2105344,
file=0xffffffff814d4f74
"/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c",
line=2074)
at /usr/src/sys/kern/vfs_vnops.c:1533
1533 /usr/src/sys/kern/vfs_vnops.c: No such file or directory.
(kgdb) print *vp
$1 = {v_tag = 0xffffffff8144af45 "zfs", v_op = 0xffffffff81c64fd0
<zfs_vnodeops>, v_data = 0xfffff800247e8110,
v_mount = 0xfffff800247e1000, v_nmntvnodes = {tqe_next 0xfffff800247305a0,
tqe_prev = 0xfffff800247e1060}, {
v_mountedhere = 0x0, v_unpcb = 0x0, v_rdev = 0x0, v_fifoinfo = 0x0},
v_hashlist = {le_next = 0x0,
le_prev = 0x0}, v_cache_src = {lh_first = 0xfffff8016f1ff5b0},
v_cache_dst = {tqh_first = 0x0,
tqh_last = 0xfffff800247307d0}, v_cache_dd = 0x0, v_lock = {lock_object
= {
lo_name = 0xffffffff8144af45 "zfs", lo_flags = 117112832,
lo_data 0, lo_witness = 0x0},
lk_lock = 18446735294395164034, lk_exslpfail = 0, lk_timo = 51, lk_pri
= 96}, v_interlock = {lock_object = {
lo_name = 0xffffffff814e4508 "vnode interlock", lo_flags =
16973824,
lo_data = 0, lo_witness = 0x0},
mtx_lock = 0}, v_vnlock = 0xfffff800247307e8, v_actfreelist = {tqe_next
= 0x0,
tqe_prev = 0xfffff80024730660}, v_bufobj = {bo_lock = {lock_object = {
lo_name = 0xffffffff814a9e5f "bufobj interlock", lo_flags
86179840, lo_data = 0, lo_witness = 0x0},
rw_lock = 1}, bo_ops = 0xffffffff81d38600 <buf_ops_bio>, bo_object
0xfffff80006f5d100, bo_synclist = {
le_next = 0x0, le_prev = 0x0}, bo_private = 0xfffff80024730780,
bo_clean = {bv_hd = {tqh_first = 0x0,
tqh_last = 0xfffff80024730898}, bv_root = {pt_root = 0}, bv_cnt 0},
bo_dirty = {bv_hd = {
tqh_first = 0x0, tqh_last = 0xfffff800247308b8}, bv_root = {pt_root
= 0}, bv_cnt = 0}, bo_numoutput = 0,
bo_flag = 0, bo_domain = 1, bo_bsize = 131072}, v_pollinfo = 0x0,
v_label = 0x0, v_lockf = 0x0, v_rl = {
rl_waiters = {tqh_first = 0x0, tqh_last = 0xfffff80024730908},
rl_currdep = 0x0}, v_cstart = 0, v_lasta = 0,
v_lastw = 0, v_clen = 0, v_holdcnt = 241, v_usecount = 240, v_iflag 512,
v_vflag = 1, v_mflag = 0,
v_writecount = 0, v_hash = 4, v_type = VDIR}
(kgdb)
ld# procstat -kka
PID TID COMM TDNAME KSTACK
0 100000 kernel swapper mi_switch+0xe1
sleepq_timedwait+0x2f _sleep+0x217 swapper+0x6b btext+0x2c
0 100012 kernel thread taskq mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100014 kernel kqueue_ctx taskq mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100015 kernel config_0 mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
0 100017 kernel aiod_kick taskq mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100018 kernel if_io_tqg_0 mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
0 100019 kernel if_io_tqg_1 mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
0 100020 kernel if_io_tqg_2 mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
0 100021 kernel if_io_tqg_3 mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
0 100022 kernel if_io_tqg_4 mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
0 100023 kernel if_io_tqg_5 mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
0 100024 kernel if_io_tqg_6 mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
0 100025 kernel if_io_tqg_7 mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
0 100026 kernel softirq_0 mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
0 100027 kernel softirq_1 mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
0 100028 kernel softirq_2 mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
0 100029 kernel softirq_3 mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
0 100030 kernel softirq_4 mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
0 100031 kernel softirq_5 mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
0 100032 kernel softirq_6 mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
0 100033 kernel softirq_7 mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
0 100034 kernel if_config_tqg_0 mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 gtaskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
0 100056 kernel firmware taskq mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100057 kernel crypto_0 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100058 kernel crypto_1 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100059 kernel crypto_2 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100060 kernel crypto_3 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100061 kernel crypto_4 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100062 kernel crypto_5 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100063 kernel crypto_6 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100064 kernel crypto_7 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100096 kernel system_taskq_0 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100097 kernel system_taskq_1 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100098 kernel system_taskq_2 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100099 kernel system_taskq_3 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100100 kernel system_taskq_4 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100101 kernel system_taskq_5 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100102 kernel system_taskq_6 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100103 kernel system_taskq_7 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100104 kernel mca taskq mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 taskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
0 100109 kernel arc_prune_0 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100110 kernel arc_prune_1 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100111 kernel arc_prune_2 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100112 kernel arc_prune_3 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100113 kernel arc_prune_4 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100114 kernel arc_prune_5 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100115 kernel arc_prune_6 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100116 kernel arc_prune_7 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100119 kernel dbu_evict mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100121 kernel z_vdev_file_0 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100122 kernel z_vdev_file_1 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100123 kernel z_vdev_file_2 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100124 kernel z_vdev_file_3 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100125 kernel z_vdev_file_4 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100126 kernel z_vdev_file_5 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100127 kernel z_vdev_file_6 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100128 kernel z_vdev_file_7 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100129 kernel z_vdev_file_8 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100130 kernel z_vdev_file_9 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100131 kernel z_vdev_file_10 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100132 kernel z_vdev_file_11 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100133 kernel z_vdev_file_12 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100134 kernel z_vdev_file_13 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100135 kernel z_vdev_file_14 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100136 kernel z_vdev_file_15 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100138 kernel zfsvfs mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100143 kernel acpi_task_0 mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 taskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
0 100144 kernel acpi_task_1 mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 taskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
0 100145 kernel acpi_task_2 mi_switch+0xe1
sleepq_wait+0x2c msleep_spin_sbt+0x177 taskqueue_thread_loop+0xc3
fork_exit+0x83 fork_trampoline+0xe
0 100146 kernel CAM taskq mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100167 kernel zio_null_issue mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100168 kernel zio_null_intr mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100169 kernel zio_read_issue_0 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100170 kernel zio_read_issue_1 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100171 kernel zio_read_issue_2 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100172 kernel zio_read_issue_3 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100173 kernel zio_read_issue_4 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100174 kernel zio_read_issue_5 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100175 kernel zio_read_issue_6 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100176 kernel zio_read_issue_7 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100177 kernel zio_read_intr_0_0 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100178 kernel zio_read_intr_0_1 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100179 kernel zio_read_intr_0_2 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100180 kernel zio_read_intr_0_3 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100181 kernel zio_read_intr_0_4 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100182 kernel zio_read_intr_0_5 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100183 kernel zio_read_intr_0_6 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100184 kernel zio_read_intr_0_7 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100185 kernel zio_read_intr_0_8 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100186 kernel zio_read_intr_0_9 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100187 kernel zio_read_intr_0_10 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100188 kernel zio_read_intr_0_11 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100189 kernel zio_read_intr_1_0 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100393 kernel zio_free_intr mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100394 kernel zio_claim_issue mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100395 kernel zio_claim_intr mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100396 kernel zio_ioctl_issue mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100397 kernel zio_ioctl_intr mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100403 kernel dp_sync_taskq_0 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100404 kernel dp_sync_taskq_1 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100405 kernel dp_sync_taskq_2 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100406 kernel dp_sync_taskq_3 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100407 kernel dp_sync_taskq_4 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100408 kernel dp_sync_taskq_5 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100409 kernel dp_zil_clean_taskq_ mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100410 kernel dp_zil_clean_taskq_ mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100411 kernel dp_zil_clean_taskq_ mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100412 kernel dp_zil_clean_taskq_ mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100413 kernel dp_zil_clean_taskq_ mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100414 kernel dp_zil_clean_taskq_ mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100415 kernel dp_zil_clean_taskq_ mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100416 kernel dp_zil_clean_taskq_ mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100417 kernel zfs_vn_rele_taskq mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100418 kernel metaslab_group_task mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100419 kernel metaslab_group_task mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100420 kernel metaslab_group_task mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
0 100421 kernel metaslab_group_task mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 taskqueue_thread_loop+0xf1 fork_exit+0x83
fork_trampoline+0xe
1 100002 init - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
kern_wait6+0x66d sys_wait4+0x78 amd64_syscall+0x369
fast_syscall_common+0x101
2 100065 crypto - mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 crypto_proc+0x2e5 fork_exit+0x83
fork_trampoline+0xe
3 100066 crypto returns 0 - mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 crypto_ret_proc+0x1cd fork_exit+0x83
fork_trampoline+0xe
4 100067 crypto returns 1 - mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 crypto_ret_proc+0x1cd fork_exit+0x83
fork_trampoline+0xe
5 100068 crypto returns 2 - mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 crypto_ret_proc+0x1cd fork_exit+0x83
fork_trampoline+0xe
6 100069 crypto returns 3 - mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 crypto_ret_proc+0x1cd fork_exit+0x83
fork_trampoline+0xe
7 100070 crypto returns 4 - mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 crypto_ret_proc+0x1cd fork_exit+0x83
fork_trampoline+0xe
8 100071 crypto returns 5 - mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 crypto_ret_proc+0x1cd fork_exit+0x83
fork_trampoline+0xe
9 100072 crypto returns 6 - mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 crypto_ret_proc+0x1cd fork_exit+0x83
fork_trampoline+0xe
10 100001 audit - mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 audit_worker+0x73 fork_exit+0x83
fork_trampoline+0xe
11 100003 idle idle: cpu0 acpi_cpu_idle+0x2e7
cpu_idle_acpi+0x3f cpu_idle+0xa7 sched_idletd+0x515 fork_exit+0x83
fork_trampoline+0xe
11 100004 idle idle: cpu1 mi_switch+0xe1
critical_exit_preempt+0x58 sched_idletd+0x515 fork_exit+0x83
fork_trampoline+0xe
11 100005 idle idle: cpu2 acpi_cpu_idle+0x2e7
cpu_idle_acpi+0x3f cpu_idle+0xa7 sched_idletd+0x515 fork_exit+0x83
fork_trampoline+0xe
11 100006 idle idle: cpu3 mi_switch+0xe1
critical_exit_preempt+0x58 sched_idletd+0x515 fork_exit+0x83
fork_trampoline+0xe
11 100007 idle idle: cpu4 acpi_cpu_idle+0x2e7
cpu_idle_acpi+0x3f cpu_idle+0xa7 sched_idletd+0x515 fork_exit+0x83
fork_trampoline+0xe
11 100008 idle idle: cpu5 mi_switch+0xe1
critical_exit_preempt+0x58 sched_idletd+0x515 fork_exit+0x83
fork_trampoline+0xe
11 100009 idle idle: cpu6 acpi_cpu_idle+0x2e7
cpu_idle_acpi+0x3f cpu_idle+0xa7 sched_idletd+0x515 fork_exit+0x83
fork_trampoline+0xe
11 100010 idle idle: cpu7 acpi_cpu_idle+0x2e7
cpu_idle_acpi+0x3f cpu_idle+0xa7 sched_idletd+0x515 fork_exit+0x83
fork_trampoline+0xe
12 100011 intr swi6: Giant taskq mi_switch+0xe1
ithread_loop+0x393 fork_exit+0x83 fork_trampoline+0xe
12 100013 intr swi5: fast taskq
12 100016 intr swi6: task queue mi_switch+0xe1
ithread_loop+0x393 fork_exit+0x83 fork_trampoline+0xe
12 100035 intr swi4: clock (0) mi_switch+0xe1
ithread_loop+0x393 fork_exit+0x83 fork_trampoline+0xe
12 100036 intr swi4: clock (1)
12 100037 intr swi4: clock (2)
12 100038 intr swi4: clock (3)
12 100039 intr swi4: clock (4)
12 100040 intr swi4: clock (5)
12 100041 intr swi4: clock (6)
12 100042 intr swi4: clock (7)
12 100043 intr swi1: netisr 0 mi_switch+0xe1
ithread_loop+0x393 fork_exit+0x83 fork_trampoline+0xe
12 100044 intr swi3: vm
12 100077 intr irq274: xhci0 mi_switch+0xe1
ithread_loop+0x393 fork_exit+0x83 fork_trampoline+0xe
12 100083 intr irq18: ehci0 ehci1 mi_switch+0xe1
ithread_loop+0x393 fork_exit+0x83 fork_trampoline+0xe
12 100094 intr irq305: ahci0 mi_switch+0xe1
ithread_loop+0x393 fork_exit+0x83 fork_trampoline+0xe
12 100095 intr swi0: uart uart mi_switch+0xe1
ithread_loop+0x393 fork_exit+0x83 fork_trampoline+0xe
12 100141 intr swi1: pf send mi_switch+0xe1
ithread_loop+0x393 fork_exit+0x83 fork_trampoline+0xe
13 100045 ng_queue ng_queue0 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 ngthread+0x9c fork_exit+0x83
fork_trampoline+0xe
13 100046 ng_queue ng_queue1 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 ngthread+0x9c fork_exit+0x83
fork_trampoline+0xe
13 100047 ng_queue ng_queue2 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 ngthread+0x9c fork_exit+0x83
fork_trampoline+0xe
13 100048 ng_queue ng_queue3 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 ngthread+0x9c fork_exit+0x83
fork_trampoline+0xe
13 100049 ng_queue ng_queue4 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 ngthread+0x9c fork_exit+0x83
fork_trampoline+0xe
13 100050 ng_queue ng_queue5 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 ngthread+0x9c fork_exit+0x83
fork_trampoline+0xe
13 100051 ng_queue ng_queue6 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 ngthread+0x9c fork_exit+0x83
fork_trampoline+0xe
13 100052 ng_queue ng_queue7 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 ngthread+0x9c fork_exit+0x83
fork_trampoline+0xe
14 100053 geom g_event mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 g_run_events+0x4f fork_exit+0x83
fork_trampoline+0xe
14 100054 geom g_up mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 g_io_schedule_up+0xcc g_up_procbody+0x6d
fork_exit+0x83 fork_trampoline+0xe
14 100055 geom g_down mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 g_io_schedule_down+0xcc g_down_procbody+0x6d
fork_exit+0x83 fork_trampoline+0xe
15 100073 crypto returns 7 - mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 crypto_ret_proc+0x1cd fork_exit+0x83
fork_trampoline+0xe
16 100074 sequencer 00 - mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 seq_eventthread+0xdc fork_exit+0x83
fork_trampoline+0xe
17 100075 cam doneq0 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 xpt_done_td+0x9b fork_exit+0x83
fork_trampoline+0xe
17 100076 cam doneq1 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 xpt_done_td+0x9b fork_exit+0x83
fork_trampoline+0xe
17 100147 cam scanner mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 xpt_scanner_thread+0x99 fork_exit+0x83
fork_trampoline+0xe
18 100078 usb usbus0 mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83
fork_trampoline+0xe
18 100079 usb usbus0 mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83
fork_trampoline+0xe
18 100080 usb usbus0 mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83
fork_trampoline+0xe
18 100081 usb usbus0 mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83
fork_trampoline+0xe
18 100082 usb usbus0 mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83
fork_trampoline+0xe
18 100084 usb usbus1 mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83
fork_trampoline+0xe
18 100085 usb usbus1 mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83
fork_trampoline+0xe
18 100086 usb usbus1 mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83
fork_trampoline+0xe
18 100087 usb usbus1 mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83
fork_trampoline+0xe
18 100088 usb usbus1 mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83
fork_trampoline+0xe
18 100089 usb usbus2 mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83
fork_trampoline+0xe
18 100090 usb usbus2 mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83
fork_trampoline+0xe
18 100091 usb usbus2 mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83
fork_trampoline+0xe
18 100092 usb usbus2 mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83
fork_trampoline+0xe
18 100093 usb usbus2 mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 usb_process+0x137 fork_exit+0x83
fork_trampoline+0xe
19 100105 soaiod1 - mi_switch+0xe1
sleepq_timedwait+0x2f _sleep+0x217 soaio_kproc_loop+0x1bf fork_exit+0x83
fork_trampoline+0xe
20 100106 soaiod2 - mi_switch+0xe1
sleepq_timedwait+0x2f _sleep+0x217 soaio_kproc_loop+0x1bf fork_exit+0x83
fork_trampoline+0xe
21 100107 soaiod3 - mi_switch+0xe1
sleepq_timedwait+0x2f _sleep+0x217 soaio_kproc_loop+0x1bf fork_exit+0x83
fork_trampoline+0xe
22 100108 soaiod4 - mi_switch+0xe1
sleepq_timedwait+0x2f _sleep+0x217 soaio_kproc_loop+0x1bf fork_exit+0x83
fork_trampoline+0xe
23 100117 zfskern arc_reclaim_thread mi_switch+0xe1
sleepq_timedwait+0x2f _cv_timedwait_sbt+0x17a arc_reclaim_thread+0x146
fork_exit+0x83 fork_trampoline+0xe
23 100118 zfskern arc_dnlc_evicts_thr mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 arc_dnlc_evicts_thread+0x16f fork_exit+0x83
fork_trampoline+0xe
23 100120 zfskern dbuf_evict_thread mi_switch+0xe1
sleepq_timedwait+0x2f _cv_timedwait_sbt+0x17a dbuf_evict_thread+0x1c8
fork_exit+0x83 fork_trampoline+0xe
23 100137 zfskern l2arc_feed_thread mi_switch+0xe1
sleepq_timedwait+0x2f _cv_timedwait_sbt+0x17a l2arc_feed_thread+0x219
fork_exit+0x83 fork_trampoline+0xe
23 100398 zfskern trim zroot mi_switch+0xe1
sleepq_timedwait+0x2f _cv_timedwait_sbt+0x17a trim_thread+0x11f
fork_exit+0x83 fork_trampoline+0xe
23 100434 zfskern txg_thread_enter mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 txg_quiesce+0x21b txg_quiesce_thread+0x11b
fork_exit+0x83 fork_trampoline+0xe
23 100435 zfskern txg_thread_enter mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 txg_sync_thread+0x13b fork_exit+0x83
fork_trampoline+0xe
23 100436 zfskern solthread 0xfffffff mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 zthr_procedure+0xcc fork_exit+0x83
fork_trampoline+0xe
23 100437 zfskern solthread 0xfffffff mi_switch+0xe1
sleepq_wait+0x2c _cv_wait+0x152 zthr_procedure+0xcc fork_exit+0x83
fork_trampoline+0xe
24 100139 sctp_iterator - mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 sctp_iterator_thread+0x59 fork_exit+0x83
fork_trampoline+0xe
25 100140 pf purge - mi_switch+0xe1
sleepq_timedwait+0x2f _sleep+0x217 pf_purge_thread+0xb9 fork_exit+0x83
fork_trampoline+0xe
26 100142 rand_harvestq - mi_switch+0xe1
sleepq_timedwait+0x2f _sleep+0x217 random_kthread+0x286 fork_exit+0x83
fork_trampoline+0xe
27 100148 enc_daemon0 - mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 enc_daemon+0x349 fork_exit+0x83
fork_trampoline+0xe
28 100149 pagedaemon dom0 mi_switch+0xe1
sleepq_timedwait+0x2f _sleep+0x217 vm_pageout_worker+0x28e vm_pageout+0x15a
fork_exit+0x83 fork_trampoline+0xe
28 100151 pagedaemon laundry: dom0 mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 vm_pageout_laundry_worker+0xfdd
fork_exit+0x83 fork_trampoline+0xe
28 100153 pagedaemon uma mi_switch+0xe1
sleepq_timedwait+0x2f _sleep+0x217 uma_reclaim_worker+0xca fork_exit+0x83
fork_trampoline+0xe
29 100150 vmdaemon - mi_switch+0xe1
sleepq_wait+0x2c _sleep+0x237 vm_daemon+0x94 fork_exit+0x83
fork_trampoline+0xe
30 100152 bufdaemon - mi_switch+0xe1
sleepq_timedwait+0x2f _sleep+0x217 buf_daemon+0xe8 fork_exit+0x83
fork_trampoline+0xe
30 100156 bufdaemon bufspacedaemon-0 mi_switch+0xe1
sleepq_timedwait+0x2f _sleep+0x217 bufspace_daemon+0xa4 fork_exit+0x83
fork_trampoline+0xe
30 100157 bufdaemon bufspacedaemon-1 mi_switch+0xe1
sleepq_timedwait+0x2f _sleep+0x217 bufspace_daemon+0xa4 fork_exit+0x83
fork_trampoline+0xe
30 100158 bufdaemon bufspacedaemon-2 mi_switch+0xe1
sleepq_timedwait+0x2f _sleep+0x217 bufspace_daemon+0xa4 fork_exit+0x83
fork_trampoline+0xe
30 100159 bufdaemon bufspacedaemon-3 mi_switch+0xe1
sleepq_timedwait+0x2f _sleep+0x217 bufspace_daemon+0xa4 fork_exit+0x83
fork_trampoline+0xe
30 100160 bufdaemon bufspacedaemon-4 mi_switch+0xe1
sleepq_timedwait+0x2f _sleep+0x217 bufspace_daemon+0xa4 fork_exit+0x83
fork_trampoline+0xe
30 100161 bufdaemon bufspacedaemon-5 mi_switch+0xe1
sleepq_timedwait+0x2f _sleep+0x217 bufspace_daemon+0xa4 fork_exit+0x83
fork_trampoline+0xe
30 100162 bufdaemon bufspacedaemon-6 mi_switch+0xe1
sleepq_timedwait+0x2f _sleep+0x217 bufspace_daemon+0xa4 fork_exit+0x83
fork_trampoline+0xe
31 100154 vnlru - mi_switch+0xe1
sleepq_timedwait+0x2f _sleep+0x217 vnlru_proc+0x95 fork_exit+0x83
fork_trampoline+0xe
32 100155 syncer - mi_switch+0xe1
sleepq_timedwait+0x2f _cv_timedwait_sbt+0x17a sched_sync+0xc4
fork_exit+0x83 fork_trampoline+0xe
2925 100473 devd - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_select+0x87f
sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101
3014 100487 syslogd - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369
fast_syscall_common+0x101
3121 100443 automountd - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
autofs_ioctl+0xbd devfs_ioctl+0xad VOP_IOCTL_APV+0x7e vn_ioctl+0x1a4
devfs_ioctl_f+0x1f kern_ioctl+0x26d sys_ioctl+0x15e amd64_syscall+0x369
fast_syscall_common+0x101
3127 100461 autounmountd - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
kqueue_kevent+0x29e kern_kevent+0xb5 kern_kevent_generic+0x71
sys_kevent+0x61 amd64_syscall+0x369 fast_syscall_common+0x101
3246 100488 sshd - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369
fast_syscall_common+0x101
7053 100664 smbd - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_poll+0x43d sys_poll+0x50
amd64_syscall+0x369 fast_syscall_common+0x101
7062 100616 smbd - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_poll+0x43d sys_poll+0x50
amd64_syscall+0x369 fast_syscall_common+0x101
7063 100670 smbd - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_poll+0x43d sys_poll+0x50
amd64_syscall+0x369 fast_syscall_common+0x101
7064 100455 smbd - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_poll+0x43d sys_poll+0x50
amd64_syscall+0x369 fast_syscall_common+0x101
7065 100519 snmptrapd - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369
fast_syscall_common+0x101
10568 100483 nginx - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
kern_sigsuspend+0x174 sys_sigsuspend+0x31 amd64_syscall+0x369
fast_syscall_common+0x101
10569 100803 nginx - mi_switch+0xe1
sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e
_vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b unp_connectat+0x174
soconnect+0xad kern_connectat+0x106 sys_connect+0x77 amd64_syscall+0x369
fast_syscall_common+0x101
10570 100448 nginx - mi_switch+0xe1
sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e
_vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b unp_connectat+0x174
soconnect+0xad kern_connectat+0x106 sys_connect+0x77 amd64_syscall+0x369
fast_syscall_common+0x101
10571 100527 nginx - mi_switch+0xe1
sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e
_vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b unp_connectat+0x174
soconnect+0xad kern_connectat+0x106 sys_connect+0x77 amd64_syscall+0x369
fast_syscall_common+0x101
10572 100552 nginx - mi_switch+0xe1
sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e
_vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b unp_connectat+0x174
soconnect+0xad kern_connectat+0x106 sys_connect+0x77 amd64_syscall+0x369
fast_syscall_common+0x101
10573 100617 nginx - mi_switch+0xe1
sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e
_vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b unp_connectat+0x174
soconnect+0xad kern_connectat+0x106 sys_connect+0x77 amd64_syscall+0x369
fast_syscall_common+0x101
10574 101643 nginx - mi_switch+0xe1
sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e
_vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b unp_connectat+0x174
soconnect+0xad kern_connectat+0x106 sys_connect+0x77 amd64_syscall+0x369
fast_syscall_common+0x101
10575 100610 nginx - mi_switch+0xe1
sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e
_vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b unp_connectat+0x174
soconnect+0xad kern_connectat+0x106 sys_connect+0x77 amd64_syscall+0x369
fast_syscall_common+0x101
10576 100653 nginx - mi_switch+0xe1
sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e
_vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b unp_connectat+0x174
soconnect+0xad kern_connectat+0x106 sys_connect+0x77 amd64_syscall+0x369
fast_syscall_common+0x101
20584 100460 openvpn - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_poll+0x43d sys_poll+0x50
amd64_syscall+0x369 fast_syscall_common+0x101
21660 100615 postgres - mi_switch+0xe1
sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e
_vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b kern_utimesat+0x110
amd64_syscall+0x369 fast_syscall_common+0x101
21662 100612 postgres - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_poll+0x43d sys_poll+0x50
amd64_syscall+0x369 fast_syscall_common+0x101
21663 100626 postgres - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_poll+0x43d sys_poll+0x50
amd64_syscall+0x369 fast_syscall_common+0x101
21664 100450 postgres - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_poll+0x43d sys_poll+0x50
amd64_syscall+0x369 fast_syscall_common+0x101
21665 100638 postgres - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_poll+0x43d sys_poll+0x50
amd64_syscall+0x369 fast_syscall_common+0x101
21666 100635 postgres - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
21667 100700 postgres - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_poll+0x43d sys_poll+0x50
amd64_syscall+0x369 fast_syscall_common+0x101
22183 100518 getty - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
tty_wait+0x1c ttydisc_read+0x1f2 ttydev_read+0x64 devfs_read_f+0xdc
dofileread+0x95 sys_read+0xc3 amd64_syscall+0x369 fast_syscall_common+0x101
22184 100509 getty - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
tty_wait+0x1c ttydisc_read+0x1f2 ttydev_read+0x64 devfs_read_f+0xdc
dofileread+0x95 sys_read+0xc3 amd64_syscall+0x369 fast_syscall_common+0x101
22185 101647 getty - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
tty_wait+0x1c ttydisc_read+0x1f2 ttydev_read+0x64 devfs_read_f+0xdc
dofileread+0x95 sys_read+0xc3 amd64_syscall+0x369 fast_syscall_common+0x101
22435 100440 getty - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
tty_wait+0x1c ttydisc_read+0x1f2 ttydev_read+0x64 devfs_read_f+0xdc
dofileread+0x95 sys_read+0xc3 amd64_syscall+0x369 fast_syscall_common+0x101
24825 101365 postgres - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
24851 100599 perl - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
kern_clock_nanosleep+0x1a7 sys_nanosleep+0x5f amd64_syscall+0x369
fast_syscall_common+0x101
24855 101640 ntpd - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369
fast_syscall_common+0x101
24865 100459 radiusd - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
kqueue_kevent+0x29e kern_kevent+0xb5 kern_kevent_generic+0x71
sys_kevent+0x61 amd64_syscall+0x369 fast_syscall_common+0x101
24865 101450 radiusd - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_sem2_wait+0x68a __umtx_op_sem2_wait+0x4b
amd64_syscall+0x369 fast_syscall_common+0x101
24869 100545 ruby25 - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_select+0x87f
sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101
24869 101382 ruby25 ruby-timer-thr mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
24870 100813 ruby25 - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24870 100960 ruby25 ruby-timer-thr mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
24870 100961 ruby25 cluster.rb:253 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369
fast_syscall_common+0x101
24870 100962 ruby25 cluster.rb:287 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24870 100968 ruby25 puma 001 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24870 100973 ruby25 reactor.rb:249 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_select+0x87f
sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101
24870 100979 ruby25 thread_pool.rb* mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24870 100981 ruby25 thread_pool.rb* mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24870 100984 ruby25 server.rb:358 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369
fast_syscall_common+0x101
24871 100541 ruby25 - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24871 100665 ruby25 cluster.rb:253 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369
fast_syscall_common+0x101
24871 100986 ruby25 ruby-timer-thr mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
24871 100987 ruby25 cluster.rb:287 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24871 100989 ruby25 puma 001 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24871 101001 ruby25 reactor.rb:249 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_select+0x87f
sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101
24871 101037 ruby25 thread_pool.rb* mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24871 101047 ruby25 thread_pool.rb* mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24871 101051 ruby25 server.rb:358 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369
fast_syscall_common+0x101
24873 100637 ruby25 - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24873 100828 ruby25 cluster.rb:253 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369
fast_syscall_common+0x101
24873 101055 ruby25 ruby-timer-thr mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
24873 101064 ruby25 cluster.rb:287 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24873 101077 ruby25 puma 001 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24873 101098 ruby25 reactor.rb:249 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_select+0x87f
sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101
24873 101102 ruby25 thread_pool.rb* mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24873 101118 ruby25 thread_pool.rb* mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24873 101122 ruby25 server.rb:358 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369
fast_syscall_common+0x101
24874 100516 ruby25 - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24874 101036 ruby25 cluster.rb:253 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369
fast_syscall_common+0x101
24874 101129 ruby25 ruby-timer-thr mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
24874 101143 ruby25 cluster.rb:287 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24874 101144 ruby25 puma 001 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24874 101147 ruby25 reactor.rb:249 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_select+0x87f
sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101
24874 101194 ruby25 thread_pool.rb* mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24874 101259 ruby25 thread_pool.rb* mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24874 101267 ruby25 server.rb:358 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369
fast_syscall_common+0x101
24875 100539 ruby25 - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24875 101071 ruby25 cluster.rb:253 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369
fast_syscall_common+0x101
24875 101284 ruby25 ruby-timer-thr mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
24875 101289 ruby25 cluster.rb:287 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24875 101298 ruby25 puma 001 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24875 101334 ruby25 reactor.rb:249 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_select+0x87f
sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101
24875 101347 ruby25 thread_pool.rb* mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24875 101350 ruby25 thread_pool.rb* mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24875 101369 ruby25 server.rb:358 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369
fast_syscall_common+0x101
24876 100642 ruby25 - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24876 101384 ruby25 ruby-timer-thr mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
24876 101411 ruby25 cluster.rb:253 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369
fast_syscall_common+0x101
24876 101423 ruby25 cluster.rb:287 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24876 101429 ruby25 puma 001 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24876 101431 ruby25 reactor.rb:249 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_select+0x87f
sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101
24876 101441 ruby25 thread_pool.rb* mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24876 101444 ruby25 thread_pool.rb* mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
24876 101447 ruby25 server.rb:358 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_select+0x87f sys_select+0x56 amd64_syscall+0x369
fast_syscall_common+0x101
24885 100816 snort - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
bpfread+0x318 devfs_read_f+0xdc dofileread+0x95 sys_read+0xc3
amd64_syscall+0x369 fast_syscall_common+0x101
24885 101457 snort - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
kern_clock_nanosleep+0x1a7 sys_nanosleep+0x5f amd64_syscall+0x369
fast_syscall_common+0x101
25043 100810 ruby25 - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
25043 101658 ruby25 ruby-timer-thr mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
25044 100682 postgres - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
25422 100603 postgres - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
25433 100691 postgres - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
25442 100504 postgres - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
26015 100805 postgres - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
27660 100636 postgres - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
29381 100553 postgres - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
34928 100480 dhcpd - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_select+0x87f
sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101
36704 101146 perl - mi_switch+0xe1
sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_xlock_hard+0x19c VOP_LOCK1_APV+0x7e
_vn_lock+0x40 zfs_znode_alloc+0x434 zfs_mknode+0xa9d
zfs_freebsd_create+0x512 VOP_CREATE_APV+0x78 vn_open_cred+0x2c9
kern_openat+0x20c amd64_syscall+0x369 fast_syscall_common+0x101
36751 101709 cron - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a pipe_read+0x446
dofileread+0x95 sys_read+0xc3 amd64_syscall+0x369 fast_syscall_common+0x101
39368 101861 sendmail - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
kern_wait6+0x66d sys_wait4+0x78 amd64_syscall+0x369
fast_syscall_common+0x101
39370 101856 mail.local - mi_switch+0xe1
sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e
_vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b kern_statat+0x77
sys_fstatat+0x2f amd64_syscall+0x369 fast_syscall_common+0x101
40693 101766 perl - mi_switch+0xe1
sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e
_vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b kern_statat+0x77
sys_fstatat+0x2f amd64_syscall+0x369 fast_syscall_common+0x101
40696 101911 sshd - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
40698 101972 sshd - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_select+0x87f
sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101
40699 101310 bash - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
kern_wait6+0x66d sys_wait4+0x78 amd64_syscall+0x369
fast_syscall_common+0x101
40711 101974 cron - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a pipe_read+0x446
dofileread+0x95 sys_read+0xc3 amd64_syscall+0x369 fast_syscall_common+0x101
40712 101395 sh - mi_switch+0xe1
sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e
_vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b kern_statat+0x77
sys_fstatat+0x2f amd64_syscall+0x369 fast_syscall_common+0x101
40715 101969 su - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
kern_wait6+0x66d sys_wait4+0x78 amd64_syscall+0x369
fast_syscall_common+0x101
40716 101976 csh - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
kern_sigsuspend+0x174 sys_sigsuspend+0x31 amd64_syscall+0x369
fast_syscall_common+0x101
40722 101977 procstat -
sysctl_kern_proc_kstack+0x282 sysctl_root_handler_locked+0x8b
sysctl_root+0x257 userland_sysctl+0x17a sys___sysctl+0x5f
amd64_syscall+0x369 fast_syscall_common+0x101
81907 100546 racoon - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_select+0x87f
sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101
83204 100656 perl - mi_switch+0xe1
sleepq_wait+0x2c sleeplk+0x1c5 lockmgr_slock_hard+0x2c5 VOP_LOCK1_APV+0x7e
_vn_lock+0x40 zfs_root+0x6d lookup+0x933 namei+0x44b kern_statat+0x77
sys_fstatat+0x2f amd64_syscall+0x369 fast_syscall_common+0x101
83208 100501 postgres - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
83209 100525 postgres - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
83210 100628 postgres - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
83424 100485 ladvd - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
kqueue_kevent+0x29e kern_kevent+0xb5 kern_kevent_generic+0x71
sys_kevent+0x61 amd64_syscall+0x369 fast_syscall_common+0x101
83425 100554 ladvd - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
kqueue_kevent+0x29e kern_kevent+0xb5 kern_kevent_generic+0x71
sys_kevent+0x61 amd64_syscall+0x369 fast_syscall_common+0x101
83428 101357 perl - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
kern_clock_nanosleep+0x1a7 sys_nanosleep+0x5f amd64_syscall+0x369
fast_syscall_common+0x101
83444 100542 snmpd - <running>
83545 100486 sendmail - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14
_cv_timedwait_sig_sbt+0x17a seltdwait+0x7b kern_select+0x87f
sys_select+0x56 amd64_syscall+0x369 fast_syscall_common+0x101
83548 100814 sendmail - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
kern_sigsuspend+0x174 sys_sigsuspend+0x31 amd64_syscall+0x369
fast_syscall_common+0x101
83554 100427 named isc-worker0000 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83554 100431 named isc-worker0001 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83554 100432 named isc-worker0002 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83554 100556 named - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
kern_sigtimedwait+0x3d4 sys_sigwait+0x5c amd64_syscall+0x369
fast_syscall_common+0x101
83554 100774 named isc-worker0003 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83554 100819 named isc-worker0004 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83554 100820 named isc-worker0005 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83554 100849 named isc-worker0006 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83554 100990 named isc-worker0007 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83554 101481 named isc-timer mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83554 101534 named isc-socket mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
kqueue_kevent+0x29e kern_kevent+0xb5 kern_kevent_generic+0x71
sys_kevent+0x61 amd64_syscall+0x369 fast_syscall_common+0x101
83556 100608 named - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
kern_sigtimedwait+0x3d4 sys_sigwait+0x5c amd64_syscall+0x369
fast_syscall_common+0x101
83556 101540 named isc-worker0000 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83556 101541 named isc-worker0001 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83556 101543 named isc-worker0002 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83556 101545 named isc-worker0003 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83556 101546 named isc-worker0004 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83556 101549 named isc-worker0005 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83556 101560 named isc-worker0006 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83556 101583 named isc-worker0007 mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83556 101596 named isc-timer mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83556 101600 named isc-socket mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _sleep+0x22a
kqueue_kevent+0x29e kern_kevent+0xb5 kern_kevent_generic+0x71
sys_kevent+0x61 amd64_syscall+0x369 fast_syscall_common+0x101
83559 100514 ruby25 - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83559 101604 ruby25 ruby-timer-thr mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
83562 100463 ruby25 - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83562 101486 ruby25 ruby-timer-thr mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
83565 100821 ruby25 - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
umtxq_sleep+0x143 do_wait+0x427 __umtx_op_wait_uint_private+0x53
amd64_syscall+0x369 fast_syscall_common+0x101
83565 101606 ruby25 ruby-timer-thr mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
84111 100668 postgres - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
84112 100613 postgres - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
84113 101274 postgres - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154
seltdwait+0xc3 kern_poll+0x43d sys_poll+0x50 amd64_syscall+0x369
fast_syscall_common+0x101
84416 100550 cron - mi_switch+0xe1
sleepq_catch_signals+0x405 sleepq_timedwait_sig+0x14 _sleep+0x205
kern_clock_nanosleep+0x1a7 sys_nanosleep+0x5f amd64_syscall+0x369
fast_syscall_common+0x101