Herve Boulouis
2011-Jul-25 08:25 UTC
Sleeping thread owns a nonsleepable lock panic (& lor)
Hi list, We have 2 freebsd 8.2-STABLE (cvsuped june 22) that keeps crashing in a bad way : The are doing heavy apache / php4 web serving from a nfs mount and panic at least once a day with the following message (no crash dump produced, hand copied from the console) : Sleeping on "vmopar" with the following non-sleepable locks held: exclusive sleep mutex NFSnode lock (NFSnode lock) r = 0 (0xffffff0201798000) locked @ nfsclient/nfs_subs.c:538 lock order reversal: 1st 0xffffffff018ff6da80 turnstile lock (turnstile lock) @ kern/subr_turnstile.c:190 2nd 0xffffffffff80b52b10 scrlock (scrlock) @ dev/syscons.c:2570 lock order reversal: 1st 0xffffffff018ff6da80 turnstile lock (turnstile lock) @ kern/subr_turnstile.c:190 2nd 0xffffffffff80b78ef8 sleepq chain (sleepq chain) @ kern/subr_turnstile.c:203 lock order reversal: 1st 0xffffffffff80b78ef8 sleepq chain (sleepq chain) @ kern/subr_turnstile.c:203 2nd 0xffffffffff80b52b10 scrlock (scrlock) @ dev/syscons.c:2570 Sleeping thread (tid 100998, pid 20700) owns a non-sleepable lock panic: sleeping thread cpuid = 1 panic: bufwrite: buffer is not busy??? cpuid = 1 The 2 servers share the same load and panic consistently. I enabled WITNESS on the 2 in the hope it would allow the boxes to auto reboot after panic and get extra debug info. I got debug info but the servers still hangs after the double panic :( I also noticed that immediately after rebooting following this panic, I got the following LORs (approximatively at the time rc.d is launching ports like apache & co) lock order reversal: 1st 0xffffff81ee00e388 bufwait (bufwait) @ kern/vfs_bio.c:2636 2nd 0xffffff0006e56c00 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:285 lock order reversal: 1st 0xffffff0009c709e0 so_snd_sx (so_snd_sx) @ kern/uipc_sockbuf.c:145 2nd 0xffffff0124282620 ufs (ufs) @ kern/uipc_syscalls.c:2086 lock order reversal: 1st 0xffffff0009c709e0 so_snd_sx (so_snd_sx) @ kern/uipc_sockbuf.c:145 2nd 0xffffff01243569d0 nfs (nfs) @ kern/uipc_syscalls.c:2086 The server continued to work despite the lors so I don't know if this is related to the panics or not. What can I do from there to debug this further ? Regards, -- Herve Boulouis
Kostik Belousov
2011-Jul-25 08:59 UTC
Sleeping thread owns a nonsleepable lock panic (& lor)
On Mon, Jul 25, 2011 at 12:21:07PM +0200, Herve Boulouis wrote:> Hi list, > > We have 2 freebsd 8.2-STABLE (cvsuped june 22) that keeps crashing in a bad way : > > The are doing heavy apache / php4 web serving from a nfs mount and panic at least once a day > with the following message (no crash dump produced, hand copied from the console) : > > Sleeping on "vmopar" with the following non-sleepable locks held: > exclusive sleep mutex NFSnode lock (NFSnode lock) r = 0 (0xffffff0201798000) locked @ nfsclient/nfs_subs.c:538 > lock order reversal: > 1st 0xffffffff018ff6da80 turnstile lock (turnstile lock) @ kern/subr_turnstile.c:190 > 2nd 0xffffffffff80b52b10 scrlock (scrlock) @ dev/syscons.c:2570 > lock order reversal: > 1st 0xffffffff018ff6da80 turnstile lock (turnstile lock) @ kern/subr_turnstile.c:190 > 2nd 0xffffffffff80b78ef8 sleepq chain (sleepq chain) @ kern/subr_turnstile.c:203 > lock order reversal: > 1st 0xffffffffff80b78ef8 sleepq chain (sleepq chain) @ kern/subr_turnstile.c:203 > 2nd 0xffffffffff80b52b10 scrlock (scrlock) @ dev/syscons.c:2570 > Sleeping thread (tid 100998, pid 20700) owns a non-sleepable lock > panic: sleeping thread > cpuid = 1 > panic: bufwrite: buffer is not busy??? > cpuid = 1 > > The 2 servers share the same load and panic consistently. I enabled WITNESS on the 2 in the hope > it would allow the boxes to auto reboot after panic and get extra debug info. I got debug info > but the servers still hangs after the double panic :(Try this. Calling vnode_pager_setsize() while holding a mutex is prohibited. On the other hand, I remember that my attempt to add a strict assert that a vnode is exclusively locked in vnode_pager_setsize() had to be reversed because nfs_loadattrcache() sometimes called without vnode lock held. commit 2aa7d15c38b0c01e3f724f04d7ed02ce11c82cc0 Author: Konstantin Belousov <kostikbel@gmail.com> Date: Mon Jul 25 11:56:04 2011 +0300 Postpone the vnode_pager_setsize() call until the nfs node mutex is dropped. diff --git a/sys/nfsclient/nfs_subs.c b/sys/nfsclient/nfs_subs.c index 19fde06..351885a 100644 --- a/sys/nfsclient/nfs_subs.c +++ b/sys/nfsclient/nfs_subs.c @@ -478,7 +478,9 @@ nfs_loadattrcache(struct vnode **vpp, struct mbuf **mdp, caddr_t *dposp, struct timespec mtime, mtime_save; int v3 = NFS_ISV3(vp); int error = 0; + int do_setsize; + do_setsize = 0; md = *mdp; t1 = (mtod(md, caddr_t) + md->m_len) - *dposp; cp2 = nfsm_disct(mdp, dposp, NFSX_FATTR(v3), t1, M_WAIT); @@ -606,7 +608,7 @@ nfs_loadattrcache(struct vnode **vpp, struct mbuf **mdp, caddr_t *dposp, np->n_size = vap->va_size; np->n_flag |= NSIZECHANGED; } - vnode_pager_setsize(vp, np->n_size); + do_setsize = 1; } else { np->n_size = vap->va_size; } @@ -643,6 +645,8 @@ nfs_loadattrcache(struct vnode **vpp, struct mbuf **mdp, caddr_t *dposp, KDTRACE_NFS_ATTRCACHE_LOAD_DONE(vp, &np->n_vattr, 0); #endif mtx_unlock(&np->n_mtx); + if (do_setsize) + vnode_pager_setsize(vp, np->n_size); out: #ifdef KDTRACE_HOOKS if (error) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 196 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20110725/712cc504/attachment.pgp
Herve Boulouis
2011-Jul-26 07:53 UTC
Sleeping thread owns a nonsleepable lock panic (& lor)
Le 25/07/2011 11:59, Kostik Belousov a écrit:
Ok the patched server crashed this morning strangely : all httpd processes were
stuck in nfs or vmopar
and were unkillable. Below is the full ps.
When tried to reboot it got stuck after "All buffers synced" with LOR
:
lock order reversal:
1st 0xffffff00094a4270 ufs (ufs) @ kern/vfs_mount.c:1204
2nd 0xffffff000960f7f8 syncer (syncer) @ kern/vfs_subr.c:2232
lock order reversal:
1st 0xffffff000960f9d0 nfs (nfs) @ kern/vfs_subr.c:2120
2nd 0xffffffff80b567e0 allproc (allproc) @ kern/kern_descrip.c:2518
In my original post I forgot to add that this box (Dell M610) has 24 cores
inside (2 physicals CPUs with 6 cores each
and HT enabled).
Full ps before reboot :
UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND
0 0 0 0 8 0 0 96 - DLs ?? 0:45.66 [kernel]
0 1 0 0 60 0 3204 120 wait ILs ?? 0:00.78 /sbin/init
--
0 2 0 0 -8 0 0 16 - DL ?? 0:03.70 [g_event]
0 3 0 0 -8 0 0 16 - DL ?? 3:18.53 [g_up]
0 4 0 0 -8 0 0 16 - DL ?? 1:01.57 [g_down]
0 5 0 0 -16 0 0 16 idle DL ?? 0:00.00
[mpt_recovery0]
0 6 0 0 -16 0 0 16 idle DL ?? 0:00.00 [mpt_raid0]
0 7 0 0 -16 0 0 16 waitin DL ?? 0:00.00
[sctp_iterator]
0 8 0 0 -16 0 0 16 pftm DL ?? 0:00.38 [pfpurge]
0 9 0 0 -16 0 0 16 ccb_sc DL ?? 0:00.00 [xpt_thrd]
0 10 0 0 -16 0 0 16 audit_ DL ?? 0:00.00 [audit]
0 11 0 0 171 0 0 384 - RL ?? 21204:47.83 [idle]
0 12 0 0 -60 0 0 640 - WL ?? 21:01.69 [intr]
0 13 0 0 44 0 0 16 - DL ?? 0:42.12 [yarrow]
0 14 0 0 -64 0 0 448 - DL ?? 0:01.85 [usb]
0 15 0 0 -8 0 0 16 mdwait DL ?? 0:00.00 [md0]
0 16 0 0 52 0 0 16 psleep DL ?? 25:15.74 [pagedaemon]
0 17 0 0 53 0 0 16 psleep DL ?? 1:54.70 [vmdaemon]
0 18 0 0 76 0 0 16 pgzero DL ?? 0:00.00 [pagezero]
0 19 0 0 -16 0 0 16 psleep DL ?? 0:00.35 [bufdaemon]
0 20 0 0 46 0 0 16 syncer DL ?? 9:22.33 [syncer]
0 21 0 0 46 0 0 16 vlruwt DL ?? 1:38.87 [vnlru]
0 22 0 0 44 0 0 16 sdflus DL ?? 1:31.12
[softdepflush]
0 23 0 0 -16 0 0 16 flowcl DL ?? 0:01.61
[flowcleaner]
80 169 1 0 46 0 132088 10752 nfs D ?? 0:00.05
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 465 1 0 44 -20 175352 10652 nfs D< ?? 0:13.53
/usr/local/sbin/httpd -DNOHTTPACCEPT
0 715 1 0 44 0 3200 32 select Is ?? 0:00.00 /sbin/devd
0 888 1 0 44 0 5864 900 select Ss ?? 3:47.20
/usr/sbin/syslogd -ss
0 914 1 0 44 0 6924 936 select Ss ?? 0:11.19
/usr/sbin/rpcbind
0 1020 1 0 44 0 267976 800 select Ss ?? 0:08.92
/usr/sbin/rpc.statd
0 1028 1 0 76 0 6916 760 rpcsvc Ss ?? 0:00.09
/usr/sbin/rpc.lockd
0 1043 1 0 47 0 15200 2660 nanslp I ?? 6:01.08
/usr/local/bin/monit -c /usr/local/etc/monitrc
122 1052 1 0 76 5 11988 0 wait IWN ?? 0:00.00
zabbix_agentd: main process (zabbix_agentd)
122 1061 1052 0 49 5 11988 1244 nanslp SN ?? 2:49.93
zabbix_agentd: collector [sleeping for 1 seconds] (zabbix_agentd
122 1062 1052 0 49 5 11988 800 select SN ?? 1:44.39
zabbix_agentd: listener [waiting for connection] (zabbix_agentd)
122 1063 1052 0 49 5 11988 800 accept SN ?? 1:43.88
zabbix_agentd: listener [waiting for connection] (zabbix_agentd)
122 1064 1052 0 49 5 11988 808 accept SN ?? 1:35.01
zabbix_agentd: listener [waiting for connection] (zabbix_agentd)
122 1065 1052 0 49 5 11988 1192 nanslp SN ?? 3:00.97
zabbix_agentd: poller [sleeping for 1 second(s)] (zabbix_agentd)
0 1077 1 0 45 0 28064 2316 select S ?? 15:20.13
/usr/local/sbin/snmpd -p /var/run/snmpd.pid
0 1269 1 0 44 0 5992 536 kqread Ss ?? 0:06.57
/usr/local/libexec/postfix/master
125 1282 1269 0 44 0 5996 792 kqread S ?? 0:02.04 qmgr -l -t
fifo -u
0 1288 1 0 44 0 43724 3212 select S ?? 0:40.03
/usr/local/bin/python2.6 /usr/local/bin/fail2ban-server -b -s /v
0 1339 1 0 44 0 36936 1824 select Ss ?? 0:00.08 sshd:
root@pts/0 (sshd)
0 1340 1 0 76 0 6920 480 nanslp Is ?? 0:05.94
/usr/sbin/cron -s
80 1721 1 0 44 -20 212216 10664 nfs D< ?? 0:06.65
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 1746 1 0 60 -20 208120 10640 nfs D< ?? 0:29.05
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 1747 1 0 44 -20 156920 10692 nfs D< ?? 0:04.32
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 2324 1 0 76 -20 212216 10704 nfs D< ?? 0:01.95
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 3400 1 0 44 -20 163064 10660 nfs D< ?? 0:05.01
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 3401 1 0 44 -20 181496 10688 nfs D< ?? 0:00.80
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 3942 1 0 47 -20 181496 10620 nfs D< ?? 0:29.44
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 3978 1 0 44 -20 179448 10672 nfs D< ?? 0:03.49
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 3979 1 0 44 -20 181496 10648 nfs D< ?? 0:09.13
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 4494 1 0 44 -20 175352 10640 nfs D< ?? 0:25.07
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 4495 1 0 68 -20 183544 10624 nfs D< ?? 0:33.51
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 5110 1 0 76 -20 208120 10688 nfs D< ?? 0:01.41
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 5675 1 0 44 -20 197880 10592 nfs D< ?? 0:42.38
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 6943 1 0 68 -20 208120 10656 nfs D< ?? 0:15.50
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 6944 1 0 44 -20 183544 10644 nfs D< ?? 0:18.46
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 7350 1 0 44 0 154744 15180 nfs D ?? 0:01.61
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 7436 1 0 44 0 177400 15952 nfs D ?? 0:19.34
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 7496 1 0 44 -20 175352 10648 nfs D< ?? 0:12.21
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 7647 1 0 44 -20 152824 10660 nfs D< ?? 0:02.36
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 8127 1 0 47 -20 173304 10616 nfs D< ?? 0:46.18
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 8713 1 0 76 -20 181496 10656 nfs D< ?? 0:10.67
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 8961 1 0 44 -20 154872 10692 nfs D< ?? 0:02.43
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 9219 1 0 54 -20 173304 10612 nfs D< ?? 0:18.19
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 9236 1 0 44 0 171256 15400 nfs D ?? 0:07.30
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 9238 1 0 44 -20 199928 79656 nfs D< ?? 0:01.20
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 9267 1 0 44 -20 199928 79988 nfs D< ?? 0:34.80
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 9273 1 0 76 -20 199928 81060 nfs D< ?? 0:10.44
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 9693 1 0 76 -20 181496 10656 nfs D< ?? 0:17.64
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 9831 1 0 44 -20 173304 10672 nfs D< ?? 0:07.07
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 9871 1 0 44 -20 169208 10684 nfs D< ?? 0:05.17
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 10223 1 0 44 -20 181496 10636 nfs D< ?? 0:25.70
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 10279 1 0 63 0 175352 15724 nfs D ?? 0:16.30
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 10361 1 0 44 -20 191736 10672 nfs D< ?? 0:08.45
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 10655 1 0 59 -20 181496 61448 nfs D< ?? 0:01.71
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 10656 1 0 71 -20 199928 79648 nfs D< ?? 0:05.74
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 10697 1 0 59 -20 199928 79980 nfs D< ?? 0:12.17
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 10994 1 0 62 0 171256 15868 nfs D ?? 0:11.99
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 11044 1 0 72 -20 199928 79952 nfs D< ?? 0:27.56
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 11078 1 0 59 -20 181496 61992 nfs D< ?? 0:57.98
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 11093 1 0 59 -20 199928 79968 nfs D< ?? 0:12.68
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 11245 1 0 44 -20 161016 10696 nfs D< ?? 0:01.55
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 11274 1 0 64 -20 181496 10684 nfs D< ?? 0:09.52
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 11300 1 0 76 -20 208120 10708 nfs D< ?? 0:01.58
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 11301 1 0 76 -20 201976 10688 nfs D< ?? 0:08.91
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 11303 1 0 56 -20 181496 11256 nfs D< ?? 0:48.15
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 11307 1 0 70 -20 201976 11072 nfs D< ?? 0:23.23
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 11661 1 0 76 -20 201976 82356 nfs D< ?? 0:38.99
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 11978 1 0 73 -20 201976 11028 nfs D< ?? 0:23.06
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 12423 1 0 65 -20 210168 12328 nfs D< ?? 0:20.99
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 12441 1 0 56 -20 201976 81556 nfs D< ?? 1:23.33
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 12757 1 0 65 0 158968 14772 nfs D ?? 0:00.98
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 13281 1 0 67 -20 201848 11304 nfs D< ?? 0:02.11
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 13855 1 0 53 -20 173304 10680 nfs D< ?? 0:09.40
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 13874 1 0 44 -20 175352 10640 nfs D< ?? 0:15.75
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 14008 1 0 58 -20 185720 14796 nfs D< ?? 0:48.99
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 14041 1 0 54 -20 179448 10664 nfs D< ?? 0:16.97
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 14043 1 0 44 -20 173304 10648 nfs D< ?? 0:15.35
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 14082 1 0 44 -20 171256 10672 nfs D< ?? 0:07.50
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 14264 1 0 60 0 173304 15328 nfs D ?? 0:04.78
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 14327 1 0 44 -20 169208 10656 nfs D< ?? 0:14.71
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 14510 1 0 58 0 173304 15800 nfs D ?? 0:17.82
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 14701 1 0 64 -20 199928 79836 nfs D< ?? 0:04.98
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 14891 1 0 44 -20 200056 80060 nfs D< ?? 0:26.68
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 14930 1 0 46 0 132088 10760 nfs D ?? 0:00.02
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 15102 1 0 44 -20 183544 10648 nfs D< ?? 0:36.67
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 15118 1 0 46 -20 201976 80812 nfs D< ?? 0:42.98
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 15119 1 0 44 -20 165112 10664 nfs D< ?? 0:05.73
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 15275 1 0 76 -20 201976 81900 nfs D< ?? 0:06.18
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 15583 1 0 44 -20 136184 10688 nfs D< ?? 0:00.12
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 15617 1 0 52 0 165112 15532 nfs D ?? 0:03.31
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 15956 1 0 56 -20 181496 10636 nfs D< ?? 0:28.79
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 15957 1 0 44 -20 136184 10700 nfs D< ?? 0:00.06
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 16395 1 0 59 0 175352 15604 nfs D ?? 0:06.21
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 16635 1 0 44 -20 156920 10680 nfs D< ?? 0:00.76
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 16680 1 0 58 -20 212216 10680 nfs D< ?? 0:07.00
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 16937 1 0 52 -20 199928 80224 nfs D< ?? 0:18.39
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 17044 1 0 44 -20 181496 10748 nfs D< ?? 0:39.98
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 17173 1 0 44 -20 167160 11100 nfs D< ?? 0:11.24
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 17250 1 0 45 0 163064 15148 nfs D ?? 0:03.63
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 17251 1 0 44 0 169208 15564 nfs D ?? 0:05.59
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 17466 1 0 57 -20 199928 79908 nfs D< ?? 0:04.42
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 17483 1 0 62 -20 181496 61652 nfs D< ?? 0:12.78
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 17489 1 0 58 0 187640 15816 nfs D ?? 0:20.67
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 17506 1 0 44 -20 204024 78260 nfs D< ?? 0:03.26
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 17656 1 0 46 -20 201976 10804 nfs D< ?? 0:16.42
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 17657 1 0 72 -20 202104 10704 nfs D< ?? 0:19.79
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 18163 1 0 45 0 132088 10756 nfs D ?? 0:00.01
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 18900 1 0 65 -20 201976 81880 nfs D< ?? 0:12.15
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 18956 1 0 44 -20 181496 61316 nfs D< ?? 0:15.83
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 18986 1 0 65 -20 212216 31548 nfs D< ?? 0:07.71
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 19127 1 0 76 -20 201976 10696 nfs D< ?? 0:01.33
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 19307 1 0 44 0 185592 15512 nfs D ?? 0:28.67
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 19489 1 0 61 -20 181496 33452 nfs D< ?? 0:05.68
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 19517 1 0 44 0 163064 13824 nfs D ?? 0:03.39
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 19566 1 0 60 -20 199928 52336 nfs D< ?? 0:53.69
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 19580 1 0 44 -20 171256 10932 nfs D< ?? 0:08.18
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 19593 1 0 76 -20 201976 10888 nfs D< ?? 0:09.98
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 19595 1 0 44 -20 167160 10760 nfs D< ?? 0:07.23
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 19596 1 0 51 -20 173304 10700 nfs D< ?? 0:08.23
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 20109 1 0 76 -20 201848 11664 nfs D< ?? 0:01.64
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 20293 1 0 65 -20 199928 79584 nfs D< ?? 0:01.05
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 20570 1 0 76 -20 201976 10680 nfs D< ?? 0:02.63
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 20773 1 0 60 -20 199928 10684 nfs D< ?? 0:02.42
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 20774 1 0 44 -20 167160 10860 nfs D< ?? 0:20.15
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 20913 1 0 44 -20 204024 84968 nfs D< ?? 0:22.02
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 21013 1 0 72 -20 210168 42436 nfs D< ?? 0:16.85
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 21091 1 0 54 -20 201976 81764 nfs D< ?? 0:21.27
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 21551 1 0 45 -20 201976 81896 nfs D< ?? 0:07.26
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 21581 1 0 61 -20 201976 81884 nfs D< ?? 0:32.97
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 21692 1 0 66 -20 199928 80132 nfs D< ?? 0:44.93
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 22467 1 0 48 -20 200056 80068 vmopar D< ?? 0:23.91
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 22473 1 0 44 -20 181496 58076 nfs D< ?? 0:06.67
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 22474 1 0 49 -20 181496 60980 nfs D< ?? 0:06.15
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 22722 1 0 46 -20 161016 10676 nfs D< ?? 0:01.12
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 22751 1 0 54 -20 212216 10896 nfs D< ?? 0:24.68
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 22808 1 0 44 -20 179448 10892 nfs D< ?? 0:26.90
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 22874 1 0 49 -20 181496 10648 nfs D< ?? 0:13.46
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 22949 1 0 48 -20 181368 60744 nfs D< ?? 0:00.73
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 23342 1 0 45 0 132088 10900 nfs D ?? 0:00.03
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 23353 1 0 51 -20 193784 10680 nfs D< ?? 0:05.29
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 23758 1 0 72 -20 201976 81968 nfs D< ?? 0:36.37
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 23779 1 0 53 0 175352 15680 nfs D ?? 0:15.83
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 23906 1 0 44 -20 158968 10676 nfs D< ?? 0:03.77
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 23948 1 0 76 -20 199928 79984 nfs D< ?? 0:25.26
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 23956 1 0 62 0 154872 15204 nfs D ?? 0:05.36
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 23976 1 0 67 -20 181368 10692 nfs D< ?? 0:00.79
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 24713 1 0 44 -20 175352 10680 nfs D< ?? 0:04.47
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 25264 1 0 59 0 163064 15208 nfs D ?? 0:05.28
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 25290 1 0 51 0 163064 15716 nfs D ?? 0:07.25
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 25731 1 0 59 0 163064 15600 nfs D ?? 0:02.72
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 25739 1 0 76 -20 201976 81956 nfs D< ?? 0:05.77
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 25986 1 0 45 -20 181496 10692 nfs D< ?? 0:03.28
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 26873 1 0 44 -20 199928 78284 nfs D< ?? 0:36.38
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 26877 1 0 76 -20 181496 61292 nfs D< ?? 0:05.44
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 27215 1 0 49 -20 181496 10676 nfs D< ?? 0:03.17
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 27435 1 0 63 -20 201976 81992 nfs D< ?? 0:20.09
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 27543 1 0 44 0 154872 14984 vmopar D ?? 0:03.65
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 27562 1 0 60 -20 201848 81788 nfs D< ?? 0:03.04
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 27648 1 0 51 0 163064 13392 nfs D ?? 0:04.92
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 27667 1 0 44 -20 183544 14120 nfs D< ?? 0:18.14
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 27840 1 0 44 0 156920 14824 nfs D ?? 0:06.36
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 28078 1 0 63 -20 181496 15416 nfs D< ?? 0:11.50
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 28090 1 0 46 -20 181496 10672 vmopar D< ?? 0:09.44
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 28155 1 0 50 0 169208 15200 nfs D ?? 0:04.77
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 28162 1 0 54 0 167160 5548 nfs D ?? 0:06.27
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 28625 1 0 46 -20 201976 81752 nfs D< ?? 0:08.19
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 28895 1 0 76 0 163064 14828 nfs D ?? 0:02.60
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 28953 1 0 45 0 132088 12660 nfs D ?? 0:00.02
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 29015 1 0 76 -20 199928 79512 nfs D< ?? 0:11.14
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 29210 1 0 45 -20 181496 10796 nfs D< ?? 0:08.80
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 29429 1 0 57 0 136184 13440 nfs D ?? 0:01.68
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 29473 1 0 47 0 136184 13636 nfs D ?? 0:01.19
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 29688 1 0 59 0 173304 8336 nfs D ?? 0:13.10
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 29879 1 0 58 0 132088 12752 nfs D ?? 0:00.39
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 30280 1 0 76 0 132088 12752 vmopar D ?? 0:00.03
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 30439 1 0 76 0 132088 12820 nfs D ?? 0:00.10
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 30595 1 0 44 -20 201976 81104 nfs D< ?? 0:33.13
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 31237 1 0 44 -20 181368 61032 nfs D< ?? 0:01.83
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 31238 1 0 76 -20 201976 82036 nfs D< ?? 1:15.62
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 31312 1 0 76 -20 181496 21012 nfs D< ?? 0:06.26
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 31438 1 0 64 -20 201976 81716 nfs D< ?? 0:14.72
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 31572 1 0 57 0 169208 15444 nfs D ?? 0:21.23
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 31617 1 0 76 0 175352 11000 nfs D ?? 0:22.85
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 31938 1 0 46 -20 181496 61108 nfs D< ?? 0:03.33
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 32070 1 0 57 0 173304 15600 nfs D ?? 0:15.80
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 32071 1 0 65 -20 201976 81628 nfs D< ?? 0:27.76
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 32451 1 0 59 0 158968 14864 nfs D ?? 0:07.64
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 32572 1 0 44 0 161016 14104 nfs D ?? 0:06.11
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 32980 1 0 46 -20 181496 54796 nfs D< ?? 0:08.39
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 33042 1 0 61 0 169208 15384 nfs D ?? 0:03.58
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 33333 1 0 76 -20 199928 75412 nfs D< ?? 0:13.56
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 33353 1 0 53 -20 199928 62204 nfs D< ?? 0:40.47
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 33682 1 0 76 0 136184 13664 nfs D ?? 0:03.65
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 33794 1 0 55 0 165112 14860 nfs D ?? 0:01.62
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 34043 1 0 76 0 136184 9184 nfs D ?? 0:06.89
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 34373 1 0 51 0 132088 12772 nfs D ?? 0:00.03
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 34833 1 0 55 0 132088 12676 nfs D ?? 0:00.88
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 35460 1 0 49 0 132088 12900 nfs D ?? 0:00.20
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 35508 1 0 57 0 136184 12836 nfs D ?? 0:00.70
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 35573 1 0 49 -20 189688 59980 nfs D< ?? 0:19.03
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 35646 1 0 64 0 136184 14216 nfs D ?? 0:03.65
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 35921 1 0 51 0 132088 12828 nfs D ?? 0:00.17
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 36368 1 0 63 0 181496 16032 nfs D ?? 0:14.35
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 36391 1 0 57 -20 181496 60416 nfs D< ?? 0:12.79
/usr/local/sbin/httpd -DNOHTTPACCEPT
0 36410 1 0 76 0 25004 3212 select Ss ?? 0:05.12
/usr/sbin/sshd
80 36431 1 0 44 0 163064 15404 nfs D ?? 0:22.64
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 36497 1 0 70 0 136184 13352 nfs D ?? 0:02.78
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 37113 1 0 69 -20 199928 79680 nfs D< ?? 0:18.34
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 37358 1 0 44 0 132088 12680 nfs D ?? 0:00.02
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 38182 1 0 76 0 185592 14604 nfs D ?? 0:51.43
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 38907 1 0 57 -20 199928 79748 nfs D< ?? 0:12.44
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 40607 1 0 57 -20 181496 50724 nfs D< ?? 0:09.08
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 41682 1 0 64 0 175352 14088 nfs D ?? 0:07.51
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 41705 1 0 76 0 181496 14436 nfs D ?? 0:27.59
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 43380 1 0 44 -20 181496 33816 nfs D< ?? 0:02.26
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 43381 1 0 45 0 154872 13536 nfs D ?? 0:01.87
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 43671 1 0 46 0 132088 12600 nfs D ?? 0:00.02
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 43794 1 0 65 0 177400 13604 nfs D ?? 0:11.35
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 44522 1 0 57 0 169208 14128 nfs D ?? 0:06.29
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 45082 1 0 45 0 181496 15528 nfs D ?? 0:24.88
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 45421 1 0 58 -20 199928 55868 nfs D< ?? 0:02.67
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 46241 1 0 70 -20 201976 59196 nfs D< ?? 0:10.51
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 46243 1 0 56 -20 199928 70824 nfs D< ?? 0:11.06
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 46604 1 0 71 -20 181496 49116 nfs D< ?? 0:02.93
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 46649 1 0 44 -20 181496 20004 nfs D< ?? 0:07.93
/usr/local/sbin/httpd -DNOHTTPACCEPT
0 46791 1 0 44 0 36936 1652 select Ss ?? 0:29.76 sshd:
root@pts/1 (sshd)
80 47119 1 0 71 -20 201976 57668 nfs D< ?? 0:48.74
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 47124 1 0 44 -20 185720 54048 nfs D< ?? 0:42.99
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 47542 1 0 55 -20 201976 71656 nfs D< ?? 0:44.89
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 48148 1 0 71 -20 201976 42480 nfs D< ?? 1:24.99
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 48657 1 0 54 0 173304 15340 nfs D ?? 0:03.34
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 49373 1 0 44 -20 204024 68580 nfs D< ?? 0:07.56
/usr/local/sbin/httpd -DNOHTTPACCEPT
0 49523 1 0 44 0 36936 1636 select Is ?? 0:22.79 sshd:
root@pts/2 (sshd)
80 49624 1 0 44 -20 201848 81272 nfs D< ?? 0:02.94
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 49641 1 0 45 0 156920 14616 nfs D ?? 0:08.58
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 49726 1 0 53 0 165112 15344 nfs D ?? 0:10.71
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 49866 1 0 44 0 161016 8728 nfs D ?? 0:05.48
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 49880 1 0 63 0 169208 13820 nfs D ?? 0:51.78
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 51064 1 0 49 -20 199928 73000 nfs D< ?? 0:08.01
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 52192 1 0 76 -20 191736 25312 nfs D< ?? 0:22.50
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 52481 1 0 44 -20 207208 10492 vmopar D<L ?? 1:21.11
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 52934 1 0 76 -20 204024 39632 nfs D< ?? 0:08.90
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 52935 1 0 76 -20 201848 81204 nfs D< ?? 0:01.55
/usr/local/sbin/httpd -DNOHTTPACCEPT
0 53192 1 0 44 0 36936 1468 select Is ?? 0:00.04 sshd:
root@pts/3 (sshd)
80 53410 1 0 76 -20 201976 55824 nfs D< ?? 0:12.22
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 53829 1 0 65 0 185592 14904 nfs D ?? 0:21.92
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 53835 1 0 76 -20 201976 21428 nfs D< ?? 0:22.51
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 54180 1 0 60 -20 181496 13520 nfs D< ?? 0:09.00
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 54198 1 0 66 0 163064 11232 nfs D ?? 0:05.44
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 54298 1 0 60 0 181624 11760 nfs D ?? 0:24.73
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 54321 1 0 76 -20 201976 61460 nfs D< ?? 0:11.69
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 54346 1 0 53 -20 181496 44508 nfs D< ?? 0:41.25
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 55246 1 0 76 -20 201976 81152 nfs D< ?? 0:10.75
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 55686 1 0 44 0 183544 14476 nfs D ?? 0:16.99
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 55699 1 0 49 0 158968 15192 nfs D ?? 0:02.57
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 56094 1 0 76 -20 181368 17428 nfs D< ?? 0:00.89
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 56952 1 0 44 -20 181368 53532 nfs D< ?? 0:04.73
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 56953 1 0 76 -20 185592 60580 nfs D< ?? 0:22.06
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 56961 1 0 44 0 142456 14904 nfs D ?? 0:02.78
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 57013 1 0 44 -20 189560 51968 nfs D< ?? 0:06.81
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 57061 1 0 44 -20 181368 56200 nfs D< ?? 0:09.06
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 57072 1 0 76 -20 206072 80004 nfs D< ?? 0:06.24
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 57094 1 0 44 0 132088 12660 nfs D ?? 0:00.10
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 57110 1 0 76 -20 193784 66972 nfs D< ?? 0:31.98
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 57126 1 0 64 0 179448 9868 nfs D ?? 0:13.10
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 57132 1 0 44 -20 181368 51388 nfs D< ?? 0:04.66
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 57134 1 0 44 -20 181368 52056 nfs D< ?? 0:07.09
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 57135 1 0 44 -20 181368 47016 nfs D< ?? 0:04.62
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 57137 1 0 44 -20 187512 55680 nfs D< ?? 0:04.50
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 57138 1 0 44 -20 181368 49836 nfs D< ?? 0:04.38
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 57145 1 0 44 -20 181368 48688 nfs D< ?? 0:04.00
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 57182 1 0 76 -20 210168 84148 nfs D< ?? 0:32.90
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 57185 1 0 44 0 132088 12604 nfs D ?? 0:00.04
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 57244 1 0 64 0 165112 15108 nfs D ?? 0:23.07
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 57292 1 0 46 0 132088 5760 nfs D ?? 0:00.22
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 57305 1 0 44 -20 181496 45992 nfs D< ?? 0:06.88
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 57535 1 0 62 -20 185464 55548 nfs D< ?? 0:06.21
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 57572 1 0 76 -20 181496 56476 nfs D< ?? 0:05.72
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 57573 1 0 76 -20 197752 69464 nfs D< ?? 0:04.78
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 57690 1 0 72 0 171256 14076 nfs D ?? 0:07.04
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 57740 1 0 44 -20 181368 55496 nfs D< ?? 0:02.97
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 68934 1 0 44 -20 183800 10664 nfs D< ?? 0:09.17
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 69007 1 0 64 -20 210424 10648 nfs D< ?? 0:09.66
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 69049 1 0 72 -20 214392 10616 nfs D< ?? 0:28.82
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 70180 1 0 64 -20 201976 10616 nfs D< ?? 0:18.46
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 70216 1 0 44 -20 177656 10688 vmopar D< ?? 0:02.01
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 71260 1 0 44 -20 169208 10672 nfs D< ?? 0:04.64
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 71289 1 0 44 -20 181496 10664 nfs D< ?? 0:03.72
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 73640 1 0 44 -20 171256 10652 nfs D< ?? 0:15.90
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 77161 1 0 64 -20 201976 10608 nfs D< ?? 0:44.65
/usr/local/sbin/httpd -DNOHTTPACCEPT
125 78345 1269 0 44 0 5992 1404 kqread S ?? 0:00.01 pickup -l -t
fifo -u
80 78972 1 0 44 -20 183544 10648 nfs D< ?? 0:28.91
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 79156 1 0 76 -20 210168 10640 nfs D< ?? 0:17.09
/usr/local/sbin/httpd -DNOHTTPACCEPT
0 80923 36410 0 76 0 36936 3872 sbwait Is ?? 0:00.02 sshd: YYYY
[priv] (sshd)
1002 80925 80923 0 46 0 36936 3964 select S ?? 0:02.11 sshd:
YYYY@notty (sshd)
1002 80926 80925 0 68 0 10916 6672 select Ss ?? 0:14.96 rsync
--server -lgDtpre.iLf . /data/XXXX/YYYY/Root/
1002 80927 80926 0 45 0 35492 13660 select S ?? 0:01.71 rsync
--server -lgDtpre.iLf . /data/XXXX/YYYY/Root/
0 81065 0 0 46 0 0 16 - IL ?? 0:00.00 [nfsiod 0]
0 81070 1 0 44 0 130004 20952 select Ss ?? 0:00.13
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 81082 81070 0 66 0 130004 20988 accept I ?? 0:00.00
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 81083 81070 0 67 0 130004 20988 accept I ?? 0:00.00
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 81084 81070 0 68 0 130004 20988 accept I ?? 0:00.00
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 81085 81070 0 68 0 130004 20988 accept I ?? 0:00.00
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 81086 81070 0 68 0 130004 20988 accept I ?? 0:00.00
/usr/local/sbin/httpd -DNOHTTPACCEPT
125 81097 1269 0 44 0 5992 1692 kqread S ?? 0:00.01 smtpd -n
smtp -t inet -u -o stress 125 81098 1269 0 44 0 5996 1416 kqread S
?? 0:00.00 proxymap -t unix -u
125 81099 1269 0 44 0 5996 1436 kqread S ?? 0:00.01
trivial-rewrite -n rewrite -t unix -u
125 81100 1269 0 44 0 5996 1544 kqread S ?? 0:00.01 cleanup -z
-t unix -u
125 81101 1269 0 44 0 5996 1592 kqread I ?? 0:00.01 smtp -t unix
-u
80 82983 1 0 76 -20 208120 10656 nfs D< ?? 0:17.83
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 82984 1 0 44 -20 179448 10632 nfs D< ?? 0:29.56
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 86598 1 0 44 -20 183544 10620 nfs D< ?? 0:17.89
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 86639 1 0 44 -20 171256 10656 nfs D< ?? 0:19.92
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 87552 1 0 44 -20 185592 10644 nfs D< ?? 0:14.60
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 87672 1 0 58 -20 208120 10648 nfs D< ?? 0:16.56
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 88435 1 0 55 -20 199928 10668 nfs D< ?? 0:02.46
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 89274 1 0 52 -20 181496 10632 nfs D< ?? 0:16.55
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 92402 1 0 44 -20 171256 10660 nfs D< ?? 0:07.15
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 93189 1 0 55 -20 181496 10672 nfs D< ?? 0:11.48
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 93191 1 0 44 -20 181496 10684 nfs D< ?? 0:00.86
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 93222 1 0 73 -20 181496 10664 nfs D< ?? 0:09.86
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 93223 1 0 44 -20 175352 10652 nfs D< ?? 0:12.19
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 93926 1 0 69 -20 212216 10664 nfs D< ?? 0:06.61
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 94716 1 0 50 -20 208248 10628 nfs D< ?? 0:25.57
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 95314 1 0 44 -20 173304 10664 nfs D< ?? 0:09.51
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 96163 1 0 55 -20 181496 10672 nfs D< ?? 0:03.65
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 96350 1 0 76 -20 208120 10660 nfs D< ?? 0:18.28
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 97819 1 0 44 -20 175352 10640 nfs D< ?? 0:26.13
/usr/local/sbin/httpd -DNOHTTPACCEPT
80 97820 1 0 44 -20 169208 10592 nfs D< ?? 0:35.71
/usr/local/sbin/httpd -DNOHTTPACCEPT
0 1418 1 0 76 0 5860 708 ttyin Is+ v0 0:00.01
/usr/libexec/getty Pc ttyv0
0 1419 1 0 76 0 5860 708 ttyin Is+ v1 0:00.01
/usr/libexec/getty Pc ttyv1
0 1420 1 0 76 0 5860 708 ttyin Is+ v2 0:00.01
/usr/libexec/getty Pc ttyv2
0 1421 1 0 76 0 5860 708 ttyin Is+ v3 0:00.01
/usr/libexec/getty Pc ttyv3
0 1422 1 0 76 0 5860 708 ttyin Is+ v4 0:00.01
/usr/libexec/getty Pc ttyv4
0 1423 1 0 76 0 5860 708 ttyin Is+ v5 0:00.01
/usr/libexec/getty Pc ttyv5
0 1424 1 0 76 0 5860 708 ttyin Is+ v6 0:00.01
/usr/libexec/getty Pc ttyv6
0 1425 1 0 76 0 5860 708 ttyin Is+ v7 0:00.01
/usr/libexec/getty Pc ttyv7
0 1426 1339 0 60 0 2120 0 pause IWs 0 0:00.00 -zsh (zsh)
0 1440 1426 0 44 0 9184 1908 wait S 0 0:00.06 bash
0 81185 1440 0 44 0 6988 1496 - R+ 0 0:00.00 ps -alx
0 46794 46791 0 76 0 2172 808 pause Ss 1 10:53.04 -zsh (zsh)
0 81182 46794 0 98 0 5872 1612 - R+ 1 0:00.98 fstat
0 81183 46794 0 67 0 8060 1136 piperd S+ 1 0:00.06 grep httpd
0 81184 46794 0 69 0 1712 700 piperd S+ 1 0:00.07 wc -l
0 49525 49523 0 44 0 2140 0 pause IWs 2 0:00.00 -zsh (zsh)
0 49845 49525 0 44 0 4772 0 kqread IW+ 2 0:00.00 tail -f
/home/logs/agregat/access_log
0 53204 53192 0 44 0 2168 312 ttyin Is+ 3 0:00.06 -zsh (zsh)
Regards,
--
Herve Boulouis
Kostik Belousov
2011-Jul-26 09:07 UTC
Sleeping thread owns a nonsleepable lock panic (& lor)
On Tue, Jul 26, 2011 at 11:49:13AM +0200, Herve Boulouis wrote:> Le 25/07/2011 11:59, Kostik Belousov a ?crit: > > Ok the patched server crashed this morning strangely : all httpd processes were stuck in nfs or vmopar > and were unkillable. Below is the full ps.Please see the http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-deadlocks.html for information required to debug the deadlocks. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 196 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20110726/ad3c5125/attachment.pgp
Herve Boulouis
2011-Jul-26 09:22 UTC
Sleeping thread owns a nonsleepable lock panic (& lor)
Le 26/07/2011 12:06, Kostik Belousov a écrit:> On Tue, Jul 26, 2011 at 11:49:13AM +0200, Herve Boulouis wrote: > > Le 25/07/2011 11:59, Kostik Belousov a ?crit: > > > > Ok the patched server crashed this morning strangely : all httpd processes were stuck in nfs or vmopar > > and were unkillable. Below is the full ps. > > Please see the > http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-deadlocks.html > for information required to debug the deadlocks.the box was not stricly deadlocked since I was able to interact with it but I suppose you want me to break into debugger when the symptoms appears again and report all the commands listed in the handbook deadlock section ? Regards, -- Herve Boulouis
Kostik Belousov
2011-Jul-26 09:33 UTC
Sleeping thread owns a nonsleepable lock panic (& lor)
On Tue, Jul 26, 2011 at 01:17:52PM +0200, Herve Boulouis wrote:> Le 26/07/2011 12:06, Kostik Belousov a ?crit: > > On Tue, Jul 26, 2011 at 11:49:13AM +0200, Herve Boulouis wrote: > > > Le 25/07/2011 11:59, Kostik Belousov a ?crit: > > > > > > Ok the patched server crashed this morning strangely : all httpd processes were stuck in nfs or vmopar > > > and were unkillable. Below is the full ps. > > > > Please see the > > http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-deadlocks.html > > for information required to debug the deadlocks. > > the box was not stricly deadlocked since I was able to interact with it but I suppose you want me to > break into debugger when the symptoms appears again and report all the commands listed in the handbook > deadlock section ?Exactly. I think everything was hung that accessed an nfs mount point. From the usermode, procstat -kk could catch some interesting information, but it is redundant if ddb output is captured. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 196 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20110726/4500dc25/attachment.pgp