Hi, folks.
I want to add some additional information about problem related
processes in block state. I see it on my server Dual CPU: Intel(R)
Xeon(TM) CPU 2.40GHz (2399.33-MHz 686-class CPU) with amrd device.
FreeBSD 6.1-RELEASE-p1
I see some httpd daemons in D state:
procs memory page disk faults cpu
r b w avm fre flt re pi po fr sr am0 in sy cs us sy
id
0 527 0 2173832 94028 49 0 0 0 2722 2658 0 9219 4140 17350 1 7 92
0 528 0 2172836 93636 66 0 0 0 218 0 6 504 512 1221 0 2 98
0 523 0 2171828 93360 56 0 0 0 188 0 6 551 298 1286 0 2 98
0 523 0 2171972 92612 29 0 0 0 189 0 6 520 395 1250 0 2 98
30 494 0 2170976 91836 92 0 0 0 333 0 11 448 250 970 0 2 98
0 518 0 2171156 90580 43 1 0 0 314 0 10 640 758 1676 0 3 97
0 510 0 2170244 89452 61 0 0 0 402 0 14 615 644 1667 0 3 97
1 512 0 2170420 89292 1 0 0 0 0 0 0 493 307 1133 0 2 98
0 517 0 2169560 86868 110 0 0 0 735 0 27 623 852 1584 0 3 96
0 514 0 2168552 87176 53 0 0 0 198 0 0 487 200 1078 0 1 99
0 517 0 2178752 84040 1523 0 0 0 1261 0 0 512 2660 1320 2 3 96
vmstat -s
3129790854 cpu context switches
1663101011 device interrupts
5504551 software interrupts
9321603 traps
746764672 system calls
54 kernel threads created
66887 fork() calls
9669 vfork() calls
0 rfork() calls
4784 swap pager pageins
5635 swap pager pages paged in
1902 swap pager pageouts
3590 swap pager pages paged out
24668 vnode pager pageins
64050 vnode pager pages paged in
0 vnode pager pageouts
0 vnode pager pages paged out
28140 page daemon wakeups
479450699 pages examined by the page daemon
75270 pages reactivated
2461680 copy-on-write faults
3805 copy-on-write optimized faults
4531914 zero fill pages zeroed
2732630 zero fill pages prezeroed
18095 intransit blocking page faults
8922821 total VM faults taken
0 pages affected by kernel thread creation
8018826 pages affected by fork()
1107169 pages affected by vfork()
0 pages affected by rfork()
491039188 pages freed
393 pages freed by daemon
5744590 pages freed by exiting processes
201969 pages active
184930 pages inactive
17190 pages in VM cache
107624 pages wired down
750 pages free
4096 bytes per page
30979017 total name lookups
cache hits (82% pos + 1% neg) system 1% per-directory
deletions 0%, falsehits 0%, toolong 0%
vmstat -z
ITEM SIZE LIMIT USED FREE REQUESTS
UMA Kegs: 140, 0, 83, 13, 83
UMA Zones: 480, 0, 83, 5, 83
UMA Slabs: 64, 0, 2537, 177, 225611
UMA RCntSlabs: 104, 0, 32769, 13, 2949442
UMA Hash: 128, 0, 3, 27, 6
16 Bucket: 76, 0, 27, 23, 73
32 Bucket: 140, 0, 23, 33, 64
64 Bucket: 268, 0, 17, 39, 100
128 Bucket: 524, 0, 223, 71, 1036
VM OBJECT: 132, 0, 30655, 2318, 1833813
MAP: 192, 0, 7, 33, 7
KMAP ENTRY: 68, 65520, 285, 11251, 78210624
MAP ENTRY: 68, 0, 215821, 8011, 3882740
PV ENTRY: 24, 2155135, 1391603, 119877, 42095182
DP fakepg: 72, 0, 0, 53, 6
mt_zone: 1024, 0, 174, 126, 174
16: 16, 0, 2158, 684, 578060
32: 32, 0, 3343, 386, 87211
64: 64, 0, 7323, 819, 1449967
128: 128, 0, 3507, 333, 519528
256: 256, 0, 6226, 509, 181376
512: 512, 0, 64, 96, 77544
1024: 1024, 0, 55, 73, 38283
2048: 2048, 0, 141, 27, 25379
4096: 4096, 0, 1990, 135, 84846
Files: 72, 0, 2924, 1475, 1665330
PROC: 524, 0, 1936, 220, 76612
THREAD: 372, 0, 2156, 14, 2156
KSEGRP: 88, 0, 2156, 84, 2156
UPCALL: 44, 0, 0, 0, 0
VMSPACE: 300, 0, 1886, 337, 76512
mbuf_packet: 256, 0, 66135, 0, 1979970489
mbuf: 256, 0, 7, 1253, 3511131461
mbuf_cluster: 2048, 65536, 65538, 0, 1966671403
mbuf_jumbo_pagesize: 4096, 0, 0, 0, 0
mbuf_jumbo_9k: 9216, 0, 0, 0, 0
mbuf_jumbo_16k: 16384, 0, 0, 0, 0
ACL UMA zone: 388, 0, 0, 0, 0
g_bio: 132, 0, 0, 87, 79264695
ata_request: 204, 0, 0, 0, 0
ata_composite: 196, 0, 0, 0, 0
VNODE: 272, 0, 27508, 7072, 4868108
VNODEPOLL: 76, 0, 1, 49, 1
S VFS Cache: 68, 0, 27595, 2869, 4582987
L VFS Cache: 291, 0, 1708, 203, 314868
NAMEI: 1024, 0, 3, 129, 12912949
DIRHASH: 1024, 0, 495, 41, 4186
NFSMOUNT: 480, 0, 2, 14, 3
NFSNODE: 460, 0, 13, 27, 1525
PIPE: 408, 0, 28, 35, 31594
KNOTE: 68, 0, 0, 168, 5762
socket: 356, 16434, 2327, 1699, 277319
unpcb: 140, 65548, 42, 70, 12935
ipq: 32, 2147, 0, 0, 2235
udpcb: 180, 16434, 5, 127, 5531
inpcb: 180, 16434, 2259, 1393, 258840
tcpcb: 464, 16424, 2258, 1454, 258840
tcptw: 48, 3354, 1, 389, 38232
syncache: 100, 15366, 17, 334, 322283
hostcache: 76, 15400, 246, 304, 6938
tcpreass: 20, 4225, 1, 168, 4936
sackhole: 20, 0, 8, 499, 23975962
ripcb: 180, 16434, 0, 44, 11
rtentry: 132, 0, 5, 53, 54
pfsrctrpl: 100, 0, 0, 0, 0
pfrulepl: 604, 0, 0, 0, 0
pfstatepl: 260, 10005, 0, 0, 0
pfaltqpl: 128, 0, 0, 0, 0
pfpooladdrpl: 68, 0, 0, 0, 0
pfrktable: 1240, 0, 0, 0, 0
pfrkentry: 156, 0, 0, 0, 0
pfrkentry2: 156, 0, 0, 0, 0
pffrent: 16, 5075, 0, 0, 0
pffrag: 48, 0, 0, 0, 0
pffrcache: 48, 10062, 0, 0, 0
pffrcent: 12, 50141, 0, 0, 0
pfstatescrub: 28, 0, 0, 0, 0
pfiaddrpl: 92, 0, 0, 0, 0
pfospfen: 108, 0, 0, 0, 0
pfosfp: 28, 0, 0, 0, 0
IPFW dynamic rule zone: 108, 0, 0, 0, 0
SWAPMETA: 276, 121576, 36, 20, 1613
FFS inode: 132, 0, 27459, 1280, 4866112
FFS1 dinode: 128, 0, 0, 0, 0
FFS2 dinode: 256, 0, 27459, 1206, 4866112
Intresting thing that everytime i use "vmstat 1" problem gone
and i save my uptime.
Server is still under load and i see only one process in D state
85815 ?? D 0:00.00 /usr/local/sbin/megarc -ldInfo -LAll -a0
Also i have found that it's much easy to trigger this state for httpd
when i use low (default) value for MaxSpareServers (I have about
300 ~ 500 connections all the time)
--
=========================================================================
= Best regards, Nikolay Pavlov. <<<------------------------------------
=
=========================================================================
On Friday, 7 July 2006 at 16:40:26 +0300, Nikolay Pavlov wrote:> Hi, folks. > I want to add some additional information about problem related > processes in block state. I see it on my server Dual CPU: Intel(R) > Xeon(TM) CPU 2.40GHz (2399.33-MHz 686-class CPU) with amrd device.It seems my issue could be related this PR http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/94139 i have submitted follow-up report with backtraces.> > FreeBSD 6.1-RELEASE-p1 > > I see some httpd daemons in D state: > > procs memory page disk faults cpu > r b w avm fre flt re pi po fr sr am0 in sy cs us sy > id > 0 527 0 2173832 94028 49 0 0 0 2722 2658 0 9219 4140 17350 1 7 92 > 0 528 0 2172836 93636 66 0 0 0 218 0 6 504 512 1221 0 2 98 > 0 523 0 2171828 93360 56 0 0 0 188 0 6 551 298 1286 0 2 98 > 0 523 0 2171972 92612 29 0 0 0 189 0 6 520 395 1250 0 2 98 > 30 494 0 2170976 91836 92 0 0 0 333 0 11 448 250 970 0 2 98 > 0 518 0 2171156 90580 43 1 0 0 314 0 10 640 758 1676 0 3 97 > 0 510 0 2170244 89452 61 0 0 0 402 0 14 615 644 1667 0 3 97 > 1 512 0 2170420 89292 1 0 0 0 0 0 0 493 307 1133 0 2 98 > 0 517 0 2169560 86868 110 0 0 0 735 0 27 623 852 1584 0 3 96 > 0 514 0 2168552 87176 53 0 0 0 198 0 0 487 200 1078 0 1 99 > 0 517 0 2178752 84040 1523 0 0 0 1261 0 0 512 2660 1320 2 3 96 > > vmstat -s > > 3129790854 cpu context switches > 1663101011 device interrupts > 5504551 software interrupts > 9321603 traps > 746764672 system calls > 54 kernel threads created > 66887 fork() calls > 9669 vfork() calls > 0 rfork() calls > 4784 swap pager pageins > 5635 swap pager pages paged in > 1902 swap pager pageouts > 3590 swap pager pages paged out > 24668 vnode pager pageins > 64050 vnode pager pages paged in > 0 vnode pager pageouts > 0 vnode pager pages paged out > 28140 page daemon wakeups > 479450699 pages examined by the page daemon > 75270 pages reactivated > 2461680 copy-on-write faults > 3805 copy-on-write optimized faults > 4531914 zero fill pages zeroed > 2732630 zero fill pages prezeroed > 18095 intransit blocking page faults > 8922821 total VM faults taken > 0 pages affected by kernel thread creation > 8018826 pages affected by fork() > 1107169 pages affected by vfork() > 0 pages affected by rfork() > 491039188 pages freed > 393 pages freed by daemon > 5744590 pages freed by exiting processes > 201969 pages active > 184930 pages inactive > 17190 pages in VM cache > 107624 pages wired down > 750 pages free > 4096 bytes per page > 30979017 total name lookups > cache hits (82% pos + 1% neg) system 1% per-directory > deletions 0%, falsehits 0%, toolong 0% > > > vmstat -z > > ITEM SIZE LIMIT USED FREE REQUESTS > > UMA Kegs: 140, 0, 83, 13, 83 > UMA Zones: 480, 0, 83, 5, 83 > UMA Slabs: 64, 0, 2537, 177, 225611 > UMA RCntSlabs: 104, 0, 32769, 13, 2949442 > UMA Hash: 128, 0, 3, 27, 6 > 16 Bucket: 76, 0, 27, 23, 73 > 32 Bucket: 140, 0, 23, 33, 64 > 64 Bucket: 268, 0, 17, 39, 100 > 128 Bucket: 524, 0, 223, 71, 1036 > VM OBJECT: 132, 0, 30655, 2318, 1833813 > MAP: 192, 0, 7, 33, 7 > KMAP ENTRY: 68, 65520, 285, 11251, 78210624 > MAP ENTRY: 68, 0, 215821, 8011, 3882740 > PV ENTRY: 24, 2155135, 1391603, 119877, 42095182 > DP fakepg: 72, 0, 0, 53, 6 > mt_zone: 1024, 0, 174, 126, 174 > 16: 16, 0, 2158, 684, 578060 > 32: 32, 0, 3343, 386, 87211 > 64: 64, 0, 7323, 819, 1449967 > 128: 128, 0, 3507, 333, 519528 > 256: 256, 0, 6226, 509, 181376 > 512: 512, 0, 64, 96, 77544 > 1024: 1024, 0, 55, 73, 38283 > 2048: 2048, 0, 141, 27, 25379 > 4096: 4096, 0, 1990, 135, 84846 > Files: 72, 0, 2924, 1475, 1665330 > PROC: 524, 0, 1936, 220, 76612 > THREAD: 372, 0, 2156, 14, 2156 > KSEGRP: 88, 0, 2156, 84, 2156 > UPCALL: 44, 0, 0, 0, 0 > VMSPACE: 300, 0, 1886, 337, 76512 > mbuf_packet: 256, 0, 66135, 0, 1979970489 > mbuf: 256, 0, 7, 1253, 3511131461 > mbuf_cluster: 2048, 65536, 65538, 0, 1966671403 > mbuf_jumbo_pagesize: 4096, 0, 0, 0, 0 > mbuf_jumbo_9k: 9216, 0, 0, 0, 0 > mbuf_jumbo_16k: 16384, 0, 0, 0, 0 > ACL UMA zone: 388, 0, 0, 0, 0 > g_bio: 132, 0, 0, 87, 79264695 > ata_request: 204, 0, 0, 0, 0 > ata_composite: 196, 0, 0, 0, 0 > VNODE: 272, 0, 27508, 7072, 4868108 > VNODEPOLL: 76, 0, 1, 49, 1 > S VFS Cache: 68, 0, 27595, 2869, 4582987 > L VFS Cache: 291, 0, 1708, 203, 314868 > NAMEI: 1024, 0, 3, 129, 12912949 > DIRHASH: 1024, 0, 495, 41, 4186 > NFSMOUNT: 480, 0, 2, 14, 3 > NFSNODE: 460, 0, 13, 27, 1525 > PIPE: 408, 0, 28, 35, 31594 > KNOTE: 68, 0, 0, 168, 5762 > socket: 356, 16434, 2327, 1699, 277319 > unpcb: 140, 65548, 42, 70, 12935 > ipq: 32, 2147, 0, 0, 2235 > udpcb: 180, 16434, 5, 127, 5531 > inpcb: 180, 16434, 2259, 1393, 258840 > tcpcb: 464, 16424, 2258, 1454, 258840 > tcptw: 48, 3354, 1, 389, 38232 > syncache: 100, 15366, 17, 334, 322283 > hostcache: 76, 15400, 246, 304, 6938 > tcpreass: 20, 4225, 1, 168, 4936 > sackhole: 20, 0, 8, 499, 23975962 > ripcb: 180, 16434, 0, 44, 11 > rtentry: 132, 0, 5, 53, 54 > pfsrctrpl: 100, 0, 0, 0, 0 > pfrulepl: 604, 0, 0, 0, 0 > pfstatepl: 260, 10005, 0, 0, 0 > pfaltqpl: 128, 0, 0, 0, 0 > pfpooladdrpl: 68, 0, 0, 0, 0 > pfrktable: 1240, 0, 0, 0, 0 > pfrkentry: 156, 0, 0, 0, 0 > pfrkentry2: 156, 0, 0, 0, 0 > pffrent: 16, 5075, 0, 0, 0 > pffrag: 48, 0, 0, 0, 0 > pffrcache: 48, 10062, 0, 0, 0 > pffrcent: 12, 50141, 0, 0, 0 > pfstatescrub: 28, 0, 0, 0, 0 > pfiaddrpl: 92, 0, 0, 0, 0 > pfospfen: 108, 0, 0, 0, 0 > pfosfp: 28, 0, 0, 0, 0 > IPFW dynamic rule zone: 108, 0, 0, 0, 0 > SWAPMETA: 276, 121576, 36, 20, 1613 > FFS inode: 132, 0, 27459, 1280, 4866112 > FFS1 dinode: 128, 0, 0, 0, 0 > FFS2 dinode: 256, 0, 27459, 1206, 4866112 > > Intresting thing that everytime i use "vmstat 1" problem gone > and i save my uptime. > > Server is still under load and i see only one process in D state > > 85815 ?? D 0:00.00 /usr/local/sbin/megarc -ldInfo -LAll -a0 > > Also i have found that it's much easy to trigger this state for httpd > when i use low (default) value for MaxSpareServers (I have about > 300 ~ 500 connections all the time) > > > -- > ========================================================================= > = Best regards, Nikolay Pavlov. <<<------------------------------------ = > ========================================================================= > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"-- ========================================================================= = Best regards, Nikolay Pavlov. <<<------------------------------------ = =========================================================================