I've been trying to track down a deadlock on some newish production
servers running FreeBSD 6.3-RELEASE-p2. The deadlock occurs on a
specific (although mundane) hardware configuration, and each of several
servers running this hardware deadlock about once per week.
Although I suspect that this is not hardware related, from a (naive)
perusal of the attached stack traces.
Forgive me if my interpretation of this is all wrong, but I'm pretty
desperate for help. So here's my basic understanding of the deadlock:
These processes seem to be waiting on the page queue mutex:
sendmail (in vm_mmap > vm_map_find > vm_map_insert >
vm_map_pmap_enter)
bsnmpd (in malloc, uma_large_malloc > page_alloc > kmem_malloc)
httpd (in trap > trap_pfault > vm_fault)
[g_up] (in g_vfs_done > bufdone)
The page queue mutex is held by rsync process:
rsync (in trap > trap_pfault > vm_fault > pmap_enter)
Rsync kernel process (in pmap_enter) was interrupted while holding the
page queue lock?
Giant is enabled in loader.conf due to the needs of the pf firewall when
dealing with user credentials lookups. I do not believe that Giant plays
into this deadlock. Kernel config attached.
Any and all help or info is welcome. Thanks in advance.
Stef Walter
-------------- next part --------------
# --------------------------------------------------------------------
# FREEBSD BUILD
FreeBSD m2.npubs.com 6.3-RELEASE-p2 FreeBSD 6.3-RELEASE-p2 #0: Tue Jun 10
02:06:41 UTC 2008 nate@m2.npubs.com:/usr/obj/usr/src/sys/RACK2DBG i386
# --------------------------------------------------------------------
#
# --------------------------------------------------------------------
# PROCESS LIST
db> ps
pid ppid pgrp uid state wmesg wchan cmd
35638 35637 4448 80 LJ *vm page 0xc8d5f780 sendmail
35637 40546 4448 80 SJ wait 0xc9e79218 sh
35620 1801 1801 80 SJ lockf 0xcceb4d00 httpd
35593 1801 1801 80 SJ lockf 0xcce46900 httpd
35566 1801 1801 80 SJ kqread 0xc8f8e000 httpd
35438 15070 15063 70 SJ sbwait 0xd017220c postgres
35401 4527 4527 10219 SJ select 0xc080b7e4 cleanup
35337 4527 4527 10219 SJ select 0xc080b7e4 pickup
35144 4413 4413 80 SJ select 0xc080b7e4 httpd
34797 8611 8611 0 SJ select 0xc080b7e4 virtual
34784 8611 8611 125 SJ select 0xc080b7e4 smtpd
34781 8611 8611 125 SJ select 0xc080b7e4 smtp
34761 8611 8611 125 SJ select 0xc080b7e4 proxymap
34760 8611 8611 125 SJ select 0xc080b7e4 smtpd
34757 8611 8611 125 SJ select 0xc080b7e4 smtp
34750 8611 8611 125 SJ select 0xc080b7e4 cleanup
34087 6435 6435 125 SJ kqread 0xcae9f900 anvil
34075 6435 6435 125 SJ kqread 0xcfb5b600 smtpd
33801 15070 15063 70 SJ sbwait 0xd01a8bc8 postgres
33796 4527 4527 10219 SJ select 0xc080b7e4 bounce
33423 1801 1801 80 SJ select 0xc080b7e4 httpd
33417 15070 15063 70 SJ sbwait 0xd087f638 postgres
33392 1801 1801 80 SJ select 0xc080b7e4 httpd
33207 4527 4527 10219 SJ select 0xc080b7e4 flush
33200 4527 4527 10219 SJ select 0xc080b7e4 bounce
33125 4527 4527 10219 SJ kqread 0xcb2ff900 smtp
33118 4527 4527 10219 SJ select 0xc080b7e4 smtp
33113 4527 4527 10219 SJ select 0xc080b7e4 smtp
33108 4527 4527 10219 SJ select 0xc080b7e4 smtp
33101 4527 4527 10219 SJ select 0xc080b7e4 smtp
33094 4527 4527 10219 SJ select 0xc080b7e4 smtp
33090 4527 4527 10219 SJ select 0xc080b7e4 smtp
33070 33067 33067 0 LJ *Giant 0xcaaf97c0 lynx
33067 33066 33067 0 SsJ wait 0xc9fd9648 sh
33066 4564 4564 0 SJ piperd 0xd0ab7000 cron
33056 4527 4527 10219 SJ select 0xc080b7e4 smtp
33055 4527 4527 10219 SJ lockf 0xc992f480 bounce
32994 15070 15063 70 SJ sbwait 0xc9cdc370 postgres
32976 15070 15063 70 SJ sbwait 0xd0f8e4d4 postgres
32971 4527 4527 10219 SJ select 0xc080b7e4 proxymap
32970 4527 4527 10219 SJ select 0xc080b7e4 smtpd
32869 15070 15063 70 SJ sbwait 0xc8fcc370 postgres
32852 9298 9298 10102 SJ select 0xc080b7e4 cleanup
32837 32836 32836 0 S select 0xc080b7e4 rsync
32836 7802 32836 0 RLs rsync
32774 4527 4527 0 SJ select 0xc080b7e4 local
32754 1801 1801 80 LLJ *vm page 0xc8d5f780 httpd
32753 15070 15063 70 SJ sbwait 0xc9c8ce90 postgres
32747 15070 15063 70 SJ sbwait 0xd01a8370 postgres
32739 15070 15063 70 SJ sbwait 0xccdd3d2c postgres
32710 1 32710 0 Ss biord 0xdcb814d4 jail-measure
32633 1801 1801 80 SJ select 0xc080b7e4 httpd
32623 1801 1801 80 SJ select 0xc080b7e4 httpd
32599 1801 1801 80 SJ select 0xc080b7e4 httpd
32598 1801 1801 80 SJ select 0xc080b7e4 httpd
32597 1801 1801 80 SJ select 0xc080b7e4 httpd
32596 1801 1801 80 SJ lockf 0xcceb43c0 httpd
32595 1801 1801 80 SJ select 0xc080b7e4 httpd
32301 4413 4413 80 SJ lockf 0xcce44680 httpd
32255 4413 4413 80 SJ lockf 0xcce44340 httpd
32206 15070 15063 70 SJ sbwait 0xc9cdf638 postgres
32163 1801 1801 80 SJ select 0xc080b7e4 httpd
32147 4527 4527 10219 SJ select 0xc080b7e4 trivial-rewrite
31910 8611 8611 125 SJ select 0xc080b7e4 smtpd
31740 8611 8611 125 SJ select 0xc080b7e4 trivial-rewrite
30378 4413 4413 80 SJ lockf 0xcd551b40 httpd
30355 4413 4413 80 SJ lockf 0xcceb41c0 httpd
29801 4413 4413 80 SJ lockf 0xcb331340 httpd
29375 9298 9298 10102 SJ select 0xc080b7e4 pickup
28283 4413 4413 80 SJ lockf 0xcb3317c0 httpd
28247 4413 4413 80 SJ kqread 0xcb092200 httpd
28113 9298 9298 10102 SJ select 0xc080b7e4 smtpd
28067 4413 4413 80 SJ lockf 0xd0ff87c0 httpd
26846 4413 4413 80 SJ select 0xc080b7e4 httpd
26581 9298 9298 10102 SJ select 0xc080b7e4 smtpd
26529 26528 8216 125 SJ select 0xc080b7e4 imapd
26528 8216 8216 0 SJ select 0xc080b7e4 couriertls
26424 4413 4413 80 SJ lockf 0xc92fe900 httpd
23902 15070 15063 70 SJ sbwait 0xccdcb0a8 postgres
23891 1801 1801 80 SJ select 0xc080b7e4 httpd
23721 8611 8611 125 SJ select 0xc080b7e4 pickup
22472 6435 6435 125 SJ kqread 0xcfb0cb00 pickup
21951 8958 8958 125 SJ kqread 0xcfb5b200 pickup
19724 3711 3711 125 SJ kqread 0xcfbee800 pickup
17335 9298 9298 10102 SJ select 0xc080b7e4 proxymap
17334 9298 9298 10102 SJ select 0xc080b7e4 trivial-rewrite
11733 40103 40103 80 SJ accept 0xd087d03a httpd
11732 40103 40103 80 SJ accept 0xd087d03a httpd
11731 40103 40103 80 SJ accept 0xd087d03a httpd
11730 40103 40103 80 SJ accept 0xd087d03a httpd
11728 40103 40103 80 SJ accept 0xd087d03a httpd
11727 40103 40103 80 SJ accept 0xd087d03a httpd
11726 40103 40103 80 SJ accept 0xd087d03a httpd
2181 1 2181 181 SsJ (threaded) nagios
100858 S select 0xc080b7e4 nagios
100434 S nanslp 0xc07be46c nagios
87785 7530 7530 80 SJ accept 0xca004b5a httpd
61004 8697 8697 80 SJ accept 0xca96519e httpd
56340 1917 1917 80 SJ lockf 0xcb3311c0 httpd
53973 8611 8611 125 SJ select 0xc080b7e4 anvil
52531 17166 52531 0 R+J CPU 3 ee
45059 7530 7530 80 SJ accept 0xca004b5a httpd
41862 4448 4448 80 SJ accept 0xc9c7d5ca httpd
41181 4448 4448 80 SJ accept 0xc9c7d5ca httpd
41180 4448 4448 80 SJ accept 0xc9c7d5ca httpd
41179 4448 4448 80 SJ accept 0xc9c7d5ca httpd
41178 7530 7530 80 SJ accept 0xca004b5a httpd
41175 4448 4448 80 SJ accept 0xc9c7d5ca httpd
40748 9036 9036 80 SJ accept 0xca965466 httpd
40742 9036 9036 80 SJ accept 0xca965466 httpd
40739 9036 9036 80 SJ accept 0xca965466 httpd
40726 9036 9036 80 SJ accept 0xca965466 httpd
40546 4448 4448 80 SJ wait 0xd024c000 httpd
40444 1917 1917 80 SJ lockf 0xcaa00580 httpd
40443 1917 1917 80 SJ kqread 0xcf271100 httpd
40442 1917 1917 80 SJ lockf 0xcce44500 httpd
40441 1917 1917 80 SJ lockf 0xcd551480 httpd
40440 1917 1917 80 SJ lockf 0xc8dba6c0 httpd
40428 8697 8697 80 SJ accept 0xca96519e httpd
40427 8697 8697 80 SJ accept 0xca96519e httpd
40426 8697 8697 80 SJ accept 0xca96519e httpd
40425 8697 8697 80 SJ accept 0xca96519e httpd
40424 8697 8697 80 SJ accept 0xca96519e httpd
40384 9036 9036 80 SJ accept 0xca965466 httpd
40378 7530 7530 80 SJ accept 0xca004b5a httpd
40377 7530 7530 80 SJ accept 0xca004b5a httpd
40376 7530 7530 80 SJ accept 0xca004b5a httpd
40375 7530 7530 80 SJ accept 0xca004b5a httpd
40374 7530 7530 80 SJ accept 0xca004b5a httpd
40372 9036 9036 80 SJ accept 0xca965466 httpd
40371 9036 9036 80 SJ accept 0xca965466 httpd
40370 9036 9036 80 SJ accept 0xca965466 httpd
40369 9036 9036 80 SJ accept 0xca965466 httpd
40368 9036 9036 80 SJ accept 0xca965466 httpd
40334 40332 4448 0 SJ piperd 0xcca2e198 cronolog
40333 4448 4448 80 SJ accept 0xc9c7d5ca httpd
40332 4448 4448 0 SJ wait 0xcb2cf218 sh
40310 40309 40309 0 SJ piperd 0xc9257cc0 sockin
40309 8742 40309 0 SsJ wait 0xd0171430 sh
40294 40291 40291 0 SJ piperd 0xc8fe7cc0 sockin
40291 6919 40291 0 SsJ wait 0xcdccba78 sh
36263 1 36263 0 SsJ (threaded) rrdbotd
100452 S kserel 0xce155d54 rrdbotd
100188 S select 0xc080b7e4 rrdbotd
100984 S kserel 0xce155d54 rrdbotd
100311 S kserel 0xce155d54 rrdbotd
100271 S select 0xc080b7e4 rrdbotd
100268 S kserel 0xce155d54 rrdbotd
100624 S ksesigwa 0xcaa27780 rrdbotd
17166 17165 17166 0 S+J wait 0xc994d000 bash
17165 17155 17152 0 S+J wait 0xd0d11218 su
17155 17152 17152 0 S+ wait 0xc9ec0a78 sh
17152 1980 17152 0 S+ wait 0xc8a42648 sh
15079 15078 15063 70 S+J select 0xc080b7e4 postgres
15078 15070 15063 70 S+J select 0xc080b7e4 postgres
15077 15070 15063 70 S+J select 0xc080b7e4 postgres
15074 15070 15063 70 S+J select 0xc080b7e4 postgres
15070 1 15063 70 S+J select 0xc080b7e4 postgres
1980 1 1980 0 S+ wait 0xcaa27430 bash
31591 2354 2354 80 SJ lockf 0xc8c8ee80 httpd
31578 2354 2354 80 SJ lockf 0xc8c8e640 httpd
40103 1 40103 0 SsJ nanslp 0xc07be46c httpd
38293 1 38293 0 SsJ nanslp 0xc07be46c cron
38286 1 38286 0 SsJ select 0xc080b7e4 sshd
38224 1 38224 0 SsJ select 0xc080b7e4 syslogd
91887 1 91886 0 SJ select 0xc080b7e4 ruby18
91876 1 91875 0 SJ select 0xc080b7e4 ruby18
61971 61968 8645 65534 SJ piperd 0xcd2d3000 cat
61970 61967 8645 65534 SJ piperd 0xcca51cc0 cat
61968 1 8645 65534 SJ wait 0xce938c90 sh
61967 1 8645 65534 SJ wait 0xc9d2c860 sh
64190 2354 2354 80 SJ kqread 0xcbe8e500 httpd
19433 19430 8645 65534 SJ piperd 0xcffe74c8 cat
19432 19431 8645 65534 SJ piperd 0xcad1d000 cat
19431 1 8645 65534 SJ wait 0xca5b1860 sh
19430 1 8645 65534 SJ wait 0xd0195430 sh
72427 3606 3606 80 SJ select 0xc080b7e4 httpd
95713 3606 3606 80 SJ lockf 0xcd551800 httpd
79740 3606 3606 80 SJ lockf 0xc8c8e7c0 httpd
36833 1 36833 0 SsJ select 0xc080b7e4 ruby
15574 3606 3606 80 SJ lockf 0xcaa00680 httpd
13381 6677 6677 80 SJ accept 0xc9df4466 httpd
11234 9367 9367 80 SJ lockf 0xc8d16a80 httpd
10994 9367 9367 80 SJ lockf 0xd00c2900 httpd
7909 6752 6677 80 SJ select 0xc080b7e4 ruby
73129 6677 6677 80 SJ accept 0xc9df4466 httpd
58056 6752 6677 80 SJ select 0xc080b7e4 ruby
41698 3606 3606 80 SJ lockf 0xc8a53740 httpd
41694 8569 8569 80 SJ accept 0xca965b5a httpd
37070 3266 3266 80 SJ kqread 0xcadb3700 httpd
37069 3266 3266 80 SJ lockf 0xcb7f4280 httpd
37068 3266 3266 80 SJ lockf 0xcec83700 httpd
30486 9367 9367 80 SJ lockf 0xd0ff8800 httpd
30262 9367 9367 80 SJ lockf 0xca6e3d40 httpd
11692 8743 8743 80 SJ lockf 0xcb7f4b80 httpd
11690 8743 8743 80 SJ select 0xc080b7e4 httpd
11599 8743 8743 80 SJ lockf 0xcb331bc0 httpd
11214 8743 8743 80 SJ lockf 0xcb0df340 httpd
11169 8743 8743 80 SJ lockf 0xd0043c00 httpd
9660 6677 6677 80 SJ accept 0xc9df4466 httpd
9659 6752 6677 80 SJ select 0xc080b7e4 ruby
9445 3266 3266 80 SJ lockf 0xcb0df900 httpd
9407 8611 8611 125 SJ select 0xc080b7e4 tlsmgr
9404 2354 2354 80 SJ lockf 0xc95798c0 httpd
9401 9367 9367 80 SJ lockf 0xcb7f4ac0 httpd
9400 9367 9367 80 SJ lockf 0xc95628c0 httpd
9399 9367 9367 80 SJ lockf 0xd05bd080 httpd
9398 9367 9367 80 SJ kqread 0xca619400 httpd
9397 9367 9367 80 SJ lockf 0xc9b5b440 httpd
9382 1 9382 0 SsJ nanslp 0xc07be46c cron
9375 1 9375 0 SsJ select 0xc080b7e4 sshd
9367 1 9367 0 SsJ nanslp 0xc07be46c httpd
9359 3266 3266 80 SJ lockf 0xc957a0c0 httpd
9334 9333 8760 70 SJ select 0xc080b7e4 postgres
9333 8760 8760 70 SJ select 0xc080b7e4 postgres
9332 8760 8760 70 SJ select 0xc080b7e4 postgres
9328 1 9328 0 SsJ nanslp 0xc07be46c cron
9320 1 9320 0 SsJ select 0xc080b7e4 inetd
9315 1 9315 0 SsJ accept 0xcaec119e saslauthd1
9304 9298 9298 10102 SJ select 0xc080b7e4 qmgr
9298 1 9298 0 SsJ select 0xc080b7e4 master
9268 2354 2354 80 SJ lockf 0xcec83ac0 httpd
9241 1 9241 0 SsJ select 0xc080b7e4 bsnmpd
9214 1 9214 0 SsJ nanslp 0xc07be46c cron
9202 1 9202 0 SsJ select 0xc080b7e4 sshd
9139 9107 9090 88 SJ (threaded) mysqld
100427 S kserel 0xc9a0b4b4 mysqld
100308 S kserel 0xc9a0b4b4 mysqld
100276 S kserel 0xc9a0b4b4 mysqld
100582 S kserel 0xc9a0b4b4 mysqld
100572 S select 0xc080b7e4 mysqld
100568 S kserel 0xcadf8394 mysqld
100571 S sigwait 0xeca04c14 mysqld
100564 S ksesigwa 0xc9b7f780 mysqld
9107 1 9090 88 SJ wait 0xcac42a78 sh
9093 1 9093 225 SsJ select 0xc080b7e4 perl5.8.8
9068 1 9068 0 SsJ nanslp 0xc07be46c cron
9058 1 9058 0 SsJ select 0xc080b7e4 sshd
9042 1 41 0 SJ piperd 0xcad1dcc0 logger
9041 1 41 0 SJ fifoor 0xca9e6c58 tail
9036 1 9036 0 SsJ select 0xc080b7e4 httpd
9035 8996 8980 88 SJ (threaded) mysqld
100855 S kserel 0xc8cee394 mysqld
101020 S sbwait 0xc950fd2c mysqld
100994 S select 0xc080b7e4 mysqld
101012 S sbwait 0xcce6320c mysqld
101021 S sbwait 0xd08750a8 mysqld
100393 S kserel 0xc8cee394 mysqld
100999 S sbwait 0xd0eab370 mysqld
101189 S sbwait 0xd0aa1bc8 mysqld
100578 S kserel 0xc8cee394 mysqld
101017 S kserel 0xc8cee394 mysqld
100573 S kserel 0xcadf82d4 mysqld
100558 S sigwait 0xec83bc14 mysqld
100555 S ksesigwa 0xc97f4780 mysqld
8996 1 8980 88 SJ wait 0xca0c4860 sh
8963 8958 8958 125 SJ kqread 0xcabf6400 qmgr
8958 1 8958 0 SsJ kqread 0xc8d71500 master
8893 1 8893 0 SsJ nanslp 0xc07be46c cron
8874 1 8874 0 SsJ select 0xc080b7e4 sshd
8844 1 8844 0 SsJ select 0xc080b7e4 sshd
8811 1 8811 0 SsJ nanslp 0xc07be46c cron
8804 8569 8569 80 SJ accept 0xca965b5a httpd
8802 8569 8569 80 SJ accept 0xca965b5a httpd
8801 8569 8569 80 SJ accept 0xca965b5a httpd
8799 8569 8569 80 SJ accept 0xca965b5a httpd
8798 1 8798 0 SsJ select 0xc080b7e4 sshd
8797 8569 8569 80 SJ accept 0xca965b5a httpd
8791 1 8791 0 SsJ select 0xc080b7e4 proftpd
8769 8743 8743 80 SJ lockf 0xcd6aeb00 httpd
8768 8743 8743 80 SJ lockf 0xcb7f4880 httpd
8767 8743 8743 80 SJ lockf 0xcb06a580 httpd
8766 8743 8743 80 SJ lockf 0xcb0df140 httpd
8765 8743 8743 80 SJ lockf 0xc8fb1700 httpd
8760 1 8760 70 SsJ select 0xc080b7e4 postgres
8743 1 8743 0 SsJ select 0xc080b7e4 httpd
8742 1 8742 0 SsJ select 0xc080b7e4 syslogd
8715 1 8715 0 SsJ nanslp 0xc07be46c cron
8706 1 8706 0 SsJ select 0xc080b7e4 sshd
8697 1 8697 0 SsJ nanslp 0xc07be46c httpd
8645 1 8645 65534 SsJ (threaded) proxsmtpd
100309 S kserel 0xca663154 proxsmtpd
100975 S kserel 0xca663154 proxsmtpd
101018 S kserel 0xca663154 proxsmtpd
100649 S kserel 0xca663154 proxsmtpd
101013 S accept 0xca9655ca proxsmtpd
100581 S ksesigwa 0xcaaf7138 proxsmtpd
8617 8611 8611 125 SJ select 0xc080b7e4 qmgr
8611 1 8611 0 SsJ select 0xc080b7e4 master
8569 1 8569 0 SsJ nanslp 0xc07be46c httpd
8496 1 8496 0 SsJ select 0xc080b7e4 syslogd
8437 1 8437 0 SsJ select 0xc080b7e4 syslogd
8300 8170 8169 0 SJ select 0xc080b7e4 authdaemond
8299 8170 8169 0 SJ select 0xc080b7e4 authdaemond
8298 8170 8169 0 SJ select 0xc080b7e4 authdaemond
8295 1 8295 0 SsJ select 0xc080b7e4 syslogd
8272 8271 8272 0 SJ select 0xc080b7e4 couriertcpd
8271 1 8271 0 SJ piperd 0xca366b28 courierlogger
8253 8252 8253 0 SJ select 0xc080b7e4 couriertcpd
8252 1 8252 0 SJ piperd 0xc904f990 courierlogger
8229 8228 8229 0 SJ select 0xc080b7e4 couriertcpd
8228 1 8228 0 SJ piperd 0xc9e1cb28 courierlogger
8216 8215 8216 0 SJ select 0xc080b7e4 couriertcpd
8215 1 8215 0 SJ piperd 0xca610330 courierlogger
8170 8169 8169 0 SJ select 0xc080b7e4 authdaemond
8169 1 8169 0 SJ piperd 0xc9dc1cc0 courierlogger
8121 1 8121 0 SsJ nanslp 0xc07be46c cron
8105 1 8105 0 SsJ select 0xc080b7e4 sshd
8082 1 8082 0 SsJ select 0xc080b7e4 syslogd
8081 1 8081 0 SsJ select 0xc080b7e4 syslogd
7979 1 7979 0 SsJ nanslp 0xc07be46c cron
7972 1 7972 0 SsJ select 0xc080b7e4 sshd
7848 1 7848 0 Ss+ ttyin 0xc8a6e410 getty
7847 1 7847 0 Ss+ ttyin 0xc8a74810 getty
7846 1 7846 0 Ss+ ttyin 0xc8a73010 getty
7845 1 7845 0 Ss+ ttyin 0xc8a73410 getty
7844 1 7844 0 Ss+ ttyin 0xc8a70c10 getty
7843 1 7843 0 Ss+ ttyin 0xc8a73c10 getty
7842 1 7842 0 Ss+ ttyin 0xc8a76010 getty
7841 1 7841 0 Ss+ ttyin 0xc8a75010 getty
7840 1 7840 0 Ss+ ttyin 0xc8a74410 getty
7829 1 7829 0 Ls *vm page 0xc8d5f780 bsnmpd
7802 1 7802 0 Ss select 0xc080b7e4 inetd
7695 1 7695 0 SsJ select 0xc080b7e4 syslogd
7530 1 7530 0 SsJ nanslp 0xc07be46c httpd
7302 1 7302 0 SsJ select 0xc080b7e4 bsnmpd
6919 1 6919 0 SsJ select 0xc080b7e4 syslogd
6754 6677 6677 80 SJ accept 0xc9df4466 httpd
6753 6677 6677 80 SJ accept 0xc9df4466 httpd
6752 6677 6677 80 SJ select 0xc080b7e4 httpd
6694 1 6694 0 SsJ nanslp 0xc07be46c cron
6687 1 6687 0 SsJ select 0xc080b7e4 sshd
6677 1 6677 0 SsJ nanslp 0xc07be46c httpd
6639 6578 6577 88 S+J piperd 0xca3517f8 logger
6638 6578 6577 88 S+J (threaded) mysqld
100566 S kserel 0xc8e765d4 mysqld
100383 S kserel 0xc8e765d4 mysqld
100988 S kserel 0xc8e765d4 mysqld
101056 S kserel 0xc8e765d4 mysqld
101005 S select 0xc080b7e4 mysqld
100392 S sbwait 0xcceda370 mysqld
100387 S sigwait 0xec6adc14 mysqld
100381 S ksesigwa 0xc9a09568 mysqld
6578 1 6577 88 S+J wait 0xca0c6430 sh
6568 6567 6568 0 S+J select 0xc080b7e4 couriertcpd
6567 1 6567 0 S+J piperd 0xca365660 courierlogger
6509 6508 6509 0 S+J select 0xc080b7e4 couriertcpd
6508 1 6508 0 S+J piperd 0xc99fe990 courierlogger
6505 1 6505 0 SsJ select 0xc080b7e4 syslogd
6454 6435 6435 125 SJ kqread 0xca2a1400 qmgr
6435 1 6435 0 SsJ kqread 0xca122500 master
5653 5529 5521 0 S+J select 0xc080b7e4 authdaemond
5652 5529 5521 0 S+J select 0xc080b7e4 authdaemond
5651 5529 5521 0 S+J select 0xc080b7e4 authdaemond
5650 5529 5521 0 S+J select 0xc080b7e4 authdaemond
5649 5529 5521 0 S+J select 0xc080b7e4 authdaemond
5529 5521 5521 0 S+J select 0xc080b7e4 authdaemond
5521 1 5521 0 S+J piperd 0xc8ee7cc0 courierlogger
5337 1 5337 0 SsJ select 0xc080b7e4 syslogd
5166 1 5166 0 SsJ nanslp 0xc07be46c cron
5153 1 5153 0 SsJ select 0xc080b7e4 sshd
5117 1 5117 0 SsJ (threaded) httpauthd
100563 S kserel 0xc8cee2d4 httpauthd
100652 S kserel 0xc8cee2d4 httpauthd
100468 S kserel 0xc8cee2d4 httpauthd
100444 S kserel 0xc8cee2d4 httpauthd
100183 S sbwait 0xc9df4bc8 httpauthd
100277 S sbwait 0xcb305370 httpauthd
100382 S sbwait 0xcafa0638 httpauthd
100314 S sbwait 0xcb1054d4 httpauthd
100499 S sbwait 0xcae5aa64 httpauthd
100223 S sbwait 0xcae4d4d4 httpauthd
100310 S sbwait 0xcb0bfd2c httpauthd
100973 S sbwait 0xcebd34d4 httpauthd
100388 S sbwait 0xcae4d20c httpauthd
100646 S sbwait 0xd0aa0900 httpauthd
100658 S sbwait 0xc8d3779c httpauthd
100471 S sbwait 0xcae4ca64 httpauthd
100985 S sbwait 0xc9ce420c httpauthd
100175 S sbwait 0xd0cdae90 httpauthd
100317 S sbwait 0xcebce4d4 httpauthd
100580 S sbwait 0xcb105638 httpauthd
100587 S accept 0xc9e0319e httpauthd
100272 S sbwait 0xcb0aa0a8 httpauthd
100556 S sbwait 0xcae59e90 httpauthd
100562 S sbwait 0xcb154e90 httpauthd
100306 S sbwait 0xd0173e90 httpauthd
100221 S sbwait 0xcebc4370 httpauthd
100448 S sbwait 0xd01744d4 httpauthd
100380 S sbwait 0xca004638 httpauthd
100318 S sbwait 0xc9c7da64 httpauthd
100560 S sbwait 0xcaec0370 httpauthd
100205 S sbwait 0xc9dd9e90 httpauthd
100394 S sbwait 0xcb0aaa64 httpauthd
100251 S sbwait 0xcabc120c httpauthd
101072 S sbwait 0xd07560a8 httpauthd
100575 S sbwait 0xccdd7bc8 httpauthd
101207 S sbwait 0xcfd28370 httpauthd
101211 S sbwait 0xd087c370 httpauthd
100469 S sbwait 0xd0177d2c httpauthd
101004 S sbwait 0xc8fa0638 httpauthd
100253 S sbwait 0xc9e04d2c httpauthd
100585 S sbwait 0xca265a64 httpauthd
100270 S sbwait 0xcaf8f20c httpauthd
100313 S sbwait 0xd01780a8 httpauthd
100673 S sbwait 0xcaf8ce90 httpauthd
100651 S sbwait 0xcae6f638 httpauthd
100324 S sbwait 0xcfd27370 httpauthd
100259 S sbwait 0xc9ce0638 httpauthd
100391 S sbwait 0xc9dd9370 httpauthd
100321 S sbwait 0xd01a8900 httpauthd
100476 S sbwait 0xcafa0a64 httpauthd
101190 S sbwait 0xcaf8fbc8 httpauthd
100316 S sbwait 0xc9ce379c httpauthd
100657 S sbwait 0xcb090638 httpauthd
100594 S sbwait 0xd01720a8 httpauthd
100559 S sbwait 0xd01a9638 httpauthd
100583 S sbwait 0xcaf8b900 httpauthd
100574 S sbwait 0xcb154a64 httpauthd
100364 S sbwait 0xc9cd8bc8 httpauthd
100567 S sbwait 0xcf485e90 httpauthd
100971 S sbwait 0xca254900 httpauthd
101016 S sbwait 0xccdf2d2c httpauthd
100977 S sbwait 0xcb0aa638 httpauthd
100447 S sbwait 0xc9cd9d2c httpauthd
100856 S sbwait 0xd0176d2c httpauthd
100586 S sbwait 0xcebcfe90 httpauthd
100249 S sbwait 0xc8d3b370 httpauthd
100972 S sbwait 0xc8d3ba64 httpauthd
100450 S sbwait 0xcaa86370 httpauthd
100384 S sbwait 0xcebcebc8 httpauthd
100981 S sbwait 0xccdcba64 httpauthd
100591 S sbwait 0xd01760a8 httpauthd
100998 S sbwait 0xccdf279c httpauthd
100320 S sbwait 0xcab88d2c httpauthd
101031 S sbwait 0xcdb35a64 httpauthd
100349 S sbwait 0xcf4a7bc8 httpauthd
100589 S sbwait 0xcaf8b79c httpauthd
100386 S sbwait 0xd0cd60a8 httpauthd
100979 S sbwait 0xcae4de90 httpauthd
100590 S sbwait 0xccdcbd2c httpauthd
100569 S sbwait 0xcacd2bc8 httpauthd
101008 S sbwait 0xcebd3638 httpauthd
100672 S sbwait 0xcb153900 httpauthd
100557 S sbwait 0xcb153a64 httpauthd
100541 S sbwait 0xcae59bc8 httpauthd
100389 S sbwait 0xca265900 httpauthd
100415 S sbwait 0xcaf8e0a8 httpauthd
100577 S sbwait 0xce18cbc8 httpauthd
101025 S sbwait 0xcf4a7d2c httpauthd
100561 S sbwait 0xcebd0d2c httpauthd
100385 S sbwait 0xd01774d4 httpauthd
100256 S sbwait 0xcceb74d4 httpauthd
100284 S sbwait 0xcaf8cd2c httpauthd
101040 S sbwait 0xc9cdcbc8 httpauthd
100319 S sbwait 0xd0172d2c httpauthd
100274 S sbwait 0xcb0bfe90 httpauthd
100269 S sbwait 0xd0cd84d4 httpauthd
101011 S sbwait 0xcb3294d4 httpauthd
101069 S sbwait 0xd0cdb79c httpauthd
100449 S sbwait 0xcce6379c httpauthd
100970 S sbwait 0xc8f75a64 httpauthd
101002 S sbwait 0xcb2de0a8 httpauthd
100983 S sbwait 0xc8d3b638 httpauthd
100312 S ksesigwa 0xc97f4bb0 httpauthd
5000 1 5000 389 SsJ (threaded) slapd
100993 S kserel 0xc97f6c34 slapd
100445 S kserel 0xc97f6c34 slapd
100694 S kserel 0xc97f6c34 slapd
100247 S kserel 0xc97f6c34 slapd
100857 S select 0xc080b7e4 slapd
100305 S ksesigwa 0xc9e24138 slapd
4867 1 4867 0 SsJ select 0xc080b7e4 inetd
4847 1 4847 0 SsJ accept 0xc9c7de22 svnserve
4835 1 4834 0 SJ select 0xc080b7e4 ruby18
4796 1 4796 0 SsJ select 0xc080b7e4 syslogd
4712 1 4711 0 SJ select 0xc080b7e4 ruby18
4707 1 4706 0 SJ select 0xc080b7e4 ruby18
4701 1 4700 0 SJ select 0xc080b7e4 ruby18
4648 1 4647 0 SJ select 0xc080b7e4 ruby18
4564 1 4564 0 SsJ nanslp 0xc07be46c cron
4555 1 4555 0 SsJ select 0xc080b7e4 inetd
4550 1 4550 0 SsJ accept 0xc9c7dcbe saslauthd1
4542 1 4541 0 SJ select 0xc080b7e4 ruby18
4535 4527 4527 10219 SJ select 0xc080b7e4 qmgr
4527 1 4527 0 SsJ select 0xc080b7e4 master
4454 1 41 0 S+J piperd 0xc8ee07f8 logger
4453 1 41 0 S+J fifoor 0xc8d04a38 tail
4448 1 4448 0 SsJ select 0xc080b7e4 httpd
4428 1 4428 0 SsJ nanslp 0xc07be46c cron
4421 1 4421 0 SsJ select 0xc080b7e4 sshd
4413 1 4413 0 SsJ nanslp 0xc07be46c httpd
4396 1 4396 0 SsJ select 0xc080b7e4 sshd
4335 1 4335 0 SsJ select 0xc080b7e4 syslogd
4174 4106 4105 88 S+J (threaded) mysqld
100565 S kserel 0xc9a0b394 mysqld
100519 S kserel 0xc9a0b394 mysqld
100976 S kserel 0xc9a0b394 mysqld
100980 S kserel 0xc9a0b394 mysqld
100859 S select 0xc080b7e4 mysqld
101014 S sbwait 0xc950fe90 mysqld
100991 S sbwait 0xd0cd7638 mysqld
101003 S sbwait 0xcdb354d4 mysqld
101001 S sbwait 0xd074820c mysqld
100588 S sbwait 0xccdcb370 mysqld
100454 S sbwait 0xcaf8c79c mysqld
100273 S sigwait 0xec480c14 mysqld
100267 S ksesigwa 0xc9b7fdc8 mysqld
4106 1 4105 88 S+J wait 0xc97f4218 sh
4033 1 4033 0 SsJ select 0xc080b7e4 syslogd
3879 1 3879 0 SsJ select 0xc080b7e4 inetd
3865 1 3865 0 SsJ nanslp 0xc07be46c cron
3858 1 3858 0 SsJ select 0xc080b7e4 sshd
3857 3820 3817 88 S+J (threaded) mysqld
100252 S kserel 0xc8cee4b4 mysqld
100365 S kserel 0xc8cee4b4 mysqld
100245 S select 0xc080b7e4 mysqld
100323 S kserel 0xc8cee4b4 mysqld
100650 S kserel 0xc8cee4b4 mysqld
100250 S kserel 0xc9a0b634 mysqld
100244 S sigwait 0xe9ef5c14 mysqld
100242 S ksesigwa 0xc97f4138 mysqld
3820 1 3817 88 S+J wait 0xc994d430 sh
3818 3606 3606 80 SJ lockf 0xc97e7180 httpd
3816 3606 3606 80 SJ lockf 0xccecb780 httpd
3815 3606 3606 80 SJ lockf 0xc9579c80 httpd
3814 3606 3606 80 SJ lockf 0xcb06a880 httpd
3813 3606 3606 80 SJ lockf 0xc9ec7900 httpd
3729 3711 3711 125 SJ kqread 0xc9920300 qmgr
3711 1 3711 0 SsJ kqread 0xc9914200 master
3613 1 3613 0 SsJ select 0xc080b7e4 proftpd
3606 1 3606 0 SsJ select 0xc080b7e4 httpd
3472 1 3472 0 SsJ select 0xc080b7e4 syslogd
3436 3266 3266 80 SJ lockf 0xd00437c0 httpd
3435 3266 3266 80 SJ lockf 0xcec83d80 httpd
3434 3266 3266 80 SJ lockf 0xc8dba200 httpd
3433 3266 3266 80 SJ lockf 0xd0210540 httpd
3432 3266 3266 80 SJ lockf 0xc8fd8b80 httpd
3398 1 3398 0 SsJ select 0xc080b7e4 inetd
3379 1 3379 0 SsJ nanslp 0xc07be46c cron
3344 1 3344 0 SsJ select 0xc080b7e4 sshd
3288 3279 3266 0 SJ piperd 0xc927ab28 cronolog
3287 3276 3266 0 SJ piperd 0xc8d54330 cronolog
3284 3275 3266 0 SJ piperd 0xc927a990 cronolog
3283 3272 3266 0 SJ piperd 0xc8d55330 cronolog
3279 3266 3266 0 SJ wait 0xc90c7430 sh
3276 3266 3266 0 SJ wait 0xc95a5c90 sh
3275 3266 3266 0 SJ wait 0xc97ef000 sh
3272 3266 3266 0 SJ wait 0xc97ef218 sh
3266 1 3266 0 SsJ nanslp 0xc07be46c httpd
3128 3073 3072 88 S+J (threaded) mysqld
100198 S kserel 0xc8950e74 mysqld
100204 S kserel 0xc8950e74 mysqld
100206 S kserel 0xc8950e74 mysqld
100202 S select 0xc080b7e4 mysqld
100203 S kserel 0xc8950e74 mysqld
100200 S sigwait 0xec3dac14 mysqld
100197 S ksesigwa 0xc8cc0780 mysqld
3073 1 3072 88 S+J wait 0xc90c6a78 sh
3022 1 3022 0 SsJ select 0xc080b7e4 syslogd
2545 2354 2354 80 SJ lockf 0xc8c8e700 httpd
2542 2354 2354 80 SJ lockf 0xc8c8e340 httpd
2541 2354 2354 80 SJ lockf 0xccebf4c0 httpd
2539 2354 2354 80 SJ lockf 0xc8c8e740 httpd
2538 2354 2354 80 SJ lockf 0xc8c8eb40 httpd
2369 1 2369 0 SsJ nanslp 0xc07be46c cron
2362 1 2362 0 SsJ select 0xc080b7e4 sshd
2354 1 2354 0 SsJ nanslp 0xc07be46c httpd
2259 2258 2255 70 SJ select 0xc080b7e4 postgres
2258 2255 2255 70 SJ select 0xc080b7e4 postgres
2257 2255 2255 70 SJ select 0xc080b7e4 postgres
2255 1 2255 70 SsJ select 0xc080b7e4 postgres
2199 1 2199 0 SsJ select 0xc080b7e4 syslogd
1917 1 1917 0 SsJ nanslp 0xc07be46c httpd
1892 1 1892 0 SsJ nanslp 0xc07be46c cron
1885 1 1885 0 SsJ select 0xc080b7e4 sshd
1842 1 1842 0 SsJ nanslp 0xc07be46c cron
1833 1 1833 0 SsJ select 0xc080b7e4 sshd
1801 1 1801 0 SsJ nanslp 0xc07be46c httpd
1800 1 1800 0 SsJ select 0xc080b7e4 syslogd
1476 1 1476 0 SsJ select 0xc080b7e4 syslogd
1161 1 41 0 S+J piperd 0xc9257660 logger
1160 1 41 0 S+J fifoor 0xc8c9ca58 tail
1152 1 1152 0 SsJ nanslp 0xc07be46c cron
1090 1 1090 0 SsJ select 0xc080b7e4 syslogd
680 1 680 0 Ss nanslp 0xc07be46c cron
673 1 673 0 Ss select 0xc080b7e4 sshd
644 1 643 0 S nanslp 0xc07be46c python
613 1 613 0 Ss select 0xc080b7e4 ntpd
524 1 524 0 Ss select 0xc080b7e4 syslogd
469 1 469 0 Ss select 0xc080b7e4 devd
40 0 0 0 SL - 0xec126d04 [schedcpu]
39 0 0 0 SL sdflush 0xc081caf4 [softdepflush]
38 0 0 0 SL syncer 0xc07be1dc [syncer]
37 0 0 0 SL vlruwt 0xc8a58000 [vnlru]
36 0 0 0 SL psleep 0xc080bd60 [bufdaemon]
35 0 0 0 SL pgzero 0xc081dac4 [pagezero]
34 0 0 0 SL psleep 0xc081d5cc [vmdaemon]
33 0 0 0 SL psleep 0xc081d588 [pagedaemon]
32 0 0 0 WL [swi0: sio]
31 0 0 0 WL [irq1: atkbd0]
30 0 0 0 WL [irq19: atapci1]
29 0 0 0 WL [irq15: ata1]
28 0 0 0 WL [irq14: ata0]
27 0 0 0 LL *Giant 0xcaaf97c0 [irq17: em1]
26 0 0 0 LL *Giant 0xcaaf97c0 [irq16: em0]
25 0 0 0 WL [irq9: acpi0]
24 0 0 0 SL - 0xc89a3c80 [acpi_task_2]
23 0 0 0 SL - 0xc89a3c80 [acpi_task_1]
22 0 0 0 SL - 0xc89a3c80 [acpi_task_0]
21 0 0 0 WL [swi2: cambio]
9 0 0 0 SL ccb_scan 0xc07b8b84 [xpt_thrd]
8 0 0 0 SL - 0xc89a3e00 [kqueue taskq]
20 0 0 0 WL [swi5: +]
7 0 0 0 SL - 0xc89d7080 [thread taskq]
19 0 0 0 WL [swi6: Giant taskq]
18 0 0 0 WL [swi6: task queue]
17 0 0 0 SL - 0xc07bae00 [yarrow]
6 0 0 0 SL crypto_r 0xc081c7c4 [crypto returns]
5 0 0 0 SL crypto_w 0xc081c784 [crypto]
4 0 0 0 SL - 0xc07bb928 [g_down]
3 0 0 0 LL *vm page 0xc8d5f780 [g_up]
2 0 0 0 SL - 0xc07bb91c [g_event]
16 0 0 0 WL [swi1: net]
15 0 0 0 WL [swi3: vm]
14 0 0 0 LL *Giant 0xcaaf97c0 [swi4: clock sio]
13 0 0 0 RL CPU 0 [idle: cpu0]
12 0 0 0 RL CPU 1 [idle: cpu1]
11 0 0 0 RL CPU 2 [idle: cpu2]
10 0 0 0 RL [idle: cpu3]
1 0 1 0 SLs wait 0xc8951000 [init]
0 0 0 0 WLs [swapper]
# --------------------------------------------------------------------
# CPU LISTING
db> show pcpu
cpuid = 1
curthread = 0xc894c900: pid 12 "idle: cpu1"
curpcb = 0xe81f2d90
fpcurthread = none
idlethread = 0xc894c900: pid 12 "idle: cpu1"
APIC ID = 1
currentldt = 0x50
spin locks held:
db> show allpcpu
Current CPU: 1
cpuid = 0
curthread = 0xc894ca80: pid 13 "idle: cpu0"
curpcb = 0xe81f5d90
fpcurthread = none
idlethread = 0xc894ca80: pid 13 "idle: cpu0"
APIC ID = 0
currentldt = 0x50
spin locks held:
cpuid = 1
curthread = 0xc894c900: pid 12 "idle: cpu1"
curpcb = 0xe81f2d90
fpcurthread = none
idlethread = 0xc894c900: pid 12 "idle: cpu1"
APIC ID = 1
currentldt = 0x50
spin locks held:
cpuid = 2
curthread = 0xc894c780: pid 11 "idle: cpu2"
curpcb = 0xe81efd90
fpcurthread = none
idlethread = 0xc894c780: pid 11 "idle: cpu2"
APIC ID = 2
currentldt = 0x50
spin locks held:
cpuid = 3
curthread = 0xca268900: pid 52531 "ee"
curpcb = 0xec710d90
fpcurthread = none
idlethread = 0xc894c600: pid 10 "idle: cpu3"
APIC ID = 3
currentldt = 0x50
spin locks held:
# --------------------------------------------------------------------
# LOCK LISTING
db> show locks
db> show alllocks
Process 35638 (sendmail) thread 0xd0530600 (101044)
exclusive sleep mutex vm object (standard object) r = 0 (0xc9c01318) locked @
/usr/src/sys/vm/vm_map.c:1472
exclusive sx user map r = 0 (0xc8fb7734) locked @ /usr/src/sys/vm/vm_map.c:1208
Process 32836 (rsync) thread 0xc9ec1900 (100360)
exclusive sleep mutex pmap r = 0 (0xca89f560) locked @
/usr/src/sys/i386/i386/pmap.c:2041
exclusive sleep mutex vm page queue mutex r = 0 (0xc081d540) locked @
/usr/src/sys/i386/i386/pmap.c:2040
exclusive sx user map r = 0 (0xca89f4e4) locked @ /usr/src/sys/vm/vm_map.c:3105
Process 32754 (httpd) thread 0xcfcc5600 (101170)
exclusive sleep mutex vm object (standard object) r = 0 (0xcc8378c4) locked @
/usr/src/sys/vm/vm_fault.c:297
exclusive sx user map r = 0 (0xc984a4e4) locked @ /usr/src/sys/vm/vm_map.c:3105
Process 7829 (bsnmpd) thread 0xca196180 (100409)
exclusive sleep mutex vm object (kmem object) r = 0 (0xc081d440) locked @
/usr/src/sys/vm/vm_kern.c:325
exclusive sleep mutex system map r = 0 (0xc1068144) locked @
/usr/src/sys/vm/vm_kern.c:295
exclusive sx sysctl lock r = 0 (0xc07be240) locked @
/usr/src/sys/kern/kern_sysctl.c:1375
exclusive sleep mutex Giant r = 0 (0xc07bdb80) locked @
/usr/src/sys/kern/kern_sysctl.c:1313
Process 3 (g_up) thread 0xc894ec00 (100015)
exclusive sleep mutex vm object (standard object) r = 0 (0xcaf6e8c4) locked @
/usr/src/sys/kern/vfs_bio.c:3120
db> show lockedvnods
Locked vnodes
0xcdcce6cc: tag ufs, type VDIR
usecount 2, writecount 0, refcount 4 mountedhere 0
flags ()
v_object 0xcaf6e8c4 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xc97f1000 (pid 32710)#0 0xc054d7f1
at lockmgr+0x4ed
#1 0xc06a0f96 at ffs_lock+0x76
#2 0xc07170ab at VOP_LOCK_APV+0x87
#3 0xc05bd370 at vn_lock+0xac
#4 0xc05b9f0b at getdirentries+0xff
#5 0xc07046b3 at syscall+0x25b
#6 0xc06efccf at Xint0x80_syscall+0x1f
ino 20398955, on dev ad8s1e
# --------------------------------------------------------------------
# PROCESS STACK TRACES
# SENDMAIL
db> trace 35638
Tracing pid 35638 tid 101044 td 0xd0530600
sched_switch(d0530600,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
turnstile_wait(c081d540,c9ec1900,0,c081d540,2,c07497e6,233) at
turnstile_wait+0x35a
_mtx_lock_sleep(c081d540,d0530600,0,c0768214,5f8) at _mtx_lock_sleep+0x14c
_mtx_lock_flags(c081d540,0,c0768214,5f8) at _mtx_lock_flags+0xa2
vm_map_pmap_enter(c8fb76f0,28095000,5,c9c01318,0,0,a000,10) at
vm_map_pmap_enter+0x1df
vm_map_insert(c8fb76f0,c9c01318,0,0,28095000,2809f000,5,7,112) at
vm_map_insert+0x28c
vm_map_find(c8fb76f0,c9c01318,0,0,ed810cc8,a000,1,5,7,112) at vm_map_find+0x86
vm_mmap(c8fb76f0,ed810cc8,a000,5,7,20002,2,c9cd0ae0,0,0) at vm_mmap+0x266
mmap(d0530600,ed810d04) at mmap+0x31f
syscall(2807003b,2806003b,bfbf003b,28082060,0,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (198, FreeBSD ELF32, nosys), eip = 0x2806cd63, esp = 0xbfbfea7c, ebp
= 0xbfbfeab8 ---
# RSYNC
db> trace 32836
Tracing pid 32836 tid 100360 td 0xc9ec1900
sched_switch(c9ec1900,0,2) at sched_switch+0x177
mi_switch(2,0,c07bdb40,0,c074c42d,...) at mi_switch+0x270
critical_exit(1,c9ec1900,0,9d6f000,bfc275bc,...) at critical_exit+0x8b
intr_execute_handlers(c893f6e4,ec552b6c,13,ec552bd4,c06f0033,...) at
intr_execute_handlers+0x129
lapic_handle_intr(35) at lapic_handle_intr+0x2e
Xapic_isr1() at Xapic_isr1+0x33
--- interrupt, eip = 0xc0700897, esp = 0xec552bb0, ebp = 0xec552bd4 ---
pmap_enter(ca89f560,9d6f000,c332e380,7,0,ced476b4,0,c0767c8b,383) at
pmap_enter+0x1bf
vm_fault(ca89f4a0,9d6f000,2,8,c9ec1900,...) at vm_fault+0x10b8
trap_pfault(ec552d38,1,9d6ffb8,9d6ffb8,0,...) at trap_pfault+0xce
trap(bfbf003b,3b,bfbf003b,9d6ffb8,0,...) at trap+0x1eb
calltrap() at calltrap+0x5
--- trap 0xc, eip = 0x2818527f, esp = 0xbfbf9670, ebp = 0xbfbf9b08 ---
# HTTPD
db> trace 32754
Tracing pid 32754 tid 101170 td 0xcfcc5600
sched_switch(cfcc5600,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
turnstile_wait(c081d540,c9ec1900,0,c081d540,2,c07497e6,233) at
turnstile_wait+0x35a
_mtx_lock_sleep(c081d540,cfcc5600,0,c0767c8b,356) at _mtx_lock_sleep+0x14c
_mtx_lock_flags(c081d540,0,c0767c8b,356,c07bdb40,...) at _mtx_lock_flags+0xa2
vm_fault(c984a4a0,88d1000,2,8,cfcc5600,...) at vm_fault+0xfa7
trap_pfault(ed840d38,1,88d1d90,88d1d90,0,...) at trap_pfault+0xce
trap(80a003b,875003b,bfbf003b,2360,8382000,...) at trap+0x1eb
calltrap() at calltrap+0x5
--- trap 0xc, eip = 0x286a34c1, esp = 0xbfbfb860, ebp = 0xbfbfb898 ---
# BSNMPD
db> trace 7829
Tracing pid 7829 tid 100409 td 0xca196180
sched_switch(ca196180,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
turnstile_wait(c081d540,c9ec1900,0,c081d540,2,c07497e6,233) at
turnstile_wait+0x35a
_mtx_lock_sleep(c081d540,ca196180,0,c076806c,16f) at _mtx_lock_sleep+0x14c
_mtx_lock_flags(c081d540,0,c076806c,16f,10928000,...) at _mtx_lock_flags+0xa2
kmem_malloc(c10680c0,7000,102,ec649b64,c06b330f,...) at kmem_malloc+0x282
page_alloc(0,7000,ec649b57,102) at page_alloc+0x1a
uma_large_malloc(7000,102,64f0,c05469d6,ec649bf4,...) at uma_large_malloc+0x3b
malloc(64f0,c078fb40,102,c057de5a,ec649bf4,...) at malloc+0xf3
sysctl_jail_list(c078f220,0,0,ec649bf4,c078f220,...) at sysctl_jail_list+0x8e
sysctl_root(0,ec649c74,3,ec649bf4) at sysctl_root+0x11b
userland_sysctl(ca196180,ec649c74,3,0,bfbfc46c,0,0,0,ec649c70,0,c07bdb80,0,c074b5cd,521)
at userland_sysctl+0xf4
__sysctl(ca196180,ec649d04) at __sysctl+0x77
syscall(bfbe003b,2818003b,bfbe003b,3,bfbfc46c,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (202, FreeBSD ELF32, __sysctl), eip = 0x2814e77b, esp = 0xbfbfc34c,
ebp = 0xbfbfc388 ---
# G_UP
db> trace 3
Tracing pid 3 tid 100015 td 0xc894ec00
sched_switch(c894ec00,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
turnstile_wait(c081d540,c9ec1900,0,c081d540,2,c07497e6,233) at
turnstile_wait+0x35a
_mtx_lock_sleep(c081d540,c894ec00,0,c0752afd,c43) at _mtx_lock_sleep+0x14c
_mtx_lock_flags(c081d540,0,c0752afd,c43,c06b29a0,...) at _mtx_lock_flags+0xa2
bufdone(dcb814d4) at bufdone+0x176
g_vfs_done(cc0806b4) at g_vfs_done+0x8a
biodone(cc0806b4) at biodone+0x58
g_io_schedule_up(c894ec00) at g_io_schedule_up+0xcb
g_up_procbody(0,e8216d38,0,c0523700,0,...) at g_up_procbody+0x5a
fork_exit(c0523700,0,e8216d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe8216d6c, ebp = 0 ---
# EE
db> trace 52531
Tracing pid 52531 tid 100453 td 0xca268900
sched_switch(3dc,c074f76f,ec710b90,c0550d8e,c080b7c0,...) at sched_switch+0x177
sellock(c055154c,c07497db,a,c055154c,0,...) at sellock
(null)(786574,50414d4b,544e4520,75005952,20726573,...) at 0x9
# SWI4: CLOCK SIO
db> trace 14
Tracing pid 14 tid 100002 td 0xc894cc00
sched_switch(c894cc00,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
turnstile_wait(c07bdb80,ca196180,0,c07bdb80,2,c07497e6,233) at
turnstile_wait+0x35a
_mtx_lock_sleep(c07bdb80,c894cc00,0,c074bcf0,102) at _mtx_lock_sleep+0x14c
_mtx_lock_flags(c07bdb80,0,c074bcf0,102) at _mtx_lock_flags+0xa2
softclock(0) at softclock+0x18f
ithread_execute_handlers(c894b430,c89a2b80) at ithread_execute_handlers+0xe6
ithread_loop(c89288e0,e81f8d38,c89288e0,c05456e0,0,...) at ithread_loop+0x67
fork_exit(c05456e0,c89288e0,e81f8d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe81f8d6c, ebp = 0 ---
# EM0
db> trace 27
Tracing pid 27 tid 100017 td 0xc894e900
sched_switch(c894e900,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
turnstile_wait(c07bdb80,ca196180,0,c07bdb80,2,c07497e6,233) at
turnstile_wait+0x35a
_mtx_lock_sleep(c07bdb80,c894e900,0,c0747942,2a9) at _mtx_lock_sleep+0x14c
_mtx_lock_flags(c07bdb80,0,c0747942,2a9) at _mtx_lock_flags+0xa2
ithread_execute_handlers(c89a4648,c89a1480) at ithread_execute_handlers+0xde
ithread_loop(c8a438d0,e8210d38,c8a438d0,c05456e0,0,...) at ithread_loop+0x67
fork_exit(c05456e0,c8a438d0,e8210d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe8210d6c, ebp = 0 ---
# em1
db> trace 26
Tracing pid 26 tid 100018 td 0xc894e780
sched_switch(c894e780,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
turnstile_wait(c07bdb80,ca196180,0,c07bdb80,2,c07497e6,233) at
turnstile_wait+0x35a
_mtx_lock_sleep(c07bdb80,c894e780,0,c0747942,2a9) at _mtx_lock_sleep+0x14c
_mtx_lock_flags(c07bdb80,0,c0747942,2a9) at _mtx_lock_flags+0xa2
ithread_execute_handlers(c89a4860,c89a1500) at ithread_execute_handlers+0xde
ithread_loop(c8a3f6a0,e820dd38,c8a3f6a0,c05456e0,0,...) at ithread_loop+0x67
fork_exit(c05456e0,c8a3f6a0,e820dd38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe820dd6c, ebp = 0 ---
# LYNX
db> trace 33070
Tracing pid 33070 tid 100668 td 0xca02f600
sched_switch(ca02f600,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
turnstile_wait(c07bdb80,ca196180,0,c07bdb80,2,c07497e6,233) at
turnstile_wait+0x35a
_mtx_lock_sleep(c07bdb80,ca02f600,0,c074fb3c,ec) at _mtx_lock_sleep+0x14c
_mtx_lock_flags(c07bdb80,0,c074fb3c,ec) at _mtx_lock_flags+0xa2
soo_poll(d015b000,40,d06e1b80,ca02f600) at soo_poll+0x2a
selscan(ca02f600,ec5d5b94,ec5d5b84,14,c080b7c0,0,c074f76f,300) at selscan+0x1a5
kern_select(ca02f600,14,bfbfe304,0,0,...) at kern_select+0x33b
select(ca02f600,ec5d5d04) at select+0x44
syscall(3b,bfbf003b,bfbf003b,0,bfbfe304,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (93, FreeBSD ELF32, select), eip = 0x281e6be4, esp = 0xbfbfe2a8, ebp
= 0xbfbfe384 ---
-------------- next part --------------
#
# RACK2 -- M2 config file
#
machine i386
cpu I686_CPU
ident RACK2
# To statically compile in device wiring instead of /boot/device.hints
#hints "GENERIC.hints" # Default places to look for devices.
makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols
options SCHED_4BSD # 4BSD scheduler
options PREEMPTION # Enable kernel thread preemption
options INET # InterNETworking
options INET6 # IPv6 communications protocols
options FFS # Berkeley Fast Filesystem
options SOFTUPDATES # Enable FFS soft updates support
options UFS_ACL # Support for access control lists
options UFS_DIRHASH # Improve performance on big directories
options MD_ROOT # MD is a potential root device
options NFSCLIENT # Network Filesystem Client
options NFSSERVER # Network Filesystem Server
options NFS_ROOT # NFS usable as /, requires NFSCLIENT
options MSDOSFS # MSDOS Filesystem
options CD9660 # ISO 9660 Filesystem
options PROCFS # Process filesystem (requires PSEUDOFS)
options PSEUDOFS # Pseudo-filesystem framework
options GEOM_GPT # GUID Partition Tables.
options COMPAT_43 # Compatible with BSD 4.3 [KEEP THIS!]
options COMPAT_FREEBSD4 # Compatible with FreeBSD4
options COMPAT_FREEBSD5 # Compatible with FreeBSD5
options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI
options KTRACE # ktrace(1) support
options SYSVSHM # SYSV-style shared memory
options SYSVMSG # SYSV-style message queues
options SYSVSEM # SYSV-style semaphores
options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions
options KBD_INSTALL_CDEV # install a CDEV entry in /dev
options ADAPTIVE_GIANT # Giant mutex is adaptive.
options SMP
options FAST_IPSEC
options IPFIREWALL
options IPFIREWALL_DEFAULT_TO_ACCEPT
options ALTQ
options ALTQ_CBQ
options ALTQ_RED
options ALTQ_RIO
options ALTQ_HFSC
options ALTQ_CDNR
options ALTQ_PRIQ
options KDB
options KDB_TRACE
options KDB_UNATTENDED
options DDB
options INVARIANTS
options INVARIANT_SUPPORT
options WITNESS
options WITNESS_SKIPSPIN
options DIAGNOSTIC
options BREAK_TO_DEBUGGER
options DEBUG_LOCKS
options DEBUG_VFS_LOCKS
device crypto
device apic # I/O APIC
# Bus support.
device eisa
device pci
# Floppy drives
device fdc
# ATA and ATAPI devices
device ata
device atadisk # ATA disk drives
device ataraid # ATA RAID drives
device atapicd # ATAPI CDROM drives
device atapifd # ATAPI floppy drives
device atapist # ATAPI tape drives
options ATA_STATIC_ID # Static device numbering
# SCSI Controllers
#device ahb # EISA AHA1742 family
device ahc # AHA2940 and onboard AIC7xxx devices
#options AHC_REG_PRETTY_PRINT # Print register bitfields in debug
# output. Adds ~128k to driver.
device ahd # AHA39320/29320 and onboard AIC79xx devices
#options AHD_REG_PRETTY_PRINT # Print register bitfields in debug
# output. Adds ~215k to driver.
#device amd # AMD 53C974 (Tekram DC-390(T))
#device isp # Qlogic family
#device ispfw # Firmware for QLogic HBAs- normally a module
#device mpt # LSI-Logic MPT-Fusion
#device ncr # NCR/Symbios Logic
#device sym # NCR/Symbios Logic (newer chipsets + those of `ncr')
#device trm # Tekram DC395U/UW/F DC315U adapters
#device adv # Advansys SCSI adapters
#device adw # Advansys wide SCSI adapters
device aha # Adaptec 154x SCSI adapters
device aic # Adaptec 15[012]x SCSI adapters, AIC-6[23]60.
#device bt # Buslogic/Mylex MultiMaster SCSI adapters
#device ncv # NCR 53C500
#device nsp # Workbit Ninja SCSI-3
#device stg # TMC 18C30/18C50
# SCSI peripherals
device scbus # SCSI bus (required for SCSI)
#device ch # SCSI media changers
device da # Direct Access (disks)
#device sa # Sequential Access (tape etc)
device cd # CD
device pass # Passthrough device (direct SCSI access)
#device ses # SCSI Environmental Services (and SAF-TE)
# RAID controllers interfaced to the SCSI subsystem
#device amr # AMI MegaRAID
#device arcmsr # Areca SATA II RAID
#device asr # DPT SmartRAID V, VI and Adaptec SCSI RAID
#device ciss # Compaq Smart RAID 5*
#device dpt # DPT Smartcache III, IV - See NOTES for options
#device hptmv # Highpoint RocketRAID 182x
#device hptrr # Highpoint RocketRAID 17xx, 22xx, 23xx, 25xx
#device rr232x # Highpoint RocketRAID 232x
#device iir # Intel Integrated RAID
#device ips # IBM (Adaptec) ServeRAID
#device mly # Mylex AcceleRAID/eXtremeRAID
#device twa # 3ware 9000 series PATA/SATA RAID
# RAID controllers
device aac # Adaptec FSA RAID
device aacp # SCSI passthrough for aac (requires CAM)
#device ida # Compaq Smart RAID
#device mfi # LSI MegaRAID SAS
#device mlx # Mylex DAC960 family
#device pst # Promise Supertrak SX6000
#device twe # 3ware ATA RAID
# atkbdc0 controls both the keyboard and the PS/2 mouse
device atkbdc # AT keyboard controller
device atkbd # AT keyboard
device psm # PS/2 mouse
device kbdmux # keyboard multiplexer
device vga # VGA video card driver
device splash # Splash screen and screen saver support
# syscons is the default console driver, resembling an SCO console
device sc
# Enable this for the pcvt (VT220 compatible) console driver
#device vt
#options XSERVER # support for X server on a vt console
#options FAT_CURSOR # start with block cursor
device agp # support several AGP chipsets
# Power management support (see NOTES for more options)
#device apm
# Add suspend/resume support for the i8254.
device pmtimer
# PCCARD (PCMCIA) support
# PCMCIA and cardbus bridge support
#device cbb # cardbus (yenta) bridge
#device pccard # PC Card (16-bit) bus
#device cardbus # CardBus (32-bit) bus
# Serial (COM) ports
device sio # 8250, 16[45]50 based serial ports
# Parallel port
#device ppc
#device ppbus # Parallel port bus (required)
#device lpt # Printer
#device plip # TCP/IP over parallel
#device ppi # Parallel port interface device
#device vpo # Requires scbus and da
# If you've got a "dumb" serial or parallel PCI card that is
# supported by the puc(4) glue driver, uncomment the following
# line to enable it (connects to the sio and/or ppc drivers):
#device puc
# PCI Ethernet NICs.
#device de # DEC/Intel DC21x4x (``Tulip'')
device em # Intel PRO/1000 adapter Gigabit Ethernet Card
device ixgb # Intel PRO/10GbE Ethernet Card
#device txp # 3Com 3cR990 (``Typhoon'')
#device vx # 3Com 3c590, 3c595 (``Vortex'')
# PCI Ethernet NICs that use the common MII bus controller code.
# NOTE: Be sure to keep the 'device miibus' line in order to use these
NICs!
device miibus # MII bus support
#device bce # Broadcom BCM5706/BCM5708 Gigabit Ethernet
#device bfe # Broadcom BCM440x 10/100 Ethernet
device bge # Broadcom BCM570xx Gigabit Ethernet
device dc # DEC/Intel 21143 and various workalikes
device fxp # Intel EtherExpress PRO/100B (82557, 82558)
#device lge # Level 1 LXT1001 gigabit Ethernet
#device msk # Marvell/SysKonnect Yukon II Gigabit Ethernet
#device nge # NatSemi DP83820 gigabit Ethernet
#device nve # nVidia nForce MCP on-board Ethernet Networking
#device pcn # AMD Am79C97x PCI 10/100(precedence over 'lnc')
device re # RealTek 8139C+/8169/8169S/8110S
device rl # RealTek 8129/8139
#device sf # Adaptec AIC-6915 (``Starfire'')
device sis # Silicon Integrated Systems SiS 900/SiS 7016
#device sk # SysKonnect SK-984x & SK-982x gigabit Ethernet
#device ste # Sundance ST201 (D-Link DFE-550TX)
#device stge # Sundance/Tamarack TC9021 gigabit Ethernet
#device ti # Alteon Networks Tigon I/II gigabit Ethernet
#device tl # Texas Instruments ThunderLAN
#device tx # SMC EtherPower II (83c170 ``EPIC'')
#device vge # VIA VT612x gigabit Ethernet
device vr # VIA Rhine, Rhine II
#device wb # Winbond W89C840F
device xl # 3Com 3c90x (``Boomerang'', ``Cyclone'')
# ISA Ethernet NICs. pccard NICs included.
#device cs # Crystal Semiconductor CS89x0 NIC
# 'device ed' requires 'device miibus'
#device ed # NE[12]000, SMC Ultra, 3c503, DS8390 cards
#device ex # Intel EtherExpress Pro/10 and Pro/10+
#device ep # Etherlink III based cards
#device fe # Fujitsu MB8696x based cards
#device ie # EtherExpress 8/16, 3C507, StarLAN 10 etc.
device lnc # NE2100, NE32-VL Lance Ethernet cards
#device sn # SMC's 9000 series of Ethernet chips
#device xe # Xircom pccard Ethernet
# Wireless NIC cards
#device wlan # 802.11 support
#device wlan_wep # 802.11 WEP support
#device wlan_ccmp # 802.11 CCMP support
#device wlan_tkip # 802.11 TKIP support
#device an # Aironet 4500/4800 802.11 wireless NICs.
#device ath # Atheros pci/cardbus NIC's
#device ath_hal # Atheros HAL (Hardware Access Layer)
#device ath_rate_sample # SampleRate tx rate control for ath
#device awi # BayStack 660 and others
#device ral # Ralink Technology RT2500 wireless NICs.
#device wi # WaveLAN/Intersil/Symbol 802.11 wireless NICs.
#device wl # Older non 802.11 Wavelan wireless NIC.
# Pseudo devices.
device loop # Network loopback
device random # Entropy device
device ether # Ethernet support
device sl # Kernel SLIP
device ppp # Kernel PPP
device tun # Packet tunnel.
device pty # Pseudo-ttys (telnet etc)
device md # Memory "disks"
device gif # IPv6 and IPv4 tunneling
device faith # IPv6-to-IPv4 relaying (translation)
# The `bpf' device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
# Note that 'bpf' is required for DHCP.
device bpf # Berkeley packet filter
# USB support
#device uhci # UHCI PCI->USB interface
#device ohci # OHCI PCI->USB interface
#device ehci # EHCI PCI->USB interface (USB 2.0)
#device usb # USB Bus (required)
#device udbp # USB Double Bulk Pipe devices
#device ugen # Generic
#device uhid # "Human Interface Devices"
#device ukbd # Keyboard
#device ulpt # Printer
#device umass # Disks/Mass storage - Requires scbus and da
#device ums # Mouse
#device ural # Ralink Technology RT2500USB wireless NICs
#device urio # Diamond Rio 500 MP3 player
#device uscanner # Scanners
# USB Ethernet, requires miibus
#device aue # ADMtek USB Ethernet
#device axe # ASIX Electronics USB Ethernet
#device cdce # Generic USB over Ethernet
#device cue # CATC USB Ethernet
#device kue # Kawasaki LSI USB Ethernet
#device rue # RealTek RTL8150 USB Ethernet
# FireWire support
#device firewire # FireWire bus code
#device sbp # SCSI over FireWire (Requires scbus and da)
#device fwe # Ethernet over FireWire (non-standard!)
Stef Walter
2008-Jun-16 11:10 UTC
FreeBSD 6.3 deadlock (interrupted pmap_remove_pages) with DDB output
Stef Walter wrote:> I've been trying to track down a deadlock on some newish production > servers running FreeBSD 6.3-RELEASE-p2. The deadlock occurs on a > specific (although mundane) hardware configuration, and each of several > servers running this hardware deadlock about once per week. > > Although I suspect that this is not hardware related, from a (naive) > perusal of the attached stack traces.Here's another similar deadlock that occurred on an similar machine. Last time pmap_enter was preempted by an interrupt, this time pmap_remove_pages was preempted by a timer. I tried changing machdep.cpu_idle_hlt to zero, based on a hint [1], but that didn't seem to make a difference. Again in this case the processes are waiting on the page queue mutex: python (in trap > trap_pfault > vm_fault) rsync (in ... cluster_read > getblk > getnewbuf > vfs_vmio_release) And another rsync process is holding the vm page queue mutex: rsync (in exit1 > vmspace_exit > pmap_remove_pages) The rsync process in pmap_remove_pages was interrupted by a timer while holding the vm page queue lock and then was not scheduled again. Kernel config and full details attched. I may try building the kernel without PREEMPTION to see if it fixes this problem. Again, any and all advice or hints are welcome. I really appreciate any time spent on this. Stef Walter [1] http://unix.derkeiler.com/Mailing-Lists/FreeBSD/current/2005-02/0508.html -------------- next part -------------- # # RACK1 # machine i386 cpu I686_CPU ident RACK1 # To statically compile in device wiring instead of /boot/device.hints #hints "GENERIC.hints" # Default places to look for devices. makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols options SCHED_4BSD # 4BSD scheduler options PREEMPTION # Enable kernel thread preemption options INET # InterNETworking options INET6 # IPv6 communications protocols options FFS # Berkeley Fast Filesystem options SOFTUPDATES # Enable FFS soft updates support options UFS_ACL # Support for access control lists options UFS_DIRHASH # Improve performance on big directories options MD_ROOT # MD is a potential root device options NFSCLIENT # Network Filesystem Client options NFSSERVER # Network Filesystem Server options NFS_ROOT # NFS usable as /, requires NFSCLIENT options MSDOSFS # MSDOS Filesystem options CD9660 # ISO 9660 Filesystem options PROCFS # Process filesystem (requires PSEUDOFS) options PSEUDOFS # Pseudo-filesystem framework options GEOM_GPT # GUID Partition Tables. options COMPAT_43 # Compatible with BSD 4.3 [KEEP THIS!] options COMPAT_FREEBSD4 # Compatible with FreeBSD4 options COMPAT_FREEBSD5 # Compatible with FreeBSD5 options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI options KTRACE # ktrace(1) support options SYSVSHM # SYSV-style shared memory options SYSVMSG # SYSV-style message queues options SYSVSEM # SYSV-style semaphores options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions options KBD_INSTALL_CDEV # install a CDEV entry in /dev options ADAPTIVE_GIANT # Giant mutex is adaptive. device apic # I/O APIC options SMP options QUOTA options FAST_IPSEC options IPFIREWALL options IPFIREWALL_DEFAULT_TO_ACCEPT options IPFIREWALL_VERBOSE options IPFIREWALL_FORWARD options IPDIVERT options DUMMYNET options ALTQ options ALTQ_CBQ options ALTQ_RED options ALTQ_RIO options ALTQ_HFSC options ALTQ_CDNR options ALTQ_PRIQ options KDB options KDB_TRACE options KDB_UNATTENDED options DDB options INVARIANTS options INVARIANT_SUPPORT options WITNESS options WITNESS_SKIPSPIN options DIAGNOSTIC options BREAK_TO_DEBUGGER options DEBUG_LOCKS options DEBUG_VFS_LOCKS device crypto # Bus support. device eisa device pci # Floppy drives device fdc # ATA and ATAPI devices device ata device atadisk # ATA disk drives device ataraid # ATA RAID drives device atapicd # ATAPI CDROM drives #device atapifd # ATAPI floppy drives #device atapist # ATAPI tape drives options ATA_STATIC_ID # Static device numbering # SCSI Controllers #device ahb # EISA AHA1742 family #device ahc # AHA2940 and onboard AIC7xxx devices #options AHC_REG_PRETTY_PRINT # Print register bitfields in debug # output. Adds ~128k to driver. #device ahd # AHA39320/29320 and onboard AIC79xx devices #options AHD_REG_PRETTY_PRINT # Print register bitfields in debug # output. Adds ~215k to driver. #device amd # AMD 53C974 (Tekram DC-390(T)) #device isp # Qlogic family #device ispfw # Firmware for QLogic HBAs- normally a module #device mpt # LSI-Logic MPT-Fusion #device ncr # NCR/Symbios Logic #device sym # NCR/Symbios Logic (newer chipsets + those of `ncr') #device trm # Tekram DC395U/UW/F DC315U adapters #device adv # Advansys SCSI adapters #device adw # Advansys wide SCSI adapters #device aha # Adaptec 154x SCSI adapters #device aic # Adaptec 15[012]x SCSI adapters, AIC-6[23]60. #device bt # Buslogic/Mylex MultiMaster SCSI adapters #device ncv # NCR 53C500 #device nsp # Workbit Ninja SCSI-3 #device stg # TMC 18C30/18C50 # SCSI peripherals device scbus # SCSI bus (required for SCSI) #device ch # SCSI media changers device da # Direct Access (disks) #device sa # Sequential Access (tape etc) #device cd # CD #device pass # Passthrough device (direct SCSI access) #device ses # SCSI Environmental Services (and SAF-TE) # RAID controllers interfaced to the SCSI subsystem #device amr # AMI MegaRAID #device arcmsr # Areca SATA II RAID #device asr # DPT SmartRAID V, VI and Adaptec SCSI RAID #device ciss # Compaq Smart RAID 5* #device dpt # DPT Smartcache III, IV - See NOTES for options #device hptmv # Highpoint RocketRAID 182x #device hptrr # Highpoint RocketRAID 17xx, 22xx, 23xx, 25xx #device rr232x # Highpoint RocketRAID 232x device iir # Intel Integrated RAID #device ips # IBM (Adaptec) ServeRAID #device mly # Mylex AcceleRAID/eXtremeRAID #device twa # 3ware 9000 series PATA/SATA RAID # RAID controllers #device aac # Adaptec FSA RAID #device aacp # SCSI passthrough for aac (requires CAM) #device ida # Compaq Smart RAID #device mfi # LSI MegaRAID SAS #device mlx # Mylex DAC960 family #device pst # Promise Supertrak SX6000 #device twe # 3ware ATA RAID # atkbdc0 controls both the keyboard and the PS/2 mouse device atkbdc # AT keyboard controller device atkbd # AT keyboard device psm # PS/2 mouse device kbdmux # keyboard multiplexer device vga # VGA video card driver device splash # Splash screen and screen saver support # syscons is the default console driver, resembling an SCO console device sc # Enable this for the pcvt (VT220 compatible) console driver #device vt #options XSERVER # support for X server on a vt console #options FAT_CURSOR # start with block cursor device agp # support several AGP chipsets # Power management support (see NOTES for more options) #device apm # Add suspend/resume support for the i8254. device pmtimer # PCCARD (PCMCIA) support # PCMCIA and cardbus bridge support #device cbb # cardbus (yenta) bridge #device pccard # PC Card (16-bit) bus #device cardbus # CardBus (32-bit) bus # Serial (COM) ports device sio # 8250, 16[45]50 based serial ports # Parallel port #device ppc #device ppbus # Parallel port bus (required) #device lpt # Printer #device plip # TCP/IP over parallel #device ppi # Parallel port interface device #device vpo # Requires scbus and da # If you've got a "dumb" serial or parallel PCI card that is # supported by the puc(4) glue driver, uncomment the following # line to enable it (connects to the sio and/or ppc drivers): #device puc # PCI Ethernet NICs. device de # DEC/Intel DC21x4x (``Tulip'') device em # Intel PRO/1000 adapter Gigabit Ethernet Card #device ixgb # Intel PRO/10GbE Ethernet Card #device txp # 3Com 3cR990 (``Typhoon'') #device vx # 3Com 3c590, 3c595 (``Vortex'') # PCI Ethernet NICs that use the common MII bus controller code. # NOTE: Be sure to keep the 'device miibus' line in order to use these NICs! device miibus # MII bus support #device bce # Broadcom BCM5706/BCM5708 Gigabit Ethernet #device bfe # Broadcom BCM440x 10/100 Ethernet #device bge # Broadcom BCM570xx Gigabit Ethernet device dc # DEC/Intel 21143 and various workalikes device fxp # Intel EtherExpress PRO/100B (82557, 82558) #device lge # Level 1 LXT1001 gigabit Ethernet #device msk # Marvell/SysKonnect Yukon II Gigabit Ethernet #device nge # NatSemi DP83820 gigabit Ethernet #device nve # nVidia nForce MCP on-board Ethernet Networking #device pcn # AMD Am79C97x PCI 10/100(precedence over 'lnc') #device re # RealTek 8139C+/8169/8169S/8110S #device rl # RealTek 8129/8139 #device sf # Adaptec AIC-6915 (``Starfire'') #device sis # Silicon Integrated Systems SiS 900/SiS 7016 #device sk # SysKonnect SK-984x & SK-982x gigabit Ethernet #device ste # Sundance ST201 (D-Link DFE-550TX) #device stge # Sundance/Tamarack TC9021 gigabit Ethernet #device ti # Alteon Networks Tigon I/II gigabit Ethernet #device tl # Texas Instruments ThunderLAN #device tx # SMC EtherPower II (83c170 ``EPIC'') #device vge # VIA VT612x gigabit Ethernet #device vr # VIA Rhine, Rhine II #device wb # Winbond W89C840F #device xl # 3Com 3c90x (``Boomerang'', ``Cyclone'') # ISA Ethernet NICs. pccard NICs included. #device cs # Crystal Semiconductor CS89x0 NIC # 'device ed' requires 'device miibus' #device ed # NE[12]000, SMC Ultra, 3c503, DS8390 cards #device ex # Intel EtherExpress Pro/10 and Pro/10+ #device ep # Etherlink III based cards #device fe # Fujitsu MB8696x based cards #device ie # EtherExpress 8/16, 3C507, StarLAN 10 etc. #device lnc # NE2100, NE32-VL Lance Ethernet cards #device sn # SMC's 9000 series of Ethernet chips #device xe # Xircom pccard Ethernet # Wireless NIC cards #device wlan # 802.11 support #device wlan_wep # 802.11 WEP support #device wlan_ccmp # 802.11 CCMP support #device wlan_tkip # 802.11 TKIP support #device an # Aironet 4500/4800 802.11 wireless NICs. #device ath # Atheros pci/cardbus NIC's #device ath_hal # Atheros HAL (Hardware Access Layer) #device ath_rate_sample # SampleRate tx rate control for ath #device awi # BayStack 660 and others #device ral # Ralink Technology RT2500 wireless NICs. #device wi # WaveLAN/Intersil/Symbol 802.11 wireless NICs. #device wl # Older non 802.11 Wavelan wireless NIC. # Pseudo devices. device loop # Network loopback device random # Entropy device device ether # Ethernet support #device sl # Kernel SLIP device ppp # Kernel PPP device tun # Packet tunnel. device pty # Pseudo-ttys (telnet etc) device md # Memory "disks" device gif # IPv6 and IPv4 tunneling device faith # IPv6-to-IPv4 relaying (translation) # The `bpf' device enables the Berkeley Packet Filter. # Be aware of the administrative consequences of enabling this! # Note that 'bpf' is required for DHCP. device bpf # Berkeley packet filter # USB support #device uhci # UHCI PCI->USB interface #device ohci # OHCI PCI->USB interface #device ehci # EHCI PCI->USB interface (USB 2.0) #device usb # USB Bus (required) #device udbp # USB Double Bulk Pipe devices #device ugen # Generic #device uhid # "Human Interface Devices" #device ukbd # Keyboard #device ulpt # Printer #device umass # Disks/Mass storage - Requires scbus and da #device ums # Mouse #device ural # Ralink Technology RT2500USB wireless NICs #device urio # Diamond Rio 500 MP3 player #device uscanner # Scanners # USB Ethernet, requires miibus #device aue # ADMtek USB Ethernet #device axe # ASIX Electronics USB Ethernet #device cdce # Generic USB over Ethernet #device cue # CATC USB Ethernet #device kue # Kawasaki LSI USB Ethernet #device rue # RealTek RTL8150 USB Ethernet # FireWire support #device firewire # FireWire bus code #device sbp # SCSI over FireWire (Requires scbus and da) #device fwe # Ethernet over FireWire (non-standard!) -------------- next part -------------- # -------------------------------------------------------------------- # FREEBSD BUILD FreeBSD m1.npubs.com 6.3-RELEASE-p2 FreeBSD 6.3-RELEASE-p2 #0: Tue Jun 10 02:07:06 UTC 2008 nate@m1.npubs.com:/usr/obj/usr/src/sys/RACK1DBG i386 # -------------------------------------------------------------------- # PROCESS LIST db> ps pid ppid pgrp uid state wmesg wchan cmd 69677 69671 4145 65534 LLJ *vm page 0xcf2cc140 python 69674 69654 69486 0 RE rsync 69671 69670 4145 65534 SJ piperd 0xca44a660 python 69670 4145 4145 65534 SJ wait 0xcf673430 sh 69669 4684 4684 125 SJ select 0xc07b7ce4 smtpd 69668 69664 69664 0 S select 0xc07b7ce4 rsync 69664 5663 69664 0 Ls *vm page 0xcf2cc140 rsync 69654 69511 69486 0 S wait 0xca033c90 sh 69561 45988 45982 70 SJ sbwait 0xcabb4bc8 postgres 69557 4141 4141 125 SJ select 0xc07b7ce4 smtp 69541 4141 4141 125 SJ select 0xc07b7ce4 cleanup 69540 4141 4141 125 SJ select 0xc07b7ce4 smtpd 69535 4141 4141 125 SJ select 0xc07b7ce4 trivial-rewrite 69533 4141 4141 125 SJ select 0xc07b7ce4 anvil 69523 4141 4141 125 SJ select 0xc07b7ce4 proxymap 69511 69498 69486 0 S wait 0xc92e7430 sh 69510 4141 4141 125 SJ select 0xc07b7ce4 smtpd 69504 69495 69495 0 SJ sbwait 0xcf2b1638 python 69498 69486 69486 0 S wait 0xca67b648 sh 69495 69492 69495 0 SsJ wait 0xcf6aa218 sh 69492 1374 1374 0 SJ piperd 0xc91c8198 cron 69486 69484 69486 0 Ss wait 0xc92e4c90 sh 69484 691 691 0 S piperd 0xc901b990 cron 69455 1 69455 0 Ls *Giant 0xcf52b6c0 jail-measure 69434 1940 1940 125 SJ kqread 0xc9987a00 trivial-rewrite 69415 1940 1940 0 SJ kqread 0xc9b21700 local 69414 1940 1940 125 SJ kqread 0xca77a600 smtpd 69413 1940 1940 125 SJ kqread 0xc988ee00 smtp 69412 1940 1940 125 SJ kqread 0xcf87b900 cleanup 69400 4684 4684 0 SJ select 0xc07b7ce4 virtual 69379 45988 45982 70 SJ sbwait 0xd0b004d4 postgres 69374 45988 45982 70 SJ sbwait 0xc8f7579c postgres 69372 45988 45982 70 SJ sbwait 0xd08a2d2c postgres 69361 2260 2260 80 SJ accept 0xc97fe5ca httpd 69352 1940 1940 125 SJ lockf 0xc94529c0 smtpd 69335 45988 45982 70 SJ sbwait 0xcf2d5900 postgres 69327 1940 1940 125 SJ lockf 0xc9452740 bounce 69326 1940 1940 125 SJ kqread 0xd0080100 bounce 69276 1940 1940 125 SJ lockf 0xcf5158c0 smtp 69274 1940 1940 125 SJ kqread 0xca333800 smtp 69256 45988 45982 70 SJ sbwait 0xc8ce5638 postgres 69245 45988 45982 70 SJ sbwait 0xc9e23bc8 postgres 69240 45988 45982 70 SJ sbwait 0xd0ad379c postgres 69208 4684 4684 125 SJ select 0xc07b7ce4 smtp 69184 4684 4684 125 SJ select 0xc07b7ce4 smtpd 69183 4684 4684 125 SJ select 0xc07b7ce4 smtp 69182 4684 4684 125 SJ select 0xc07b7ce4 cleanup 69175 1940 1940 125 SJ kqread 0xc993ba00 smtpd 69174 4684 4684 125 SJ select 0xc07b7ce4 trivial-rewrite 69141 45988 45982 70 SJ sbwait 0xd0b030a8 postgres 68996 45988 45982 70 SJ sbwait 0xcf2b1e90 postgres 68790 2260 2260 80 SJ accept 0xc97fe5ca httpd 68693 1940 1940 125 SJ kqread 0xcbf11a00 anvil 68338 4557 4557 125 SJ select 0xc07b7ce4 imapd 68262 45988 45982 70 SJ sbwait 0xc9f54900 postgres 68199 4557 4557 125 SJ select 0xc07b7ce4 imapd 68189 45988 45982 70 SJ sbwait 0xd0afd0a8 postgres 67971 4684 4684 125 SJ select 0xc07b7ce4 proxymap 67135 4684 4684 125 SJ select 0xc07b7ce4 pickup 67085 4557 4557 125 SJ select 0xc07b7ce4 imapd 66421 45988 45982 70 SJ sbwait 0xd0ac9900 postgres 64620 1940 1940 125 SJ kqread 0xcbfa9400 pickup 63268 45988 45982 70 SJ sbwait 0xd0af9d2c postgres 62647 45988 45982 70 SJ sbwait 0xc9f23370 postgres 62453 45988 45982 70 SJ sbwait 0xc8d6379c postgres 62324 4141 4141 125 SJ select 0xc07b7ce4 pickup 59910 2260 2260 80 SJ accept 0xc97fe5ca httpd 58656 3815 3815 80 SJ accept 0xc986d892 httpd 58655 3815 3815 80 SJ accept 0xc986d892 httpd 58641 3815 3815 80 SJ accept 0xc986d892 httpd 58523 2260 2260 80 SJ accept 0xc97fe5ca httpd 58520 2260 2260 80 SJ accept 0xc97fe5ca httpd 58398 2260 2260 80 SJ accept 0xc97fe5ca httpd 57936 2260 2260 80 SJ accept 0xc97fe5ca httpd 55518 2260 2260 80 SJ accept 0xc97fe5ca httpd 49914 2260 2260 80 SJ accept 0xc97fe5ca httpd 46007 1 46007 0 SsJ nanslp 0xc076a96c cron 45999 1 45999 0 SsJ select 0xc07b7ce4 inetd 45993 45992 45982 70 S+J select 0xc07b7ce4 postgres 45992 45988 45982 70 S+J select 0xc07b7ce4 postgres 45991 45988 45982 70 S+J select 0xc07b7ce4 postgres 45989 45988 45982 70 S+J select 0xc07b7ce4 postgres 45988 1 45982 70 S+J select 0xc07b7ce4 postgres 45983 45958 45957 88 S+J (threaded) mysqld 100467 S kserel 0xcf638694 mysqld 100473 S kserel 0xcf638694 mysqld 100358 S kserel 0xcf638694 mysqld 100362 S kserel 0xcf638694 mysqld 100355 S select 0xc07b7ce4 mysqld 100948 S kserel 0xd092cc94 mysqld 100949 S sigwait 0xec446c14 mysqld 100938 S ksesigwa 0xcf6aabb0 mysqld 45958 1 45957 88 S+J wait 0xcf644000 sh 45942 1 45942 0 SsJ select 0xc07b7ce4 sshd 45876 1 45876 0 SsJ select 0xc07b7ce4 syslogd 33626 2260 2260 80 SJ accept 0xc97fe5ca httpd 23070 1336 1336 80 SJ lockf 0xd0401e40 httpd 23069 1336 1336 80 SJ kqread 0xca0be100 httpd 23057 1336 1336 80 SJ lockf 0xcf515680 httpd 22000 1336 1336 80 SJ lockf 0xc9c80380 httpd 21992 1336 1336 80 SJ lockf 0xcf515dc0 httpd 20343 1336 1336 80 SJ lockf 0xc91dd200 httpd 20342 1336 1336 80 SJ lockf 0xd0458ac0 httpd 20341 3815 3815 80 SJ accept 0xc986d892 httpd 20340 1336 1336 80 SJ lockf 0xd0401140 httpd 20339 3815 3815 80 SJ accept 0xc986d892 httpd 20337 1336 1336 80 SJ lockf 0xd0458800 httpd 20336 3815 3815 80 SJ accept 0xc986d892 httpd 20335 3815 3815 80 SJ accept 0xc986d892 httpd 20334 1336 1336 80 SJ lockf 0xcf4eb280 httpd 20333 3815 3815 80 SJ accept 0xc986d892 httpd 20324 20323 20323 0 SJ piperd 0xd02164c8 sockin 20323 4456 20323 0 SsJ wait 0xd092e430 sh 78897 0 0 0 SL ipmireq 0xcf93bec0 [ipmi0: kcs] 65042 64222 64222 70 SJ sbwait 0xcf0f14d4 postgres 64780 64222 64222 70 SJ sbwait 0xcabb4900 postgres 64778 64222 64222 70 SJ sbwait 0xd0af10a8 postgres 64777 64222 64222 70 SJ sbwait 0xc8f96370 postgres 64769 64222 64222 70 SJ sbwait 0xca43e20c postgres 64768 64222 64222 70 SJ sbwait 0xc9a26900 postgres 64767 64222 64222 70 SJ sbwait 0xd0afd4d4 postgres 64742 64222 64222 70 SJ sbwait 0xd0afb638 postgres 64666 64222 64222 70 SJ sbwait 0xccb3579c postgres 64665 64222 64222 70 SJ sbwait 0xd0afc20c postgres 64664 64222 64222 70 SJ sbwait 0xc9e4979c postgres 64663 64222 64222 70 SJ sbwait 0xd05c0e90 postgres 64662 64222 64222 70 SJ sbwait 0xd0ad3d2c postgres 64661 64222 64222 70 SJ sbwait 0xca39a79c postgres 64484 64222 64222 70 SJ sbwait 0xc8d6a370 postgres 64414 64222 64222 70 SJ sbwait 0xc9f2379c postgres 64361 64222 64222 70 SJ sbwait 0xd08a1370 postgres 64349 64222 64222 70 SJ sbwait 0xd05c8370 postgres 64348 64222 64222 70 SJ sbwait 0xcc9b70a8 postgres 64332 64222 64222 70 SJ sbwait 0xd0509a64 postgres 64331 64222 64222 70 SJ sbwait 0xc9a2679c postgres 64227 64226 64222 70 SJ select 0xc07b7ce4 postgres 64226 64222 64222 70 SJ select 0xc07b7ce4 postgres 64225 64222 64222 70 SJ select 0xc07b7ce4 postgres 64223 64222 64222 70 SJ select 0xc07b7ce4 postgres 64222 1 64222 70 SsJ select 0xc07b7ce4 postgres 43040 1114 43040 70 SsJ sbwait 0xca50b370 postgres 75907 4167 4167 1003 RJ CPU 0 stunnel 46911 1 46911 0 SsJ nanslp 0xc076a96c cron 46904 1 46904 0 SsJ select 0xc07b7ce4 sshd 46880 1 46880 0 SsJ (threaded) pdns_server 100752 S kserel 0xcf45e0f4 pdns_server 100748 S kserel 0xcf45e0f4 pdns_server 100695 S kserel 0xcf45e0f4 pdns_server 100171 S kserel 0xcf45e0f4 pdns_server 100464 S sbwait 0xcf7be638 pdns_server 100572 S sbwait 0xca872370 pdns_server 100836 S accept 0xcac13e22 pdns_server 100834 S select 0xc07b7ce4 pdns_server 100784 S ksesigwa 0xcf564bb0 pdns_server 46845 1 46845 389 SsJ (threaded) slapd 100729 S kserel 0xcf638454 slapd 100750 S kserel 0xcf638454 slapd 100747 S select 0xc07b7ce4 slapd 100630 S kserel 0xcf638454 slapd 100512 S kserel 0xcf638454 slapd 100733 S ksesigwa 0xcf6ab998 slapd 46825 1 46825 0 SsJ select 0xc07b7ce4 syslogd 68366 5772 5772 70 SJ sbwait 0xcabb379c postgres 57454 5772 5772 70 SJ sbwait 0xca4124d4 postgres 94638 5772 5772 70 SJ sbwait 0xc9f22bc8 postgres 94626 5772 5772 70 SJ sbwait 0xd052ee90 postgres 85402 5800 5800 80 SJ accept 0xc9cf4892 httpd 85398 5772 5772 70 SJ sbwait 0xceb45e90 postgres 85396 5772 5772 70 SJ sbwait 0xca4b4638 postgres 71130 1114 71130 70 SsJ sbwait 0xc9eeaa64 postgres 71129 1114 71129 70 SsJ sbwait 0xcf2b0bc8 postgres 71128 1114 71128 70 SsJ sbwait 0xc9e484d4 postgres 71127 1114 71127 70 SsJ sbwait 0xcf7bd20c postgres 71126 1114 71126 70 SsJ sbwait 0xc8f76900 postgres 71115 1114 71115 70 SsJ sbwait 0xca50ce90 postgres 71081 71080 71080 89 SJ (threaded) java 100360 S kserel 0xcf5dbe74 java 100255 S kserel 0xcf5dbe74 java 100722 S select 0xc07b7ce4 java 100746 S kserel 0xcf5dbe74 java 100505 S kserel 0xcf5dbe74 java 100409 S sbwait 0xceb88bc8 java 100718 S sbwait 0xca609900 java 100163 S sbwait 0xcf7be79c java 100518 S sbwait 0xca6094d4 java 100778 S sbwait 0xca609370 java 100258 S sbwait 0xc93454d4 java 100506 S ksesigwa 0xcf6ad780 java 71080 1 71080 0 SsJ select 0xc07b7ce4 udb-server 18181 2388 2388 80 SJ accept 0xc986d19e httpd 16803 2388 2388 80 SJ accept 0xc986d19e httpd 16802 2388 2388 80 SJ accept 0xc986d19e httpd 16801 2388 2388 80 SJ accept 0xc986d19e httpd 16795 2388 2388 80 SJ accept 0xc986d19e httpd 7788 5772 5772 70 SJ sbwait 0xc9e240a8 postgres 7723 5772 5772 70 SJ sbwait 0xcabb620c postgres 7705 5772 5772 70 SJ sbwait 0xca8dc370 postgres 7703 5800 5800 80 SJ accept 0xc9cf4892 httpd 7700 5800 5800 80 SJ accept 0xc9cf4892 httpd 7699 5772 5772 70 SJ sbwait 0xca39aa64 postgres 7696 5772 5772 70 SJ sbwait 0xc9cf3638 postgres 7694 5800 5800 80 SJ accept 0xc9cf4892 httpd 6376 4141 4141 125 SJ select 0xc07b7ce4 tlsmgr 5969 3438 3438 80 SJ lockf 0xce4036c0 httpd 5968 3438 3438 80 SJ lockf 0xcaf5dcc0 httpd 5966 3438 3438 80 SJ select 0xc07b7ce4 httpd 5962 3438 3438 80 SJ lockf 0xc94524c0 httpd 5854 5772 5772 70 SJ sbwait 0xca314638 postgres 5852 5800 5800 80 SJ accept 0xc9cf4892 httpd 5851 5800 5800 80 SJ accept 0xc9cf4892 httpd 5850 5800 5800 80 SJ accept 0xc9cf4892 httpd 5849 5800 5800 80 SJ accept 0xc9cf4892 httpd 5848 5800 5800 80 SJ accept 0xc9cf4892 httpd 5827 5826 5826 1020 SJ (threaded) java 100390 S kserel 0xc8e0b034 java 100475 S kserel 0xc8e0b034 java 100418 S kserel 0xc8e0b034 java 100389 S select 0xc07b7ce4 java 100180 S kserel 0xc8e0b034 java 100400 S sbwait 0xc9e49a64 java 100812 S sbwait 0xc9f5420c java 100382 S sbwait 0xc9f54e90 java 100394 S sbwait 0xc9cf3900 java 100388 S sbwait 0xca314370 java 100398 S sbwait 0xca02f370 java 100386 S ksesigwa 0xc8e0d138 java 5826 1 5826 0 SsJ select 0xc07b7ce4 udb-server 5819 1 5819 0 SsJ nanslp 0xc076a96c cron 5812 1 5812 0 SsJ select 0xc07b7ce4 sshd 5809 5805 5800 0 SJ piperd 0xc9cfb7f8 cronolog 5808 5803 5800 0 SJ piperd 0xc9e44b28 cronolog 5805 5800 5800 0 SJ wait 0xc969aa78 sh 5803 5800 5800 0 SJ wait 0xca033648 sh 5800 1 5800 0 SsJ nanslp 0xc076a96c httpd 5788 5787 5772 70 SJ select 0xc07b7ce4 postgres 5787 5772 5772 70 SJ select 0xc07b7ce4 postgres 5786 5772 5772 70 SJ select 0xc07b7ce4 postgres 5772 1 5772 70 SsJ select 0xc07b7ce4 postgres 5713 1 5713 0 SsJ select 0xc07b7ce4 syslogd 5704 1 5704 0 Ss+ ttyin 0xc8a61410 getty 5703 1 5703 0 Ss+ ttyin 0xc8a69c10 getty 5702 1 5702 0 Ss+ ttyin 0xc8a6c010 getty 5701 1 5701 0 Ss+ ttyin 0xc8a6b010 getty 5700 1 5700 0 Ss+ ttyin 0xc8a6a410 getty 5699 1 5699 0 Ss+ ttyin 0xc8a6a810 getty 5698 1 5698 0 Ss+ ttyin 0xc8a60810 getty 5697 1 5697 0 Ss+ ttyin 0xc8a6c410 getty 5696 1 5696 0 Ss+ ttyin 0xc8a6b410 getty 5675 1 5675 0 Ss select 0xc07b7ce4 bsnmpd 5663 1 5663 0 Ss select 0xc07b7ce4 inetd 5443 5442 5442 1020 SJ (threaded) java 100223 S kserel 0xc998b874 java 100095 S kserel 0xc998b874 java 100385 S sbwait 0xd0509978 java 100414 S sbwait 0xd089e814 java 100699 S kserel 0xc998b874 java 100187 S kserel 0xc998b874 java 100229 S select 0xc07b7ce4 java 100736 S sbwait 0xd025379c java 100157 S sbwait 0xc9644f08 java 100338 S ksesigwa 0xc998abb0 java 5442 1 5442 0 SsJ select 0xc07b7ce4 udb-server 5435 1 5435 0 SsJ nanslp 0xc076a96c cron 5419 1 5419 0 SsJ select 0xc07b7ce4 sshd 5265 1 5265 0 SsJ select 0xc07b7ce4 syslogd 4960 1 4960 0 SsJ nanslp 0xc076a96c cron 4953 1 4953 0 SsJ select 0xc07b7ce4 sshd 4906 1 4906 53 SsJ select 0xc07b7ce4 named 4891 1 4891 0 SsJ select 0xc07b7ce4 syslogd 4834 4684 4684 125 SJ select 0xc07b7ce4 anvil 4789 4684 4684 125 SJ select 0xc07b7ce4 tlsmgr 4709 1 4709 0 SsJ nanslp 0xc076a96c cron 4699 1 4699 0 SsJ accept 0xc9dfae22 saslauthd1 4691 4684 4684 125 SJ select 0xc07b7ce4 qmgr 4684 1 4684 0 SsJ select 0xc07b7ce4 master 4598 1 4598 0 SsJ select 0xc07b7ce4 rpc.dracd 4585 4584 4585 0 S+J select 0xc07b7ce4 couriertcpd 4584 1 4584 0 S+J piperd 0xc9cb6b28 courierlogger 4581 4532 4531 0 S+J select 0xc07b7ce4 authdaemond 4579 4532 4531 0 S+J select 0xc07b7ce4 authdaemond 4578 4532 4531 0 S+J select 0xc07b7ce4 authdaemond 4577 4532 4531 0 S+J select 0xc07b7ce4 authdaemond 4576 4532 4531 0 S+J select 0xc07b7ce4 authdaemond 4569 4568 4569 0 S+J select 0xc07b7ce4 couriertcpd 4568 1 4568 0 S+J piperd 0xc8e04330 courierlogger 4557 4556 4557 0 S+J select 0xc07b7ce4 couriertcpd 4556 1 4556 0 S+J piperd 0xc8f6e000 courierlogger 4544 4543 4544 0 S+J select 0xc07b7ce4 couriertcpd 4543 1 4543 0 S+J piperd 0xc9cb64c8 courierlogger 4532 4531 4531 0 S+J select 0xc07b7ce4 authdaemond 4531 1 4531 0 S+J piperd 0xc94917f8 courierlogger 4517 1 4517 0 SsJ select 0xc07b7ce4 sshd 4500 1 4500 0 SsJ select 0xc07b7ce4 bsnmpd 4467 1 4467 0 SsJ select 0xc07b7ce4 rpcbind 4456 1 4456 0 SsJ select 0xc07b7ce4 syslogd 4180 1 4180 0 SsJ nanslp 0xc076a96c cron 4172 1 4172 0 SsJ select 0xc07b7ce4 inetd 4167 1 4167 1003 SsJ (threaded) stunnel 100735 S kserel 0xc8e0b454 stunnel 100532 S kserel 0xc8e0b454 stunnel 100189 S kserel 0xc8e0b454 stunnel 100749 S kserel 0xc8e0b454 stunnel 100412 S select 0xc07b7ce4 stunnel 100080 S ksesigwa 0xc8e05780 stunnel 4160 1 4160 0 SsJ accept 0xc986d302 saslauthd1 4149 4141 4141 125 SJ select 0xc07b7ce4 qmgr 4145 1 4145 65534 SsJ (threaded) proxsmtpd 100762 S kserel 0xc8e0b9f4 proxsmtpd 100383 S kserel 0xc8e0b9f4 proxsmtpd 100761 S kserel 0xc8e0b9f4 proxsmtpd 100765 S kserel 0xc8e0b9f4 proxsmtpd 100740 S select 0xc07b7ce4 proxsmtpd 100783 S accept 0xc9a97892 proxsmtpd 100516 S ksesigwa 0xc9b99dc8 proxsmtpd 4141 1 4141 0 SsJ select 0xc07b7ce4 master 4077 1 4077 0 SsJ select 0xc07b7ce4 sshd 4021 1 4021 0 SsJ select 0xc07b7ce4 syslogd 3844 1 3844 0 SsJ select 0xc07b7ce4 inetd 3830 1 3830 0 SsJ nanslp 0xc076a96c cron 3823 1 3823 0 SsJ select 0xc07b7ce4 sshd 3815 1 3815 0 SsJ nanslp 0xc076a96c httpd 3680 1 3680 0 SsJ select 0xc07b7ce4 syslogd 3538 3534 3438 0 SJ piperd 0xc8f69660 cronolog 3537 3531 3438 0 SJ piperd 0xc9311cc0 cronolog 3536 3532 3438 0 SJ piperd 0xc9491b28 cronolog 3535 3438 3438 80 SJ lockf 0xcdedd2c0 httpd 3534 3438 3438 0 SJ wait 0xc8d95a78 sh 3532 3438 3438 0 SJ wait 0xc969a648 sh 3531 3438 3438 0 SJ wait 0xc9828430 sh 3486 3450 3449 88 S+J (threaded) mysqld 100165 S kserel 0xc894d0f4 mysqld 100256 S kserel 0xc894d0f4 mysqld 100176 S kserel 0xc894d0f4 mysqld 100181 S select 0xc07b7ce4 mysqld 100166 S kserel 0xc894d0f4 mysqld 100252 S kserel 0xc894ea54 mysqld 100253 S sigwait 0xec476c14 mysqld 100172 S ksesigwa 0xc8a4d780 mysqld 3479 1 3479 0 SsJ nanslp 0xc076a96c cron 3460 1 3460 0 SsJ select 0xc07b7ce4 inetd 3450 1 3449 88 S+J wait 0xc92e4000 sh 3448 3438 3438 80 SJ select 0xc07b7ce4 httpd 3438 1 3438 0 SsJ select 0xc07b7ce4 httpd 3326 1 3326 0 SsJ select 0xc07b7ce4 sshd 3265 1 3265 0 SsJ select 0xc07b7ce4 syslogd 2983 1 2983 0 SsJ nanslp 0xc076a96c cron 2976 1 2976 0 SsJ select 0xc07b7ce4 sshd 2968 1 2968 0 SsJ (threaded) httpauthd 100179 S kserel 0xc93a0454 httpauthd 100741 S kserel 0xc93a0454 httpauthd 100760 S kserel 0xc93a0454 httpauthd 100730 S kserel 0xc93a0454 httpauthd 100185 S sbwait 0xca02fd2c httpauthd 100593 S sbwait 0xceb5c79c httpauthd 100404 S sbwait 0xd05c8a64 httpauthd 100191 S sbwait 0xca43fd2c httpauthd 100393 S sbwait 0xca8dc20c httpauthd 100417 S sbwait 0xca41c638 httpauthd 100359 S sbwait 0xd05c04d4 httpauthd 100468 S sbwait 0xc9f224d4 httpauthd 100273 S sbwait 0xca50b638 httpauthd 100408 S sbwait 0xca50b0a8 httpauthd 100357 S sbwait 0xcabb7a64 httpauthd 100254 S sbwait 0xcf2b3900 httpauthd 100173 S sbwait 0xcf0f2900 httpauthd 100251 S sbwait 0xca50cbc8 httpauthd 100226 S sbwait 0xd0af00a8 httpauthd 100221 S sbwait 0xc9eeabc8 httpauthd 100391 S sbwait 0xca41c900 httpauthd 100162 S sbwait 0xcca28a64 httpauthd 100352 S sbwait 0xd0b01900 httpauthd 100421 S sbwait 0xc97a64d4 httpauthd 100411 S sbwait 0xd0b0120c httpauthd 100479 S accept 0xc9800892 httpauthd 100169 S sbwait 0xca41c370 httpauthd 100396 S sbwait 0xca4110a8 httpauthd 100265 S sbwait 0xcf2d5d2c httpauthd 100402 S sbwait 0xca411e90 httpauthd 100781 S sbwait 0xcf0f2bc8 httpauthd 100183 S sbwait 0xc95b7bc8 httpauthd 100356 S sbwait 0xd0afc638 httpauthd 100502 S sbwait 0xc9df8370 httpauthd 100751 S sbwait 0xc97ff0a8 httpauthd 100500 S sbwait 0xc900b0a8 httpauthd 100513 S sbwait 0xc9a794d4 httpauthd 100170 S sbwait 0xcabaf20c httpauthd 100476 S sbwait 0xc8d6a900 httpauthd 100851 S sbwait 0xca39b79c httpauthd 100274 S sbwait 0xceb44900 httpauthd 100510 S sbwait 0xca8dc79c httpauthd 100936 S sbwait 0xc986d4d4 httpauthd 100419 S sbwait 0xcabaf900 httpauthd 100711 S sbwait 0xd05aca64 httpauthd 100533 S sbwait 0xcefa7a64 httpauthd 100250 S sbwait 0xd0aef0a8 httpauthd 100422 S sbwait 0xcabb3a64 httpauthd 100405 S sbwait 0xc91dfd2c httpauthd 100514 S sbwait 0xd0ad34d4 httpauthd 100482 S sbwait 0xc9f23bc8 httpauthd 100470 S sbwait 0xc9e4879c httpauthd 100410 S sbwait 0xc96af4d4 httpauthd 100614 S sbwait 0xca50c20c httpauthd 100801 S sbwait 0xc9eea370 httpauthd 100175 S sbwait 0xd0ac9638 httpauthd 100737 S sbwait 0xd089ee90 httpauthd 100403 S ksesigwa 0xc93a1bb0 httpauthd 2920 1 2920 389 SsJ (threaded) slapd 100186 S kserel 0xc8ddedb4 slapd 100497 S kserel 0xc8ddedb4 slapd 100763 S kserel 0xc8ddedb4 slapd 100469 S kserel 0xc8ddedb4 slapd 100471 S select 0xc07b7ce4 slapd 100136 S ksesigwa 0xc8e0d998 slapd 2900 1 2900 0 SsJ select 0xc07b7ce4 syslogd 2668 1 2668 0 SsJ select 0xc07b7ce4 slapd 2661 1 2661 0 SsJ nanslp 0xc076a96c cron 2654 1 2654 0 SsJ select 0xc07b7ce4 sshd 2599 1 2599 0 SsJ select 0xc07b7ce4 syslogd 2547 2388 2388 80 SJ accept 0xc986d19e httpd 2546 2388 2388 80 SJ accept 0xc986d19e httpd 2545 2388 2388 80 SJ accept 0xc986d19e httpd 2544 2388 2388 80 SJ accept 0xc986d19e httpd 2543 2388 2388 80 SJ accept 0xc986d19e httpd 2403 1 2403 0 SsJ nanslp 0xc076a96c cron 2396 1 2396 0 SsJ select 0xc07b7ce4 sshd 2388 1 2388 0 SsJ nanslp 0xc076a96c httpd 2289 1 2289 0 SsJ select 0xc07b7ce4 inetd 2275 1 2275 0 SsJ nanslp 0xc076a96c cron 2268 1 2268 0 SsJ select 0xc07b7ce4 sshd 2260 1 2260 0 SsJ nanslp 0xc076a96c httpd 2206 1 2206 0 SsJ select 0xc07b7ce4 syslogd 2006 1951 1950 88 S+J (threaded) mysqld 100161 S kserel 0xc93a0514 mysqld 100038 S kserel 0xc93a0514 mysqld 100167 S kserel 0xc93a0514 mysqld 100225 S select 0xc07b7ce4 mysqld 100091 S kserel 0xc93a0514 mysqld 100222 S sigwait 0xec3e1c14 mysqld 100219 S ksesigwa 0xc93a1780 mysqld 1951 1 1950 88 S+J wait 0xc939f430 sh 1949 1940 1940 125 SJ kqread 0xc97a9b00 qmgr 1940 1 1940 0 SsJ kqread 0xc969eb00 master 1800 1795 1800 70 SsJ select 0xc07b7ce4 postgres 1799 1795 1799 70 SsJ nanslp 0xc076a96c postgres 1798 1795 1798 70 SsJ nanslp 0xc076a96c postgres 1797 1795 1797 70 SsJ nanslp 0xc076a96c postgres 1795 1 1791 70 S+J select 0xc07b7ce4 postgres 1765 1760 1760 0 SJ lockf 0xcca4ed00 saslauthd 1763 1760 1760 0 SJ lockf 0xca55b240 saslauthd 1762 1760 1760 0 SJ lockf 0xcf50de40 saslauthd 1761 1760 1760 0 SJ accept 0xc8f76cbe saslauthd 1760 1 1760 0 SsJ lockf 0xcff97c00 saslauthd 1731 1 1731 0 SsJ select 0xc07b7ce4 syslogd 1374 1 1374 0 SsJ nanslp 0xc076a96c cron 1364 1 1364 0 SsJ select 0xc07b7ce4 sshd 1336 1 1336 0 SsJ nanslp 0xc076a96c httpd 1117 1114 1117 70 SsJ select 0xc07b7ce4 postgres 1116 1114 1116 70 SsJ select 0xc07b7ce4 postgres 1114 1 1114 70 SsJ select 0xc07b7ce4 postgres 1057 1 1057 0 SsJ select 0xc07b7ce4 syslogd 691 1 691 0 Ss nanslp 0xc076a96c cron 685 1 685 25 Ss pause 0xc8cf0894 sendmail 681 1 681 0 Ss select 0xc07b7ce4 sendmail 675 1 675 0 Ss select 0xc07b7ce4 sshd 655 1 654 0 S nanslp 0xc076a96c python 615 1 615 0 Ss select 0xc07b7ce4 ntpd 525 1 525 0 Ss select 0xc07b7ce4 syslogd 470 1 470 0 Ss select 0xc07b7ce4 devd 41 0 0 0 SL - 0xe9c77d04 [schedcpu] 40 0 0 0 SL sdflush 0xc07c9294 [softdepflush] 39 0 0 0 SL syncer 0xc076a6dc [syncer] 38 0 0 0 SL vlruwt 0xc8a07c90 [vnlru] 37 0 0 0 SL psleep 0xc07b8260 [bufdaemon] 36 0 0 0 SL pgzero 0xc07ca264 [pagezero] 35 0 0 0 SL psleep 0xc07c9d6c [vmdaemon] 34 0 0 0 SL psleep 0xc07c9d28 [pagedaemon] 33 0 0 0 SL - 0xc8bee580 [dummynet] 32 0 0 0 WL [swi0: sio] 31 0 0 0 WL [irq1: atkbd0] 30 0 0 0 WL [irq19: atapci1] 29 0 0 0 WL [irq15: ata1] 28 0 0 0 WL [irq14: ata0] 27 0 0 0 LL *Giant 0xcf52b6c0 [irq17: em1] 26 0 0 0 LL *Giant 0xcf52b6c0 [irq16: em0] 25 0 0 0 WL [irq9: acpi0] 24 0 0 0 SL - 0xc89a2c00 [thread taskq] 23 0 0 0 SL - 0xc89a2c80 [acpi_task_2] 22 0 0 0 SL - 0xc89a2c80 [acpi_task_1] 9 0 0 0 SL - 0xc89a2c80 [acpi_task_0] 21 0 0 0 WL [swi6: Giant taskq] 20 0 0 0 WL [swi6: task queue] 19 0 0 0 WL [swi2: cambio] 8 0 0 0 SL ccb_scan 0xc0752ae4 [xpt_thrd] 7 0 0 0 SL - 0xc89ef180 [kqueue taskq] 18 0 0 0 WL [swi5: +] 17 0 0 0 SL - 0xc0767320 [yarrow] 6 0 0 0 SL crypto_r 0xc07c8ec4 [crypto returns] 5 0 0 0 SL crypto_w 0xc07c8e84 [crypto] 4 0 0 0 SL - 0xc0767e28 [g_down] 3 0 0 0 SL - 0xc0767e24 [g_up] 2 0 0 0 SL - 0xc0767e1c [g_event] 16 0 0 0 WL [swi3: vm] 15 0 0 0 LL *Giant 0xcf52b6c0 [swi4: clock sio] 14 0 0 0 WL [swi1: net] 13 0 0 0 RL [idle: cpu0] 12 0 0 0 RL CPU 1 [idle: cpu1] 11 0 0 0 RL CPU 2 [idle: cpu2] 10 0 0 0 RL CPU 3 [idle: cpu3] 1 0 1 0 SLs wait 0xc894f000 [init] 0 0 0 0 WLs [swapper] # -------------------------------------------------------------------- # CPU INFO db> show pcpu cpuid = 1 curthread = 0xc894a900: pid 12 "idle: cpu1" curpcb = 0xe81f2d90 fpcurthread = none idlethread = 0xc894a900: pid 12 "idle: cpu1" APIC ID = 1 currentldt = 0x50 spin locks held: db> show allpcpu Current CPU: 1 cpuid = 0 curthread = 0xcf2ba780: pid 75907 "stunnel" curpcb = 0xed1b1d90 fpcurthread = none idlethread = 0xc894aa80: pid 13 "idle: cpu0" APIC ID = 0 currentldt = 0x58 spin locks held: cpuid = 1 curthread = 0xc894a900: pid 12 "idle: cpu1" curpcb = 0xe81f2d90 fpcurthread = none idlethread = 0xc894a900: pid 12 "idle: cpu1" APIC ID = 1 currentldt = 0x50 spin locks held: cpuid = 2 curthread = 0xc894a780: pid 11 "idle: cpu2" curpcb = 0xe81efd90 fpcurthread = none idlethread = 0xc894a780: pid 11 "idle: cpu2" APIC ID = 2 currentldt = 0x50 spin locks held: cpuid = 3 curthread = 0xc894a600: pid 10 "idle: cpu3" curpcb = 0xe81ecd90 fpcurthread = none idlethread = 0xc894a600: pid 10 "idle: cpu3" APIC ID = 3 currentldt = 0x50 spin locks held: # -------------------------------------------------------------------- # LOCK INFO db> show locks db> show alllocks Process 69677 (python) thread 0xca239900 (100432) exclusive sleep mutex vm object (standard object) r = 0 (0xce1f3420) locked @ /usr/src/sys/vm/vm_fault.c:688 exclusive sx user map r = 0 (0xcf67f4e4) locked @ /usr/src/sys/vm/vm_map.c:3105 Process 69674 (rsync) thread 0xc8cf5a80 (100132) exclusive sleep mutex pmap r = 0 (0xcf5b87b0) locked @ /usr/src/sys/i386/i386/pmap.c:2749 exclusive sleep mutex vm page queue mutex r = 0 (0xc07c9ce0) locked @ /usr/src/sys/i386/i386/pmap.c:2748 Process 69664 (rsync) thread 0xca239480 (100485) exclusive sleep mutex vm object (standard object) r = 0 (0xcc3ad528) locked @ /usr/src/sys/kern/vfs_bio.c:1480 exclusive sleep mutex Giant r = 0 (0xc076a080) locked @ /usr/src/sys/kern/vfs_vnops.c:492 db> show lockedvnods Locked vnodes 0xccc2e414: tag ufs, type VREG usecount 1, writecount 0, refcount 5965 mountedhere 0 flags () v_object 0xcc3ad528 ref 0 pages 81936 lock type ufs: SHARED (count 1)#0 0xc04fa690 at lockmgr+0x160 #1 0xc0649d8a at ffs_lock+0x76 #2 0xc06bf8bb at VOP_LOCK_APV+0x87 #3 0xc056a59c at vn_lock+0xac #4 0xc0569c72 at vn_read+0x132 #5 0xc052c741 at dofileread+0x85 #6 0xc052c5da at kern_readv+0x36 #7 0xc052c505 at read+0x45 #8 0xc06acec3 at syscall+0x25b #9 0xc06984df at Xint0x80_syscall+0x1f ino 13262597, on dev ad8s1e # -------------------------------------------------------------------- # PROCESS STACK TRACES # PYTHON db> trace 69677 Tracing pid 69677 tid 100432 td 0xca239900 sched_switch(ca239900,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 turnstile_wait(c07c9ce0,c8cf5a80,0,c07c9ce0,2,c06e8e2b,233) at turnstile_wait+0x35a _mtx_lock_sleep(c07c9ce0,ca239900,0,c07075e5,15a) at _mtx_lock_sleep+0x14c _mtx_lock_flags(c07c9ce0,0,c07075e5,15a,c06eba6d,...) at _mtx_lock_flags+0xa2 vm_fault(cf67f4a0,8147000,2,8,ca239900,...) at vm_fault+0x311 trap_pfault(ec755d38,1,8147f20,8147f20,0,...) at trap_pfault+0xce trap(812003b,811003b,bfbf003b,812511c,8147f20,...) at trap+0x1eb calltrap() at calltrap+0x5 --- trap 0xc, eip = 0x8073f3c, esp = 0xbfbfda00, ebp = 0xbfbfda28 --- # RSYNC db> trace 69664 Tracing pid 69664 tid 100485 td 0xca239480 sched_switch(ca239480,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 turnstile_wait(c07c9ce0,c8cf5a80,0,c07c9ce0,2,c06e8e2b,233) at turnstile_wait+0x35a _mtx_lock_sleep(c07c9ce0,ca239480,0,c06f2142,5c9) at _mtx_lock_sleep+0x14c _mtx_lock_flags(c07c9ce0,0,c06f2142,5c9,cc3ad528,...) at _mtx_lock_flags+0xa2 vfs_vmio_release(dcc09838) at vfs_vmio_release+0x35 getnewbuf(0,0,4000,4000) at getnewbuf+0x276 getblk(ccc2e414,236e,0,4000,0,...) at getblk+0x3b3 cluster_read(ccc2e414,14002000,0,236e,0,...) at cluster_read+0xde ffs_read(ec74cbf4) at ffs_read+0x2d6 VOP_READ_APV(c0741260,ec74cbf4) at VOP_READ_APV+0x9b vn_read(cb7676c0,ec74ccbc,c94d6a00,0,ca239480) at vn_read+0x1fb dofileread(ca239480,5,cb7676c0,ec74ccbc,ffffffff,...) at dofileread+0x85 kern_readv(ca239480,5,ec74ccbc,85063e8,42d10,...) at kern_readv+0x36 read(ca239480,ec74cd04) at read+0x45 syscall(3b,3b,3b,0,430f8,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (3, FreeBSD ELF32, read), eip = 0x28185a63, esp = 0xbfbf987c, ebp = 0xbfbf98c8 --- # RSYNC db> trace 69674 Tracing pid 69674 tid 100132 td 0xc8cf5a80 sched_switch(c8cf5a80,0,2) at sched_switch+0x177 mi_switch(2,0,c076a040,0,c06eba6d,...) at mi_switch+0x270 critical_exit(c5e1bd38,e9f25c30,c0698a70,0,e9f20008,...) at critical_exit+0x8b lapic_handle_timer(0) at lapic_handle_timer+0xe1 Xtimerint(c5e1bd38,cf5b87b0,28120000,e9f25c48) at Xtimerint+0x30 pmap_remove_pages(cf5b87b0,0,bfc00000) at pmap_remove_pages+0x1c2 vmspace_exit(c8cf5a80,c076a240,0,c06e6c7d,125,...) at vmspace_exit+0x97 exit1(c8cf5a80,0,e9f25d30,c06acec3,c8cf5a80,...) at exit1+0x496 exit1(c8cf5a80,e9f25d04) at exit1 syscall(3b,3b,3b,8088eda,407,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (1, FreeBSD ELF32, sys_exit), eip = 0x2818520b, esp = 0xbfbfd82c, ebp = 0xbfbfd848 --- # JAIL-MEASURE db> trace 69455 Tracing pid 69455 tid 100839 td 0xc9a5d780 sched_switch(c9a5d780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 turnstile_wait(c076a080,ca239480,0,c076a080,2,c06e8e2b,233) at turnstile_wait+0x35a _mtx_lock_sleep(c076a080,c9a5d780,0,c06f3435,c1) at _mtx_lock_sleep+0x14c _mtx_lock_flags(c076a080,0,c06f3435,c1) at _mtx_lock_flags+0xa2 namei(ece75ba0) at namei+0x227 kern_lstat(c9a5d780,80637a8,0,ece75c74) at kern_lstat+0x47 lstat(c9a5d780,ece75d04) at lstat+0x1b syscall(805003b,805003b,bfbf003b,8063748,8063700,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (190, FreeBSD ELF32, lstat), eip = 0x28159b47, esp = 0xbfbfeafc, ebp = 0xbfbfeb98 --- # EM1 db> trace 27 Tracing pid 27 tid 100017 td 0xc894c900 sched_switch(c894c900,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 turnstile_wait(c076a080,ca239480,0,c076a080,2,c06e8e2b,233) at turnstile_wait+0x35a _mtx_lock_sleep(c076a080,c894c900,0,c06e6f85,2a9) at _mtx_lock_sleep+0x14c _mtx_lock_flags(c076a080,0,c06e6f85,2a9) at _mtx_lock_flags+0xa2 ithread_execute_handlers(c89a3648,c89a0480) at ithread_execute_handlers+0xde ithread_loop(c8a2cae0,e8210d38,c8a2cae0,c04f290c,0,...) at ithread_loop+0x67 fork_exit(c04f290c,c8a2cae0,e8210d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe8210d6c, ebp = 0 --- # EM0 db> trace 26 Tracing pid 26 tid 100018 td 0xc894c780 sched_switch(c894c780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 turnstile_wait(c076a080,ca239480,0,c076a080,2,c06e8e2b,233) at turnstile_wait+0x35a _mtx_lock_sleep(c076a080,c894c780,0,c06e6f85,2a9) at _mtx_lock_sleep+0x14c _mtx_lock_flags(c076a080,0,c06e6f85,2a9) at _mtx_lock_flags+0xa2 ithread_execute_handlers(c89a3860,c89a0500) at ithread_execute_handlers+0xde ithread_loop(c8a2cc40,e820dd38,c8a2cc40,c04f290c,0,...) at ithread_loop+0x67 fork_exit(c04f290c,c8a2cc40,e820dd38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe820dd6c, ebp = 0 --- # -------------------------------------------------------------------- # OTHER INFO db> x cpu_idle_hlt cpu_idle_hlt: 0
On Sunday 15 June 2008 07:23:19 am Stef Walter wrote:> I've been trying to track down a deadlock on some newish production > servers running FreeBSD 6.3-RELEASE-p2. The deadlock occurs on a > specific (although mundane) hardware configuration, and each of several > servers running this hardware deadlock about once per week. > > Although I suspect that this is not hardware related, from a (naive) > perusal of the attached stack traces. > > Forgive me if my interpretation of this is all wrong, but I'm pretty > desperate for help. So here's my basic understanding of the deadlock: > > These processes seem to be waiting on the page queue mutex: > sendmail (in vm_mmap > vm_map_find > vm_map_insert > vm_map_pmap_enter) > bsnmpd (in malloc, uma_large_malloc > page_alloc > kmem_malloc) > httpd (in trap > trap_pfault > vm_fault) > [g_up] (in g_vfs_done > bufdone) > > The page queue mutex is held by rsync process: > rsync (in trap > trap_pfault > vm_fault > pmap_enter) > > Rsync kernel process (in pmap_enter) was interrupted while holding the > page queue lock? > > > Giant is enabled in loader.conf due to the needs of the pf firewall when > dealing with user credentials lookups. I do not believe that Giant plays > into this deadlock. Kernel config attached. > > Any and all help or info is welcome. Thanks in advance.Try this change: jhb 2007-10-27 22:07:40 UTC FreeBSD src repository Modified files: sys/kern sched_4bsd.c Log: Change the roundrobin implementation in the 4BSD scheduler to trigger a userland preemption directly from hardclock() via sched_clock() when a thread uses up a full quantum instead of using a periodic timeout to cause a userland preemption every so often. This fixes a potential deadlock when IPI_PREEMPTION isn't enabled where softclock blocks on a lock held by a thread pinned or bound to another CPU. The current thread on that CPU will never be preempted while softclock is blocked. Note that ULE already drives its round-robin userland preemption from sched_clock() as well and always enables IPI_PREEMPT. MFC after: 1 week Revision Changes Path 1.108 +8 -29 src/sys/kern/sched_4bsd.c We use it at work on 6.x. W/o this fix, round-robin stops working on 4BSD when softclock() (swi4: clock) blocks on a lock like Giant. -- John Baldwin