Been fighting this for a while. We have an older server, running
5.4-RELEASE-p8 i386 and used primarily for email, which hangs every
couple of weeks. The hang seems to be in the disk I/O system; pings
succeed, and I can continue get a login: prompt on the console until
I enter a login at which the response stops.
I'm suspecting this is a 5.8 issue as we have the same problem on
a another box running 5.4-RELEASE-p8 amd64 with a 3ware controller.
I do not have a dump on that one.
Based on the times of the hangs, the triggering event seems to be
running dump.
We have a serial console set up, I broke to the debugger and got
the following info. Since the hang is in the disk I/O system, a
dump is not possible. The many versions of inetd are likely due
to users attempting to POP their email and hanging on disk I/O.
Any suggestions or tips on how to track this down would be appreciated.
db> ps
pid proc uid ppid pgrp flag stat wmesg wchan cmd
67487 c5ea98d4 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67486 c3b8a1c4 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67485 c634c710 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67484 c62931c4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67483 c58a9388 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67482 c6293710 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67481 c58ab8d4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67480 c6292c5c 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67479 c62938d4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67478 c634f000 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67477 c62941c4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67476 c5e55c5c 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67475 c5f1fe20 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67474 c5da854c 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67473 c5f9ee20 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67472 c58a9e20 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67471 c602d8d4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67470 c61191c4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67469 c58ab1c4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67468 c5f19a98 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67467 c58c3388 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67466 c5f1f388 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67465 c5f19000 0 442 67465 0000100 [SLPQ ufs 0xc3851c04][SLP] sshd
67464 c5fa68d4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67463 c5eab54c 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67462 c6294000 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67461 c5fa6710 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67460 c6119000 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67459 c634c388 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67458 c5da8710 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67457 c3b8e388 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67456 c62948d4 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67455 c62921c4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67454 c5f1954c 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67453 c5eab388 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67452 c5f171c4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67451 c6294388 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67450 c60291c4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67449 c5ea9710 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67448 c3b8ac5c 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67447 c6293388 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67446 c5f1fa98 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67445 c5e55e20 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67444 c6117710 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67443 c5fa7a98 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67442 c6026388 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67441 c5e5a1c4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67440 c6119710 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67439 c5e5a000 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67438 c3b8e8d4 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67437 c6293e20 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67436 c5dfee20 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67435 c5ea8e20 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67434 c5fa7388 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67433 c39dde20 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67432 c6118000 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67431 c58c31c4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67430 c634fe20 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67429 c5f191c4 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67428 c58a9c5c 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67427 c63508d4 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67426 c3b8ee20 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67425 c3b8ec5c 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67424 c5eabc5c 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67423 c5e5a54c 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67422 c39de710 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67421 c6117e20 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67420 c5da454c 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67419 c5e551c4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67418 c5da88d4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67417 c611ce20 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67416 c5e55000 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67415 c5f19c5c 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67414 c5f198d4 0 574 574 0000000 [SLPQ ufs 0xc3851c04][SLP] inetd
67413 c60261c4 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
67412 c6293c5c 0 574 67412 0004000 [SLPQ ufs 0xc3851c04][SLP] qpopper
67411 c58a9a98 0 574 67411 0004000 [SLPQ ufs 0xc3851c04][SLP] qpopper
67410 c38898d4 0 574 67410 0004000 [SLPQ suspfs 0xc37cac6c][SLP] qpopper
67409 c611c54c 0 574 67409 0004000 [SLPQ ufs 0xc38515d4][SLP] qpopper
67408 c634c1c4 0 574 67408 0004000 [SLPQ ufs 0xc3851c04][SLP] qpopper
67407 c5ea9000 0 574 67407 0004000 [SLPQ ufs 0xc3851c04][SLP] qpopper
67406 c58c3000 0 574 67406 0004000 [SLPQ ufs 0xc388d9f4][SLP] qpopper
67405 c5da41c4 2 67404 67401 0004100 [SLPQ ufs 0xc3851c04][SLP] mksnap_ffs
67404 c58c5a98 2 67403 67401 0004000 [SLPQ wait 0xc58c5a98][SLP] sh
67403 c5fa7e20 2 67402 67401 0004000 [SLPQ wait 0xc5fa7e20][SLP] dump
67402 c5fa71c4 2 67401 67401 0004000 [SLPQ piperd 0xc60fb600][SLP] gzip
67401 c5f19710 2 67400 67401 0004000 [SLPQ pause 0xc5f19748][SLP] tcsh
67400 c5e5554c 2 67398 67398 0000100 [SLPQ select 0xc08f1f44][SLP] sshd
67398 c611854c 0 442 67398 0000100 [SLPQ sbwait 0xc6273974][SLP] sshd
67322 c39dea98 0 551 551 0000100 [SLPQ ufs 0xc3851c04][SLP] sendmail
62457 c39dee20 0 514 514 0000100 [SLPQ accept 0xc3997916][SLP] perl5.8.6
62370 c5f1f710 0 514 514 0000100 [SLPQ accept 0xc3997916][SLP] perl5.8.6
61704 c58c5c5c 0 514 514 0000100 [SLPQ accept 0xc3997916][SLP] perl5.8.6
61031 c58a91c4 0 514 514 0000100 [SLPQ accept 0xc3997916][SLP] perl5.8.6
59856 c5da48d4 0 514 514 0000100 [SLPQ accept 0xc3997916][SLP] perl5.8.6
43502 c58c3710 5147 589 43502 0004002 [SLPQ ufs 0xc3851c04][SLP] tcsh
589 c38848d4 0 1 589 0004102 [SLPQ wait 0xc38848d4][SLP] login
588 c3884c5c 0 1 1 0004000 [SLPQ ufs 0xc3851c04][SLP] getty
587 c39dd000 0 1 1 0004000 [SLPQ ufs 0xc3851c04][SLP] getty
586 c3889e20 0 1 1 0004000 [SLPQ ufs 0xc3851c04][SLP] getty
574 c3884e20 0 1 574 0000000 [SLPQ select 0xc08f1f44][SLP] inetd
551 c39dd388 0 1 551 0000100 [SLPQ select 0xc08f1f44][SLP] sendmail
542 c39dda98 0 1 542 0008080 (threaded) spamass-milter
thread 0xc628e000 ksegrp 0xc353aaf0 [SLPQ kserel 0xc353ab30][SLP]
thread 0xc5fa1900 ksegrp 0xc353aaf0 [SLPQ select 0xc08f1f44][SLP]
thread 0xc39e0900 ksegrp 0xc353a5b0 [SLPQ ksesigwait 0xc39ddb98][SLP]
532 c39de000 25 1 532 0000100 [SLPQ pause 0xc39de038][SLP] sendmail
527 c39ddc5c 0 521 521 0000001 [SLPQ lockf 0xc38193c0][SLP] saslauthd
526 c39dd710 0 521 521 0000001 [SLPQ lockf 0xc35e4c40][SLP] saslauthd
525 c39dd54c 0 521 521 0000001 [SLPQ lockf 0xc37e2580][SLP] saslauthd
524 c39dd1c4 0 521 521 0000001 [SLPQ lockf 0xc3819140][SLP] saslauthd
521 c3889c5c 0 1 521 0000001 [SLPQ accept 0xc3997b9e][SLP] saslauthd
514 c3889a98 0 1 514 0000000 [SLPQ pause 0xc3889ad0][SLP] perl5.8.6
499 c37eda98 65534 1 499 0000100 [SLPQ select 0xc08f1f44][SLP] spamd
493 c3889000 0 1 493 0000000 [SLPQ select 0xc08f1f44][SLP] rpc.dracd
484 c3884388 106 1 484 0008180 (threaded) clamav-milter
thread 0xc5f1e900 ksegrp 0xc3448380 [SLPQ kserel 0xc34483c0][SLP]
thread 0xc6032780 ksegrp 0xc3448380 [SLPQ select 0xc08f1f44][SLP]
thread 0xc5df8780 ksegrp 0xc3448380 [SLPQ ufs 0xc3851c04][SLP]
thread 0xc39d6900 ksegrp 0xc353ae00 [SLPQ ksesigwait 0xc3884488][SLP]
477 c3884710 106 1 477 0000100 [SLPQ pause 0xc3884748][SLP] freshclam
470 c388454c 106 1 470 0008180 (threaded) clamd
thread 0xc5e34900 ksegrp 0xc3448310 [SLPQ kserel 0xc3448350][SLP]
thread 0xc5e56000 ksegrp 0xc3448310 [SLPQ accept 0xc3924e26][SLP]
thread 0xc3aa2000 ksegrp 0xc388abd0 [SLPQ ksesigwait 0xc388464c][SLP]
455 c3889388 0 1 455 0000000 [SLPQ ufs 0xc3851c04][SLP] cron
442 c38841c4 0 1 442 0000100 [SLPQ ufs 0xc3851c04][SLP] sshd
429 c3884000 0 1 429 0000000 [SLPQ select 0xc08f1f44][SLP] ntpd
338 c37ede20 0 1 338 0000000 [SLPQ select 0xc08f1f44][SLP] rpcbind
325 c3884a98 0 1 325 0000000 [SLPQ select 0xc08f1f44][SLP] syslogd
307 c38891c4 0 1 307 0000000 [SLPQ select 0xc08f1f44][SLP] devd
58 c353c54c 0 0 0 0000204 [SLPQ - 0xe67d4d18][SLP] schedcpu
57 c353c710 0 0 0 0000204 [SLPQ - 0xc08f996c][SLP] nfsiod 3
56 c353c8d4 0 0 0 0000204 [SLPQ - 0xc08f9968][SLP] nfsiod 2
55 c353ca98 0 0 0 0000204 [SLPQ - 0xc08f9964][SLP] nfsiod 1
54 c353cc5c 0 0 0 0000204 [SLPQ - 0xc08f9960][SLP] nfsiod 0
53 c353ce20 0 0 0 0000204 [SLPQ vlruwt 0xc353ce20][SLP] vnlru
52 c37ed000 0 0 0 0000204 [SLPQ syncer 0xc08ee6cc][SLP] syncer
51 c37ed1c4 0 0 0 0000204 [SLPQ psleep 0xc08f250c][SLP] bufdaemon
50 c37ed388 0 0 0 000020c [SLPQ pgzero 0xc09002d4][SLP] pagezero
49 c37ed54c 0 0 0 0000204 [SLPQ psleep 0xc0900328][SLP] vmdaemon
48 c37ed710 0 0 0 0000204 [SLPQ psleep 0xc09002e4][SLP] pagedaemon
47 c37ed8d4 0 0 0 0000204 [SLPQ m:w2 0xc37ea000][SLP] g_mirror
gm0s2
46 c349ba98 0 0 0 0000204 [SLPQ m:w2 0xc37ea500][SLP] g_mirror
gm0s1
45 c349bc5c 0 0 0 0000204 [IWAIT] swi0: sio
44 c349be20 0 0 0 0000204 [SLPQ - 0xc354223c][SLP] fdc0
43 c3538000 0 0 0 0000204 [SLPQ idle 0xc3540e00][SLP]
aic_recovery1
9 c35381c4 0 0 0 0000204 [SLPQ idle 0xc3540e00][SLP]
aic_recovery1
8 c3538388 0 0 0 0000204 [SLPQ idle 0xc3540400][SLP]
aic_recovery0
7 c353854c 0 0 0 0000204 [SLPQ idle 0xc3540400][SLP]
aic_recovery0
42 c3538710 0 0 0 0000204 [IWAIT] swi6: task queue
6 c35388d4 0 0 0 0000204 [SLPQ - 0xc352e3c0][SLP] kqueue taskq
41 c3538a98 0 0 0 0000204 [IWAIT] swi3: cambio
40 c3538c5c 0 0 0 0000204 [IWAIT] swi2: camnet
39 c3538e20 0 0 0 0000204 [IWAIT] swi6:+
5 c353c000 0 0 0 0000204 [SLPQ - 0xc352ed80][SLP] thread taskq
38 c348a54c 0 0 0 0000204 [IWAIT] swi6:+
37 c348a710 0 0 0 0000204 [SLPQ - 0xc08e4660][SLP] yarrow
4 c348a8d4 0 0 0 0000204 [SLPQ - 0xc08e8fa8][SLP] g_down
3 c348aa98 0 0 0 0000204 [SLPQ - 0xc08e8fa4][SLP] g_up
2 c348ac5c 0 0 0 0000204 [SLPQ - 0xc08e8f9c][SLP] g_event
36 c348ae20 0 0 0 0000204 [IWAIT] swi1: net
35 c349b000 0 0 0 0000204 [IWAIT] swi4: vm
34 c349b1c4 0 0 0 000020c [IWAIT] swi5: clock sio
33 c349b388 0 0 0 0000204 [IWAIT] irq0: clk
32 c349b54c 0 0 0 0000204 [IWAIT] irq22:
31 c349b710 0 0 0 0000204 [IWAIT] irq21:
30 c349b8d4 0 0 0 0000204 [IWAIT] irq20:
29 c34491c4 0 0 0 0000204 [IWAIT] irq19: fxp0
28 c3449388 0 0 0 0000204 [IWAIT] irq18:
27 c344954c 0 0 0 0000204 [IWAIT] irq17:
26 c3449710 0 0 0 0000204 [IWAIT] irq16: fxp1 ahc0+
25 c34498d4 0 0 0 0000204 [IWAIT] irq15: ata1
24 c3449a98 0 0 0 0000204 [IWAIT] irq14: ata0
23 c3449c5c 0 0 0 0000204 [IWAIT] irq13:
22 c3449e20 0 0 0 0000204 [IWAIT] irq12:
21 c348a000 0 0 0 0000204 [IWAIT] irq11:
20 c348a1c4 0 0 0 0000204 [IWAIT] irq10:
19 c348a388 0 0 0 0000204 [IWAIT] irq9:
18 c3442000 0 0 0 0000204 [IWAIT] irq8: rtc
17 c34421c4 0 0 0 0000204 [IWAIT] irq7: ppc0
16 c3442388 0 0 0 0000204 [IWAIT] irq6: fdc0
15 c344254c 0 0 0 0000204 [IWAIT] irq5:
14 c3442710 0 0 0 0000204 [IWAIT] irq4: sio0
13 c34428d4 0 0 0 0000204 [IWAIT] irq3: sio1
12 c3442a98 0 0 0 0000204 [IWAIT] irq1: atkbd0
11 c3442c5c 0 0 0 000020c [CPU 0] idle
1 c3442e20 0 0 1 0004200 [SLPQ wait 0xc3442e20][SLP] init
10 c3449000 0 0 0 0000204 [SLPQ ktrace 0xc08ec8f8][SLP] ktrace
0 c08e90a0 0 0 0 0000200 [SLPQ sched 0xc08e90a0][SLP] swapper
db> where
Tracing pid 11 tid 100003 td 0xc3443480
kdb_enter(c08479e3) at kdb_enter+0x2b
siointr1(c35d6000) at siointr1+0xd5
siointr(c35d6000) at siointr+0x38
intr_execute_handlers(c343dc90,e4d53cc8,4,e4d53d0c,c07b9483) at
intr_execute_handlers+0x7d
lapic_handle_intr(34) at lapic_handle_intr+0x2e
Xapic_isr1() at Xapic_isr1+0x33
--- interrupt, eip = 0xc07c057d, esp = 0xe4d53d0c, ebp = 0xe4d53d0c ---
cpu_idle_default(e4d53d20,c0604971,c3442c5c,e4d53d34,c0604720) at
cpu_idle_default+0x5
cpu_idle(c3442c5c,e4d53d34,c0604720,0,e4d53d48) at cpu_idle+0x1f
idle_proc(0,e4d53d48) at idle_proc+0x11
fork_exit(c0604960,0,e4d53d48) at fork_exit+0x74
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe4d53d7c, ebp = 0 ---
Relevant dmesg info:
ahc0: <Adaptec aic7896/97 Ultra2 SCSI adapter> port 0xe400-0xe4ff mem
0xffafe000-0xffafefff irq 16 at device 11.0 on pci0
aic7896/97: Ultra2 Wide Channel A, SCSI Id=7, 32/253 SCBs
ahc1: <Adaptec aic7896/97 Ultra2 SCSI adapter> port 0xe800-0xe8ff mem
0xffaff000-0xffafffff irq 16 at device 11.1 on pci0
aic7896/97: Ultra2 Wide Channel B, SCSI Id=7, 32/253 SCBs
da0 at ahc0 bus 0 target 0 lun 0
da0: <SEAGATE ST318436LW 0010> Fixed Direct Access SCSI-3 device
da0: 80.000MB/s transfers (40.000MHz, offset 31, 16bit), Tagged Queueing
Enabled
da0: 17522MB (35885168 512 byte sectors: 255H 63S/T 2233C)
da1 at ahc1 bus 0 target 0 lun 0
da1: <SEAGATE ST336938LW 0003> Fixed Direct Access SCSI-3 device
da1: 80.000MB/s transfers (40.000MHz, offset 63, 16bit), Tagged Queueing
Enabled
da1: 35242MB (72176567 512 byte sectors: 255H 63S/T 4492C)
da2 at ahc1 bus 0 target 1 lun 0
da2: <SEAGATE ST318436LW 0010> Fixed Direct Access SCSI-3 device
da2: 80.000MB/s transfers (40.000MHz, offset 31, 16bit), Tagged Queueing
Enabled
da2: 17522MB (35885168 512 byte sectors: 255H 63S/T 2233C)
GEOM_MIRROR: Device gm0s1 created (id=520792649).
GEOM_MIRROR: Device gm0s1: provider da0s1 detected.
GEOM_MIRROR: Device gm0s2 created (id=3744871543).
GEOM_MIRROR: Device gm0s2: provider da0s2 detected.
GEOM_MIRROR: Device gm0s1: provider da2s1 detected.
GEOM_MIRROR: Device gm0s1: provider da2s1 activated.
GEOM_MIRROR: Device gm0s1: provider da0s1 activated.
GEOM_MIRROR: Device gm0s1: provider mirror/gm0s1 launched.
GEOM_MIRROR: Device gm0s2: provider da2s2 detected.
GEOM_MIRROR: Device gm0s2: provider da2s2 activated.
GEOM_MIRROR: Device gm0s2: provider da0s2 activated.
GEOM_MIRROR: Device gm0s2: provider mirror/gm0s2 launched.