Robert Watson
2005-Jun-01 05:58 UTC
panic: vm_thread_swapin: cannot get kstack for proc: 643 under high memory load
Had an interesting panic on RELENG_5 today. System was under high memory
load due to a run-away pine process that was also generating a very high
memory load on the kernel due to heavy network I/O, but the swap pager
keels over due to ENOMEM (error 12). Before I knew it, syslogd was core
dumping and the system panicked due to a stack paging issue (possible that
there was no memory into which to load the stack?).
Some details below; core is available. An undesirable failure mode...
Robert N M Watson
Had an interesting panic on RELENG_5 today. System was under high memory
load due to a run-away pine process, but the swap pager keels over due to
ENOMEM (error 12). Before I knew it, syslogd was core dumping and the
system panicked due to a stack paging issue (possible that there was no
memory into which to load the stack?).
Regardless, not a desirable failure mode.
swap_pager: I/O error - pagein failed; blkno 7158,size 24576, error 12
vm_fault: pager read error, pid 1506 (pine)
swap_pager: I/O error - pagein failed; blkno 2989,size 4096, error 12
vm_fault: pager read error, pid 383 (syslogd)
swap_pager: I/O error - pagein failed; blkno 4272,size 4096, error 12
vm_fault: pager read error, pid 1506 (pine)
swap_pager: I/O error - pagein failed; blkno 2862,size 4096, error 12
vm_fault: pager read error, pid 642 (httpd)
swap_pager: I/O error - pagein failed; blkno 2862,size 4096, error 12
vm_fault: pager read error, pid 642 (httpd)
swap_pager: I/O error - pagein failed; blkno 399,size 4096, error 12
vm_fault: pager read error, pid 577 (httpd)
swap_pager: I/O error - pagein failed; blkno 399,size 4096, error 12
vm_fault: pager read error, pid 577 (httpd)
swap_pager: I/O error - pagein failed; blkno 4272,size 4096, error 12
vm_fault: pager read error, pid 1506 (pine)
swap_pager: I/O error - pagein failed; blkno 5880,size 4096, error 12
panic: vm_thread_swapin: cannot get kstack for proc: 643
KDB: enter: panic
[thread pid 0 tid 0 ]
Stopped at kdb_enter+0x2b: nop
Tracing pid 0 tid 0 td 0xc08e2820
kdb_enter(c0823105) at kdb_enter+0x2b
panic(c083a7f7,283,2,c1bd48d8,c1da9a90) at panic+0xbb
vm_thread_swapin(c28e7900) at vm_thread_swapin+0xd0
faultin(c28eac5c,c2255c34,c08bb050,c25000,c0c20d88) at faultin+0xc6
scheduler(0,c1ec00,c1e000,0,c0440e05) at scheduler+0x1b1
mi_startup() at mi_startup+0x96
begin() at begin+0x2c
db> show pcpu
cpuid = 0
curthread = 0xc08e2820: pid 0 "swapper"
curpcb = 0xc0c20d90
fpcurthread = none
idlethread = 0xc2265480: pid 11 "idle"
APIC ID = 0
currentldt = 0x28
db> ps
pid proc uid ppid pgrp flag stat wmesg wchan cmd
2520 c2b01000 0 534 534 0000100 [SLPQ select 0xc08eb504][SLP][SWAP]
sendmail
2394 c31048d4 1000 2320 2394 0015002 [RUNQ] pine
2320 c30fb000 1000 2319 2320 0004002 [SLPQ pause 0xc30fb038][SLP][SWAP] tcsh
2319 c30fb710 1000 2316 2316 0000100 [SLPQ select 0xc08eb504][SLP] sshd
2316 c241554c 0 529 2316 0000100 [SLPQ sbwait 0xc310d5a8][SLP][SWAP]
sshd
2315 c2b01a98 1000 2314 2315 0004002 [RUNQ] slogin
2314 c2a71e20 1000 2313 2314 0004002 [SLPQ pause 0xc2a71e58][SLP][SWAP] tcsh
2313 c309c8d4 1000 2311 2311 0000100 [SLPQ select 0xc08eb504][SLP] sshd
2311 c309cc5c 0 529 2311 0000100 [SLPQ sbwait 0xc310c6ec][SLP][SWAP]
sshd
2310 c3104000 0 534 534 0000100 [SLPQ select 0xc08eb504][SLP][SWAP]
sendmail
1838 c2a71710 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP] httpd
1837 c2b011c4 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP] httpd
1836 c2b0154c 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP][SWAP]
httpd
1835 c2b04c5c 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP] httpd
1834 c309c000 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP][SWAP]
httpd
1833 c2b03388 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP][SWAP]
httpd
1824 c2a718d4 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP][SWAP]
httpd
1506 c2b031c4 1000 1211 1506 0004002 [RUNQ] pine
1211 c2b03000 1000 1210 1211 0004002 [SLPQ pause 0xc2b03038][SLP][SWAP] tcsh
1210 c2b03a98 1000 1208 1208 0000100 [SLPQ select 0xc08eb504][SLP] sshd
1208 c2b01710 0 529 1208 0000100 [SLPQ sbwait 0xc2e19464][SLP][SWAP]
sshd
925 c2b01e20 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP] httpd
923 c28e6000 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP] httpd
922 c268e54c 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP][SWAP]
httpd
921 c2b03e20 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP][SWAP]
httpd
676 c2b03710 1026 1 670 0004000 [SLPQ select 0xc08eb504][SLP][SWAP]
uomapserver
668 c2a288d4 0 553 553 0000000 [SLPQ piperd 0xc26d3180][SLP] cron
663 c2a28e20 65534 577 577 0000100 [SLPQ select 0xc08eb504][SLP] httpd
662 c2a28c5c 65534 577 577 0000100 [SLPQ select 0xc08eb504][SLP] httpd
661 c2a28a98 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP][SWAP]
httpd
655 c2a6c000 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP][SWAP]
httpd
654 c2a6c1c4 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP][SWAP]
httpd
653 c2a6c388 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP][SWAP]
httpd
652 c2a6c54c 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP][SWAP]
httpd
651 c2a6c710 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP][SWAP]
httpd
650 c2a6c8d4 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP] httpd
649 c2a6ca98 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP][SWAP]
httpd
648 c2a6cc5c 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP][SWAP]
httpd
647 c2a6ce20 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP][SWAP]
httpd
646 c2a71000 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP][SWAP]
httpd
645 c2a711c4 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP][SWAP]
httpd
644 c2a71388 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP][SWAP]
httpd
643 c28eac5c 65534 577 577 0000100 [SWAP] httpd
641 c2a22000 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP][SWAP]
httpd
640 c2a221c4 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP][SWAP]
httpd
639 c2a22388 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP][SWAP]
httpd
638 c2a2254c 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP][SWAP]
httpd
637 c2a22710 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP][SWAP]
httpd
636 c2a228d4 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP][SWAP]
httpd
635 c2a22a98 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP][SWAP]
httpd
634 c2a22c5c 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP][SWAP]
httpd
633 c2a22e20 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP][SWAP]
httpd
632 c2a28000 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP][SWAP]
httpd
631 c2a281c4 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP][SWAP]
httpd
630 c2a28388 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP][SWAP]
httpd
629 c2a2854c 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP][SWAP]
httpd
628 c2a28710 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP][SWAP]
httpd
626 c28e68d4 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP][SWAP]
httpd
625 c268e8d4 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP] httpd
623 c28ea1c4 65534 577 577 0000100 [SLPQ accept 0xc275c916][SLP][SWAP]
httpd
622 c28e654c 0 1 622 0004002 [SLPQ ttyin 0xc2493410][SLP][SWAP]
getty 621 c28e61c4 0 1 621 0004002 [SLPQ ttyin 0xc2448e10][SLP][SWAP]
getty
620 c28ea000 0 1 620 0004002 [SLPQ ttyin 0xc2491010][SLP][SWAP]
getty
619 c28e6a98 0 1 619 0004002 [SLPQ ttyin 0xc2491210][SLP][SWAP]
getty
618 c28e6c5c 0 1 618 0004002 [SLPQ ttyin 0xc2491410][SLP][SWAP]
getty
617 c28ea388 0 1 617 0004002 [SLPQ ttyin 0xc2491610][SLP][SWAP]
getty
616 c28e6710 0 1 616 0004002 [SLPQ ttyin 0xc2491810][SLP][SWAP]
getty
615 c2415388 0 1 615 0004002 [SLPQ ttyin 0xc2491a10][SLP][SWAP]
getty
614 c268ea98 0 1 614 0004002 [SLPQ ttyin 0xc2409e10][SLP][SWAP]
getty
602 c28e6388 0 1 602 0000000 [SLPQ select 0xc08eb504][SLP][SWAP]
inetd
577 c28ea54c 0 1 577 0000000 [SLPQ ufs 0xc26bdd0c][SLP] httpd
553 c28eaa98 0 1 553 0000000 [SLPQ nanslp 0xc08e7f0c][SLP][SWAP]
cron
540 c268e388 25 1 540 0000100 [SLPQ pause 0xc268e3c0][SLP][SWAP]
sendmail
534 c2415c5c 0 1 534 0000100 [SLPQ select 0xc08eb504][SLP] sendmail
529 c268e000 0 1 529 0000100 [SLPQ select 0xc08eb504][SLP][SWAP]
sshd
490 c268e1c4 0 1 490 0000000 [SLPQ select 0xc08eb504][SLP] usbd
429 c24158d4 0 0 0 0000204 [SLPQ mdwait 0xc2866800][SLP] md0
399 c2415710 53 1 399 0008180 (threaded) named
thread 0xc2b06a80 ksegrp 0xc2414310 [RUNQ]
thread 0xc2b06c00 ksegrp 0xc2414310 [SLPQ select 0xc08eb504][SLP]
thread 0xc2412900 ksegrp 0xc2349c40 [SLPQ ksesigwait 0xc2415810][SLP]
383 c268e710 0 1 383 0000000 [RUNQ] syslogd
361 c2415a98 0 1 361 0000000 [SLPQ select 0xc08eb504][SLP][SWAP]
devd
157 c268ec5c 0 0 0 0000204 [RUNQ] schedcpu
156 c268ee20 0 0 0 0000204 [SLPQ - 0xc08f354c][SLP] nfsiod 3
155 c2364710 0 0 0 0000204 [SLPQ - 0xc08f3548][SLP] nfsiod 2
154 c23648d4 0 0 0 0000204 [SLPQ - 0xc08f3544][SLP] nfsiod 1
153 c2364a98 0 0 0 0000204 [SLPQ - 0xc08f3540][SLP] nfsiod 0
152 c2364c5c 0 0 0 0000204 [RUNQ] syncer
151 c2364e20 0 0 0 0000204 [RUNQ] vnlru
150 c2410000 0 0 0 0000204 [RUNQ] bufdaemon
149 c24101c4 0 0 0 000020c [SLPQ pgzero 0xc08f9eb4][SLP] pagezero
148 c2410388 0 0 0 0000204 [SLPQ psleep 0xc08f9f08][SLP] vmdaemon
147 c241054c 0 0 0 0000204 [SLPQ psleep 0xc08f9ec4][SLP]
pagedaemon
146 c2410710 0 0 0 0000204 [RUNQ] fdc0
145 c24108d4 0 0 0 0000204 [IWAIT] swi0: sio
144 c2410a98 0 0 0 0000204 [SLPQ usbevt 0xc244b210][SLP] usb2
143 c2410c5c 0 0 0 0000204 [SLPQ usbevt 0xc243f210][SLP] usb1
142 c2410e20 0 0 0 0000204 [SLPQ usbtsk 0xc08dff78][SLP] usbtask
141 c2415000 0 0 0 0000204 [SLPQ usbevt 0xc2435210][SLP] usb0
9 c24151c4 0 0 0 0000204 [SLPQ actask 0xc0a33bec][SLP]
acpi_task2
8 c234ca98 0 0 0 0000204 [SLPQ actask 0xc0a33bec][SLP]
acpi_task1
7 c234cc5c 0 0 0 0000204 [SLPQ actask 0xc0a33bec][SLP]
acpi_task0
6 c234ce20 0 0 0 0000204 [SLPQ - 0xc24077c0][SLP] kqueue taskq
140 c2360000 0 0 0 0000204 [IWAIT] swi3: cambio
139 c23601c4 0 0 0 0000204 [IWAIT] swi2: camnet
138 c2360388 0 0 0 0000204 [IWAIT] swi6: acpitaskq
137 c236054c 0 0 0 0000204 [IWAIT] swi6:+
5 c2360710 0 0 0 0000204 [SLPQ - 0xc239e800][SLP] thread taskq
136 c23608d4 0 0 0 0000204 [IWAIT] swi6:+
135 c2360a98 0 0 0 0000204 [IWAIT] swi6: task queue
134 c2360c5c 0 0 0 0000204 [RUNQ] yarrow
4 c2360e20 0 0 0 0000204 [RUNQ] g_down
3 c2364000 0 0 0 0000204 [RUNQ] g_up
2 c23641c4 0 0 0 0000204 [RUNQ] g_event
133 c2364388 0 0 0 0000204 [IWAIT] swi1: net
132 c236454c 0 0 0 0000204 [IWAIT] swi4: vm
131 c2337e20 0 0 0 000020c [RUNQ] swi5: clock sio
130 c2348000 0 0 0 0000204 [IWAIT] irq0: clk
129 c23481c4 0 0 0 0000204 [IWAIT] irq119:
128 c2348388 0 0 0 0000204 [IWAIT] irq118:
127 c234854c 0 0 0 0000204 [IWAIT] irq117:
126 c2348710 0 0 0 0000204 [IWAIT] irq116:
125 c23488d4 0 0 0 0000204 [IWAIT] irq115:
124 c2348a98 0 0 0 0000204 [IWAIT] irq114:
123 c2348c5c 0 0 0 0000204 [IWAIT] irq113:
122 c2348e20 0 0 0 0000204 [IWAIT] irq112:
121 c234c000 0 0 0 0000204 [IWAIT] irq111:
120 c234c1c4 0 0 0 0000204 [IWAIT] irq110:
119 c234c388 0 0 0 0000204 [IWAIT] irq109:
118 c234c54c 0 0 0 0000204 [IWAIT] irq108:
117 c234c710 0 0 0 0000204 [IWAIT] irq107:
116 c234c8d4 0 0 0 0000204 [IWAIT] irq106:
115 c23201c4 0 0 0 0000204 [IWAIT] irq105:
114 c2320388 0 0 0 0000204 [IWAIT] irq104:
113 c232054c 0 0 0 0000204 [IWAIT] irq103:
112 c2320710 0 0 0 0000204 [IWAIT] irq102:
111 c23208d4 0 0 0 0000204 [IWAIT] irq101:
110 c2320a98 0 0 0 0000204 [IWAIT] irq100:
109 c2320c5c 0 0 0 0000204 [IWAIT] irq99:
108 c2320e20 0 0 0 0000204 [IWAIT] irq98:
107 c2337000 0 0 0 0000204 [IWAIT] irq97:
106 c23371c4 0 0 0 0000204 [IWAIT] irq96:
105 c2337388 0 0 0 0000204 [IWAIT] irq95:
104 c233754c 0 0 0 0000204 [IWAIT] irq94:
103 c2337710 0 0 0 0000204 [IWAIT] irq93:
102 c23378d4 0 0 0 0000204 [IWAIT] irq92:
101 c2337a98 0 0 0 0000204 [IWAIT] irq91:
100 c2337c5c 0 0 0 0000204 [IWAIT] irq90:
99 c230154c 0 0 0 0000204 [IWAIT] irq89:
98 c2301710 0 0 0 0000204 [IWAIT] irq88:
97 c23018d4 0 0 0 0000204 [IWAIT] irq87:
96 c2301a98 0 0 0 0000204 [IWAIT] irq86:
95 c2301c5c 0 0 0 0000204 [IWAIT] irq85:
94 c2301e20 0 0 0 0000204 [IWAIT] irq84:
93 c231e000 0 0 0 0000204 [IWAIT] irq83:
92 c231e1c4 0 0 0 0000204 [IWAIT] irq82:
91 c231e388 0 0 0 0000204 [IWAIT] irq81:
90 c231e54c 0 0 0 0000204 [IWAIT] irq80:
89 c231e710 0 0 0 0000204 [IWAIT] irq79:
88 c231e8d4 0 0 0 0000204 [IWAIT] irq78:
87 c231ea98 0 0 0 0000204 [IWAIT] irq77:
86 c231ec5c 0 0 0 0000204 [IWAIT] irq76:
85 c231ee20 0 0 0 0000204 [IWAIT] irq75:
84 c2320000 0 0 0 0000204 [IWAIT] irq74:
83 c22efa98 0 0 0 0000204 [IWAIT] irq73:
82 c22efc5c 0 0 0 0000204 [IWAIT] irq72: twe0
81 c22efe20 0 0 0 0000204 [IWAIT] irq71:
80 c22fe000 0 0 0 0000204 [IWAIT] irq70:
79 c22fe1c4 0 0 0 0000204 [IWAIT] irq69:
78 c22fe388 0 0 0 0000204 [IWAIT] irq68:
77 c22fe54c 0 0 0 0000204 [IWAIT] irq67:
76 c22fe710 0 0 0 0000204 [IWAIT] irq66:
75 c22fe8d4 0 0 0 0000204 [IWAIT] irq65:
74 c22fea98 0 0 0 0000204 [IWAIT] irq64:
73 c22fec5c 0 0 0 0000204 [IWAIT] irq63:
72 c22fee20 0 0 0 0000204 [IWAIT] irq62:
71 c2301000 0 0 0 0000204 [IWAIT] irq61:
70 c23011c4 0 0 0 0000204 [IWAIT] irq60:
69 c2301388 0 0 0 0000204 [IWAIT] irq59:
68 c22df1c4 0 0 0 0000204 [IWAIT] irq58:
67 c22df388 0 0 0 0000204 [IWAIT] irq57:
66 c22df54c 0 0 0 0000204 [IWAIT] irq56:
65 c22df710 0 0 0 0000204 [IWAIT] irq55:
64 c22df8d4 0 0 0 0000204 [IWAIT] irq54:
63 c22dfa98 0 0 0 0000204 [IWAIT] irq53:
62 c22dfc5c 0 0 0 0000204 [IWAIT] irq52:
61 c22dfe20 0 0 0 0000204 [IWAIT] irq51:
60 c22ef000 0 0 0 0000204 [IWAIT] irq50:
59 c22ef1c4 0 0 0 0000204 [IWAIT] irq49:
58 c22ef388 0 0 0 0000204 [IWAIT] irq48:
57 c22ef54c 0 0 0 0000204 [IWAIT] irq47:
56 c22ef710 0 0 0 0000204 [IWAIT] irq46:
55 c22ef8d4 0 0 0 0000204 [IWAIT] irq45:
54 c22c7a98 0 0 0 0000204 [IWAIT] irq44:
53 c22c7c5c 0 0 0 0000204 [IWAIT] irq43:
52 c22c7e20 0 0 0 0000204 [IWAIT] irq42:
51 c22db000 0 0 0 0000204 [IWAIT] irq41:
50 c22db1c4 0 0 0 0000204 [IWAIT] irq40:
49 c22db388 0 0 0 0000204 [IWAIT] irq39:
48 c22db54c 0 0 0 0000204 [IWAIT] irq38:
47 c22db710 0 0 0 0000204 [IWAIT] irq37:
46 c22db8d4 0 0 0 0000204 [IWAIT] irq36:
45 c22dba98 0 0 0 0000204 [IWAIT] irq35:
44 c22dbc5c 0 0 0 0000204 [IWAIT] irq34:
43 c22dbe20 0 0 0 0000204 [IWAIT] irq33:
42 c22df000 0 0 0 0000204 [IWAIT] irq32:
41 c22b654c 0 0 0 0000204 [IWAIT] irq31:
40 c22b6710 0 0 0 0000204 [IWAIT] irq30:
39 c22b68d4 0 0 0 0000204 [IWAIT] irq29: em1
38 c22b6a98 0 0 0 0000204 [RUNQ] irq28: em0
37 c22b6c5c 0 0 0 0000204 [IWAIT] irq27:
36 c22b6e20 0 0 0 0000204 [IWAIT] irq26:
35 c22c7000 0 0 0 0000204 [IWAIT] irq25:
34 c22c71c4 0 0 0 0000204 [IWAIT] irq24:
33 c22c7388 0 0 0 0000204 [IWAIT] irq23:
32 c22c754c 0 0 0 0000204 [IWAIT] irq22:
31 c22c7710 0 0 0 0000204 [IWAIT] irq21:
30 c22c78d4 0 0 0 0000204 [IWAIT] irq20:
29 c226b1c4 0 0 0 0000204 [IWAIT] irq19: uhci1
28 c226b388 0 0 0 0000204 [IWAIT] irq18: uhci2
27 c226b54c 0 0 0 0000204 [IWAIT] irq17:
26 c226b710 0 0 0 0000204 [IWAIT] irq16: uhci0
25 c226b8d4 0 0 0 0000204 [IWAIT] irq15: ata1
24 c226ba98 0 0 0 0000204 [IWAIT] irq14: ata0
23 c226bc5c 0 0 0 0000204 [IWAIT] irq13:
22 c226be20 0 0 0 0000204 [IWAIT] irq12:
21 c22b6000 0 0 0 0000204 [IWAIT] irq11:
20 c22b61c4 0 0 0 0000204 [IWAIT] irq10:
19 c22b6388 0 0 0 0000204 [IWAIT] irq9: acpi0
18 c2264000 0 0 0 0000204 [IWAIT] irq8: rtc
17 c22641c4 0 0 0 0000204 [IWAIT] irq7: ppc0
16 c2264388 0 0 0 0000204 [IWAIT] irq6: fdc0
15 c226454c 0 0 0 0000204 [IWAIT] irq5:
14 c2264710 0 0 0 0000204 [IWAIT] irq4: sio0
13 c22648d4 0 0 0 0000204 [IWAIT] irq3: sio1
12 c2264a98 0 0 0 0000204 [IWAIT] irq1: atkbd0
11 c2264c5c 0 0 0 000020c [Can run] idle
1 c2264e20 0 0 1 0004200 [SLPQ wait 0xc2264e20][SLP] init
10 c226b000 0 0 0 0000204 [SLPQ ktrace 0xc08e5e98][SLP] ktrace
0 c08e2640 0 0 0 0000200 [CPU 0] swapper
670 c28ea710 1026 668 670 0006000 zomb[INACTIVE] sh
db> trace 2394
Tracing pid 2394 tid 100278 td 0xc2b07c00
sched_switch(c2b07c00,0,1) at sched_switch+0x16f
mi_switch(1,0) at mi_switch+0x264
sleepq_switch(c08eb504,c2b07c00,c08eb504,e73fcb50,c05f1814) at
sleepq_switch+0xe0
sleepq_timedwait_sig(c08eb504,0,c08eb504,c08eb504,5dd) at
sleepq_timedwait_sig+0xd
cv_timedwait_sig(c08eb504,c08eb4e0,5dd,4,e73fcba4) at cv_timedwait_sig+0x1b0
kern_select(c2b07c00,6,bfbf7f60,0,bfbf7ee0) at kern_select+0x4a9
select(c2b07c00,e73fcd04,5,4fc,206) at select+0x44
syscall(2f,bfbf002f,bfbf002f,304a,bfbfcc10) at syscall+0x227
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (93, FreeBSD ELF32, select), eip = 0x28568f6b, esp = 0xbfbf7e7c, ebp
= 0xbfbf7ff8 ---
db> trace 2315
Tracing pid 2315 tid 100238 td 0xc2a72c00
sched_switch(c2a72c00,0,1) at sched_switch+0x16f
mi_switch(1,0) at mi_switch+0x264
sleepq_switch(c08eb504,c2a72c00,e7357b54,c05f13e6,c08eb504) at
sleepq_switch+0xe0
sleepq_wait_sig(c08eb504,c08eb504,0,0,c22ab6a0) at sleepq_wait_sig+0xc
cv_wait_sig(c08eb504,c08eb4e0,4,e7357ba4,1) at cv_wait_sig+0x19e
kern_select(c2a72c00,7,806e640,806e680,0) at kern_select+0x4bf
select(c2a72c00,e7357d04,5,50,202) at select+0x44
syscall(bfbf002f,bfbf002f,bfbf002f,bfbfe890,bfbfe894) at syscall+0x227
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (93, FreeBSD ELF32, select), eip = 0x282b2f6b, esp = 0xbfbfe78c, ebp
= 0xbfbfe858 ---
db> trace 383
Tracing pid 383 tid 100171 td 0xc2417180
sched_switch(c2417180,0,1) at sched_switch+0x16f
mi_switch(1,0) at mi_switch+0x264
sleepq_switch(e4f766b0,1,e4f76680,c0619a8e,e4f766b0) at sleepq_switch+0xe0
sleepq_timedwait(e4f766b0,0,0,0,0) at sleepq_timedwait+0x34
msleep(e4f766b0,0,4c,c081da50,a) at msleep+0x28a
g_dev_strategy(d64f3a34,c087bca0,c2417180,d64f3a34,e4f766d8) at
g_dev_strategy+0xc3
dev_strategy(d64f3a34) at dev_strategy+0x6c
spec_xstrategy(c26bdd68,d64f3a34,2,e4f766f8,c05dd7d7) at spec_xstrategy+0x150
spec_specstrategy(e4f76720,e4f7673c,c0651040,e4f76720,0) at
spec_specstrategy+0x4f
spec_vnoperate(e4f76720) at spec_vnoperate+0x13
breadn(c26bdd68,c0,0,4000,0) at breadn+0xe8
bread(c26bdd68,c0,0,4000,0,e4f767d8) at bread+0x20
ffs_nodealloccg(c26c1000,0,2,0,8180) at ffs_nodealloccg+0x10d
ffs_hashalloc(c26c1000,0,2,0,8180) at ffs_hashalloc+0x28
ffs_valloc(c26bdc60,8180,c2262d80,e4f7687c,c26c1000) at ffs_valloc+0xbf
ufs_makeinode(8180,c26bdc60,e4f76c30,e4f76c44) at ufs_makeinode+0x59
ufs_create(e4f76a00,e4f76abc,c066ac84,e4f76a00,46) at ufs_create+0x26
ufs_vnoperate(e4f76a00) at ufs_vnoperate+0x13
vn_open_cred(e4f76c1c,e4f76b08,180,c2262d80,ffffffff) at vn_open_cred+0x18c
vn_open(e4f76c1c,e4f76b08,180,ffffffff,ffffffff) at vn_open+0x1e
coredump(c2417180) at coredump+0x1af
sigexit(c2417180,b,c26b2aa8,0,c0823436) at sigexit+0x53
postsig(b) at postsig+0x131
ast(e4f76d38) at ast+0x386
doreti_ast() at doreti_ast+
db> trace 152
Tracing pid 152 tid 100131 td 0xc2362180
sched_switch(c2362180,0,1) at sched_switch+0x16f
mi_switch(1,0) at mi_switch+0x264
sleepq_switch(c08e7c9c,e4efec98,c0619ab5,c08e7c9c,0) at sleepq_switch+0xe0
sleepq_wait(c08e7c9c,0,0,0,0) at sleepq_wait+0x30
msleep(c08e7c9c,0,68,c082a162,0) at msleep+0x2b1
sched_sync(0,e4efed38,0,c0660848,0) at sched_sync+0x4d7
fork_exit(c0660848,0,e4efed38) at fork_exit+0xa4
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe4efed6c, ebp = 0 ---
db> trace 151
Tracing pid 151 tid 100130 td 0xc2362000
sched_switch(c2362000,0,1) at sched_switch+0x16f
mi_switch(1,0) at mi_switch+0x264
sleepq_switch(c2364e20,1,e4efbcc0,c0619a8e,c2364e20) at sleepq_switch+0xe0
sleepq_timedwait(c2364e20,0,0,0,17f4) at sleepq_timedwait+0x34
msleep(c2364e20,0,50,c0829f15,64,c08ebf84,c08ebec0,0,c0829ee4,248) at
msleep+0x28a
vnlru_proc(0,e4efbd38,0,c065eb3c,0) at vnlru_proc+0xe1
fork_exit(c065eb3c,0,e4efbd38) at fork_exit+0xa4
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe4efbd6c, ebp = 0 ---
db> trace 150
Tracing pid 150 tid 100129 td 0xc234bd80
sched_switch(c234bd80,0,1) at sched_switch+0x16f
mi_switch(1,0) at mi_switch+0x264
sleepq_switch(c08ebacc,1,e4ef8cf0,c0619a8e,c08ebacc) at sleepq_switch+0xe0
sleepq_timedwait(c08ebacc,0,0,0,c2410000) at sleepq_timedwait+0x34
msleep(c08ebacc,c08ebae0,44,c0828bb3,64) at msleep+0x28a
buf_daemon(0,e4ef8d38,0,c06535d0,0) at buf_daemon+0x1d9
fork_exit(c06535d0,0,e4ef8d38) at fork_exit+0xa4
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe4ef8d6c, ebp = 0 ---
db> trace 146
Tracing pid 146 tid 100157 td 0xc2411a80
sched_switch(c2411a80,0,1) at sched_switch+0x16f
mi_switch(1,0) at mi_switch+0x264
sleepq_switch(c244623c,0,e4f4cca8,c0619a8e,c244623c) at sleepq_switch+0xe0
sleepq_timedwait(c244623c,0,0,0,c244623c) at sleepq_timedwait+0x34
msleep(c244623c,c24462ac,4c,c081cead,64) at msleep+0x28a
fdc_worker(c2446200,c2410710,c0789ff4,e4f4cd24,c0601c24) at fdc_worker+0x192
fdc_thread(c2446200,e4f4cd38,c2446200,c0789ff4,0) at fdc_thread+0xe
fork_exit(c0789ff4,c2446200,e4f4cd38) at fork_exit+0xa4
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe4f4cd6c, ebp = 0 ---
db> trace 4
Tracing pid 4 tid 100139 td 0xc2362d80
sched_switch(c2362d80,0,1) at sched_switch+0x16f
mi_switch(1,0) at mi_switch+0x264
sleepq_switch(e4f16cf4,0,e4f16cd8,c0619a8e,e4f16cf4) at sleepq_switch+0xe0
sleepq_timedwait(e4f16cf4,0,0,0,d61b55ac) at sleepq_timedwait+0x34
msleep(e4f16cf4,0,4c,c081e3a4,a) at msleep+0x28a
g_io_schedule_down(c2362d80) at g_io_schedule_down+0x9c
g_down_procbody(0,e4f16d38,0,c05e24fc,0) at g_down_procbody+0x32
fork_exit(c05e24fc,0,e4f16d38) at fork_exit+0xa4
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe4f16d6c, ebp = 0 ---
db> trace 3
Tracing pid 3 tid 100138 td 0xc2362c00
sched_switch(c2362c00,0,1) at sched_switch+0x16f
mi_switch(1,0) at mi_switch+0x264
sleepq_switch(c08e2544,0,e4f13cdc,c0619a8e,c08e2544) at sleepq_switch+0xe0
sleepq_timedwait(c08e2544,0,0,0,0) at sleepq_timedwait+0x34
msleep(c08e2544,c08e2468,24c,c081cead,a) at msleep+0x28a
g_io_schedule_up(c2362c00) at g_io_schedule_up+0x91
g_up_procbody(0,e4f13d38,0,c05e24c4,0) at g_up_procbody+0x32
fork_exit(c05e24c4,0,e4f13d38) at fork_exit+0xa4
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe4f13d6c, ebp = 0 ---
db> trace 131
Tracing pid 131 tid 100102 td 0xc2322300
sched_switch(c2322300,0,1) at sched_switch+0x16f
mi_switch(1,0) at mi_switch+0x264
ithread_loop(c235f580,e4ea7d38,c235f580,c060268c,0) at ithread_loop+0x21f
fork_exit(c060268c,c235f580,e4ea7d38) at fork_exit+0xa4
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe4ea7d6c, ebp = 0 ---
Alan Cox
2005-Jun-01 11:34 UTC
panic: vm_thread_swapin: cannot get kstack for proc: 643 under high memory load
On Wed, Jun 01, 2005 at 01:57:59PM +0100, Robert Watson wrote:> > Had an interesting panic on RELENG_5 today. System was under high memory > load due to a run-away pine process that was also generating a very high > memory load on the kernel due to heavy network I/O, but the swap pager > keels over due to ENOMEM (error 12). Before I knew it, syslogd was core > dumping and the system panicked due to a stack paging issue (possible that > there was no memory into which to load the stack?). > > Some details below; core is available. An undesirable failure mode... > > Robert N M Watson > > > Had an interesting panic on RELENG_5 today. System was under high memory > load due to a run-away pine process, but the swap pager keels over due to > ENOMEM (error 12). Before I knew it, syslogd was core dumping and the > system panicked due to a stack paging issue (possible that there was no > memory into which to load the stack?).We don't release the kernel virtual address range for the stack on a swap out. So, it's still available in vm_thread_swapin(). Furthermore, at this point in vm_thread_swapin(), we have already allocated the required physical page(s). The panic is a direct result of the I/O error. In other words, vm_thread_swapin() panics if there is an I/O error during page in of the stack page(s).> Regardless, not a desirable failure mode.Indeed, I'll see what I can do. Alan