Anton Yuzhaninov
2008-Nov-24 14:59 UTC
RELENG_7 panic under load: vm_page_unwire: invalid wire count: 0
Box with fresh RELENG_7 panic under heavy network load (more than 50k
connections).
This panics seems to be senfile(2) related, because when sendfile disabled in
nginx, I can't reproduce the problem.
Backtrace in all cases like this:
# kgdb kernel /spool/crash/vmcore.1
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "amd64-marcel-freebsd"...
Unread portion of the kernel message buffer:
panic: vm_page_unwire: invalid wire count: 0
cpuid = 0
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
panic() at panic+0x182
vm_page_unwire() at vm_page_unwire+0x84
sf_buf_mext() at sf_buf_mext+0x3c
mb_free_ext() at mb_free_ext+0x99
sbdrop_internal() at sbdrop_internal+0x1e8
tcp_do_segment() at tcp_do_segment+0x1512
tcp_input() at tcp_input+0x7f7
ip_input() at ip_input+0xa8
ether_demux() at ether_demux+0x1b4
ether_input() at ether_input+0x1bb
bge_intr() at bge_intr+0x3ca
ithread_loop() at ithread_loop+0x180
fork_exit() at fork_exit+0x11f
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffea28fd30, rbp = 0 ---
Uptime: 36m47s
Physical memory: 4087 MB
Dumping 708 MB: 693 677 661 645 629 613 597 581 565 549 533 517 501 485 469 453
437 421 405 389 373 357 341 325 309 293 277 261 245 229 213 197 181 165 149 133
117 101 85 69 53 37 21 5
#0 doadump () at pcpu.h:195
195 __asm __volatile("movq %%gs:0,%0" : "=r"
(td));
(kgdb) bt
#0 doadump () at pcpu.h:195
#1 0xffffffff8031adf8 in boot (howto=260) at
/usr/src/sys/kern/kern_shutdown.c:418
#2 0xffffffff8031b25c in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:574
#3 0xffffffff8044a084 in vm_page_unwire (m=Variable "m" is not
available.
) at /usr/src/sys/vm/vm_page.c:1410
#4 0xffffffff80379a4c in sf_buf_mext (addr=Variable "addr" is not
available.
) at /usr/src/sys/kern/uipc_syscalls.c:1720
#5 0xffffffff8036e9c9 in mb_free_ext (m=0xffffff0081f93d00) at
/usr/src/sys/kern/uipc_mbuf.c:257
#6 0xffffffff80372c38 in sbdrop_internal (sb=0xffffff00b4161458, len=2896) at
mbuf.h:515
#7 0xffffffff803d6532 in tcp_do_segment (m=0xffffff0075c23b00,
th=0xffffff0075c53024, so=0xffffff00b41612d0,
tp=0xffffff00b4154b60, drop_hdrlen=52, tlen=0) at
/usr/src/sys/netinet/tcp_input.c:2042
#8 0xffffffff803d7bc7 in tcp_input (m=0xffffff0075c23b00, off0=20) at
/usr/src/sys/netinet/tcp_input.c:846
#9 0xffffffff803cf108 in ip_input (m=0xffffff0075c23b00) at
/usr/src/sys/netinet/ip_input.c:665
#10 0xffffffff803b8004 in ether_demux (ifp=0xffffff0001255800,
m=0xffffff0075c23b00) at /usr/src/sys/net/if_ethersubr.c:834
#11 0xffffffff803b825b in ether_input (ifp=0xffffff0001255800,
m=0xffffff0075c23b00) at /usr/src/sys/net/if_ethersubr.c:692
#12 0xffffffff801bcf5a in bge_intr (xsc=Variable "xsc" is not
available.
) at /usr/src/sys/dev/bge/if_bge.c:3160
#13 0xffffffff802fb5f0 in ithread_loop (arg=0xffffff0003711840) at
/usr/src/sys/kern/kern_intr.c:1088
#14 0xffffffff802f7f7f in fork_exit (callout=0xffffffff802fb470
<ithread_loop>, arg=0xffffff0003711840,
frame=0xffffffffea28fc80) at /usr/src/sys/kern/kern_fork.c:804
#15 0xffffffff8045b88e in fork_trampoline () at
/usr/src/sys/amd64/amd64/exception.S:455
#16 0x0000000000000000 in ?? ()
#17 0x0000000000000000 in ?? ()
#18 0x0000000000000001 in ?? ()
in /boot/loader.conf I have:
vm.kmem_size=1536M
# 2 Mb KVA/kmem
net.inet.tcp.tcbhashsize=131072
# 64M KVA
kern.maxbcache=64M
# 4M KVA
kern.ipc.maxpipekva=4M
#
net.inet.tcp.syncache.hashsize=1024
net.inet.tcp.syncache.bucketlimit=100
in /etc/sysctl.conf
# 576 Mb KVA/kmem
kern.ipc.nmbclusters=262144
kern.ipc.nmbjumbop=65536
kern.ipc.maxsockets=307200
kern.ipc.somaxconn=4096
kern.maxfiles=307200
kern.maxfilesperproc=102400
$ sysctl vm.kvm_free
vm.kvm_free: 327151616
netstat -m output, several seconds before panic:
380270/63895/444165 mbufs in use (current/cache/total)
14141/29273/43414/262144 mbuf clusters in use (current/cache/total/max)
14141/29251 mbuf+clusters out of packet secondary zone in use (current/cache)
0/9/9/65536 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/6400 9k jumbo clusters in use (current/cache/total/max)
0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
123349K/74555K/197905K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/0/0 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
1 requests for I/O initiated by sendfile
0 calls to protocol drain routines
--
Anton Yuzhaninov
Anton Yuzhaninov
2008-Nov-24 15:38 UTC
RELENG_7 panic under load: vm_page_unwire: invalid wire count: 0
On 25.11.2008 01:48, Anton Yuzhaninov wrote:> Box with fresh RELENG_7 panic under heavy network load (more than 50k > connections). > > This panics seems to be senfile(2) related, because when sendfile > disabled in nginx, I can't reproduce the problem. > > Backtrace in all cases like this: > > # kgdb kernel /spool/crash/vmcore.1 > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you > are > welcome to change it and/or distribute copies of it under certain > conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "amd64-marcel-freebsd"... > > Unread portion of the kernel message buffer: > panic: vm_page_unwire: invalid wire count: 0 > cpuid = 0 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > panic() at panic+0x182 > vm_page_unwire() at vm_page_unwire+0x84 > sf_buf_mext() at sf_buf_mext+0x3c > mb_free_ext() at mb_free_ext+0x99 > sbdrop_internal() at sbdrop_internal+0x1e8 > tcp_do_segment() at tcp_do_segment+0x1512 > tcp_input() at tcp_input+0x7f7 > ip_input() at ip_input+0xa8 > ether_demux() at ether_demux+0x1b4 > ether_input() at ether_input+0x1bb > bge_intr() at bge_intr+0x3ca > ithread_loop() at ithread_loop+0x180 > fork_exit() at fork_exit+0x11f > fork_trampoline() at fork_trampoline+0xe > --- trap 0, rip = 0, rsp = 0xffffffffea28fd30, rbp = 0 --- > Uptime: 36m47s > Physical memory: 4087 MB > Dumping 708 MB: 693 677 661 645 629 613 597 581 565 549 533 517 501 485 > 469 453 437 421 405 389 373 357 341 325 309 293 277 261 245 229 213 197 > 181 165 149 133 117 101 85 69 53 37 21 5 > > #0 doadump () at pcpu.h:195 > 195 __asm __volatile("movq %%gs:0,%0" : "=r" (td)); > (kgdb) bt > #0 doadump () at pcpu.h:195 > #1 0xffffffff8031adf8 in boot (howto=260) at > /usr/src/sys/kern/kern_shutdown.c:418 > #2 0xffffffff8031b25c in panic (fmt=Variable "fmt" is not available. > ) at /usr/src/sys/kern/kern_shutdown.c:574 > #3 0xffffffff8044a084 in vm_page_unwire (m=Variable "m" is not available. > ) at /usr/src/sys/vm/vm_page.c:1410 > #4 0xffffffff80379a4c in sf_buf_mext (addr=Variable "addr" is not > available. > ) at /usr/src/sys/kern/uipc_syscalls.c:1720 > #5 0xffffffff8036e9c9 in mb_free_ext (m=0xffffff0081f93d00) at > /usr/src/sys/kern/uipc_mbuf.c:257May be it is wire_count integer overflow? wire_count type is u_short... -- Anton Yuzhaninov