Daniel Dvořák
2006-Sep-02 01:21 UTC
atheros driver under high load, panics and even more freezes
Hi all,
first of all, I?m sorry maybe for my bad English.
We have 2 routers which I maintain in our mesh wireless community network.
The Router 1 has 2 atheros adapters, ath0=wistron cm9, ath1=wistron cm10, of
course some sisX, fxpX and so on.
The Router 2 has 1 atheros adapter, ath1=wistron CM10.
My R1 panics and even more it freezes very often. Maybe the reason for
panicing and freezing is the same and maybe not.
I started (only after vmcore.5, so vmcore.6 is with this option) to use
"option SW_WATCHDOG" in both my custom kernels on the R1 and R2
recently in
hope, it is some walkaround for freezing at least if not for panicing.
This router was installed on the 1st of April 2006.
Statistics:
9 panics with 8 kernel dumps, 1 missed
10 freezes
I think that all panics some how connected to athX taskq process, page fault
in kernel panic and sbflush_locked.
I guess that panic comes when router transmits and receives datas at the
maximum throughput for setted nominal media rate speed, exactly 24Mbps, more
I do not use, because there are problems with quagga
ospfd packets, it is known issue.
Today I did a small test with throughput.
Router 1 executed this command:
# ping -i 0.001 -c 100000 -s 1472 ANY IP
As you see, it is not even flood ping, it is almost flood, but not flood.
Throughput was about 1,13-1,2 MB/s as bmon showed me. I notice there is not
any qos and icmp.limit is so high net.inet.icmp.icmplim: 2147483647
net.ineticmp.icmplim_output: 0.
First 5 s latency was about 1,1-1,7 ms
After it goes to 10-30, 50-70, 110-130, 270-300, up 300ms and packet loss
.... some seconds ....
panic
here it is:
# kgdb kernel.debug /var/crash/vmcore.6
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so:
Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "i386-marcel-freebsd".
Unread portion of the kernel message buffer:
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0xc
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc05a47bb
stack pointer = 0x28:0xd447db18
frame pointer = 0x28:0xd447db3c
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 24 (ath1 taskq)
trap number = 12
panic: page fault
Uptime: 1d7h56m6s
Dumping 511 MB (2 chunks)
chunk 0: 1MB (159 pages) ... ok
chunk 1: 511MB (130800 pages) 495 479 463 447 431 415 399 383 367 351 335
319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15
#0 doadump () at pcpu.h:165
165 __asm __volatile("movl %%fs:0,%0" : "=r"
(td));
(kgdb) backtrace
#0 doadump () at pcpu.h:165
#1 0xc056da25 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:402
#2 0xc056dcbc in panic (fmt=0xc07a4185 "%s") at
/usr/src/sys/kern/kern_shutdown.c:558
#3 0xc07658c8 in trap_fatal (frame=0xd447dad8, eva=12) at
/usr/src/sys/i386/i386/trap.c:836
#4 0xc076562f in trap_pfault (frame=0xd447dad8, usermode=0, eva=12) at
/usr/src/sys/i386/i386/trap.c:744
#5 0xc076528d in trap (frame {tf_fs = 8, tf_es = 40, tf_ds = 40, tf_edi =
-979275436, tf_esi = 370,
tf_ebp = -733488324, tf_isp = -733488380, tf_ebx = -979275520, tf_edx = 0,
tf_ecx = -1012252656, tf_eax = 0, tf_trapno = 12, tf_err = 0, tf_eip
-1067825221, tf_cs = 32, tf_eflags = 590342, tf_esp = 0, tf_ss -733488320})
at /usr/src/sys/i386/i386/trap.c:434
#6 0xc0754a6a in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#7 0xc05a47bb in m_copym (m=0x0, off0=1500, len=1480, wait=1) at
/usr/src/sys/kern/uipc_mbuf.c:400
#8 0xc06204b8 in ip_fragment (ip=0xc3aa4010, m_frag=0xd447dbec,
mtu=-979275520, if_hwassist_flags=0, sw_csum=1) at
/usr/src/sys/netinet/ip_output.c:975
#9 0xc0610c4e in ip_fastforward (m=0xc3a7ba00) at
/usr/src/sys/netinet/ip_fastfwd.c:561
#10 0xc05de953 in ether_demux (ifp=0xc33ac400, m=0xc3a7ba00) at
/usr/src/sys/net/if_ethersubr.c:766
#11 0xc05de715 in ether_input (ifp=0xc33ac400, m=0xc3a7ba00) at
/usr/src/sys/net/if_ethersubr.c:620
#12 0xc05f5604 in ieee80211_deliver_data (ic=0xc33ad230, ni=0xc5602000,
m=0xc3a7ba00) at /usr/src/sys/net80211/ieee80211_input.c:717
#13 0xc05f507d in ieee80211_input (ic=0xc33ad230, m=0xc3a7ba00,
ni=0xc5602000, rssi=30, rstamp=27616) at
/usr/src/sys/net80211/ieee80211_input.c:481
#14 0xc04afd66 in ath_rx_proc (arg=0xc33ad000, npending=1) at
/usr/src/sys/dev/ath/if_ath.c:2977
#15 0xc058de89 in taskqueue_run (queue=0xc32f4780) at
/usr/src/sys/kern/subr_taskqueue.c:217
#16 0xc058e03a in taskqueue_thread_loop (arg=0x0) at
/usr/src/sys/kern/subr_taskqueue.c:276
#17 0xc0558068 in fork_exit (callout=0xc058dff8 <taskqueue_thread_loop>,
arg=0xc33adee0, frame=0xd447dd38) at /usr/src/sys/kern/kern_fork.c:805
#18 0xc0754acc in fork_trampoline () at
/usr/src/sys/i386/i386/exception.s:208
(kgdb) quit
I notice packets were send out through ath0, so I expect ath0 taskq, through
ath1 I was logged in the box.
I can reproduce it with ping command, with high load.
There are earlier kernel dumps which is connected to high load some how, but
kgdb does not work with them.
# kgdb kernel.debug /var/crash/vmcore.0
kgdb: kvm_read: invalid address (0x18)
kgdb: kvm_read: invalid address (0x18)
kgdb: kvm_read: invalid address (0x18)
^C
# kgdb kernel.debug /var/crash/vmcore.1
kgdb: cannot read PTD
# kgdb kernel.debug /var/crash/vmcore.1
kgdb: cannot read PTD
# kgdb kernel.debug /var/crash/vmcore.1
kgdb: cannot read PTD
# kgdb kernel.debug /var/crash/vmcore.2
kgdb: cannot read PTD
# kgdb kernel.debug /var/crash/vmcore.3
kgdb: cannot read PTD
# kgdb kernel.debug /var/crash/vmcore.4
kgdb: cannot read PTD
# kgdb kernel.debug /var/crash/vmcore.5
kgdb: cannot read PTD
I have info?s files, here they are in attachments.
I guess it is about mem_buf, ath.c, ath taskq, something about memory, when
there is HIGH LOAD.
ANY HELP IS VERY APPRECIATED.
Daniel
P.S.: I am not currently subscribed in the freebsd-stable mailling list, so
use my e-mail address. I am ok with freebsd-current mailling list.
-------------- next part --------------
# kgdb kernel.debug /var/crash/vmcore.0
kgdb: kvm_read: invalid address (0x18)
kgdb: kvm_read: invalid address (0x18)
kgdb: kvm_read: invalid address (0x18)
^C
# kgdb kernel.debug /var/crash/vmcore.1
kgdb: cannot read PTD
# kgdb kernel.debug /var/crash/vmcore.1
kgdb: cannot read PTD
# kgdb kernel.debug /var/crash/vmcore.1
kgdb: cannot read PTD
# kgdb kernel.debug /var/crash/vmcore.2
kgdb: cannot read PTD
# kgdb kernel.debug /var/crash/vmcore.3
kgdb: cannot read PTD
# kgdb kernel.debug /var/crash/vmcore.4
kgdb: cannot read PTD
# kgdb kernel.debug /var/crash/vmcore.5
kgdb: cannot read PTD
# cat info.0
Dump header from device /dev/ad0s1b
Architecture: i386
Architecture Version: 2
Dump Length: 267976704B (255 MB)
Blocksize: 512
Dumptime: Mon May 22 01:24:01 2006
Hostname:
Magic: FreeBSD Kernel Dump
Version String: FreeBSD 6.1-RELEASE #0: Wed May 10 14:02:17 CEST 2006
Panic String: page fault
Dump Parity: 3252354328
Bounds: 0
Dump Status: good
# cat info.1
Dump header from device /dev/ad0s1b
Architecture: i386
Architecture Version: 2
Dump Length: 267976704B (255 MB)
Blocksize: 512
Dumptime: Fri Jun 16 18:01:27 2006
Hostname:
Magic: FreeBSD Kernel Dump
Version String: FreeBSD 6.1-RELEASE-p2 #2: Fri Jun 16 02:17:00 CEST 2006
Panic String: ffs_blkfree: freeing free block
Dump Parity: 3481480040
Bounds: 1
Dump Status: good
# cat info.2
Dump header from device /dev/ad0s1b
Architecture: i386
Architecture Version: 2
Dump Length: 267976704B (255 MB)
Blocksize: 512
Dumptime: Wed Jun 28 23:08:28 2006
Hostname:
Magic: FreeBSD Kernel Dump
Version String: FreeBSD 6.1-RELEASE-p2 #4: Mon Jun 26 14:36:29 CEST 2006
Panic String: sbflush_locked: cc 0 || mb 0xc23e7500 || mbcnt 2304
Dump Parity: 2760206361
Bounds: 2
Dump Status: good
# cat info.3
Dump header from device /dev/ad0s1b
Architecture: i386
Architecture Version: 2
Dump Length: 267976704B (255 MB)
Blocksize: 512
Dumptime: Thu Jun 29 09:25:44 2006
Hostname:
Magic: FreeBSD Kernel Dump
Version String: FreeBSD 6.1-RELEASE-p2 #4: Mon Jun 26 14:36:29 CEST 2006
Panic String: page fault
Dump Parity: 1992958036
Bounds: 3
Dump Status: good
# cat info.4
Dump header from device /dev/ad0s1b
Architecture: i386
Architecture Version: 2
Dump Length: 267976704B (255 MB)
Blocksize: 512
Dumptime: Sat Jul 1 14:02:53 2006
Hostname:
Magic: FreeBSD Kernel Dump
Version String: FreeBSD 6.1-RELEASE-p2 #4: Mon Jun 26 14:36:29 CEST 2006
Panic String: page fault
Dump Parity: 1663968596
Bounds: 4
Dump Status: good
# cat info.5
Dump header from device /dev/ad0s1b
Architecture: i386
Architecture Version: 2
Dump Length: 267976704B (255 MB)
Blocksize: 512
Dumptime: Sat Aug 12 03:01:25 2006
Hostname:
Magic: FreeBSD Kernel Dump
Version String: FreeBSD 6.1-RELEASE-p2 #4: Mon Jun 26 14:36:29 CEST 2006
Panic String: page fault
Dump Parity: 1801608788
Bounds: 5
Dump Status: good
-------------- next part --------------
# cat info.6
Dump header from device /dev/ad0s1b
Architecture: i386
Architecture Version: 2
Dump Length: 536412160B (511 MB)
Blocksize: 512
Dumptime: Fri Sep 1 21:44:58 2006
Hostname:
Magic: FreeBSD Kernel Dump
Version String: FreeBSD 6.1-RELEASE-p5 #0: Tue Aug 29 03:09:23 CEST 2006
Panic String: page fault
Dump Parity: 2714335061
Bounds: 6
Dump Status: good
# cat info.7
Dump header from device /dev/ad0s1b
Architecture: i386
Architecture Version: 2
Dump Length: 536412160B (511 MB)
Blocksize: 512
Dumptime: Fri Sep 1 21:49:55 2006
Hostname:
Magic: FreeBSD Kernel Dump
Version String: FreeBSD 6.1-RELEASE-p5 #0: Tue Aug 29 03:09:23 CEST 2006
Panic String: watchdog timeout
Dump Parity: 4123728435
Bounds: 7
Dump Status: good
-------------- next part --------------
# kgdb kernel.debug /var/crash/vmcore.6
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so:
Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "i386-marcel-freebsd".
Unread portion of the kernel message buffer:
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0xc
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc05a47bb
stack pointer = 0x28:0xd447db18
frame pointer = 0x28:0xd447db3c
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 24 (ath1 taskq)
trap number = 12
panic: page fault
Uptime: 1d7h56m6s
Dumping 511 MB (2 chunks)
chunk 0: 1MB (159 pages) ... ok
chunk 1: 511MB (130800 pages) 495 479 463 447 431 415 399 383 367 351 335 319
303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15
#0 doadump () at pcpu.h:165
165 __asm __volatile("movl %%fs:0,%0" : "=r"
(td));
(kgdb) backtrace
#0 doadump () at pcpu.h:165
#1 0xc056da25 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:402
#2 0xc056dcbc in panic (fmt=0xc07a4185 "%s") at
/usr/src/sys/kern/kern_shutdown.c:558
#3 0xc07658c8 in trap_fatal (frame=0xd447dad8, eva=12) at
/usr/src/sys/i386/i386/trap.c:836
#4 0xc076562f in trap_pfault (frame=0xd447dad8, usermode=0, eva=12) at
/usr/src/sys/i386/i386/trap.c:744
#5 0xc076528d in trap (frame {tf_fs = 8, tf_es = 40, tf_ds = 40, tf_edi =
-979275436, tf_esi = 370, tf_ebp = -733488324, tf_isp = -733488380, tf_ebx =
-979275520, tf_edx = 0, tf_ecx = -1012252656, tf_eax = 0, tf_trapno = 12, tf_err
= 0, tf_eip = -1067825221, tf_cs = 32, tf_eflags = 590342, tf_esp = 0, tf_ss =
-733488320})
at /usr/src/sys/i386/i386/trap.c:434
#6 0xc0754a6a in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#7 0xc05a47bb in m_copym (m=0x0, off0=1500, len=1480, wait=1) at
/usr/src/sys/kern/uipc_mbuf.c:400
#8 0xc06204b8 in ip_fragment (ip=0xc3aa4010, m_frag=0xd447dbec, mtu=-979275520,
if_hwassist_flags=0, sw_csum=1) at /usr/src/sys/netinet/ip_output.c:975
#9 0xc0610c4e in ip_fastforward (m=0xc3a7ba00) at
/usr/src/sys/netinet/ip_fastfwd.c:561
#10 0xc05de953 in ether_demux (ifp=0xc33ac400, m=0xc3a7ba00) at
/usr/src/sys/net/if_ethersubr.c:766
#11 0xc05de715 in ether_input (ifp=0xc33ac400, m=0xc3a7ba00) at
/usr/src/sys/net/if_ethersubr.c:620
#12 0xc05f5604 in ieee80211_deliver_data (ic=0xc33ad230, ni=0xc5602000,
m=0xc3a7ba00) at /usr/src/sys/net80211/ieee80211_input.c:717
#13 0xc05f507d in ieee80211_input (ic=0xc33ad230, m=0xc3a7ba00, ni=0xc5602000,
rssi=30, rstamp=27616) at /usr/src/sys/net80211/ieee80211_input.c:481
#14 0xc04afd66 in ath_rx_proc (arg=0xc33ad000, npending=1) at
/usr/src/sys/dev/ath/if_ath.c:2977
#15 0xc058de89 in taskqueue_run (queue=0xc32f4780) at
/usr/src/sys/kern/subr_taskqueue.c:217
#16 0xc058e03a in taskqueue_thread_loop (arg=0x0) at
/usr/src/sys/kern/subr_taskqueue.c:276
#17 0xc0558068 in fork_exit (callout=0xc058dff8 <taskqueue_thread_loop>,
arg=0xc33adee0, frame=0xd447dd38) at /usr/src/sys/kern/kern_fork.c:805
#18 0xc0754acc in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:208
(kgdb) quit
# kgdb kernel.debug /var/crash/vmcore.7
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so:
Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "i386-marcel-freebsd".
Unread portion of the kernel message buffer:
interrupt total
irq14: ata0 34029
irq17: ath0 329111
irq18: ath1 165989
irq19: rl0 15153
irq21: uhci0 ehci0 735
irq23: vr0 110085
cpu0: timer 443218
Total 1098320
panic: watchdog timeout
Uptime: 3m42s
Dumping 511 MB (2 chunks)
chunk 0: 1MB (159 pages) ... ok
chunk 1: 511MB (130800 pages) 495 479 463 447 431 415 399 383 367 351 335 319
303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15
#0 doadump () at pcpu.h:165
165 __asm __volatile("movl %%fs:0,%0" : "=r"
(td));
(kgdb) backtrace
#0 doadump () at pcpu.h:165
#1 0xc056da25 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:402
#2 0xc056dcbc in panic (fmt=0xc07ae613 "watchdog timeout") at
/usr/src/sys/kern/kern_shutdown.c:558
#3 0xc0543e86 in watchdog_fire () at /usr/src/sys/kern/kern_clock.c:583
#4 0xc05439a5 in hardclock (frame=0xc3884900) at
/usr/src/sys/kern/kern_clock.c:279
#5 0xc075a68d in lapic_handle_timer (frame {cf_vec = 0, cf_fs = 8, cf_es =
40, cf_ds = 40, cf_edi = -1015050576, cf_esi = -1018397184, cf_ebp = -660935872,
cf_ebx = -1018397184, cf_edx = 116, cf_ecx = -1065296032, cf_eax = -1416560640,
cf_eip = -1067795618, cf_cs = 32, cf_eflags = 663, cf_esp = -1015530112, cf_ss =
-1014478592})
at /usr/src/sys/i386/i386/local_apic.c:623
#6 0xc0755050 in Xtimerint () at apic_vector.s:137
#7 0xc05abb5e in sbcompress (sb=0xc37f8eb0, m=0xc34c7e00, n=0xc34c7e00) at
/usr/src/sys/kern/uipc_socket2.c:1091
#8 0xc05ab2fb in sbappendstream_locked (sb=0xc37f8eb0, m=0xc34c7e00) at
/usr/src/sys/kern/uipc_socket2.c:745
#9 0xc05ab345 in sbappendstream (sb=0xc37f8eb0, m=0xc34c7e00) at
/usr/src/sys/kern/uipc_socket2.c:761
#10 0xc0630fc0 in tcp_usr_send (so=0xc37f8de8, flags=0, m=0xc34c7e00, nam=0x0,
control=0x0, td=0xc3884900) at /usr/src/sys/netinet/tcp_usrreq.c:665
#11 0xc05a7c0b in sosend (so=0xc37f8de8, addr=0x0, uio=0xd89aecbc,
top=0xc34c7e00, control=0x0, flags=0, td=0xc3884900)
at /usr/src/sys/kern/uipc_socket.c:836
#12 0xc0596152 in soo_write (fp=0xab910000, uio=0xd89aecbc,
active_cred=0xc3783d80, flags=0, td=0xc3884900) at
/usr/src/sys/kern/sys_socket.c:118
#13 0xc05908bb in dofilewrite (td=0xc3884900, fd=3, fp=0xc37f9000,
auio=0xd89aecbc, offset=Unhandled dwarf expression opcode 0x93
) at file.h:246
#14 0xc059075f in kern_writev (td=0xc3884900, fd=3, auio=0xd89aecbc) at
/usr/src/sys/kern/sys_generic.c:402
#15 0xc0590685 in write (td=0xc3884900, uap=0xc080df60) at
/usr/src/sys/kern/sys_generic.c:326
#16 0xc0765bdf in syscall (frame {tf_fs = 59, tf_es = 59, tf_ds = 59,
tf_edi = 134715936, tf_esi = 134681128, tf_ebp = -1077943096, tf_isp =
-660935324, tf_ebx = 671931192, tf_edx = 134681128, tf_ecx = 3, tf_eax = 4,
tf_trapno = 0, tf_err = 2, tf_eip = 674475811, tf_cs = 51, tf_eflags = 646,
tf_esp = -1077943124, tf_ss = 59})
at /usr/src/sys/i386/i386/trap.c:981
#17 0xc0754abf in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:200
#18 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) quit
-------------- next part --------------
# cat /etc/sysctl.conf
# $FreeBSD: src/etc/sysctl.conf,v 1.8 2003/03/13 18:43:50 mux Exp $
#
# This file is read when going to multi-user and its contents piped thru
# ``sysctl'' to adjust kernel values. ``man 5 sysctl.conf''
for details.
#
# Uncomment this to prevent users from seeing information about processes that
# are being run under another UID.
#security.bsd.see_other_uids=0
# IFS parametry Atheros distance 1730m and 3440m
dev.ath.0.slottime=15
dev.ath.0.acktimeout=33
dev.ath.0.ctstimeout=33
dev.ath.0.txantenna=1
dev.ath.0.rxantenna=1
dev.ath.0.diversity=0
#dev.ath.0.tpscale=0
#dev.ath.0.tpc=0
dev.ath.1.slottime=21
dev.ath.1.acktimeout=45
dev.ath.1.ctstimeout=45
dev.ath.1.txantenna=1
dev.ath.1.rxantenna=1
dev.ath.1.diversity=0
#dev.ath.1.tpscale=0
#dev.ath.1.tpc=0
kern.ipc.maxsockbuf=2097152
#net.inet6.ip6.forwarding=1
net.inet.ip.fastforwarding=1
net.inet.tcp.sendspace=65536
#net.inet.tcp.rfc1323=1
net.inet.icmp.icmplim=2147483647
net.inet.icmp.icmplim_output=0
net.inet.ip.random_id=1
-------------- next part --------------
# ifconfig -v
ath0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
inet6 fe80::20b:6bff:fe35:509a%ath0 prefixlen 64 scopeid 0x1
inet 10.40.192.17 netmask 0xfffffffc broadcast 10.40.192.19
ether 00:0b:6b:35:50:9a
media: IEEE 802.11 Wireless Ethernet OFDM/24Mbps mode 11a
<flag0,adhoc> (OFDM/24Mbps)
status: associated
ssid PtPa channel 100 (5500) bssid 00:00:00:00:00:00
authmode OPEN privacy OFF deftxkey UNDEF powersavemode OFF
powersavesleep 100 txpowmax 38 txpower 63 rtsthreshold 2346
mcastrate 1 fragthreshold 2346 pureg protmode OFF wme burst
roaming AUTO bintval 100
AC_BE cwmin 4 cwmax 10 aifs 3 txopLimit 0 -acm ack
cwmin 4 cwmax 10 aifs 3 txopLimit 0 -acm
AC_BK cwmin 4 cwmax 10 aifs 7 txopLimit 0 -acm ack
cwmin 4 cwmax 10 aifs 7 txopLimit 0 -acm
AC_VI cwmin 3 cwmax 4 aifs 2 txopLimit 94 -acm ack
cwmin 3 cwmax 4 aifs 2 txopLimit 94 -acm
AC_VO cwmin 2 cwmax 3 aifs 2 txopLimit 47 -acm ack
cwmin 2 cwmax 3 aifs 2 txopLimit 47 -acm
ath1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
inet6 fe80::20b:6bff:fe2a:b8d5%ath1 prefixlen 64 scopeid 0x2
inet 10.40.64.17 netmask 0xfffffffc broadcast 10.40.64.19
ether 00:0b:6b:2a:b8:d5
media: IEEE 802.11 Wireless Ethernet OFDM/24Mbps mode 11a
<flag0,adhoc> (OFDM/24Mbps)
status: associated
ssid PtP channel 108 (5540) bssid 00:00:00:00:00:00
authmode OPEN privacy OFF deftxkey UNDEF powersavemode OFF
powersavesleep 100 txpowmax 37 txpower 63 rtsthreshold 2346
mcastrate 1 fragthreshold 2346 pureg protmode OFF wme burst
roaming AUTO bintval 100
AC_BE cwmin 4 cwmax 10 aifs 3 txopLimit 0 -acm ack
cwmin 4 cwmax 10 aifs 3 txopLimit 0 -acm
AC_BK cwmin 4 cwmax 10 aifs 7 txopLimit 0 -acm ack
cwmin 4 cwmax 10 aifs 7 txopLimit 0 -acm
AC_VI cwmin 3 cwmax 4 aifs 2 txopLimit 94 -acm ack
cwmin 3 cwmax 4 aifs 2 txopLimit 94 -acm
AC_VO cwmin 2 cwmax 3 aifs 2 txopLimit 47 -acm ack
cwmin 2 cwmax 3 aifs 2 txopLimit 47 -acm
-------------- next part --------------
# uname -a
FreeBSD * 6.1-RELEASE-p5 FreeBSD 6.1-RELEASE-p5 #0: Tue Aug 29 03:09:23 CEST
2006 *:/usr/obj/usr/src/sys/* i386
-------------- next part --------------
this ath0 is old CM9:
ath0@pci0:14:0: class=0x020000 card=0x1012185f chip=0x0013168c rev=0x01 hdr=0x00
vendor = 'Atheros Communications Inc.'
device = 'AR5212, AR5213 802.11a/b/g Wireless Adapter'
class = network
subclass = ethernet
this ath1 is new CM10:
ath1@pci0:19:0: class=0x020000 card=0x1600185f chip=0x001b168c rev=0x01 hdr=0x00
vendor = 'Atheros Communications Inc.'
class = network
subclass = ethernet
BTW: Does somebody know where device information is ? :)
-------------- next part --------------
Copyright (c) 1992-2006 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD 6.1-RELEASE-p5 #0: Tue Aug 29 03:09:23 CEST 2006
*:/usr/obj/usr/src/sys/*
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Duron(tm) (800.03-MHz 686-class CPU)
Origin = "AuthenticAMD" Id = 0x631 Stepping = 1
Features=0x183fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR>
AMD Features=0xc0440800<SYSCALL,<b18>,MMX+,3DNow+,3DNow>
real memory = 536805376 (511 MB)
avail memory = 515940352 (492 MB)
ACPI APIC Table: <AMIINT VIA_K7 >
ioapic0 <Version 0.3> irqs 0-23 on motherboard
wlan: mac acl policy registered
kbd1 at kbdmux0
netsmb_dev: loaded
ath_hal: 0.9.16.16 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
acpi0: <AMIINT VIA_K7> on motherboard
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
agp0: <VIA 8377 (Apollo KT400/KT400A/KT600) host to PCI bridge> mem
0xe0000000-0xe1ffffff at device 0.0 on pci0
pcib1: <PCI-PCI bridge> at device 1.0 on pci0
pci1: <PCI bus> on pcib1
pci1: <display, VGA> at device 0.0 (no driver attached)
ath0: <Atheros 5212> mem 0xdfff0000-0xdfffffff irq 17 at device 6.0 on
pci0
ath0: Ethernet address: 00:0b:6b:35:50:9a
ath0: mac 5.9 phy 4.3 radio 3.6
ath1: <Atheros 5212> mem 0xdffe0000-0xdffeffff irq 18 at device 7.0 on
pci0
ath1: Ethernet address: 00:0b:6b:2a:b8:d5
ath1: mac 10.5 phy 6.1 radio 6.3
rl0: <RealTek 8139 10/100BaseTX> port 0xec00-0xecff mem
0xdffdff00-0xdffdffff irq 19 at device 8.0 on pci0
miibus0: <MII bus> on rl0
rlphy0: <RealTek internal media interface> on miibus0
rlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
rl0: Ethernet address: 00:50:fc:e4:c4:b6
uhci0: <VIA 83C572 USB controller> port 0xe800-0xe81f irq 21 at device
16.0 on pci0
uhci0: [GIANT-LOCKED]
usb0: <VIA 83C572 USB controller> on uhci0
usb0: USB revision 1.0
uhub0: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
ehci0: <VIA VT6202 USB 2.0 controller> mem 0xdffdfe00-0xdffdfeff irq 21 at
device 16.3 on pci0
ehci0: [GIANT-LOCKED]
usb1: EHCI version 1.0
usb1: companion controller, 2 ports each: usb0
usb1: <VIA VT6202 USB 2.0 controller> on ehci0
usb1: USB revision 2.0
uhub1: VIA EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
isab0: <PCI-ISA bridge> at device 17.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <VIA 8235 UDMA133 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xfc00-0xfc0f at device 17.1 on pci0
ata0: <ATA channel 0> on atapci0
ata1: <ATA channel 1> on atapci0
vr0: <VIA VT6102 Rhine II 10/100BaseTX> port 0xe400-0xe4ff mem
0xdffdfd00-0xdffdfdff irq 23 at device 18.0 on pci0
miibus1: <MII bus> on vr0
ukphy0: <Generic IEEE 802.3u media interface> on miibus1
ukphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
vr0: Ethernet address: 00:0c:76:1c:aa:b7
acpi_button1: <Sleep Button> on acpi0
pmtimer0 on isa0
orm0: <ISA Option ROM> at iomem 0xc0000-0xcb7ff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
ugen0: American Power Conversion Back-UPS RS 500 FW:30.j5.I USB FW:j5, rev
1.10/0.06, addr 2
Timecounter "TSC" frequency 800032669 Hz quality 800
Timecounters tick every 1.000 msec
IPv6 packet filtering initialized, default to accept, logging limited to 100
packets/entry
ipfw2 (+ipv6) initialized, divert loadable, rule-based forwarding enabled,
default to accept, logging limited to 100 packets/entry by default
ad0: 76319MB <WDC WD800JB-00CRA1 17.07W17> at ata0-master UDMA100
Trying to mount root from ufs:/dev/ad0s1a
Sam Leffler
2006-Sep-02 03:59 UTC
atheros driver under high load, panics and even more freezes
Daniel Dvo??k wrote:> Hi all, > > first of all, I?m sorry maybe for my bad English. > > We have 2 routers which I maintain in our mesh wireless community network. > > The Router 1 has 2 atheros adapters, ath0=wistron cm9, ath1=wistron cm10, of > course some sisX, fxpX and so on. > The Router 2 has 1 atheros adapter, ath1=wistron CM10. > > My R1 panics and even more it freezes very often. Maybe the reason for > panicing and freezing is the same and maybe not. > > I started (only after vmcore.5, so vmcore.6 is with this option) to use > "option SW_WATCHDOG" in both my custom kernels on the R1 and R2 recently in > hope, it is some walkaround for freezing at least if not for panicing. > > This router was installed on the 1st of April 2006. > > Statistics: > > 9 panics with 8 kernel dumps, 1 missed > > 10 freezes > > I think that all panics some how connected to athX taskq process, page fault > in kernel panic and sbflush_locked.Why?> > I guess that panic comes when router transmits and receives datas at the > maximum throughput for setted nominal media rate speed, exactly 24Mbps, more > I do not use, because there are problems with quagga > > ospfd packets, it is known issue. > > Today I did a small test with throughput. > > Router 1 executed this command: > > # ping -i 0.001 -c 100000 -s 1472 ANY IP > > As you see, it is not even flood ping, it is almost flood, but not flood. > > Throughput was about 1,13-1,2 MB/s as bmon showed me. I notice there is not > any qos and icmp.limit is so high net.inet.icmp.icmplim: 2147483647 > net.ineticmp.icmplim_output: 0. > > > First 5 s latency was about 1,1-1,7 ms > After it goes to 10-30, 50-70, 110-130, 270-300, up 300ms and packet loss > > .... some seconds .... > > panic<...lots of stuff deleted...> Sounds like a resource leak to me. You've got crash dumps, look at memory usage with vmstat and/or netstat. Past that it sounds like you're running 6.1 RELEASE which is now 6+ months old. Many bugs have been fixed including, I believe, some resource-related ones. Please try 6-STABLE. Sam
Daniel Dvořák
2006-Sep-06 03:45 UTC
atheros driver under high load, panics and even more freezes
And I forgot to attach netstat -m Daniel> -----Original Message----- > From: Sam Leffler [mailto:sam@errno.com] > Sent: Sunday, September 03, 2006 6:41 PM > To: dandee@volny.cz > Cc: freebsd-stable@freebsd.org > Subject: Re: atheros driver under high load, panics and even > more freezes > > Daniel Dvo??k wrote: > > Ok, I will upgrade my boxes and I will do simple ping tests again. > > > > Did you see my sysctl.conf file ? > > > > I mean these options: > > > > kern.ipc.maxsockbuf=2097152 > > net.inet.ip.fastforwarding=1 > > net.inet.tcp.sendspace=65536 > > > > Could be this connected with increasing latency up to 500ms ? > > Seems unlikely but I have little to go on. You can easily > identify whether the delays are in the OS or due to wireless > issues by sniffing traffic. Tools like athstats are also > important for diagnosing problems. > > Sam >-------------- next part -------------- before the test: 180/1005/1185 mbufs in use (current/cache/total) 172/420/592/7872 mbuf clusters in use (current/cache/total/max) 169/215 mbuf+clusters out of packet secondary zone in use (current/cache) 0/0/0/0 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/0 9k jumbo clusters in use (current/cache/total/max) 0/0/0/0 16k jumbo clusters in use (current/cache/total/max) 389K/1091K/1480K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/7/2224 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 488 calls to protocol drain routines during the test: 702/483/1185 mbufs in use (current/cache/total) 317/275/592/7872 mbuf clusters in use (current/cache/total/max) 168/216 mbuf+clusters out of packet secondary zone in use (current/cache) 0/0/0/0 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/0 9k jumbo clusters in use (current/cache/total/max) 0/0/0/0 16k jumbo clusters in use (current/cache/total/max) 809K/670K/1480K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/7/2224 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 488 calls to protocol drain routines after the test: 181/1004/1185 mbufs in use (current/cache/total) 171/421/592/7872 mbuf clusters in use (current/cache/total/max) 168/216 mbuf+clusters out of packet secondary zone in use (current/cache) 0/0/0/0 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/0 9k jumbo clusters in use (current/cache/total/max) 0/0/0/0 16k jumbo clusters in use (current/cache/total/max) 387K/1093K/1480K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/7/2224 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 488 calls to protocol drain routines
Daniel Dvořák
2006-Sep-06 03:51 UTC
atheros driver under high load, panics and even more freezes
And there is diffrence even between -i 0.002 and 0.0019 0.002 is okay 1,4ms 0,0019 means "no buffers, PL, big latency, OACTIVE flag" with -f no problem 0% PL and avg latency 1,5 ms and the throughput 0,89MB/s IN and OUT. Daniel I am sorry for dividing to 3 emails.> -----Original Message----- > From: Sam Leffler [mailto:sam@errno.com] > Sent: Sunday, September 03, 2006 6:41 PM > To: dandee@volny.cz > Cc: freebsd-stable@freebsd.org > Subject: Re: atheros driver under high load, panics and even > more freezes > > Daniel Dvo??k wrote: > > Ok, I will upgrade my boxes and I will do simple ping tests again. > > > > Did you see my sysctl.conf file ? > > > > I mean these options: > > > > kern.ipc.maxsockbuf=2097152 > > net.inet.ip.fastforwarding=1 > > net.inet.tcp.sendspace=65536 > > > > Could be this connected with increasing latency up to 500ms ? > > Seems unlikely but I have little to go on. You can easily > identify whether the delays are in the OS or due to wireless > issues by sniffing traffic. Tools like athstats are also > important for diagnosing problems. > > Sam >