Pawel Malachowski
2003-Nov-23 09:21 UTC
Panic on 4.9-PRE (mbuf/m_copydata/ippr_ftp_process related)
Hello,
My router caught kernel panic after 11 days of working. System
is 4.9-PRERELEASE and runs without problems since 28 Sep 2003.
There are two outgoing interfaces and about 10 internal interfaces
(mostly vlans); ipfw2 fwd is used to help with routing a bit (2 ISPs,
no BGP); dummynet shaping happens at external and some of internal
devices:
% ipfw pipe show | wc -l
1134
I will update to recent 4.9-STABLE, however I'm posting backtrace here
cause it may be hard for me to reproduce this panic (this is the first
time I'm seeing it). Any ideas what should I look for?
Here comes backtrace:
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0xc
fault code = supervisor read, page not present
instruction pointer = 0x8:0xc0255b50
stack pointer = 0x10:0xce4a2cbc
frame pointer = 0x10:0xce4a2cc8
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 76908 (trafd)
interrupt mask = net tty
trap number = 12
panic: page fault
syncing disks...
done
Uptime: 11d4h46m8s
dumping to dev #ad/0x30001, offset 1573024
dump ata0: resetting devices .. done
[...]
---
#0 dumpsys () at ../../kern/kern_shutdown.c:487
487 if (dumping++) {
(kgdb) bt
#0 dumpsys () at ../../kern/kern_shutdown.c:487
#1 0xc0238d33 in boot (howto=256) at ../../kern/kern_shutdown.c:316
#2 0xc0239158 in poweroff_wait (junk=0xc043516c, howto=-1069331345)
at ../../kern/kern_shutdown.c:595
#3 0xc03b0d8a in trap_fatal (frame=0xce4a2c7c, eva=12)
at ../../i386/i386/trap.c:974
#4 0xc03b0a5d in trap_pfault (frame=0xce4a2c7c, usermode=0, eva=12)
at ../../i386/i386/trap.c:867
#5 0xc03b061b in trap (frame={tf_fs = 2097168, tf_es = 16, tf_ds = -834011120,
tf_edi = 2, tf_esi = 0, tf_ebp = -833999672, tf_isp = -833999704,
tf_ebx = 2, tf_edx = -1032830968, tf_ecx = 120, tf_eax = 0, tf_trapno =
12,
tf_err = 0, tf_eip = -1071293616, tf_cs = 8, tf_eflags = 66050, tf_esp =
2,
tf_ss = -1032830860}) at ../../i386/i386/trap.c:466
#6 0xc0255b50 in m_copydata (m=0xc1178700, off=120, len=2, cp=0xc2704078
"")
at ../../kern/uipc_mbuf.c:985
#7 0xc0157b10 in ippr_ftp_process (fin=0xce4a2df0, ip=0xc11b9810,
nat=0xc1d1dc00,
ftp=0xc2704000, rv=0) at ../../contrib/ipfilter/netinet/ip_ftp_pxy.c:1052
#8 0xc0157cd6 in ippr_ftp_out (fin=0xce4a2df0, ip=0xc11b9810, aps=0xc1aa7a80,
nat=0xc1d1dc00) at ../../contrib/ipfilter/netinet/ip_ftp_pxy.c:1165
#9 0xc01591e1 in appr_check (ip=0xc11b9810, fin=0xce4a2df0, nat=0xc1d1dc00)
at ../../contrib/ipfilter/netinet/ip_proxy.c:341
#10 0xc0156426 in ip_natout (ip=0xc11b9810, fin=0xce4a2df0)
at ../../contrib/ipfilter/netinet/ip_nat.c:2555
#11 0xc014f4ea in fr_check (ip=0xc11b9810, hlen=20, ifp=0xc15c6800, out=1,
mp=0xce4a2ea0) at ../../contrib/ipfilter/netinet/fil.c:1154
#12 0xc0295bb9 in ip_output (m0=0xc2541c80, opt=0x0, ro=0xc2541cac, flags=1,
imo=0x0, inp=0x0) at ../../netinet/ip_output.c:964
#13 0xc17edb18 in ?? ()
#14 0xc17eddce in ?? ()
#15 0xc17ee267 in ?? ()
#16 0xc023edb1 in softclock () at ../../kern/kern_timeout.c:131
#17 0xc03a3543 in doreti_swi ()
#18 0x8049c91 in ?? ()
#19 0x8049edf in ?? ()
#20 0x804a5e6 in ?? ()
#21 0x804b530 in ?? ()
#22 0x28079e89 in ?? ()
#23 0x280799db in ?? ()
#24 0x80498b6 in ?? ()
#25 0x804926d in ?? ()
(kgdb) up 6
#6 0xc0255b50 in m_copydata (m=0xc1178700, off=120, len=2, cp=0xc2704078
"")
at ../../kern/uipc_mbuf.c:985
985 while (len > 0) {
(kgdb) list
980 if (off < m->m_len)
981 break;
982 off -= m->m_len;
983 m = m->m_next;
984 }
985 while (len > 0) {
986 KASSERT(m != NULL, ("m_copydata, length > size
of mbuf chain"));
987 count = min(m->m_len - off, len);
988 bcopy(mtod(m, caddr_t) + off, cp, count);
989 len -= count;
(kgdb) p m
$1 = (struct mbuf *) 0x0
(kgdb) up
#7 0xc0157b10 in ippr_ftp_process (fin=0xce4a2df0, ip=0xc11b9810,
nat=0xc1d1dc00,
ftp=0xc2704000, rv=0) at ../../contrib/ipfilter/netinet/ip_ftp_pxy.c:1052
1052 m_copydata(m, off, len, wptr);
(kgdb) list
1047 bcopy((char *)m + off, wptr, len);
1048 #else
1049 # if SOLARIS
1050 copyout_mblk(m, off, len, wptr);
1051 # else
1052 m_copydata(m, off, len, wptr);
1053 # endif
1054 #endif
1055 mlen -= len;
1056 off += len;
(kgdb) p m
$2 = (mb_t *) 0xc1178700
% netstat -m -M vmcore.26 -N /kernel.debug
156/2816/10048 mbufs in use (current/peak/max):
156 mbufs allocated to data
130/1656/2512 mbuf clusters in use (current/peak/max)
4016 Kbytes allocated to network (53% of mb_map in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
ipnat rules are like this:
map rl0 172.27.192.0/20 -> x/32 proxy port ftp ftp/tcp
map rl0 172.27.192.0/20 -> x/32 portmap tcp/udp auto
map rl0 172.27.192.0/20 -> x/32
map rl0 10.0.0.0/8 -> y/32 proxy port ftp ftp/tcp
map rl0 10.0.0.0/8 -> y/32 portmap tcp/udp auto
map rl0 10.0.0.0/8 -> y/32
map fxp0 172.27.192.0/20 -> z/32 proxy port ftp ftp/tcp
map fxp0 172.27.192.0/20 -> z/32 portmap tcp/udp auto
map fxp0 172.27.192.0/20 -> z/32
map fxp0 10.0.0.0/8 -> z/32 proxy port ftp ftp/tcp
map fxp0 10.0.0.0/8 -> z/32 portmap tcp/udp auto
map fxp0 10.0.0.0/8 -> z/32
map rl0 127.0.0.1/32 -> x/32 proxy port ftp ftp/tcp
map rl0 127.0.0.1/32 -> x/32 portmap tcp/udp auto
map rl0 127.0.0.1/32 -> x/32
map fxp0 127.0.0.1/32 -> z/32 proxy port ftp ftp/tcp
map fxp0 127.0.0.1/32 -> z/32 portmap tcp/udp auto
map fxp0 127.0.0.1/32 -> z/32
rdr fxp0 from SBD1/32 to z/32 port = XXX -> 10.1.X.X port XXX tcp
rdr rl0 from SBD2/32 to x/32 port = XXX -> 10.1.X.X port XXX tcp
rdr fxp0 from any to z/32 port = X -> 10.1.X.X port X tcp
.
.
. (similar rdrs)
.
.
rdr xl0 from X/24 to any port = 80 -> 172.27.X.X port 81 tcp
TIA,
--
Pawe? Ma?achowski
David Malone
2003-Nov-23 15:25 UTC
Panic on 4.9-PRE (mbuf/m_copydata/ippr_ftp_process related)
On Sun, Nov 23, 2003 at 06:22:59PM +0100, Pawel Malachowski wrote:> My router caught kernel panic after 11 days of working. System > is 4.9-PRERELEASE and runs without problems since 28 Sep 2003.Is 28 Sep the date of your 4.9-PRERELEASE? There was an important memory corruption bug in ipfw2 fixed before the 4.9 RELEASE (on 17/10/2003). The bug was triggered by dynamic rules, so if you're using dynamic rules then it is definitely worth trying to upgrade to 4.9-RELEASE. David.