I have a box running 9.0-RELEASE where I'm seeing a panic happen every
5-7 days. For the record, it's moving about 80-100 mbit/s of network
traffic and has several gre tunnels setup. The box has panic'd many
times, but due to unrelated (serial port) issues, I've only been able to
get a complete panic once.
I took a look at the core and did some basic debugging:
# kgdb kernel.debug /var/crash/vmcore.0
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "i386-marcel-freebsd"...
Unread portion of the kernel message buffer:
Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 06
fault virtual address = 0x44
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc0ae4eae
stack pointer = 0x28:0xe0f00ab0
frame pointer = 0x28:0xe0f00b38
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 2372 (snmpd)
trap number = 12
panic: page fault
cpuid = 2
KDB: stack backtrace:
#0 0xc0a50a47 at kdb_backtrace+0x47
#1 0xc0a1dfa7 at panic+0x117
#2 0xc0d5a243 at trap_fatal+0x323
#3 0xc0d5a2fd at trap_pfault+0xad
#4 0xc0d5b085 at trap+0x465
#5 0xc0d43fdc at calltrap+0x6
#6 0xc0a27aca at sysctl_root+0x1fa
#7 0xc0a27d83 at userland_sysctl+0x1d3
#8 0xc0a28144 at sys___sysctl+0x94
#9 0xc0d5a865 at syscall+0x355
#10 0xc0d44041 at Xint0x80_syscall+0x21
Uptime: 6d7h1m32s
Physical memory: 3567 MB
Dumping 334 MB: 319 303 287 271 255 239 223 207 191 175 159 143 127 111
95 79 63 47 31 15
Reading symbols from /boot/kernel/pf.ko...Reading symbols from
/boot/kernel/pf.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/pf.ko
Reading symbols from /boot/kernel/if_gre.ko...Reading symbols from
/boot/kernel/if_gre.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/if_gre.ko
#0 doadump (textdump=1) at pcpu.h:244
244 __asm("movl %%fs:0,%0" : "=r" (td));
(kgdb) up
#1 0xc0a1dd4a in kern_reboot (howto=260) at
/usr/src/sys/kern/kern_shutdown.c:442
442 doadump(TRUE);
(kgdb) up
#2 0xc0a1dfe1 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:607
607 kern_reboot(bootopt);
(kgdb) up
#3 0xc0d5a243 in trap_fatal (frame=0xe0f00a70, eva=68) at
/usr/src/sys/i386/i386/trap.c:975
975 panic("%s", trap_msg[type]);
(kgdb) up
#4 0xc0d5a2fd in trap_pfault (frame=0xe0f00a70, usermode=0, eva=68) at
/usr/src/sys/i386/i386/trap.c:839
839 trap_fatal(frame, eva);
(kgdb) up
#5 0xc0d5b085 in trap (frame=0xe0f00a70) at
/usr/src/sys/i386/i386/trap.c:558
558 (void) trap_pfault(frame, FALSE, eva);
(kgdb) up
#6 0xc0d43fdc in calltrap () at /usr/src/sys/i386/i386/exception.s:168
168 call trap
Current language: auto; currently asm
(kgdb) up
#7 0xc0ae4eae in sysctl_rtsock (oidp=0xc1031560, arg1=0xe0f00c08,
arg2=4, req=0xe0f00b94) at /usr/src/sys/net/rtsock.c:1594
1594 ifam->ifam_index
ifa->ifa_ifp->if_index;
Current language: auto; currently c
(kgdb) i li 1594
Line 1594 of "/usr/src/sys/net/rtsock.c" starts at address 0xc0ae4eab
<sysctl_rtsock+1035> and ends at 0xc0ae4eb6 <sysctl_rtsock+1046>.
(kgdb) disas 0xc0ae4eab 0xc0ae4eb6
Dump of assembler code from 0xc0ae4eab to 0xc0ae4eb6:
0xc0ae4eab <sysctl_rtsock+1035>: mov 0x5c(%ebx),%eax
0xc0ae4eae <sysctl_rtsock+1038>: movzwl 0x44(%eax),%eax
0xc0ae4eb2 <sysctl_rtsock+1042>: mov %ax,0xc(%edx)
End of assembler dump.
(kgdb) p *(struct ifaddr *)$ebx
$1 = {ifa_addr = 0xc827c7a8, ifa_dstaddr = 0xc827c7b8, ifa_netmask 0xc77c8ca8,
if_data = {ifi_type = 1 '\001', ifi_physical = 13 '\r',
ifi_addrlen = 0 '\0', ifi_hdrlen = 0 '\0', ifi_link_state = 0
'\0',
ifi_spare_char1 = 0 '\0',
ifi_spare_char2 = 0 '\0', ifi_datalen = 0 '\0', ifi_mtu
3426383120, ifi_metric = 0, ifi_baudrate = 0, ifi_ipackets = 3346381610,
ifi_ierrors = 284187, ifi_opackets = 4294901815, ifi_oerrors = 0,
ifi_collisions = 0,
ifi_ibytes = 9385256, ifi_obytes = 4620, ifi_imcasts = 0,
ifi_omcasts = 3358050108, ifi_iqdrops = 4294967295, ifi_noproto 4294967295,
ifi_hwassist = 4294967295, ifi_epoch = 0, ifi_lastchange {tv_sec = 0, tv_usec =
0}},
ifa_ifp = 0x0, ifa_link = {tqe_next = 0x0, tqe_prev = 0xc827c760},
ifa_rtrequest = 0, ifa_flags = 51048, ifa_refcnt = 25499, ifa_metric -936917136,
ifa_claim_addr = 0, ifa_mtx = {lock_object = {lo_name 0xc827c778 "",
lo_flags = 0, lo_data = 3358050176, lo_witness = 0x0}, mtx_lock = 6}}
(kgdb)
Sorry for the bad formatting there, but it seems like ifa->ifa_ifp is
null. For the record, net-snmpd is being polled every 60 seconds. I see
that the locking in this file has changed a little since 9.0-RELEASE, so
I'm currently working on upgrading it to 9.0-STABLE in hopes of avoiding
the problem. Any comments would be appreciated. I'll also submit a PR in
case that's helpful.
Thanks,
Steve