Hello,
My server is experiencing occasional kernel panics when under moderate
load. I'm attaching a crash dump and the dmesg output. I'm not sure how
to read the kernel backtrace but it looks like the Intel NIC (em0)
caused the problem. Occasionally, I used to get a "em0: watchdog timeout
-- resetting" error message, but I never had a kernel panic. The problem
started last week when I updated the server to the latest 7-STABLE. Can
anyone help pinpoint the problem?
Thanks,
Yani
kgdb output:
[GDB will not be able to debug user-mode threads:
/usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "i386-marcel-freebsd".
There is no member named pathname.
Reading symbols from /boot/kernel/linux.ko...Reading symbols from
/boot/kernel/linux.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/linux.ko
Reading symbols from /boot/kernel/miibus.ko...Reading symbols from
/boot/kernel/miibus.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/miibus.ko
Reading symbols from /boot/kernel/if_vr.ko...Reading symbols from
/boot/kernel/if_vr.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/if_vr.ko
Reading symbols from /boot/kernel/accf_http.ko...Reading symbols from
/boot/kernel/accf_http.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/accf_http.ko
Reading symbols from /boot/kernel/bridgestp.ko...Reading symbols from
/boot/kernel/bridgestp.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/bridgestp.ko
Reading symbols from /boot/kernel/if_bridge.ko...Reading symbols from
/boot/kernel/if_bridge.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/if_bridge.ko
Reading symbols from /boot/kernel/wlan_wep.ko...Reading symbols from
/boot/kernel/wlan_wep.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/wlan_wep.ko
Reading symbols from /boot/kernel/wlan_tkip.ko...Reading symbols from
/boot/kernel/wlan_tkip.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/wlan_tkip.ko
Reading symbols from /boot/kernel/wlan_ccmp.ko...Reading symbols from
/boot/kernel/wlan_ccmp.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/wlan_ccmp.ko
Reading symbols from /boot/kernel/wlan_xauth.ko...Reading symbols from
/boot/kernel/wlan_xauth.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/wlan_xauth.ko
Reading symbols from /boot/kernel/wlan_acl.ko...Reading symbols from
/boot/kernel/wlan_acl.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/wlan_acl.ko
Reading symbols from /boot/kernel/aio.ko...Reading symbols from
/boot/kernel/aio.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/aio.ko
Reading symbols from /boot/kernel/acpi.ko...Reading symbols from
/boot/kernel/acpi.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/acpi.ko
Reading symbols from /boot/kernel/linprocfs.ko...Reading symbols from
/boot/kernel/linprocfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/linprocfs.ko
Reading symbols from /boot/kernel/nfslockd.ko...Reading symbols from
/boot/kernel/nfslockd.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/nfslockd.ko
Reading symbols from /boot/kernel/krpc.ko...Reading symbols from
/boot/kernel/krpc.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/krpc.ko
Unread portion of the kernel message buffer:
em0: watchdog timeout -- resetting
<5>em0: link state changed to DOWN
kernel trap 12 with interrupts disabled
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0x14
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc05bc167
stack pointer = 0x28:0xe403ec10
frame pointer = 0x28:0xe403ec2c
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = resume, IOPL = 0
current process = 23 (em0 taskq)
trap number = 12
panic: page fault
Uptime: 1d2h53m51s
Physical memory: 1015 MB
Dumping 224 MB: 209 193 177 161 145 129 113 97 81 65 49 33 17 1
#0 doadump () at pcpu.h:195
195 __asm __volatile("movl %%fs:0,%0" : "=r"
(td));
(kgdb) list
190 static __inline struct thread *
191 __curthread(void)
192 {
193 struct thread *td;
194
195 __asm __volatile("movl %%fs:0,%0" : "=r"
(td));
196 return (td);
197 }
198 #define curthread (__curthread())
199
(kgdb) backtrace
#0 doadump () at pcpu.h:195
#1 0xc058ad84 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418
#2 0xc058af84 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:572
#3 0xc07ccb5c in trap_fatal (frame=0xe403ebd0, eva=20)
at /usr/src/sys/i386/i386/trap.c:899
#4 0xc07cd4dd in trap (frame=0xe403ebd0) at
/usr/src/sys/i386/i386/trap.c:280
#5 0xc07b721b in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#6 0xc05bc167 in propagate_priority (td=0xc3ab7880)
at /usr/src/sys/kern/subr_turnstile.c:272
#7 0xc05bcab8 in turnstile_wait (ts=0xc3a9d370, owner=0xc3ab7880,
queue=Variable "queue" is not available.
)
at /usr/src/sys/kern/subr_turnstile.c:739
#8 0xc057e88d in _mtx_lock_sleep (m=0xc3baa2f4, tid=3283517440, opts=0,
file=0x0, line=0) at /usr/src/sys/kern/kern_mutex.c:416
#9 0xc04b02d7 in em_handle_rxtx (context=0xc3baa000, pending=1)
at /usr/src/sys/dev/em/if_em.c:1666
#10 0xc05bb032 in taskqueue_run (queue=0xc3b94100)
at /usr/src/sys/kern/subr_taskqueue.c:255
#11 0xc05bb20d in taskqueue_thread_loop (arg=0xc3baa368)
at /usr/src/sys/kern/subr_taskqueue.c:374
#12 0xc0569d34 in fork_exit (callout=0xc05bb190 <taskqueue_thread_loop>,
arg=0xc3baa368, frame=0xe403ed38) at /usr/src/sys/kern/kern_fork.c:783
#13 0xc07b7290 in fork_trampoline () at
/usr/src/sys/i386/i386/exception.s:205
dmesg:
Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.0-STABLE #11: Tue May 6 15:34:11 EEST 2008
root@techserver.pi-greece.eu:/usr/obj/usr/src/sys/TECHSERVER7
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Sempron(tm) 2300+ (1585.75-MHz 686-class CPU)
Origin = "AuthenticAMD" Id = 0x681 Stepping = 1
Features=0x383fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE>
AMD Features=0xc0480800<SYSCALL,MP,MMX+,3DNow!+,3DNow!>
real memory = 1073676288 (1023 MB)
avail memory = 1041485824 (993 MB)
ACPI APIC Table: <GBT AWRDACPI>
ioapic0 <Version 0.3> irqs 0-23 on motherboard
wlan: mac acl policy registered
kbd1 at kbdmux0
acpi0: <GBT AWRDACPI> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, 3fef0000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
agp0: <VIA 8377 (Apollo KT400/KT400A/KT600) host to PCI bridge> on hostb0
agp0: aperture size is 32M
pcib1: <PCI-PCI bridge> at device 1.0 on pci0
pci1: <PCI bus> on pcib1
vgapci0: <VGA-compatible display> port 0xa000-0xa0ff mem
0xd0000000-0xd7ffffff,0xe0000000-0xe000ffff at device 0.0 on pci1
em0: <Intel(R) PRO/1000 Network Connection 6.9.0> port 0xb000-0xb03f mem
0xe1000000-0xe101ffff,0xe1020000-0xe103ffff irq 17 at device 9.0 on pci0
em0: [FILTER]
em0: Ethernet address: 00:07:e9:49:18:b3
aac0: <HP ML110 G2 (Adaptec 2610SA)> mem 0xd8000000-0xdbffffff irq 18 at
device 10.0 on pci0
aac0: Enable Raw I/O
aac0: New comm. interface enabled
aac0: [ITHREAD]
aac0: Adaptec 2610SA, aac driver 2.0.0-1
ahc0: <Adaptec 2940 Ultra SCSI adapter> port 0xb400-0xb4ff mem
0xe1042000-0xe1042fff irq 19 at device 11.0 on pci0
ahc0: [ITHREAD]
aic7880: Ultra Wide Channel A, SCSI Id=7, 16/253 SCBs
ral0: <Ralink Technology RT2560> mem 0xe1040000-0xe1041fff irq 17 at
device 13.0 on pci0
ral0: MAC/BBP RT2560 (rev 0x04), RF RT2525
ral0: Ethernet address: 00:11:50:14:b7:66
ral0: [ITHREAD]
atapci0: <VIA 6420 SATA150 controller> port
0xb800-0xb807,0xbc00-0xbc03,0xc000-0xc007,0xc400-0xc403,0xc800-0xc80f,0xcc00-0xccff
irq 20 at device 15.0 on pci0
atapci0: [ITHREAD]
ata2: <ATA channel 0> on atapci0
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci0
ata3: [ITHREAD]
atapci1: <VIA 8237 UDMA133 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xd000-0xd00f at device 15.1 on pci0
ata0: <ATA channel 0> on atapci1
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci1
ata1: [ITHREAD]
uhci0: <VIA 83C572 USB controller> port 0xd400-0xd41f irq 21 at device
16.0 on pci0
uhci0: [GIANT-LOCKED]
uhci0: [ITHREAD]
usb0: <VIA 83C572 USB controller> on uhci0
usb0: USB revision 1.0
uhub0: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 2 ports with 2 removable, self powered
uhci1: <VIA 83C572 USB controller> port 0xd800-0xd81f irq 21 at device
16.1 on pci0
uhci1: [GIANT-LOCKED]
uhci1: [ITHREAD]
usb1: <VIA 83C572 USB controller> on uhci1
usb1: USB revision 1.0
uhub1: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1
uhub1: 2 ports with 2 removable, self powered
uhci2: <VIA 83C572 USB controller> port 0xdc00-0xdc1f irq 21 at device
16.2 on pci0
uhci2: [GIANT-LOCKED]
uhci2: [ITHREAD]
usb2: <VIA 83C572 USB controller> on uhci2
usb2: USB revision 1.0
uhub2: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb2
uhub2: 2 ports with 2 removable, self powered
uhci3: <VIA 83C572 USB controller> port 0xe000-0xe01f irq 21 at device
16.3 on pci0
uhci3: [GIANT-LOCKED]
uhci3: [ITHREAD]
usb3: <VIA 83C572 USB controller> on uhci3
usb3: USB revision 1.0
uhub3: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb3
uhub3: 2 ports with 2 removable, self powered
ehci0: <VIA VT6202 USB 2.0 controller> mem 0xe1043000-0xe10430ff irq 21
at device 16.4 on pci0
ehci0: [GIANT-LOCKED]
ehci0: [ITHREAD]
usb4: EHCI version 1.0
usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3
usb4: <VIA VT6202 USB 2.0 controller> on ehci0
usb4: USB revision 2.0
uhub4: <VIA EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb4
uhub4: 8 ports with 8 removable, self powered
isab0: <PCI-ISA bridge> at device 17.0 on pci0
isa0: <ISA bus> on isab0
vr0: <VIA VT6102 Rhine II 10/100BaseTX> port 0xe400-0xe4ff mem
0xe1044000-0xe10440ff irq 23 at device 18.0 on pci0
vr0: Quirks: 0x0
vr0: Revision: 0x78
miibus0: <MII bus> on vr0
ukphy0: <Generic IEEE 802.3u media interface> PHY 1 on miibus0
ukphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
vr0: Ethernet address: 00:0f:ea:e0:eb:cc
vr0: [ITHREAD]
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on
acpi0
fdc0: [FILTER]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on
acpi0
sio0: type 16550A
sio0: [FILTER]
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
sio1: [FILTER]
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: [ITHREAD]
psm0: model IntelliMouse Explorer, device ID 4
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem
0xc0000-0xcafff,0xcc000-0xccfff,0xcd000-0xd0fff,0xd1000-0xd17ff pnpid
ORM0000 on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppbus0: <Parallel port bus> on ppc0
ppbus0: [ITHREAD]
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
ppc0: [GIANT-LOCKED]
ppc0: [ITHREAD]
Timecounter "TSC" frequency 1585747239 Hz quality 800
Timecounters tick every 1.000 msec
Waiting 5 seconds for SCSI devices to settle
ad0: 305244MB <Seagate ST3320620A 3.AAF> at ata0-master UDMA100
acd0: DVDR <HL-DT-ST DVDRAM GSA-4163B/A106> at ata1-master UDMA33
aacd0: <RAID 5> on aac0
aacd0: 953812MB (1953406976 sectors)
sa0 at ahc0 bus 0 target 3 lun 0
sa0: <HP Ultrium 1-SCSI N15G> Removable Sequential Access SCSI-3 device
sa0: 20.000MB/s transfers (10.000MHz, offset 8, 16bit)
Trying to mount root from ufs:/dev/ad0s1a
WARNING: / was not properly dismounted
bridge0: Ethernet address: 86:5b:d2:24:75:72
em0: link state changed to UP
