I'm not sure if this is the right place to post this. I
hope 5.2.1-RELEASE is considered part of this list. If
not, I apologize. I don't know much about debuging kernel
crashes in FreeBSD so here goes trying to provide enough
information about the problem:
I am running 5.2.1-RELEASE-p9.
FreeBSD hgdownload 5.2.1-RELEASE-p9 FreeBSD 5.2.1-RELEASE-p9 #1: Thu Jul 29
08:29:56 PDT 2004 root@hgdownload:/usr/src/sys/i386/compile/HGDOWNLOAD i386
On a Dell 2650 with a PERC3/QC. Both data area and boot/swap drive is on
the PERC3/QC drives. Separate logical drives.
Recently, under heavy load (rsync download server for genome data) system
has repeatably crashed with the following gdb -k output every time:
(NOTE, the nullfs module is because we do a few loopback mounts)
panic: mtx_lock() of spin mutex (null) @ ../../../ufs/ufs/ufs_dirhash.c:282
panic messages:
---
panic: mtx_lock() of spin mutex (null) @ ../../../ufs/ufs/ufs_dirhash.c:282
Stack backtrace:
Uptime: 7h24m50s
Dumping 2047 MB
16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 256 272 288 304 320 336
352 368 384 400 416 432 448 464 480 496 512 528 544 560 576 592 608 624 640 656
672 688 704 720 736 752 768 784 800 816 832 848 864 880 896 912 928 944 960 976
992 1008 1024 1040 1056 1072 1088 1104 1120 1136 1152 1168 1184 1200 1216 1232
1248 1264 1280 1296 1312 1328 1344 1360 1376 1392 1408 1424 1440 1456 1472 1488
1504 1520 1536 1552 1568 1584 1600 1616 1632 1648 1664 1680 1696 1712 1728 1744
1760 1776 1792 1808 1824 1840 1856 1872 1888 1904 1920 1936 1952 1968 1984 2000
2016 2032
---
Reading symbols from
/usr/src/sys/i386/compile/HGDOWNLOAD/modules/usr/src/sys/modules/acpi/acpi.ko.debug...done.
Loaded symbols for
/usr/src/sys/i386/compile/HGDOWNLOAD/modules/usr/src/sys/modules/acpi/acpi.ko.debug
Reading symbols from
/usr/src/sys/i386/compile/HGDOWNLOAD/modules/usr/src/sys/modules/nullfs/nullfs.ko.debug...done.
Loaded symbols for
/usr/src/sys/i386/compile/HGDOWNLOAD/modules/usr/src/sys/modules/nullfs/nullfs.ko.debug
#0 doadump () at ../../../kern/kern_shutdown.c:240
240 dumping++;
(kgdb) where
#0 doadump () at ../../../kern/kern_shutdown.c:240
#1 0xc04c8158 in boot (howto=260) at ../../../kern/kern_shutdown.c:372
#2 0xc04c84e7 in panic () at ../../../kern/kern_shutdown.c:550
#3 0xc04be773 in _mtx_lock_flags (m=0x0, opts=0,
file=0xc0674938 "../../../ufs/ufs/ufs_dirhash.c", line=282)
at ../../../kern/kern_mutex.c:215
#4 0xc05d9eab in ufsdirhash_free (ip=0xc8c85578)
at ../../../ufs/ufs/ufs_dirhash.c:282
#5 0xc05dbf44 in ufs_reclaim (ap=0x0) at ../../../ufs/ufs/ufs_inode.c:182
#6 0xc05e3eb8 in ufs_vnoperate (ap=0x0) at ../../../ufs/ufs/ufs_vnops.c:2793
#7 0xc0521c08 in vclean (vp=0xc8c9f514, flags=8, td=0xc8248640)
at vnode_if.h:981
#8 0xc0522171 in vgonel (vp=0xc8c9f514, td=0x0)
at ../../../kern/vfs_subr.c:2788
#9 0xc051e15a in vtryrecycle (vp=0xc8c9f514) at ../../../kern/vfs_subr.c:898
#10 0xc051e3e9 in getnewvnode (tag=0xc066da2c "ufs", mp=0x0, vops=0x0,
vpp=0x0)
at ../../../kern/vfs_subr.c:960
#11 0xc05d4be5 in ffs_vget (mp=0xc8235c00, ino=187006288, flags=2,
vpp=0xee5aea4c) at ../../../ufs/ffs/ffs_vfsops.c:1268
#12 0xc05dcd12 in ufs_lookup (ap=0xee5aeb10)
at ../../../ufs/ufs/ufs_lookup.c:603
#13 0xc05e3eb8 in ufs_vnoperate (ap=0x0) at ../../../ufs/ufs/ufs_vnops.c:2793
#14 0xc0514ca1 in vfs_cache_lookup (ap=0x0) at vnode_if.h:82
#15 0xc05e3eb8 in ufs_vnoperate (ap=0x0) at ../../../ufs/ufs/ufs_vnops.c:2793
#16 0xc0519ed2 in lookup (ndp=0xee5aec28) at vnode_if.h:52
#17 0xc05198be in namei (ndp=0xee5aec28) at ../../../kern/vfs_lookup.c:183
#18 0xc0527262 in lstat (td=0xc8248640, uap=0xee5aed14)
at ../../../kern/vfs_syscalls.c:2030
#19 0xc06339e0 in syscall (frame {tf_fs = 134742063, tf_es = 134742063,
tf_ds = -1078001617, tf_edi = 134563840, tf_esi = 134563912, tf_ebp =
-1077940936, tf_isp = -296030860, tf_ebx = 672370976, tf_edx = 134524928, tf_ecx
= 0, tf_eax = 190, tf_trapno = 12, tf_err = 2, tf_eip = 671863887, tf_cs = 31,
tf_eflags = 582, tf_esp = -1077941092, tf_ss = 47}) at
../../../i386/i386/trap.c:1010
#20 0xc062483d in Xint0x80_syscall () at {standard input}:136
---Can't read userspace from dump, or kernel process---
Kernel Config is as follows, note we are running it uniprocessor
to try to rule out SMP problems. We have also turned off logical
processors (hyperthreading).
machine i386
cpu I686_CPU
ident HGDOWNLOAD
options SCHED_4BSD #4BSD scheduler
options INET #InterNETworking
options INET6 #IPv6 communications protocols
options FFS #Berkeley Fast Filesystem
options SOFTUPDATES #Enable FFS soft updates support
options UFS_ACL #Support for access control lists
options UFS_DIRHASH #Improve performance on big directories
options MD_ROOT #MD is a potential root device
options NFSCLIENT #Network Filesystem Client
options NFSSERVER #Network Filesystem Server
options NFS_ROOT #NFS usable as /, requires NFSCLIENT
options MSDOSFS #MSDOS Filesystem
options CD9660 #ISO 9660 Filesystem
options PROCFS #Process filesystem (requires PSEUDOFS)
options PSEUDOFS #Pseudo-filesystem framework
options COMPAT_43 #Compatible with BSD 4.3 [KEEP THIS!]
options COMPAT_FREEBSD4 #Compatible with FreeBSD4
options SCSI_DELAY=15000 #Delay (in ms) before probing SCSI
options KTRACE #ktrace(1) support
options SYSVSHM #SYSV-style shared memory
options SYSVMSG #SYSV-style message queues
options SYSVSEM #SYSV-style semaphores
options _KPOSIX_PRIORITY_SCHEDULING #Posix P1003_1B real-time extensions
options KBD_INSTALL_CDEV # install a CDEV entry in /dev
options AHC_REG_PRETTY_PRINT # Print register bitfields in debug
# output. Adds ~128k to driver.
options AHD_REG_PRETTY_PRINT # Print register bitfields in debug
# output. Adds ~215k to driver.
options PFIL_HOOKS # pfil(9) framework
makeoptions DEBUG=-g #Build kernel with gdb(1) debug symbols
options DDB
options DDB_TRACE
options DDB_UNATTENDED
ptions INVARIANTS #Enable calls of extra sanity checking
options INVARIANT_SUPPORT #Extra sanity checks of internal structur
# To make an SMP kernel, the next two are needed
#options SMP # Symmetric MultiProcessor Kernel
#device apic # I/O APIC
device isa
device pci
device scbus # SCSI bus (required for SCSI)
device ch # SCSI media changers
device da # Direct Access (disks)
device sa # Sequential Access (tape etc)
device cd # CD
device pass # Passthrough device (direct SCSI access)
device ses # SCSI Environmental Services (and SAF-TE)
device amr # AMI MegaRAID
device atkbdc # AT keyboard controller
device atkbd # AT keyboard
device psm # PS/2 mouse
device vga # VGA video card driver
device splash # Splash screen and screen saver support
# syscons is the default console driver, resembling an SCO console
device sc
device agp # support several AGP chipsets
# Floating point support - do not disable.
device npx
# Add suspend/resume support for the i8254.
device pmtimer
# Serial (COM) ports
device sio # 8250, 16[45]50 based serial ports
# Parallel port
device ppc
device ppbus # Parallel port bus (required)
device lpt # Printer
device ppi # Parallel port interface device
# PCI Ethernet NICs that use the common MII bus controller code.
# NOTE: Be sure to keep the 'device miibus' line in order to use these
NICs!
device miibus # MII bus support
device bfe # Broadcom BCM440x 10/100 ethernet
device bge # Broadcom BCM570xx Gigabit Ethernet
device random # Entropy device
device loop # Network loopback
device ether # Ethernet support
device ppp # Kernel PPP
device tun # Packet tunnel.
device pty # Pseudo-ttys (telnet etc)
device md # Memory "disks"
device gif # IPv6 and IPv4 tunneling
device faith # IPv6-to-IPv4 relaying (translation)
Dmesg output follows:
Copyright (c) 1992-2004 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD 5.2.1-RELEASE-p9 #1: Thu Jul 29 08:29:56 PDT 2004
root@hgdownload:/usr/src/sys/i386/compile/HGDOWNLOAD
Preloaded elf kernel "/boot/kernel/kernel" at 0xc07a8000.
Preloaded elf module "/boot/kernel/acpi.ko" at 0xc07a821c.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2784.39-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0xf29 Stepping = 9
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,C
MOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
real memory = 2147352576 (2047 MB)
avail memory = 2084798464 (1988 MB)
Pentium Pro MTRR support enabled
npx0: [FAST]
npx0: <math processor> on motherboard
npx0: INT 16 interface
acpi0: <DELL PE2650 > on motherboard
acpi0: Power Button (fixed)
Timecounter "ACPI-safe" frequency 3579545 Hz quality 1000
pcibios: BIOS version 2.10
Using $PIR table, 9 entries at 0xc00fc410
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
acpi_cpu0: <CPU> on acpi0
acpi_cpu1: <CPU> on acpi0
device_probe_and_attach: acpi_cpu1 attach returned 6
acpi_cpu1: <CPU> on acpi0
device_probe_and_attach: acpi_cpu1 attach returned 6
acpi_cpu1: <CPU> on acpi0
device_probe_and_attach: acpi_cpu1 attach returned 6
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib0: slot 4 INTA is routed to irq 11
pcib0: slot 4 INTB is routed to irq 10
pcib0: slot 4 INTC is routed to irq 7
pcib0: slot 15 INTA is routed to irq 5
pci0: <unknown> at device 4.0 (no driver attached)
pci0: <unknown> at device 4.1 (no driver attached)
pci0: <unknown> at device 4.2 (no driver attached)
pci0: <display, VGA> at device 14.0 (no driver attached)
pci0: <mass storage, ATA> at device 15.1 (no driver attached)
pci0: <serial bus, USB> at device 15.2 (no driver attached)
isab0: <PCI-ISA bridge> at device 15.3 on pci0
isa0: <ISA bus> on isab0
pcib1: <ACPI Host-PCI bridge> on acpi0
pci6: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 8.0 on pci6
pci7: <ACPI PCI bus> on pcib2
pcib2: slot 6 INTA is routed to irq 7
pcib2: slot 6 INTB is routed to irq 11
pci7: <mass storage, SCSI> at device 6.0 (no driver attached)
pci7: <mass storage, SCSI> at device 6.1 (no driver attached)
pcib3: <ACPI Host-PCI bridge> on acpi0
pci5: <ACPI PCI bus> on pcib3
pcib3: slot 6 INTA is routed to irq 11
pcib3: slot 8 INTA is routed to irq 10
bge0: <Broadcom BCM5703 Gigabit Ethernet, ASIC rev. 0x1002> mem
0xeff10000-0xeff
1ffff irq 11 at device 6.0 on pci5
bge0: Ethernet address: 00:0d:56:ba:cd:45
miibus0: <MII bus> on bge0
brgphy0: <BCM5703 10/100/1000baseTX PHY> on miibus0
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX
-FDX, auto
bge1: <Broadcom BCM5703 Gigabit Ethernet, ASIC rev. 0x1002> mem
0xeff00000-0xeff
0ffff irq 10 at device 8.0 on pci5
bge1: Ethernet address: 00:0d:56:ba:cd:46
miibus1: <MII bus> on bge1
brgphy1: <BCM5703 10/100/1000baseTX PHY> on miibus1
brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX
-FDX, auto
pcib4: <ACPI Host-PCI bridge> on acpi0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI Host-PCI bridge> on acpi0
pci1: <ACPI PCI bus> on pcib5
pcib6: <PCI-PCI bridge> at device 8.0 on pci1
pci2: <PCI bus> on pcib6
pcib5: slot 8 INTB is routed to irq 11
pcib6: slot 1 INTA is routed to irq 11
pcib5: slot 8 INTC is routed to irq 10
pcib6: slot 2 INTA is routed to irq 10
pcib7: <PCI-PCI bridge> at device 0.0 on pci2
pci3: <PCI bus> on pcib7
pcib5: slot 8 INTA is routed to irq 7
pcib6: slot 0 INTA is routed to irq 7
pcib7: slot 0 INTA is routed to irq 7
amr0: <LSILogic MegaRAID> mem 0xf0000000-0xf7ffffff irq 7 at device 0.0 on
pci3
amr0: <LSILogic PERC 3/QC> Firmware 196T, BIOS 3.33, 128MB RAM
pci2: <mass storage, SCSI> at device 1.0 (no driver attached)
pci2: <mass storage, SCSI> at device 2.0 (no driver attached)
fdc0: <Enhanced floppy controller (i82077, NE72065 or clone)> port
0x3f7,0x3f0-0
x3f5 irq 6 drq 2 on acpi0
fdc0: FIFO enabled, 8 bytes threshold
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
sio0 port 0x3f8-0x3ff irq 4 on acpi0
sio0: type 16550A
sio1 port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
acpi_cpu1: <CPU> on acpi0
device_probe_and_attach: acpi_cpu1 attach returned 6
acpi_cpu1: <CPU> on acpi0
device_probe_and_attach: acpi_cpu1 attach returned 6
acpi_cpu1: <CPU> on acpi0
device_probe_and_attach: acpi_cpu1 attach returned 6
pmtimer0 on isa0
orm0: <Option ROMs> at iomem
0xec000-0xeffff,0xca800-0xcbfff,0xc8000-0xc8fff,0xc
0000-0xc7fff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x64,0x60 on isa0
ppc0: parallel port not found.
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounter "TSC" frequency 2784393264 Hz quality 800
Timecounters tick every 10.000 msec
amrd0: <LSILogic MegaRAID logical drive> on amr0
amrd0: 34680MB (71024640 sectors) RAID 1 (optimal)
GEOM: create disk amrd0 dp=0xc81fa40c
amrd1: <LSILogic MegaRAID logical drive> on amr0
amrd1: 1818700MB (3724697600 sectors) RAID 5 (optimal)
GEOM: create disk amrd1 dp=0xc81fa30c
Mounting root from ufs:/dev/amrd0s1a
WARNING: / was not properly dismounted
bge0: gigabit link up
Does anyone have suggestions on what I can do to debug this further?
The unit did have some hardware problems, but we've swapped out all
the components involved and updated firmware of critical components.
At least we think we have to proper levels.
Any help anyone could provide would be greatly appreciated.
--------------------------------------------------------------------
Paul Tatarsky paul@cse.ucsc.edu
UC Santa Cruz http://www.cse.ucsc.edu/~paul
CBSE Cluster Project Sysadmin SOE Network Security Advisor
--------------------------------------------------------------------