Alex Popa
2008-Mar-14 19:50 UTC
Lock Order Reversal on 7.0-STABLE with pf and ipfw / dummynet
World was cvsupped on March 6th, around 18:00 GMT.
Built and installed kernel + world, with options WITNESS and
WITNESS_SKIPSPIN.
Short background: 7.0-RELEASE had excellent performance on the machine,
but it would randomly lock up after some hours (usually over 10 hours).
The lockups were hard, meaning nothing seemed to work (NumLock didn't
toggle the keyboard LED, no replies to ping, no disk activity). We
changed the motherboard and RAM and had the same behaviour. 6.2-REL is
rock solid on this machine (had over 50 days uptime), but upgrading to
6.3-REL made it lock up just like 7.0 (so we put 6.2 back and accepted
the lower performance for the time being).
The LOR messages from dmesg of 7.0-STABLE are as follows:
lock order reversal:
1st 0xffffffffb19e0680 pf task mtx (pf task mtx) @
/usr/src/sys/modules/pf/../../contrib/pf/net/pf.c:6729
2nd 0xffffff00042ea0f0 radix node head (radix node head) @
/usr/src/sys/net/route.c:147
lock order reversal:
1st 0xffffffff80938508 PFil hook read/write mutex (PFil hook read/write mutex)
@ /usr/src/sys/net/pfil.c:73
2nd 0xffffffff80938c48 tcp (tcp) @ /usr/src/sys/netinet/tcp_input.c:400
More details about the machine in the attached dmesg. It's a SMP with
4GB of RAM, 3 gigabit cards (em0, em1 and, depending on the motherboard
we used, either bge0 or msk0). Only em0 is linked to a gigabit port,
the others are 100Mbits/s
My setup has in-kernel IPFIREWALL, IPFIREWALL_VERBOSE,
IPFIREWALL_DEFAULT_TO_ACCEPT, DUMMYNET. I have commented out INET6,
SCTP and the wireless interfaces. WITNESS and WITNESS_SKIPSPIN were
only added in the hope of figuring out what locks it up, and they did
signal these 2 LORs.
pf and pflog are loaded as modules (pf_enable and pflog_enable set to
yes in rc.conf).
- The ipfw/dummynet side:
I use net.link.ether.ipfw = 1 for MAC address checking, ipfw + dummynet
for traffic shaping (4 queues at 95Mbits/s for the 2 external interfaces
in/out, and 4 more queues for traffic that goes outside the AS group for
which we have fast access). Deciding which queue traffic goes in
depends on its source address and whether its destination is in ipfw
tables 1, 2 or none. These tables are synchronized from pf tables via a
custom script in crontab, which runs every 3 minutes. The pf tables
used as source for these are controlled by OpenBGPD.
- The pf side:
Filtering is done here, as is policy routing. Filtering also contains
redirecting to a transparent squid proxy of traffic destined to port 80
but not bound for networks received via BGP and saved to tables <metro>
and <special>. Metro and special port 80 traffic goes directly to
the destination server.
Traffic from net1 and net2 is routed via the "other" external
interface,
which doesn't contain the default route... with the exception of traffic
to pf table <special> (from BGP, same as table 2 in ipfw). Traffic to
<special> is routed via fastroute in pf (meaning using the default
route).
Attached are full dmesg and the kernel config.
I still have access to the hard drive with 7.0-STABLE on it, but not the
motherboard/CPU and the network cards... they are running off the hard
drive with 6.2 on it.
--
"Computer science is no more about computers
than astronomy is about telescopes" -- E. W. Dijkstra
-------------- next part --------------
#
# GENERIC -- Generic kernel configuration file for FreeBSD/amd64
#
# For more information on this file, please read the handbook section on
# Kernel Configuration Files:
#
#
http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html
#
# The handbook is also available locally in /usr/share/doc/handbook
# if you've installed the doc distribution, otherwise always see the
# FreeBSD World Wide Web server (http://www.FreeBSD.org/) for the
# latest information.
#
# An exhaustive list of options and more detailed explanations of the
# device lines is also present in the ../../conf/NOTES and NOTES files.
# If you are in doubt as to the purpose or necessity of a line, check first
# in NOTES.
#
# $FreeBSD: src/sys/amd64/conf/GENERIC,v 1.484.2.2.2.1 2008/02/06 03:24:28
scottl Exp $
cpu HAMMER
ident GENERIC
# To statically compile in device wiring instead of /boot/device.hints
#hints "GENERIC.hints" # Default places to look for devices.
makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols
options WITNESS
options WITNESS_SKIPSPIN
options SCHED_4BSD # 4BSD scheduler
options PREEMPTION # Enable kernel thread preemption
options INET # InterNETworking
#options INET6 # IPv6 communications protocols
#options SCTP # Stream Control Transmission Protocol
options FFS # Berkeley Fast Filesystem
options SOFTUPDATES # Enable FFS soft updates support
options UFS_ACL # Support for access control lists
options UFS_DIRHASH # Improve performance on big directories
options UFS_GJOURNAL # Enable gjournal-based UFS journaling
options MD_ROOT # MD is a potential root device
options NFSCLIENT # Network Filesystem Client
options NFSSERVER # Network Filesystem Server
options NFS_ROOT # NFS usable as /, requires NFSCLIENT
options NTFS # NT File System
options MSDOSFS # MSDOS Filesystem
options CD9660 # ISO 9660 Filesystem
options PROCFS # Process filesystem (requires PSEUDOFS)
options PSEUDOFS # Pseudo-filesystem framework
options GEOM_PART_GPT # GUID Partition Tables.
options GEOM_LABEL # Provides labelization
options COMPAT_43TTY # BSD 4.3 TTY compat [KEEP THIS!]
options COMPAT_IA32 # Compatible with i386 binaries
options COMPAT_FREEBSD4 # Compatible with FreeBSD4
options COMPAT_FREEBSD5 # Compatible with FreeBSD5
options COMPAT_FREEBSD6 # Compatible with FreeBSD6
options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI
options KTRACE # ktrace(1) support
options SYSVSHM # SYSV-style shared memory
options SYSVMSG # SYSV-style message queues
options SYSVSEM # SYSV-style semaphores
options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions
options KBD_INSTALL_CDEV # install a CDEV entry in /dev
options ADAPTIVE_GIANT # Giant mutex is adaptive.
options STOP_NMI # Stop CPUS using NMI instead of IPI
options AUDIT # Security event auditing
options DUMMYNET
options IPFIREWALL
options IPFIREWALL_VERBOSE
options IPFIREWALL_DEFAULT_TO_ACCEPT
# Make an SMP-capable kernel by default
options SMP # Symmetric MultiProcessor Kernel
# CPU frequency control
device cpufreq
# Bus support.
device acpi
device pci
# Floppy drives
device fdc
# ATA and ATAPI devices
device ata
device atadisk # ATA disk drives
device ataraid # ATA RAID drives
device atapicd # ATAPI CDROM drives
device atapifd # ATAPI floppy drives
device atapist # ATAPI tape drives
options ATA_STATIC_ID # Static device numbering
# SCSI Controllers
device ahc # AHA2940 and onboard AIC7xxx devices
options AHC_REG_PRETTY_PRINT # Print register bitfields in debug
# output. Adds ~128k to driver.
device ahd # AHA39320/29320 and onboard AIC79xx devices
options AHD_REG_PRETTY_PRINT # Print register bitfields in debug
# output. Adds ~215k to driver.
device amd # AMD 53C974 (Tekram DC-390(T))
device hptiop # Highpoint RocketRaid 3xxx series
device isp # Qlogic family
#device ispfw # Firmware for QLogic HBAs- normally a module
device mpt # LSI-Logic MPT-Fusion
#device ncr # NCR/Symbios Logic
device sym # NCR/Symbios Logic (newer chipsets + those of `ncr')
device trm # Tekram DC395U/UW/F DC315U adapters
device adv # Advansys SCSI adapters
device adw # Advansys wide SCSI adapters
device aic # Adaptec 15[012]x SCSI adapters, AIC-6[23]60.
device bt # Buslogic/Mylex MultiMaster SCSI adapters
# SCSI peripherals
device scbus # SCSI bus (required for SCSI)
device ch # SCSI media changers
device da # Direct Access (disks)
device sa # Sequential Access (tape etc)
device cd # CD
device pass # Passthrough device (direct SCSI access)
device ses # SCSI Environmental Services (and SAF-TE)
# RAID controllers interfaced to the SCSI subsystem
device amr # AMI MegaRAID
device arcmsr # Areca SATA II RAID
device ciss # Compaq Smart RAID 5*
device dpt # DPT Smartcache III, IV - See NOTES for options
device hptmv # Highpoint RocketRAID 182x
device hptrr # Highpoint RocketRAID 17xx, 22xx, 23xx, 25xx
device iir # Intel Integrated RAID
device ips # IBM (Adaptec) ServeRAID
device mly # Mylex AcceleRAID/eXtremeRAID
device twa # 3ware 9000 series PATA/SATA RAID
# RAID controllers
device aac # Adaptec FSA RAID
device aacp # SCSI passthrough for aac (requires CAM)
device ida # Compaq Smart RAID
device mfi # LSI MegaRAID SAS
device mlx # Mylex DAC960 family
#XXX pointer/int warnings
#device pst # Promise Supertrak SX6000
device twe # 3ware ATA RAID
# atkbdc0 controls both the keyboard and the PS/2 mouse
device atkbdc # AT keyboard controller
device atkbd # AT keyboard
device psm # PS/2 mouse
device kbdmux # keyboard multiplexer
device vga # VGA video card driver
device splash # Splash screen and screen saver support
# syscons is the default console driver, resembling an SCO console
device sc
device agp # support several AGP chipsets
# PCCARD (PCMCIA) support
# PCMCIA and cardbus bridge support
device cbb # cardbus (yenta) bridge
device pccard # PC Card (16-bit) bus
device cardbus # CardBus (32-bit) bus
# Serial (COM) ports
device sio # 8250, 16[45]50 based serial ports
device uart # Generic UART driver
# Parallel port
device ppc
device ppbus # Parallel port bus (required)
device lpt # Printer
device plip # TCP/IP over parallel
device ppi # Parallel port interface device
#device vpo # Requires scbus and da
# If you've got a "dumb" serial or parallel PCI card that is
# supported by the puc(4) glue driver, uncomment the following
# line to enable it (connects to sio, uart and/or ppc drivers):
#device puc
# PCI Ethernet NICs.
device de # DEC/Intel DC21x4x (``Tulip'')
device em # Intel PRO/1000 adapter Gigabit Ethernet Card
device ixgb # Intel PRO/10GbE Ethernet Card
device le # AMD Am7900 LANCE and Am79C9xx PCnet
device txp # 3Com 3cR990 (``Typhoon'')
device vx # 3Com 3c590, 3c595 (``Vortex'')
# PCI Ethernet NICs that use the common MII bus controller code.
# NOTE: Be sure to keep the 'device miibus' line in order to use these
NICs!
device miibus # MII bus support
device bce # Broadcom BCM5706/BCM5708 Gigabit Ethernet
device bfe # Broadcom BCM440x 10/100 Ethernet
device bge # Broadcom BCM570xx Gigabit Ethernet
device dc # DEC/Intel 21143 and various workalikes
device fxp # Intel EtherExpress PRO/100B (82557, 82558)
device lge # Level 1 LXT1001 gigabit Ethernet
device msk # Marvell/SysKonnect Yukon II Gigabit Ethernet
device nfe # nVidia nForce MCP on-board Ethernet
device nge # NatSemi DP83820 gigabit Ethernet
#device nve # nVidia nForce MCP on-board Ethernet Networking
device pcn # AMD Am79C97x PCI 10/100 (precedence over 'le')
device re # RealTek 8139C+/8169/8169S/8110S
device rl # RealTek 8129/8139
device sf # Adaptec AIC-6915 (``Starfire'')
device sis # Silicon Integrated Systems SiS 900/SiS 7016
device sk # SysKonnect SK-984x & SK-982x gigabit Ethernet
device ste # Sundance ST201 (D-Link DFE-550TX)
device ti # Alteon Networks Tigon I/II gigabit Ethernet
device tl # Texas Instruments ThunderLAN
device tx # SMC EtherPower II (83c170 ``EPIC'')
device vge # VIA VT612x gigabit Ethernet
device vr # VIA Rhine, Rhine II
device wb # Winbond W89C840F
device xl # 3Com 3c90x (``Boomerang'', ``Cyclone'')
# ISA Ethernet NICs. pccard NICs included.
device cs # Crystal Semiconductor CS89x0 NIC
# 'device ed' requires 'device miibus'
device ed # NE[12]000, SMC Ultra, 3c503, DS8390 cards
device ex # Intel EtherExpress Pro/10 and Pro/10+
device ep # Etherlink III based cards
device fe # Fujitsu MB8696x based cards
device sn # SMC's 9000 series of Ethernet chips
device xe # Xircom pccard Ethernet
## Wireless NIC cards
#device wlan # 802.11 support
#device wlan_wep # 802.11 WEP support
#device wlan_ccmp # 802.11 CCMP support
#device wlan_tkip # 802.11 TKIP support
#device wlan_amrr # AMRR transmit rate control algorithm
#device wlan_scan_ap # 802.11 AP mode scanning
#device wlan_scan_sta # 802.11 STA mode scanning
#device an # Aironet 4500/4800 802.11 wireless NICs.
#device ath # Atheros pci/cardbus NIC's
#device ath_hal # Atheros HAL (Hardware Access Layer)
#device ath_rate_sample # SampleRate tx rate control for ath
#device awi # BayStack 660 and others
#device ral # Ralink Technology RT2500 wireless NICs.
#device wi # WaveLAN/Intersil/Symbol 802.11 wireless NICs.
# Pseudo devices.
device loop # Network loopback
device random # Entropy device
device ether # Ethernet support
device sl # Kernel SLIP
device ppp # Kernel PPP
device tun # Packet tunnel.
device pty # Pseudo-ttys (telnet etc)
device md # Memory "disks"
device gif # IPv6 and IPv4 tunneling
device faith # IPv6-to-IPv4 relaying (translation)
device firmware # firmware assist module
# The `bpf' device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
# Note that 'bpf' is required for DHCP.
device bpf # Berkeley packet filter
# USB support
device uhci # UHCI PCI->USB interface
device ohci # OHCI PCI->USB interface
device ehci # EHCI PCI->USB interface (USB 2.0)
device usb # USB Bus (required)
#device udbp # USB Double Bulk Pipe devices
device ugen # Generic
device uhid # "Human Interface Devices"
device ukbd # Keyboard
device ulpt # Printer
device umass # Disks/Mass storage - Requires scbus and da
device ums # Mouse
#device ural # Ralink Technology RT2500USB wireless NICs
device urio # Diamond Rio 500 MP3 player
device uscanner # Scanners
# USB Ethernet, requires miibus
device aue # ADMtek USB Ethernet
device axe # ASIX Electronics USB Ethernet
device cdce # Generic USB over Ethernet
device cue # CATC USB Ethernet
device kue # Kawasaki LSI USB Ethernet
device rue # RealTek RTL8150 USB Ethernet
# FireWire support
device firewire # FireWire bus code
device sbp # SCSI over FireWire (Requires scbus and da)
device fwe # Ethernet over FireWire (non-standard!)
device fwip # IP over FireWire (RFC 2734,3146)
device dcons # Dumb console driver
device dcons_crom # Configuration ROM for dcons
-------------- next part --------------
Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.0-STABLE #0: Fri Mar 7 01:31:39 EET 2008
root@censored.censored.ro:/usr/obj/usr/src/sys/KERNEL
WARNING: WITNESS option enabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Pentium(R) D CPU 2.80GHz (2810.56-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0xf44 Stepping = 4
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x641d<SSE3,RSVD2,MON,DS_CPL,CNXT-ID,CX16,xTPR>
AMD Features=0x20100800<SYSCALL,NX,LM>
Cores per package: 2
usable memory = 4282994688 (4084 MB)
avail memory = 4129546240 (3938 MB)
ACPI APIC Table: <A_M_I_ OEMAPIC >
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
hptrr: HPT RocketRAID controller driver v1.1 (Mar 7 2008 01:31:23)
acpi0: <A_M_I_ OEMXSDT> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, dff00000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on
acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
cpu0: <ACPI CPU> on acpi0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
cpu1: <ACPI CPU> on acpi0
p4tcc1: <CPU Frequency Thermal Control> on cpu1
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
uhci0: <UHCI (generic) USB controller> port 0xa800-0xa81f irq 16 at device
26.0 on pci0
uhci0: [GIANT-LOCKED]
uhci0: [ITHREAD]
usb0: <UHCI (generic) USB controller> on uhci0
usb0: USB revision 1.0
uhub0: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 2 ports with 2 removable, self powered
uhci1: <UHCI (generic) USB controller> port 0xa880-0xa89f irq 21 at device
26.1 on pci0
uhci1: [GIANT-LOCKED]
uhci1: [ITHREAD]
usb1: <UHCI (generic) USB controller> on uhci1
usb1: USB revision 1.0
uhub1: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1
uhub1: 2 ports with 2 removable, self powered
uhci2: <UHCI (generic) USB controller> port 0xac00-0xac1f irq 18 at device
26.2 on pci0
uhci2: [GIANT-LOCKED]
uhci2: [ITHREAD]
usb2: <UHCI (generic) USB controller> on uhci2
usb2: USB revision 1.0
uhub2: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb2
uhub2: 2 ports with 2 removable, self powered
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xfbbffc00-0xfbbfffff irq
18 at device 26.7 on pci0
ehci0: [GIANT-LOCKED]
ehci0: [ITHREAD]
usb3: EHCI version 1.0
usb3: companion controllers, 2 ports each: usb0 usb1 usb2
usb3: <EHCI (generic) USB 2.0 controller> on ehci0
usb3: USB revision 2.0
uhub3: <Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb3
uhub3: 6 ports with 6 removable, self powered
pcib1: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
pci5: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 18 at device 28.2 on pci0
pci4: <ACPI PCI bus> on pcib2
em0: <Intel(R) PRO/1000 Network Connection Version - 6.7.3> port
0xec00-0xec1f mem 0xfbfe0000-0xfbffffff,0xfbfc0000-0xfbfdffff irq 18 at device
0.0 on pci4
em0: Using MSI interrupt
em0: Ethernet address: 00:15:17:0e:73:3d
em0: [FILTER]
pcib3: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0
pci3: <ACPI PCI bus> on pcib3
em1: <Intel(R) PRO/1000 Network Connection Version - 6.7.3> port
0xdc00-0xdc1f mem 0xfbee0000-0xfbefffff,0xfbec0000-0xfbedffff irq 19 at device
0.0 on pci3
em1: Using MSI interrupt
em1: Ethernet address: 00:15:17:0e:74:14
em1: [FILTER]
pcib4: <ACPI PCI-PCI bridge> irq 17 at device 28.4 on pci0
pci2: <ACPI PCI bus> on pcib4
atapci0: <Marvell 88SX6101 UDMA133 controller> port
0xcc00-0xcc07,0xc880-0xc883,0xc800-0xc807,0xc480-0xc483,0xc400-0xc40f mem
0xfbdffc00-0xfbdffdff irq 16 at device 0.0 on pci2
atapci0: [ITHREAD]
ata2: <ATA channel 0> on atapci0
ata2: [ITHREAD]
pcib5: <ACPI PCI-PCI bridge> irq 16 at device 28.5 on pci0
pci1: <ACPI PCI bus> on pcib5
mskc0: <Marvell Yukon 88E8056 Gigabit Ethernet> port 0xb800-0xb8ff mem
0xfbcfc000-0xfbcfffff irq 17 at device 0.0 on pci1
msk0: <Marvell Technology Group Ltd. Yukon EC Ultra Id 0xb4 Rev 0x03> on
mskc0
msk0: Ethernet address: 00:1e:8c:5d:48:4b
miibus0: <MII bus> on msk0
e1000phy0: <Marvell 88E1149 Gigabit PHY> PHY 0 on miibus0
e1000phy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX-FDX, auto
mskc0: [FILTER]
uhci3: <UHCI (generic) USB controller> port 0xa080-0xa09f irq 23 at device
29.0 on pci0
uhci3: [GIANT-LOCKED]
uhci3: [ITHREAD]
usb4: <UHCI (generic) USB controller> on uhci3
usb4: USB revision 1.0
uhub4: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb4
uhub4: 2 ports with 2 removable, self powered
uhci4: <UHCI (generic) USB controller> port 0xa400-0xa41f irq 19 at device
29.1 on pci0
uhci4: [GIANT-LOCKED]
uhci4: [ITHREAD]
usb5: <UHCI (generic) USB controller> on uhci4
usb5: USB revision 1.0
uhub5: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb5
uhub5: 2 ports with 2 removable, self powered
uhci5: <UHCI (generic) USB controller> port 0xa480-0xa49f irq 18 at device
29.2 on pci0
uhci5: [GIANT-LOCKED]
uhci5: [ITHREAD]
usb6: <UHCI (generic) USB controller> on uhci5
usb6: USB revision 1.0
uhub6: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb6
uhub6: 2 ports with 2 removable, self powered
ehci1: <EHCI (generic) USB 2.0 controller> mem 0xfbbff800-0xfbbffbff irq
23 at device 29.7 on pci0
ehci1: [GIANT-LOCKED]
ehci1: [ITHREAD]
usb7: EHCI version 1.0
usb7: companion controllers, 2 ports each: usb4 usb5 usb6
usb7: <EHCI (generic) USB 2.0 controller> on ehci1
usb7: USB revision 2.0
uhub7: <Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb7
uhub7: 6 ports with 6 removable, self powered
pcib6: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci6: <ACPI PCI bus> on pcib6
vgapci0: <VGA-compatible display> mem 0xfd000000-0xfdffffff at device 1.0
on pci6
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci1: <Intel ICH9 SATA300 controller> port
0x9000-0x9007,0x8c00-0x8c03,0x8880-0x8887,0x8800-0x8803,0x8480-0x848f,0x8400-0x840f
irq 22 at device 31.2 on pci0
atapci1: [ITHREAD]
ata3: <ATA channel 0> on atapci1
ata3: [ITHREAD]
ata4: <ATA channel 1> on atapci1
ata4: [ITHREAD]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
atapci2: <Intel ICH9 SATA300 controller> port
0xa000-0xa007,0x9c00-0x9c03,0x9880-0x9887,0x9800-0x9803,0x9480-0x948f,0x9400-0x940f
irq 22 at device 31.5 on pci0
atapci2: [ITHREAD]
ata5: <ATA channel 0> on atapci2
ata5: [ITHREAD]
ata6: <ATA channel 1> on atapci2
ata6: [ITHREAD]
acpi_button0: <Power Button> on acpi0
fdc0: <floppy drive controller (FDE)> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2
on acpi0
fdc0: [FILTER]
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on
acpi0
sio0: type 16550A
sio0: [FILTER]
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xca7ff on isa0
ppc0: cannot reserve I/O port range
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
ipfw2 initialized, divert loadable, rule-based forwarding disabled, default to
accept, logging unlimited
hptrr: no controller detected.
ad4: 76319MB <WDC WD800JB-00JJC0 05.01C05> at ata2-master UDMA100
ad6: 70910MB <WDC WD740ADFD-00NLR5 21.07QR5> at ata3-master SATA150
SMP: AP CPU #1 Launched!
WARNING: WITNESS option enabled, expect reduced performance.
GEOM_LABEL: Label for provider ad6s1a is ufs/root64.
GEOM_LABEL: Label for provider ad6s1d is ufs/var64.
GEOM_LABEL: Label for provider ad6s1e is ufs/tmp64.
GEOM_LABEL: Label for provider ad6s1f is ufs/usr64.
Trying to mount root from ufs:/dev/ufs/root64
msk0: link state changed to UP
em0: link state changed to UP
lock order reversal:
1st 0xffffffffb19e0680 pf task mtx (pf task mtx) @
/usr/src/sys/modules/pf/../../contrib/pf/net/pf.c:6729
2nd 0xffffff00042ea0f0 radix node head (radix node head) @
/usr/src/sys/net/route.c:147
lock order reversal:
1st 0xffffffff80938508 PFil hook read/write mutex (PFil hook read/write mutex)
@ /usr/src/sys/net/pfil.c:73
2nd 0xffffffff80938c48 tcp (tcp) @ /usr/src/sys/netinet/tcp_input.c:400
msk0: Rx FIFO overrun!
msk0: Rx FIFO overrun!
-------------- next part --------------
Id Refs Address Size Name
1 3 0xffffffff80100000 984658 kernel
Contains modules:
Id Name
1 cam
2 probe
3 xpt
4 cd
5 ch
6 da
7 pass
8 sa
9 ses
10 aac/aacp
11 aac/aacd
12 pci/aacch
13 pci/aac
14 nexus/acpi
15 acpi/acpi_acad
16 acpi/acpi_button
17 acpi/acpi_cmbat
18 acpi/cpu
19 acpi/acpi_ec
20 acpi/acpi_hpet
21 acpi/acpi_isab
22 acpi/acpi_lid
23 pcib/acpi_pci
24 acpi/acpi_pci_link
25 acpi/acpi_pcib
26 pci/acpi_pcib
27 cpu/acpi_perf
28 acpi/acpi_sysresource
29 acpi/acpi_smbat
30 acpi/acpi_tz
31 cpu/acpi_throttle
32 acpi/acpi_timer
33 pci/adv
34 pci/adw
35 pccard/aic
36 isa/ahc_isa
37 cardbus/ahc_pci
38 pci/ahc_pci
39 cardbus/ahd
40 pci/ahd
41 ahd
42 ahc
43 pci/amd
44 amr/amrd
45 pci/amr
46 ata
47 pccard/ata
48 ata/ad
49 isa/ata
50 atapci/ata
51 pci/atapci
52 ataraid
53 ad/subdisk
54 ata/acd
55 ata/afd
56 ata/ast
57 bce/miibus
58 pci/bce
59 bfe/miibus
60 pci/bfe
61 bge/miibus
62 pci/bge
63 isa/bt
64 pci/bt
65 cbb/cardbus
66 pci/ciss
67 pci/ichss_pci
68 cpu/ichss
69 isa/cs
70 pccard/cs
71 dc/miibus
72 pci/dc
73 cardbus/dc
74 miibus/dcphy
75 miibus/pnphy
76 firewire/dcons_crom
77 dcons
78 pci/de
79 pci/dpt
80 ed/miibus
81 pccard/ed
82 pci/ed
83 pci/em
84 isa/ep
85 pccard/ep
86 isa/ex
87 pccard/ex
88 exca
89 pccard/fe
90 fwohci/firewire
91 cardbus/fwohci
92 pci/fwohci
93 firewire/fwe
94 firewire/fwip
95 firewire/sbp
96 fxp/miibus
97 cardbus/fxp
98 pci/fxp
99 ida/idad
100 pci/ida
101 pci/iir
102 ips/ipsd
103 pci/ips
104 pci/isp
105 pci/if_ixgb
106 kbdmux
107 pci/le
108 lge/miibus
109 pci/lge
110 g_md
111 mem
112 pci/mfi
113 mfi/mfid
114 miibus/acphy
115 miibus/amphy
116 miibus/bmtphy
117 miibus/brgphy
118 miibus/ciphy
119 miibus/e1000phy
120 miibus/xlphy
121 miibus/gentbi
122 miibus/icsphy
123 miibus/inphy
124 miibus/ip1000phy
125 miibus/lxtphy
126 miibus/mlphy
127 miibus/nsgphy
128 miibus/nsphy
129 miibus/nsphyter
130 miibus/pnaphy
131 miibus/qsphy
132 miibus/rgephy
133 miibus/rlphy
134 miibus/ruephy
135 miibus/tdkphy
136 miibus/tlphy
137 miibus/ukphy
138 miibus/xmphy
139 mlx/mlxd
140 pci/mlx
141 pci/mly
142 mpt_core
143 mpt_cam
144 pci/mpt
145 mpt_raid
146 msk/miibus
147 mskc/msk
148 pci/mskc
149 nge/miibus
150 pci/nge
151 null
152 cbb/pccard
153 pcic/pccard
154 isa/cbb
155 pci/cbb
156 pci/fixup_pci
157 pci/hostb
158 pci/ignore_pci
159 pci/isab
160 pcib/pci
161 pci/pcib
162 pci/vgapci
163 ppbus/plip
164 ppbus/lpt
165 ppc/ppbus
166 ppbus/ppi
167 acpi/ppc
168 isa/ppc
169 pci/ppc
170 random
171 re/miibus
172 cardbus/re
173 pci/re
174 sk/miibus
175 skc/sk
176 pci/skc
177 isa/sn
178 pccard/sn
179 pci/sym
180 pci/ti
181 pci/trm
182 pci/twa
183 twe/twed
184 pci/twe
185 tx/miibus
186 pci/tx
187 pci/txp
188 acpi/uart
189 isa/uart
190 pccard/uart
191 cardbus/uart
192 pci/uart
193 cardbus/ehci
194 pci/ehci
195 aue/miibus
196 uhub/aue
197 axe/miibus
198 uhub/axe
199 uhub/cdce
200 uhub/cue
201 uhub/kue
202 rue/miibus
203 uhub/rue
204 cardbus/ohci
205 pci/ohci
206 uhub/ugen
207 cardbus/uhci
208 pci/uhci
209 uhub/uhid
210 uhub/uhub
211 usb/uhub
212 uhub/ukbd
213 uhub/ulpt
214 uhub/umass
215 uhub/ums
216 uhub/urio
217 slhci/usb
218 ehci/usb
219 uhci/usb
220 ohci/usb
221 uhub/uscanner
222 vge/miibus
223 cardbus/vge
224 pci/vge
225 pci/vx
226 watchdog
227 pccard/xe
228 devfs
229 msdosfs
230 ntfs
231 procfs
232 pseudofs
233 g_bsd
234 g_dev
235 g_disk
236 g_mbrext
237 g_mbr
238 g_vfs
239 g_label
240 g_part
241 eisab/isa
242 isab/isa
243 isa/isahint
244 isa/orm
245 cd9660
246 elf64
247 shell
248 cpu/cpufreq
249 rootbus
250 firmware
251 sysvmsg
252 msgrcv
253 msgsnd
254 msgget
255 msgctl
256 msgsys
257 sysvsem
258 semop
259 semget
260 __semctl
261 semsys
262 sysvshm
263 shmget
264 shmdt
265 shmctl
266 shmat
267 shmsys
268 ether
269 if_faith
270 if_firewire
271 if_gif
272 loop
273 if_ppp
274 if_sl
275 if_tun
276 dummynet
277 ipfw
278 nfs4
279 nfs
280 nfslock
281 nfsserver
282 pcn/miibus
283 pci/pcn
284 rl/miibus
285 cardbus/rl
286 pci/rl
287 sf/miibus
288 pci/sf
289 sis/miibus
290 pci/sis
291 ste/miibus
292 pci/ste
293 tl/miibus
294 pci/tl
295 vr/miibus
296 pci/vr
297 wb/miibus
298 pci/wb
299 xl/miibus
300 pci/xl
301 cardbus/xl
302 ufs
303 g_class
304 acpi/fpupnp
305 nexus/apic
306 pci/ioapic
307 legacy/cpu
308 nexus/legacy
309 isa/sysresource
310 nexus/ram
311 root/nexus
312 acpi/attimer
313 isa/attimer
314 legacy/isa
315 acpi/atdma
316 isa/atdma
317 isa/pcibus_pnp
318 legacy/pcib
319 pci/arcmsr
320 atkbdc/atkbd
321 acpi/atkbdc
322 isa/atkbdc
323 acpi/psmcpnp
324 isa/psmcpnp
325 atkbdc/psm
326 isa/ed
327 io
328 fdc/fd
329 acpi/fdc
330 isa/fdc
331 pccard/fdc
332 pci/hptmv
333 pci/hptrr
334 nfe/miibus
335 pci/nfe
336 acpi/sio
337 isa/sio
338 pccard/sio
339 cardbus/sio
340 pci/sio
341 scterm-sc
342 scrndr-vga
343 isa/sc
344 isa/vga
345 hostb/agp_amd64
346 vgapci/agp_i810
347 hostb/agp_intel
348 elf32
349 cpu/powernow
350 cpu/est
351 cpu/p4tcc
2 1 0xffffffffb19bb000 912 pflog.ko
Contains modules:
Id Name
353 pflog
3 1 0xffffffffb19bc000 25abc pf.ko
Contains modules:
Id Name
352 pf
Alex Popa
2008-Mar-15 12:37 UTC
Lock Order Reversal on 7.0-STABLE with pf and ipfw / dummynet (extra details)
Extra details: The machine shows no problems when running
6.2-RELEASE-p7 / i386.
Upgrading to 6.3-RELEASE / i386 caused the machine to start randomly
locking up, much faster than 7.0 (about 4 hours versus 12+)
Have tried 7.0-RELEASE/i386, 7.0-RELEASE/amd64, both with SMP support.
I've also tried 7.0-RELEASE/i386 without SMP support. All of them
exhibited the lockups (amd64 with SMP locked up after most time, the
others were much faster to lock up but this isn't statistically proven).
Alex
--
"Computer science is no more about computers
than astronomy is about telescopes" -- E. W. Dijkstra
Robert Watson
2008-Mar-15 20:36 UTC
Lock Order Reversal on 7.0-STABLE with pf and ipfw / dummynet
On Fri, 14 Mar 2008, Alex Popa wrote:> World was cvsupped on March 6th, around 18:00 GMT. > > Built and installed kernel + world, with options WITNESS and > WITNESS_SKIPSPIN. > > Short background: 7.0-RELEASE had excellent performance on the machine, but > it would randomly lock up after some hours (usually over 10 hours). The > lockups were hard, meaning nothing seemed to work (NumLock didn't toggle the > keyboard LED, no replies to ping, no disk activity). We changed the > motherboard and RAM and had the same behaviour. 6.2-REL is rock solid on > this machine (had over 50 days uptime), but upgrading to 6.3-REL made it > lock up just like 7.0 (so we put 6.2 back and accepted the lower performance > for the time being). > > The LOR messages from dmesg of 7.0-STABLE are as follows: > > lock order reversal: > 1st 0xffffffffb19e0680 pf task mtx (pf task mtx) @ /usr/src/sys/modules/pf/../../contrib/pf/net/pf.c:6729 > 2nd 0xffffff00042ea0f0 radix node head (radix node head) @ /usr/src/sys/net/route.c:147 > lock order reversal: > 1st 0xffffffff80938508 PFil hook read/write mutex (PFil hook read/write mutex) @ /usr/src/sys/net/pfil.c:73 > 2nd 0xffffffff80938c48 tcp (tcp) @ /usr/src/sys/netinet/tcp_input.c:400Dear Alex, Thanks for this report, and sorry about the problem. It could well be that the lock order warning from WITNESS is related to the hang, and might reflect a recursion-related bug in the pf policy routing code. I'm not sure to what extent you can tolerate further downtime, but it would be useful to gather some more information about the hang itself to try and confirm the involvement of lock order. In particular, if it's feasible, it would be very helpful if you could boot back to the 7-STABLE kernel (keeping the 6.2-STABLE userspace should be fine, I think), and when the hang occurs, use the console debuggger (ideally hooked up to serial or firewire) to run the following debugging commands: show pcpu show allpcpu trace alltrace show allocks show witness show lockedvnods show uma show malloc A shot-in-the-dark guess is that something about pf's interactions with the protocol stack is involved here, but unfortunately I suspect we'll need some more information to track it down. Also, could you confirm if you're using any credential-related firewall rules with either ipfw or pf? These would be uid/gid/jail matching rules. Robert N M Watson Computer Laboratory University of Cambridge> > More details about the machine in the attached dmesg. It's a SMP with > 4GB of RAM, 3 gigabit cards (em0, em1 and, depending on the motherboard > we used, either bge0 or msk0). Only em0 is linked to a gigabit port, > the others are 100Mbits/s > > My setup has in-kernel IPFIREWALL, IPFIREWALL_VERBOSE, > IPFIREWALL_DEFAULT_TO_ACCEPT, DUMMYNET. I have commented out INET6, > SCTP and the wireless interfaces. WITNESS and WITNESS_SKIPSPIN were > only added in the hope of figuring out what locks it up, and they did > signal these 2 LORs. > > pf and pflog are loaded as modules (pf_enable and pflog_enable set to > yes in rc.conf). > > - The ipfw/dummynet side: > > I use net.link.ether.ipfw = 1 for MAC address checking, ipfw + dummynet > for traffic shaping (4 queues at 95Mbits/s for the 2 external interfaces > in/out, and 4 more queues for traffic that goes outside the AS group for > which we have fast access). Deciding which queue traffic goes in > depends on its source address and whether its destination is in ipfw > tables 1, 2 or none. These tables are synchronized from pf tables via a > custom script in crontab, which runs every 3 minutes. The pf tables > used as source for these are controlled by OpenBGPD. > > - The pf side: > > Filtering is done here, as is policy routing. Filtering also contains > redirecting to a transparent squid proxy of traffic destined to port 80 > but not bound for networks received via BGP and saved to tables <metro> > and <special>. Metro and special port 80 traffic goes directly to > the destination server. > > Traffic from net1 and net2 is routed via the "other" external interface, > which doesn't contain the default route... with the exception of traffic > to pf table <special> (from BGP, same as table 2 in ipfw). Traffic to > <special> is routed via fastroute in pf (meaning using the default > route). > > Attached are full dmesg and the kernel config. > > I still have access to the hard drive with 7.0-STABLE on it, but not the > motherboard/CPU and the network cards... they are running off the hard > drive with 6.2 on it. > > -- > "Computer science is no more about computers > than astronomy is about telescopes" -- E. W. Dijkstra >