Hello,
Recently I updated my RELENG_8 to RELENG_9. Since then, the server hangs
from time to time for 5 minutes. When I run a top in a remote terminal,
I can see that it hangs so strong, that the clock hangs too. When it
continues to run , the time continues from the when it 'hanged'. TCP
connections are also dropped with timeout at that time. However, no
kernel panic, and i can't see anything in the dmesg log too.
A strange thing is, the server continues working when I press a key at
the physical console (I'm doing this with a remote IP console). More
strange thing is, when I do a reboot, the server flushes all its disks,
and then does a panic, instead of rebooting.
I have to revert to the RELENG_8 kernel (userland is RELENG_9 now), I
have no other choice.
I hardly can get the configuration and log out from it these times,
because of the hangs.
Hardware details:
This has 4 SAMSUNG disk (1.5TB each) array, driven by a 3ware Raid
controller, each disk exported as is. It also has an OCZ Revodrive as a
disk cache (zfs L2ARC cache) limited to SATA1 speed (strange kernel
panics because of disk timeouts when using at full speed), 8GB RAM,
AMD64 processor.
FreeBSD details:
The server runs on the 4-disk zfs array, boots from it and uses the zfs
array also as root media. It has 4 jails, connections handled by pf.
Kernel configuration:
cpu HAMMER
ident MYSERVER
machine amd64
options SCHED_ULE # ULE scheduler
options PREEMPTION # Enable kernel thread preemption
options INET # InterNETworking
options INET6 # IPv6 communications protocols
options SCTP # Stream Control Transmission
Protocol
options FFS # Berkeley Fast Filesystem
options SOFTUPDATES # Enable FFS soft updates support
options UFS_ACL # Support for access control lists
options UFS_DIRHASH # Improve performance on big
directories
options UFS_GJOURNAL # Enable gjournal-based UFS
journaling
options NFSCLIENT # Network Filesystem Client
options NFSLOCKD # Network Lock Manager
options MSDOSFS # MSDOS Filesystem
options GEOM_PART_GPT # GUID Partition Tables.
options GEOM_LABEL # Provides labelization
options KTRACE # ktrace(1) support
options STACK # stack(9) support
options SYSVSHM # SYSV-style shared memory
options SYSVMSG # SYSV-style message queues
options SYSVSEM # SYSV-style semaphores
options P1003_1B_SEMAPHORES # POSIX-style semaphores
options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time
extensions
options PRINTF_BUFR_SIZE=128 # Prevent printf output being
interspersed.
options KBD_INSTALL_CDEV # install a CDEV entry in /dev
options HWPMC_HOOKS # Necessary kernel hooks for
hwpmc(4)
options AUDIT # Security event auditing
options MAC # TrustedBSD MAC Framework
options FLOWTABLE # per-cpu routing cache
options INCLUDE_CONFIG_FILE # Include this file in kernel
options SMP # Symmetric MultiProcessor Kernel
device cpufreq
device acpi
device pci
device ata
device atadisk # ATA disk drives
device ataraid # ATA RAID drives
options ATA_STATIC_ID # Static device numbering
device scbus # SCSI bus (required for SCSI)
device da # Direct Access (disks)
device twa # 3ware 9000 series PATA/SATA RAID
device atkbdc # AT keyboard controller
device atkbd # AT keyboard
device psm # PS/2 mouse
device kbdmux # keyboard multiplexer
device vga # VGA video card driver
device splash # Splash screen and screen saver support
device sc
device agp # support several AGP chipsets
device uart # Generic UART driver
device ppc
device ppbus # Parallel port bus (required)
device lpt # Printer
device miibus # MII bus support
device re # RealTek 8139C+/8169/8169S/8110S
device loop # Network loopback
device random # Entropy device
device ether # Ethernet support
device tun # Packet tunnel.
device pty # BSD-style compatibility pseudo ttys
device md # Memory "disks"
device gif # IPv6 and IPv4 tunneling
device faith # IPv6-to-IPv4 relaying (translation)
device firmware # firmware assist module
device bpf # Berkeley packet filter
options USB_DEBUG # enable debug msgs
device uhci # UHCI PCI->USB interface
device ohci # OHCI PCI->USB interface
device ehci # EHCI PCI->USB interface (USB 2.0)
device xhci # XHCI PCI->USB interface (USB 3.0)
device usb # USB Bus (required)
device uhid # "Human Interface Devices"
device ukbd # Keyboard
device ulpt # Printer
device umass # Disks/Mass storage - Requires scbus and da
device ums # Mouse
device firewire # FireWire bus code
device pf
device pflog
device pfsync
device siis
device ahci
device pass
device ada
dmesg log, from boot to now:
Copyright (c) 1992-2012 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.0-STABLE #20: Thu Jan 19 12:26:03 CET 2012
root@tilos.hu:/usr/obj/usr/src/sys/MYKERNEL amd64
CPU: AMD Phenom(tm) II X4 945 Processor (3013.69-MHz K8-class CPU)
Origin = "AuthenticAMD" Id = 0x100f42 Family = 10 Model = 4
Stepping = 2
Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
Features2=0x802009<SSE3,MON,CX16,POPCNT>
AMD
Features=0xee500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM,3DNow!+,3DNow!>
AMD
Features2=0x37ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT>
TSC: P-state invariant
real memory = 8589934592 (8192 MB)
avail memory = 8118177792 (7742 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <GBT GBTUACPI>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 2
cpu3 (AP): APIC ID: 3
ioapic0: Changing APIC ID to 2
ioapic0 <Version 2.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <GBT GBTUACPI> on motherboard
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, d7ce0000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pcib0: Length mismatch for 3 range: 26d00000 vs 26e10000
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
vgapci0: <VGA-compatible display> port 0xce00-0xceff mem
0xd8000000-0xdfffffff,0xfdbf0000-0xfdbfffff,0xfda00000-0xfdafffff irq 18
at device 5.0 on pci1
pcib2: <ACPI PCI-PCI bridge> irq 18 at device 2.0 on pci0
pci2: <ACPI PCI bus> on pcib2
3ware device driver for 9000 series storage controllers, version:
3.80.06.003
twa0: <3ware 9000 series Storage Controller> port 0xbe00-0xbeff mem
0xfa000000-0xfbffffff,0xfd7ff000-0xfd7fffff irq 18 at device 0.0 on pci2
twa0: INFO: (0x15: 0x1300): Controller details:: Model 9650SE-8LPML, 8
ports, Firmware FE9X 3.08.00.016, BIOS BE9X 3.08.00.004
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 4.0 on pci0
pci3: <ACPI PCI bus> on pcib3
pcib4: <PCI-PCI bridge> irq 16 at device 0.0 on pci3
pci4: <PCI bus> on pcib4
siis0: <SiI3124 SATA controller> port 0xef00-0xef0f mem
0xfdfff000-0xfdfff07f,0xfdff0000-0xfdff7fff irq 16 at device 0.0 on pci4
siisch0: <SIIS channel> at channel 0 on siis0
siisch1: <SIIS channel> at channel 1 on siis0
siisch2: <SIIS channel> at channel 2 on siis0
siisch3: <SIIS channel> at channel 3 on siis0
pcib5: <ACPI PCI-PCI bridge> irq 18 at device 10.0 on pci0
pci5: <ACPI PCI bus> on pcib5
re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F PCIe Gigabit Ethernet> port
0xde00-0xdeff mem 0xfdcff000-0xfdcfffff,0xfdce0000-0xfdceffff irq 18 at
device 0.0 on pci5
re0: Using 1 MSI-X message
re0: Chip rev. 0x3c000000
re0: MAC rev. 0x00400000
miibus0: <MII bus> on re0
rgephy0: <RTL8169S/8110S/8211 1000BASE-T media interface> PHY 1 on miibus0
rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX,
100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master,
1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow,
1000baseT-FDX-flow-master, auto, auto-flow
re0: Ethernet address: 00:24:1d:d5:9e:b0
ahci0: <ATI IXP700 AHCI SATA controller> port
0xff00-0xff07,0xfe00-0xfe03,0xfd00-0xfd07,0xfc00-0xfc03,0xfb00-0xfb0f
mem 0xfe02f000-0xfe02f3ff irq 22 at device 17.0 on pci0
ahci0: AHCI v1.10 with 4 3Gbps ports, Port Multiplier supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
pci0: <serial bus, USB> at device 18.0 (no driver attached)
pci0: <serial bus, USB> at device 18.1 (no driver attached)
pci0: <serial bus, USB> at device 18.2 (no driver attached)
pci0: <serial bus, USB> at device 19.0 (no driver attached)
pci0: <serial bus, USB> at device 19.1 (no driver attached)
pci0: <serial bus, USB> at device 19.2 (no driver attached)
pci0: <serial bus, SMBus> at device 20.0 (no driver attached)
atapci0: <ATI IXP700/800 UDMA133 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xfa00-0xfa0f at device 20.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
isab0: <PCI-ISA bridge> at device 20.3 on pci0
isa0: <ISA bus> on isab0
pcib6: <ACPI PCI-PCI bridge> at device 20.4 on pci0
pci6: <ACPI PCI bus> on pcib6
fwohci0: <Texas Instruments TSB43AB23> mem
0xfd9ff000-0xfd9ff7ff,0xfd9f8000-0xfd9fbfff irq 22 at device 14.0 on pci6
fwohci0: OHCI version 1.10 (ROM=0)
fwohci0: No. of Isochronous channels is 4.
fwohci0: EUI64 00:4f:4d:c7:00:00:24:1d
fwohci0: Phy 1394a available S400, 3 ports.
fwohci0: Link S400, max_rec 2048 bytes.
firewire0: <IEEE1394(FireWire) bus> on fwohci0
fwohci0: Initiate bus reset
fwohci0: fwohci_intr_core: BUS reset
fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=1,
CYCLEMASTER mode
pci0: <serial bus, USB> at device 20.5 (no driver attached)
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77b irq 7 drq 3 on acpi0
ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/16 bytes threshold
ppbus0: <Parallel port bus> on ppc0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
attimer0: <AT timer> port 0x40-0x43 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq 0,8
on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 450
Event timer "HPET1" frequency 14318180 Hz quality 450
Event timer "HPET2" frequency 14318180 Hz quality 450
atrtc0: <AT realtime clock> port 0x70-0x73 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
orm0: <ISA Option ROMs> at iomem 0xd0000-0xd4fff,0xd5000-0xd6fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
hwpstate0: <Cool`n'Quiet 2.0> on cpu0
firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0) (me)
firewire0: bus manager 0
(noperiph:siisch0:0:-1:-1): rescan already queued
(noperiph:siisch1:0:-1:-1): rescan already queued
ZFS filesystem version 5
ZFS storage pool version 28
Timecounters tick every 1.000 msec
da0 at twa0 bus 0 scbus0 target 0 lun 0
da0: <AMCC 9650SE-8LP DISK 3.08> Fixed Direct Access SCSI-5 device
da0: 100.000MB/s transfers
da0: 1430501MB (2929666048 512 byte sectors: 255H 63S/T 182363C)
da1 at twa0 bus 0 scbus0 target 1 lun 0
da1: <AMCC 9650SE-8LP DISK 3.08> Fixed Direct Access SCSI-5 device
da1: 100.000MB/s transfers
da1: 1430501MB (2929666048 512 byte sectors: 255H 63S/T 182363C)
da2 at twa0 bus 0 scbus0 target 2 lun 0
da2: <AMCC 9650SE-8LP DISK 3.08> Fixed Direct Access SCSI-5 device
da2: 100.000MB/s transfers
da2: 1430501MB (2929666048 512 byte sectors: 255H 63S/T 182363C)
da3 at twa0 bus 0 scbus0 target 3 lun 0
da3: <AMCC 9650SE-8LP DISK 3.08> Fixed Direct Access SCSI-5 device
da3: 100.000MB/s transfers
da3: 1430501MB (2929666048 512 byte sectors: 255H 63S/T 182363C)
ada0 at siisch0 bus 0 scbus1 target 0 lun 0
ada0: <OCZ-REVODRIVE 1.20> ATA-8 SATA 2.x device
ada0: 150.000MB/s transfers (SATA 1.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 38164MB (78161328 512 byte sectors: 16H 63S/T 16383C)
ada1 at siisch1 bus 0 scbus2 target 0 lun 0
ada1: <OCZ-REVODRIVE 1.20> ATA-8 SATA 2.x device
ada1: 150.000MB/s transfers (SATA 1.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 38164MB (78161328 512 byte sectors: 16H 63S/T 16383C)
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #3 Launched!
Timecounter "TSC-low" frequency 11772208 Hz quality 800
GEOM_RAID: SiI-110316131611: Array SiI-110316131611 created.
GEOM_RAID: SiI-110316131611: Disk ada0 state changed from NONE to ACTIVE.
GEOM_RAID: SiI-110316131611: Subdisk SiI Raid0 Set:0-ada0 state changed
from NONE to STALE.
GEOM_RAID: SiI-110316131611: Disk ada1 state changed from NONE to ACTIVE.
GEOM_RAID: SiI-110316131611: Subdisk SiI Raid0 Set:1-ada1 state changed
from NONE to STALE.
GEOM_RAID: SiI-110316131611: Array started.
GEOM_RAID: SiI-110316131611: Subdisk SiI Raid0 Set:0-ada0 state changed
from STALE to ACTIVE.
GEOM_RAID: SiI-110316131611: Subdisk SiI Raid0 Set:1-ada1 state changed
from STALE to ACTIVE.
GEOM_RAID: SiI-110316131611: Volume SiI Raid0 Set state changed from
STARTING to OPTIMAL.
GEOM_RAID: SiI-110316131611: Provider raid/r0 for volume SiI Raid0 Set
created.
Trying to mount root from zfs:pool/root []...
/boot/loader.conf:
zfs_load="YES"
vfs.root.mountfrom="zfs:pool/root"
vfs.zfs.vdev.max_pending="8"
geom_raid_load="YES"
hint.siisch.0.sata_rev=1
hint.siisch.1.sata_rev=1
/etc/sysctl.conf:
vfs.zfs.l2arc_noprefetch=0
/etc/make.conf, the kernel was compiled with this settings:
CPUTYPE?=athlon64
I'd highly appreciate any help, as I am clueless with this one.
--
L?szl? K?ROLYI
http://linkedin.com/in/karolyi