On the serial console I see
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 74, size: 4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 128, size: 20480
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 69, size: 4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 6, size: 4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 74, size: 4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 128, size: 20480
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 69, size: 4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 6, size: 4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 74, size: 4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 128, size: 20480
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 69, size: 4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 6, size: 4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 74, size: 4096
swap_pager: indefinite wait buffer: bufobj: 0, blkno: 128, size: 20480
and on a session I had open from before
# killall -9 watchdogd
just hangs, I guess because its having trouble reading from the disk.
If I hit CTRL+t, I see
load: 0.00 cmd: csh 73167 [vnread] 22.32r 0.00u 0.00s 0% 3232k
load: 0.00 cmd: csh 73167 [vnread] 22.65r 0.00u 0.00s 0% 3232k
load: 0.00 cmd: csh 73167 [vnread] 22.96r 0.00u 0.00s 0% 3232k
load: 0.00 cmd: csh 73167 [vnread] 23.20r 0.00u 0.00s 0% 3232k
load: 0.00 cmd: csh 73167 [vnread] 23.40r 0.00u 0.00s 0% 3232k
load: 0.00 cmd: csh 73167 [vnread] 23.61r 0.00u 0.00s 0% 3232k
Its RELENG_8 amd64 from July 13th and the swap is on an ARECA drive
and I dont see any errors on any of the raidset members. I also have
a large zfs spool and a small mount point on a 3ware controller but
unfortunately, nothing in the logs post reboot and nothing from smartctl
cat /var/run/dmesg.boot
Copyright (c) 1992-2010 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.1-PRERELEASE #0: Tue Jul 13 09:55:48 EDT 2010
mdtancsa@backup3.sentex.ca:/usr/obj/usr/src/sys/backup amd64
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz (2400.10-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x6fb Family = 6 Model = f
Stepping = 11
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0xe3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM>
AMD Features=0x20100800<SYSCALL,NX,LM>
AMD Features2=0x1<LAHF>
TSC: P-state invariant
real memory = 8589934592 (8192 MB)
avail memory = 8267673600 (7884 MB)
ACPI APIC Table: <A_M_I_ OEMAPIC >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 2
cpu3 (AP): APIC ID: 3
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <A_M_I_ OEMXSDT> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of fed08000, 1000 (3) failed
acpi0: reservation of fed1c000, 4000 (3) failed
acpi0: reservation of fed20000, 20000 (3) failed
acpi0: reservation of fed50000, 40000 (3) failed
acpi0: reservation of ffc00000, 300000 (3) failed
acpi0: reservation of fec00000, 1000 (3) failed
acpi0: reservation of fee00000, 1000 (3) failed
acpi0: reservation of e0000000, 10000000 (3) failed
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, dff00000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
cpu0: <ACPI CPU> on acpi0
ACPI Warning: Incorrect checksum in table [OEMB] - 0xD1, should be
0xD0 (20100331/tbutils-354)
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on
acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <PCI-PCI bridge> at device 0.0 on pci1
pci3: <PCI bus> on pcib2
arcmsr0: <Areca SATA Host Adapter RAID Controller
> mem 0xfc9ff000-0xfc9fffff irq 18 at device 14.0 on pci3
ARECA RAID ADAPTER0: Driver Version 1.20.00.16 2009-10-10
ARECA RAID ADAPTER0: FIRMWARE VERSION V1.44 2008-2-1
arcmsr0: [ITHREAD]
pcib3: <PCI-PCI bridge> at device 0.2 on pci1
pci2: <PCI bus> on pcib3
uhci0: <Intel 82801JI (ICH10) USB controller USB-D> port
0x7800-0x781f irq 16 at device 26.0 on pci0
uhci0: [ITHREAD]
usbus0: <Intel 82801JI (ICH10) USB controller USB-D> on uhci0
uhci1: <Intel 82801JI (ICH10) USB controller USB-E> port
0x7880-0x789f irq 21 at device 26.1 on pci0
uhci1: [ITHREAD]
usbus1: <Intel 82801JI (ICH10) USB controller USB-E> on uhci1
uhci2: <Intel 82801JI (ICH10) USB controller USB-F> port
0x7c00-0x7c1f irq 18 at device 26.2 on pci0
uhci2: [ITHREAD]
usbus2: <Intel 82801JI (ICH10) USB controller USB-F> on uhci2
ehci0: <Intel 82801JI (ICH10) USB 2.0 controller USB-B> mem
0xfc8ffc00-0xfc8fffff irq 18 at device 26.7 on pci0
ehci0: [ITHREAD]
usbus3: EHCI version 1.0
usbus3: <Intel 82801JI (ICH10) USB 2.0 controller USB-B> on ehci0
pci0: <multimedia, HDA> at device 27.0 (no driver attached)
pcib4: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
pci9: <ACPI PCI bus> on pcib4
em0: <Intel(R) PRO/1000 Network Connection 7.0.5> port 0xdc00-0xdc1f
mem 0xfcfe0000-0xfcffffff,0xfcf00000-0xfcf7ffff,0xfcfdc000-0xfcfdffff
irq 16 at device 0.0 on pci9
em0: Using MSI interrupt
em0: [FILTER]
em0: Ethernet address: 00:1b:21:3f:62:72
pcib5: <ACPI PCI-PCI bridge> irq 16 at device 28.1 on pci0
pci8: <ACPI PCI bus> on pcib5
siis0: <SiI3132 SATA controller> port 0xcc00-0xcc7f mem
0xfceffc00-0xfceffc7f,0xfcef8000-0xfcefbfff irq 17 at device 0.0 on pci8
siis0: [ITHREAD]
siisch0: <SIIS channel> at channel 0 on siis0
siisch0: [ITHREAD]
siisch1: <SIIS channel> at channel 1 on siis0
siisch1: [ITHREAD]
pcib6: <ACPI PCI-PCI bridge> irq 18 at device 28.2 on pci0
pci7: <ACPI PCI bus> on pcib6
3ware device driver for 9000 series storage controllers, version: 3.80.06.002
twa0: <3ware 9000 series Storage Controller> port 0xb800-0xb8ff mem
0xfa000000-0xfbffffff,0xfcdff000-0xfcdfffff irq 18 at device 0.0 on pci7
twa0: [ITHREAD]
twa0: WARNING: (0x04: 0x0008): Unclean shutdown detected: unit=0
twa0: INFO: (0x15: 0x1300): Controller details:: Model 9650SE-2LP, 2
ports, Firmware FE9X 3.08.00.016, BIOS BE9X 3.08.00.004
pcib7: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0
pci6: <ACPI PCI bus> on pcib7
fwohci0: <1394 Open Host Controller Interface> port 0xa800-0xa8ff mem
0xfccff800-0xfccfffff irq 19 at device 0.0 on pci6
fwohci0: [ITHREAD]
fwohci0: OHCI version 1.10 (ROM=1)
fwohci0: No. of Isochronous channels is 4.
fwohci0: EUI64 00:1e:8c:00:00:c4:10:80
fwohci0: Phy 1394a available S400, 2 ports.
fwohci0: Link S400, max_rec 2048 bytes.
firewire0: <IEEE1394(FireWire) bus> on fwohci0
dcons_crom0: <dcons configuration ROM> on firewire0
dcons_crom0: bus_addr 0x8eacc0
fwe0: <Ethernet over FireWire> on firewire0
if_fwe0: Fake Ethernet address: 02:1e:8c:c4:10:80
fwe0: Ethernet address: 02:1e:8c:c4:10:80
fwip0: <IP over FireWire> on firewire0
fwip0: Firewire address: 00:1e:8c:00:00:c4:10:80 @ 0xfffe00000000,
S400, maxrec 2048
fwohci0: Initiate bus reset
fwohci0: fwohci_intr_core: BUS reset
fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=1, CYCLEMASTER mode
pcib8: <ACPI PCI-PCI bridge> irq 17 at device 28.4 on pci0
pci5: <ACPI PCI bus> on pcib8
ahci0: <JMicron JMB361 AHCI SATA controller> mem
0xfcbfa000-0xfcbfbfff irq 16 at device 0.0 on pci5
ahci0: [ITHREAD]
ahci0: AHCI v1.00 with 2 3Gbps ports, Port Multiplier supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich0: [ITHREAD]
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich1: [ITHREAD]
atapci0: <JMicron JMB361 UDMA133 controller> port
0x9c00-0x9c07,0x9880-0x9883,0x9800-0x9807,0x9480-0x9483,0x9400-0x940f
irq 17 at device 0.1 on pci5
atapci0: [ITHREAD]
ata2: <ATA channel 0> on atapci0
ata2: [ITHREAD]
pcib9: <ACPI PCI-PCI bridge> irq 16 at device 28.5 on pci0
pci4: <ACPI PCI bus> on pcib9
ale0: <Atheros AR8121/AR8113/AR8114 PCIe Ethernet> port 0x8c00-0x8c7f
mem 0xfcac0000-0xfcafffff irq 17 at device 0.0 on pci4
ale0: 960 Tx FIFO, 1024 Rx FIFO
ale0: Using 1 MSI messages.
miibus0: <MII bus> on ale0
atphy0: <Atheros F1 10/100/1000 PHY> PHY 0 on miibus0
atphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, auto
ale0: Ethernet address: e0:cb:4e:42:4b:37
ale0: [FILTER]
uhci3: <Intel 82801JI (ICH10) USB controller USB-A> port
0x7080-0x709f irq 23 at device 29.0 on pci0
uhci3: [ITHREAD]
usbus4: <Intel 82801JI (ICH10) USB controller USB-A> on uhci3
uhci4: <Intel 82801JI (ICH10) USB controller USB-B> port
0x7400-0x741f irq 19 at device 29.1 on pci0
uhci4: [ITHREAD]
usbus5: <Intel 82801JI (ICH10) USB controller USB-B> on uhci4
uhci5: <Intel 82801JI (ICH10) USB controller USB-C> port
0x7480-0x749f irq 18 at device 29.2 on pci0
uhci5: [ITHREAD]
usbus6: <Intel 82801JI (ICH10) USB controller USB-C> on uhci5
ehci1: <Intel 82801JI (ICH10) USB 2.0 controller USB-A> mem
0xfc8ff800-0xfc8ffbff irq 23 at device 29.7 on pci0
ehci1: [ITHREAD]
usbus7: EHCI version 1.0
usbus7: <Intel 82801JI (ICH10) USB 2.0 controller USB-A> on ehci1
pcib10: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci10: <ACPI PCI bus> on pcib10
vgapci0: <VGA-compatible display> port 0xe000-0xe0ff mem
0xfd000000-0xfdffffff,0xfebff000-0xfebfffff irq 16 at device 0.0 on pci10
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
ahci1: <Intel ICH10 AHCI SATA controller> port
0x6c00-0x6c07,0x6880-0x6883,0x6800-0x6807,0x6480-0x6483,0x6400-0x641f
mem 0xfc8fe800-0xfc8fefff irq 19 at device 31.2 on pci0
ahci1: [ITHREAD]
ahci1: AHCI v1.20 with 6 3Gbps ports, Port Multiplier supported
ahcich2: <AHCI channel> at channel 0 on ahci1
ahcich2: [ITHREAD]
ahcich3: <AHCI channel> at channel 1 on ahci1
ahcich3: [ITHREAD]
ahcich4: <AHCI channel> at channel 2 on ahci1
ahcich4: [ITHREAD]
ahcich5: <AHCI channel> at channel 3 on ahci1
ahcich5: [ITHREAD]
ahcich6: <AHCI channel> at channel 4 on ahci1
ahcich6: [ITHREAD]
ahcich7: <AHCI channel> at channel 5 on ahci1
ahcich7: [ITHREAD]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
uart0: console (9600,n,8,1)
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
orm0: <ISA Option ROMs> at iomem
0xc0000-0xc97ff,0xc9800-0xca7ff,0xca800-0xcc7ff,0xd4800-0xd77ff,0xd7800-0xd87ff
on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
est1: <Enhanced SpeedStep Frequency Control> on cpu1
p4tcc1: <CPU Frequency Thermal Control> on cpu1
est2: <Enhanced SpeedStep Frequency Control> on cpu2
p4tcc2: <CPU Frequency Thermal Control> on cpu2
est3: <Enhanced SpeedStep Frequency Control> on cpu3
p4tcc3: <CPU Frequency Thermal Control> on cpu3
Timecounters tick every 1.000 msec
firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0) (me)
firewire0: bus manager 0
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
usbus4: 12Mbps Full Speed USB v1.0
usbus5: 12Mbps Full Speed USB v1.0
usbus6: 12Mbps Full Speed USB v1.0
usbus7: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <Intel> at usbus3
uhub3: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
ugen4.1: <Intel> at usbus4
uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
ugen5.1: <Intel> at usbus5
uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5
ugen6.1: <Intel> at usbus6
uhub6: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6
ugen7.1: <Intel> at usbus7
uhub7: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus7
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub4: 2 ports with 2 removable, self powered
uhub5: 2 ports with 2 removable, self powered
uhub6: 2 ports with 2 removable, self powered
(probe16:arcmsr0:0:16:0): inquiry data fails comparison at DV1 step
da0 at arcmsr0 bus 0 scbus0 target 0 lun 0
da0: <Areca usrvar R001> Fixed Direct Access SCSI-5 device
da0: 166.666MB/s transfers (83.333MHz, offset 32, 16bit)
da0: Command Queueing enabled
da0: 76293MB (156249600 512 byte sectors: 255H 63S/T 9726C)
da1 at arcmsr0 bus 0 scbus0 target 0 lun 1
da1: <Areca backup1 R001> Fixed Direct Access SCSI-5 device
da1: 166.666MB/s transfers (83.333MHz, offset 32, 16bit)
da1: Command Queueing enabled
da1: 2784728MB (5703123456 512 byte sectors: 255H 63S/T 355003C)
ada0 at ahcich2 bus 0 scbus6 target 0 lun 0da2 at twa0 bus 0 scbus3
target 0 lun 0
da2: <AMCC 9650SE-2LP DISK 3.08> Fixed Direct Access SCSI-5 device
da2: 100.000MB/s transfers
da2: 66747MB (136697856 512 byte sectors: 255H 63S/T 8509C)
ada0: <ST31000340AS SD1A> ATA-8 SATA 2.x device
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada1 at ahcich3 bus 0 scbus7 target 0 lun 0
ada1: <ST31000340AS SD15> ATA-8 SATA 2.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada2 at ahcich4 bus 0 scbus8 target 0 lun 0
ada2: <ST31000333AS SD35> ATA-8 SATA 2.x device
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada3 at ahcich5 bus 0 scbus9 target 0 lun 0
ada3: <ST31000528AS CC35> ATA-8 SATA 2.x device
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
pass2 at arcmsr0 bus 0 scbus0 target 16 lun 0
pass2: <Areca RAID controller R001> Fixed Processor SCSI-0 device
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #3 Launched!
uhub3: 6 ports with 6 removable, self powered
uhub7: 6 ports with 6 removable, self powered
Root mount waiting for: usbus7
Trying to mount root from ufs:/dev/da2s1a
WARNING: / was not properly dismounted
ZFS filesystem version 3
ZFS storage pool version 14
ugen5.2: <American Power Conversion> at usbus5
twa0: INFO: (0x04: 0x000C): Initialize started: unit=0
em0: link state changed to UP
ale0: link state changed to UP
---Mike
--------------------------------------------------------------------
Mike Tancsa, tel +1 519 651 3400
Sentex Communications, mike@sentex.net
Providing Internet since 1994 www.sentex.net
Cambridge, Ontario Canada www.sentex.net/mike
On Sun, Jul 18, 2010 at 05:08:09PM -0400, Mike Tancsa wrote:> > > On the serial console I see > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 74, size: 4096 > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 128, size: 20480 > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 69, size: 4096 > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 6, size: 4096 > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 74, size: 4096 > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 128, size: 20480 > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 69, size: 4096 > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 6, size: 4096 > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 74, size: 4096 > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 128, size: 20480 > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 69, size: 4096 > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 6, size: 4096 > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 74, size: 4096 > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 128, size: 20480 > [...] > load: 0.00 cmd: csh 73167 [vnread] 22.32r 0.00u 0.00s 0% 3232k > load: 0.00 cmd: csh 73167 [vnread] 22.65r 0.00u 0.00s 0% 3232k > load: 0.00 cmd: csh 73167 [vnread] 22.96r 0.00u 0.00s 0% 3232k > load: 0.00 cmd: csh 73167 [vnread] 23.20r 0.00u 0.00s 0% 3232k > load: 0.00 cmd: csh 73167 [vnread] 23.40r 0.00u 0.00s 0% 3232k > load: 0.00 cmd: csh 73167 [vnread] 23.61r 0.00u 0.00s 0% 3232kWhere exactly is your swap partition? I ask because of the below paragraph starting with "Its RELENG_8 amd64 from July 13th ...".> Its RELENG_8 amd64 from July 13th and the swap is on an ARECA drive > and I dont see any errors on any of the raidset members. I also have > a large zfs spool and a small mount point on a 3ware controller but > unfortunately, nothing in the logs post reboot and nothing from > smartctlIf you Google for "swap_pager: indefinite wait buffer: bufobj" you'll find this is a pretty well-established problem, but the situation varies per person. A common one is here (read the entire thread): http://www.mail-archive.com/freebsd-questions@freebsd.org/msg192481.html I have no advice as far as how to solve this problem. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
> > just hangs, I guess because its having trouble reading from the disk. > If I hit CTRL+t, I see > > load: 0.00 cmd: csh 73167 [vnread] 22.32r 0.00u 0.00s 0% 3232k > load: 0.00 cmd: csh 73167 [vnread] 22.65r 0.00u 0.00s 0% 3232k > load: 0.00 cmd: csh 73167 [vnread] 22.96r 0.00u 0.00s 0% 3232k > load: 0.00 cmd: csh 73167 [vnread] 23.20r 0.00u 0.00s 0% 3232k > load: 0.00 cmd: csh 73167 [vnread] 23.40r 0.00u 0.00s 0% 3232k > load: 0.00 cmd: csh 73167 [vnread] 23.61r 0.00u 0.00s 0% 3232k > >Hi, i have similar problems on a i7 system with a 3ware 9650SE controller and a simple 2 disk RAID1 configuration. I can trigger it by just extracting the ports tree onto the raid. It usually stops several times for over a minute doing nothing before continuing. While building KDE it stopped for good when extracting a port. The bsdtar process was hanging there in the wdrain status. Waited over 60 minutes before interrupting the process. I haven't seen any messages in dmesg. I'll try to build with debug support tonight and see if it makes a difference. The version was the one delivered with the PC-BSD 8.1RC discs but also after upgrading to the newest RELENG_8 sources the problem persists. The hardware is fairly new and other OSes show no problems so i'm inclined to say that the hardware isn't faulty ;) I could also try to install a 8.0 system if it helps to determine if a regression in 8.1 is the problem.
On 3/25/2011 6:29 AM, Steven Hartland wrote:> ----- Original Message ----- From: "Jeremy Chadwick" > <freebsd@jdc.parodius.com> > > Was there any conclusion from this guys, was there a bad disk > causing the issue?You mean this old thread ? http://lists.freebsd.org/pipermail/freebsd-stable/2010-July/057874.html I would say probably the disk mostly. Perhaps a driver or firmware bug on the Areca. Hard to say. The drive totally failed a month or so later. Also, moved to a later firmware on the areaca controller after that and all has been quite stable on the box except for an odd em driver bug. However, version 7.2.2 fixed that. Here is the dmesg of the box right now. Copyright (c) 1992-2011 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.2-PRERELEASE #0: Wed Feb 23 10:00:14 EST 2011 mdtancsa@backup3.sentex.ca:/usr/obj/usr/src/sys/backup amd64 module_register: module pci/em already exists! Module pci/em failed to register: 17 module_register: module pci/lem already exists! Module pci/lem failed to register: 17 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Core(TM) i5 CPU 660 @ 3.33GHz (3333.31-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x20652 Family = 6 Model = 25 Stepping = 2 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x298e3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT,AESNI> AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM> AMD Features2=0x1<LAHF> TSC: P-state invariant real memory = 8589934592 (8192 MB) avail memory = 8186777600 (7807 MB) ACPI APIC Table: <INTEL S3420GPX> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 SMT threads cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 4 cpu3 (AP): APIC ID: 5 ioapic0 <Version 2.0> irqs 0-23 on motherboard lapic0: Forcing LINT1 to edge trigger kbd1 at kbdmux0 cryptosoft0: <software crypto> on motherboard aesni0: <AES-CBC,AES-XTS> on motherboard acpi0: <INTEL S3420GPX> on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 cpu0: <ACPI CPU> on acpi0 cpu1: <ACPI CPU> on acpi0 cpu2: <ACPI CPU> on acpi0 cpu3: <ACPI CPU> on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0 pci1: <ACPI PCI bus> on pcib1 pcib2: <ACPI PCI-PCI bridge> at device 0.0 on pci1 pci2: <ACPI PCI bus> on pcib2 pcib3: <ACPI PCI-PCI bridge> at device 2.0 on pci2 pci3: <ACPI PCI bus> on pcib3 pcib4: <ACPI PCI-PCI bridge> at device 3.0 on pci2 pci4: <ACPI PCI bus> on pcib4 pcib5: <ACPI PCI-PCI bridge> at device 0.0 on pci4 pci5: <ACPI PCI bus> on pcib5 siis0: <SiI3124 SATA controller> port 0x3000-0x300f mem 0xb4408000-0xb440807f,0xb4400000-0xb4407fff irq 19 at device 0.0 on pci5 siis0: [ITHREAD] siisch0: <SIIS channel> at channel 0 on siis0 siisch0: [ITHREAD] siisch1: <SIIS channel> at channel 1 on siis0 siisch1: [ITHREAD] siisch2: <SIIS channel> at channel 2 on siis0 siisch2: [ITHREAD] siisch3: <SIIS channel> at channel 3 on siis0 siisch3: [ITHREAD] pcib6: <ACPI PCI-PCI bridge> at device 4.0 on pci2 pci6: <ACPI PCI bus> on pcib6 bge0: <HPQ 10/100/1000 Copper Based Gigabit Adapter, ASIC rev. 0x004001> mem 0xb4300000-0xb430ffff irq 16 at device 0.0 on pci6 bge0: CHIP ID 0x00004001; ASIC REV 0x04; CHIP REV 0x40; PCI-E miibus0: <MII bus> on bge0 brgphy0: <BCM5750 10/100/1000baseTX PHY> PHY 1 on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow bge0: Ethernet address: 00:10:18:14:27:d5 bge0: [ITHREAD] pcib7: <ACPI PCI-PCI bridge> irq 16 at device 6.0 on pci0 pci7: <ACPI PCI bus> on pcib7 pcib8: <ACPI PCI-PCI bridge> at device 0.0 on pci7 pci8: <ACPI PCI bus> on pcib8 arcmsr0: <Areca SATA Host Adapter RAID Controller> mem 0xb4200000-0xb4200fff irq 18 at device 14.0 on pci8ARECA RAID ADAPTER0: Driver Version 1.20.00.19 2010-11-11 ARECA RAID ADAPTER0: FIRMWARE VERSION V1.44 2008-2-1 arcmsr0: [ITHREAD] pcib9: <ACPI PCI-PCI bridge> at device 0.2 on pci7 pci9: <ACPI PCI bus> on pcib9 em0: <Intel(R) PRO/1000 Network Connection 7.2.2> port 0x4040-0x405f mem 0xb4500000-0xb451ffff,0xb4525000-0xb4525fff irq 16 at device 25.0 on pci0 em0: Using an MSI interrupt em0: [FILTER] em0: Ethernet address: 00:15:17:ed:68:a5 ehci0: <Intel PCH USB 2.0 controller USB-B> mem 0xb4522000-0xb45223ff irq 21 at device 26.0 on pci0 ehci0: [ITHREAD] usbus0: EHCI version 1.0 usbus0: <Intel PCH USB 2.0 controller USB-B> on ehci0 pcib10: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0 pci10: <ACPI PCI bus> on pcib10 pcib11: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0 pci11: <ACPI PCI bus> on pcib11 em1: <Intel(R) PRO/1000 Network Connection 7.2.2> port 0x2000-0x201f mem 0xb4100000-0xb411ffff,0xb4120000-0xb4123fff irq 16 at device 0.0 on pci11 em1: Using MSIX interrupts with 3 vectors em1: [ITHREAD] em1: [ITHREAD] em1: [ITHREAD] em1: Ethernet address: 00:15:17:ed:68:a4 pcib12: <ACPI PCI-PCI bridge> irq 18 at device 28.6 on pci0 pci12: <ACPI PCI bus> on pcib12 vgapci0: <VGA-compatible display> mem 0xb2000000-0xb2ffffff,0xb3800000-0xb3803fff,0xb3000000-0xb37fffff irq 17 at device 0.0 on pci12 pcib13: <ACPI PCI-PCI bridge> irq 19 at device 28.7 on pci0 pci13: <ACPI PCI bus> on pcib13 3ware device driver for 9000 series storage controllers, version: 3.80.06.003 twa0: <3ware 9000 series Storage Controller> port 0x1000-0x10ff mem 0xb0000000-0xb1ffffff,0xb4000000-0xb4000fff irq 19 at device 0.0 on pci13 twa0: [ITHREAD] twa0: INFO: (0x15: 0x1300): Controller details:: Model 9650SE-2LP, 2 ports, Firmware FE9X 3.08.00.016, BIOS BE9X 3.08.00.004 ehci1: <Intel PCH USB 2.0 controller USB-A> mem 0xb4521000-0xb45213ff irq 23 at device 29.0 on pci0 ehci1: [ITHREAD] usbus1: EHCI version 1.0 usbus1: <Intel PCH USB 2.0 controller USB-A> on ehci1 pcib14: <ACPI PCI-PCI bridge> at device 30.0 on pci0 pci14: <ACPI PCI bus> on pcib14 isab0: <PCI-ISA bridge> at device 31.0 on pci0 isa0: <ISA bus> on isab0 ahci0: <Intel 5 Series/3400 Series AHCI SATA controller> port 0x4068-0x406f,0x4074-0x4077,0x4060-0x4067,0x4070-0x4073,0x4020-0x403f mem 0xb4520000-0xb45207ff irq 18 at device 31.2 on pci0 ahci0: [ITHREAD] ahci0: AHCI v1.30 with 6 3Gbps ports, Port Multiplier not supported ahcich0: <AHCI channel> at channel 0 on ahci0 ahcich0: [ITHREAD] ahcich1: <AHCI channel> at channel 1 on ahci0 ahcich1: [ITHREAD] ahcich2: <AHCI channel> at channel 2 on ahci0 ahcich2: [ITHREAD] ahcich3: <AHCI channel> at channel 3 on ahci0 ahcich3: [ITHREAD] ahcich4: <AHCI channel> at channel 4 on ahci0 ahcich4: [ITHREAD] ahcich5: <AHCI channel> at channel 5 on ahci0 ahcich5: [ITHREAD] pci0: <serial bus, SMBus> at device 31.3 (no driver attached) acpi_button0: <Sleep Button> on acpi0 atrtc0: <AT realtime clock> port 0x70-0x71,0x74-0x77 irq 8 on acpi0 acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 900 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] uart0: console (115200,n,8,1) uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 uart1: [FILTER] orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xcb000-0xcffff,0xd0000-0xd1fff,0xd3800-0xd57ff on isa0 sc0: <System console> at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd: unable to get the current command byte value. atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] psm0: unable to get the current command byte value. device_attach: acpi_perf1 attach returned 6 device_attach: acpi_perf3 attach returned 6 coretemp0: <CPU On-Die Thermal Sensors> on cpu0 coretemp0: Tj(target) value 105 does not seem right. est0: <Enhanced SpeedStep Frequency Control> on cpu0 p4tcc0: <CPU Frequency Thermal Control> on cpu0 device_attach: acpi_perf1 attach returned 6 coretemp1: <CPU On-Die Thermal Sensors> on cpu1 coretemp1: Tj(target) value 105 does not seem right. est1: <Enhanced SpeedStep Frequency Control> on cpu1 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 19 device_attach: est1 attach returned 6 p4tcc1: <CPU Frequency Thermal Control> on cpu1 coretemp2: <CPU On-Die Thermal Sensors> on cpu2 coretemp2: Tj(target) value 105 does not seem right. est2: <Enhanced SpeedStep Frequency Control> on cpu2 p4tcc2: <CPU Frequency Thermal Control> on cpu2 device_attach: acpi_perf3 attach returned 6 coretemp3: <CPU On-Die Thermal Sensors> on cpu3 coretemp3: Tj(target) value 105 does not seem right. est3: <Enhanced SpeedStep Frequency Control> on cpu3 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 19 device_attach: est3 attach returned 6 p4tcc3: <CPU Frequency Thermal Control> on cpu3 Timecounters tick every 1.000 msec IPsec: Initialized Security Association Processing.(noperiph:siisch0:0:-1:-1): rescan already queued usbus0: 480Mbps High Speed USB v2.0 usbus1: 480Mbps High Speed USB v2.0 ugen0.1: <Intel> at usbus0 uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0 ugen1.1: <Intel> at usbus1 uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1 pmp0 at siisch0 bus 0 scbus0 target 15 lun 0 pmp0: <Port Multiplier 47261095 1f06> ATA-0 device pmp0: 300.000MB/s transfers (SATA 2.x, NONE, PIO 8192bytes) pmp0: 5 fan-out ports uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered ugen0.2: <vendor 0x8087> at usbus0 uhub2: <vendor 0x8087 product 0x0020, class 9/0, rev 2.00/0.00, addr 2> on usbus0 ugen1.2: <vendor 0x8087> at usbus1 uhub3: <vendor 0x8087 product 0x0020, class 9/0, rev 2.00/0.00, addr 2> on usbus1 uhub2: 6 ports with 6 removable, self powered uhub3: 8 ports with 8 removable, self powered ugen0.3: <American Power Conversion> at usbus0 ugen1.3: <American Megatrends Inc.> at usbus1 ukbd1: <Keyboard Interface> on usbus1 kbd2 at ukbd1 ums0: <Mouse Interface> on usbus1 ums0: 3 buttons and [Z] coordinates ID=0 da0 at arcmsr0 bus 0 scbus4 target 0 lun 0 da0: <Areca usrvar R001> Fixed Direct Access SCSI-5 device da0: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) da0: Command Queueing enabled da0: 76293MB (156249600 512 byte sectors: 255H 63S/T 9726C) da1 at arcmsr0 bus 0 scbus4 target 0 lun 1 da1: <Areca backup1 R001> Fixed Direct Access SCSI-5 device da1: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) da1: Command Queueing enabled da1: 2784728MB (5703123456 512 byte sectors: 255H 63S/T 355003C) ada0 at siisch0 bus 0 scbus0 target 0 lun 0da2 at twa0 bus 0 scbus5 target 0 lun 0 da2: <AMCC 9650SE-2LP DISK 3.08> Fixed Direct Access SCSI-5 device da2: 100.000MB/s transfers da2: 66747MB (136697856 512 byte sectors: 255H 63S/T 8509C) ada0: <WDC WD2001FASS-00U0B0 01.00101> ATA-8 SATA 2.x device ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada1 at siisch0 bus 0 scbus0 target 1 lun 0 ada1: <WDC WD2001FASS-00U0B0 01.00101> ATA-8 SATA 2.x device ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada1: Command Queueing enabled ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada2 at siisch0 bus 0 scbus0 target 2 lun 0 ada2: <WDC WD2001FASS-00U0B0 01.00101> ATA-8 SATA 2.x device ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada2: Command Queueing enabled ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada3 at siisch0 bus 0 scbus0 target 3 lun 0 ada3: <WDC WD2001FASS-00U0B0 01.00101> ATA-8 SATA 2.x device ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada3: Command Queueing enabled ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada4 at ahcich0 bus 0 scbus6 target 0 lun 0 ada4: <ST31000333AS SD35> ATA-8 SATA 2.x device ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada4: Command Queueing enabled ada4: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) ada5 at ahcich1 bus 0 scbus7 target 0 lun 0 ada5: <ST31000528AS CC35> ATA-8 SATA 2.x device ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada5: Command Queueing enabled ada5: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) ada6 at ahcich2 bus 0 scbus8 target 0 lun 0 ada6: <ST31000340AS SD1A> ATA-8 SATA 2.x device ada6: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada6: Command Queueing enabled ada6: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) ada7 at ahcich5 bus 0 scbus11 target 0 lun 0 ada7: <WDC WD1002FAEX-00Z3A0 05.01D05> ATA-8 SATA 3.x device ada7: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada7: Command Queueing enabled ada7: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) pass7 at arcmsr0 bus 0 scbus4 target 16 lun 0 pass7: <Areca RAID controller R001> Fixed Processor SCSI-0 device lapic1: Forcing LINT1 to edge trigger SMP: AP CPU #1 Launched! lapic5: Forcing LINT1 to edge trigger SMP: AP CPU #3 Launched! lapic4: Forcing LINT1 to edge trigger SMP: AP CPU #2 Launched! ugen1.4: <USB> at usbus1 ukbd0: <USB USB Keykoard, class 0/0, rev 1.10/1.10, addr 4> on usbus1 kbd3 at ukbd0 uhid0: <USB USB Keykoard, class 0/0, rev 1.10/1.10, addr 4> on usbus1 Root mount waiting for: usbus0 Trying to mount root from ufs:/dev/da2s1a ZFS filesystem version 4 ZFS storage pool version 15 em0: link state changed to UP em1: link state changed to UP ---Mike -- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/
----- Original Message ----- From: "Mike Tancsa" <mike@sentex.net>> I would say probably the disk mostly. Perhaps a driver or firmware bug > on the Areca. Hard to say. The drive totally failed a month or so > later. Also, moved to a later firmware on the areaca controller after > that and all has been quite stable on the box except for an odd em > driver bug. However, version 7.2.2 fixed that.Thanks for that Mike, been having some problems on a core box here where it would just hang for periods of time during which disk IO would drop to nothing and then it would just suddenly recover. We suspect one of the disks is at fault but as with you said disk hasnt failed so we where just going on the decreased smart values. Confirmation that you had a disk failure soon after is really helpful :)> Copyright (c) 1992-2011 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 8.2-PRERELEASE #0: Wed Feb 23 10:00:14 EST 2011 > mdtancsa@backup3.sentex.ca:/usr/obj/usr/src/sys/backup amd64 > module_register: module pci/em already exists! > Module pci/em failed to register: 17 > module_register: module pci/lem already exists! > Module pci/lem failed to register: 17Looks like your trying to load an em module when its already compiled into the kernel here, so you many running the driver you think you are ;-) Thanks again. Regards Steve ===============================================This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk.