FreeBSD dash.ish.com.au 7.0-RELEASE FreeBSD 7.0-RELEASE #10
i386 with PAE (5Gb RAM)
We've had fairly reproducible freezes. After several hours of stress
testing or even overnight not doing anything, everything locks up
including the console.
We then installed a debugging kernel (without INVARIANTS since that
prevented the kernel from compiling at all) and obtained this LOR when
it froze:
LOR:
1st 0x807d3d90 sleepq chain (sleepq chain) @/usr/src/sys/kern/
subr_sleepqueue.c:773
2nd 0x807c8110 scrlock (scrlock) @/usr/src/sys/dev/syscons/syscons.c:
2526
I have taken photographs of the KDB output following this but have not
transcribed it until someone says that it will be useful to them. I
could put it up as slightly fuzzy screen photographs on our web site.
I am somewhat concerned about the combination of PAE (being slightly
old technology now) and zfs (being cutting edge), not having a huge
amount of testing against each other. However nothing in this lock
seems to suggest zfs to me. The only unusual thing on this box is this:
hw.physmem: 1063911424
actual memory is half that, but I thought this might be a side effect
of PAE.
Then, restarting the machine and it hit a couple of LOR errors in
quick succession. I can't see any exact matches to
http://sources.zabbadoz.net/freebsd/lor.html
However the box boots normally after these LOR, so they may not be
fatal and may not be related to the above, but just in case...
lock order reversal:
1st 0x862f9204 inp (udpinp) @ /usr/src/sys/netinet/udp_usrreq.c:843
2nd 0x8081f498 PFil hook read/write mutex (PFil hook read/write
mutex) @ /usr/src/sys/net/pfil.c:73
KDB: stack backtrace:
db_trace_self_wrapper(807430c0,c3d639fc,
80438ff5,80744463,8081f498,...) at
db_trace_self_wrapper+0x26
kdb_backtrace(80744463,8081f498,8074a8c5,8074a8c5,8074a8ad,...) at
kdb_backtrace+0x29
witness_checkorder(8081f498,1,8074a8ad,49,807528d7,...) at
witness_checkorder+0x5e5
_rw_rlock(8081f498,8074a8ad,49,0,c3d63ab8,...) at _rw_rlock+0x2a
pfil_run_hooks(8081f480,c3d63ad8,83931800,2,862f9168,...) at
pfil_run_hooks+0x35
ip_output(867ea700,0,c3d63a9c,0,0,...) at ip_output+0x86f
udp_send(862f77bc,0,867ea700,83c91bb0,0,...) at udp_send+0x57b
sosend_dgram(862f77bc,83c91bb0,c3d63bd4,867ea700,0,...) at sosend_dgram
+0x356
sosend(862f77bc,83c91bb0,c3d63bd4,0,0,...) at sosend+0x3f
kern_sendit(84cad660,1e,c3d63c58,0,0,...) at kern_sendit+0x106
sendit(0,1,c3d63c54,28,83c91c60,...) at sendit+0xb1
sendmsg(84cad660,c3d63cfc,c,84cad660,807845c0,...) at sendmsg+0x71
syscall(c3d63d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (28, FreeBSD ELF32, sendmsg), eip = 0x2842415b, esp =
0x7f3fc7fc, ebp = 0x7f3fc818 ---
lock order reversal:
1st 0x8082010c tcp (tcp) @ /usr/src/sys/netinet/tcp_input.c:400
2nd 0x8081f498 PFil hook read/write mutex (PFil hook read/write
mutex) @ /usr/src/sys/net/pfil.c:73
KDB: stack backtrace:
db_trace_self_wrapper(807430c0,c16039ec,
80438ff5,80744463,8081f498,...) at db_trace_self_wrapper+0x26
kdb_backtrace(80744463,8081f498,8074a8c5,8074a8c5,8074a8ad,...) at
kdb_backtrace+0x29
witness_checkorder(8081f498,1,8074a8ad,49,807528d7,...) at
witness_checkorder+0x5e5
_rw_rlock(8081f498,8074a8ad,49,0,c1603aa8,...) at _rw_rlock+0x2a
pfil_run_hooks(8081f480,c1603ac8,83935000,2,0,...) at pfil_run_hooks
+0x35
ip_output(8394ed00,0,c1603a8c,
0,0,0,80796f90,0,0,0,804b2971,80796f94,80796f9c,c8) at ip_output+0x86f
tcp_respond(0,83984830,83984844,8394ed00,46ca580,...) at tcp_respond
+0x395
tcp_dropwithreset(1,3,99e2,873e1dcb,1600,...) at tcp_dropwithreset+0x126
tcp_input(8394ed00,14,83935000,1,0,...) at tcp_input+0xcf9
ip_input(8394ed00,14e,800,83935000,800,...) at ip_input+0x64a
netisr_dispatch(2,8394ed00,10,3,0,...) at netisr_dispatch+0x55
ether_demux(83935000,8394ed00,3,0,3,...) at ether_demux+0x1c1
ether_input(83935000,8394ed00,8072d03c,6a9,83922014,...) at ether_input
+0x323
fxp_intr(83922000,0,8073e924,471,8384a764,...) at fxp_intr+0x237
ithread_loop(839219b0,c1603d38,8073e756,305,838fc804,...) at
ithread_loop+0x145
fork_exit(803ee250,839219b0,c1603d38) at fork_exit+0x94
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc1603d70, ebp = 0 ---
lock order reversal:
1st 0x86c5809c inp (tcpinp) @ /usr/src/sys/netinet/tcp_usrreq.c:470
2nd 0x8081f498 PFil hook read/write mutex (PFil hook read/write
mutex) @ /usr/src/sys/net/pfil.c:73
KDB: stack backtrace:
db_trace_self_wrapper
(807430c0,c39d7a30,80438ff5,80744463,8081f498,...) at
db_trace_self_wrapper+0x26
kdb_backtrace(80744463,8081f498,8074a8c5,8074a8c5,8074a8ad,...) at
kdb_backtrace+0x29
witness_checkorder(8081f498,1,8074a8ad,49,807528d7,...) at
witness_checkorder+0x5e5
_rw_rlock(8081f498,8074a8ad,49,0,c39d7aec,...) at _rw_rlock+0x2a
pfil_run_hooks(8081f480,c39d7b0c,839d3400,2,86c58000,...) at
pfil_run_hooks+0x35
ip_output(867ce300,0,c39d7ad0,0,0,...) at ip_output+0x86f
tcp_output(86c59000,0,8074feb4,1d6,86c5809c,...) at tcp_output+0x12dc
tcp_usr_connect(862f7ad4,86572bd0,83d5b000,25,c39d7c60,...) at
tcp_usr_connect+0xf4
soconnect(862f7ad4,86572bd0,83d5b000,10,16,...) at soconnect+0x52
kern_connect(83d5b000,5a,86572bd0,86572bd0,282cd72c,...) at
kern_connect+0x59
connect(83d5b000,c39d7cfc,c,80744e40,80784c50,...) at connect+0x46
syscall(c39d7d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (98, FreeBSD ELF32, connect), eip = 0x2838a11b, esp =
0x7f7fda4c, ebp = 0x7f7fda68 ---
dmesg:
Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.0-RELEASE #10: Tue Apr 8 12:54:38 EST 2008
root@dash.ish.com.au:/usr/obj/usr/src/sys/PAE
WARNING: WITNESS option enabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) XEON(TM) CPU 2.40GHz (2395.92-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0xf24 Stepping = 4
Features
=
0x3febfbff
<
FPU
,VME
,DE
,PSE
,TSC
,MSR
,PAE
,MCE
,CX8
,APIC
,SEP
,MTRR
,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM>
Logical CPUs per core: 2
real memory = 1073741824 (1024 MB)
avail memory = 1038790656 (990 MB)
MPTable: < Kings Canyon>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 6
cpu3 (AP): APIC ID: 7
WITNESS: spin lock intrcnt not in order list
ioapic0: Assuming intbase of 0
ioapic1: Assuming intbase of 24
ioapic2: Assuming intbase of 48
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
ioapic2 <Version 2.0> irqs 48-71 on motherboard
registered firmware set <isp_1040>
registered firmware set <isp_1040_it>
registered firmware set <isp_1080>
registered firmware set <isp_1080_it>
registered firmware set <isp_12160>
registered firmware set <isp_12160_it>
registered firmware set <isp_2100>
registered firmware set <isp_2200>
registered firmware set <isp_2300>
registered firmware set <isp_2322>
registered firmware set <isp_2400>
kbd1 at kbdmux0
cryptosoft0: <software crypto> on motherboard
cpu0 on motherboard
cpu1 on motherboard
cpu2 on motherboard
cpu3 on motherboard
pcib0: <MPTable Host-PCI bridge> pcibus 0 on motherboard
pci0: <PCI bus> on pcib0
pci0: <unknown> at device 0.1 (no driver attached)
pcib1: <PCI-PCI bridge> at device 2.0 on pci0
pci1: <PCI bus> on pcib1
pcib2: <MPTable PCI-PCI bridge> at device 29.0 on pci1
pci2: <PCI bus> on pcib2
pci2: <network, ethernet> at device 1.0 (no driver attached)
pcib3: <MPTable PCI-PCI bridge> at device 31.0 on pci1
pci3: <PCI bus> on pcib3
ahc0: <Adaptec aic7899 Ultra160 SCSI adapter> port 0x4000-0x40ff mem
0xfc300000-0xfc300fff irq 28 at device 2.0 on pci3
ahc0: [ITHREAD]
aic7899: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs
ahc1: <Adaptec aic7899 Ultra160 SCSI adapter> port 0x4400-0x44ff mem
0xfc301000-0xfc301fff irq 29 at device 2.1 on pci3
ahc1: [ITHREAD]
aic7899: Ultra160 Wide Channel B, SCSI Id=7, 32/253 SCBs
em0: <Intel(R) PRO/1000 Network Connection Version - 6.7.3> port
0x4800-0x481f mem 0xfc340000-0xfc35ffff,0xfc320000-0xfc33ffff irq 31
at device 4.0 on pci3
em0: Ethernet address: 00:30:48:24:aa:6f
em0: [FILTER]
pci0: <serial bus, USB> at device 29.0 (no driver attached)
pci0: <serial bus, USB> at device 29.1 (no driver attached)
pci0: <serial bus, USB> at device 29.2 (no driver attached)
pcib4: <MPTable PCI-PCI bridge> at device 30.0 on pci0
pci4: <PCI bus> on pcib4
vgapci0: <VGA-compatible display> port 0x5000-0x50ff mem
0xfd000000-0xfdffffff,0xfc400000-0xfc400fff irq 16 at device 1.0 on pci4
fxp0: <Intel 82550 Pro/100 Ethernet> port 0x5400-0x543f mem
0xfc401000-0xfc401fff,0xfc420000-0xfc43ffff irq 17 at device 2.0 on pci4
miibus0: <MII bus> on fxp0
inphy0: <i82555 10/100 media interface> PHY 1 on miibus0
inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp0: Ethernet address: 00:30:48:24:aa:70
fxp0: [ITHREAD]
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH3 UDMA100 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x2060-0x206f at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc8fff,
0xdc000-0xdffff,0xe0000-0xe3fff pnpid ORM0000 on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
fdc0: <Enhanced floppy controller> at port 0x3f0-0x3f5,0x3f7 irq 6 drq
2 on isa0
fdc0: [FILTER]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 16550A
sio0: [FILTER]
sio1 at port 0x2f8-0x2ff irq 3 on isa0
sio1: type 16550A
sio1: [FILTER]
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on
isa0
unknown: <PNP0c02> can't assign resources (memory)
unknown: <PNP0c01> can't assign resources (memory)
unknown: <PNP0303> can't assign resources (port)
unknown: <PNP0c02> can't assign resources (memory)
unknown: <PNP0c02> can't assign resources (memory)
unknown: <PNP0c02> can't assign resources (memory)
unknown: <PNP0501> can't assign resources (port)
unknown: <PNP0501> can't assign resources (port)
unknown: <PNP0700> can't assign resources (port)
WARNING: ZFS is considered to be an experimental feature in FreeBSD.
Timecounters tick every 1.000 msec
Fast IPsec: Initialized Security Association Processing.
ZFS filesystem version 6
ZFS storage pool version 6
acd0: CDROM <MATSHITA CR-177/7T0D> at ata1-master UDMA33
Waiting 5 seconds for SCSI devices to settle
ses0 at ahc0 bus 0 target 6 lun 0
ses0: <SDR GEM318 0> Fixed Processor SCSI-2 device
ses0: 3.300MB/s transfers
ses0: SAF-TE Compliant Device
da0 at ahc0 bus 0 target 0 lun 0
da0: <FUJITSU MAW3073NC 0104> Fixed Direct Access SCSI-3 device
da0: 160.000MB/s transfers (80.000MHz DT, offset 127, 16bit)
da0: Command Queueing Enabled
da0: 70136MB (143638992 512 byte sectors: 255H 63S/T 8941C)
da1 at ahc0 bus 0 target 1 lun 0
da1: <FUJITSU MAW3073NC 0104> Fixed Direct Access SCSI-3 device
da1: 160.000MB/s transfers (80.000MHz DT, offset 127, 16bit)
da1: Command Queueing Enabled
da1: 70136MB (143638992 512 byte sectors: 255H 63S/T 8941C)
da2 at ahc0 bus 0 target 2 lun 0
da2: <FUJITSU MAP3367NC 0108> Fixed Direct Access SCSI-3 device
da2: 160.000MB/s transfers (80.000MHz DT, offset 127, 16bit)
da2: Command Queueing Enabled
da2: 35046MB (71775284 512 byte sectors: 255H 63S/T 4467C)
SMP: AP CPU #2 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
WARNING: WITNESS option enabled, expect reduced performance.
Trying to mount root from zfs:tank/root
WARNING: /mnt/otherhdd was not properly dismounted
-------------------------->
ish
http://www.ish.com.au
Level 1, 30 Wilson Street Newtown 2042 Australia
phone +61 2 9550 5001 fax +61 2 9550 4001
GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A