thr3ads.net - freebsd stable - Hard Hang, nothing in logs / no panics [May 2007]

If this information is useful, please help other people find it:
Share via:

Roger Miranda

2007-May-11 20:00 UTC

Hard Hang, nothing in logs / no panics

Good Day everyone.

I have this one system setup with If_bridge to filter traffic. It does work 
quite good.  I am running FreeBSD 6.2 but as a TINYBSD Image.  The one 
problem I have is I place the machine at the perimeter on our network with 27 
seats.  At that time anywhere between 15min - 24hours the entire system goes 
into a Hard Lock (Physical reboot needed).  The thing is there is no logs or 
kernel panics or anything.  No IRQ Conflicts exists.  

I am looking for any inputs or any ways to go after looking how to even 
diagnosed this.

Here is a copy of my dmesg.

Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 6.2-RELEASE #0: Wed May  9 17:48:30 UTC 2007
    root@:/usr/obj/usr/src/sys/TINYBSD
WARNING: MPSAFE network stack disabled, expect reduced performance.
ACPI APIC Table: <IntelR AWRDACPI>
Timecounter "i8254" frequency 1193520 Hz quality 0
CPU: Intel(R) Celeron(R) CPU 2.80GHz (2792.85-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0xf34  Stepping = 4
  
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x441d<SSE3,RSVD2,MON,DS_CPL,CNTX-ID,<b14>>
real memory  = 535691264 (510 MB)
avail memory = 519000064 (494 MB)
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
acpi0: <IntelR AWRDACPI> on motherboard
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 3.0 on pci0
pci1: <ACPI PCI bus> on pcib1
em0: <Intel(R) PRO/1000 Network Connection Version - 6.3.9> port
0xc000-0xc01f
mem 0xf2000000-0xf201ffff irq 18 at device 1.0 on pci1
em0: Ethernet address: 00:30:48:86:97:62
em0: [GIANT-LOCKED]
pcib2: <ACPI PCI-PCI bridge> at device 28.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pci0: <serial bus, USB> at device 29.0 (no driver attached)
pci0: <serial bus, USB> at device 29.1 (no driver attached)
pci0: <base peripheral> at device 29.4 (no driver attached)
pci0: <base peripheral, interrupt controller> at device 29.5 (no driver 
attached)
pci0: <serial bus, USB> at device 29.7 (no driver attached)
pcib3: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci3: <ACPI PCI bus> on pcib3
pci3: <display, VGA> at device 9.0 (no driver attached)
em1: <Intel(R) PRO/1000 Network Connection Version - 6.3.9> port
0xd100-0xd13f
mem 0xf1000000-0xf101ffff irq 19 at device 10.0 on pci3
em1: Ethernet address: 00:30:48:86:97:63
em1: [GIANT-LOCKED]
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel 6300ESB SATA150 controller> port 
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xf000-0xf00f irq 18 at device 31.2 on 
pci0
ata0: <ATA channel 0> on atapci0
ata1: <ATA channel 1> on atapci0
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_tz0: <Thermal Zone> on acpi0
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on
acpi0
sio0: type 16550A
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
pmtimer0 on isa0
orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounter "TSC" frequency 2792849472 Hz quality 800
Timecounters tick every 1.000 msec
IP Filter: v4.1.13 initialized.  Default = pass all, Logging = enabled
ipfw2 (+ipv6) initialized, divert loadable, rule-based forwarding disabled, 
default to accept, logging limited to 100 packets/entry by default
ad0: 977MB <SanDisk SDCFH-1024 HDX 3.19> at ata0-master PIO4
ad2: 76319MB <Seagate ST380811AS 3.AAB> at ata1-master SATA150
Trying to mount root from ufs:/dev/ad0s1a
em1: link state changed to DOWN
em0: link state changed to DOWN
bridge0: Ethernet address: 46:e0:af:c9:e6:b7
em0: promiscuous mode enabled
em1: promiscuous mode enabled
em0: link state changed to UP
em1: link state changed to UP
em1: link state changed to DOWN
em1: link state changed to UP
Waiting (max 60 seconds) for system process `vnlru' to stop...done
Waiting (max 60 seconds) for system process `bufdaemon' to stop...done
Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining...1 0 0 done
All buffers synced.
Uptime: 47m5s
Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 6.2-RELEASE #0: Wed May  9 17:48:30 UTC 2007
    root@:/usr/obj/usr/src/sys/TINYBSD
WARNING: MPSAFE network stack disabled, expect reduced performance.
ACPI APIC Table: <IntelR AWRDACPI>
Timecounter "i8254" frequency 1193516 Hz quality 0
CPU: Intel(R) Celeron(R) CPU 2.80GHz (2792.84-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0xf34  Stepping = 4
  
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x441d<SSE3,RSVD2,MON,DS_CPL,CNTX-ID,<b14>>
real memory  = 535691264 (510 MB)
avail memory = 519000064 (494 MB)
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
acpi0: <IntelR AWRDACPI> on motherboard
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 3.0 on pci0
pci1: <ACPI PCI bus> on pcib1
em0: <Intel(R) PRO/1000 Network Connection Version - 6.3.9> port
0xc000-0xc01f
mem 0xf2000000-0xf201ffff irq 18 at device 1.0 on pci1
em0: Ethernet address: 00:30:48:86:97:62
em0: [GIANT-LOCKED]
pcib2: <ACPI PCI-PCI bridge> at device 28.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pci0: <serial bus, USB> at device 29.0 (no driver attached)
pci0: <serial bus, USB> at device 29.1 (no driver attached)
pci0: <base peripheral> at device 29.4 (no driver attached)
pci0: <base peripheral, interrupt controller> at device 29.5 (no driver 
attached)
pci0: <serial bus, USB> at device 29.7 (no driver attached)
pcib3: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci3: <ACPI PCI bus> on pcib3
pci3: <display, VGA> at device 9.0 (no driver attached)
em1: <Intel(R) PRO/1000 Network Connection Version - 6.3.9> port
0xd100-0xd13f
mem 0xf1000000-0xf101ffff irq 19 at device 10.0 on pci3
em1: Ethernet address: 00:30:48:86:97:63
em1: [GIANT-LOCKED]
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel 6300ESB SATA150 controller> port 
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xf000-0xf00f irq 18 at device 31.2 on 
pci0
ata0: <ATA channel 0> on atapci0
ata1: <ATA channel 1> on atapci0
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_tz0: <Thermal Zone> on acpi0
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on
acpi0
sio0: type 16550A
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
pmtimer0 on isa0
orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounter "TSC" frequency 2792842836 Hz quality 800
Timecounters tick every 1.000 msec
IP Filter: v4.1.13 initialized.  Default = pass all, Logging = enabled
ipfw2 (+ipv6) initialized, divert loadable, rule-based forwarding disabled, 
default to accept, logging limited to 100 packets/entry by default
ad0: 977MB <SanDisk SDCFH-1024 HDX 3.19> at ata0-master PIO4
ad2: 76319MB <Seagate ST380811AS 3.AAB> at ata1-master SATA150
Trying to mount root from ufs:/dev/ad0s1a

Kris Kennaway

2007-May-11 21:00 UTC

head link

Hard Hang, nothing in logs / no panics

On Fri, May 11, 2007 at 02:42:51PM -0500, Roger Miranda
wrote:> Good Day everyone.
> 
> I have this one system setup with If_bridge to filter traffic. It does work
> quite good.  I am running FreeBSD 6.2 but as a TINYBSD Image.  The one 
> problem I have is I place the machine at the perimeter on our network with
27
> seats.  At that time anywhere between 15min - 24hours the entire system
goes
> into a Hard Lock (Physical reboot needed).  The thing is there is no logs
or
> kernel panics or anything.  No IRQ Conflicts exists.  
> 
> I am looking for any inputs or any ways to go after looking how to even 
> diagnosed this.
See the developers handbook chapter on kernel debugging.

Kris (I really need to make a keyboard shortcut for typing this phrase)

Roger Miranda

2007-May-11 21:20 UTC

head link

Hard Hang, nothing in logs / no panics

On Friday 11 May 2007 16:00, Kris Kennaway wrote:> See the developers handbook chapter on kernel debugging.
Kris,  I have gone through the kernel debugging sections of the developers 
handbook.  The one problem is when I get a hard hang, I do not get any error 
or panics.  And there is no crash or dump data on reboot. Yes. I have enabled 
DDB and KDB and a dumpdir (in /etc/rc.conf)

Am I missing something in the kernel debugging section?  I see "11.9
Debugging
Deadlocks" talk about Deadlocks.  But at the time of the lock I have no way
of doing a ps or really anything as the system is locked up solid.

Roger

Kris Kennaway

2007-May-11 21:38 UTC

head link

Hard Hang, nothing in logs / no panics

On Fri, May 11, 2007 at 04:21:25PM -0500, Roger Miranda
wrote:> On Friday 11 May 2007 16:00, Kris Kennaway wrote:
> > See the developers handbook chapter on kernel debugging.
> 
> Kris,  I have gone through the kernel debugging sections of the developers 
> handbook.  The one problem is when I get a hard hang, I do not get any
error
> or panics.  And there is no crash or dump data on reboot. Yes. I have
enabled
> DDB and KDB and a dumpdir (in /etc/rc.conf)
> 
> Am I missing something in the kernel debugging section?  I see "11.9
Debugging
> Deadlocks" talk about Deadlocks.  But at the time of the lock I have
no way
> of doing a ps or really anything as the system is locked up solid.
You missed that the debugger is there to debug bugs (including
deadlocks).  Break to the debugger and obtain the necessary debugging
:)

Kris

freebsd stable - May 2007 - Hard Hang, nothing in logs / no panics

Hard Hang, nothing in logs / no panics

Hard Hang, nothing in logs / no panics

Hard Hang, nothing in logs / no panics

Hard Hang, nothing in logs / no panics