I'm having a problem with a surprisingly easily provoked panic
on an SMP machine. (If I try it with a non-SMP machine, I can
get it to hang but not panic.)
The easiest way I've found to do it is to login to target SMP
machine "drum" from two different windows on some other machine
"remote" using ssh.
In the first window:
remote % ssh drum
<Scary banner>
Password: <whatever>
Last login: Sat Aug 26 12:02:07 2006
Copyright (c) 1980, 1983, 1986, 1988, 1990, 1991, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD 6.1-STABLE (DRUM6) #0: Fri Aug 25 10:07:09 CDT 2006
$
In the second window, log in:
remote % ssh drum
<Scary banner>
Password: <whatever>
Last login: Sat Aug 26 12:02:07 2006
Copyright (c) 1980, 1983, 1986, 1988, 1990, 1991, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD 6.1-STABLE (DRUM6) #0: Fri Aug 25 10:07:09 CDT 2006
$ ls -l `tty`
crw------- 1 jpc tty 0, 142 Aug 26 18:58 /dev/ttyp1
$ exit
Now go back to the first window and write to the other
terminal's revoked tty:
$ echo hello > /dev/ttyp1
Go to the second window and log in again, or try to:
remote % ssh drum
<Scary banner>
Password: <whatever>
...and that's as far as I get. drum has panicked.
drum# kgdb kernel.debug /usr/crash/vmcore.13
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so:
Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "i386-marcel-freebsd".
Unread portion of the kernel message buffer:
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x0
fault code = supervisor read, page not present
instruction pointer = 0x20:0x0
stack pointer = 0x28:0xe921f974
frame pointer = 0x28:0xe921f988
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 1287 (sshd)
trap number = 12
panic: page fault
cpuid = 0
Uptime: 8m57s
Dumping 2047 MB (2 chunks)
chunk 0: 1MB (159 pages) ... ok
chunk 1: 2047MB (524032 pages) <blah blah>
#0 doadump () at pcpu.h:165
165 __asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb) x/80xw 0xe921f974
0xe921f974: 0xc0571543 0x00000000 0xc6b74800 0xc6b74800
0xe921f984: 0xc6bb6c00 0xe921f9a4 0xc05c01a9 0xc6b74888
0xe921f994: 0x00000000 0x00000000 0xc6b74810 0xc6b74800
0xe921f9a4: 0xe921f9bc 0xc05beda0 0xc6b74800 0xc6b74800
0xe921f9b4: 0xc73b0e80 0xc6b74800 0xe921f9d8 0xc05c32ad
0xe921f9c4: 0xc6b74800 0x00000001 0xc6ebc400 0xe921fbcc
0xe921f9d4: 0xc07c0520 0xe921f9f4 0xc056315b 0xc6ebc400
0xe921f9e4: 0x00000003 0x00002000 0xc6bb6c00 0x00000000
0xe921f9f4: 0xe921fa34 0xc0533c6b 0xc6ebc400 0x00000003
0xe921fa04: 0x00002000 0xc6bb6c00 0xc6ebc400 0xc71ff440
0xe921fa14: 0xc6bb6c00 0xc07de2c0 0xc71ff440 0x00000000
0xe921fa24: 0xc6bb6c00 0x00000000 0xe921fbcc 0x00000003
0xe921fa34: 0xe921fa40 0xc073e0b8 0xe921fa64 0xe921fb20
0xe921fa44: 0xc05f33a8 0xc07b2a60 0xe921fa64 0x00000000
0xe921fa54: 0x00000180 0xc6bb6c00 0xc71ff440 0xe921fa78
0xe921fa64: 0xc07ddcc0 0xc71ff440 0x00000003 0xc6f71680
0xe921fa74: 0xc6bb6c00 0x00000006 0xe921fa84 0xc146cd20
0xe921fa84: 0x00000400 0xc6b74800 0xe921fa94 0xc0582668
0xe921fa94: 0xe921faac 0xc05827ed 0xc07bfe80 0x00000400
0xe921faa4: 0xc6b74800 0x00000000 0xe921fad0 0xc05c0c73
(kgdb) quit
drum# addr2line -e kernel.debug 0xc0571543
../../../kern/kern_event.c:1534
drum# addr2line -e kernel.debug 0xc05c01a9
../../../kern/tty.c:2427
drum# addr2line -e kernel.debug 0xc05beda0
../../../kern/tty.c:1681
drum# addr2line -e kernel.debug 0xc05c32ad
../../../sys/linedisc.h:136
drum# addr2line -e kernel.debug 0xc056315b
../../../kern/kern_conf.c:242
drum# addr2line -e kernel.debug 0xc0533c6b
../../../fs/devfs/devfs_vnops.c:680
drum# addr2line -e kernel.debug 0xc05f33a8
./vnode_if.h:198
drum# addr2line -e kernel.debug 0xc0582668
../../../kern/kern_malloc.c:251
drum# addr2line -e kernel.debug 0xc05827ed
../../../kern/kern_malloc.c:404
drum# addr2line -e kernel.debug 0xc05c0c73
../../../kern/tty.c:2829
Copyright (c) 1992-2006 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD 6.1-STABLE #0: Fri Aug 25 10:07:09 CDT 2006
jpc@drum.msfc.nasa.gov:/usr/src/sys/i386/compile/DRUM6
MPTable: <AMI GCHE >
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2800.12-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0xf25 Stepping = 5
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x4400<CNTX-ID,<b14>>
Logical CPUs per core: 2
real memory = 2147483648 (2048 MB)
avail memory = 2091847680 (1994 MB)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 6
cpu3 (AP): APIC ID: 7
ioapic0: Assuming intbase of 0
ioapic1: Assuming intbase of 16
ioapic2: Assuming intbase of 32
ioapic3: Assuming intbase of 48
ioapic0 <Version 1.1> irqs 0-15 on motherboard
ioapic1 <Version 1.1> irqs 16-31 on motherboard
ioapic2 <Version 1.1> irqs 32-47 on motherboard
ioapic3 <Version 1.1> irqs 48-63 on motherboard
netsmb_dev: loaded
cpu0 on motherboard
cpu1 on motherboard
cpu2 on motherboard
cpu3 on motherboard
pcib0: <MPTable Host-PCI bridge> pcibus 0 on motherboard
pci0: <PCI bus> on pcib0
pcib1: <MPTable PCI-PCI bridge> at device 7.0 on pci0
pci1: <PCI bus> on pcib1
fwohci0: <VIA Fire II (VT6306)> port 0xa800-0xa87f mem
0xfc8ff000-0xfc8ff7ff irq 24 at device 8.0 on pci1
fwohci0: OHCI version 1.0 (ROM=1)
fwohci0: No. of Isochronous channels is 8.
fwohci0: EUI64 00:11:06:00:00:00:50:66
fwohci0: Phy 1394a available S400, 3 ports.
fwohci0: Link S400, max_rec 2048 bytes.
firewire0: <IEEE1394(FireWire) bus> on fwohci0
sbp0: <SBP-2/SCSI over FireWire> on firewire0
fwe0: <Ethernet over FireWire> on firewire0
if_fwe0: Fake Ethernet address: 02:11:06:00:50:66
fwe0: Ethernet address: 02:11:06:00:50:66
fwe0: if_start running deferred for Giant
fwip0: <IP over FireWire> on firewire0
fwip0: Firewire address: 00:11:06:00:00:00:50:66 @ 0xfffe00000000, S400, maxrec
2048
fwohci0: Initiate bus reset
fwohci0: node_id=0xc800ffc1, gen=1, CYCLEMASTER mode
firewire0: 2 nodes, maxhop <= 1, cable IRM = 1 (me)
firewire0: bus manager 1 (me)
ohci0: <NEC uPD 9210 USB controller> mem 0xfc8fd000-0xfc8fdfff irq 25 at
device 9.0 on pci1
ohci0: [GIANT-LOCKED]
usb0: OHCI version 1.0
usb0: <NEC uPD 9210 USB controller> on ohci0
usb0: USB revision 1.0
uhub0: NEC OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 3 ports with 3 removable, self powered
ohci1: <NEC uPD 9210 USB controller> mem 0xfc8fe000-0xfc8fefff irq 20 at
device 9.1 on pci1
ohci1: [GIANT-LOCKED]
usb1: OHCI version 1.0
usb1: <NEC uPD 9210 USB controller> on ohci1
usb1: USB revision 1.0
uhub1: NEC OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
ehci0: <NEC uPD 720100 USB 2.0 controller> mem 0xfc8ffc00-0xfc8ffcff irq
21 at device 9.2 on pci1
ehci0: [GIANT-LOCKED]
usb2: EHCI version 0.95
usb2: companion controllers, 3 ports each: usb0 usb1
usb2: <NEC uPD 720100 USB 2.0 controller> on ehci0
usb2: USB revision 2.0
uhub2: NEC EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub2: 5 ports with 5 removable, self powered
em0: <Intel(R) PRO/1000 Network Connection Version - 6.0.5> port
0xc800-0xc83f mem 0xfea60000-0xfea7ffff irq 28 at device 8.0 on pci0
em0: Ethernet address: 00:30:48:53:37:e4
em0: [FAST]
em1: <Intel(R) PRO/1000 Network Connection Version - 6.0.5> port
0xd000-0xd03f mem 0xfeba0000-0xfebbffff irq 26 at device 9.0 on pci0
em1: Ethernet address: 00:30:48:53:37:e5
em1: [FAST]
ahd0: <Adaptec AIC7902 Ultra320 SCSI adapter> port
0xd800-0xd8ff,0xd400-0xd4ff mem 0xfebfa000-0xfebfbfff irq 30 at device 10.0 on
pci0
ahd0: [GIANT-LOCKED]
aic7902: Ultra320 Wide Channel A, SCSI Id=7, PCI 33 or 66Mhz, 512 SCBs
ahd1: <Adaptec AIC7902 Ultra320 SCSI adapter> port
0xe400-0xe4ff,0xe000-0xe0ff mem 0xfebfc000-0xfebfdfff irq 31 at device 10.1 on
pci0
ahd1: [GIANT-LOCKED]
aic7902: Ultra320 Wide Channel B, SCSI Id=7, PCI 33 or 66Mhz, 512 SCBs
drm0: <Rage XL> port 0xe800-0xe8ff mem
0xfd000000-0xfdffffff,0xfebff000-0xfebfffff irq 29 at device 11.0 on pci0
info: [drm] Initialized mach64 1.0.0 20020904
atapci0: <ServerWorks CSB6 UDMA100 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 15.1 on pci0
ata0: <ATA channel 0> on atapci0
ata1: <ATA channel 1> on atapci0
ohci2: <OHCI (generic) USB controller> mem 0xfebfe000-0xfebfefff irq 17 at
device 15.2 on pci0
ohci2: [GIANT-LOCKED]
usb3: OHCI version 1.0, legacy support
usb3: <OHCI (generic) USB controller> on ohci2
usb3: USB revision 1.0
uhub3: (0x1166) OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub3: 4 ports with 4 removable, self powered
isab0: <PCI-ISA bridge> at device 15.3 on pci0
isa0: <ISA bus> on isab0
pcib255: <ServerWorks host to PCI bridge(unknown chipset)> pcibus 255 on
motherboard
pci255: <PCI bus> on pcib255
pmtimer0 on isa0
orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model Generic PS/2 mouse, device ID 0
fdc0: <Enhanced floppy controller> at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2
on isa0
fdc0: [FAST]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0
ppc0: Generic chipset (ECP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/8 bytes threshold
ppbus0: <Parallel port bus> on ppc0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 16550A
sio1 at port 0x2f8-0x2ff irq 3 on isa0
sio1: type 16550A
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
unknown: <PNP0303> can't assign resources (port)
speaker0: <PC speaker> at port 0x61 on isa0
unknown: <PNP0f13> can't assign resources (irq)
unknown: <PNP0501> can't assign resources (port)
unknown: <PNP0501> can't assign resources (port)
unknown: <PNP0401> can't assign resources (port)
unknown: <PNP0700> can't assign resources (port)
Timecounters tick every 1.000 msec
acd0: DVDR <PIONEER DVD-RW DVR-108/1.10> at ata0-master UDMA66
Waiting 5 seconds for SCSI devices to settle
firewire0: New S400 device ID:0010b920007ad726
sa0 at ahd1 bus 0 target 5 lun 0
sa0: <SONY SDT-11000 0200> Removable Sequential Access SCSI-2 device
sa0: 40.000MB/s transfers (20.000MHz, offset 15, 16bit)
da0 at ahd0 bus 0 target 0 lun 0
da0: <FUJITSU MAP3367NP 0108> Fixed Direct Access SCSI-3 device
da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit), Tagged Queueing
Enabled
da0: 35046MB (71775284 512 byte sectors: 255H 63S/T 4467C)
da1 at ahd0 bus 0 target 1 lun 0
da1: <FUJITSU MAP3367NP 0108> Fixed Direct Access SCSI-3 device
da1: 320.000MB/s transfers (160.000MHz, offset 127, 16bit), Tagged Queueing
Enabled
da1: 35046MB (71775284 512 byte sectors: 255H 63S/T 4467C)
da2 at sbp0 bus 0 target 0 lun 0
da2: <Maxtor OneTouch 0000> Fixed Direct Access SCSI-4 device
da2: 50.000MB/s transfers
da2: 286103MB (585938944 512 byte sectors: 255H 63S/T 36473C)
cd0 at ata0 bus 0 target 0 lun 0
cd0: <PIONEER DVD-RW DVR-108 1.10> Removable CD-ROM SCSI-0 device
cd0: 66.000MB/s transfers
cd0: Attempt to query device size failed: NOT READY, Medium not present
SMP: AP CPU #3 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
Trying to mount root from ufs:/dev/da0s1a
WARNING: / was not properly dismounted
--
J. Porter Clark j.porter.clark@nasa.gov
NASA/MSFC Flight & Ground Computers Branch (EI31)
Phone (256)544-3661 Fax (256)544-6193