thr3ads.net - freebsd stable - Page fault, GEOM problem?? [Nov 2005]

If this information is useful, please help other people find it:
Share via:

Johan Ström

2005-Nov-17 09:20 UTC

Page fault, GEOM problem??

Ok, just got this not so very nice error on a RELENG_6_0 box (built  
from sources this morning, GENERIC kernel minus drivers I dont use):

Nov 17 15:35:43 elfi kernel: subdisk10: detached
Nov 17 15:35:43 elfi kernel: ad10: detached
Nov 17 15:35:43 elfi kernel: unknown: TIMEOUT - READ_DMA retrying (1  
retry left) LBA=85720528
Nov 17 15:35:43 elfi kernel: GEOM_MIRROR: Device gm0s1: provider  
ad10s1 disconnected.
Nov 17 15:35:43 elfi kernel: GEOM_MIRROR: Request failed (error=6).  
ad10s1[WRITE(offset=134356992, length=16384)]
Nov 17 15:35:43 elfi kernel: GEOM_MIRROR: Request failed (error=6).  
ad10s1[WRITE(offset=134373376, length=16384)]
Nov 17 15:35:43 elfi kernel: GEOM_MIRROR: Request failed (error=6).  
ad10s1[WRITE(offset=134438912, length=16384)]
Nov 17 15:35:43 elfi kernel: GEOM_MIRROR: Request failed (error=6).  
ad10s1[WRITE(offset=268591104, length=16384)]
Nov 17 15:35:43 elfi kernel: GEOM_MIRROR: Request failed (error=6).  
ad10s1[WRITE(offset=268607488, length=16384)]
Nov 17 15:35:43 elfi kernel: GEOM_MIRROR: Request failed (error=6).  
ad10s1[WRITE(offset=268623872, length=16384)]
Nov 17 15:35:43 elfi kernel: GEOM_MIRROR: Request failed (error=6).  
ad10s1[WRITE(offset=268640256, length=16384)]
Nov 17 15:35:43 elfi kernel: GEOM_MIRROR: Request failed (error=6).  
ad10s1[WRITE(offset=20151026176, length=2048)]
Nov 17 15:35:43 elfi kernel: GEOM_MIRROR: Request failed (error=6).  
ad10s1[WRITE(offset=32299655680, length=8192)]
Nov 17 15:35:43 elfi kernel: GEOM_MIRROR: Request failed (error=6).  
ad10s1[READ(offset=37363671552, length=16384)]
Nov 17 15:35:43 elfi kernel: GEOM_MIRROR: Request failed (error=6).  
ad10s1[READ(offset=38349087232, length=16384)]
Nov 17 15:35:43 elfi kernel: GEOM_MIRROR: Request failed (error=6).  
ad10s1[READ(offset=45453566464, length=16384)]
Nov 17 15:35:43 elfi kernel: GEOM_MIRROR: Request failed (error=6).  
ad10s1[READ(offset=54459458048, length=131072)]
Nov 17 17:59:18 elfi syslogd: kernel boot file is /boot/kernel/kernel
Nov 17 17:59:18 elfi kernel:
Nov 17 17:59:18 elfi kernel:
Nov 17 17:59:18 elfi kernel: Fatal trap 12: page fault while in  
kernel mode
Nov 17 17:59:18 elfi kernel: fault virtual address      = 0x48
Nov 17 17:59:18 elfi kernel: fault code         = supervisor read,  
page not present
Nov 17 17:59:18 elfi kernel: instruction pointer        =  
0x20:0xc0506b92
Nov 17 17:59:18 elfi kernel: stack pointer              =  
0x28:0xd56d7c9c
Nov 17 17:59:18 elfi kernel: frame pointer              =  
0x28:0xd56d7c9c
Nov 17 17:59:18 elfi kernel: code segment               = base 0x0,  
limit 0xfffff, type 0x1b
Nov 17 17:59:18 elfi kernel: = DPL 0, pres 1, def32 1, gran 1
Nov 17 17:59:18 elfi kernel: processor eflags   = interrupt enabled,  
resume, IOPL = 0
Nov 17 17:59:18 elfi kernel: current process            = 36 (swi4:  
clock sio)
Nov 17 17:59:18 elfi kernel: trap number                = 12
Nov 17 17:59:18 elfi kernel: panic: page fault
Nov 17 17:59:18 elfi kernel: Uptime: 8h55m1s

ad10 and ad6, 2 brand new Maxtor Maxline 300GB SATA, attached to a  
Promise PDC40518 SATA150 controller, makes a GEOM mirror gm0s1.
I've been running this stuff in another "test" machine (MSI K8N
neo
Platinum, KT333 chip I believe), and I havent had a single problem. I  
moved the disks/controllercard to my "real" server 24 hours ago, with
the only apparent "problem" I seemd to have was this:

Nov 17 07:06:12 elfi kernel: xl0: transmission error: 90
Nov 17 07:06:12 elfi kernel: xl0: tx underrun, increasing tx start  
threshold to 120 bytes
Nov 17 07:06:18 elfi kernel: xl0: watchdog timeout
Nov 17 07:06:18 elfi kernel: xl0: link state changed to DOWN
Nov 17 07:06:18 elfi kernel: vlan5: link state changed to DOWN
Nov 17 07:06:20 elfi kernel: xl0: link state changed to UP
Nov 17 07:06:20 elfi kernel: vlan5: link state changed to UP

Comming and going... these problems just apperade during first 20-30  
minutes after boot, then they dissapeared totally (and yes there was  
plenty of IO on the net going on both during and after these  
messages). Sometimes i just got the first two messages and nothing  
"happened", but sometimes the watchdog message came and the network  
died for a minute or so.

Here is dmesg from last boot (directly after crash):

Copyright (c) 1992-2005 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
         The Regents of the University of California. All rights  
reserved.
FreeBSD 6.0-RELEASE #0: Thu Nov 17 00:49:29 CET 2005
     johan@elfi.stromnet.org:/usr/obj/usr/src/sys/ELFI
ACPI APIC Table: <ASUS   A7V333  >
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Athlon(TM) XP 1900+ (1599.56-MHz 686-class CPU)
   Origin = "AuthenticAMD"  Id = 0x662  Stepping = 2
    
Features=0x383fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE, 
MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE>
   AMD Features=0xc0480800<SYSCALL,MP,MMX+,3DNow+,3DNow>
real memory  = 536854528 (511 MB)
avail memory = 516014080 (492 MB)
ioapic0: Changing APIC ID to 2
ioapic0 <Version 0.2> irqs 0-23 on motherboard
npx0: [FAST]
npx0: <math processor> on motherboard
npx0: INT 16 interface
acpi0: <ASUS A7V333> on motherboard
acpi0: Power Button (fixed)
pci_link0: <ACPI PCI Link LNKA> irq 11 on acpi0
pci_link1: <ACPI PCI Link LNKB> irq 10 on acpi0
pci_link2: <ACPI PCI Link LNKC> irq 0 on acpi0
pci_link3: <ACPI PCI Link LNKD> irq 12 on acpi0
pci_link4: <ACPI PCI Link LNKE> irq 5 on acpi0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <32-bit timer at 3.579545MHz> port 0xe408-0xe40b on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
agp0: <VIA 8367 (KT266/KY266x/KT333) host to PCI bridge> mem  
0xe0000000-0xe3ffffff at device 0.0 on pci0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pci0: <multimedia, audio> at device 5.0 (no driver attached)
fwohci0: <Texas Instruments TSB43AB21/A/AI/A-EP> mem  
0xdf000000-0xdf0007ff,0xde800000-0xde803fff irq 17 at device 7.0 on pci0
fwohci0: OHCI version 1.10 (ROM=1)
fwohci0: No. of Isochronous channels is 4.
fwohci0: EUI64 00:e0:18:00:00:02:7e:fe
fwohci0: Phy 1394a available S400, 1 ports.
fwohci0: Link S400, max_rec 2048 bytes.
firewire0: <IEEE1394(FireWire) bus> on fwohci0
sbp0: <SBP-2/SCSI over FireWire> on firewire0
fwe0: <Ethernet over FireWire> on firewire0
if_fwe0: Fake Ethernet address: 02:e0:18:02:7e:fe
fwe0: Ethernet address: 02:e0:18:02:7e:fe
fwe0: if_start running deferred for Giant
fwohci0: Initiate bus reset
fwohci0: node_id=0xc800ffc0, gen=1, CYCLEMASTER mode
firewire0: 1 nodes, maxhop <= 0, cable IRM = 0 (me)
firewire0: bus manager 0 (me)
uhci0: <VIA 83C572 USB controller> port 0xd400-0xd41f irq 19 at  
device 9.0 on pci0
uhci0: [GIANT-LOCKED]
usb0: <VIA 83C572 USB controller> on uhci0
usb0: USB revision 1.0
uhub0: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1: <VIA 83C572 USB controller> port 0xd000-0xd01f irq 16 at  
device 9.1 on pci0
uhci1: [GIANT-LOCKED]
usb1: <VIA 83C572 USB controller> on uhci1
usb1: USB revision 1.0
uhub1: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
ehci0: <VIA VT6202 USB 2.0 controller> mem 0xde000000-0xde0000ff irq  
17 at device 9.2 on pci0
ehci0: [GIANT-LOCKED]
usb2: EHCI version 0.95
usb2: companion controllers, 2 ports each: usb0 usb1
usb2: <VIA VT6202 USB 2.0 controller> on ehci0
usb2: USB revision 2.0
uhub2: VIA EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub2: 4 ports with 4 removable, self powered
pci0: <display, VGA> at device 12.0 (no driver attached)
atapci0: <Promise PDC40518 SATA150 controller> port 0xb400-0xb47f, 
0xb000-0xb0ff mem 0xdc000000-0xdc000fff,0xdb800000-0xdb81ffff irq 17  
at device 14.0 on pci0
ata2: <ATA channel 0> on atapci0
ata3: <ATA channel 1> on atapci0
ata4: <ATA channel 2> on atapci0
ata5: <ATA channel 3> on atapci0
xl0: <3Com 3c905C-TX Fast Etherlink XL> port 0xa800-0xa87f mem  
0xdb000000-0xdb00007f irq 19 at device 16.0 on pci0
miibus0: <MII bus> on xl0
xlphy0: <3c905C 10/100 internal PHY> on miibus0
xlphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
xl0: Ethernet address: 00:04:76:ef:c6:36
isab0: <PCI-ISA bridge> at device 17.0 on pci0
isa0: <ISA bus> on isab0
atapci1: <VIA 8233A UDMA133 controller> port  
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xa400-0xa40f at device 17.1 on pci0
ata0: <ATA channel 0> on atapci1
ata1: <ATA channel 1> on atapci1
uhci2: <VIA 83C572 USB controller> port 0xa000-0xa01f at device 17.2  
on pci0
uhci2: [GIANT-LOCKED]
usb3: <VIA 83C572 USB controller> on uhci2
usb3: USB revision 1.0
uhub3: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub3: 2 ports with 2 removable, self powered
uhci3: <VIA 83C572 USB controller> port 0x9800-0x981f irq 21 at  
device 17.3 on pci0
uhci3: [GIANT-LOCKED]
usb4: <VIA 83C572 USB controller> on uhci3
usb4: USB revision 1.0
uhub4: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub4: 2 ports with 2 removable, self powered
ppc0: <ECP parallel printer port> port 0x378-0x37f,0x778-0x77b irq 7  
drq 3 on acpi0
ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/9 bytes threshold
ppbus0: <Parallel port bus> on ppc0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10  
on acpi0
sio0: type 16550A
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xccfff, 
0xd0000-0xd07ff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on  
isa0
Timecounter "TSC" frequency 1599556047 Hz quality 800
Timecounters tick every 1.000 msec
acd0: CDROM <CD-ROM CDU701-F/1.0q> at ata1-master PIO4
ad6: 286188MB <Maxtor 7L300S0 BANC1G10> at ata3-master SATA150
ad10: 286188MB <Maxtor 7L300S0 BANC1G10> at ata5-master SATA150
GEOM_MIRROR: Device gm0s1 created (id=4118114647).
GEOM_MIRROR: Device gm0s1: provider ad6s1 detected.
GEOM_MIRROR: Device gm0s1: provider ad10s1 detected.
GEOM_MIRROR: Device gm0s1: provider ad6s1 activated.
GEOM_MIRROR: Device gm0s1: provider mirror/gm0s1 launched.
GEOM_MIRROR: Device gm0s1: rebuilding provider ad10s1.
Trying to mount root from ufs:/dev/mirror/gm0s1a
WARNING: / was not properly dismounted
WARNING: /tmp was not properly dismounted
WARNING: /usr was not properly dismounted
/usr: mount pending error: blocks 8076 files 28
WARNING: /var was not properly dismounted
/var: mount pending error: blocks 4508 files 2
xl0: transmission error: 90
xl0: tx underrun, increasing tx start threshold to 120 bytes
xl0: transmission error: 90
xl0: tx underrun, increasing tx start threshold to 180 bytes


The network card is the exact same model as the one I used in the  
"test" machine, didn't have any problems there..

So, any ideas what this can be? If there were a disk crash, wish I  
have a hard time believing since I ran powermax (maxtor test program)  
on both of these disk 3 weeks ago and they have been running fine w/o  
a single problem since I started using them, why didn't just GEOM  
kick in and run on the other disk? Pagefaulting is not a way to react  
if a disk goes dead..

Hope someone can help me/this problem doesn't occur any more... but I  
suppose that is to much to hope for...

Thanks
Johan

Xin LI

2005-Nov-18 01:18 UTC

head link

Page fault, GEOM problem??

On 11/18/05, Johan Str?m <johan@stromnet.org>
wrote:> Ok, just got this not so very nice error on a RELENG_6_0 box (built
> from sources this morning, GENERIC kernel minus drivers I dont use):
> The network card is the exact same model as the one I used in the
> "test" machine, didn't have any problems there..
[...]> So, any ideas what this can be? If there were a disk crash, wish I
> have a hard time believing since I ran powermax (maxtor test program)
> on both of these disk 3 weeks ago and they have been running fine w/o
> a single problem since I started using them, why didn't just GEOM
> kick in and run on the other disk? Pagefaulting is not a way to react
> if a disk goes dead..
>
> Hope someone can help me/this problem doesn't occur any more... but I
> suppose that is to much to hope for...
Would you please consider trying to obtain a crashdump and send the
backtrace so we can investigate more?

(Hints can be found at
http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug.html#KERNELDEBUG-OBTAIN)

Thanks,
--
Xin LI <delphij@delphij.net> http://www.delphij.net

Johan Ström

2006-Jan-22 21:43 UTC

head link

Page fault, GEOM problem??

On 23 jan 2006, at 01.17, Michael S. Eubanks wrote:

> On Sun, 2006-01-22 at 23:51 +0100, Johan Str?m wrote:
>
> ...snip...
>
>
>> On 22 jan 2006, at 22.58, Michael S. Eubanks wrote:
>> This card does afaik dont have raid functionalitys (I've never read
>> anything about it either on the web, the cards box or anywhere  
>> else..).
>> I'm running GENERIC, which does include ataraid..
>> What does your dmesg identify your card as?
>>
>> atapci0: <Promise PDC40518 SATA150 controller> port
0xb800-0xb87f,
>> 0xb400-0xb4ff mem 0xfb800000-0xfb800fff,0xfb000000-0xfb01ffff irq 19
>> at device 12.0 on pci0
>>
>> Is it the same PDC chipset?
>>
>> --
>> Johan
>>
>>
>
> No, I have a different controller.  My mistake.  I think what is
> happening is the DMA read command is failing, therefore causing the
> device to be disconnected, and the kernel can't write to the disk from
> that point on (this is somewhat obvious given the output below).
>
>
>>> Nov 29 20:36:54 elfi kernel: subdisk10: detached
>>> Nov 29 20:36:54 elfi kernel: ad10: detached
>>> Nov 29 20:36:54 elfi kernel: unknown: TIMEOUT - READ_DMA48 retrying
>>> (1 retry left) LBA=426562704
>>> Nov 29 20:36:54 elfi kernel: GEOM_MIRROR: Device gm0s1: provider
>>> ad10s1 disconnected.
>>>
>
> The message seen from the last line above is generated in any of the
> following scenarios (from g_mirror.c):
>   1. Device wasn't running yet, but disk disappear.
>   2. Disk was active and disapppear.
>   3. Disk disappear during synchronization process.
>
>
>>> Nov 29 20:36:54 elfi kernel: GEOM_MIRROR: Request failed (error=6).
>>> ad10s1[WRITE(offset=134356992, length=16384)]
>>>
>
> As far as recovering the disk, I remember seeing something about  
> booting
> to single user mode and using fsck after a core dump in a previous  
> post.
> I'm assuming the disks worked initially and that you were able to  
> label
> them etc?  Is there any possibility that the disk state may be altered
> by a power saving feature or setting in the BIOS and FreeBSD just
> doesn't know when it happens until the next time it tries to access  
> the
> disk?
>
For recovering, i've always done a direct reboot, the gmirror  
rebuilds the mirror and fsck is run.
No problems reading labels etc, and never has been, only problem has  
been these sporadic crashes.. And the read/write performance (see  
earlier in thread)...

This is a server, so all bios setting for powersaving is (should be)  
shut of. Bios should thus never make the disk go to sleep.

>
>
> -Michael
>
>
Thanks for trying to help!
--
Johan

Johan Ström

2006-Jan-23 12:11 UTC

head link

Page fault, GEOM problem??

On 23 jan 2006, at 20.16, Paul T. Root wrote:> My friends disks are SATA. The jumper was to force
> the drives to use the SATA 1.x 1.5 gig standard instead
> of the faster SATA 2.x standard. Older cards can have
> trouble recognizing newer disks.
>
> His were recognized, but very flaky. They've been solid
> since.
>
These disk should be SATA150 afaik (Maxtor MaXLine III 300Gb).
The promise card is named SATAII 150..
So shouldnt be any missmatching. Both card and disks supports NCQ..  
Dunno about freebsd on the other hand..Havent found a way to enable/ 
disable this

> Johan Str?m wrote:
>
>> On 23 jan 2006, at 15.29, Paul T. Root wrote:
>>
>>> I'm coming in very late here, and only have some
>>> hearsay. But, a friend of mine has built a new hobby
>>> machine, with twin 160G drives on a 3Ware 8006, working as
>>> a stripe. He had a bunch of problems with stability of the drives
>>> until I gave him a couple of tiny (half size) jumpers, that he
>>> put on the drive. Smooth sailing since them. If needed, I can find
>>> what the jumpers did. But looking through the controllers doco
>>> should give you a clue.
>>>
>> As far as I know, SATA drives doesnt have jumpers.. Mine doesnt  
>> seem to do atleast.. There are two unused pins but i doubt they  
>> are for jumpers..
>>
>
> -- 
> Paul Root
> "Few people know what to do when hula girls attack." - Sam, age 8
>
>

freebsd stable - Nov 2005 - Page fault, GEOM problem??

Page fault, GEOM problem??

Page fault, GEOM problem??

Page fault, GEOM problem??

Page fault, GEOM problem??