freebsd stable - Sep 2005 - Stress testing and TIMEOUT - WRITE_DMA

I'm not seeing much in the way of responses to this post from
freebsd-questions, so I thought I'd take it to freebsd-stable, where it
is probably more relevant. ;-)

Please see my original thread on freebsd-questions for context.

On Fri, 26 Aug 2005 03:21:35 -0600 Anthony Chavez <acc@anthonychavez.org>
wrote:
> My question is simply this: is the fact that I received 4 TIMEOUT
> warnings in the space of roughly 2 weeks significant cause for concern?
Apparently, the fact that the stress tool produced so few warnings may
have given me a false sense of security.  I'm being treated to the
following messages (81 in total) today, after 8 days uptime:

Sep  6 11:35:27 mybox kernel: ad0: TIMEOUT - WRITE_DMA retrying (2 retries left)
LBA=8348191
...
Sep  6 18:59:09 mybox kernel: ad0: TIMEOUT - WRITE_DMA retrying (2 retries left)
LBA=8348383
Sep  6 19:04:58 mybox kernel: ad0: TIMEOUT - READ_DMA retrying (2 retries left)
LBA=61749183

The READ_DMA timeouts are happening very infrequently, but it's worth
mentioning that I'm seeing them now in addition.

This is quite disturbing, particularly when the machine in question is
*in*production.*

Has anyone who has experienced this pain found solace in 5-STABLE's ATA
drivers?

dmesg below.

-- 
Anthony Chavez                                 http://anthonychavez.org/
mailto:acc@anthonychavez.org         jabber:acc@jabber.anthonychavez.org

Copyright (c) 1992-2005 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD 5.4-RELEASE-p6 #0: Fri Aug 26 02:23:19 MDT 2005
    root@mybox:/usr/obj/usr/src/sys/GENERIC
ACPI APIC Table: <DELL   2400   >
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Celeron(R) CPU 2.40GHz (2392.25-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0xf29  Stepping = 9
 
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
real memory  = 266813440 (254 MB)
avail memory = 251445248 (239 MB)
ioapic0: Changing APIC ID to 1
ioapic0 <Version 2.0> irqs 0-23 on motherboard
npx0: <math processor> on motherboard
npx0: INT 16 interface
acpi0: <DELL 2400   > on motherboard
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
agp0: <Intel 82845G (845G GMCH) SVGA controller> mem
0xfeb80000-0xfebfffff,0xe8000000-0xefffffff irq 16 at device 2.0 on pci0
agp0: detected 892k stolen memory
agp0: aperture size is 128M
uhci0: <Intel 82801DB (ICH4) USB controller USB-A> port 0xff80-0xff9f irq
16 at device 29.0 on pci0
usb0: <Intel 82801DB (ICH4) USB controller USB-A> on uhci0
usb0: USB revision 1.0
uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1: <Intel 82801DB (ICH4) USB controller USB-B> port 0xff60-0xff7f irq
19 at device 29.1 on pci0
usb1: <Intel 82801DB (ICH4) USB controller USB-B> on uhci1
usb1: USB revision 1.0
uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhci2: <Intel 82801DB (ICH4) USB controller USB-C> port 0xff40-0xff5f irq
18 at device 29.2 on pci0
usb2: <Intel 82801DB (ICH4) USB controller USB-C> on uhci2
usb2: USB revision 1.0
uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
pci0: <serial bus, USB> at device 29.7 (no driver attached)
pcib1: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pci1: <simple comms, generic modem> at device 5.0 (no driver attached)
xl0: <3Com 3c900-TPO Etherlink XL> port 0xddc0-0xddff irq 18 at device 6.0
on pci1
xl0: selecting 10baseT transceiver, half duplex
xl0: Ethernet address: 00:60:97:74:a8:6d
bfe0: <Broadcom BCM4401 Fast Ethernet> mem 0xfe9fe000-0xfe9fffff irq 17 at
device 9.0 on pci1
miibus0: <MII bus> on bfe0
bmtphy0: <BCM4401 10/100baseTX PHY> on miibus0
bmtphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
bfe0: Ethernet address: 00:12:3f:d4:21:75
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH4 UDMA100 controller> port
0xffa0-0xffaf,0x376,0x170-0x177,0x3f6,0x1f0-0x1f7 irq 18 at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata1: <ATA channel 1> on atapci0
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
pci0: <multimedia, audio> at device 31.5 (no driver attached)
fdc0: <floppy drive controller> port 0x3f7,0x3f0-0x3f5 irq 6 drq 2 on
acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x64,0x60 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on
acpi0
sio0: type 16550A
ppc0: <ECP parallel printer port> port 0x778-0x77f,0x378-0x37f irq 7 on
acpi0
ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/8 bytes threshold
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
orm0: <ISA Option ROMs> at iomem
0xcd000-0xcffff,0xcb800-0xccfff,0xc0000-0xcb7ff on isa0
pmtimer0 on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounter "TSC" frequency 2392248384 Hz quality 800
Timecounters tick every 10.000 msec
ad0: 76293MB <SAMSUNG SP0802N TK100-28> at ata0-master UDMA100
acd0: CDROM <GCR-8483B/1.07> at ata1-master UDMA33
ATA PseudoRAID loaded
Mounting root from ufs:/dev/ad0s1a
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 477 bytes
Desc: not available
Url :
http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20050911/1f565f69/attachment.bin

Hi Anthony,

Sunday, September 11, 2005, 10:18:36 PM, you has on mind:
> I'm not seeing much in the way of responses to this post from
> freebsd-questions, so I thought I'd take it to freebsd-stable, where it
> is probably more relevant. ;-)
> Please see my original thread on freebsd-questions for context.
> On Fri, 26 Aug 2005 03:21:35 -0600 Anthony Chavez
<acc@anthonychavez.org> wrote:
>> My question is simply this: is the fact that I received 4 TIMEOUT
>> warnings in the space of roughly 2 weeks significant cause for concern?
> Apparently, the fact that the stress tool produced so few warnings may
> have given me a false sense of security.  I'm being treated to the
> following messages (81 in total) today, after 8 days uptime:
> Sep  6 11:35:27 mybox kernel: ad0: TIMEOUT - WRITE_DMA retrying (2 retries
left) LBA=8348191
> ...
> Sep  6 18:59:09 mybox kernel: ad0: TIMEOUT - WRITE_DMA retrying (2 retries
left) LBA=8348383
> Sep  6 19:04:58 mybox kernel: ad0: TIMEOUT - READ_DMA retrying (2 retries
left) LBA=61749183
> The READ_DMA timeouts are happening very infrequently, but it's worth
> mentioning that I'm seeing them now in addition.
> This is quite disturbing, particularly when the machine in question is
> *in*production.*
I thing you should really quickly look for backuping your data. When
I was seeing this kind of messages last time, my disk died after 3
days from time they started showing up in my log files. I wasn't able
to write any data to the disk (system just sudennly paniced, when
I tried to mount it rw, but I was able to mount it ro and copy most of
the data) Note, that I wasn't able to copy about 10GB out of 30GB. So
don't ignore them and have a good luck.
> Has anyone who has experienced this pain found solace in 5-STABLE's ATA
> drivers?
> dmesg below.
-- 
Best regards

  DanGer, ICQ: 261701668  | e-mail protecting at: http://www.2pu.net/
  http://danger.rulez.sk  | proxy list at:        http://www.proxy-web.com/
                          | FreeBSD - The Power to Serve!

[ I am not Mr. Tator! I am not the entertainment! - Dictator to Yakko ]

Anthony Chavez wrote:
> Sep  6 11:35:27 mybox kernel: ad0: TIMEOUT - WRITE_DMA retrying (2 retries
left) LBA=8348191
[...]
> Has anyone who has experienced this pain found solace in 5-STABLE's ATA
> drivers?
Is this with the ATA mkIII patches?
I assume you're acquainted with the ATA DMA timeout discussions of the 
last couple months concerning 5.x.

mkb.

On Fri, 26 Aug 2005 03:21:35 -0600 Anthony Chavez <acc@anthonychavez.org> 
wrote:
> My question is simply this: is the fact that I received 4 TIMEOUT
> warnings in the space of roughly 2 weeks significant cause for concern?
Hi,
You may have a look at this pr :85603  (FS corruption and
'uncorrectable' DMA
errors on ATA disks after unclean shutdown) and see if that applies for you.

Are you running a kernel built around mid June this year?
Did your machine paniced before the DMA problems appears (I think a power 
faillure can do the trick too)?

We were severall usenet user experiencing this kind of problems 
(news://comp.unix.bsd.freebsd.misc thread was named "Disaster Recovery? and
started 30 Aug 05). If you have the same problem as us, the fix is easy:
- backup your data with tar (will take a while due to timeouts)
- fdisk + newfs 
- reinstall your backup
- cvsup + upgrade your kernel
and thats all... And I was surprised to see my PostgreSQL database coming 
online without a single error message Pg really hate when theFS is 
inconsistent...

In our case this problem was fixed by newfs, even smartctl 
(sysutils/smartmontool) did report errors at the drive level. After
newfs'ing
the disk no more message (but they still in the drive's log). 

Hope this is relevant to your problem...
--
MaXX

I tested my drive as follow:
On comp.unix.bsd.freebsd.misc MaXX wrote:
> I will stress test the drive to see if it still reliable for some purpose.
I've finished some tests on the drive:

1. filled the drive with huge files (11,25,30,10Gb) 3 simultaneous writes =>
no DMA_READ or DMA_WRITE errors; fsck OK

2. copied 18 times /usr/ports with some distfiles and work folders (2
simultaneous copies , 9
times about 4 596 000 files) => no DMA_READ or DMA_WRITE errors; fsck NOT
OK: a bunch of errors which seem to be only at the file system level.

3. md5 sum of 4 596 000 files before corrective fsck: no errors, burning hot
drive

4. clean reboot + fsck: ok; fsck skipped checks.

5. compare md5 before and after reboot: OK, no missing files/folders, newsum
== oldsum.

I the tried to reproduce the initial problem, no way to do it... I killed
init, pulled the plug while writing or reading. No way to get those DMA_*
errors back (Note: the kernel was not the same as the failled one)...

I give up...

Conclusion: the disk is reliable enough to go back to work with a good
backup policy (maybe in a vinum mirror to be sure). The problem seem to be
bound to the kernel the machine was running since mid June 05.

MaXX wrote:
> started 30 Aug 05). If you have the same problem as us, the fix is easy:
[...]

Does the problem still exist then with newer versions? Or was it 
specific to the snapshot "around June"?

mkb.