Hi guys, after a scrub my raidz array status showed:
# zpool status
pool: pool
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using ''zpool clear'' or replace the device with
''zpool replace''.
see: http://www.sun.com/msg/ZFS-8000-9P
scan: scrub repaired 85.5K in 1h21m with 0 errors on Mon Dec 19 06:24:25 2011
config:
NAME STATE READ WRITE CKSUM
pool ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ad18 ONLINE 0 0 1
ad19 ONLINE 0 0 0
ad10 ONLINE 0 0 1
ad4 ONLINE 0 0 0
errors: No known data errors
I assume the checksum counts are current and irreconcilable. (Why does
the scan say ''repaired with 0 errors'' then?).
What should one do at this point?
I rebooted and ran another scrub, this time it came up with 0 errors
and 0 checksum counts, what does that mean?
I then backed up the array, kicked out ad18 and resilvered it from scratch:
# zpool status
pool: pool
state: DEGRADED
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://www.sun.com/msg/ZFS-8000-8A
scan: resilvered 218G in 1h25m with 14 errors on Wed Dec 21 14:48:47 2011
config:
NAME STATE READ WRITE CKSUM
pool DEGRADED 0 0 14
raidz1-0 DEGRADED 0 0 28
replacing-0 OFFLINE 0 0 0
ad18/old OFFLINE 0 0 0
ad18 ONLINE 0 0 0
ad19 ONLINE 0 0 0
ad10 ONLINE 0 0 0
ad4 ONLINE 0 0 0
errors: 11 data errors, use ''-v'' for a list
and ''zpool status -v'' gives me a list of affected files.
I assume I delete those files, then follow the same procedure on ad10?
# uname -a
FreeBSD file 8.2-STABLE FreeBSD 8.2-STABLE #0: Sat Nov 12 17:51:22 SAST 2011
root at file:/usr/obj/usr/src/sys/COWNEL amd64
ZFS filesystem version 5
ZFS storage pool version 28
ps. I did get 1 disk alert in the logs during this whole process, half an hour
before resilvering:
Dec 21 12:41:48 file kernel: ad10: WARNING - READ_DMA48 UDMA ICRC error
(retrying request) LBA=306763504
Dec 21 12:41:48 file kernel: ad10: FAILURE - READ_DMA48
status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=306763504
On Wed, 21 Dec 2011, Gareth de Vaux wrote:> > I assume the checksum counts are current and irreconcilable. (Why does > the scan say ''repaired with 0 errors'' then?).One of your disks failed to return a sector. Due to redundancy, the original data was recreated from the remaining disks. This is normal good behavior (other than the disk failing to read the sector).> What should one do at this point?Normally one would have done nothing other than ''zpool clear'' after thinking about the issue for a while. If there were many failures to read from that disk, or the failures continue to accumulate for the same disk, then that would have been cause to replace it. Doing a ''zpool scrub'' is very much recommended in order to flush out data which fails to read while redundancy is still available.> see: http://www.sun.com/msg/ZFS-8000-8A > scan: resilvered 218G in 1h25m with 14 errors on Wed Dec 21 14:48:47 2011 > config: > > NAME STATE READ WRITE CKSUM > pool DEGRADED 0 0 14 > raidz1-0 DEGRADED 0 0 28 > replacing-0 OFFLINE 0 0 0 > ad18/old OFFLINE 0 0 0 > ad18 ONLINE 0 0 0 > ad19 ONLINE 0 0 0 > ad10 ONLINE 0 0 0 > ad4 ONLINE 0 0 0 > > errors: 11 data errors, use ''-v'' for a list > > > and ''zpool status -v'' gives me a list of affected files.This is a bummer. If you had a spare slot (or spare installed disk) you could have installed a new disk and done a replace with the existing disk still live. Then you would be much less likely to encounter a data error since the original disk was still working. Raidz1 is not very robust when used with large disks and with one drive totally failed. Bob -- Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
On Dec 21, 2011, at 11:45 AM, Gareth de Vaux wrote:> Hi guys, after a scrub my raidz array status showed: > > # zpool status > pool: pool > state: ONLINE > status: One or more devices has experienced an unrecoverable error. An > attempt was made to correct the error. Applications are unaffected. > action: Determine if the device needs to be replaced, and clear the errors > using ''zpool clear'' or replace the device with ''zpool replace''. > see: http://www.sun.com/msg/ZFS-8000-9P > scan: scrub repaired 85.5K in 1h21m with 0 errors on Mon Dec 19 06:24:25 2011 > config: > > NAME STATE READ WRITE CKSUM > pool ONLINE 0 0 0 > raidz1-0 ONLINE 0 0 0 > ad18 ONLINE 0 0 1 > ad19 ONLINE 0 0 0 > ad10 ONLINE 0 0 1 > ad4 ONLINE 0 0 0 > > errors: No known data errors > > > I assume the checksum counts are current and irreconcilable. (Why does > the scan say ''repaired with 0 errors'' then?). > > What should one do at this point?Be happy. Dance a jig. Buy a lottery ticket. Notice: scrub repaired 85.5K in 1h21m with 0 errors on Mon Dec 19 06:24:25 2011 ZFS found corruption and fixed it.> > I rebooted and ran another scrub, this time it came up with 0 errors > and 0 checksum counts, what does that mean?ZFS found corruption and fixed it.> > I then backed up the array, kicked out ad18 and resilvered it from scratch:oops... tempting the fates? Transient errors do occur, frequently. Not all errors are persistent or fatal. Given the information presented here, IMHO, this system did not warrant further action.> > # zpool status > pool: pool > state: DEGRADED > status: One or more devices has experienced an error resulting in data > corruption. Applications may be affected. > action: Restore the file in question if possible. Otherwise restore the > entire pool from backup. > see: http://www.sun.com/msg/ZFS-8000-8A > scan: resilvered 218G in 1h25m with 14 errors on Wed Dec 21 14:48:47 2011 > config: > > NAME STATE READ WRITE CKSUM > pool DEGRADED 0 0 14 > raidz1-0 DEGRADED 0 0 28 > replacing-0 OFFLINE 0 0 0 > ad18/old OFFLINE 0 0 0 > ad18 ONLINE 0 0 0 > ad19 ONLINE 0 0 0 > ad10 ONLINE 0 0 0 > ad4 ONLINE 0 0 0 > > errors: 11 data errors, use ''-v'' for a list > > > and ''zpool status -v'' gives me a list of affected files. > > I assume I delete those files, then follow the same procedure on ad10? > > > # uname -a > FreeBSD file 8.2-STABLE FreeBSD 8.2-STABLE #0: Sat Nov 12 17:51:22 SAST 2011 root at file:/usr/obj/usr/src/sys/COWNEL amd64 > > ZFS filesystem version 5 > ZFS storage pool version 28 > > > ps. I did get 1 disk alert in the logs during this whole process, half an hour before resilvering: > > Dec 21 12:41:48 file kernel: ad10: WARNING - READ_DMA48 UDMA ICRC error (retrying request) LBA=306763504 > Dec 21 12:41:48 file kernel: ad10: FAILURE - READ_DMA48 status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=306763504This appears to be a [S]ATA error generated by the drive. If LBA 306763504 is a legal LBA, then this can be one of the factors contributing to the original checksum error. -- richard -- ZFS and performance consulting http://www.RichardElling.com
On Thu 2011-12-22 (10:09), Bob Friesenhahn wrote:> One of your disks failed to return a sector. Due to redundancy, the > original data was recreated from the remaining disks. This is normal > good behavior (other than the disk failing to read the sector).So those checksum counts were historical?> Normally one would have done nothing other than ''zpool clear'' after > thinking about the issue for a while. If there were many failures to > read from that disk, or the failures continue to accumulate for the > same disk, then that would have been cause to replace it. > > Doing a ''zpool scrub'' is very much recommended in order to flush out > data which fails to read while redundancy is still available.I did a scrub and what worries me is that it came back with 0 issues when clearly there were considering what happens when I kick 1 disk out. Similarly I''ve seen that ''zpool clear'' just sets you up for problems down the line. It just pretends there aren''t errors. I managed to get the array back by resilvering, deleting affected files, resilvering other disk, scrubbing etc. but I''m wondering whether there''s a way to diagnose and do this without completely removing and resilvering an entire disk at a time?> This is a bummer. If you had a spare slot (or spare installed disk) > you could have installed a new disk and done a replace with the > existing disk still live.Right, didn''t think of that. Would''ve made it safer, but question above still stands.
On Thu 2011-12-22 (09:13), Richard Elling wrote:> Be happy. Dance a jig. Buy a lottery ticket. > Notice: scrub repaired 85.5K in 1h21m with 0 errors on Mon Dec 19 06:24:25 2011 > ZFS found corruption and fixed it.lol, will do next time.> oops... tempting the fates? > Transient errors do occur, frequently. Not all errors are persistent or fatal. > Given the information presented here, IMHO, this system did not warrant further action.Right, so I guess what happened is the array fixed itself, then 1 of the drives picked up an error just before I decided to take a different drive offline - so I caused the corruption. Only noticed the kernel message afterwards :/
On Thu, 22 Dec 2011, Gareth de Vaux wrote:> On Thu 2011-12-22 (10:09), Bob Friesenhahn wrote: >> One of your disks failed to return a sector. Due to redundancy, the >> original data was recreated from the remaining disks. This is normal >> good behavior (other than the disk failing to read the sector). > > So those checksum counts were historical?Yes. When a problem is detected and there is enough redundancy to resolve the problem, then the bad data block is not used any more and the corrected data is relocated somewhere else on the drive (I not sure if zfs does this, or if it requests that drive firmware do this). The count reflects that problems were found but does not reflect if a correction was made. Other text describes any continuing issues which were found.> I did a scrub and what worries me is that it came back with 0 issues > when clearly there were considering what happens when I kick 1 disk > out.Zero issues seems like a good thing. Resilvering the disk in the pool performed most of the functions that scrub does so it should not surprise that there are no more issues remaining.> Similarly I''ve seen that ''zpool clear'' just sets you up for problems > down the line. It just pretends there aren''t errors.As far as I am aware, the data cleared by ''zpool clear'' is for administrators to confirm they are aware the issue occured. A good administrator will consider any implications. The decision made for a high capacity SATA drive should likely be different than that made for a low-capacity enterprise SAS drive. Studies by Google suggest that SATA drives will experience many more block errors than enterprise SAS drives, and that higher error rates should be allowed from SATA drives than SAS drives when considering to replace the disk. Drives which experience continually more block failures are doomed to fail. Bob -- Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/