zfs discuss - Aug 2009 - Why so many data errors with raidz2 config and one failing drive?

Added a third raidz2 vdev to my pool:
  pool: tww
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: http://www.sun.com/msg/ZFS-8000-8A
 scrub: resilver in progress for 0h57m, 13.36% done, 6h9m to go
config:

        NAME                                         STATE     READ
WRITE CKSUM
        tww                                          DEGRADED     0
0 16.9K
          raidz2                                     ONLINE       0
0     0
            c6t600A0B8000299966000005964668CB39d0    ONLINE       0
0     0
            c6t600A0B8000299CCC000006C84744C892d0    ONLINE       0
0     0
            c6t600A0B8000299CCC000005B44668CC6Ad0    ONLINE       0
0     0
            c6t600A0B8000299966000005A44668CC3Fd0    ONLINE       0
0     0
            c6t600A0B8000299CCC000005BA4668CD2Ed0    ONLINE       0
0     0
            c6t600A0B8000299966000005AA4668CDB1d0    ONLINE       0
0     0
            c6t600A0B80002999660000073547C5CED9d0    ONLINE       0
0     0
          raidz2                                     ONLINE       0
0     0
            c6t600A0B8000299966000005B04668F17Dd0    ONLINE       0
0     0
            c6t600A0B8000299CCC0000099E4A400B94d0    ONLINE       0
0     0
            c6t600A0B8000299966000005B64668F26Fd0    ONLINE       0
0     0
            c6t600A0B8000299CCC000005CC4668F30Ed0    ONLINE       0
0     0
            c6t600A0B8000299966000005BC4668F305d0    ONLINE       0
0     0
            c6t600A0B8000299CCC0000099B4A400A9Cd0    ONLINE       0
0     0
            c6t600A0B8000299966000005C24668F39Bd0    ONLINE       0
0     0
          raidz2                                     DEGRADED     0
0 34.0K
            c6t600A0B8000299CCC00000A154A89E426d0    ONLINE       0
0     0
            c6t600A0B8000299966000009F74A89E1A5d0    ONLINE       0
0     7  4K resilvered
            c6t600A0B8000299CCC00000A174A89E520d0    ONLINE       0
0     2  4K resilvered
            c6t600A0B8000299966000009F94A89E24Bd0    ONLINE       0
0    48  24.5K resilvered
            replacing                                DEGRADED     0
0 78.7K
              c6t600A0B8000299CCC00000A194A89E634d0  UNAVAIL     20
277K     0  experienced I/O failures
              c6t600A0B8000299966000009EE4A89DA51d0  ONLINE       0
0     0  38.1M resilvered
            c6t600A0B8000299CCC00000A0C4A89DDE8d0    ONLINE       0
0     6  6K resilvered
            c6t600A0B8000299966000009F04A89DB1Bd0    ONLINE       0
0    86  92K resilvered
        spares
          c6t600A0B8000299CCC000005D84668F448d0      AVAIL   
          c6t600A0B8000299966000005C84668F461d0      AVAIL   

errors: 17097 data errors, use ''-v'' for a list


Seems some of the new drives are having problems, resulting in CKSUM
errors. I don''t understand why I have so many data errors though. Why
does the third raidz2 vdev report 34.0K CKSUM errors?

The number of data errors appears to be increasing as well as the
resilver process continues.

-- 
albert chin (china at thewrittenword.com)

On Mon, 24 Aug 2009, Albert Chin wrote:
>
> Seems some of the new drives are having problems, resulting in CKSUM
> errors. I don''t understand why I have so many data errors though.
Why
> does the third raidz2 vdev report 34.0K CKSUM errors?
Is it possible that this third raidz2 is inflicted with a shared 
problem such as a cable, controller, backplane, or power supply? 
Only one drive is reported as being unscathed.
> The number of data errors appears to be increasing as well as the
> resilver process continues.
Bad news.

Do you periodically scrub your array?

Bob
--
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

On Mon, Aug 24, 2009 at 02:01:39PM -0500, Bob Friesenhahn
wrote:
> On Mon, 24 Aug 2009, Albert Chin wrote:
>>
>> Seems some of the new drives are having problems, resulting in CKSUM
>> errors. I don''t understand why I have so many data errors
though. Why
>> does the third raidz2 vdev report 34.0K CKSUM errors?
>
> Is it possible that this third raidz2 is inflicted with a shared
> problem such as a cable, controller, backplane, or power supply? Only
> one drive is reported as being unscathed.
Well, we''re just using unused drives on the existing array. No other
changes.
> Do you periodically scrub your array?
No. Guess we will now :) But, I think all of the data loss is a result
of the new drives, not ones that were already part of the two previous
vdevs.

-- 
albert chin (china at thewrittenword.com)