zfs discuss - Aug 2012 - Zpool recovery after too many failed disks

RAIDz set, lost a disk, replaced it... lost another disk during resilver. 
Replaced it, ran another resilver, and now it shows all disks with too many
errors.

Safe to say this is getting rebuilt and restored, or is there hope to recover
some of the data?  I assume this is the case because rpool/filemover has errors,
is that fixable?



# zpool status -v
  pool: rpool
state: DEGRADED
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: http://www.sun.com/msg/ZFS-8000-8A
scrub: resilver completed after 4h51m with 190449 errors on Sat Aug 25 05:45:12
2012
config:

        NAME              STATE     READ WRITE CKSUM
        rpool             DEGRADED  455K     0     0
          raidz1          DEGRADED  455K     0     0
            c3t0d0        DEGRADED     0     0     0  too many errors
            c2t1d0        DEGRADED     0     0     0  too many errors
            replacing     UNAVAIL      0     0     0  insufficient replicas
              c2t0d0s0/o  FAULTED      0     0     0  too many errors
              c2t0d0      FAULTED      0     0     0  too many errors
            c3t1d0        DEGRADED     0     0     0  too many errors
            c4t0d0        DEGRADED     0     0     0  too many errors
            c4t1d0        DEGRADED     0     0     0  too many errors

errors: Permanent errors have been detected in the following files:

        rpool/filemover:<0x1>

# zfs list
NAME              USED  AVAIL  REFER  MOUNTPOINT
rpool            6.64T      0  29.9K  /rpool
rpool/filemover  6.64T   323G  6.32T  -

Thanks
Mark

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20120827/b0ce3781/attachment.html>

On 08/27/2012 09:02 PM, Mark Wolek wrote:
> RAIDz set, lost a disk, replaced it... lost another disk during resilver. 
Replaced it, ran another resilver, and now it shows all disks with too many
errors.
> 
> Safe to say this is getting rebuilt and restored, or is there hope to
recover some of the data?  I assume this is the case because rpool/filemover has
errors, is that fixable?
It seems you fell into the standard two-disk failure mode during
resilver. If this is really the case, it seems like your pool is lost,
because raidz works by treating each block as a single raidz stripe and
spreading it over the component devices - as a result, most of your
blocks will probably be missing some data.

You can try and retrieve as much data from the pool as possible (via
something like rsync or tar), though I''m not exactly certain how well
(or whether at all) that will work.

--
Saso