Greetings,
It seems that ZFS ate my data. I was wondering if one of you smart
people could help me get it back.
This happened on a Sun x4500 (Thumper) with one big RAID-10 zpool.
Solaris 10 update 4, patched to kernel 137138-09, zpool was version
10.
1) I was preparing for a Thumper-to-Thor hardware upgrade, and
performed a zpool export, as specified in the Sun-provided upgrade
document.
2) For irrelevant reasons, the hardware upgrade was canceled. The
server was never powered down or rebooted. 2-3 hours after the
export, I attempted to re-import the zpool (named "data"), and got the
error:
# zpool import data
cannot import ''data'': one or more devices is currently
unavailable
So I have a disk failure? No problem, it''s RAID 10, I''ll just
replace
the bad drive and re-silver, right? *bonk* wrong answer.
Since the zpool can''t be imported, none of the recovery commands are
available. None of these work on un-imported pools:
zpool clear
zpool replace
zpool offline
zpool scrub
Here is the zpool import output:
dsdw2-1 /root# zpool import
pool: data
id: 6825159954177683445
state: UNAVAIL
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
devices and try again.
see: http://www.sun.com/msg/ZFS-8000-6X
config:
data UNAVAIL missing device
mirror ONLINE
c5t3d0 ONLINE
c4t3d0 ONLINE
mirror ONLINE
c5t7d0 ONLINE
c4t7d0 ONLINE
mirror ONLINE
c7t3d0 ONLINE
c6t3d0 ONLINE
mirror ONLINE
c7t7d0 ONLINE
c6t7d0 ONLINE
mirror DEGRADED
c1t3d0 FAULTED corrupted data
c0t3d0 ONLINE
mirror ONLINE
c1t7d0 ONLINE
c0t7d0 ONLINE
mirror ONLINE
c5t2d0 ONLINE
c4t2d0 ONLINE
mirror ONLINE
c5t6d0 ONLINE
c4t6d0 ONLINE
mirror ONLINE
c7t2d0 ONLINE
c6t2d0 ONLINE
mirror ONLINE
c7t6d0 ONLINE
c6t6d0 ONLINE
mirror ONLINE
c1t2d0 ONLINE
c0t2d0 ONLINE
mirror ONLINE
c1t6d0 ONLINE
c0t6d0 ONLINE
mirror ONLINE
c5t1d0 ONLINE
c4t1d0 ONLINE
mirror ONLINE
c5t5d0 ONLINE
c4t5d0 ONLINE
mirror ONLINE
c7t1d0 ONLINE
c6t1d0 ONLINE
mirror ONLINE
c7t5d0 ONLINE
c6t5d0 ONLINE
mirror ONLINE
c1t1d0 ONLINE
c0t1d0 ONLINE
mirror ONLINE
c1t5d0 ONLINE
c0t5d0 ONLINE
mirror ONLINE
c4t0d0 ONLINE
c4t4d0 ONLINE
mirror ONLINE
c7t0d0 ONLINE
c7t4d0 ONLINE
mirror ONLINE
c6t0d0 ONLINE
c6t4d0 ONLINE
Additional devices are known to be part of this pool, though their
exact configuration cannot be determined.
Now I don''t know what missing devices it''s talking about,
because I
have found no hardware errors via fmadm or the message log, or the
LOM. format shows all the disks just fine, including the one ZFS does
not like, c1t3d0.
Any advice is welcome.
S^2