I am in a rather unique situation. I''ve inherited a zpool composed of
two vdevs. One vdev was roughly 9TB on one RAID 5 array, and the
other vdev is roughly 2TB on a different RAID 5 array. The 9TB
array crashed and was sent to a data recovery firm, and they''ve given
me a dd image. I''ve also taken a dd image of the other side.
-rw-r--r-- 1 root wheel 9.2T Aug 18 07:53 deep_Lun0.dd
-rw-r--r-- 1 root wheel 2.1T Aug 23 15:55 deep_san01_lun.dd
zpool import identifies that both files are members of the pool, but
there are ''insufficient replicas'' and it can not import.
zpool import -d . -F
pool: deep
id: 8026270260630449340
state: UNAVAIL
status: The pool was last accessed by another system.
action: The pool cannot be imported due to damaged devices or data.
see: http://www.sun.com/msg/ZFS-8000-EY
config:
deep UNAVAIL
insufficient replicas
/jbod1-diskbackup/restore/deep_san01_lun.dd UNAVAIL cannot open
/jbod1-diskbackup/restore/deep_Lun0.dd UNAVAIL cannot open
Using zdb, I can see both images have 4 labels. What are my options
in terms of attempting a recovery? I had hoped that ZFS could import
the pool even if the vdevs are now files instead of devices, but
unfortunately it was not so easy.
zdb -l deep_Lun0.dd
--------------------------------------------
LABEL 0
--------------------------------------------
version: 15
name: ''deep''
state: 0
txg: 322458504
pool_guid: 8026270260630449340
hostid: 713828554
hostname: ''vali.REDACTED
top_guid: 7224951874042153155
guid: 7224951874042153155
vdev_tree:
type: ''disk''
id: 1
guid: 7224951874042153155
path:
''/dev/dsk/c4t526169645765622E436F6D202020202030303330383933323030303130363120d0s0''
devid: ''id1,sd at TRaidWeb.Com_____003089320001061_/a''
phys_path:
''/scsi_vhci/disk at
g526169645765622e436f6d202020202030303330383933323030303130363120:a''
whole_disk: 0
metaslab_array: 148
metaslab_shift: 36
ashift: 9
asize: 10076008480768
is_log: 0
DTL: 213
--------------------------------------------
LABEL 1
--------------------------------------------
version: 15
name: ''deep''
state: 0
txg: 322458504
pool_guid: 8026270260630449340
hostid: 713828554
hostname: ''vali.REDACTED
top_guid: 7224951874042153155
guid: 7224951874042153155
vdev_tree:
type: ''disk''
id: 1
guid: 7224951874042153155
path:
''/dev/dsk/c4t526169645765622E436F6D202020202030303330383933323030303130363120d0s0''
devid: ''id1,sd at TRaidWeb.Com_____003089320001061_/a''
phys_path:
''/scsi_vhci/disk at
g526169645765622e436f6d202020202030303330383933323030303130363120:a''
whole_disk: 0
metaslab_array: 148
metaslab_shift: 36
ashift: 9
asize: 10076008480768
is_log: 0
DTL: 213
--------------------------------------------
LABEL 2
--------------------------------------------
version: 15
name: ''deep''
state: 0
txg: 322458504
pool_guid: 8026270260630449340
hostid: 713828554
hostname: ''vali.REDACTED
top_guid: 7224951874042153155
guid: 7224951874042153155
vdev_tree:
type: ''disk''
id: 1
guid: 7224951874042153155
path:
''/dev/dsk/c4t526169645765622E436F6D202020202030303330383933323030303130363120d0s0''
devid: ''id1,sd at TRaidWeb.Com_____003089320001061_/a''
phys_path:
''/scsi_vhci/disk at
g526169645765622e436f6d202020202030303330383933323030303130363120:a''
whole_disk: 0
metaslab_array: 148
metaslab_shift: 36
ashift: 9
asize: 10076008480768
is_log: 0
DTL: 213
--------------------------------------------
LABEL 3
--------------------------------------------
version: 15
name: ''deep''
state: 0
txg: 322458504
pool_guid: 8026270260630449340
hostid: 713828554
hostname: ''vali.REDACTED
top_guid: 7224951874042153155
guid: 7224951874042153155
vdev_tree:
type: ''disk''
id: 1
guid: 7224951874042153155
path:
''/dev/dsk/c4t526169645765622E436F6D202020202030303330383933323030303130363120d0s0''
devid: ''id1,sd at TRaidWeb.Com_____003089320001061_/a''
phys_path:
''/scsi_vhci/disk at
g526169645765622e436f6d202020202030303330383933323030303130363120:a''
whole_disk: 0
metaslab_array: 148
metaslab_shift: 36
ashift: 9
asize: 10076008480768
is_log: 0
DTL: 213
zdb -l deep_san01_lun.dd
--------------------------------------------
LABEL 0
--------------------------------------------
version: 15
name: ''deep''
state: 0
txg: 322458504
pool_guid: 8026270260630449340
hostid: 713828554
hostname: ''vali.REDACTED
top_guid: 6101030958963841536
guid: 6101030958963841536
vdev_tree:
type: ''disk''
id: 0
guid: 6101030958963841536
path:
''/dev/dsk/c4t526169645765622E436F6D202020202030303131383933323030303130354220d0s0''
devid: ''id1,sd at TRaidWeb.Com_____00118932000105B_/a''
phys_path:
''/scsi_vhci/disk at
g526169645765622e436f6d202020202030303131383933323030303130354220:a''
whole_disk: 0
metaslab_array: 13
metaslab_shift: 34
ashift: 9
asize: 2329718554624
is_log: 0
DTL: 214
--------------------------------------------
LABEL 1
--------------------------------------------
version: 15
name: ''deep''
state: 0
txg: 322458504
pool_guid: 8026270260630449340
hostid: 713828554
hostname: ''vali.REDACTED
top_guid: 6101030958963841536
guid: 6101030958963841536
vdev_tree:
type: ''disk''
id: 0
guid: 6101030958963841536
path:
''/dev/dsk/c4t526169645765622E436F6D202020202030303131383933323030303130354220d0s0''
devid: ''id1,sd at TRaidWeb.Com_____00118932000105B_/a''
phys_path:
''/scsi_vhci/disk at
g526169645765622e436f6d202020202030303131383933323030303130354220:a''
whole_disk: 0
metaslab_array: 13
metaslab_shift: 34
ashift: 9
asize: 2329718554624
is_log: 0
DTL: 214
--------------------------------------------
LABEL 2
--------------------------------------------
version: 15
name: ''deep''
state: 0
txg: 322458504
pool_guid: 8026270260630449340
hostid: 713828554
hostname: ''vali.REDACTED
top_guid: 6101030958963841536
guid: 6101030958963841536
vdev_tree:
type: ''disk''
id: 0
guid: 6101030958963841536
path:
''/dev/dsk/c4t526169645765622E436F6D202020202030303131383933323030303130354220d0s0''
devid: ''id1,sd at TRaidWeb.Com_____00118932000105B_/a''
phys_path:
''/scsi_vhci/disk at
g526169645765622e436f6d202020202030303131383933323030303130354220:a''
whole_disk: 0
metaslab_array: 13
metaslab_shift: 34
ashift: 9
asize: 2329718554624
is_log: 0
DTL: 214
--------------------------------------------
LABEL 3
--------------------------------------------
version: 15
name: ''deep''
state: 0
txg: 322458504
pool_guid: 8026270260630449340
hostid: 713828554
hostname: ''vali.REDACTED
top_guid: 6101030958963841536
guid: 6101030958963841536
vdev_tree:
type: ''disk''
id: 0
guid: 6101030958963841536
path:
''/dev/dsk/c4t526169645765622E436F6D202020202030303131383933323030303130354220d0s0''
devid: ''id1,sd at TRaidWeb.Com_____00118932000105B_/a''
phys_path:
''/scsi_vhci/disk at
g526169645765622e436f6d202020202030303131383933323030303130354220:a''
whole_disk: 0
metaslab_array: 13
metaslab_shift: 34
ashift: 9
asize: 2329718554624
is_log: 0
DTL: 214
Hi Kelsey, I haven''t had to do this myself so someone who has done this before might have a better suggestion. I wonder if you need to make links from the original device name to the new device names. You can see from the zdb -l output below that the device path is pointing to the original device names (really long device names). Thanks, Cindy On 08/24/11 12:11, Kelsey Damas wrote:> I am in a rather unique situation. I''ve inherited a zpool composed of > two vdevs. One vdev was roughly 9TB on one RAID 5 array, and the > other vdev is roughly 2TB on a different RAID 5 array. The 9TB > array crashed and was sent to a data recovery firm, and they''ve given > me a dd image. I''ve also taken a dd image of the other side. > > -rw-r--r-- 1 root wheel 9.2T Aug 18 07:53 deep_Lun0.dd > -rw-r--r-- 1 root wheel 2.1T Aug 23 15:55 deep_san01_lun.dd > > zpool import identifies that both files are members of the pool, but > there are ''insufficient replicas'' and it can not import. > > zpool import -d . -F > pool: deep > id: 8026270260630449340 > state: UNAVAIL > status: The pool was last accessed by another system. > action: The pool cannot be imported due to damaged devices or data. > see: http://www.sun.com/msg/ZFS-8000-EY > config: > > deep UNAVAIL > insufficient replicas > /jbod1-diskbackup/restore/deep_san01_lun.dd UNAVAIL cannot open > /jbod1-diskbackup/restore/deep_Lun0.dd UNAVAIL cannot open > > Using zdb, I can see both images have 4 labels. What are my options > in terms of attempting a recovery? I had hoped that ZFS could import > the pool even if the vdevs are now files instead of devices, but > unfortunately it was not so easy. > > > zdb -l deep_Lun0.dd > -------------------------------------------- > LABEL 0 > -------------------------------------------- > version: 15 > name: ''deep'' > state: 0 > txg: 322458504 > pool_guid: 8026270260630449340 > hostid: 713828554 > hostname: ''vali.REDACTED > top_guid: 7224951874042153155 > guid: 7224951874042153155 > vdev_tree: > type: ''disk'' > id: 1 > guid: 7224951874042153155 > path: ''/dev/dsk/c4t526169645765622E436F6D202020202030303330383933323030303130363120d0s0'' > devid: ''id1,sd at TRaidWeb.Com_____003089320001061_/a'' > phys_path: > ''/scsi_vhci/disk at g526169645765622e436f6d202020202030303330383933323030303130363120:a'' > whole_disk: 0 > metaslab_array: 148 > metaslab_shift: 36 > ashift: 9 > asize: 10076008480768 > is_log: 0 > DTL: 213 > -------------------------------------------- > LABEL 1 > -------------------------------------------- > version: 15 > name: ''deep'' > state: 0 > txg: 322458504 > pool_guid: 8026270260630449340 > hostid: 713828554 > hostname: ''vali.REDACTED > top_guid: 7224951874042153155 > guid: 7224951874042153155 > vdev_tree: > type: ''disk'' > id: 1 > guid: 7224951874042153155 > path: ''/dev/dsk/c4t526169645765622E436F6D202020202030303330383933323030303130363120d0s0'' > devid: ''id1,sd at TRaidWeb.Com_____003089320001061_/a'' > phys_path: > ''/scsi_vhci/disk at g526169645765622e436f6d202020202030303330383933323030303130363120:a'' > whole_disk: 0 > metaslab_array: 148 > metaslab_shift: 36 > ashift: 9 > asize: 10076008480768 > is_log: 0 > DTL: 213 > -------------------------------------------- > LABEL 2 > -------------------------------------------- > version: 15 > name: ''deep'' > state: 0 > txg: 322458504 > pool_guid: 8026270260630449340 > hostid: 713828554 > hostname: ''vali.REDACTED > top_guid: 7224951874042153155 > guid: 7224951874042153155 > vdev_tree: > type: ''disk'' > id: 1 > guid: 7224951874042153155 > path: ''/dev/dsk/c4t526169645765622E436F6D202020202030303330383933323030303130363120d0s0'' > devid: ''id1,sd at TRaidWeb.Com_____003089320001061_/a'' > phys_path: > ''/scsi_vhci/disk at g526169645765622e436f6d202020202030303330383933323030303130363120:a'' > whole_disk: 0 > metaslab_array: 148 > metaslab_shift: 36 > ashift: 9 > asize: 10076008480768 > is_log: 0 > DTL: 213 > -------------------------------------------- > LABEL 3 > -------------------------------------------- > version: 15 > name: ''deep'' > state: 0 > txg: 322458504 > pool_guid: 8026270260630449340 > hostid: 713828554 > hostname: ''vali.REDACTED > top_guid: 7224951874042153155 > guid: 7224951874042153155 > vdev_tree: > type: ''disk'' > id: 1 > guid: 7224951874042153155 > path: ''/dev/dsk/c4t526169645765622E436F6D202020202030303330383933323030303130363120d0s0'' > devid: ''id1,sd at TRaidWeb.Com_____003089320001061_/a'' > phys_path: > ''/scsi_vhci/disk at g526169645765622e436f6d202020202030303330383933323030303130363120:a'' > whole_disk: 0 > metaslab_array: 148 > metaslab_shift: 36 > ashift: 9 > asize: 10076008480768 > is_log: 0 > DTL: 213 > > > > zdb -l deep_san01_lun.dd > -------------------------------------------- > LABEL 0 > -------------------------------------------- > version: 15 > name: ''deep'' > state: 0 > txg: 322458504 > pool_guid: 8026270260630449340 > hostid: 713828554 > hostname: ''vali.REDACTED > top_guid: 6101030958963841536 > guid: 6101030958963841536 > vdev_tree: > type: ''disk'' > id: 0 > guid: 6101030958963841536 > path: ''/dev/dsk/c4t526169645765622E436F6D202020202030303131383933323030303130354220d0s0'' > devid: ''id1,sd at TRaidWeb.Com_____00118932000105B_/a'' > phys_path: > ''/scsi_vhci/disk at g526169645765622e436f6d202020202030303131383933323030303130354220:a'' > whole_disk: 0 > metaslab_array: 13 > metaslab_shift: 34 > ashift: 9 > asize: 2329718554624 > is_log: 0 > DTL: 214 > -------------------------------------------- > LABEL 1 > -------------------------------------------- > version: 15 > name: ''deep'' > state: 0 > txg: 322458504 > pool_guid: 8026270260630449340 > hostid: 713828554 > hostname: ''vali.REDACTED > top_guid: 6101030958963841536 > guid: 6101030958963841536 > vdev_tree: > type: ''disk'' > id: 0 > guid: 6101030958963841536 > path: ''/dev/dsk/c4t526169645765622E436F6D202020202030303131383933323030303130354220d0s0'' > devid: ''id1,sd at TRaidWeb.Com_____00118932000105B_/a'' > phys_path: > ''/scsi_vhci/disk at g526169645765622e436f6d202020202030303131383933323030303130354220:a'' > whole_disk: 0 > metaslab_array: 13 > metaslab_shift: 34 > ashift: 9 > asize: 2329718554624 > is_log: 0 > DTL: 214 > -------------------------------------------- > LABEL 2 > -------------------------------------------- > version: 15 > name: ''deep'' > state: 0 > txg: 322458504 > pool_guid: 8026270260630449340 > hostid: 713828554 > hostname: ''vali.REDACTED > top_guid: 6101030958963841536 > guid: 6101030958963841536 > vdev_tree: > type: ''disk'' > id: 0 > guid: 6101030958963841536 > path: ''/dev/dsk/c4t526169645765622E436F6D202020202030303131383933323030303130354220d0s0'' > devid: ''id1,sd at TRaidWeb.Com_____00118932000105B_/a'' > phys_path: > ''/scsi_vhci/disk at g526169645765622e436f6d202020202030303131383933323030303130354220:a'' > whole_disk: 0 > metaslab_array: 13 > metaslab_shift: 34 > ashift: 9 > asize: 2329718554624 > is_log: 0 > DTL: 214 > -------------------------------------------- > LABEL 3 > -------------------------------------------- > version: 15 > name: ''deep'' > state: 0 > txg: 322458504 > pool_guid: 8026270260630449340 > hostid: 713828554 > hostname: ''vali.REDACTED > top_guid: 6101030958963841536 > guid: 6101030958963841536 > vdev_tree: > type: ''disk'' > id: 0 > guid: 6101030958963841536 > path: ''/dev/dsk/c4t526169645765622E436F6D202020202030303131383933323030303130354220d0s0'' > devid: ''id1,sd at TRaidWeb.Com_____00118932000105B_/a'' > phys_path: > ''/scsi_vhci/disk at g526169645765622e436f6d202020202030303131383933323030303130354220:a'' > whole_disk: 0 > metaslab_array: 13 > metaslab_shift: 34 > ashift: 9 > asize: 2329718554624 > is_log: 0 > DTL: 214 > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
On Wed, Aug 24, 2011 at 1:23 PM, Cindy Swearingen <cindy.swearingen at oracle.com> wrote:> I wonder if you need to make links from the original device > name to the new device names. > > You can see from the zdb -l output below that the device path > is pointing to the original device names (really long device > names).Thank you for this suggestion. I''ve created symlinks from the /dev/dsk directory pointing to the dd image, but zpool import says the same thing. However, look at the error message below. zpool can see the hostname of the last system to use the pool, but the date looks like the epoch (Dec 31 1969). Is this meaningful? # ln -s /jbod1-diskbackup/restore/deep_Lun0.dd /dev/dsk/c4t526169645765622E436F6D202020202030303330383933323030303130363120d0s0 # ln -s /jbod1-diskbackup/restore/deep_san01_lun.dd /dev/dsk/c4t526169645765622E436F6D202020202030303131383933323030303130354220d0s0 # zpool import -d . pool: deep id: 8026270260630449340 state: UNAVAIL status: The pool was last accessed by another system. action: The pool cannot be imported due to damaged devices or data. see: http://www.sun.com/msg/ZFS-8000-EY config: deep UNAVAIL insufficient replicas /jbod1-diskbackup/restore/deep_san01_lun.dd UNAVAIL cannot open /jbod1-diskbackup/restore/deep_Lun0.dd UNAVAIL cannot open # zpool import -d . -F deep cannot import ''deep'': pool may be in use from other system, it was last accessed by vali.REDACTED (hostid: 0x2a8c28ca) on Wed Dec 31 16:00:00 1969 use ''-f'' to import anyway # zpool import -d . -F -f deep cannot import ''deep'': one or more devices is currently unavailable
On 24 August, 2011 - Kelsey Damas sent me these 1,8K bytes:> On Wed, Aug 24, 2011 at 1:23 PM, Cindy Swearingen > <cindy.swearingen at oracle.com> wrote: > > > I wonder if you need to make links from the original device > > name to the new device names. > > > > You can see from the zdb -l output below that the device path > > is pointing to the original device names (really long device > > names). > > Thank you for this suggestion. I''ve created symlinks from the > /dev/dsk directory pointing to the dd image, but zpool import says the > same thing. However, look at the error message below. zpool can > see the hostname of the last system to use the pool, but the date > looks like the epoch (Dec 31 1969). Is this meaningful? > > # ln -s /jbod1-diskbackup/restore/deep_Lun0.dd > /dev/dsk/c4t526169645765622E436F6D202020202030303330383933323030303130363120d0s0 > # ln -s /jbod1-diskbackup/restore/deep_san01_lun.dd > /dev/dsk/c4t526169645765622E436F6D202020202030303131383933323030303130354220d0s0 > > # zpool import -d .Just for fun, try an absolute path. /Tomas -- Tomas Forsman, stric at acc.umu.se, http://www.acc.umu.se/~stric/ |- Student at Computing Science, University of Ume? `- Sysadmin at {cs,acc}.umu.se
> Just for fun, try an absolute path.Thank you again for the suggestions. I was able to make this work with lofiadm to mount the images. Then, be sure to give zpool the -d flag to scan /dev/lofi # lofiadm -a /jbod1-diskbackup/restore/deep_Lun0.dd /dev/lofi/1 # lofiadm -a /jbod1-diskbackup/restore/deep_san01_lun.dd /dev/lofi/2 # zpool import -d /dev/lofi pool: deep id: 8026270260630449340 state: ONLINE status: The pool was last accessed by another system. action: The pool can be imported using its name or numeric identifier and the ''-f'' flag. see: http://www.sun.com/msg/ZFS-8000-EY config: deep ONLINE /dev/lofi/2 ONLINE /dev/lofi/1 ONLINE
markm wrote:> Because the vdev tree is calling them ''disk'', zfs is attempting to open > them using disk i/o instead of file i/o.This was correct, thank you. lofiadm was useful to loopback mount the image files to provide disk i/o.> ZFS has much more opportunity to recover from device failure when it has a > redundant config. Splitting the 9T & 2T LUNs into a few separate LUNs and > then using raidz would be highly desirable.Yes, I completely agree. This configuration is something I inherited from a previous colleague, and did not make good use of ZFS. The underlying RAID system suffered a catastrophic failure, but even prior to that, we would consistently have the pools become DEGRADED when ZFS would report corruption that the underlying RAID system could not detect and repair. Thank you to all responders, and also thank you to Drive Savers who recovered the image for us. If you find yourself in the regrettable situation of having a dozen terabytes go missing one day without proper backups, I highly recommend their services. -- Kelsey