We have a production SunFireV240 that had a zfs mirror until this week. One of
the drives (c1t3d0) in the mirror failed.
The system was shutdown and the bad disk replaced without an export.
I don''t know what happened next but by the time I got involved there
was no evidence that the remaining good disk (c1t2d0) had ever been part of a
ZFS mirror.
Using dd on the raw device I can see data on slice 6 of the good disk but
can''t import it.
Is there any way to recover from this or are they SOL?
Thanks in advance
# zpool status
no pools available
# zpool import
# ls /etc/zfs
#
ls /dev/dsk
c0t0d0s0 c0t0d0s3 c0t0d0s6 c1t0d0s1 c1t0d0s4 c1t0d0s7 c1t1d0s2 c1t1d0s5
c1t2d0 c1t2d0s2 c1t2d0s5 c1t3d0s0 c1t3d0s3 c1t3d0s6
c0t0d0s1 c0t0d0s4 c0t0d0s7 c1t0d0s2 c1t0d0s5 c1t1d0s0 c1t1d0s3 c1t1d0s6
c1t2d0s0 c1t2d0s3 c1t2d0s6 c1t3d0s1 c1t3d0s4
c0t0d0s2 c0t0d0s5 c1t0d0s0 c1t0d0s3 c1t0d0s6 c1t1d0s1 c1t1d0s4 c1t1d0s7
c1t2d0s1 c1t2d0s4 c1t3d0 c1t3d0s2 c1t3d0s5
# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c1t0d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424> bootdisk
/pci at 1c,600000/scsi at 2/sd at 0,0
1. c1t1d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424> bootmirr
/pci at 1c,600000/scsi at 2/sd at 1,0
2. c1t2d0 <SUN146G cyl 14087 alt 2 hd 24 sec 848>
/pci at 1c,600000/scsi at 2/sd at 2,0
3. c1t3d0 <SUN146G cyl 14087 alt 2 hd 24 sec 848>
/pci at 1c,600000/scsi at 2/sd at 3,0
Specify disk (enter its number): 2
selecting c1t2d0
format> inquiry
Vendor: FUJITSU
Product: MAT3147N SUN146G
Revision: 1703
format> current
Current Disk = c1t2d0
<SUN146G cyl 14087 alt 2 hd 24 sec 848>
/pci at 1c,600000/scsi at 2/sd at 2,0
format> verify
Primary label contents:
Volume name = < >
ascii name = <SUN146G cyl 14087 alt 2 hd 24 sec 848>
pcyl = 14089
ncyl = 14087
acyl = 2
nhead = 24
nsect = 848
Part Tag Flag Cylinders Size Blocks
0 root wm 0 - 12 129.19MB (13/0/0) 264576
1 swap wu 13 - 25 129.19MB (13/0/0) 264576
2 backup wu 0 - 14086 136.71GB (14087/0/0) 286698624
3 unassigned wm 0 0 (0/0/0) 0
4 unassigned wm 0 0 (0/0/0) 0
5 unassigned wm 0 0 (0/0/0) 0
6 usr wm 26 - 14086 136.46GB (14061/0/0) 286169472
7 unassigned wm 0 0 (0/0/0) 0
format> quit
# dd if=/dev/rdsk/c1t2d0s6 count=1 | od -x
1+0 records in
1+0 records out
0000000 7215 2b79 046c 8ddc 3e31 6966 caa4 6950
0000020 9c60 4514 7d4a 2a13 9b66 e69e d484 a327
0000040 4eb0 220e 9c7f 6604 6182 7b39 1310 9c5c
0000060 4584 c7c6 bd51 aba9 7b4d ec9a 99b2 6bc2
0000100 6cab 7a88 46d7 937d 5026 86cd 4cf9 ae83
0000120 20f3 44ec c22e d322 e6cc 2c09 f598 caf4
0000140 a9c5 85ad a695 8862 c6cc 124d bb72 d540
0000160 8886 2173 57cc 9759 a209 d78e 9a11 df4d
0000200 cdc4 5c99 259a 56e5 a301 d540 e691 182b
0000220 b354 93a9 bc33 085e 1fb6 0445 ac95 59aa
0000240 fb5a dd66 21de 2f18 24e7 d4c9 c464 99a5
0000260 9ae4 628a a434 7b96 d1a0 d761 3c21 3ed5
0000300 c417 5364 e5a3 837a dfd6 266c 50a6 4b10
0000320 95d5 2952 0f8f cb30 9ef0 23ab 6abc 6872
0000340 ed58 1977 79ff 9a89 0533 530e 6b83 95aa
0000360 630b f638 8508 02b1 6266 ca8a 6990 8ad4
0000400 47c2 7db3 9d9c 62cc ccb4 db3a 0803 ef35
0000420 0bd3 46b3 04bb d778 c471 9d65 de1b 1861
0000440 e0b9 ae27 d084 19da 716d b0ca 67be 07ea
0000460 5650 268e eb2c d7cc 083d c1a8 55ac 4c3c
0000500 d699 f558 d353 dc61 e25b 2bb8 7d8c 249c
0000520 c853 258a 01cd b366 bad3 2599 f8ac b3dc
0000540 6783 72eb 9029 926b 72e6 c84c 3cd7 59e1
0000560 f122 f20e f8d8 f32f 8226 ceeb acd0 ccf0
0000600 df3c f3f5 1e71 5d67 da75 1d84 b177 d21b
0000620 5fa8 a340 6404 2bec 2884 1d62 83cc 2498
0000640 4288 cf67 c6de 0970 75fe 9e05 8ed8 2173
0000660 fd30 4ec8 9ea0 63ee bd3f 7a07 b01a d04b
0000700 8045 29a6 6203 9ed3 9c16 740f 335e 53d8
0000720 c70e 9c73 981a f0f1 3547 8b84 0651 b1fb
0000740 b5c8 4887 dafe 15ab 721b 60d2 c1d8 8441
0000760 eee2 1896 2311 76da 1bfb 4422 3439 07e5
0001000
--
This message posted from opensolaris.org
Jim Sloey wrote:> We have a production SunFireV240 that had a zfs mirror until this week. One of the drives (c1t3d0) in the mirror failed. > The system was shutdown and the bad disk replaced without an export. > I don''t know what happened next but by the time I got involved there was no evidence that the remaining good disk (c1t2d0) had ever been part of a ZFS mirror. > Using dd on the raw device I can see data on slice 6 of the good disk but can''t import it.Did you use entire disks c1t2d0 and c1t3d0 to create your pool? If yes, then current labeling on c1t2d0 suggests that it got relabeled somehow in the process.> Is there any way to recover from this or are they SOL?First I''d make a full copy of c1t2d0 content (so you can try recovery several times). Then first thing I''d try is to relabel c1t2d0 with EFI label and check if the pool is there. regards, victor> > Thanks in advance > > # zpool status > no pools available > # zpool import > # ls /etc/zfs > # > > ls /dev/dsk > c0t0d0s0 c0t0d0s3 c0t0d0s6 c1t0d0s1 c1t0d0s4 c1t0d0s7 c1t1d0s2 c1t1d0s5 c1t2d0 c1t2d0s2 c1t2d0s5 c1t3d0s0 c1t3d0s3 c1t3d0s6 > c0t0d0s1 c0t0d0s4 c0t0d0s7 c1t0d0s2 c1t0d0s5 c1t1d0s0 c1t1d0s3 c1t1d0s6 c1t2d0s0 c1t2d0s3 c1t2d0s6 c1t3d0s1 c1t3d0s4 > c0t0d0s2 c0t0d0s5 c1t0d0s0 c1t0d0s3 c1t0d0s6 c1t1d0s1 c1t1d0s4 c1t1d0s7 c1t2d0s1 c1t2d0s4 c1t3d0 c1t3d0s2 c1t3d0s5 > > # format > Searching for disks...done > > > AVAILABLE DISK SELECTIONS: > 0. c1t0d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424> bootdisk > /pci at 1c,600000/scsi at 2/sd at 0,0 > 1. c1t1d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424> bootmirr > /pci at 1c,600000/scsi at 2/sd at 1,0 > 2. c1t2d0 <SUN146G cyl 14087 alt 2 hd 24 sec 848> > /pci at 1c,600000/scsi at 2/sd at 2,0 > 3. c1t3d0 <SUN146G cyl 14087 alt 2 hd 24 sec 848> > /pci at 1c,600000/scsi at 2/sd at 3,0 > Specify disk (enter its number): 2 > selecting c1t2d0 > > format> inquiry > Vendor: FUJITSU > Product: MAT3147N SUN146G > Revision: 1703 > > format> current > Current Disk = c1t2d0 > <SUN146G cyl 14087 alt 2 hd 24 sec 848> > /pci at 1c,600000/scsi at 2/sd at 2,0 > > format> verify > > Primary label contents: > > Volume name = < > > ascii name = <SUN146G cyl 14087 alt 2 hd 24 sec 848> > pcyl = 14089 > ncyl = 14087 > acyl = 2 > nhead = 24 > nsect = 848 > Part Tag Flag Cylinders Size Blocks > 0 root wm 0 - 12 129.19MB (13/0/0) 264576 > 1 swap wu 13 - 25 129.19MB (13/0/0) 264576 > 2 backup wu 0 - 14086 136.71GB (14087/0/0) 286698624 > 3 unassigned wm 0 0 (0/0/0) 0 > 4 unassigned wm 0 0 (0/0/0) 0 > 5 unassigned wm 0 0 (0/0/0) 0 > 6 usr wm 26 - 14086 136.46GB (14061/0/0) 286169472 > 7 unassigned wm 0 0 (0/0/0) 0 > > format> quit > > # dd if=/dev/rdsk/c1t2d0s6 count=1 | od -x > 1+0 records in > 1+0 records out > 0000000 7215 2b79 046c 8ddc 3e31 6966 caa4 6950 > 0000020 9c60 4514 7d4a 2a13 9b66 e69e d484 a327 > 0000040 4eb0 220e 9c7f 6604 6182 7b39 1310 9c5c > 0000060 4584 c7c6 bd51 aba9 7b4d ec9a 99b2 6bc2 > 0000100 6cab 7a88 46d7 937d 5026 86cd 4cf9 ae83 > 0000120 20f3 44ec c22e d322 e6cc 2c09 f598 caf4 > 0000140 a9c5 85ad a695 8862 c6cc 124d bb72 d540 > 0000160 8886 2173 57cc 9759 a209 d78e 9a11 df4d > 0000200 cdc4 5c99 259a 56e5 a301 d540 e691 182b > 0000220 b354 93a9 bc33 085e 1fb6 0445 ac95 59aa > 0000240 fb5a dd66 21de 2f18 24e7 d4c9 c464 99a5 > 0000260 9ae4 628a a434 7b96 d1a0 d761 3c21 3ed5 > 0000300 c417 5364 e5a3 837a dfd6 266c 50a6 4b10 > 0000320 95d5 2952 0f8f cb30 9ef0 23ab 6abc 6872 > 0000340 ed58 1977 79ff 9a89 0533 530e 6b83 95aa > 0000360 630b f638 8508 02b1 6266 ca8a 6990 8ad4 > 0000400 47c2 7db3 9d9c 62cc ccb4 db3a 0803 ef35 > 0000420 0bd3 46b3 04bb d778 c471 9d65 de1b 1861 > 0000440 e0b9 ae27 d084 19da 716d b0ca 67be 07ea > 0000460 5650 268e eb2c d7cc 083d c1a8 55ac 4c3c > 0000500 d699 f558 d353 dc61 e25b 2bb8 7d8c 249c > 0000520 c853 258a 01cd b366 bad3 2599 f8ac b3dc > 0000540 6783 72eb 9029 926b 72e6 c84c 3cd7 59e1 > 0000560 f122 f20e f8d8 f32f 8226 ceeb acd0 ccf0 > 0000600 df3c f3f5 1e71 5d67 da75 1d84 b177 d21b > 0000620 5fa8 a340 6404 2bec 2884 1d62 83cc 2498 > 0000640 4288 cf67 c6de 0970 75fe 9e05 8ed8 2173 > 0000660 fd30 4ec8 9ea0 63ee bd3f 7a07 b01a d04b > 0000700 8045 29a6 6203 9ed3 9c16 740f 335e 53d8 > 0000720 c70e 9c73 981a f0f1 3547 8b84 0651 b1fb > 0000740 b5c8 4887 dafe 15ab 721b 60d2 c1d8 8441 > 0000760 eee2 1896 2311 76da 1bfb 4422 3439 07e5 > 0001000
No. Only slice 6 from what I understand. I didn''t create this (the person who did has left the company) and all I know is that the pool was mounted on /oraprod before it faulted. -- This message posted from opensolaris.org
Never mind. It looks like the controller is flakey. Neither disk in the mirror is clean. Attempts to backup and recover the remaining disk produced I/O errors that were traced to the controller. Thanks for your help Victor. -- This message posted from opensolaris.org