Miles Nordin
2011-Apr-05 00:49 UTC
[zfs-discuss] ''cannot import ''andaman'': I/O error'', and failure to follow my own advice
I have a Solaris Express snv_130 box that imports a zpool from two
iSCSI targets, and after some power problems I cannot import the pool.
When I found the machine, the pool was FAULTED with half of most
mirrors shoring CORRUPTED DATA and half showing UNAVAIL. One of
the two iSCSI enclosures was on, while the other was off. When I
brought the other iSCSI enclosure up, bringing all the devices in each
of the seven mirror vdev''s online, the box paniced.
It went into a panic loop every time it tried to import the problem
pool at boot. I disabled all the iSCSI targets that make up the
problem pool and brought the box up, then saved a copy of
/etc/zfs/zpool.cache and exported the UNAVAIL pool. Then I turned the
host off, brought back all the iSCSI targets, and booted without a
crash, hoping I could ''zpool import'' the problem pool.
(Another mirrored pool on the same pair of iSCSI enclosures came back
fine and scrubbed with no errors. <shrug>)
Here is what I get typing some basic commands:
-----8<-----
terabithia:/# zpool import
pool: andaman
id: 7400719929021713582
state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:
andaman ONLINE
mirror-0 ONLINE
c3t43d0 ONLINE
c3t48d0 ONLINE
mirror-1 ONLINE
c3t45d0 ONLINE
c3t47d0 ONLINE
mirror-2 ONLINE
c3t52d0 ONLINE
c3t59d0 ONLINE
mirror-3 ONLINE
c3t46d0 ONLINE
c3t49d0 ONLINE
mirror-4 ONLINE
c3t50d0 ONLINE
c3t44d0 ONLINE
mirror-5 ONLINE
c3t57d0 ONLINE
c3t53d0 ONLINE
mirror-6 ONLINE
c3t54d0 ONLINE
c3t51d0 ONLINE
terabithia:/# zpool import andaman
cannot import ''andaman'': I/O error
Destroy and re-create the pool from
a backup source.
terabithia:/# zpool status
pool: aboveground
state: ONLINE
status: The pool is formatted using an older on-disk format. The pool can
still be used, but some features are unavailable.
action: Upgrade the pool using ''zpool upgrade''. Once this is
done, the
pool will no longer be accessible on older software versions.
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
aboveground ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c3t10d0 ONLINE 0 0 0
c3t16d0 ONLINE 0 0 0
errors: No known data errors
pool: rpool
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c1t0d0s0 ONLINE 0 0 0
c1t1d0s0 ONLINE 0 0 0
errors: No known data errors
terabithia:/# zpool import -F andaman
cannot import ''andaman'': I/O error
Destroy and re-create the pool from
a backup source.
terabithia:/# zdb -ve andaman
Configuration for import:
version: 22
pool_guid: 7400719929021713582
name: ''andaman''
state: 0
hostid: 2200768359
hostname: ''terabithia.th3h.inner.chaos''
vdev_children: 7
vdev_tree:
type: ''root''
id: 0
guid: 7400719929021713582
children[0]:
type: ''mirror''
id: 0
guid: 337393226491877361
whole_disk: 0
metaslab_array: 14
metaslab_shift: 33
ashift: 9
asize: 1000191557632
is_log: 0
children[0]:
type: ''disk''
id: 0
guid: 1781150413433362160
phys_path: ''/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h.fishstick%3Asd-andaman0001,0:a''
whole_disk: 1
DTL: 91
path: ''/dev/dsk/c3t43d0s0''
devid: ''id1,sd at
t49455400000000000000000002000000591000000f000000/a''
children[1]:
type: ''disk''
id: 1
guid: 7841235598547702997
phys_path: ''/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h%3Aoldfishstick%3Asd-andaman0001,1:a''
whole_disk: 1
DTL: 215
path: ''/dev/dsk/c3t48d0s0''
devid: ''id1,sd at
t49455400000000000000000002000000880e00000f000000/a''
children[1]:
type: ''mirror''
id: 1
guid: 1953060080997571723
whole_disk: 0
metaslab_array: 210
metaslab_shift: 33
ashift: 9
asize: 1000191557632
is_log: 0
children[0]:
type: ''disk''
id: 0
guid: 14567891505260370684
phys_path: ''/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h.fishstick%3Asd-andaman0001,3:a''
whole_disk: 1
DTL: 1395
path: ''/dev/dsk/c3t45d0s0''
devid: ''id1,sd at
t49455400000000000000000002000000731000000f000000/a''
children[1]:
type: ''disk''
id: 1
guid: 12913430974955856922
phys_path: ''/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h%3Aoldfishstick%3Asd-andaman0001,0:a''
whole_disk: 1
DTL: 217
path: ''/dev/dsk/c3t47d0s0''
devid: ''id1,sd at
t494554000000000000000000020000005c0f00000f000000/a''
children[2]:
type: ''mirror''
id: 2
guid: 17555572171866652488
metaslab_array: 330
metaslab_shift: 31
ashift: 9
asize: 1000191557632
is_log: 0
children[0]:
type: ''disk''
id: 0
guid: 10644297178963807622
phys_path: ''/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h%3Aoldfishstick%3Asd-andaman0001,7:a''
whole_disk: 1
DTL: 1167
path: ''/dev/dsk/c3t52d0s0''
devid: ''id1,sd at
t49455400000000000000000002000000090f00000f000000/a''
children[1]:
type: ''disk''
id: 1
guid: 2378339860349039724
phys_path: ''/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h.fishstick%3Asd-andaman0001,10:a''
whole_disk: 1
DTL: 1168
path: ''/dev/dsk/c3t59d0s0''
devid: ''id1,sd at
t49455400000000000000000002000000061100000f000000/a''
children[3]:
type: ''mirror''
id: 3
guid: 5261242283366420554
metaslab_array: 332
metaslab_shift: 31
ashift: 9
asize: 250046054400
is_log: 0
children[0]:
type: ''disk''
id: 0
guid: 17534037132307391717
phys_path: ''/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h.fishstick%3Asd-andaman0001,7:a''
whole_disk: 1
DTL: 335
path: ''/dev/dsk/c3t46d0s0''
devid: ''id1,sd at
t494554000000000000000000020000008a1000000f000000/a''
children[1]:
type: ''disk''
id: 1
guid: 2758124488221280115
phys_path: ''/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h%3Aoldfishstick%3Asd-andaman0001,2:a''
whole_disk: 1
DTL: 336
path: ''/dev/dsk/c3t49d0s0''
devid: ''id1,sd at
t49455400000000000000000002000000af0e00000f000000/a''
children[4]:
type: ''mirror''
id: 4
guid: 2126473073638606315
whole_disk: 0
metaslab_array: 510
metaslab_shift: 31
ashift: 9
asize: 750142881792
is_log: 0
children[0]:
type: ''disk''
id: 0
guid: 7272191256105073467
phys_path: ''/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h%3Aoldfishstick%3Asd-andaman0001,3:a''
whole_disk: 1
DTL: 660
path: ''/dev/dsk/c3t50d0s0''
devid: ''id1,sd at
t49455400000000000000000002000000c20e00000f000000/a''
children[1]:
type: ''disk''
id: 1
guid: 5437265705837840703
phys_path: ''/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h.fishstick%3Asd-andaman0001,1:a''
whole_disk: 1
DTL: 513
path: ''/dev/dsk/c3t44d0s0''
devid: ''id1,sd at
t49455400000000000000000002000000c91000000f000000/a''
children[5]:
type: ''mirror''
id: 5
guid: 6278268859805291262
whole_disk: 0
metaslab_array: 509
metaslab_shift: 30
ashift: 9
asize: 2000385474560
is_log: 0
children[0]:
type: ''disk''
id: 0
guid: 2444845073767152521
phys_path: ''/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h.fishstick%3Asd-andaman0001,5:a''
whole_disk: 1
DTL: 4188
path: ''/dev/dsk/c3t57d0s0''
devid: ''id1,sd at
t49455400000000000000000002000000a11000000f000000/a''
children[1]:
type: ''disk''
id: 1
guid: 7384627338137907176
phys_path: ''/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h%3Aoldfishstick%3Asd-andaman0001,8:a''
whole_disk: 1
DTL: 4192
path: ''/dev/dsk/c3t53d0s0''
devid: ''id1,sd at
t49455400000000000000000002000000990f00000f000000/a''
children[6]:
type: ''mirror''
id: 6
guid: 16251593073754009033
metaslab_array: 961
metaslab_shift: 31
ashift: 9
asize: 320059473920
is_log: 0
children[0]:
type: ''disk''
id: 0
guid: 2494043808582069183
phys_path: ''/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h.fishstick%3Asd-andaman0001,11:a''
whole_disk: 1
DTL: 5289
path: ''/dev/dsk/c3t54d0s0''
devid: ''id1,sd at
t49455400000000000000000002000000ef1000000f000000/a''
children[1]:
type: ''disk''
id: 1
guid: 12236206099593455135
phys_path: ''/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h%3Aoldfishstick%3Asd-andaman0001,6:a''
whole_disk: 1
DTL: 964
path: ''/dev/dsk/c3t51d0s0''
devid: ''id1,sd at
t494554000000000000000000020000004d0f00000f000000/a''
[long pause]
zdb: can''t open ''andaman'': I/O error
terabithia:/# format -e
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c1t0d0 <NETAPP-X231_SCHT5036F10-NA07 cyl 28837 alt 2 hd 4 sec
607> boot-a
/pci at 8,600000/SUNW,qlc at 4/fp at 0,0/ssd at w2100002037f86b53,0
1. c1t1d0 <NETAPP-X231_SCHT5036F10-NA07 cyl 28837 alt 2 hd 4 sec
607> boot-b
/pci at 8,600000/SUNW,qlc at 4/fp at 0,0/ssd at w2100002037f87b4c,0
2. c3t10d0 <IET-VIRTUAL-DISK-0-298.09GB>
/iscsi/disk at 0000iqn.2006-11.chaos.inner.th3h.fishstick%3Asd00001,0
3. c3t16d0 <IET-VIRTUAL-DISK-0-298.09GB>
/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h%3Aoldfishstick%3Asd30001,0
4. c3t43d0 <IET-VIRTUAL-DISK-0-931.51GB>
/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h.fishstick%3Asd-andaman0001,0
5. c3t44d0 <IET-VIRTUAL-DISK-0-698.64GB>
/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h.fishstick%3Asd-andaman0001,1
6. c3t45d0 <IET-VIRTUAL-DISK-0-931.51GB>
/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h.fishstick%3Asd-andaman0001,3
7. c3t46d0 <IET-VIRTUAL-DISK-0-233.76GB>
/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h.fishstick%3Asd-andaman0001,7
8. c3t47d0 <IET-VIRTUAL-DISK-0-931.51GB>
/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h%3Aoldfishstick%3Asd-andaman0001,0
9. c3t48d0 <IET-VIRTUAL-DISK-0-931.51GB>
/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h%3Aoldfishstick%3Asd-andaman0001,1
10. c3t49d0 <IET-VIRTUAL-DISK-0-232.89GB>
/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h%3Aoldfishstick%3Asd-andaman0001,2
11. c3t50d0 <IET-VIRTUAL-DISK-0-698.64GB>
/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h%3Aoldfishstick%3Asd-andaman0001,3
12. c3t51d0 <IET-VIRTUAL-DISK-0-298.09GB>
/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h%3Aoldfishstick%3Asd-andaman0001,6
13. c3t52d0 <IET-VIRTUAL-DISK-0-931.51GB>
/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h%3Aoldfishstick%3Asd-andaman0001,7
14. c3t53d0 <IET-VIRTUAL-DISK-0-1.82TB>
/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h%3Aoldfishstick%3Asd-andaman0001,8
15. c3t54d0 <IET-VIRTUAL-DISK-0-931.51GB>
/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h.fishstick%3Asd-andaman0001,11
16. c3t57d0 <IET-VIRTUAL-DISK-0-1.82TB>
/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h.fishstick%3Asd-andaman0001,5
17. c3t59d0 <IET-VIRTUAL-DISK-0-931.51GB>
/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h.fishstick%3Asd-andaman0001,10
Specify disk (enter its number): ^C
terabithia:/# exit
-----8<-----
Here is ''zdb -C'' before I exported the problem pool:
-----8<-----
aboveground:
version: 8
name: ''aboveground''
state: 0
txg: 27093860
pool_guid: 14919333615317928292
hostid: 2200768359
hostname: ''terabithia.th3h.inner.chaos''
vdev_children: 1
vdev_tree:
type: ''root''
id: 0
guid: 14919333615317928292
children[0]:
type: ''mirror''
id: 0
guid: 16297033706428251175
whole_disk: 0
metaslab_array: 14
metaslab_shift: 31
ashift: 9
asize: 320059473920
is_log: 0
children[0]:
type: ''disk''
id: 0
guid: 6677977776636178990
path: ''/dev/dsk/c3t10d0s0''
devid: ''id1,sd at
t49455400000000000000000001000000541100000f000000/a''
phys_path: ''/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h.fishstick%3Asd00001,0:a''
whole_disk: 1
DTL: 300
children[1]:
type: ''disk''
id: 1
guid: 8268759604762550338
path: ''/dev/dsk/c3t16d0s0''
devid: ''id1,sd at
t49455400000000000000000004000000680e00000f000000/a''
phys_path: ''/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h%3Aoldfishstick%3Asd30001,0:a''
whole_disk: 1
DTL: 19
andaman:
version: 22
name: ''andaman''
state: 0
txg: 27092168
pool_guid: 7400719929021713582
hostid: 2200768359
hostname: ''terabithia.th3h.inner.chaos''
vdev_children: 7
vdev_tree:
type: ''root''
id: 0
guid: 7400719929021713582
stats[0]: 1085420200
stats[1]: 4
stats[2]: 3
stats[3]: 0
stats[4]: 0
stats[5]: 0
stats[6]: 0
stats[7]: 0
stats[8]: 0
stats[9]: 0
stats[10]: 0
stats[11]: 0
stats[12]: 0
stats[13]: 0
stats[14]: 0
stats[15]: 0
stats[16]: 0
stats[17]: 0
stats[18]: 0
stats[19]: 0
stats[20]: 0
stats[21]: 0
stats[22]: 0
stats[23]: 0
stats[24]: 0
stats[25]: 0
stats[26]: 0
stats[27]: 0
stats[28]: 0
stats[29]: 0
children[0]:
type: ''mirror''
id: 0
guid: 337393226491877361
whole_disk: 0
metaslab_array: 14
metaslab_shift: 33
ashift: 9
asize: 1000191557632
is_log: 0
stats[0]: 1085422400
stats[1]: 4
stats[2]: 3
stats[3]: 0
stats[4]: 0
stats[5]: 0
stats[6]: 996432412672
stats[7]: 0
stats[8]: 0
stats[9]: 0
stats[10]: 0
stats[11]: 0
stats[12]: 0
stats[13]: 0
stats[14]: 0
stats[15]: 0
stats[16]: 0
stats[17]: 0
stats[18]: 0
stats[19]: 0
stats[20]: 0
stats[21]: 0
stats[22]: 0
stats[23]: 0
stats[24]: 0
stats[25]: 0
stats[26]: 0
stats[27]: 0
stats[28]: 0
stats[29]: 0
children[0]:
type: ''disk''
id: 0
guid: 1781150413433362160
path: ''/dev/dsk/c3t43d0s0''
devid: ''id1,sd at
t49455400000000000000000002000000521000000f000000/a''
phys_path: ''/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h.fishstick%3Asd-andaman0001,0:a''
whole_disk: 1
DTL: 91
stats[0]: 1085420200
stats[1]: 4
stats[2]: 1
stats[3]: 0
stats[4]: 0
stats[5]: 0
stats[6]: 996437131264
stats[7]: 0
stats[8]: 0
stats[9]: 0
stats[10]: 0
stats[11]: 0
stats[12]: 0
stats[13]: 0
stats[14]: 0
stats[15]: 0
stats[16]: 0
stats[17]: 0
stats[18]: 0
stats[19]: 0
stats[20]: 0
stats[21]: 0
stats[22]: 0
stats[23]: 0
stats[24]: 0
stats[25]: 0
stats[26]: 0
stats[27]: 0
stats[28]: 0
stats[29]: 0
children[1]:
type: ''disk''
id: 1
guid: 7841235598547702997
path: ''/dev/dsk/c3t48d0s0''
devid: ''id1,sd at
t494554000000000000000000020000009b0e00000f000000/a''
phys_path: ''/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h%3Aoldfishstick%3Asd-andaman0001,1:a''
whole_disk: 1
DTL: 215
stats[0]: 1085415000
stats[1]: 4
stats[2]: 1
stats[3]: 0
stats[4]: 0
stats[5]: 0
stats[6]: 996437131264
stats[7]: 0
stats[8]: 0
stats[9]: 0
stats[10]: 0
stats[11]: 0
stats[12]: 0
stats[13]: 0
stats[14]: 0
stats[15]: 0
stats[16]: 0
stats[17]: 0
stats[18]: 0
stats[19]: 0
stats[20]: 0
stats[21]: 0
stats[22]: 0
stats[23]: 0
stats[24]: 0
stats[25]: 0
stats[26]: 0
stats[27]: 0
stats[28]: 0
stats[29]: 0
children[1]:
type: ''mirror''
id: 1
guid: 1953060080997571723
whole_disk: 0
metaslab_array: 210
metaslab_shift: 33
ashift: 9
asize: 1000191557632
is_log: 0
stats[0]: 1085431600
stats[1]: 4
stats[2]: 3
stats[3]: 0
stats[4]: 0
stats[5]: 0
stats[6]: 996432412672
stats[7]: 0
stats[8]: 0
stats[9]: 0
stats[10]: 0
stats[11]: 0
stats[12]: 0
stats[13]: 0
stats[14]: 0
stats[15]: 0
stats[16]: 0
stats[17]: 0
stats[18]: 0
stats[19]: 0
stats[20]: 0
stats[21]: 0
stats[22]: 0
stats[23]: 0
stats[24]: 0
stats[25]: 0
stats[26]: 0
stats[27]: 0
stats[28]: 0
stats[29]: 0
children[0]:
type: ''disk''
id: 0
guid: 14567891505260370684
path: ''/dev/dsk/c3t45d0s0''
devid: ''id1,sd at
t49455400000000000000000002000000601000000f000000/a''
phys_path: ''/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h.fishstick%3Asd-andaman0001,3:a''
whole_disk: 1
DTL: 1395
stats[0]: 1085428200
stats[1]: 4
stats[2]: 1
stats[3]: 0
stats[4]: 0
stats[5]: 0
stats[6]: 996437131264
stats[7]: 0
stats[8]: 0
stats[9]: 0
stats[10]: 0
stats[11]: 0
stats[12]: 0
stats[13]: 0
stats[14]: 0
stats[15]: 0
stats[16]: 0
stats[17]: 0
stats[18]: 0
stats[19]: 0
stats[20]: 0
stats[21]: 0
stats[22]: 0
stats[23]: 0
stats[24]: 0
stats[25]: 0
stats[26]: 0
stats[27]: 0
stats[28]: 0
stats[29]: 0
children[1]:
type: ''disk''
id: 1
guid: 12913430974955856922
path: ''/dev/dsk/c3t47d0s0''
devid: ''id1,sd at
t49455400000000000000000002000000960f00000f000000/a''
phys_path: ''/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h%3Aoldfishstick%3Asd-andaman0001,0:a''
whole_disk: 1
DTL: 217
stats[0]: 1085270000
stats[1]: 4
stats[2]: 1
stats[3]: 0
stats[4]: 0
stats[5]: 0
stats[6]: 996437131264
stats[7]: 0
stats[8]: 0
stats[9]: 0
stats[10]: 0
stats[11]: 0
stats[12]: 0
stats[13]: 0
stats[14]: 0
stats[15]: 0
stats[16]: 0
stats[17]: 0
stats[18]: 0
stats[19]: 0
stats[20]: 0
stats[21]: 0
stats[22]: 0
stats[23]: 0
stats[24]: 0
stats[25]: 0
stats[26]: 0
stats[27]: 0
stats[28]: 0
stats[29]: 0
children[2]:
type: ''mirror''
id: 2
guid: 17555572171866652488
metaslab_array: 330
metaslab_shift: 31
ashift: 9
asize: 1000191557632
is_log: 0
stats[0]: 1085290200
stats[1]: 4
stats[2]: 3
stats[3]: 0
stats[4]: 0
stats[5]: 0
stats[6]: 998579896320
stats[7]: 0
stats[8]: 0
stats[9]: 0
stats[10]: 0
stats[11]: 0
stats[12]: 0
stats[13]: 0
stats[14]: 0
stats[15]: 0
stats[16]: 0
stats[17]: 0
stats[18]: 0
stats[19]: 0
stats[20]: 0
stats[21]: 0
stats[22]: 0
stats[23]: 0
stats[24]: 0
stats[25]: 0
stats[26]: 0
stats[27]: 0
stats[28]: 0
stats[29]: 0
children[0]:
type: ''disk''
id: 0
guid: 10644297178963807622
path: ''/dev/dsk/c3t52d0s0''
devid: ''id1,sd at
t49455400000000000000000002000000310f00000f000000/a''
phys_path: ''/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h%3Aoldfishstick%3Asd-andaman0001,7:a''
whole_disk: 1
DTL: 1167
stats[0]: 1085289600
stats[1]: 4
stats[2]: 1
stats[3]: 0
stats[4]: 0
stats[5]: 0
stats[6]: 998584614912
stats[7]: 0
stats[8]: 0
stats[9]: 0
stats[10]: 0
stats[11]: 0
stats[12]: 0
stats[13]: 0
stats[14]: 0
stats[15]: 0
stats[16]: 0
stats[17]: 0
stats[18]: 0
stats[19]: 0
stats[20]: 0
stats[21]: 0
stats[22]: 0
stats[23]: 0
stats[24]: 0
stats[25]: 0
stats[26]: 0
stats[27]: 0
stats[28]: 0
stats[29]: 0
children[1]:
type: ''disk''
id: 1
guid: 2378339860349039724
path: ''/dev/dsk/c3t59d0s0''
devid: ''id1,sd at
t49455400000000000000000002000000a41000000f000000/a''
phys_path: ''/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h.fishstick%3Asd-andaman0001,10:a''
whole_disk: 1
DTL: 1168
stats[0]: 1085287600
stats[1]: 4
stats[2]: 1
stats[3]: 0
stats[4]: 0
stats[5]: 0
stats[6]: 998584614912
stats[7]: 0
stats[8]: 0
stats[9]: 0
stats[10]: 0
stats[11]: 0
stats[12]: 0
stats[13]: 0
stats[14]: 0
stats[15]: 0
stats[16]: 0
stats[17]: 0
stats[18]: 0
stats[19]: 0
stats[20]: 0
stats[21]: 0
stats[22]: 0
stats[23]: 0
stats[24]: 0
stats[25]: 0
stats[26]: 0
stats[27]: 0
stats[28]: 0
stats[29]: 0
children[3]:
type: ''mirror''
id: 3
guid: 5261242283366420554
metaslab_array: 332
metaslab_shift: 31
ashift: 9
asize: 250046054400
is_log: 0
stats[0]: 1085311200
stats[1]: 4
stats[2]: 3
stats[3]: 0
stats[4]: 0
stats[5]: 0
stats[6]: 249108103168
stats[7]: 0
stats[8]: 0
stats[9]: 0
stats[10]: 0
stats[11]: 0
stats[12]: 0
stats[13]: 0
stats[14]: 0
stats[15]: 0
stats[16]: 0
stats[17]: 0
stats[18]: 0
stats[19]: 0
stats[20]: 0
stats[21]: 0
stats[22]: 0
stats[23]: 0
stats[24]: 0
stats[25]: 0
stats[26]: 0
stats[27]: 0
stats[28]: 0
stats[29]: 0
children[0]:
type: ''disk''
id: 0
guid: 17534037132307391717
path: ''/dev/dsk/c3t46d0s0''
devid: ''id1,sd at
t494554000000000000000000020000003a1000000f000000/a''
phys_path: ''/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h.fishstick%3Asd-andaman0001,7:a''
whole_disk: 1
DTL: 335
stats[0]: 1085276000
stats[1]: 4
stats[2]: 1
stats[3]: 0
stats[4]: 0
stats[5]: 0
stats[6]: 249112821760
stats[7]: 0
stats[8]: 0
stats[9]: 0
stats[10]: 0
stats[11]: 0
stats[12]: 0
stats[13]: 0
stats[14]: 0
stats[15]: 0
stats[16]: 0
stats[17]: 0
stats[18]: 0
stats[19]: 0
stats[20]: 0
stats[21]: 0
stats[22]: 0
stats[23]: 0
stats[24]: 0
stats[25]: 0
stats[26]: 0
stats[27]: 0
stats[28]: 0
stats[29]: 0
children[1]:
type: ''disk''
id: 1
guid: 2758124488221280115
path: ''/dev/dsk/c3t49d0s0''
devid: ''id1,sd at
t49455400000000000000000002000000b00e00000f000000/a''
phys_path: ''/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h%3Aoldfishstick%3Asd-andaman0001,2:a''
whole_disk: 1
DTL: 336
stats[0]: 1085275800
stats[1]: 4
stats[2]: 1
stats[3]: 0
stats[4]: 0
stats[5]: 0
stats[6]: 249112821760
stats[7]: 0
stats[8]: 0
stats[9]: 0
stats[10]: 0
stats[11]: 0
stats[12]: 0
stats[13]: 0
stats[14]: 0
stats[15]: 0
stats[16]: 0
stats[17]: 0
stats[18]: 0
stats[19]: 0
stats[20]: 0
stats[21]: 0
stats[22]: 0
stats[23]: 0
stats[24]: 0
stats[25]: 0
stats[26]: 0
stats[27]: 0
stats[28]: 0
stats[29]: 0
children[4]:
type: ''mirror''
id: 4
guid: 2126473073638606315
whole_disk: 0
metaslab_array: 510
metaslab_shift: 31
ashift: 9
asize: 750142881792
is_log: 0
stats[0]: 1085300800
stats[1]: 4
stats[2]: 3
stats[3]: 0
stats[4]: 0
stats[5]: 0
stats[6]: 749471793152
stats[7]: 0
stats[8]: 0
stats[9]: 0
stats[10]: 0
stats[11]: 0
stats[12]: 0
stats[13]: 0
stats[14]: 0
stats[15]: 0
stats[16]: 0
stats[17]: 0
stats[18]: 0
stats[19]: 0
stats[20]: 0
stats[21]: 0
stats[22]: 0
stats[23]: 0
stats[24]: 0
stats[25]: 0
stats[26]: 0
stats[27]: 0
stats[28]: 0
stats[29]: 0
children[0]:
type: ''disk''
id: 0
guid: 7272191256105073467
path: ''/dev/dsk/c3t50d0s0''
devid: ''id1,sd at
t494554000000000000000000020000000b0f00000f000000/a''
phys_path: ''/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h%3Aoldfishstick%3Asd-andaman0001,3:a''
whole_disk: 1
DTL: 660
stats[0]: 1085299800
stats[1]: 4
stats[2]: 1
stats[3]: 0
stats[4]: 0
stats[5]: 0
stats[6]: 749476511744
stats[7]: 0
stats[8]: 0
stats[9]: 0
stats[10]: 0
stats[11]: 0
stats[12]: 0
stats[13]: 0
stats[14]: 0
stats[15]: 0
stats[16]: 0
stats[17]: 0
stats[18]: 0
stats[19]: 0
stats[20]: 0
stats[21]: 0
stats[22]: 0
stats[23]: 0
stats[24]: 0
stats[25]: 0
stats[26]: 0
stats[27]: 0
stats[28]: 0
stats[29]: 0
children[1]:
type: ''disk''
id: 1
guid: 5437265705837840703
path: ''/dev/dsk/c3t44d0s0''
devid: ''id1,sd at
t494554000000000000000000020000007f1000000f000000/a''
phys_path: ''/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h.fishstick%3Asd-andaman0001,1:a''
whole_disk: 1
DTL: 513
stats[0]: 1085299800
stats[1]: 4
stats[2]: 1
stats[3]: 0
stats[4]: 0
stats[5]: 0
stats[6]: 749476511744
stats[7]: 0
stats[8]: 0
stats[9]: 0
stats[10]: 0
stats[11]: 0
stats[12]: 0
stats[13]: 0
stats[14]: 0
stats[15]: 0
stats[16]: 0
stats[17]: 0
stats[18]: 0
stats[19]: 0
stats[20]: 0
stats[21]: 0
stats[22]: 0
stats[23]: 0
stats[24]: 0
stats[25]: 0
stats[26]: 0
stats[27]: 0
stats[28]: 0
stats[29]: 0
children[5]:
type: ''mirror''
id: 5
guid: 6278268859805291262
whole_disk: 0
metaslab_array: 509
metaslab_shift: 30
ashift: 9
asize: 2000385474560
is_log: 0
stats[0]: 1085291400
stats[1]: 4
stats[2]: 3
stats[3]: 0
stats[4]: 0
stats[5]: 0
stats[6]: 2000381018112
stats[7]: 0
stats[8]: 0
stats[9]: 0
stats[10]: 0
stats[11]: 0
stats[12]: 0
stats[13]: 0
stats[14]: 0
stats[15]: 0
stats[16]: 0
stats[17]: 0
stats[18]: 0
stats[19]: 0
stats[20]: 0
stats[21]: 0
stats[22]: 0
stats[23]: 0
stats[24]: 0
stats[25]: 0
stats[26]: 0
stats[27]: 0
stats[28]: 0
stats[29]: 0
children[0]:
type: ''disk''
id: 0
guid: 2444845073767152521
path: ''/dev/dsk/c3t57d0s0''
devid: ''id1,sd at
t49455400000000000000000002000000211000000f000000/a''
phys_path: ''/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h.fishstick%3Asd-andaman0001,5:a''
whole_disk: 1
DTL: 4188
stats[0]: 1085290200
stats[1]: 4
stats[2]: 1
stats[3]: 0
stats[4]: 0
stats[5]: 0
stats[6]: 2000385736704
stats[7]: 0
stats[8]: 0
stats[9]: 0
stats[10]: 0
stats[11]: 0
stats[12]: 0
stats[13]: 0
stats[14]: 0
stats[15]: 0
stats[16]: 0
stats[17]: 0
stats[18]: 0
stats[19]: 0
stats[20]: 0
stats[21]: 0
stats[22]: 0
stats[23]: 0
stats[24]: 0
stats[25]: 0
stats[26]: 0
stats[27]: 0
stats[28]: 0
stats[29]: 0
children[1]:
type: ''disk''
id: 1
guid: 7384627338137907176
path: ''/dev/dsk/c3t53d0s0''
devid: ''id1,sd at
t49455400000000000000000002000000b80f00000f000000/a''
phys_path: ''/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h%3Aoldfishstick%3Asd-andaman0001,8:a''
whole_disk: 1
DTL: 4192
stats[0]: 1085289400
stats[1]: 4
stats[2]: 1
stats[3]: 0
stats[4]: 0
stats[5]: 0
stats[6]: 2000385736704
stats[7]: 0
stats[8]: 0
stats[9]: 0
stats[10]: 0
stats[11]: 0
stats[12]: 0
stats[13]: 0
stats[14]: 0
stats[15]: 0
stats[16]: 0
stats[17]: 0
stats[18]: 0
stats[19]: 0
stats[20]: 0
stats[21]: 0
stats[22]: 0
stats[23]: 0
stats[24]: 0
stats[25]: 0
stats[26]: 0
stats[27]: 0
stats[28]: 0
stats[29]: 0
children[6]:
type: ''mirror''
id: 6
guid: 16251593073754009033
metaslab_array: 961
metaslab_shift: 31
ashift: 9
asize: 320059473920
is_log: 0
stats[0]: 1085390000
stats[1]: 4
stats[2]: 3
stats[3]: 0
stats[4]: 0
stats[5]: 0
stats[6]: 319975063552
stats[7]: 0
stats[8]: 0
stats[9]: 0
stats[10]: 0
stats[11]: 0
stats[12]: 0
stats[13]: 0
stats[14]: 0
stats[15]: 0
stats[16]: 0
stats[17]: 0
stats[18]: 0
stats[19]: 0
stats[20]: 0
stats[21]: 0
stats[22]: 0
stats[23]: 0
stats[24]: 0
stats[25]: 0
stats[26]: 0
stats[27]: 0
stats[28]: 0
stats[29]: 0
children[0]:
type: ''disk''
id: 0
guid: 2494043808582069183
path: ''/dev/dsk/c3t54d0s0''
devid: ''id1,sd at
t49455400000000000000000002000000271100000f000000/a''
phys_path: ''/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h.fishstick%3Asd-andaman0001,11:a''
whole_disk: 1
DTL: 5289
stats[0]: 1085390800
stats[1]: 4
stats[2]: 1
stats[3]: 0
stats[4]: 0
stats[5]: 0
stats[6]: 319979782144
stats[7]: 0
stats[8]: 0
stats[9]: 0
stats[10]: 0
stats[11]: 0
stats[12]: 0
stats[13]: 0
stats[14]: 0
stats[15]: 0
stats[16]: 0
stats[17]: 0
stats[18]: 0
stats[19]: 0
stats[20]: 0
stats[21]: 0
stats[22]: 0
stats[23]: 0
stats[24]: 0
stats[25]: 0
stats[26]: 0
stats[27]: 0
stats[28]: 0
stats[29]: 0
children[1]:
type: ''disk''
id: 1
guid: 12236206099593455135
path: ''/dev/dsk/c3t51d0s0''
devid: ''id1,sd at
t49455400000000000000000002000000750f00000f000000/a''
phys_path: ''/iscsi/disk at
0000iqn.2006-11.chaos.inner.th3h%3Aoldfishstick%3Asd-andaman0001,6:a''
whole_disk: 1
DTL: 964
stats[0]: 1085358200
stats[1]: 4
stats[2]: 1
stats[3]: 0
stats[4]: 0
stats[5]: 0
stats[6]: 319979782144
stats[7]: 0
stats[8]: 0
stats[9]: 0
stats[10]: 0
stats[11]: 0
stats[12]: 0
stats[13]: 0
stats[14]: 0
stats[15]: 0
stats[16]: 0
stats[17]: 0
stats[18]: 0
stats[19]: 0
stats[20]: 0
stats[21]: 0
stats[22]: 0
stats[23]: 0
stats[24]: 0
stats[25]: 0
stats[26]: 0
stats[27]: 0
stats[28]: 0
stats[29]: 0
rpool:
version: 22
name: ''rpool''
state: 0
txg: 18614338
pool_guid: 16346529704285805676
hostid: 2200768359
hostname: ''''
vdev_children: 1
vdev_tree:
type: ''root''
id: 0
guid: 16346529704285805676
create_txg: 4
children[0]:
type: ''mirror''
id: 0
guid: 15963611323458091535
whole_disk: 0
metaslab_array: 23
metaslab_shift: 28
ashift: 9
asize: 35843473408
is_log: 0
create_txg: 4
children[0]:
type: ''disk''
id: 0
guid: 10495950740462619950
path: ''/dev/dsk/c1t0d0s0''
devid: ''id1,ssd at n2000002037f86b53/a''
phys_path: ''/pci at 8,600000/SUNW,qlc at 4/fp at
0,0/ssd at w2100002037f86b53,0:a''
whole_disk: 0
DTL: 102
create_txg: 4
children[1]:
type: ''disk''
id: 1
guid: 12455868191384207452
path: ''/dev/dsk/c1t1d0s0''
devid: ''id1,ssd at n2000002037f87b4c/a''
phys_path: ''/pci at 8,600000/SUNW,qlc at 4/fp at
0,0/ssd at w2100002037f87b4c,0:a''
whole_disk: 0
DTL: 180
create_txg: 4
-----8<-----
I have a coredump from the first panic, but for some reason dumpadm is
not writing ''unix.5'' for me. I don''t know how to get
a backtrace with
''mdb -k''---any advice?
-----8<-----
terabithia:/# ls -l /var/crash/terabithia.th3h.inner.chaos/
total 5854958
-rw-r--r-- 1 root root 2 Apr 4 02:28 bounds
-rw-r--r-- 1 root root 686161920 Aug 1 2010 vmdump.0
-rw-r--r-- 1 root root 684130304 Aug 8 2010 vmdump.1
-rw-r--r-- 1 root root 684261376 Aug 22 2010 vmdump.2
-rw-r--r-- 1 root root 680525824 Sep 5 2010 vmdump.3
-rw-r--r-- 1 root root 178847744 Oct 31 12:39 vmdump.4
-rw-r--r-- 1 root root 132055040 Apr 4 02:28 vmdump.5
-----8<-----
Are there things I might try to recover the data? Backups exist, but
are several years old.
I can buy and house enough drives to make a copy of the pool, if
needed. I thought I might try ''zpool import -V andaman'' from
solaris
express 11 or openindiana b148, but I wonder if there''s something
smarter I should be doing.
--
00
(--)
( || )
^^~~^^
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 304 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110404/8056a5ee/attachment-0001.bin>
Miles Nordin
2011-Apr-05 21:29 UTC
[zfs-discuss] ''cannot import ''andaman'': I/O error'', and failure to follow my own advice
>>>>> "c" == Miles Nordin <carton at Ivy.NET> writes:c> terabithia:/# zpool import andaman c> cannot import ''andaman'': I/O error c> Destroy and re-create the pool from c> a backup source. snv_151, the proprietary release, was able to fix this. I didn''t try oi_148 first, so there''s a chance it would''ve worked too if I''d given it a chance. root at solaris:~# zpool import -n -F -f 7400719929021713582 Would be able to return andaman to its state as of April 3, 2011 03:53:23 PM PDT. Would discard approximately 31 seconds of transactions. root at solaris:~# zpool import -F -f 7400719929021713582 Pool andaman returned to its state as of April 3, 2011 03:53:23 PM PDT. Discarded approximately 31 seconds of transactions. so, ftr, seems not all ''import -F'' are created equal. :) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 304 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110405/bc767a53/attachment.bin>