I have searched the forums and google wide, but cannot find a fix for the issue
I''m currently experiencing. Long story short - I''m now at a
point where I cannot even import my zpool (zpool import -f tank) without causing
a kernel panic
I''m running OpenSolaris snv_111b and the zpool is version 14.
This is the panic from /var/adm/messages; (full output attached);
genunix: [ID 361072 kern.notice] zfs: freeing free segment (offset=3540185931776
size=22528)
This is the output I get from zpool import;
# zpool import
pool: tank
id: 15136317365944618902
state: ONLINE
status: The pool was last accessed by another system.
action: The pool can be imported using its name or numeric identifier and
the ''-f'' flag.
see: http://www.sun.com/msg/ZFS-8000-EY
config:
tank ONLINE
raidz1 ONLINE
c9t4d0 ONLINE
c9t5d0 ONLINE
c9t6d0 ONLINE
c9t7d0 ONLINE
raidz1 ONLINE
c9t0d0 ONLINE
c9t1d0 ONLINE
c9t2d0 ONLINE
c9t3d0 ONLINE
I tried pulling back some info via this zdb command, but i''m not sure
if i''m on the right track here (as zpool import seems to see the zpool
without issue). This result is similar from all drives;
# zdb -l /dev/dsk/c9t4d0
--------------------------------------------
LABEL 0
--------------------------------------------
failed to unpack label 0
--------------------------------------------
LABEL 1
--------------------------------------------
failed to unpack label 1
--------------------------------------------
LABEL 2
--------------------------------------------
failed to unpack label 2
--------------------------------------------
LABEL 3
--------------------------------------------
failed to unpack label 3
I also can complete zdb -e tank without issues ? it lists all my snapshots and
various objects without problem (this is still running on the machine at the
moment)
I have put the following into /etc/system;
set zfs:zfs_recover=1
set aok=1
i''ve also tried mounting the zpool read only with zpool import -f -o ro
tank but no luck..
I dont know where to go next? ? am I meant to try and recover using an older
txg? Errrr.
I would be extremely grateful to anyone who can offer advice on how to resolve
this issue as the pool contains irreplaceable photos. Unfortunately I have not
done any backups for a while as I thought raidz would be my savour. :(
please help
--
This message posted from opensolaris.org
Darren Taylor wrote:> I have searched the forums and google wide, but cannot find a fix for the issue I''m currently experiencing. Long story short - I''m now at a point where I cannot even import my zpool (zpool import -f tank) without causing a kernel panic > > I''m running OpenSolaris snv_111b and the zpool is version 14. > > This is the panic from /var/adm/messages; (full output attached); > > genunix: [ID 361072 kern.notice] zfs: freeing free segment (offset=3540185931776 size=22528) > >Have you tried importing to a system running a more recent build? The problem may have been fixed... -- Ian.
Hi Ian, I''m currently downloading build 124 to see if that helps... the download is running a bit slow so wont know until later tomorrow. Just an update that i have also tried; (forgot to mention above) * Pulling out each disk - tried mounting in degraded state - same kernel panic * Deleting the zpool.cache Fingers crossed i get something different with the newer build. Very strange, as i don''t think this was a hardware issue? -- as all the drives appear to be working without issue and "zpool import" list all drives as ONLINE without any information pointing to corruption. -- This message posted from opensolaris.org
On 11.10.09 12:59, Darren Taylor wrote:> I have searched the forums and google wide, but cannot find a fix for the issue I''m currently experiencing. Long story short - I''m now at a point where I cannot even import my zpool (zpool import -f tank) without causing a kernel panic > > I''m running OpenSolaris snv_111b and the zpool is version 14. > > This is the panic from /var/adm/messages; (full output attached);Where is full stack back trace? I do not see any attachment. victor> > genunix: [ID 361072 kern.notice] zfs: freeing free segment (offset=3540185931776 size=22528) > > This is the output I get from zpool import; > > # zpool import > pool: tank > id: 15136317365944618902 > state: ONLINE > status: The pool was last accessed by another system. > action: The pool can be imported using its name or numeric identifier and > the ''-f'' flag. > see: http://www.sun.com/msg/ZFS-8000-EY > config: > > tank ONLINE > raidz1 ONLINE > c9t4d0 ONLINE > c9t5d0 ONLINE > c9t6d0 ONLINE > c9t7d0 ONLINE > raidz1 ONLINE > c9t0d0 ONLINE > c9t1d0 ONLINE > c9t2d0 ONLINE > c9t3d0 ONLINE > > I tried pulling back some info via this zdb command, but i''m not sure if i''m on the right track here (as zpool import seems to see the zpool without issue). This result is similar from all drives; > > # zdb -l /dev/dsk/c9t4d0 > -------------------------------------------- > LABEL 0 > -------------------------------------------- > failed to unpack label 0 > -------------------------------------------- > LABEL 1 > -------------------------------------------- > failed to unpack label 1 > -------------------------------------------- > LABEL 2 > -------------------------------------------- > failed to unpack label 2 > -------------------------------------------- > LABEL 3 > -------------------------------------------- > failed to unpack label 3 > > I also can complete zdb -e tank without issues ? it lists all my snapshots and various objects without problem (this is still running on the machine at the moment) > > I have put the following into /etc/system; > > set zfs:zfs_recover=1 > set aok=1 > > i''ve also tried mounting the zpool read only with zpool import -f -o ro tank but no luck.. > > I dont know where to go next? ? am I meant to try and recover using an older txg? Errrr. > > I would be extremely grateful to anyone who can offer advice on how to resolve this issue as the pool contains irreplaceable photos. Unfortunately I have not done any backups for a while as I thought raidz would be my savour. :( > > please help
Hi Victor, i have tried to re-attach the detail from /var/adm/messages -- This message posted from opensolaris.org -------------- next part -------------- Oct 11 17:16:55 opensolaris unix: [ID 836849 kern.notice] Oct 11 17:16:55 opensolaris ^Mpanic[cpu0]/thread=ffffff000b6f7c60: Oct 11 17:16:55 opensolaris genunix: [ID 361072 kern.notice] zfs: freeing free segment (offset=3540185931776 size=22528) Oct 11 17:16:55 opensolaris unix: [ID 100000 kern.notice] Oct 11 17:16:55 opensolaris genunix: [ID 655072 kern.notice] ffffff000b6f75f0 genunix:vcmn_err+2c () Oct 11 17:16:55 opensolaris genunix: [ID 655072 kern.notice] ffffff000b6f76e0 zfs:zfs_panic_recover+ae () Oct 11 17:16:55 opensolaris genunix: [ID 655072 kern.notice] ffffff000b6f7770 zfs:space_map_remove+13c () Oct 11 17:16:55 opensolaris genunix: [ID 655072 kern.notice] ffffff000b6f7820 zfs:space_map_load+260 () Oct 11 17:16:55 opensolaris genunix: [ID 655072 kern.notice] ffffff000b6f7860 zfs:metaslab_activate+64 () Oct 11 17:16:55 opensolaris genunix: [ID 655072 kern.notice] ffffff000b6f7920 zfs:metaslab_group_alloc+2b7 () Oct 11 17:16:55 opensolaris genunix: [ID 655072 kern.notice] ffffff000b6f7a00 zfs:metaslab_alloc_dva+295 () Oct 11 17:16:55 opensolaris genunix: [ID 655072 kern.notice] ffffff000b6f7aa0 zfs:metaslab_alloc+9b () Oct 11 17:16:55 opensolaris genunix: [ID 655072 kern.notice] ffffff000b6f7ad0 zfs:zio_dva_allocate+3e () Oct 11 17:16:55 opensolaris genunix: [ID 655072 kern.notice] ffffff000b6f7b00 zfs:zio_execute+a0 () Oct 11 17:16:55 opensolaris genunix: [ID 655072 kern.notice] ffffff000b6f7b60 zfs:zio_notify_parent+a6 () Oct 11 17:16:55 opensolaris genunix: [ID 655072 kern.notice] ffffff000b6f7b90 zfs:zio_ready+188 () Oct 11 17:16:55 opensolaris genunix: [ID 655072 kern.notice] ffffff000b6f7bc0 zfs:zio_execute+a0 () Oct 11 17:16:55 opensolaris genunix: [ID 655072 kern.notice] ffffff000b6f7c40 genunix:taskq_thread+193 () Oct 11 17:16:55 opensolaris genunix: [ID 655072 kern.notice] ffffff000b6f7c50 unix:thread_start+8 () Oct 11 17:16:55 opensolaris unix: [ID 100000 kern.notice] Oct 11 17:16:55 opensolaris genunix: [ID 672855 kern.notice] syncing file systems... Oct 11 17:16:55 opensolaris genunix: [ID 904073 kern.notice] done Oct 11 17:16:56 opensolaris genunix: [ID 111219 kern.notice] dumping to /dev/zvol/dsk/rpool/dump, offset 65536, content: kernel Oct 11 17:17:09 opensolaris genunix: [ID 409368 kern.notice] ^M100% done: 168706 pages dumped, compression ratio 3.58, Oct 11 17:17:09 opensolaris genunix: [ID 851671 kern.notice] dump succeeded
i have re run zdb -l /dev/dsk/c9t4d0s0 as i should have the first time (thanks
Nicolas).
Attached output.
--
This message posted from opensolaris.org
-------------- next part --------------
# zdb -l /dev/dsk/c9t4d0s0
--------------------------------------------
LABEL 0
--------------------------------------------
version=14
name=''tank''
state=0
txg=119170
pool_guid=15136317365944618902
hostid=290968
hostname=''lexx''
top_guid=1561201926038510280
guid=11292568128772689834
vdev_tree
type=''raidz''
id=0
guid=1561201926038510280
nparity=1
metaslab_array=23
metaslab_shift=35
ashift=9
asize=4000766230528
is_log=0
children[0]
type=''disk''
id=0
guid=11292568128772689834
path=''/dev/dsk/c9t4d0s0''
devid=''id1,sd at n50014ee2588170a5/a''
phys_path=''/pci at 0,0/pci1022,9603 at 2/pci15d9,a580
at 0/sd at 4,0:a''
whole_disk=1
children[1]
type=''disk''
id=1
guid=10678319508898151547
path=''/dev/dsk/c9t5d0s0''
devid=''id1,sd at n50014ee2032b9b04/a''
phys_path=''/pci at 0,0/pci1022,9603 at 2/pci15d9,a580
at 0/sd at 5,0:a''
whole_disk=1
children[2]
type=''disk''
id=2
guid=16523383997370950474
path=''/dev/dsk/c9t6d0s0''
devid=''id1,sd at n50014ee2032b9b75/a''
phys_path=''/pci at 0,0/pci1022,9603 at 2/pci15d9,a580
at 0/sd at 6,0:a''
whole_disk=1
children[3]
type=''disk''
id=3
guid=1710422830365926220
path=''/dev/dsk/c9t7d0s0''
devid=''id1,sd at n50014ee2add68f2c/a''
phys_path=''/pci at 0,0/pci1022,9603 at 2/pci15d9,a580
at 0/sd at 7,0:a''
whole_disk=1
--------------------------------------------
LABEL 1
--------------------------------------------
version=14
name=''tank''
state=0
txg=119170
pool_guid=15136317365944618902
hostid=290968
hostname=''lexx''
top_guid=1561201926038510280
guid=11292568128772689834
vdev_tree
type=''raidz''
id=0
guid=1561201926038510280
nparity=1
metaslab_array=23
metaslab_shift=35
ashift=9
asize=4000766230528
is_log=0
children[0]
type=''disk''
id=0
guid=11292568128772689834
path=''/dev/dsk/c9t4d0s0''
devid=''id1,sd at n50014ee2588170a5/a''
phys_path=''/pci at 0,0/pci1022,9603 at 2/pci15d9,a580
at 0/sd at 4,0:a''
whole_disk=1
children[1]
type=''disk''
id=1
guid=10678319508898151547
path=''/dev/dsk/c9t5d0s0''
devid=''id1,sd at n50014ee2032b9b04/a''
phys_path=''/pci at 0,0/pci1022,9603 at 2/pci15d9,a580
at 0/sd at 5,0:a''
whole_disk=1
children[2]
type=''disk''
id=2
guid=16523383997370950474
path=''/dev/dsk/c9t6d0s0''
devid=''id1,sd at n50014ee2032b9b75/a''
phys_path=''/pci at 0,0/pci1022,9603 at 2/pci15d9,a580
at 0/sd at 6,0:a''
whole_disk=1
children[3]
type=''disk''
id=3
guid=1710422830365926220
path=''/dev/dsk/c9t7d0s0''
devid=''id1,sd at n50014ee2add68f2c/a''
phys_path=''/pci at 0,0/pci1022,9603 at 2/pci15d9,a580
at 0/sd at 7,0:a''
whole_disk=1
--------------------------------------------
LABEL 2
--------------------------------------------
version=14
name=''tank''
state=0
txg=119170
pool_guid=15136317365944618902
hostid=290968
hostname=''lexx''
top_guid=1561201926038510280
guid=11292568128772689834
vdev_tree
type=''raidz''
id=0
guid=1561201926038510280
nparity=1
metaslab_array=23
metaslab_shift=35
ashift=9
asize=4000766230528
is_log=0
children[0]
type=''disk''
id=0
guid=11292568128772689834
path=''/dev/dsk/c9t4d0s0''
devid=''id1,sd at n50014ee2588170a5/a''
phys_path=''/pci at 0,0/pci1022,9603 at 2/pci15d9,a580
at 0/sd at 4,0:a''
whole_disk=1
children[1]
type=''disk''
id=1
guid=10678319508898151547
path=''/dev/dsk/c9t5d0s0''
devid=''id1,sd at n50014ee2032b9b04/a''
phys_path=''/pci at 0,0/pci1022,9603 at 2/pci15d9,a580
at 0/sd at 5,0:a''
whole_disk=1
children[2]
type=''disk''
id=2
guid=16523383997370950474
path=''/dev/dsk/c9t6d0s0''
devid=''id1,sd at n50014ee2032b9b75/a''
phys_path=''/pci at 0,0/pci1022,9603 at 2/pci15d9,a580
at 0/sd at 6,0:a''
whole_disk=1
children[3]
type=''disk''
id=3
guid=1710422830365926220
path=''/dev/dsk/c9t7d0s0''
devid=''id1,sd at n50014ee2add68f2c/a''
phys_path=''/pci at 0,0/pci1022,9603 at 2/pci15d9,a580
at 0/sd at 7,0:a''
whole_disk=1
--------------------------------------------
LABEL 3
--------------------------------------------
version=14
name=''tank''
state=0
txg=119170
pool_guid=15136317365944618902
hostid=290968
hostname=''lexx''
top_guid=1561201926038510280
guid=11292568128772689834
vdev_tree
type=''raidz''
id=0
guid=1561201926038510280
nparity=1
metaslab_array=23
metaslab_shift=35
ashift=9
asize=4000766230528
is_log=0
children[0]
type=''disk''
id=0
guid=11292568128772689834
path=''/dev/dsk/c9t4d0s0''
devid=''id1,sd at n50014ee2588170a5/a''
phys_path=''/pci at 0,0/pci1022,9603 at 2/pci15d9,a580
at 0/sd at 4,0:a''
whole_disk=1
children[1]
type=''disk''
id=1
guid=10678319508898151547
path=''/dev/dsk/c9t5d0s0''
devid=''id1,sd at n50014ee2032b9b04/a''
phys_path=''/pci at 0,0/pci1022,9603 at 2/pci15d9,a580
at 0/sd at 5,0:a''
whole_disk=1
children[2]
type=''disk''
id=2
guid=16523383997370950474
path=''/dev/dsk/c9t6d0s0''
devid=''id1,sd at n50014ee2032b9b75/a''
phys_path=''/pci at 0,0/pci1022,9603 at 2/pci15d9,a580
at 0/sd at 6,0:a''
whole_disk=1
children[3]
type=''disk''
id=3
guid=1710422830365926220
path=''/dev/dsk/c9t7d0s0''
devid=''id1,sd at n50014ee2add68f2c/a''
phys_path=''/pci at 0,0/pci1022,9603 at 2/pci15d9,a580
at 0/sd at 7,0:a''
whole_disk=1
# zdb -l /dev/dsk/c9t0d0s0
--------------------------------------------
LABEL 0
--------------------------------------------
version=14
name=''tank''
state=0
txg=118998
pool_guid=15136317365944618902
hostid=290968
hostname=''lexx''
top_guid=17530628911224651473
guid=11914115076705860870
vdev_tree
type=''raidz''
id=1
guid=17530628911224651473
nparity=1
metaslab_array=322
metaslab_shift=34
ashift=9
asize=3000567332864
is_log=0
children[0]
type=''disk''
id=0
guid=11914115076705860870
path=''/dev/dsk/c9t0d0s0''
devid=''id1,sd at n50000f000b039399/a''
phys_path=''/pci at 0,0/pci1022,9603 at 2/pci15d9,a580
at 0/sd at 0,0:a''
whole_disk=1
children[1]
type=''disk''
id=1
guid=11267830253346467192
path=''/dev/dsk/c9t1d0s0''
devid=''id1,sd at n5000c5000262bfd7/a''
phys_path=''/pci at 0,0/pci1022,9603 at 2/pci15d9,a580
at 0/sd at 1,0:a''
whole_disk=1
children[2]
type=''disk''
id=2
guid=11201530646814503514
path=''/dev/dsk/c9t2d0s0''
devid=''id1,sd at n50014ee257d868ae/a''
phys_path=''/pci at 0,0/pci1022,9603 at 2/pci15d9,a580
at 0/sd at 2,0:a''
whole_disk=1
children[3]
type=''disk''
id=3
guid=4440015903126305102
path=''/dev/dsk/c9t3d0s0''
devid=''id1,sd at n50014ee20204f0c7/a''
phys_path=''/pci at 0,0/pci1022,9603 at 2/pci15d9,a580
at 0/sd at 3,0:a''
whole_disk=1
--------------------------------------------
LABEL 1
--------------------------------------------
version=14
name=''tank''
state=0
txg=118998
pool_guid=15136317365944618902
hostid=290968
hostname=''lexx''
top_guid=17530628911224651473
guid=11914115076705860870
vdev_tree
type=''raidz''
id=1
guid=17530628911224651473
nparity=1
metaslab_array=322
metaslab_shift=34
ashift=9
asize=3000567332864
is_log=0
children[0]
type=''disk''
id=0
guid=11914115076705860870
path=''/dev/dsk/c9t0d0s0''
devid=''id1,sd at n50000f000b039399/a''
phys_path=''/pci at 0,0/pci1022,9603 at 2/pci15d9,a580
at 0/sd at 0,0:a''
whole_disk=1
children[1]
type=''disk''
id=1
guid=11267830253346467192
path=''/dev/dsk/c9t1d0s0''
devid=''id1,sd at n5000c5000262bfd7/a''
phys_path=''/pci at 0,0/pci1022,9603 at 2/pci15d9,a580
at 0/sd at 1,0:a''
whole_disk=1
children[2]
type=''disk''
id=2
guid=11201530646814503514
path=''/dev/dsk/c9t2d0s0''
devid=''id1,sd at n50014ee257d868ae/a''
phys_path=''/pci at 0,0/pci1022,9603 at 2/pci15d9,a580
at 0/sd at 2,0:a''
whole_disk=1
children[3]
type=''disk''
id=3
guid=4440015903126305102
path=''/dev/dsk/c9t3d0s0''
devid=''id1,sd at n50014ee20204f0c7/a''
phys_path=''/pci at 0,0/pci1022,9603 at 2/pci15d9,a580
at 0/sd at 3,0:a''
whole_disk=1
--------------------------------------------
LABEL 2
--------------------------------------------
version=14
name=''tank''
state=0
txg=118998
pool_guid=15136317365944618902
hostid=290968
hostname=''lexx''
top_guid=17530628911224651473
guid=11914115076705860870
vdev_tree
type=''raidz''
id=1
guid=17530628911224651473
nparity=1
metaslab_array=322
metaslab_shift=34
ashift=9
asize=3000567332864
is_log=0
children[0]
type=''disk''
id=0
guid=11914115076705860870
path=''/dev/dsk/c9t0d0s0''
devid=''id1,sd at n50000f000b039399/a''
phys_path=''/pci at 0,0/pci1022,9603 at 2/pci15d9,a580
at 0/sd at 0,0:a''
whole_disk=1
children[1]
type=''disk''
id=1
guid=11267830253346467192
path=''/dev/dsk/c9t1d0s0''
devid=''id1,sd at n5000c5000262bfd7/a''
phys_path=''/pci at 0,0/pci1022,9603 at 2/pci15d9,a580
at 0/sd at 1,0:a''
whole_disk=1
children[2]
type=''disk''
id=2
guid=11201530646814503514
path=''/dev/dsk/c9t2d0s0''
devid=''id1,sd at n50014ee257d868ae/a''
phys_path=''/pci at 0,0/pci1022,9603 at 2/pci15d9,a580
at 0/sd at 2,0:a''
whole_disk=1
children[3]
type=''disk''
id=3
guid=4440015903126305102
path=''/dev/dsk/c9t3d0s0''
devid=''id1,sd at n50014ee20204f0c7/a''
phys_path=''/pci at 0,0/pci1022,9603 at 2/pci15d9,a580
at 0/sd at 3,0:a''
whole_disk=1
--------------------------------------------
LABEL 3
--------------------------------------------
version=14
name=''tank''
state=0
txg=118998
pool_guid=15136317365944618902
hostid=290968
hostname=''lexx''
top_guid=17530628911224651473
guid=11914115076705860870
vdev_tree
type=''raidz''
id=1
guid=17530628911224651473
nparity=1
metaslab_array=322
metaslab_shift=34
ashift=9
asize=3000567332864
is_log=0
children[0]
type=''disk''
id=0
guid=11914115076705860870
path=''/dev/dsk/c9t0d0s0''
devid=''id1,sd at n50000f000b039399/a''
phys_path=''/pci at 0,0/pci1022,9603 at 2/pci15d9,a580
at 0/sd at 0,0:a''
whole_disk=1
children[1]
type=''disk''
id=1
guid=11267830253346467192
path=''/dev/dsk/c9t1d0s0''
devid=''id1,sd at n5000c5000262bfd7/a''
phys_path=''/pci at 0,0/pci1022,9603 at 2/pci15d9,a580
at 0/sd at 1,0:a''
whole_disk=1
children[2]
type=''disk''
id=2
guid=11201530646814503514
path=''/dev/dsk/c9t2d0s0''
devid=''id1,sd at n50014ee257d868ae/a''
phys_path=''/pci at 0,0/pci1022,9603 at 2/pci15d9,a580
at 0/sd at 2,0:a''
whole_disk=1
children[3]
type=''disk''
id=3
guid=4440015903126305102
path=''/dev/dsk/c9t3d0s0''
devid=''id1,sd at n50014ee20204f0c7/a''
phys_path=''/pci at 0,0/pci1022,9603 at 2/pci15d9,a580
at 0/sd at 3,0:a''
whole_disk=1
We have the same problem since of today. The pool was to be "renamed" width zpool export, after an import it didn''t come back online. A import -f results in a kernel panic. zpool status -v freports a degraded drive also. I''ll also try to supply som,e traces and logs. -- This message posted from opensolaris.org
Marc Althoff wrote:> We have the same problem since of today. The pool was to be "renamed" width zpool export, after an import it didn''t come back online. A import -f results in a kernel panic. > > zpool status -v freports a degraded drive also. > > I''ll also try to supply som,e traces and logs. >Please provide at least stack trace from console or /var/adm/messages for a start, please try to make sure that crashdump from the first panic is saved. victor
dear all, victor, i am most happy to report that the problems were somehwat hardware-related, caused by a damaged / dangling SATA cable which apparently caused long delays (sometimes working, disk on, disk off, ...) during normal zfs operations. Why the -f produced a kernel panic I''m unsure. Interestingly it all fit some symptoms other people have with a bad uberlblock, a defect spanned metadata structure (?) detected after a scrube tc. anyway, great that you guys answered to quickly. there was 6 TB of data on that pool. I stress-tested it for a week and 30 minutes prior to the incident deleted the old RAID set ... imagine my horror ;) have a good one marc -- This message posted from opensolaris.org