thr3ads.net - zfs discuss - [zfs-discuss] Metadata corrupted [Oct 2006]

If this information is useful, please help other people find it:
Share via:

Siegfried Nikolaivich

2006-Oct-08 05:19 UTC

[zfs-discuss] Metadata corrupted

I was in the middle of doing a large transfer to my ZFS pool over CIFS.  Near
the end of the transfer, the Solaris machine froze.  Both ethernet links were
down.

I walked over to the machine and pushed the reset button, as it
wouldn''t respond to any key-presses.  After the machine booted up, I
got an unpleasant surprise:

FServe% zpool status
  pool: tank
 state: FAULTED
status: The pool metadata is corrupted and the pool cannot be opened.
action: Destroy and re-create the pool from a backup source.
   see: http://www.sun.com/msg/ZFS-8000-CS
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        tank        FAULTED      0     0     6  corrupted data
          raidz     ONLINE       0     0     6
            c0t0d0  ONLINE       0     0     0
            c0t1d0  ONLINE       0     0     0
            c0t2d0  ONLINE       0     0     0
            c0t4d0  ONLINE       0     0     0

The machine has worked fine for months, and now after a freeze/reboot all my
data is gone?  Is there a way to recover any of it?  I have some very important
files on it.

I am running Solaris 10 06/06.


Thanks
 
 
This message posted from opensolaris.org

Siegfried Nikolaivich

2006-Oct-09 20:55 UTC

head link

[zfs-discuss] Re: Metadata corrupted

> status: The pool metadata is corrupted and the pool
> cannot be opened.
Is there at least a way to determine what caused this error?  Is it a hardware
issue?  Is it a possible defect in ZFS?

I don''t think it''s a hardware issue because it seems to be
still working fine, and has been for months.

It''s important to have this information so that I/we can prevent it
from happening next time.


Thanks,
Siegfried
 
 
This message posted from opensolaris.org

Matthew Ahrens

2006-Oct-10 05:13 UTC

head link

[zfs-discuss] Re: Metadata corrupted

Siegfried Nikolaivich wrote:>> status: The pool metadata is corrupted and the pool cannot be
>> opened.
> 
> Is there at least a way to determine what caused this error?  Is it a
> hardware issue?  Is it a possible defect in ZFS?
My best guess would be that it''s a hardware issue.
> I don''t think it''s a hardware issue because it seems to
be still
> working fine, and has been for months.
"Working fine", except that you can''t access your pool,
right? :-)

We might be able to figure out more exactly what went wrong if you can 
send the output of:

zpool status -x
zdb -v tank
   (which might not work)
zdb -l /dev/dsk/c0t1d0
zdb -l /dev/dsk/... (for each of the other devices in the pool)

--matt

Siegfried Nikolaivich

2006-Oct-10 05:33 UTC

head link

[zfs-discuss] Re: Re: Metadata corrupted

Thanks for the response Matthew.
> > I don''t think it''s a hardware issue because it
> seems to be still
> > working fine, and has been for months.
> 
> "Working fine", except that you can''t access your
> pool, right? :-)
Well the computer and disk controller work fine when I tried it in Linux with a
different set of disks.  Even if one of the disks or the controller failed, I do
not think that this should destroy the pool, should it?
> We might be able to figure out more exactly what went
> wrong if you can 
> send the output of:
> 
> zpool status -x
> zdb -v tank
>    (which might not work)
>  -l /dev/dsk/c0t1d0
> db -l /dev/dsk/... (for each of the other devices in
> the pool)
# zpool status -x
  pool: tank
 state: FAULTED
status: The pool metadata is corrupted and the pool cannot be opened.
action: Destroy and re-create the pool from a backup source.
   see: http://www.sun.com/msg/ZFS-8000-CS
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        tank        FAULTED      0     0     6  corrupted data
          raidz     ONLINE       0     0     6
            c0t0d0  ONLINE       0     0     0
            c0t1d0  ONLINE       0     0     0
            c0t2d0  ONLINE       0     0     0
            c0t4d0  ONLINE       0     0     0

# zdb -v tank
    version=2
    name=''tank''
    state=0
    txg=4
    pool_guid=13701012839440790608
    vdev_tree
        type=''root''
        id=0
        guid=13701012839440790608
        children[0]
                type=''raidz''
                id=0
                guid=4302888056402016629
                metaslab_array=13
                metaslab_shift=33
                ashift=9
                asize=1280238944256
                children[0]
                        type=''disk''
                        id=0
                        guid=15193057146179069576
                        path=''/dev/dsk/c0t0d0s0''
                        whole_disk=1
                children[1]
                        type=''disk''
                        id=1
                        guid=5113010407673419800
                        path=''/dev/dsk/c0t1d0s0''
                        whole_disk=1
                children[2]
                        type=''disk''
                        id=2
                        guid=11908389868437050464
                        path=''/dev/dsk/c0t2d0s0''
                        whole_disk=1
                children[3]
                        type=''disk''
                        id=3
                        guid=800140628824658935
                        path=''/dev/dsk/c0t4d0s0''
                        devid=''id1,sd at
f2d72e6394515e8000005433e0003/a''
                        whole_disk=1

# zdb -l /dev/dsk/c0t0d0
--------------------------------------------
LABEL 0
--------------------------------------------
failed to unpack label 0
--------------------------------------------
LABEL 1
--------------------------------------------
failed to unpack label 1
--------------------------------------------
LABEL 2
--------------------------------------------
failed to unpack label 2
--------------------------------------------
LABEL 3
--------------------------------------------
failed to unpack label 3


The result is the same for the other 3 disks (c0t1d0, c0t2d0 and c0t4d0).  I am
new to ZFS, so I am not sure what these results tell, they don''t look
too good.  What I find strange is that zdb -v tank has a devid for the 4th
child, but not for the others.

Any ideas?


Thanks,
Siegfried
 
 
This message posted from opensolaris.org

Siegfried Nikolaivich

2006-Oct-10 05:37 UTC

head link

[zfs-discuss] Re: Re: Metadata corrupted

> > zdb -v tank
Forgot to add "zdb: can''t open tank: error 5" to the end of
the output of that command.
 
 
This message posted from opensolaris.org

Siegfried Nikolaivich

2006-Oct-10 05:46 UTC

head link

[zfs-discuss] Re: Re: Metadata corrupted

>  zdb -l /dev/dsk/c0t1d0
Sorry for posting again, but I think you might have meant /dev/dsk/c0t1d0s0
there.  The only difference between the following outputs is the guid for each
device.

# zdb -l /dev/dsk/c0t0d0s0
--------------------------------------------
LABEL 0
--------------------------------------------
    version=2
    name=''tank''
    state=0
    txg=225992
    pool_guid=13701012839440790608
    top_guid=4302888056402016629
    guid=15193057146179069576
    vdev_tree
        type=''raidz''
        id=0
        guid=4302888056402016629
        metaslab_array=13
        metaslab_shift=33
        ashift=9
        asize=1280238944256
        children[0]
                type=''disk''
                id=0
                guid=15193057146179069576
                path=''/dev/dsk/c0t0d0s0''
                whole_disk=1
                DTL=122
        children[1]
                type=''disk''
                id=1
                guid=5113010407673419800
                path=''/dev/dsk/c0t1d0s0''
                whole_disk=1
                DTL=121
        children[2]
                type=''disk''
                id=2
                guid=11908389868437050464
                path=''/dev/dsk/c0t2d0s0''
                whole_disk=1
                DTL=120
        children[3]
                type=''disk''
                id=3
                guid=800140628824658935
                path=''/dev/dsk/c0t4d0s0''
                devid=''id1,sd at
f2d72e6394515e8000005433e0003/a''
                whole_disk=1
                DTL=119
--------------------------------------------
LABEL 1
--------------------------------------------
    version=2
    name=''tank''
    state=0
    txg=225992
    pool_guid=13701012839440790608
    top_guid=4302888056402016629
    guid=15193057146179069576
    vdev_tree
        type=''raidz''
        id=0
        guid=4302888056402016629
        metaslab_array=13
        metaslab_shift=33
        ashift=9
        asize=1280238944256
        children[0]
                type=''disk''
                id=0
                guid=15193057146179069576
                path=''/dev/dsk/c0t0d0s0''
                whole_disk=1
                DTL=122
        children[1]
                type=''disk''
                id=1
                guid=5113010407673419800
                path=''/dev/dsk/c0t1d0s0''
                whole_disk=1
                DTL=121
        children[2]
                type=''disk''
                id=2
                guid=11908389868437050464
                path=''/dev/dsk/c0t2d0s0''
                whole_disk=1
                DTL=120
        children[3]
                type=''disk''
                id=3
                guid=800140628824658935
                path=''/dev/dsk/c0t4d0s0''
                devid=''id1,sd at
f2d72e6394515e8000005433e0003/a''
                whole_disk=1
                DTL=119
--------------------------------------------
LABEL 2
--------------------------------------------
    version=2
    name=''tank''
    state=0
    txg=225992
    pool_guid=13701012839440790608
    top_guid=4302888056402016629
    guid=15193057146179069576
    vdev_tree
        type=''raidz''
        id=0
        guid=4302888056402016629
        metaslab_array=13
        metaslab_shift=33
        ashift=9
        asize=1280238944256
        children[0]
                type=''disk''
                id=0
                guid=15193057146179069576
                path=''/dev/dsk/c0t0d0s0''
                whole_disk=1
                DTL=122
        children[1]
                type=''disk''
                id=1
                guid=5113010407673419800
                path=''/dev/dsk/c0t1d0s0''
                whole_disk=1
                DTL=121
        children[2]
                type=''disk''
                id=2
                guid=11908389868437050464
                path=''/dev/dsk/c0t2d0s0''
                whole_disk=1
                DTL=120
        children[3]
                type=''disk''
                id=3
                guid=800140628824658935
                path=''/dev/dsk/c0t4d0s0''
                devid=''id1,sd at
f2d72e6394515e8000005433e0003/a''
                whole_disk=1
                DTL=119
--------------------------------------------
LABEL 3
--------------------------------------------
    version=2
    name=''tank''
    state=0
    txg=225992
    pool_guid=13701012839440790608
    top_guid=4302888056402016629
    guid=15193057146179069576
    vdev_tree
        type=''raidz''
        id=0
        guid=4302888056402016629
        metaslab_array=13
        metaslab_shift=33
        ashift=9
        asize=1280238944256
        children[0]
                type=''disk''
                id=0
                guid=15193057146179069576
                path=''/dev/dsk/c0t0d0s0''
                whole_disk=1
                DTL=122
        children[1]
                type=''disk''
                id=1
                guid=5113010407673419800
                path=''/dev/dsk/c0t1d0s0''
                whole_disk=1
                DTL=121
        children[2]
                type=''disk''
                id=2
                guid=11908389868437050464
                path=''/dev/dsk/c0t2d0s0''
                whole_disk=1
                DTL=120
        children[3]
                type=''disk''
                id=3
                guid=800140628824658935
                path=''/dev/dsk/c0t4d0s0''
                devid=''id1,sd at
f2d72e6394515e8000005433e0003/a''
                whole_disk=1
                DTL=119

# zdb -l /dev/dsk/c0t1d0s0
--------------------------------------------
LABEL 0
--------------------------------------------
    version=2
    name=''tank''
    state=0
    txg=225992
    pool_guid=13701012839440790608
    top_guid=4302888056402016629
    guid=5113010407673419800
    vdev_tree
        type=''raidz''
        id=0
        guid=4302888056402016629
        metaslab_array=13
        metaslab_shift=33
        ashift=9
        asize=1280238944256
        children[0]
                type=''disk''
                id=0
                guid=15193057146179069576
                path=''/dev/dsk/c0t0d0s0''
                whole_disk=1
                DTL=122
        children[1]
                type=''disk''
                id=1
                guid=5113010407673419800
                path=''/dev/dsk/c0t1d0s0''
                whole_disk=1
                DTL=121
        children[2]
                type=''disk''
                id=2
                guid=11908389868437050464
                path=''/dev/dsk/c0t2d0s0''
                whole_disk=1
                DTL=120
        children[3]
                type=''disk''
                id=3
                guid=800140628824658935
                path=''/dev/dsk/c0t4d0s0''
                devid=''id1,sd at
f2d72e6394515e8000005433e0003/a''
                whole_disk=1
                DTL=119
--------------------------------------------
LABEL 1
--------------------------------------------
    version=2
    name=''tank''
    state=0
    txg=225992
    pool_guid=13701012839440790608
    top_guid=4302888056402016629
    guid=5113010407673419800
    vdev_tree
        type=''raidz''
        id=0
        guid=4302888056402016629
        metaslab_array=13
        metaslab_shift=33
        ashift=9
        asize=1280238944256
        children[0]
                type=''disk''
                id=0
                guid=15193057146179069576
                path=''/dev/dsk/c0t0d0s0''
                whole_disk=1
                DTL=122
        children[1]
                type=''disk''
                id=1
                guid=5113010407673419800
                path=''/dev/dsk/c0t1d0s0''
                whole_disk=1
                DTL=121
        children[2]
                type=''disk''
                id=2
                guid=11908389868437050464
                path=''/dev/dsk/c0t2d0s0''
                whole_disk=1
                DTL=120
        children[3]
                type=''disk''
                id=3
                guid=800140628824658935
                path=''/dev/dsk/c0t4d0s0''
                devid=''id1,sd at
f2d72e6394515e8000005433e0003/a''
                whole_disk=1
                DTL=119
--------------------------------------------
LABEL 2
--------------------------------------------
    version=2
    name=''tank''
    state=0
    txg=225992
    pool_guid=13701012839440790608
    top_guid=4302888056402016629
    guid=5113010407673419800
    vdev_tree
        type=''raidz''
        id=0
        guid=4302888056402016629
        metaslab_array=13
        metaslab_shift=33
        ashift=9
        asize=1280238944256
        children[0]
                type=''disk''
                id=0
                guid=15193057146179069576
                path=''/dev/dsk/c0t0d0s0''
                whole_disk=1
                DTL=122
        children[1]
                type=''disk''
                id=1
                guid=5113010407673419800
                path=''/dev/dsk/c0t1d0s0''
                whole_disk=1
                DTL=121
        children[2]
                type=''disk''
                id=2
                guid=11908389868437050464
                path=''/dev/dsk/c0t2d0s0''
                whole_disk=1
                DTL=120
        children[3]
                type=''disk''
                id=3
                guid=800140628824658935
                path=''/dev/dsk/c0t4d0s0''
                devid=''id1,sd at
f2d72e6394515e8000005433e0003/a''
                whole_disk=1
                DTL=119
--------------------------------------------
LABEL 3
--------------------------------------------
    version=2
    name=''tank''
    state=0
    txg=225992
    pool_guid=13701012839440790608
    top_guid=4302888056402016629
    guid=5113010407673419800
    vdev_tree
        type=''raidz''
        id=0
        guid=4302888056402016629
        metaslab_array=13
        metaslab_shift=33
        ashift=9
        asize=1280238944256
        children[0]
                type=''disk''
                id=0
                guid=15193057146179069576
                path=''/dev/dsk/c0t0d0s0''
                whole_disk=1
                DTL=122
        children[1]
                type=''disk''
                id=1
                guid=5113010407673419800
                path=''/dev/dsk/c0t1d0s0''
                whole_disk=1
                DTL=121
        children[2]
                type=''disk''
                id=2
                guid=11908389868437050464
                path=''/dev/dsk/c0t2d0s0''
                whole_disk=1
                DTL=120
        children[3]
                type=''disk''
                id=3
                guid=800140628824658935
                path=''/dev/dsk/c0t4d0s0''
                devid=''id1,sd at
f2d72e6394515e8000005433e0003/a''
                whole_disk=1
                DTL=119

# zdb -l /dev/dsk/c0t2d0s0
--------------------------------------------
LABEL 0
--------------------------------------------
    version=2
    name=''tank''
    state=0
    txg=225992
    pool_guid=13701012839440790608
    top_guid=4302888056402016629
    guid=11908389868437050464
    vdev_tree
        type=''raidz''
        id=0
        guid=4302888056402016629
        metaslab_array=13
        metaslab_shift=33
        ashift=9
        asize=1280238944256
        children[0]
                type=''disk''
                id=0
                guid=15193057146179069576
                path=''/dev/dsk/c0t0d0s0''
                whole_disk=1
                DTL=122
        children[1]
                type=''disk''
                id=1
                guid=5113010407673419800
                path=''/dev/dsk/c0t1d0s0''
                whole_disk=1
                DTL=121
        children[2]
                type=''disk''
                id=2
                guid=11908389868437050464
                path=''/dev/dsk/c0t2d0s0''
                whole_disk=1
                DTL=120
        children[3]
                type=''disk''
                id=3
                guid=800140628824658935
                path=''/dev/dsk/c0t4d0s0''
                devid=''id1,sd at
f2d72e6394515e8000005433e0003/a''
                whole_disk=1
                DTL=119
--------------------------------------------
LABEL 1
--------------------------------------------
    version=2
    name=''tank''
    state=0
    txg=225992
    pool_guid=13701012839440790608
    top_guid=4302888056402016629
    guid=11908389868437050464
    vdev_tree
        type=''raidz''
        id=0
        guid=4302888056402016629
        metaslab_array=13
        metaslab_shift=33
        ashift=9
        asize=1280238944256
        children[0]
                type=''disk''
                id=0
                guid=15193057146179069576
                path=''/dev/dsk/c0t0d0s0''
                whole_disk=1
                DTL=122
        children[1]
                type=''disk''
                id=1
                guid=5113010407673419800
                path=''/dev/dsk/c0t1d0s0''
                whole_disk=1
                DTL=121
        children[2]
                type=''disk''
                id=2
                guid=11908389868437050464
                path=''/dev/dsk/c0t2d0s0''
                whole_disk=1
                DTL=120
        children[3]
                type=''disk''
                id=3
                guid=800140628824658935
                path=''/dev/dsk/c0t4d0s0''
                devid=''id1,sd at
f2d72e6394515e8000005433e0003/a''
                whole_disk=1
                DTL=119
--------------------------------------------
LABEL 2
--------------------------------------------
    version=2
    name=''tank''
    state=0
    txg=225992
    pool_guid=13701012839440790608
    top_guid=4302888056402016629
    guid=11908389868437050464
    vdev_tree
        type=''raidz''
        id=0
        guid=4302888056402016629
        metaslab_array=13
        metaslab_shift=33
        ashift=9
        asize=1280238944256
        children[0]
                type=''disk''
                id=0
                guid=15193057146179069576
                path=''/dev/dsk/c0t0d0s0''
                whole_disk=1
                DTL=122
        children[1]
                type=''disk''
                id=1
                guid=5113010407673419800
                path=''/dev/dsk/c0t1d0s0''
                whole_disk=1
                DTL=121
        children[2]
                type=''disk''
                id=2
                guid=11908389868437050464
                path=''/dev/dsk/c0t2d0s0''
                whole_disk=1
                DTL=120
        children[3]
                type=''disk''
                id=3
                guid=800140628824658935
                path=''/dev/dsk/c0t4d0s0''
                devid=''id1,sd at
f2d72e6394515e8000005433e0003/a''
                whole_disk=1
                DTL=119
--------------------------------------------
LABEL 3
--------------------------------------------
    version=2
    name=''tank''
    state=0
    txg=225992
    pool_guid=13701012839440790608
    top_guid=4302888056402016629
    guid=11908389868437050464
    vdev_tree
        type=''raidz''
        id=0
        guid=4302888056402016629
        metaslab_array=13
        metaslab_shift=33
        ashift=9
        asize=1280238944256
        children[0]
                type=''disk''
                id=0
                guid=15193057146179069576
                path=''/dev/dsk/c0t0d0s0''
                whole_disk=1
                DTL=122
        children[1]
                type=''disk''
                id=1
                guid=5113010407673419800
                path=''/dev/dsk/c0t1d0s0''
                whole_disk=1
                DTL=121
        children[2]
                type=''disk''
                id=2
                guid=11908389868437050464
                path=''/dev/dsk/c0t2d0s0''
                whole_disk=1
                DTL=120
        children[3]
                type=''disk''
                id=3
                guid=800140628824658935
                path=''/dev/dsk/c0t4d0s0''
                devid=''id1,sd at
f2d72e6394515e8000005433e0003/a''
                whole_disk=1
                DTL=119

# zdb -l /dev/dsk/c0t4d0s0
--------------------------------------------
LABEL 0
--------------------------------------------
    version=2
    name=''tank''
    state=0
    txg=225992
    pool_guid=13701012839440790608
    top_guid=4302888056402016629
    guid=800140628824658935
    vdev_tree
        type=''raidz''
        id=0
        guid=4302888056402016629
        metaslab_array=13
        metaslab_shift=33
        ashift=9
        asize=1280238944256
        children[0]
                type=''disk''
                id=0
                guid=15193057146179069576
                path=''/dev/dsk/c0t0d0s0''
                whole_disk=1
                DTL=122
        children[1]
                type=''disk''
                id=1
                guid=5113010407673419800
                path=''/dev/dsk/c0t1d0s0''
                whole_disk=1
                DTL=121
        children[2]
                type=''disk''
                id=2
                guid=11908389868437050464
                path=''/dev/dsk/c0t2d0s0''
                whole_disk=1
                DTL=120
        children[3]
                type=''disk''
                id=3
                guid=800140628824658935
                path=''/dev/dsk/c0t4d0s0''
                devid=''id1,sd at
f2d72e6394515e8000005433e0003/a''
                whole_disk=1
                DTL=119
--------------------------------------------
LABEL 1
--------------------------------------------
    version=2
    name=''tank''
    state=0
    txg=225992
    pool_guid=13701012839440790608
    top_guid=4302888056402016629
    guid=800140628824658935
    vdev_tree
        type=''raidz''
        id=0
        guid=4302888056402016629
        metaslab_array=13
        metaslab_shift=33
        ashift=9
        asize=1280238944256
        children[0]
                type=''disk''
                id=0
                guid=15193057146179069576
                path=''/dev/dsk/c0t0d0s0''
                whole_disk=1
                DTL=122
        children[1]
                type=''disk''
                id=1
                guid=5113010407673419800
                path=''/dev/dsk/c0t1d0s0''
                whole_disk=1
                DTL=121
        children[2]
                type=''disk''
                id=2
                guid=11908389868437050464
                path=''/dev/dsk/c0t2d0s0''
                whole_disk=1
                DTL=120
        children[3]
                type=''disk''
                id=3
                guid=800140628824658935
                path=''/dev/dsk/c0t4d0s0''
                devid=''id1,sd at
f2d72e6394515e8000005433e0003/a''
                whole_disk=1
                DTL=119
--------------------------------------------
LABEL 2
--------------------------------------------
    version=2
    name=''tank''
    state=0
    txg=225992
    pool_guid=13701012839440790608
    top_guid=4302888056402016629
    guid=800140628824658935
    vdev_tree
        type=''raidz''
        id=0
        guid=4302888056402016629
        metaslab_array=13
        metaslab_shift=33
        ashift=9
        asize=1280238944256
        children[0]
                type=''disk''
                id=0
                guid=15193057146179069576
                path=''/dev/dsk/c0t0d0s0''
                whole_disk=1
                DTL=122
        children[1]
                type=''disk''
                id=1
                guid=5113010407673419800
                path=''/dev/dsk/c0t1d0s0''
                whole_disk=1
                DTL=121
        children[2]
                type=''disk''
                id=2
                guid=11908389868437050464
                path=''/dev/dsk/c0t2d0s0''
                whole_disk=1
                DTL=120
        children[3]
                type=''disk''
                id=3
                guid=800140628824658935
                path=''/dev/dsk/c0t4d0s0''
                devid=''id1,sd at
f2d72e6394515e8000005433e0003/a''
                whole_disk=1
                DTL=119
--------------------------------------------
LABEL 3
--------------------------------------------
    version=2
    name=''tank''
    state=0
    txg=225992
    pool_guid=13701012839440790608
    top_guid=4302888056402016629
    guid=800140628824658935
    vdev_tree
        type=''raidz''
        id=0
        guid=4302888056402016629
        metaslab_array=13
        metaslab_shift=33
        ashift=9
        asize=1280238944256
        children[0]
                type=''disk''
                id=0
                guid=15193057146179069576
                path=''/dev/dsk/c0t0d0s0''
                whole_disk=1
                DTL=122
        children[1]
                type=''disk''
                id=1
                guid=5113010407673419800
                path=''/dev/dsk/c0t1d0s0''
                whole_disk=1
                DTL=121
        children[2]
                type=''disk''
                id=2
                guid=11908389868437050464
                path=''/dev/dsk/c0t2d0s0''
                whole_disk=1
                DTL=120
        children[3]
                type=''disk''
                id=3
                guid=800140628824658935
                path=''/dev/dsk/c0t4d0s0''
                devid=''id1,sd at
f2d72e6394515e8000005433e0003/a''
                whole_disk=1
                DTL=119


Thanks,
Siegfried
 
 
This message posted from opensolaris.org

Matthew Ahrens

2006-Oct-10 06:08 UTC

head link

[zfs-discuss] Re: Re: Metadata corrupted

Siegfried Nikolaivich wrote:>> zdb -l /dev/dsk/c0t1d0
> 
> Sorry for posting again, but I think you might have meant
> /dev/dsk/c0t1d0s0 there.  The only difference between the following
> outputs is the guid for each device.
Yeah, good catch.  So this means that it seems to be able to read the 
label off of each device OK, and the labels look good.  I''m not sure 
what else would cause us to be unable to open the pool...  Can you try 
running ''zpool status -v''?

--matt

Siegfried Nikolaivich

2006-Oct-10 06:37 UTC

head link

[zfs-discuss] Re: Re: Re: Metadata corrupted

> Yeah, good catch.  So this means that it seems to be
> able to read the 
> label off of each device OK, and the labels look
> good.  I''m not sure 
> what else would cause us to be unable to open the
> pool...  Can you try 
> running ''zpool status -v''?
The command seems to return the same thing:

% zpool status -v
  pool: tank
 state: FAULTED
status: The pool metadata is corrupted and the pool cannot be opened.
action: Destroy and re-create the pool from a backup source.
   see: http://www.sun.com/msg/ZFS-8000-CS
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        tank        FAULTED      0     0     6  corrupted data
          raidz     ONLINE       0     0     6
            c0t0d0  ONLINE       0     0     0
            c0t1d0  ONLINE       0     0     0
            c0t2d0  ONLINE       0     0     0
            c0t4d0  ONLINE       0     0     0


I can provide you with SSH access if you want.

Thanks,
Siegfried
 
 
This message posted from opensolaris.org

Eric Schrock

2006-Oct-10 14:37 UTC

head link

[zfs-discuss] Re: Re: Metadata corrupted

On Mon, Oct 09, 2006 at 11:08:14PM -0700, Matthew Ahrens
wrote:> Siegfried Nikolaivich wrote:
> >>zdb -l /dev/dsk/c0t1d0
> >
> >Sorry for posting again, but I think you might have meant
> >/dev/dsk/c0t1d0s0 there.  The only difference between the following
> >outputs is the guid for each device.
> 
> Yeah, good catch.  So this means that it seems to be able to read the 
> label off of each device OK, and the labels look good.  I''m not
sure
> what else would cause us to be unable to open the pool...  Can you try 
> running ''zpool status -v''?
You may also want to try ''fmdump -eV'' to get an idea of what
those
faults were.

- Eric

--
Eric Schrock, Solaris Kernel Development       http://blogs.sun.com/eschrock

Siegfried Nikolaivich

2006-Oct-11 20:55 UTC

head link

[zfs-discuss] Re: Re: Re: Metadata corrupted

> On Mon, Oct 09, 2006 at 11:08:14PM -0700, Matthew
> Ahrens wrote:
> You may also want to try ''fmdump -eV'' to get an idea
> of what those
> faults were.
I am not sure how to interpret the results, maybe you can help me.  It looks
like the following with many more similar pages following:

% fmdump -eV
TIME                           CLASS
Oct 07 2006 17:28:48.265102839 ereport.fs.zfs.checksum
nvlist version: 0
        class = ereport.fs.zfs.checksum
        ena = 0x933872163a00001
        detector = (embedded nvlist)
        nvlist version: 0
                version = 0x0
                scheme = zfs
                pool = 0xbe23c6961def3450
                vdev = 0x46f50fe03a3fd818
        (end detector)

        pool = tank
        pool_guid = 0xbe23c6961def3450
        pool_context = 0
        vdev_guid = 0x46f50fe03a3fd818
        vdev_type = disk
        vdev_path = /dev/dsk/c0t1d0s0
        parent_guid = 0x3bb6ede3be1cf975
        parent_type = raidz
        zio_err = 0
        zio_offset = 0x1c3644ae00
        zio_size = 0xac00
        zio_objset = 0x20
        zio_object = 0x78
        zio_level = 0
        zio_blkid = 0xafaf
        __ttl = 0x1
        __tod = 0x45284640 0xfcd25f7

Oct 07 2006 17:31:24.616729701 ereport.fs.zfs.checksum
nvlist version: 0
        class = ereport.fs.zfs.checksum
        ena = 0xb7a0bad55900401
        detector = (embedded nvlist)
        nvlist version: 0
                version = 0x0
                scheme = zfs
                pool = 0xbe23c6961def3450
                vdev = 0xa543197df30d1460
        (end detector)

        pool = tank
        pool_guid = 0xbe23c6961def3450
        pool_context = 0
        vdev_guid = 0xa543197df30d1460
        vdev_type = disk
        vdev_path = /dev/dsk/c0t2d0s0
        parent_guid = 0x3bb6ede3be1cf975
        parent_type = raidz
        zio_err = 0
        zio_offset = 0x30d218e00
        zio_size = 0xac00
        zio_objset = 0x20
        zio_object = 0xea
        zio_level = 0
        zio_blkid = 0x7577
        __ttl = 0x1
        __tod = 0x452846dc 0x24c28c65

Oct 07 2006 17:31:24.903968466 ereport.fs.zfs.checksum
nvlist version: 0
        class = ereport.fs.zfs.checksum
        ena = 0xb7b1da392500001
        detector = (embedded nvlist)
        nvlist version: 0
                version = 0x0
                scheme = zfs
                pool = 0xbe23c6961def3450
                vdev = 0x46f50fe03a3fd818
        (end detector)

        pool = tank
        pool_guid = 0xbe23c6961def3450
        pool_context = 0
        vdev_guid = 0x46f50fe03a3fd818
        vdev_type = disk
        vdev_path = /dev/dsk/c0t1d0s0
        parent_guid = 0x3bb6ede3be1cf975
        parent_type = raidz
        zio_err = 0
        zio_offset = 0x30e558800
        zio_size = 0xac00
        zio_objset = 0x20
        zio_object = 0xea
        zio_level = 0
        zio_blkid = 0x7724
        __ttl = 0x1
        __tod = 0x452846dc 0x35e176d2

Oct 07 2006 17:31:52.178481693 ereport.fs.zfs.checksum
nvlist version: 0
        class = ereport.fs.zfs.checksum
        ena = 0xbe0bb6f3b100001
        detector = (embedded nvlist)
        nvlist version: 0
                version = 0x0
                scheme = zfs
                pool = 0xbe23c6961def3450
                vdev = 0xa543197df30d1460
        (end detector)

        pool = tank
        pool_guid = 0xbe23c6961def3450
        pool_context = 0
        vdev_guid = 0xa543197df30d1460
        vdev_type = disk
        vdev_path = /dev/dsk/c0t2d0s0
        parent_guid = 0x3bb6ede3be1cf975
        parent_type = raidz
        zio_err = 0
        zio_offset = 0x375e12800
        zio_size = 0xac00
        zio_objset = 0x20
        zio_object = 0xec
        zio_level = 0
        zio_blkid = 0x7788
        __ttl = 0x1
        __tod = 0x452846f8 0xaa36a1d

Cheers,
Albert
 
 
This message posted from opensolaris.org

Alasdair Lumsden

2008-Mar-28 13:47 UTC

head link

[zfs-discuss] Metadata corrupted

Hi There,

Were you able to fix this problem in the end?
 
 
This message posted from opensolaris.org

Siegfried Nikolaivich

2008-Apr-29 02:42 UTC

head link

[zfs-discuss] Metadata corrupted

> Were you able to fix this problem in the end?
Unfortunately, no.  I believe Matthew Ahrens took a look at it and
couldn''t find the cause or how to fix it.  We had to destroy the pool
and re-create it from scratch.

Fortunately, this was during the ZFS testing period, and no critically important
data was lost, but I am still a bit shaken by the incident.  Since then we did
eventually adopt ZFS, and it has been running well without further such problems
for over a year now.  This leads me to believe it was either a software bug, or
a hardware failure that triggered a fatal condition in the software that is not
resilient to error in a redundant configuration.  I am sincerely hoping that
this has been fixed, on purpose or by accident.

Cheers,
Siegfried
 
 
This message posted from opensolaris.org

Łukasz

2008-Apr-30 10:31 UTC

head link

[zfs-discuss] Metadata corrupted

Did you see http://www.opensolaris.org/jive/thread.jspa?messageID=220125

I managed to recover my lost data with simple mdb commands.

--Lukas
 
 
This message posted from opensolaris.org

zfs discuss - Oct 2006 - Metadata corrupted

[zfs-discuss] Metadata corrupted

[zfs-discuss] Re: Metadata corrupted

[zfs-discuss] Re: Metadata corrupted

[zfs-discuss] Re: Re: Metadata corrupted

[zfs-discuss] Re: Re: Metadata corrupted

[zfs-discuss] Re: Re: Metadata corrupted

[zfs-discuss] Re: Re: Metadata corrupted

[zfs-discuss] Re: Re: Re: Metadata corrupted

[zfs-discuss] Re: Re: Metadata corrupted

[zfs-discuss] Re: Re: Re: Metadata corrupted

[zfs-discuss] Metadata corrupted

[zfs-discuss] Metadata corrupted

[zfs-discuss] Metadata corrupted