thr3ads.net - zfs discuss - [zfs-discuss] FAULTED ZFS volume even though it is mirrored [Feb 2007]

If this information is useful, please help other people find it:
Share via:

Stuart Low

2007-Feb-28 23:31 UTC

[zfs-discuss] FAULTED ZFS volume even though it is mirrored

Hi there,

We have been using ZFS for our backup storage since August last year.
Overall it''s been very good, handling transient data issues and even
drop outs of connectivity to the iscsi arrays we are using for storage.
However, I logged in this morning to discover that the ZFS volume could
not be read. In addition, it appears to have marked all drives, mirrors
& the volume itself as ''corrupted''.
>From what I can tell I''m completely unable to perform a zpool
import andthe only solution according to SUN is to ''destroy and recreate the
volume from a backup''. That''s not overly helpful considering
this IS our
backup source AND the fact we thought we had redundancy covered through
the implementation of multiple mirrors. The arrays themselves are
configured in 5 x 880GB RAID5 LUNs per array. These 5 are then mirrored
to their matching 5 on the other array.

To be honest, I''m pretty disappointed this occurred. I was of the
assumption that by setting up a set of mirrored volumes I would avoid
this exact problem. Now, for whatever reason (perhaps triggered by an
iscsi connection timeout) ZFS has decided all devices in the entire
array are corrupt. I guess that''s why assumptions are the mother of
all ......s. :)

While this data isn''t mission critical it IS of significant use to us
for historical analysis purposes so my assistance would be greatly
appreciated.

I''ve included a dump of the zpool import response below.

Thanks for your help!

Stuart

----

[root at solaris1 ~]$ zpool import
  pool: ax150
    id: 6217526921542582188
 state: FAULTED
status: One or more devices contains corrupted data.
action: The pool cannot be imported due to damaged devices or data.
   see: http://www.sun.com/msg/ZFS-8000-5E
config:

ax150                                      FAULTED   corrupted data
mirror                                   FAULTED   corrupted data
  c5t6006016031E0180032F8E9868E30DB11d0  FAULTED   corrupted data
  c5t6006016071851800B86C8EE05831DB11d0  FAULTED   corrupted data
mirror                                   FAULTED   corrupted data
  c5t6006016031E01800AC7E34918E30DB11d0  FAULTED   corrupted data
  c5t600601607185180010A65FE75831DB11d0  FAULTED   corrupted data
mirror                                   FAULTED   corrupted data
  c5t6006016031E0180026057F9B8E30DB11d0  FAULTED   corrupted data
  c5t6006016071851800CA1D94EF5831DB11d0  FAULTED   corrupted data
mirror                                   FAULTED   corrupted data
  c5t6006016031E018005A9B74A48E30DB11d0  FAULTED   corrupted data
  c5t600601607185180064063BF85831DB11d0  FAULTED   corrupted data
mirror                                   FAULTED   corrupted data
  c5t6006016031E018003810E7AC8E30DB11d0  FAULTED   corrupted data
  c5t60060160718518009A7926FF5831DB11d0  FAULTED   corrupted data
[root at solaris1 ~]$

Eric Schrock

2007-Feb-28 23:53 UTC

head link

[zfs-discuss] FAULTED ZFS volume even though it is mirrored

That''s quite strange.  What version of ZFS are you running?  What does
''zdb -l /dev/dsk/c5t6006016031E0180032F8E9868E30DB11d0s0''
show?

- eric

On Thu, Mar 01, 2007 at 09:31:05AM +1000, Stuart Low
wrote:> Hi there,
> 
> We have been using ZFS for our backup storage since August last year.
> Overall it''s been very good, handling transient data issues and
even
> drop outs of connectivity to the iscsi arrays we are using for storage.
> However, I logged in this morning to discover that the ZFS volume could
> not be read. In addition, it appears to have marked all drives, mirrors
> & the volume itself as ''corrupted''.
> 
> >From what I can tell I''m completely unable to perform a zpool
import and
> the only solution according to SUN is to ''destroy and recreate the
> volume from a backup''. That''s not overly helpful
considering this IS our
> backup source AND the fact we thought we had redundancy covered through
> the implementation of multiple mirrors. The arrays themselves are
> configured in 5 x 880GB RAID5 LUNs per array. These 5 are then mirrored
> to their matching 5 on the other array.
> 
> To be honest, I''m pretty disappointed this occurred. I was of the
> assumption that by setting up a set of mirrored volumes I would avoid
> this exact problem. Now, for whatever reason (perhaps triggered by an
> iscsi connection timeout) ZFS has decided all devices in the entire
> array are corrupt. I guess that''s why assumptions are the mother
of
> all ......s. :)
> 
> While this data isn''t mission critical it IS of significant use to
us
> for historical analysis purposes so my assistance would be greatly
> appreciated.
> 
> I''ve included a dump of the zpool import response below.
> 
> Thanks for your help!
> 
> Stuart
> 
> ----
> 
> [root at solaris1 ~]$ zpool import
>   pool: ax150
>     id: 6217526921542582188
>  state: FAULTED
> status: One or more devices contains corrupted data.
> action: The pool cannot be imported due to damaged devices or data.
>    see: http://www.sun.com/msg/ZFS-8000-5E
> config:
> 
> ax150                                      FAULTED   corrupted data
> mirror                                   FAULTED   corrupted data
>   c5t6006016031E0180032F8E9868E30DB11d0  FAULTED   corrupted data
>   c5t6006016071851800B86C8EE05831DB11d0  FAULTED   corrupted data
> mirror                                   FAULTED   corrupted data
>   c5t6006016031E01800AC7E34918E30DB11d0  FAULTED   corrupted data
>   c5t600601607185180010A65FE75831DB11d0  FAULTED   corrupted data
> mirror                                   FAULTED   corrupted data
>   c5t6006016031E0180026057F9B8E30DB11d0  FAULTED   corrupted data
>   c5t6006016071851800CA1D94EF5831DB11d0  FAULTED   corrupted data
> mirror                                   FAULTED   corrupted data
>   c5t6006016031E018005A9B74A48E30DB11d0  FAULTED   corrupted data
>   c5t600601607185180064063BF85831DB11d0  FAULTED   corrupted data
> mirror                                   FAULTED   corrupted data
>   c5t6006016031E018003810E7AC8E30DB11d0  FAULTED   corrupted data
>   c5t60060160718518009A7926FF5831DB11d0  FAULTED   corrupted data
> [root at solaris1 ~]$
> 
> 
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
--
Eric Schrock, Solaris Kernel Development       http://blogs.sun.com/eschrock

Stuart Low

2007-Mar-01 00:26 UTC

head link

[zfs-discuss] FAULTED ZFS volume even though it is mirrored

Heya,

Firstly thanks for your help.
> That''s quite strange.  
Your telling me! :) I like ZFS I really do but this has dented my love
of it. :-/
> What version of ZFS are you running?  
[root at solaris1 ~]$ pkginfo -l SUNWzfsu
   PKGINST:  SUNWzfsu
      NAME:  ZFS (Usr)
  CATEGORY:  system
      ARCH:  i386
   VERSION:  11.10.0,REV=2006.05.18.01.46
   BASEDIR:  /
    VENDOR:  Sun Microsystems, Inc.
      DESC:  ZFS libraries and commands
    PSTAMP:  on10-patch-x20060302165447
  INSTDATE:  Nov 28 2006 16:53
   HOTLINE:  Please contact your local service provider
    STATUS:  completely installed
     FILES:       45 installed pathnames
                  14 shared pathnames
                   1 linked files
                  16 directories
                  13 executables
                3612 blocks used (approx)

[root at solaris1 ~]$
> What does
> ''zdb -l /dev/dsk/c5t6006016031E0180032F8E9868E30DB11d0s0''
show?
Included below. I don''t suppose there a full manual of zdb?

Stuart

----

[root at solaris1 ~]$ zdb
-l /dev/dsk/c5t6006016031E0180032F8E9868E30DB11d0s0
--------------------------------------------
LABEL 0
--------------------------------------------
    version=2
    name=''ax150''
    state=0
    txg=2063663
    pool_guid=6217526921542582188
    top_guid=11119705179868573891
    guid=10975169994332783304
    vdev_tree
        type=''mirror''
        id=0
        guid=11119705179868573891
        metaslab_array=13
        metaslab_shift=33
        ashift=9
        asize=944879435776
        children[0]
                type=''disk''
                id=0
                guid=10975169994332783304
               
path=''/dev/dsk/c4t6006016031E0180032F8E9868E30DB11d0s0''
                devid=''id1,sd at
n6006016031e0180032f8e9868e30db11/a''
                whole_disk=1
                DTL=314
        children[1]
                type=''disk''
                id=1
                guid=5119251555573000616
               
path=''/dev/dsk/c4t6006016071851800B86C8EE05831DB11d0s0''
                devid=''id1,sd at
n6006016071851800b86c8ee05831db11/a''
                whole_disk=1
                DTL=313
--------------------------------------------
LABEL 1
--------------------------------------------
    version=2
    name=''ax150''
    state=0
    txg=2063663
    pool_guid=6217526921542582188
    top_guid=11119705179868573891
    guid=10975169994332783304
    vdev_tree
        type=''mirror''
        id=0
        guid=11119705179868573891
        metaslab_array=13
        metaslab_shift=33
        ashift=9
        asize=944879435776
        children[0]
                type=''disk''
                id=0
                guid=10975169994332783304
               
path=''/dev/dsk/c4t6006016031E0180032F8E9868E30DB11d0s0''
                devid=''id1,sd at
n6006016031e0180032f8e9868e30db11/a''
                whole_disk=1
                DTL=314
        children[1]
                type=''disk''
                id=1
                guid=5119251555573000616
               
path=''/dev/dsk/c4t6006016071851800B86C8EE05831DB11d0s0''
                devid=''id1,sd at
n6006016071851800b86c8ee05831db11/a''
                whole_disk=1
                DTL=313
--------------------------------------------
LABEL 2
--------------------------------------------
    version=2
    name=''ax150''
    state=0
    txg=2063663
    pool_guid=6217526921542582188
    top_guid=11119705179868573891
    guid=10975169994332783304
    vdev_tree
        type=''mirror''
        id=0
        guid=11119705179868573891
        metaslab_array=13
        metaslab_shift=33
        ashift=9
        asize=944879435776
        children[0]
                type=''disk''
                id=0
                guid=10975169994332783304
               
path=''/dev/dsk/c4t6006016031E0180032F8E9868E30DB11d0s0''
                devid=''id1,sd at
n6006016031e0180032f8e9868e30db11/a''
                whole_disk=1
                DTL=314
        children[1]
                type=''disk''
                id=1
                guid=5119251555573000616
               
path=''/dev/dsk/c4t6006016071851800B86C8EE05831DB11d0s0''
                devid=''id1,sd at
n6006016071851800b86c8ee05831db11/a''
                whole_disk=1
                DTL=313
--------------------------------------------
LABEL 3
--------------------------------------------
    version=2
    name=''ax150''
    state=0
    txg=2063663
    pool_guid=6217526921542582188
    top_guid=11119705179868573891
    guid=10975169994332783304
    vdev_tree
        type=''mirror''
        id=0
        guid=11119705179868573891
        metaslab_array=13
        metaslab_shift=33
        ashift=9
        asize=944879435776
        children[0]
                type=''disk''
                id=0
                guid=10975169994332783304
               
path=''/dev/dsk/c4t6006016031E0180032F8E9868E30DB11d0s0''
                devid=''id1,sd at
n6006016031e0180032f8e9868e30db11/a''
                whole_disk=1
                DTL=314
        children[1]
                type=''disk''
                id=1
                guid=5119251555573000616
               
path=''/dev/dsk/c4t6006016071851800B86C8EE05831DB11d0s0''
                devid=''id1,sd at
n6006016071851800b86c8ee05831db11/a''
                whole_disk=1
                DTL=313
[root at solaris1 ~]$

Eric Schrock

2007-Mar-01 00:33 UTC

head link

[zfs-discuss] FAULTED ZFS volume even though it is mirrored

The label looks sane.  Can you try running:

# dtrace -n vdev_set_state:entry''{@[args[2], args[3], stack()] =
count()}''

While executing ''zpool import'' and send the output?  Can you
also send
''::dis'' output (from ''mdb -k'') of the
function immediately above
vdev_set_state() in the above stacks?  I think the function should be
vdev_validate(), but I don''t remember if it''s the same in the
ZFS
version you have.

- Eric

--
Eric Schrock, Solaris Kernel Development       http://blogs.sun.com/eschrock

Stuart Low

2007-Mar-01 00:46 UTC

head link

[zfs-discuss] FAULTED ZFS volume even though it is mirrored

Heya,
> The label looks sane.  Can you try running:
Not sure if I should be reassured by that but I''ll hold my hopes
high. :)
> # dtrace -n vdev_set_state:entry''{@[args[2], args[3], stack()] =
count()}''
> While executing ''zpool import'' and send the output?  Can
you also send
> ''::dis'' output (from ''mdb -k'') of the
function immediately above
> vdev_set_state() in the above stacks?  I think the function should be
> vdev_validate(), but I don''t remember if it''s the same in
the ZFS
> version you have.
Attached below. Forgive my ignorance but I don''t understand what
you''re
requesting with regard to ''mdb -k''?
>From what I can see ''vdev_propagate_state'' appears to be
the functionimmediately before vdev_set_state

Thanks again for your assistance, greatly appreciated.

Stuart

---
[root at solaris1 ~]$ dtrace -n vdev_set_state:entry''{@[args[2],
args[3],
stack()] = count()}'' -c ''zpool import''
dtrace: description ''vdev_set_state:entry'' matched 1 probe
  pool: ax150
<snip>
dtrace: pid 726 has exited

        4        0
              zfs`vdev_root_state_change+0x61
              zfs`vdev_propagate_state+0x8f
              zfs`vdev_set_state+0xa8
              zfs`vdev_mirror_state_change+0x32
              zfs`vdev_propagate_state+0x8f
              zfs`vdev_set_state+0xa8
              zfs`vdev_load+0x69
              zfs`vdev_load+0x25
              zfs`vdev_load+0x25
              zfs`spa_load+0x487
              zfs`spa_tryimport+0x87
              zfs`zfs_ioc_pool_tryimport+0x4b
              zfs`zfsdev_ioctl+0x144
              genunix`cdev_ioctl+0x1d
              specfs`spec_ioctl+0x50
              genunix`fop_ioctl+0x1a
              genunix`ioctl+0xac
              unix`_sys_sysenter_post_swapgs+0x139
                1
        3        2
              zfs`vdev_propagate_state+0xc6
              zfs`vdev_set_state+0xa8
              zfs`vdev_mirror_state_change+0x32
              zfs`vdev_propagate_state+0x8f
              zfs`vdev_set_state+0xa8
              zfs`vdev_load+0x69
              zfs`vdev_load+0x25
              zfs`vdev_load+0x25
              zfs`spa_load+0x487
              zfs`spa_tryimport+0x87
              zfs`zfs_ioc_pool_tryimport+0x4b
              zfs`zfsdev_ioctl+0x144
              genunix`cdev_ioctl+0x1d
              specfs`spec_ioctl+0x50
              genunix`fop_ioctl+0x1a
              genunix`ioctl+0xac
              unix`_sys_sysenter_post_swapgs+0x139
                4
        3        2
              zfs`vdev_propagate_state+0xc6
              zfs`vdev_set_state+0xa8
              zfs`vdev_mirror_state_change+0x45
              zfs`vdev_propagate_state+0x8f
              zfs`vdev_set_state+0xa8
              zfs`vdev_load+0x69
              zfs`vdev_load+0x25
              zfs`vdev_load+0x25
              zfs`spa_load+0x487
              zfs`spa_tryimport+0x87
              zfs`zfs_ioc_pool_tryimport+0x4b
              zfs`zfsdev_ioctl+0x144
              genunix`cdev_ioctl+0x1d
              specfs`spec_ioctl+0x50
              genunix`fop_ioctl+0x1a
              genunix`ioctl+0xac
              unix`_sys_sysenter_post_swapgs+0x139
                4
        3        3
              zfs`vdev_root_state_change+0x40
              zfs`vdev_propagate_state+0x8f
              zfs`vdev_set_state+0xa8
              zfs`vdev_mirror_state_change+0x32
              zfs`vdev_propagate_state+0x8f
              zfs`vdev_set_state+0xa8
              zfs`vdev_load+0x69
              zfs`vdev_load+0x25
              zfs`vdev_load+0x25
              zfs`spa_load+0x487
              zfs`spa_tryimport+0x87
              zfs`zfs_ioc_pool_tryimport+0x4b
              zfs`zfsdev_ioctl+0x144
              genunix`cdev_ioctl+0x1d
              specfs`spec_ioctl+0x50
              genunix`fop_ioctl+0x1a
              genunix`ioctl+0xac
              unix`_sys_sysenter_post_swapgs+0x139
                4
        3        2
              zfs`vdev_load+0x96
              zfs`vdev_load+0x25
              zfs`spa_load+0x487
              zfs`spa_tryimport+0x87
              zfs`zfs_ioc_pool_tryimport+0x4b
              zfs`zfsdev_ioctl+0x144
              genunix`cdev_ioctl+0x1d
              specfs`spec_ioctl+0x50
              genunix`fop_ioctl+0x1a
              genunix`ioctl+0xac
              unix`_sys_sysenter_post_swapgs+0x139
                5
        3        3
              zfs`vdev_root_state_change+0x40
              zfs`vdev_propagate_state+0x8f
              zfs`vdev_set_state+0xa8
              zfs`vdev_mirror_state_change+0x45
              zfs`vdev_propagate_state+0x8f
              zfs`vdev_set_state+0xa8
              zfs`vdev_load+0x69
              zfs`vdev_load+0x25
              zfs`vdev_load+0x25
              zfs`spa_load+0x487
              zfs`spa_tryimport+0x87
              zfs`zfs_ioc_pool_tryimport+0x4b
              zfs`zfsdev_ioctl+0x144
              genunix`cdev_ioctl+0x1d
              specfs`spec_ioctl+0x50
              genunix`fop_ioctl+0x1a
              genunix`ioctl+0xac
              unix`_sys_sysenter_post_swapgs+0x139
                5
        3        3
              zfs`vdev_mirror_state_change+0x45
              zfs`vdev_propagate_state+0x8f
              zfs`vdev_set_state+0xa8
              zfs`vdev_load+0x69
              zfs`vdev_load+0x25
              zfs`vdev_load+0x25
              zfs`spa_load+0x487
              zfs`spa_tryimport+0x87
              zfs`zfs_ioc_pool_tryimport+0x4b
              zfs`zfsdev_ioctl+0x144
              genunix`cdev_ioctl+0x1d
              specfs`spec_ioctl+0x50
              genunix`fop_ioctl+0x1a
              genunix`ioctl+0xac
              unix`_sys_sysenter_post_swapgs+0x139
                5
        4        0
              zfs`vdev_mirror_state_change+0x32
              zfs`vdev_propagate_state+0x8f
              zfs`vdev_set_state+0xa8
              zfs`vdev_load+0x69
              zfs`vdev_load+0x25
              zfs`vdev_load+0x25
              zfs`spa_load+0x487
              zfs`spa_tryimport+0x87
              zfs`zfs_ioc_pool_tryimport+0x4b
              zfs`zfsdev_ioctl+0x144
              genunix`cdev_ioctl+0x1d
              specfs`spec_ioctl+0x50
              genunix`fop_ioctl+0x1a
              genunix`ioctl+0xac
              unix`_sys_sysenter_post_swapgs+0x139
                5
        3        2
              zfs`vdev_load+0x69
              zfs`vdev_load+0x25
              zfs`vdev_load+0x25
              zfs`spa_load+0x487
              zfs`spa_tryimport+0x87
              zfs`zfs_ioc_pool_tryimport+0x4b
              zfs`zfsdev_ioctl+0x144
              genunix`cdev_ioctl+0x1d
              specfs`spec_ioctl+0x50
              genunix`fop_ioctl+0x1a
              genunix`ioctl+0xac
              unix`_sys_sysenter_post_swapgs+0x139
               10
[root at solaris1 ~]$

Eric Schrock

2007-Mar-01 00:49 UTC

head link

[zfs-discuss] FAULTED ZFS volume even though it is mirrored

On Thu, Mar 01, 2007 at 10:46:02AM +1000, Stuart Low
wrote:> Heya,
> 
> > The label looks sane.  Can you try running:
> 
> Not sure if I should be reassured by that but I''ll hold my hopes
> high. :)
> 
> > # dtrace -n vdev_set_state:entry''{@[args[2], args[3],
stack()] = count()}''
> > While executing ''zpool import'' and send the output? 
Can you also send
> > ''::dis'' output (from ''mdb -k'') of
the function immediately above
> > vdev_set_state() in the above stacks?  I think the function should be
> > vdev_validate(), but I don''t remember if it''s the
same in the ZFS
> > version you have.
> 
> Attached below. Forgive my ignorance but I don''t understand what
you''re
> requesting with regard to ''mdb -k''?
Sorry.  Try ''echo vdev_load::dis | mdb -k''.  This will give
the
disassembly for vdev_load() in your current kernel (which will help us
pinpoint what vdev_load+0x69 is really doing).

- Eric

--
Eric Schrock, Solaris Kernel Development       http://blogs.sun.com/eschrock

Stuart Low

2007-Mar-01 00:50 UTC

head link

[zfs-discuss] FAULTED ZFS volume even though it is mirrored

Heya,
> Sorry.  Try ''echo vdev_load::dis | mdb -k''.  This will
give the
> disassembly for vdev_load() in your current kernel (which will help us
> pinpoint what vdev_load+0x69 is really doing).
Ahh, thanks for that.

Attached.

Stuart

---

[root at solaris1 ~]$ echo vdev_load::dis | mdb -k
vdev_load:                      pushq  %rbp
vdev_load+1:                    movq   %rsp,%rbp
vdev_load+4:                    pushq  %r12
vdev_load+6:                    movq   %rdi,%r12
vdev_load+9:                    pushq  %rbx
vdev_load+0xa:                  xorl   %ebx,%ebx
vdev_load+0xc:                  cmpq   $0x0,0x68(%rdi)
vdev_load+0x11:                 je     +0x1e    <vdev_load+0x2f>
vdev_load+0x13:                 xorl   %edx,%edx
vdev_load+0x15:                 movq   0x60(%r12),%rax
vdev_load+0x1a:                 incl   %ebx
vdev_load+0x1c:                 movq   (%rax,%rdx,8),%rdi
vdev_load+0x20:                 call   -0x20    <vdev_load>
vdev_load+0x25:                 movslq %ebx,%rdx
vdev_load+0x28:                 cmpq   0x68(%r12),%rdx
vdev_load+0x2d:                 jb     -0x18    <vdev_load+0x15>
vdev_load+0x2f:                 cmpq   %r12,0x50(%r12)
vdev_load+0x34:                 je     +0x3a    <vdev_load+0x6e>
vdev_load+0x36:                 movq   0x38(%r12),%rax
vdev_load+0x3b:                 movl   0x40(%rax),%r8d
vdev_load+0x3f:                 testl  %r8d,%r8d
vdev_load+0x42:                 jne    +0x7     <vdev_load+0x49>
vdev_load+0x44:                 popq   %rbx
vdev_load+0x45:                 popq   %r12
vdev_load+0x47:                 leave
vdev_load+0x48:                 ret
vdev_load+0x49:                 movq   %r12,%rdi
vdev_load+0x4c:                 call   -0x35c   <vdev_dtl_load>
vdev_load+0x51:                 testl  %eax,%eax
vdev_load+0x53:                 je     -0xf     <vdev_load+0x44>
vdev_load+0x55:                 movq   %r12,%rdi
vdev_load+0x58:                 movl   $0x2,%ecx
vdev_load+0x5d:                 movl   $0x3,%edx
vdev_load+0x62:                 xorl   %esi,%esi
vdev_load+0x64:                 call   +0xb7c   <vdev_set_state>
vdev_load+0x69:                 popq   %rbx
vdev_load+0x6a:                 popq   %r12
vdev_load+0x6c:                 leave
vdev_load+0x6d:                 ret
vdev_load+0x6e:                 cmpq   $0x0,0x20(%r12)
vdev_load+0x74:                 je     +0xe     <vdev_load+0x82>
vdev_load+0x76:                 cmpq   $0x0,0x18(%r12)
vdev_load+0x7c:                 nop
vdev_load+0x80:                 jne    +0x18    <vdev_load+0x98>
vdev_load+0x82:                 movl   $0x2,%ecx
vdev_load+0x87:                 movl   $0x3,%edx
vdev_load+0x8c:                 xorl   %esi,%esi
vdev_load+0x8e:                 movq   %r12,%rdi
vdev_load+0x91:                 call   +0xb4f   <vdev_set_state>
vdev_load+0x96:                 jmp    -0x60    <vdev_load+0x36>
vdev_load+0x98:                 xorl   %esi,%esi
vdev_load+0x9a:                 movq   %r12,%rdi
vdev_load+0x9d:                 call   -0xefd   <vdev_metaslab_init>
vdev_load+0xa2:                 testl  %eax,%eax
vdev_load+0xa4:                 je     -0x6e    <vdev_load+0x36>
vdev_load+0xa6:                 jmp    -0x24    <vdev_load+0x82>
[root at solaris1 ~]$

Eric Schrock

2007-Mar-01 00:55 UTC

head link

[zfs-discuss] FAULTED ZFS volume even though it is mirrored

On Thu, Mar 01, 2007 at 10:50:28AM +1000, Stuart Low
wrote:> Heya,
> 
> > Sorry.  Try ''echo vdev_load::dis | mdb -k''.  This
will give the
> > disassembly for vdev_load() in your current kernel (which will help us
> > pinpoint what vdev_load+0x69 is really doing).
> 
> Ahh, thanks for that.
> 
Hmmm.  This would indicate that vdev_dtl_load() is failing, which
suggests that something vital got corrupted to the point where
dmu_bonus_hold() or space_map_load() is failing.  I don''t know exactly
how this is supposed to work, or how exactly to debug from
here, so I''ll let one of the other ZFS experts chime in...

- Eric

--
Eric Schrock, Solaris Kernel Development       http://blogs.sun.com/eschrock

Stuart Low

2007-Mar-01 02:43 UTC

head link

[zfs-discuss] FAULTED ZFS volume even though it is mirrored

Heya,
> Hmmm.  This would indicate that vdev_dtl_load() is failing, which
> suggests that something vital got corrupted to the point where
> dmu_bonus_hold() or space_map_load() is failing.  I don''t know
exactly
> how this is supposed to work, or how exactly to debug from
> here, so I''ll let one of the other ZFS experts chime in...
:( Bummer.

I look forward to their reply (hopefully I''ll get one). In the meantime
what I was going to try was prepping a more recent install of Solaris 10
(ie. 11/06) on another host and see if it had any further
''info'' to give
me. :-/

Thanks again!

Stuart

Stuart Low

2007-Mar-01 03:25 UTC

head link

[zfs-discuss] FAULTED ZFS volume even though it is mirrored

Further to this, I''ve considered doing the following:

1) Doing a zpool destroy on the volume
2) Doing a zpool import -D on the volume

It would appear to me that primarily what has occurred is one or all of
the metadata stores ZFS has created have become corrupt? Will a zpool
import -D ignore metadata and rebuild using some magic foo?

I still don''t understand how even if an entire array became broken
these
issues were then synced to the backup array.

Confused and bewildered,

Stuart :)

On Wed, 2007-02-28 at 16:55 -0800, Eric Schrock wrote:> On Thu, Mar 01, 2007 at 10:50:28AM +1000, Stuart Low wrote:
> > Heya,
> > 
> > > Sorry.  Try ''echo vdev_load::dis | mdb -k''. 
This will give the
> > > disassembly for vdev_load() in your current kernel (which will
help us
> > > pinpoint what vdev_load+0x69 is really doing).
> > 
> > Ahh, thanks for that.
> > <snip>

Jeff Bonwick

2007-Mar-01 08:19 UTC

head link

[zfs-discuss] FAULTED ZFS volume even though it is mirrored

> However, I logged in this morning to discover that the ZFS volume could
> not be read. In addition, it appears to have marked all drives, mirrors
> & the volume itself as ''corrupted''.
One possibility: I''ve seen this happen when a system doesn''t
shut down
cleanly after the last change to the pool configuration.  In this case,
what can happen is that the boot archive (an annoying implementation
detail of the new boot architecture) can be out of date relative to
your pool.  In particular, the stale boot archive may contain an old
version of /etc/zfs/zpool.cache, which confuses the initial pool open.

The workaround for this is simple enough: export the pool and then
import it.  Assuming this works, you can fix the stupid boot archive
is by running ''bootadm update-archive''.  Please let us know if
this helps.

Jeff

Stuart Low

2007-Mar-01 08:32 UTC

head link

[zfs-discuss] FAULTED ZFS volume even though it is mirrored

Hi Jeff,
> One possibility: I''ve seen this happen when a system
doesn''t shut down
> cleanly after the last change to the pool configuration.  In this case,
> what can happen is that the boot archive (an annoying implementation
> detail of the new boot architecture) can be out of date relative to
> your pool.  In particular, the stale boot archive may contain an old
> version of /etc/zfs/zpool.cache, which confuses the initial pool open.
> The workaround for this is simple enough: export the pool and then
> import it.  Assuming this works, you can fix the stupid boot archive
> is by running ''bootadm update-archive''.  Please let us
know if this helps.
On the newly deployed system I don''t even have a zpool.cache.
I''ve
already tried the zpool export then zpool import but to no avail. :(

Thanks for the help!

Stuart

Robert Milkowski

2007-Mar-01 11:49 UTC

head link

[zfs-discuss] FAULTED ZFS volume even though it is mirrored

Hello Stuart,

Thursday, March 1, 2007, 4:25:14 AM, you wrote:

SL> Further to this, I''ve considered doing the following:

SL> 1) Doing a zpool destroy on the volume
SL> 2) Doing a zpool import -D on the volume

SL> It would appear to me that primarily what has occurred is one or all of
SL> the metadata stores ZFS has created have become corrupt? Will a zpool
SL> import -D ignore metadata and rebuild using some magic foo?

It won''t help. zpool import -D doesn''t do anything special
compared to
standard import - it just ignores flag indicating pool was destroyed.

-- 
Best regards,
 Robert                            mailto:rmilkowski at task.gda.pl
                                       http://milek.blogspot.com

Stuart Low

2007-Mar-01 13:02 UTC

head link

[zfs-discuss] FAULTED ZFS volume even though it is mirrored

Heya,
> SL> 1) Doing a zpool destroy on the volume
> SL> 2) Doing a zpool import -D on the volume
> SL> It would appear to me that primarily what has occurred is one or all
of
> SL> the metadata stores ZFS has created have become corrupt? Will a
zpool
> SL> import -D ignore metadata and rebuild using some magic foo?
> It won''t help. zpool import -D doesn''t do anything
special compared to
> standard import - it just ignores flag indicating pool was destroyed.
I suspected this but thought it was worth putting out there. The other
thought I had was possibly doing a zpool destroy then a zpool create
with half of the segments (ie. just 1 of the arrays) in the same order.
Given that these are basically a concat of mirrors I wondered if this
would mean that the data would at least be partially accessible.

I suspect however that the metadb would not recognise any of the files
on the unit (ie. it''d simply assume it was ''null''
data ready for being
overwritten)? I wonder whether there''s something like ext3''s
''use
alternate superblock''? I do assume though that if the array is already
in ''FAULTED'' mode ZFS has checked and concluded that all
alternate
copies of the metadb are corrupt?

Is it not at all possible to search for all defined files and regenerate
the metadb? Utopian perhaps but I''m still completely baffled as to how
on earth ZFS managed to sync corruption to our mirror. :-/

Realistically, I wouldn''t be too concerned if I could at least
''mark''
all parts of 1 of the arrays as ''clean'' to a point where I can
at least
perform an import and sync as much data as possible off the array.
Possible?

Cheers,

Stuart

Robert Milkowski

2007-Mar-01 15:43 UTC

head link

Re[4]: FAULTED ZFS volume even though it ismirrored

Hello Stuart,

Thursday, March 1, 2007, 2:02:26 PM, you wrote:

&gt;

Heya,

&gt; SL&gt; 1) Doing a zpool destroy on the volume

&gt; SL&gt; 2) Doing a zpool import -D on the volume

&gt; SL&gt; It would appear to me that primarily what has occurred is
one or all of

&gt; SL&gt; the metadata stores ZFS has created have become corrupt?
Will a zpool

&gt; SL&gt; import -D ignore metadata and rebuild using some magic foo?

&gt; It won''t help. zpool import -D doesn''t do anything
special compared to

&gt; standard import - it just ignores flag indicating pool was destroyed.

I suspected this but thought it was worth putting out there. The other

thought I had was possibly doing a zpool destroy then a zpool create

with half of the segments (ie. just 1 of the arrays) in the same order.

Given that these are basically a concat of mirrors I wondered if this

would mean that the data would at least be partially accessible.

It''s not a concat - it''s rather similar to striping.

However recreating pool on halve of the mirrors won''t help.

If you still hope to restore your data then DO NOT re-create

pool.

&gt;

I suspect however that the metadb would not recognise any of the files

on the unit (ie. it''d simply assume it was ''null''
data ready for being

overwritten)? I wonder whether there''s something like ext3''s
''use

alternate superblock''? I do assume though that if the array is already

in ''FAULTED'' mode ZFS has checked and concluded that all
alternate

copies of the metadb are corrupt?

It''s called uberblock here and during import zfs tries all valid copies
automatically.

&gt;

Is it not at all possible to search for all defined files and regenerate

the metadb? Utopian perhaps but I''m still completely baffled as to how

on earth ZFS managed to sync corruption to our mirror. :-/

Realistically, I wouldn''t be too concerned if I could at least
''mark''

all parts of 1 of the arrays as ''clean'' to a point where I can
at least

perform an import and sync as much data as possible off the array.

Possible?

It''s not possible.

Your only hope now probably is some clever ZFS developer...

-- 

Best regards,

 Robert                            mailto:rmilkowski@task.gda.pl

                                       http://milek.blogspot.com


_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Frank Cusack

2007-Mar-01 18:54 UTC

head link

[zfs-discuss] FAULTED ZFS volume even though it is mirrored

On March 1, 2007 12:19:22 AM -0800 Jeff Bonwick <Jeff.Bonwick at sun.com> 
wrote:> import it.  Assuming this works, you can fix the stupid boot archive
thank you.  i hate the boot archive.  i have just spent MANY unnecessary
hours on some machines thanks to the stupid boot archive.

(sorry, OT)

-frank

Luke Scharf

2007-Mar-01 19:15 UTC

head link

[zfs-discuss] FAULTED ZFS volume even though it is mirrored

Frank Cusack wrote:> On March 1, 2007 12:19:22 AM -0800 Jeff Bonwick <Jeff.Bonwick at
sun.com>
> wrote:
>> import it.  Assuming this works, you can fix the stupid boot archive
>
> thank you.  i hate the boot archive.  i have just spent MANY unnecessary
> hours on some machines thanks to the stupid boot archive.
>
> (sorry, OT)
This is also OT -- but what is the boot-archive, really?

Is it analogous to the initrd on Linux?

Thanks,
-Luke

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3271 bytes
Desc: S/MIME Cryptographic Signature
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20070301/cb6119aa/attachment.bin>

Dan Mick

2007-Mar-01 23:42 UTC

head link

[zfs-discuss] FAULTED ZFS volume even though it is mirrored

Luke Scharf wrote:
> This is also OT -- but what is the boot-archive, really?
> 
> Is it analogous to the initrd on Linux?
precisely.

zfs discuss - Feb 2007 - FAULTED ZFS volume even though it is mirrored

[zfs-discuss] FAULTED ZFS volume even though it is mirrored

[zfs-discuss] FAULTED ZFS volume even though it is mirrored

[zfs-discuss] FAULTED ZFS volume even though it is mirrored

[zfs-discuss] FAULTED ZFS volume even though it is mirrored

[zfs-discuss] FAULTED ZFS volume even though it is mirrored

[zfs-discuss] FAULTED ZFS volume even though it is mirrored

[zfs-discuss] FAULTED ZFS volume even though it is mirrored

[zfs-discuss] FAULTED ZFS volume even though it is mirrored

[zfs-discuss] FAULTED ZFS volume even though it is mirrored

[zfs-discuss] FAULTED ZFS volume even though it is mirrored

[zfs-discuss] FAULTED ZFS volume even though it is mirrored

[zfs-discuss] FAULTED ZFS volume even though it is mirrored

[zfs-discuss] FAULTED ZFS volume even though it is mirrored

[zfs-discuss] FAULTED ZFS volume even though it is mirrored

Re[4]: FAULTED ZFS volume even though it ismirrored

[zfs-discuss] FAULTED ZFS volume even though it is mirrored

[zfs-discuss] FAULTED ZFS volume even though it is mirrored

[zfs-discuss] FAULTED ZFS volume even though it is mirrored