zfs discuss - Apr 2009 - Importing zpool after one side of mirror was destroyed

Hello all,

I have a customer asking the following question.  Does anyone have any 
suggestions? I have searched SunSolve
and have not had any luck finding documentation to assist this customer. 
Any assistance would be greatly appreciated!

Thank you in advance,
Chris

-------------------------Customer''s problem 
statement----------------------------------------
I''m unable to import a mirrored ZFS pool.  The system was shut down
with
a functional mirrored ZFS pool using c1t0d0s6 and c1t1d0s6.  Solaris was 
then re-installed from DVD media using c1t0d0 as the root disk.  The 
root disk was repartitioned during install, which destroyed slice 6 
formerly used as a Zpool mirror component.  I need to recover data from 
the mirrored Zpool using the one remaining mirror.

Based on this behavior, I''m also concerned that a system with a
mirrored
ZFS root disk could be difficult to recover.  Example:  Server X boots 
from mirrored zpool "rpool".  Server X is shutdown for maintenance,
and
1 of the 2 disks in the mirrored Zpool fails when the server is powered 
back on.  Now the mirrored zpool is unable to be imported and the server 
fails to boot.  What is the recovery procedure?

# zpool import
  pool: local
      id: xxxxxxxxxxxxxxx
 state: UNAVAIL
status: The pool is formatted using an older on-disk version.
actiion: The pool cannot be imported due to damaged devices or data.
config:

       local               UNAVAIL  insufficient replicas
          mirror          UNAVAIL  corrupted data
             c1d0s6     ONLINE
             c1t1d0s6  ONLINE

# zpool import -o failmode=continue -f local
cannot import ''local'': invalid vdev configuratiion

# zdb -l /dev/rdsk/c1t1d0s6
---------------------------------------------------------------------
LABEL 0 
---------------------------------------------------------------------
   version=4
   name=''local''
   state=0
   txg=4
   pool_guid=xxxxxxxxx
   top_guid=xxxxxxxxxx
   guid=xxxxxxxxx
   vdev_tree
         type=''mirror''
         id=0
         guid=xxxxxxxx
         metaslab_array=14
         metaslab_shift=31
         ashift=9
         asize=xxxxxxxxxxxx
         children[0]
                 type=''disk''
                 id=0
                 guid=xxxxxxxxxxxx
                 path=''/dev/dsk/c1d0s6''
                 devid=''id1,cmdk at xxxxxxxxxxxxxxx/g''
                 whole_disk=0
         children[1]
                 type=''disk''
                 id=1
                 guid=xxxxxxxxxxxx
                 path=''/dev/dsk/c2d0s6''
                 devid=''id1,cmdk at xxxxxxxxxxxxxxx/g''
                 whole_disk=0

<  --- The remaining 3 labels on this disk are also valid --- >

-- 
Christopher West, OS Administration
Email: Christopher.West at sun.com
Phone: 1-800-USA-4SUN
My Working Hours : 8am-5pm MT, Monday thru Friday
My Manager <Email>: Michael Ventimiglia <venti at sun.com>
==========================================TO REACH THE NEXT AVAILABLE ENGINEER:
1. Call 1-800-USA-4SUN choose opt 2 and enter your SR number.
2. Wait for my voice mail message to begin.
3. Press "0" during my message to reach the next available engineer.
4. You will hear hold music until the next engineer answers.
===========================================Submit, check and update tickets at
http://www.sun.com/osc
===========================================This email may contain confidential
and privileged material for the sole
use of the intended recipient. Any review or distribution by others is
strictly prohibited. If you are not the intended recipient please
contact the sender and delete all copies.
============================================

Chris,

I believe the trouble is stemming from the fact both devices in the pool
are seen as available, but as customer noted one side of the mirror was
destroyed by the install.

If the untouched device were alone then the pool should be listed as
being in a degraded state as with a mirror the pool should survive a
single lost device.   Then the pool should at least import and allow
access to the data.

I set up a mirrored root ZFS system and one of the devices on boot takes
longer to probe than the other so is listed as unavailable and the root
pool is listed as degraded but never has the system not booted because
of this.

I originally setup my ZFS root partitions incorrectly by trying to
assign the entire disk (c#t#d#) to the zpool.   This worked as I was
migrating from UFS to ZFS but after getting both devices listed as
entire disks  could not boot the system because EFI labeling was in play
and currently an EFI labeled disk cannot be booted from.   I then
booted  to the Update 6 media and imported the pool and detached one of
the devices then used format to relabel the detached device to SMI
assigning slice 1 the entire disk.  Then imported the single device pool
again and attached the new SMI device which resilvered from the original
device.  I added the boot blocks for ZFS via the install boot command. 
After the resilver finished I detached the remaining EFI labeled device
and used format to create the SMI label and assigned the single slice
the entire disk.  Then attached that device to the pool and again added
the ZFS boot blocks to the new device and waited for the resilver to
complete.  Then tested by booting system and everything came back as
expected.

So the management of a mirrored ZFS root pools was not that difficult. 
Making sure the correct device was detached and fixed and attached later
was the toughest part.

So in customers you hope that the pool in its current configuration does
not get imported as the resilver most likely would finish the wipe of
the good data.   As customer has the OS loaded on c1t0d0 boot from OS
media with this device removed so it is not found issue the "zpool
import" and the pool should be listed as degraded with the expected
device listed as unavailable.  Then issue "zpool import -f -R /mnt
<zpool ID or name>"  and verify the data still exists.  Then issue a
"zpool detach <missing device> <pool name>" so you no
longer have a
mirror, and export the pool.

At this point boot from disk into single user mode and move the
/etc/zfs/zpool.cache file to a different name.   Finish the boot into
multi user and issue the "zpool import" the pool should be seen and
available.  Then finish the "zpool import <pool name or ID>" and
if you
want to make a mirror again issue the "zpool attach <pool name>
<device>
<new device>".

This what I believe would get out of the trouble seen.

Geoff


On Tue, 2009-04-07 at 13:59, Christopher West wrote:
> Hello all,
> 
> I have a customer asking the following question.  Does anyone have any
> suggestions? I have searched SunSolve
> and have not had any luck finding documentation to assist this
> customer. 
> Any assistance would be greatly appreciated!
> 
> Thank you in advance,
> Chris
> 
> -------------------------Customer''s problem 
> statement----------------------------------------
> I''m unable to import a mirrored ZFS pool.  The system was shut
down
> with 
> a functional mirrored ZFS pool using c1t0d0s6 and c1t1d0s6.  Solaris
> was 
> then re-installed from DVD media using c1t0d0 as the root disk.  The 
> root disk was repartitioned during install, which destroyed slice 6 
> formerly used as a Zpool mirror component.  I need to recover data
> from 
> the mirrored Zpool using the one remaining mirror.
> 
> Based on this behavior, I''m also concerned that a system with a
> mirrored 
> ZFS root disk could be difficult to recover.  Example:  Server X boots
> from mirrored zpool "rpool".  Server X is shutdown for
maintenance,
> and 
> 1 of the 2 disks in the mirrored Zpool fails when the server is
> powered 
> back on.  Now the mirrored zpool is unable to be imported and the
> server 
> fails to boot.  What is the recovery procedure?
> 
> # zpool import
>   pool: local
>       id: xxxxxxxxxxxxxxx
>  state: UNAVAIL
> status: The pool is formatted using an older on-disk version.
> actiion: The pool cannot be imported due to damaged devices or data.
> config:
> 
>        local               UNAVAIL  insufficient replicas
>           mirror          UNAVAIL  corrupted data
>              c1d0s6     ONLINE
>              c1t1d0s6  ONLINE
> 
> # zpool import -o failmode=continue -f local
> cannot import ''local'': invalid vdev configuratiion
> 
> # zdb -l /dev/rdsk/c1t1d0s6
> ---------------------------------------------------------------------
> LABEL 0 
> ---------------------------------------------------------------------
>    version=4
>    name=''local''
>    state=0
>    txg=4
>    pool_guid=xxxxxxxxx
>    top_guid=xxxxxxxxxx
>    guid=xxxxxxxxx
>    vdev_tree
>          type=''mirror''
>          id=0
>          guid=xxxxxxxx
>          metaslab_array=14
>          metaslab_shift=31
>          ashift=9
>          asize=xxxxxxxxxxxx
>          children[0]
>                  type=''disk''
>                  id=0
>                  guid=xxxxxxxxxxxx
>                  path=''/dev/dsk/c1d0s6''
>                  devid=''id1,cmdk at xxxxxxxxxxxxxxx/g''
>                  whole_disk=0
>          children[1]
>                  type=''disk''
>                  id=1
>                  guid=xxxxxxxxxxxx
>                  path=''/dev/dsk/c2d0s6''
>                  devid=''id1,cmdk at xxxxxxxxxxxxxxx/g''
>                  whole_disk=0
> 
> <  --- The remaining 3 labels on this disk are also valid --- >
> 
> -- 
> Christopher West, OS Administration
> Email: Christopher.West at sun.com
> Phone: 1-800-USA-4SUN
> My Working Hours : 8am-5pm MT, Monday thru Friday
> My Manager <Email>: Michael Ventimiglia <venti at sun.com>
> ==========================================> TO REACH THE NEXT AVAILABLE
ENGINEER:
> 1. Call 1-800-USA-4SUN choose opt 2 and enter your SR number.
> 2. Wait for my voice mail message to begin.
> 3. Press "0" during my message to reach the next available
engineer.
> 4. You will hear hold music until the next engineer answers.
> ===========================================> Submit, check and update
tickets at http://www.sun.com/osc
> ===========================================> This email may contain
confidential and privileged material for the
> sole
> use of the intended recipient. Any review or distribution by others is
> strictly prohibited. If you are not the intended recipient please
> contact the sender and delete all copies.
> ===========================================> 
> 
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
-- 
Geoff Shipman - (303) 272-9955
Systems Technology Service Center - Operating System
Solaris and Network Technology Domain
Americas Systems Technology Service Center

>>>>> "gs" == Geoff Shipman <Geoff.Shipman at
Sun.COM> writes:
    gs> At this point boot from disk into single user mode and move
    gs> the /etc/zfs/zpool.cache file to a different name.

and these days ``boot single-user'''' seems to often mean 
''boot -m milestone=none''.  The old ''boot -s''
will, AFAICT, still read
zpool.cache and ``patiently wait'''' for some kinds of devices
to
appear, forever (although apparently not for the other half of your
mirror to come up, Geoff).  anyway I ususally now use this incantation
instead of boot -s without thinking.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 304 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090408/bfec2100/attachment.bin>