zfs discuss - Mar 2010 - ZFS: clarification on meaning of the autoreplace property

>From pages 29,83,86,90 and 284 of the 10/09 Solaris ZFS Administration
guide, it sounds like a disk designated as a hot spare will:
1. Automatically take the place of a bad drive when needed
2. The spare will automatically be detached back to the spare
   pool when a new device is inserted and brought up to replace the
   original compromised one.

Should this work the same way for slices?

I have four active disks in a RAID 10 configuration,
for a storage pool, and the same disks are used
for mirrored root configurations, but only
only one of the possible mirrored root slice
pairs is currently active.

I wanted to designate slices on a 5th disk as
hot spares for the two existing pools, so
after partitioning the 5th disk (#4) identical
to the four existing disks, I ran:

# zpool add rpool spare c0t4d0s0
# zpool add store1 spare c0t4d0s7
# zpool status
  pool: rpool
 state: ONLINE
 scrub: none requested
config:

        NAME          STATE     READ WRITE CKSUM
        rpool         ONLINE       0     0     0
          mirror      ONLINE       0     0     0
            c0t0d0s0  ONLINE       0     0     0
            c0t1d0s0  ONLINE       0     0     0
        spares
          c0t4d0s0    AVAIL

errors: No known data errors

  pool: store1
 state: ONLINE
 scrub: none requested
config:

        NAME          STATE     READ WRITE CKSUM
        store1        ONLINE       0     0     0
          mirror      ONLINE       0     0     0
            c0t0d0s7  ONLINE       0     0     0
            c0t1d0s7  ONLINE       0     0     0
          mirror      ONLINE       0     0     0
            c0t2d0s7  ONLINE       0     0     0
            c0t3d0s7  ONLINE       0     0     0
        spares
          c0t4d0s7    AVAIL

errors: No known data errors
--
So It looked like everything was set up how I was
hoping until I emulated a disk failure by pulling
one of the online disks. The root pool responded
how I expected, but the storage pool, on slice 7,
did not appear to perform the autoreplace:

Not too long after pulling one of the online disks:

--------------------
# zpool status
  pool: rpool
 state: DEGRADED
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using ''zpool clear'' or replace the device with
''zpool replace''.
   see: http://www.sun.com/msg/ZFS-8000-9P
 scrub: resilver in progress for 0h0m, 10.02% done, 0h5m to go
config:

        NAME            STATE     READ WRITE CKSUM
        rpool           DEGRADED     0     0     0
          mirror        DEGRADED     0     0     0
            c0t0d0s0    ONLINE       0     0     0
            spare       DEGRADED    84     0     0
              c0t1d0s0  REMOVED      0     0     0
              c0t4d0s0  ONLINE       0     0    84  329M resilvered
        spares
          c0t4d0s0      INUSE     currently in use

errors: No known data errors

  pool: store1
 state: ONLINE
 scrub: none requested
config:

        NAME          STATE     READ WRITE CKSUM
        store1        ONLINE       0     0     0
          mirror      ONLINE       0     0     0
            c0t0d0s7  ONLINE       0     0     0
            c0t1d0s7  ONLINE       0     0     0
          mirror      ONLINE       0     0     0
            c0t2d0s7  ONLINE       0     0     0
            c0t3d0s7  ONLINE       0     0     0
        spares
          c0t4d0s7    AVAIL

errors: No known data errors
--------------------
I was able to convert the state of store1 to DEGRADED by
writing to a file in that storage pool, but it always listed
the spare as available. This at the same time as showing
c0t1d0s7 as REMOVED in the same pool

Based on the manual, I expected the system to bring a
reinserted disk back on line automatically, but zpool status
still showed it as "REMOVED". To get it back on line:

# zpool detach rpool c0t4d0s0
# zpool clear rpool
# zpool clear store1

Then status showed *both* pools resilvering. So the questions are:

1. Does autoreplace work on slices, or just complete disks?
2. Is there a problem replacing a "bad" disk with the same disk
   to get the autoreplace function to work?
-- 
This message posted from opensolaris.org

Hi Dave,

I''m unclear about the autoreplace behavior with one spare that is
connected to two pools. I don''t see how it could work if the
autoreplace
property is enabled on both pools, which formats and replaces a spare
disk that might be in-use in another pool (?) Maybe I misunderstand.

1. I think autoreplace behavior might be inconsistent when a device is
removed. CR 6935332 was filed recently but is not available yet through
our public bug database.

2. The current issue with adding a spare disk to a ZFS root pool is that 
if a root pool mirror disk fails and the spare kicks in, the bootblock
is not applied automatically. We''re working on improving this
experience.

My advice would be to create a 3-way mirrored root pool until we have a
better solution for root pool spares.

3. For simplicity and ease of recovery, consider using your disks as
whole disks, even though you must use slices for the root pool.
If one disk is part of two pools and it fails, two pool are impacted. 
The beauty of ZFS is no longer having to deal with slice administration, 
except for the root pool.

I like your mirror pool configurations but I would simplify it by
converting store1 to using whole disks, and keep separate spare disks.`
One for the store1 pool, and either create a 3-way mirrored root pool
or keep a spare disk connected to the system but unconfigured.

Thanks,

Cindy



On 03/17/10 10:25, Dave Johnson wrote:
> From pages 29,83,86,90 and 284 of the 10/09 Solaris ZFS Administration
> guide, it sounds like a disk designated as a hot spare will:
> 1. Automatically take the place of a bad drive when needed
> 2. The spare will automatically be detached back to the spare
>    pool when a new device is inserted and brought up to replace the
>    original compromised one.
> 
> Should this work the same way for slices?
> 
> I have four active disks in a RAID 10 configuration,
> for a storage pool, and the same disks are used
> for mirrored root configurations, but only
> only one of the possible mirrored root slice
> pairs is currently active.
> 
> I wanted to designate slices on a 5th disk as
> hot spares for the two existing pools, so
> after partitioning the 5th disk (#4) identical
> to the four existing disks, I ran:
> 
> # zpool add rpool spare c0t4d0s0
> # zpool add store1 spare c0t4d0s7
> # zpool status
>   pool: rpool
>  state: ONLINE
>  scrub: none requested
> config:
> 
>         NAME          STATE     READ WRITE CKSUM
>         rpool         ONLINE       0     0     0
>           mirror      ONLINE       0     0     0
>             c0t0d0s0  ONLINE       0     0     0
>             c0t1d0s0  ONLINE       0     0     0
>         spares
>           c0t4d0s0    AVAIL
> 
> errors: No known data errors
> 
>   pool: store1
>  state: ONLINE
>  scrub: none requested
> config:
> 
>         NAME          STATE     READ WRITE CKSUM
>         store1        ONLINE       0     0     0
>           mirror      ONLINE       0     0     0
>             c0t0d0s7  ONLINE       0     0     0
>             c0t1d0s7  ONLINE       0     0     0
>           mirror      ONLINE       0     0     0
>             c0t2d0s7  ONLINE       0     0     0
>             c0t3d0s7  ONLINE       0     0     0
>         spares
>           c0t4d0s7    AVAIL
> 
> errors: No known data errors
> --
> So It looked like everything was set up how I was
> hoping until I emulated a disk failure by pulling
> one of the online disks. The root pool responded
> how I expected, but the storage pool, on slice 7,
> did not appear to perform the autoreplace:
> 
> Not too long after pulling one of the online disks:
> 
> --------------------
> # zpool status
>   pool: rpool
>  state: DEGRADED
> status: One or more devices has experienced an unrecoverable error.  An
>         attempt was made to correct the error.  Applications are
unaffected.
> action: Determine if the device needs to be replaced, and clear the errors
>         using ''zpool clear'' or replace the device with
''zpool replace''.
>    see: http://www.sun.com/msg/ZFS-8000-9P
>  scrub: resilver in progress for 0h0m, 10.02% done, 0h5m to go
> config:
> 
>         NAME            STATE     READ WRITE CKSUM
>         rpool           DEGRADED     0     0     0
>           mirror        DEGRADED     0     0     0
>             c0t0d0s0    ONLINE       0     0     0
>             spare       DEGRADED    84     0     0
>               c0t1d0s0  REMOVED      0     0     0
>               c0t4d0s0  ONLINE       0     0    84  329M resilvered
>         spares
>           c0t4d0s0      INUSE     currently in use
> 
> errors: No known data errors
> 
>   pool: store1
>  state: ONLINE
>  scrub: none requested
> config:
> 
>         NAME          STATE     READ WRITE CKSUM
>         store1        ONLINE       0     0     0
>           mirror      ONLINE       0     0     0
>             c0t0d0s7  ONLINE       0     0     0
>             c0t1d0s7  ONLINE       0     0     0
>           mirror      ONLINE       0     0     0
>             c0t2d0s7  ONLINE       0     0     0
>             c0t3d0s7  ONLINE       0     0     0
>         spares
>           c0t4d0s7    AVAIL
> 
> errors: No known data errors
> --------------------
> I was able to convert the state of store1 to DEGRADED by
> writing to a file in that storage pool, but it always listed
> the spare as available. This at the same time as showing
> c0t1d0s7 as REMOVED in the same pool
> 
> Based on the manual, I expected the system to bring a
> reinserted disk back on line automatically, but zpool status
> still showed it as "REMOVED". To get it back on line:
> 
> # zpool detach rpool c0t4d0s0
> # zpool clear rpool
> # zpool clear store1
> 
> Then status showed *both* pools resilvering. So the questions are:
> 
> 1. Does autoreplace work on slices, or just complete disks?
> 2. Is there a problem replacing a "bad" disk with the same disk
>    to get the autoreplace function to work?

> Hi Dave,
> 
> I''m unclear about the autoreplace behavior with one
> spare that is
> connected to two pools. I don''t see how it could work
> if the autoreplace 
> property is enabled on both pools, which formats and
> replaces a spare
Because I already partitioned the disk into slices. Then
I indicated the proper slice as the spare.
> disk that might be in-use in another pool (?) Maybe I
> misunderstand.
> 
> 1. I think autoreplace behavior might be inconsistent
> when a device is
> removed. CR 6935332 was filed recently but is not
> available yet through
> our public bug database.
> 
> 2. The current issue with adding a spare disk to a
> ZFS root pool is that 
> if a root pool mirror disk fails and the spare kicks
> in, the bootblock
> is not applied automatically. We''re working on
> improving this
> experience.
While the bootblock may not have been applied automatically,
the root pool did show resilvering, but the storage pool
did not (at least per the status report)
> 
> My advice would be to create a 3-way mirrored root
> pool until we have a
> better solution for root pool spares.
That would be sort of a different topic. I''m just interested
in understanding the functionality of the hot spare at this
point.
> 
> 3. For simplicity and ease of recovery, consider
> using your disks as
> whole disks, even though you must use slices for the
> root pool.
I can''t do this with a RAID 10 configuration on the
storage pool, and a mirrored root pool. I only have
places for 5 disks on a 2RU/ 3.5" drive server
> If one disk is part of two pools and it fails, two
> pool are impacted. 
Yes. This is why I used slices instead of a whole disk
for the hot spare.
> The beauty of ZFS is no longer having to deal with
> slice administration, 
> except for the root pool.
> 
> I like your mirror pool configurations but I would
> simplify it by
> converting store1 to using whole disks, and keep
> separate spare disks.`
I would have done that from the beginning with more
chassis space.
> One for the store1 pool, and either create a 3-way
> mirrored root pool
> or keep a spare disk connected to the system but
> unconfigured.
I still need confirmation on whether the hot spare function
will work with slices. I saw no errors when executing the commands
for the hot spare slices, but I got this funny response when I ran the 
test
> 
Dave
-- 
This message posted from opensolaris.org