zfs discuss - Oct 2006 - adding to a raidz pool and its discontents

we thought we would try adding a disk to an existing raidz pool
named backup:

# zpool status
...
         NAME        STATE     READ WRITE CKSUM
         backup      ONLINE       0     0     0
           raidz     ONLINE       0     0     0
             c4t2d0  ONLINE       0     0     0
		...
             c4t2d5  ONLINE       0     0     0

add the next disk, we need a bit more space:

# zpool add backup c4t2d6
# zpool status
...
         NAME        STATE     READ WRITE CKSUM
         backup      ONLINE       0     0     0
           raidz     ONLINE       0     0     0
             c4t2d0  ONLINE       0     0     0
		...
           c4t2d6    ONLINE       0     0     0


ah. looks like the new disk is not a part of the raidz set,
yet the diskspace in the pool increased appropriately. if this
pool is used as is, can it still offer the raidz guarantee?
only for the bits that happen to fall into the raidz set but
not the additional disk?

i am confused about the overall failure characteristics of such
a setup. [maybe it is a boneheaded thing to do, but one gets no
warning from zpool if it is...]

thanks

oz
-- 
ozan s. yigit | oz at somanetworks.com | http://nextbit.blogspot.com
you take a banana, you get a lunar landscape. -- j. van wijk

On 10/25/06, ozan s. yigit <oz at somanetworks.com> wrote:
> ah. looks like the new disk is not a part of the raidz set,
> yet the diskspace in the pool increased appropriately.
Yes.  A pool can hold many vdevs, an uses the space in all of them.
> if this
> pool is used as is, can it still offer the raidz guarantee?
No.
> only for the bits that happen to fall into the raidz set but
> not the additional disk?
Yes (and you can''t control or determine where in the pool a file
landed, so you might as well regard that answer as "no" for all
practical purposes).  Files will be spread across vdevs potentially.
> i am confused about the overall failure characteristics of such
> a setup. [maybe it is a boneheaded thing to do, but one gets no
> warning from zpool if it is...]
I will stop short of calling it boneheaded, but...let''s say I
can''t
think of a reason you''d want such a pool in production, other than
utter emergency shortage of diskpace coupled with shortage of places
to put new drives.

I got myself into this situation in my own early experimentation as
well.  Non-production system, just playing around, so no harm done.

It''s easy to do by mistake/confusion, and I do tend to agree that
zpool should issue a warning for that drastically differing a level of
redundancy on vdevs in a pool.
-- 
David Dyer-Bennet, <mailto:dd-b at dd-b.net>,
<http://www.dd-b.net/dd-b/>
RKBA: <http://www.dd-b.net/carry/>
Pics: <http://www.dd-b.net/dd-b/SnapshotAlbum/>
Dragaera/Steven Brust: <http://dragaera.info/>

thanks for confirming my observation vis a vis the failure characteristics
of the pool. this is strictly experimentation as well; we thought there may
be no way to extend a raidz pool seamlessly, but wanted to test to see what
zpool did.

since the fundamental safety property of the pool is blown away by the
addition, it would be useful to generate a warning and maybe require
-f or something...

oz

David Dyer-Bennet wrote:
> On 10/25/06, ozan s. yigit <oz at somanetworks.com> wrote:
> 
>> ah. looks like the new disk is not a part of the raidz set,
>> yet the diskspace in the pool increased appropriately.
> 
> Yes.  A pool can hold many vdevs, an uses the space in all of them.
> 
>> if this
>> pool is used as is, can it still offer the raidz guarantee?
> 
> No.
> 
>> only for the bits that happen to fall into the raidz set but
>> not the additional disk?
> 
> Yes (and you can''t control or determine where in the pool a file
> landed, so you might as well regard that answer as "no" for all
> practical purposes).  Files will be spread across vdevs potentially.
> 
>> i am confused about the overall failure characteristics of such
>> a setup. [maybe it is a boneheaded thing to do, but one gets no
>> warning from zpool if it is...]
> 
> I will stop short of calling it boneheaded, but...let''s say I
can''t
> think of a reason you''d want such a pool in production, other than
> utter emergency shortage of diskpace coupled with shortage of places
> to put new drives.
> 
> I got myself into this situation in my own early experimentation as
> well.  Non-production system, just playing around, so no harm done.
> 
> It''s easy to do by mistake/confusion, and I do tend to agree that
> zpool should issue a warning for that drastically differing a level of
> redundancy on vdevs in a pool.
-- 
ozan s. yigit | oz at somanetworks.com | http://nextbit.blogspot.com
you take a banana, you get a lunar landscape. -- j. van wijk

ozan s. yigit wrote:
> we thought we would try adding a disk to an existing raidz pool
> named backup:
> 
> # zpool status
> ...
>         NAME        STATE     READ WRITE CKSUM
>         backup      ONLINE       0     0     0
>           raidz     ONLINE       0     0     0
>             c4t2d0  ONLINE       0     0     0
>         ...
>             c4t2d5  ONLINE       0     0     0
> 
> add the next disk, we need a bit more space:
> 
> # zpool add backup c4t2d6
> # zpool status
> ...
>         NAME        STATE     READ WRITE CKSUM
>         backup      ONLINE       0     0     0
>           raidz     ONLINE       0     0     0
>             c4t2d0  ONLINE       0     0     0
>         ...
>           c4t2d6    ONLINE       0     0     0
> 
> 
> ah. looks like the new disk is not a part of the raidz set,
> yet the diskspace in the pool increased appropriately. if this
> pool is used as is, can it still offer the raidz guarantee?
> only for the bits that happen to fall into the raidz set but
> not the additional disk?
> 
> i am confused about the overall failure characteristics of such
> a setup. [maybe it is a boneheaded thing to do, but one gets no
> warning from zpool if it is...]
> 
What bits are you running?

I don''t think this has changed since ZFS integrated but on recent
nevada
bits you do get an error:

thumper-12tb# zpool create -f backup raidz c1t0d0 c1t1d0 c1t2d0 c1t3d0
thumper-12tb# zpool status
   pool: backup
  state: ONLINE
  scrub: none requested
config:

         NAME        STATE     READ WRITE CKSUM
         backup      ONLINE       0     0     0
           raidz1    ONLINE       0     0     0
             c1t0d0  ONLINE       0     0     0
             c1t1d0  ONLINE       0     0     0
             c1t2d0  ONLINE       0     0     0
             c1t3d0  ONLINE       0     0     0

errors: No known data errors

thumper-12tb# zpool add backup c7t7d0
invalid vdev specification
use ''-f'' to override the following errors:
mismatched replication level: pool uses raidz and new vdev is disk

thumper-12tb# uname -a
SunOS thumper-12tb 5.11 onnv-gate:2006-10-23 i86pc i386 i86pc
thumper-12tb#

Did you use the -f flag when you added the single disk vdev?

eric

eric kustarz wrote [in part]
> What bits are you running?
s10r2.
> thumper-12tb# zpool add backup c7t7d0
> invalid vdev specification
> use ''-f'' to override the following errors:
> mismatched replication level: pool uses raidz and new vdev is disk
interesting. that helps.
> Did you use the -f flag when you added the single disk vdev?
nope. thanks for the demo from nevada build.
> 
> eric
-- 
ozan s. yigit | oz at somanetworks.com | http://nextbit.blogspot.com
you take a banana, you get a lunar landscape. -- j. van wijk