zfs discuss - Mar 2010 - Q : recommendations for zpool configuration

Greetings

I would like to get your recommendation how setup new pool.

I have 4 new 1.5TB disks reserved to new zpool.
I planned to crow/replace existing small 4 disks ( raidz ) setup with new bigger
one.

As new pool will be bigger and will have more personally important data to be
stored long time, i like to ask your recommendations should i create recreate
pool or just replace existing devices.

I have noted there is now raidz2 and been thinking witch woul be better.
A pool with 2 mirrors or one pool with 4 disks raidz2

So at least could some explain these new raidz configurations

Thanks
-- 
This message posted from opensolaris.org

A pool with a 4-wide raidz2 is a completely nonsensical idea. It has the
same amount of accessible storage as two striped mirrors. And would be
slower in terms of IOPS, and be harder to upgrade in the future (you''d
need
to keep adding four drives for every expansion with raidz2 - with mirrors
you only need to add another two drives to the pool).

Just my $0.02

On 19 March 2010 18:28, homerun <petri.j.kunnari at gmail.com> wrote:
> Greetings
>
> I would like to get your recommendation how setup new pool.
>
> I have 4 new 1.5TB disks reserved to new zpool.
> I planned to crow/replace existing small 4 disks ( raidz ) setup with new
> bigger one.
>
> As new pool will be bigger and will have more personally important data to
> be stored long time, i like to ask your recommendations should i create
> recreate pool or just replace existing devices.
>
> I have noted there is now raidz2 and been thinking witch woul be better.
> A pool with 2 mirrors or one pool with 4 disks raidz2
>
> So at least could some explain these new raidz configurations
>
> Thanks
> --
> This message posted from opensolaris.org
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100319/76286a6b/attachment.html>

On Fri, Mar 19, 2010 at 2:34 PM, taemun <taemun at gmail.com>
wrote:
> A pool with a 4-wide raidz2 is a completely nonsensical idea. It has the
> same amount of accessible storage as two striped mirrors. And would be
> slower in terms of IOPS, and be harder to upgrade in the future
(you''d need
> to keep adding four drives for every expansion with raidz2 - with mirrors
> you only need to add another two drives to the pool).
> Just my $0.02
>
but it can survive on failure of 2 random disks in the pool.

In striped mirror:
mirror1
  diskA
  diskB
mirror2
  diskC
  diskD

In event diskA and diskB (or diskC and diskD) failed together, entire
pool is lost.

In raidz2:
raidz2-1
  diskA
  diskB
  diskC
  diskD

Any combination of 2 disks can fail at same time and the pool will still intact.


-- 
O< ascii ribbon campaign - stop html mail - www.asciiribbon.org

On Fri, Mar 19, 2010 at 06:34:50PM +1100, taemun wrote:
> A pool with a 4-wide raidz2 is a completely nonsensical idea.
No, it''s not - not completely.
> It has the same amount of accessible storage as two striped mirrors. And 
> would be slower in terms of IOPS, and be harder to upgrade in the future 
All that is true.

If those things weren''t as important to you as error recovery, raidz2
make fine sense: a 4-way raidz2 can tolerate the loss of any 2 disks.
The mirror pool may die with the loss of the wrong 2 disks.
> Just my $0.02
Cost and benefit valuation are left to the user according to their
circumstances. 

--
Dan.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100319/41663ec7/attachment.bin>

Thanks for comments

So possible choises are :

1) 2 2-way mirros
2) 4 disks raidz2

BTW , can raidz have spare ? so is there one posible choise more :
3 disks raidz with 1 spare ?

Here i prefer data availibility not performance.
And if need sometime to expand / change setup it is then that time problem....
-- 
This message posted from opensolaris.org

On Fri, Mar 19, 2010 at 12:59:39AM -0700, homerun wrote:
> Thanks for comments
> 
> So possible choises are :
> 
> 1) 2 2-way mirros
> 2) 4 disks raidz2
> 
> BTW , can raidz have spare ? so is there one posible choise more :
> 3 disks raidz with 1 spare ?
raidz2 is basically this, with a pre-silvered spare.  With an
unsilvered spare, you have no redundancy until the resilver completes,
and if there are latent errors in the remaining non-redundant disks
you may lose data. 

Other choices:

 - 4way raidz3
 - 4way mirror

Same space and fault tolerance, different performance.  This is an
easier choice, closer (but still not completely) to the nonsensical.

Another choice again:

 - 2 separate pools, each a 2-disk mirror

Data in one pool, backed up regularly by snapshot replication to the
second. Same space as a 4-way mirror, but this has tolerance to some
other kinds of problems that a single pool does not. 

Better still would be a backup pool in another machine/site. Perhaps
the disks you are replacing can go to this purpose?

--
Dan.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100319/7e59d904/attachment.bin>

Thanks !!

I will go to raidz2 , seems to be best choise for me..
Data fault tolerance vs amont of space suite for me.

Thanks all !!!
-- 
This message posted from opensolaris.org

On Fri, March 19, 2010 02:28, homerun wrote:
> Greetings
>
> I would like to get your recommendation how setup new pool.
>
> I have 4 new 1.5TB disks reserved to new zpool.
> I planned to crow/replace existing small 4 disks ( raidz ) setup with new
> bigger one.
>
> As new pool will be bigger and will have more personally important data to
> be stored long time, i like to ask your recommendations should i create
> recreate pool or just replace existing devices.
Replacing existing drives runs risks to the data -- you''re deliberately
reducing yourself to no redundancy for a while (while the resilver
happens).  It would probably be faster, and definitely safer, to back up
the data, recreate the pool, and restore the data.
> I have noted there is now raidz2 and been thinking witch woul be better.
> A pool with 2 mirrors or one pool with 4 disks raidz2
A pool with 2 mirrors will have the same available space as a 4-disk
raidz2.  It will generally perform better.

For small numbers of disks, I''m a big fan of using mirrors rather than
RAIDZ.  I''ve got an 8-disk hot-swap bay currently occupied by 3 2-disk
pairs (with 2 slots for future expansion; maybe a hot spare, and a space
to attach an additional disk during upgrades).

When expanding a vdev by replacing devices, it can be done much more
safely with a mirror than a RAIDZ group.  With a mirror, you can attach a
THIRD disk (in fact you can attach any number; one guy wrote about
creating a 47-way mirror).  So, instead of replacing one disk with a
bigger one (eliminating your redundancy during the resilver), attach the
bigger one as a third disk.  When that resilver is done, you can attach
the other new disk, if you have bay space; or detach one of the small
disks and THEN attach the other new disk.  When the second resilver is
done, detach the last small disk, and you have now increased  your mirror
vdev size without ever reducing your redundancy below 2 copies. 
There''s
no equivalent process for a RAIDZ group.
> So at least could some explain these new raidz configurations
RAIDZ is "single parity" -- one drive is redundant data.  A RAIDZ vdev
will withstand the failure of one drive without loss of data, but NOT the
failure of 2 or more.  A RAIDZ pool of N drives (all the same size) has
N-1 drives worth of available capacity.

RAIDZ2 is "double parity" -- two drives are given to redundant data. 
A
RAIDZ2 vdev will withstand the failure of one or two drives without loss
of data, but NOT the failure of 3 or more.  A RAIDZ2 pool of N drives (all
the same size) has N-2 drives worth of available capacity.

A problem with modern large drives is that they take a long time to
"resilver" in case of failure and replacement.  During that period, if
you
started with one redundant drive, you''re down to no redundant drives,
meaning that a failure during the resilver could lose your data.  (This is
one of the many reasons you should have backups *in addition* to using
redundant vdevs).  This has driven people to develop higher levels of
redundancy in parity schemes, such as RAIDZ2 (and RAIDZ3).
-- 
David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/
Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/
Photos: http://dd-b.net/photography/gallery/
Dragaera: http://dragaera.info

> I have noted there is now raidz2 and been thinking witch woul be
> better.
> A pool with 2 mirrors or one pool with 4 disks raidz2
If you use raidz2, made of 4 disks, you will have usable capacity of 2
disks, and you can tolerate any 2 disks failing.

If you use 2 mirrors, you will have a total of 4 disks and usable capacity
of 2 disks.  Your redundancy is not quite as good as above ... You could
survive a failed disk in the first mirror, and a failed disk in the second
mirror, but you could not survive two failed disks that are in the same
mirror.

If you use raidz2, your reliability might be a little bit higher.
If you use 2 mirrors, your performance will certainly be higher for random
IO operations.

So you must choose what you care about more:  Performance or reliability.
Both ways are good ways.

> A pool with a 4-wide raidz2 is a completely nonsensical idea. It has
> the same amount of accessible storage as two striped mirrors. And would
> be slower in terms of IOPS, and be harder to upgrade in the future
> (you''d need to keep adding four drives for every expansion with
raidz2
> - with mirrors you only need to add another two drives to the pool).
> 
> Just my $0.02
Here''s my $0.04:

Suppose you had 4 disks, configured as 2 mirrors.  And you want to expand by
adding another mirror.  No problem.

Suppose you had 4 disks, configured as raidz2.  And you want to expand by
adding a mirror.  No problem.

I went through this determination when setting up my pool.  I decided to go with
mirrors instead of raidz2 after considering the following:

1.  Drive capacity in my box.  At most, I can realistically cram 10 drives in my
box and I am not interested in expanding outside of the box.  I could go with
2.5 inch drives and fit a lot more, but I don''t feel the necessity to
do so.  That being said, given the historic trend for mass storage drives to
become cheaper over time, I have a feeling that I will be replacing drives to
expand storage space long before the drives themselves start failing.  The added
redundancy of raidz2 is great, but I am betting that, barring a poorly
manufactured drive, I will be replacing the drives with bigger drives before
they have a chance to reach the end of their life.

2.  Taking into account the above, it''s a great deal easier on the
pocket book to expand two drives at a time instead of four at a time.  As bigger
drives are always getting cheaper, I feel that I have a lot more flexibility
with mirrors when it comes to expanding.  If you have limitless physical space
for drives, you might feel differently.

3.  Mirrors are going to perform better than raidz.  Again, redundancy is great,
but so is performance.  My setup is for home use.  I want to keep my data safe
but at the same time I am limited by cost and space.  I think that given the
tradeoff between the two, mirrors win.  I feel that the chances of two drives in
a mirror failing simultaneously are remote enough that I''ll take the
risk.

4.  Again, I''m running this at home.  It''s not mission
critical to me to have my data available 24/7.  Redundancy is a convenience and
not a necessity.  Regardless of what you choose, backups are what will save your
ass in the event of catastrophe.  Having said that, I currently don''t
have a good backup solution and how to implement a good backup solution seems to
be a hot topic on this list lately.  Figuring out how to easily, effectively and
cheaply back up multiple terabytes of storage is my number one priority at the
moment.

So anyways, all things considered, I prefer the better performance and easier
expansion of storage space vs my physical space over a relatively small layer of
extra redundancy.  If you aren''t doing anything that necessitates the
added redundancy of raidz2, go with mirrors.  Either way, if you care about your
data, back it up.

eric
-- 
This message posted from opensolaris.org

On Sat, 20 Mar 2010, Eric Andersen wrote:
>
> 2.  Taking into account the above, it''s a great deal easier on the
> pocket book to expand two drives at a time instead of four at a 
> time.  As bigger drives are always getting cheaper, I feel that I 
> have a lot more flexibility with mirrors when it comes to expanding. 
> If you have limitless physical space for drives, you might feel 
> differently.
I agree with your arguments.  Just make sure that you have a way to 
expand a mirror pair without losing redundancy.  For example, make 
sure that there is a way to add a new device to act as the replacement 
without taking existing devices off line.  Otherwise there is some 
possibility of data loss during the replacement.

Bob
--
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/