zfs discuss - Jun 2007 - 6 disk raidz2 or 3 stripe 2 way mirror

I''m playing around with ZFS and want to figure out the best use of my
6x
300GB SATA drives. The purpose of the drives is to store all of my data at
home (video, photos, music, etc). I''m debating between:

6x 300GB disks in a single raidz2 pool

--or--

2x (3x 300GB disks in a pool) mirrored


I''ve read up a lot on ZFS, but I can''t really figure out which
is ideal. I
know that both ways will work but it seems like the 3 disk strip in a 2 way
mirror is the most flexible as far as adding drives. Any help is
appreciated.

Running Solaris 10 with latest patches.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20070617/a67d536d/attachment.html>

Joe S wrote:
> I''m playing around with ZFS and want to figure out the best use of
my
> 6x 300GB SATA drives. The purpose of the drives is to store all of my
> data at home (video, photos, music, etc). I''m debating between:
>
> 6x 300GB disks in a single raidz2 pool
>
> --or--
>
> 2x (3x 300GB disks in a pool) mirrored
>
I currently use 4x 2 way mirrors, which for my workload gave the best
performance/capacity ratio.  It probably is the easiest configuration to
expand if required.

Ian

Ok. That helps. I''m trying to wrap my brain around the best way to do
this.
I''m going to try to duplicate what you are doing and see how that works
out.
I''m going to create 3x 2-way mirrors. I guess I don''t really
*need* the
raidz at this point. My biggest concern with raidz is getting locked into a
configuration i can''t grow out of. I like the idea of adding more 2 way
mirrors to a pool.



On 6/17/07, Ian Collins <ian at ianshome.com>
wrote:
>
> Joe S wrote:
> > I''m playing around with ZFS and want to figure out the best
use of my
> > 6x 300GB SATA drives. The purpose of the drives is to store all of my
> > data at home (video, photos, music, etc). I''m debating
between:
> >
> > 6x 300GB disks in a single raidz2 pool
> >
> > --or--
> >
> > 2x (3x 300GB disks in a pool) mirrored
> >
> I currently use 4x 2 way mirrors, which for my workload gave the best
> performance/capacity ratio.  It probably is the easiest configuration to
> expand if required.
>
> Ian
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20070617/3503cf4a/attachment.html>

>Ok. That helps. I''m trying to wrap my brain around the best way to
do this.
>I''m going to try to duplicate what you are doing and see how that
works out.
>I''m going to create 3x 2-way mirrors. I guess I don''t
really *need* the
>raidz at this point. My biggest concern with raidz is getting locked into a
>configuration i can''t grow out of. I like the idea of adding more 2
way
>mirrors to a pool.
Just as with a mirror you can add a raidz stripe:

	zpool add tank raidz c1t0d0 c1t1d0 c1t2d0

Casper

Joe S <js.lists <at> gmail.com> writes:
> 
> I''m going to create 3x 2-way mirrors. I guess I don''t
really *need* the
> raidz at this point. My biggest concern with raidz is getting locked into
> a configuration i can''t grow out of. I like the idea of adding
more
> 2 way mirrors to a pool.
The raidz2 option will *not* restrict your possibilities of expansion.

For example, it is perfectly possible to add a mirror to a pool consisting
of single raidz2 vdev.

Plus, compared to 3 2-disk mirrors, a 6-disk raidz2 offers more usable space 
and is more reliable.

-marc

My understanding is that if I create a 6 disk raidz2,

# zpool create tank raidz2 disk0 disk1 disk2 disk3 disk4 disk5

I cannot add more disks to this set. I cannot expand this. I have to destroy
the raidz2 it and recreate it if I want to increase capacity.
Whereas with 2 way mirrors, i can just keep adding pairs of drives, or
replace old 2 way pairs with new larger capacity pairs.

At first this raidz stuff was really simple, until I thought about adding
more drives or replacing the old drives with larger drives in the future.
I''m trying to make this scale in the future, but now it looks like
I''m not
understanding raidz.



On 6/17/07, Marc Bevand <m.bevand at gmail.com>
wrote:
>
> Joe S <js.lists <at> gmail.com> writes:
> >
> > I''m going to create 3x 2-way mirrors. I guess I
don''t really *need* the
> > raidz at this point. My biggest concern with raidz is getting locked
> into
> > a configuration i can''t grow out of. I like the idea of
adding more
> > 2 way mirrors to a pool.
>
> The raidz2 option will *not* restrict your possibilities of expansion.
>
> For example, it is perfectly possible to add a mirror to a pool consisting
> of single raidz2 vdev.
>
> Plus, compared to 3 2-disk mirrors, a 6-disk raidz2 offers more usable
> space
> and is more reliable.
>
> -marc
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20070617/4b491e69/attachment.html>

ok if its just storing stuff raidz2 is probably the best use of space.
raidz2 on 5 disk and one spare - this can take 3 disk failing before you lose
your data.

The three strip mirror will give you nice performance but from the sounds of it
you don''t need it.
 
 
This message posted from opensolaris.org

After getting a few replies on this, I''ve realized the following:

If I have 3 disks in a raidz vdev and I want to add more storage in the
future, I would need add another 3 disk raidz vdev.
If I have 4 disks in a raidz2 vdev and I want to add more storage in the
future, I would need add another 4 disk raidz2 vdev.
If I have multiple 2 disk mirrors in a pool and I want to add more storage
in the future, I would need to add another 2 disk mirror.

Mirroring seems to be my least favorite option since it uses up so much
space. I could also mix and match the above, but I think I will get confused
later on down the road. ;-)

My goal is to have some sort of fault tolerance for this data, but I know I
will need to grow it someday as drives only last so long and larger drives
will be more affordable and readily available.

Thanks for the replies so far. I want to do this right the first time. I
didn''t realize the task of choosing what kind fault tolerance to use in
ZFS
would be this difficult! But I am very impressed with how quickly and easily
I can destroy and recreate pools.



On 6/17/07, Joe S <js.lists at gmail.com> wrote:
>
> I''m playing around with ZFS and want to figure out the best use of
my 6x
> 300GB SATA drives. The purpose of the drives is to store all of my data at
> home (video, photos, music, etc). I''m debating between:
>
> 6x 300GB disks in a single raidz2 pool
>
> --or--
>
> 2x (3x 300GB disks in a pool) mirrored
>
>
> I''ve read up a lot on ZFS, but I can''t really figure out
which is ideal. I
> know that both ways will work but it seems like the 3 disk strip in a 2 way
> mirror is the most flexible as far as adding drives. Any help is
> appreciated.
>
> Running Solaris 10 with latest patches.
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20070617/eb4dbc49/attachment.html>

Joe S wrote:
> I''m playing around with ZFS and want to figure out the best use of
my 6x
> 300GB SATA drives. The purpose of the drives is to store all of my data 
> at home (video, photos, music, etc). I''m debating between:
> 
> 6x 300GB disks in a single raidz2 pool
> 
> --or--
> 
> 2x (3x 300GB disks in a pool) mirrored
> 
> 
> I''ve read up a lot on ZFS, but I can''t really figure out
which is ideal.
> I know that both ways will work but it seems like the 3 disk strip in a 
> 2 way mirror is the most flexible as far as adding drives. Any help is 
> appreciated.
This is becoming one of the most frequently asked questions.
The good news is that you have options.  The bad news is that
you have options.  Deciding which option is best forces you to
prioritize, and different people value different features.  I''ve
put together a series of blogs which you might find useful to
help make such decisions (space vs. RAS vs. performance)
http://blogs.sun.com/relling/entry/zfs_raid_recommendations_space_performance1

  -- richard

Ah, so you are the Richard behind those articles I''ve been mulling
over! :-)
You blog posts helped me to realize there was much more forethought required
when setting up my ZFS pool(s).

I''m glad I''m not the only person with this question.

Whatever I decide, I will include in this thread.



On 6/17/07, Richard Elling <Richard.Elling at sun.com>
wrote:
>
> Joe S wrote:
> > I''m playing around with ZFS and want to figure out the best
use of my 6x
> > 300GB SATA drives. The purpose of the drives is to store all of my
data
> > at home (video, photos, music, etc). I''m debating between:
> >
> > 6x 300GB disks in a single raidz2 pool
> >
> > --or--
> >
> > 2x (3x 300GB disks in a pool) mirrored
> >
> >
> > I''ve read up a lot on ZFS, but I can''t really figure
out which is ideal.
> > I know that both ways will work but it seems like the 3 disk strip in
a
> > 2 way mirror is the most flexible as far as adding drives. Any help is
> > appreciated.
>
> This is becoming one of the most frequently asked questions.
> The good news is that you have options.  The bad news is that
> you have options.  Deciding which option is best forces you to
> prioritize, and different people value different features.  I''ve
> put together a series of blogs which you might find useful to
> help make such decisions (space vs. RAS vs. performance)
>
>
http://blogs.sun.com/relling/entry/zfs_raid_recommendations_space_performance1
>
>   -- richard
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20070617/9ea880c2/attachment.html>

I''m going to try 5 disks in raidz2 with 1 hot spare.

I read about this here:

http://prefetch.net/blog/index.php/2007/02/09/using-raidz2-and-hot-spares-on-older-sun-storage-arrays/

I don''t have older disks, but they are consumer grade disks, and
I''ve been
bitten by disks going dead before, thus this whole thread.



On 6/17/07, Joe S <js.lists at gmail.com> wrote:
>
> Ah, so you are the Richard behind those articles I''ve been mulling
over!
> :-)
> You blog posts helped me to realize there was much more forethought
> required when setting up my ZFS pool(s).
>
> I''m glad I''m not the only person with this question.
>
> Whatever I decide, I will include in this thread.
>
>
>
> On 6/17/07, Richard Elling <Richard.Elling at sun.com > wrote:
> >
> > Joe S wrote:
> > > I''m playing around with ZFS and want to figure out the
best use of my
> > 6x
> > > 300GB SATA drives. The purpose of the drives is to store all of
my
> > data
> > > at home (video, photos, music, etc). I''m debating
between:
> > >
> > > 6x 300GB disks in a single raidz2 pool
> > >
> > > --or--
> > >
> > > 2x (3x 300GB disks in a pool) mirrored
> > >
> > >
> > > I''ve read up a lot on ZFS, but I can''t really
figure out which is
> > ideal.
> > > I know that both ways will work but it seems like the 3 disk
strip in
> > a
> > > 2 way mirror is the most flexible as far as adding drives. Any
help is
> > > appreciated.
> >
> > This is becoming one of the most frequently asked questions.
> > The good news is that you have options.  The bad news is that
> > you have options.  Deciding which option is best forces you to
> > prioritize, and different people value different features. 
I''ve
> > put together a series of blogs which you might find useful to
> > help make such decisions (space vs. RAS vs. performance)
> >
> >
http://blogs.sun.com/relling/entry/zfs_raid_recommendations_space_performance1
> >
> >   -- richard
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20070617/0efc3cb0/attachment.html>

> After getting a few replies on this, I''ve realized the following:
> 
> If I have 3 disks in a raidz vdev and I want to add more storage in the
> future, I would need add another 3 disk raidz vdev.
This is not a constraint that ZFS forces on you.
> If I have 4 disks in a raidz2 vdev and I want to add more storage in the
> future, I would need add another 4 disk raidz2 vdev.
> If I have multiple 2 disk mirrors in a pool and I want to add more storage
> in the future, I would need to add another 2 disk mirror.
And neither are these.

The configuration of any vdev that you create does not constrain you
with any vdevs you want to add to the pool in the future.  You can start
with any of your three choices above and then add any of the other three
to the same pool.

Now some additions may not make sense for what you want to do, but they
are available. 

-- 
Darren Dunham                                           ddunham at taos.com
Senior Technical Consultant         TAOS            http://www.taos.com/
Got some Dr Pepper?                           San Francisco, CA bay area
         < This line left intentionally blank to confuse you. >

On Sun, Jun 17, 2007 at 01:22:34PM -0700, Darren Dunham
wrote:
> 
> The configuration of any vdev that you create does not constrain you
> with any vdevs you want to add to the pool in the future.  You can start
> with any of your three choices above and then add any of the other three
> to the same pool.
''zpool add'' will complain if the number of disks
aren''t equal, forcing you
to use -f to get it to work.  I wonder why this is?  Does this have to do
with the way zfs determines what goes where?  Does whatever system that is
used work better with similar vdevs?
> Now some additions may not make sense for what you want to do, but they
> are available. 
I setup a test zfs sytem at work for some of the other non-solaris savvy
linux admins and it had a 6x36g raidz, a 6x73g raidz and a 143g mirror.
It''s not something we pushed hard, so I don''t know what the
performance
impact would be, but it certainly did work.  :)

Which raises the question.....

What are the performance implications of pairing different "types" of
vdevs in a single pool?

-brian
-- 
"Perl can be fast and elegant as much as J2EE can be fast and elegant.
In the hands of a skilled artisan, it can and does happen; it''s just
that most of the shit out there is built by people who''d be better
suited to making sure that my burger is cooked thoroughly."  -- Jonathan
Patschke

> > The configuration of any vdev that you create does not constrain you
> > with any vdevs you want to add to the pool in the future.  You can
start
> > with any of your three choices above and then add any of the other
three
> > to the same pool.
> 
> ''zpool add'' will complain if the number of disks
aren''t equal, forcing you
> to use -f to get it to work.  I wonder why this is?  Does this have to do
> with the way zfs determines what goes where?  Does whatever system that is
> used work better with similar vdevs?
I think it''s mainly to keep you from doing something silly without
meaning to.

If you have the same type and columns, then you have the same
availability expectations.  If instead you take a 5 disk raidz and add a
2 disk stripe, you''re dramatically changing the availability
expectations.  So the force is required.
> > Now some additions may not make sense for what you want to do, but
they
> > are available. 
> 
> I setup a test zfs sytem at work for some of the other non-solaris savvy
> linux admins and it had a 6x36g raidz, a 6x73g raidz and a 143g mirror.
> It''s not something we pushed hard, so I don''t know what
the performance
> impact would be, but it certainly did work.  :)
> 
> Which raises the question.....
> 
> What are the performance implications of pairing different
"types" of
> vdevs in a single pool?
It''s going to try to stripe across them without regard for performance
today.  So one especially slow vdev (maybe a single disk?) could slow
the overall performance for a pool that has other faster components.

-- 
Darren Dunham                                           ddunham at taos.com
Senior Technical Consultant         TAOS            http://www.taos.com/
Got some Dr Pepper?                           San Francisco, CA bay area
         < This line left intentionally blank to confuse you. >

On Mon, Jun 18, 2007 at 08:57:29AM -0700, Darren Dunham
wrote:
> 
> I think it''s mainly to keep you from doing something silly without
> meaning to.
That''s certainly a good reason. :)
> If you have the same type and columns, then you have the same
> availability expectations.  If instead you take a 5 disk raidz and add a
> 2 disk stripe, you''re dramatically changing the availability
> expectations.  So the force is required.
Certainly, but adding a mirror doesn''t degrade the availability
expectations,
so it''s a fairly safe addition.
> It''s going to try to stripe across them without regard for
performance
> today.  So one especially slow vdev (maybe a single disk?) could slow
> the overall performance for a pool that has other faster components.
Ok, so performance will be based on the slowest vdev for the most part.
I can live with that. ;)

I''m sure this is far from simple, but it would be interesting to see
zfs
able to figure out the relative performance characteristics of each of
its vdevs and spread the load accordingly. :)

-brian
-- 
"Perl can be fast and elegant as much as J2EE can be fast and elegant.
In the hands of a skilled artisan, it can and does happen; it''s just
that most of the shit out there is built by people who''d be better
suited to making sure that my burger is cooked thoroughly."  -- Jonathan
Patschke

> > If you have the same type and columns, then you have the same
> > availability expectations.  If instead you take a 5 disk raidz and add
a
> > 2 disk stripe, you''re dramatically changing the availability
> > expectations.  So the force is required.
> 
> Certainly, but adding a mirror doesn''t degrade the availability
expectations,
> so it''s a fairly safe addition.
Nonetheless, it''s different.  It does change the cost expectations, so
you could consider that it''s a warning for that reason then.  
> I''m sure this is far from simple, but it would be interesting to
see zfs
> able to figure out the relative performance characteristics of each of
> its vdevs and spread the load accordingly. :)
There are quite a few ideas around that.  Simply the ability to
recognize "slow" storage would be a good idea.  Then you can make
other
decisions (lower it for write priority?  temporarily not use it for
writes?  Decide it''s so bad that you should fail it?....)  

-- 
Darren Dunham                                           ddunham at taos.com
Senior Technical Consultant         TAOS            http://www.taos.com/
Got some Dr Pepper?                           San Francisco, CA bay area
         < This line left intentionally blank to confuse you. >

Thanks for all the comments. Very helpful.

I have another question. The six disk raidz2 pool works, but I noticed in
Richard Elling''s blog that a raidz/raidz2 pool has the read performance
of a
single drive (unless I misread something). What if I create 2x three disk
raidz vdevs and put them in a pool? That would be like RAID 5-0 right? Any
disadvantage to doing this? It looks like I''ll get the read performance
of 2
disks.


-joe

On 6/17/07, Joe S <js.lists at gmail.com> wrote:
>
> I''m playing around with ZFS and want to figure out the best use of
my 6x
> 300GB SATA drives. The purpose of the drives is to store all of my data at
> home (video, photos, music, etc). I''m debating between:
>
> 6x 300GB disks in a single raidz2 pool
>
> --or--
>
> 2x (3x 300GB disks in a pool) mirrored
>
>
> I''ve read up a lot on ZFS, but I can''t really figure out
which is ideal. I
> know that both ways will work but it seems like the 3 disk strip in a 2 way
> mirror is the most flexible as far as adding drives. Any help is
> appreciated.
>
> Running Solaris 10 with latest patches.
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20070618/6fc132b4/attachment.html>

Joe S wrote:
> Thanks for all the comments. Very helpful.
>
> I have another question. The six disk raidz2 pool works, but I noticed
> in Richard Elling''s blog that a raidz/raidz2 pool has the read
> performance of a single drive (unless I misread something). What if I
> create 2x three disk raidz vdevs and put them in a pool? That would be
> like RAID 5-0 right? Any disadvantage to doing this? It looks like
> I''ll get the read performance of 2 disks.
>
Try it!

Ian

On 6/19/07, Joe S <js.lists at gmail.com> wrote:
> Thanks for all the comments. Very helpful.
>
> I have another question. The six disk raidz2 pool works, but I noticed in
> Richard Elling''s blog that a raidz/raidz2 pool has the read
performance of a
> single drive (unless I misread something). What if I create 2x three disk
> raidz vdevs and put them in a pool? That would be like RAID 5-0 right? Any
> disadvantage to doing this? It looks like I''ll get the read
performance of 2
> disks.
If you do small, random reads, then his numbers hold.  If you do
full-stripe reads and writes (which (for example) playing a movie
does) then you need to look elsewhere for your numbers.

The other thing to consider is that for most home uses, single-disk
performance is plenty.  Even if you only get performance like that of
one disk, it''s overkill even for such demanding-sounding things as
streaming four HD movies at once*.  Sequential reads are plenty fast.
It''s generally only when things get small and/or random that disk I/O
becomes a bottleneck.

Will

*Blu-ray maximum mux rate is 54 megabits per second.  That''s about 7
MB/s.  HD-DVD is lower.  x264 is lower still, generally.  Disks are,
generally speaking, fast enough to do a couple streaming reads at that
speed.