I''m playing around with ZFS and want to figure out the best use of my 6x 300GB SATA drives. The purpose of the drives is to store all of my data at home (video, photos, music, etc). I''m debating between: 6x 300GB disks in a single raidz2 pool --or-- 2x (3x 300GB disks in a pool) mirrored I''ve read up a lot on ZFS, but I can''t really figure out which is ideal. I know that both ways will work but it seems like the 3 disk strip in a 2 way mirror is the most flexible as far as adding drives. Any help is appreciated. Running Solaris 10 with latest patches. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20070617/a67d536d/attachment.html>
Joe S wrote:> I''m playing around with ZFS and want to figure out the best use of my > 6x 300GB SATA drives. The purpose of the drives is to store all of my > data at home (video, photos, music, etc). I''m debating between: > > 6x 300GB disks in a single raidz2 pool > > --or-- > > 2x (3x 300GB disks in a pool) mirrored >I currently use 4x 2 way mirrors, which for my workload gave the best performance/capacity ratio. It probably is the easiest configuration to expand if required. Ian
Ok. That helps. I''m trying to wrap my brain around the best way to do this. I''m going to try to duplicate what you are doing and see how that works out. I''m going to create 3x 2-way mirrors. I guess I don''t really *need* the raidz at this point. My biggest concern with raidz is getting locked into a configuration i can''t grow out of. I like the idea of adding more 2 way mirrors to a pool. On 6/17/07, Ian Collins <ian at ianshome.com> wrote:> > Joe S wrote: > > I''m playing around with ZFS and want to figure out the best use of my > > 6x 300GB SATA drives. The purpose of the drives is to store all of my > > data at home (video, photos, music, etc). I''m debating between: > > > > 6x 300GB disks in a single raidz2 pool > > > > --or-- > > > > 2x (3x 300GB disks in a pool) mirrored > > > I currently use 4x 2 way mirrors, which for my workload gave the best > performance/capacity ratio. It probably is the easiest configuration to > expand if required. > > Ian > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20070617/3503cf4a/attachment.html>
Casper.Dik at Sun.COM
2007-Jun-17 10:27 UTC
[zfs-discuss] 6 disk raidz2 or 3 stripe 2 way mirror
>Ok. That helps. I''m trying to wrap my brain around the best way to do this. >I''m going to try to duplicate what you are doing and see how that works out. >I''m going to create 3x 2-way mirrors. I guess I don''t really *need* the >raidz at this point. My biggest concern with raidz is getting locked into a >configuration i can''t grow out of. I like the idea of adding more 2 way >mirrors to a pool.Just as with a mirror you can add a raidz stripe: zpool add tank raidz c1t0d0 c1t1d0 c1t2d0 Casper
Marc Bevand
2007-Jun-17 10:47 UTC
[zfs-discuss] Re: 6 disk raidz2 or 3 stripe 2 way mirror
Joe S <js.lists <at> gmail.com> writes:> > I''m going to create 3x 2-way mirrors. I guess I don''t really *need* the > raidz at this point. My biggest concern with raidz is getting locked into > a configuration i can''t grow out of. I like the idea of adding more > 2 way mirrors to a pool.The raidz2 option will *not* restrict your possibilities of expansion. For example, it is perfectly possible to add a mirror to a pool consisting of single raidz2 vdev. Plus, compared to 3 2-disk mirrors, a 6-disk raidz2 offers more usable space and is more reliable. -marc
My understanding is that if I create a 6 disk raidz2, # zpool create tank raidz2 disk0 disk1 disk2 disk3 disk4 disk5 I cannot add more disks to this set. I cannot expand this. I have to destroy the raidz2 it and recreate it if I want to increase capacity. Whereas with 2 way mirrors, i can just keep adding pairs of drives, or replace old 2 way pairs with new larger capacity pairs. At first this raidz stuff was really simple, until I thought about adding more drives or replacing the old drives with larger drives in the future. I''m trying to make this scale in the future, but now it looks like I''m not understanding raidz. On 6/17/07, Marc Bevand <m.bevand at gmail.com> wrote:> > Joe S <js.lists <at> gmail.com> writes: > > > > I''m going to create 3x 2-way mirrors. I guess I don''t really *need* the > > raidz at this point. My biggest concern with raidz is getting locked > into > > a configuration i can''t grow out of. I like the idea of adding more > > 2 way mirrors to a pool. > > The raidz2 option will *not* restrict your possibilities of expansion. > > For example, it is perfectly possible to add a mirror to a pool consisting > of single raidz2 vdev. > > Plus, compared to 3 2-disk mirrors, a 6-disk raidz2 offers more usable > space > and is more reliable. > > -marc > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20070617/4b491e69/attachment.html>
John-Paul Drawneek
2007-Jun-17 15:42 UTC
[zfs-discuss] Re: 6 disk raidz2 or 3 stripe 2 way mirror
ok if its just storing stuff raidz2 is probably the best use of space. raidz2 on 5 disk and one spare - this can take 3 disk failing before you lose your data. The three strip mirror will give you nice performance but from the sounds of it you don''t need it. This message posted from opensolaris.org
After getting a few replies on this, I''ve realized the following: If I have 3 disks in a raidz vdev and I want to add more storage in the future, I would need add another 3 disk raidz vdev. If I have 4 disks in a raidz2 vdev and I want to add more storage in the future, I would need add another 4 disk raidz2 vdev. If I have multiple 2 disk mirrors in a pool and I want to add more storage in the future, I would need to add another 2 disk mirror. Mirroring seems to be my least favorite option since it uses up so much space. I could also mix and match the above, but I think I will get confused later on down the road. ;-) My goal is to have some sort of fault tolerance for this data, but I know I will need to grow it someday as drives only last so long and larger drives will be more affordable and readily available. Thanks for the replies so far. I want to do this right the first time. I didn''t realize the task of choosing what kind fault tolerance to use in ZFS would be this difficult! But I am very impressed with how quickly and easily I can destroy and recreate pools. On 6/17/07, Joe S <js.lists at gmail.com> wrote:> > I''m playing around with ZFS and want to figure out the best use of my 6x > 300GB SATA drives. The purpose of the drives is to store all of my data at > home (video, photos, music, etc). I''m debating between: > > 6x 300GB disks in a single raidz2 pool > > --or-- > > 2x (3x 300GB disks in a pool) mirrored > > > I''ve read up a lot on ZFS, but I can''t really figure out which is ideal. I > know that both ways will work but it seems like the 3 disk strip in a 2 way > mirror is the most flexible as far as adding drives. Any help is > appreciated. > > Running Solaris 10 with latest patches. > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20070617/eb4dbc49/attachment.html>
Joe S wrote:> I''m playing around with ZFS and want to figure out the best use of my 6x > 300GB SATA drives. The purpose of the drives is to store all of my data > at home (video, photos, music, etc). I''m debating between: > > 6x 300GB disks in a single raidz2 pool > > --or-- > > 2x (3x 300GB disks in a pool) mirrored > > > I''ve read up a lot on ZFS, but I can''t really figure out which is ideal. > I know that both ways will work but it seems like the 3 disk strip in a > 2 way mirror is the most flexible as far as adding drives. Any help is > appreciated.This is becoming one of the most frequently asked questions. The good news is that you have options. The bad news is that you have options. Deciding which option is best forces you to prioritize, and different people value different features. I''ve put together a series of blogs which you might find useful to help make such decisions (space vs. RAS vs. performance) http://blogs.sun.com/relling/entry/zfs_raid_recommendations_space_performance1 -- richard
Ah, so you are the Richard behind those articles I''ve been mulling over! :-) You blog posts helped me to realize there was much more forethought required when setting up my ZFS pool(s). I''m glad I''m not the only person with this question. Whatever I decide, I will include in this thread. On 6/17/07, Richard Elling <Richard.Elling at sun.com> wrote:> > Joe S wrote: > > I''m playing around with ZFS and want to figure out the best use of my 6x > > 300GB SATA drives. The purpose of the drives is to store all of my data > > at home (video, photos, music, etc). I''m debating between: > > > > 6x 300GB disks in a single raidz2 pool > > > > --or-- > > > > 2x (3x 300GB disks in a pool) mirrored > > > > > > I''ve read up a lot on ZFS, but I can''t really figure out which is ideal. > > I know that both ways will work but it seems like the 3 disk strip in a > > 2 way mirror is the most flexible as far as adding drives. Any help is > > appreciated. > > This is becoming one of the most frequently asked questions. > The good news is that you have options. The bad news is that > you have options. Deciding which option is best forces you to > prioritize, and different people value different features. I''ve > put together a series of blogs which you might find useful to > help make such decisions (space vs. RAS vs. performance) > > http://blogs.sun.com/relling/entry/zfs_raid_recommendations_space_performance1 > > -- richard >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20070617/9ea880c2/attachment.html>
I''m going to try 5 disks in raidz2 with 1 hot spare. I read about this here: http://prefetch.net/blog/index.php/2007/02/09/using-raidz2-and-hot-spares-on-older-sun-storage-arrays/ I don''t have older disks, but they are consumer grade disks, and I''ve been bitten by disks going dead before, thus this whole thread. On 6/17/07, Joe S <js.lists at gmail.com> wrote:> > Ah, so you are the Richard behind those articles I''ve been mulling over! > :-) > You blog posts helped me to realize there was much more forethought > required when setting up my ZFS pool(s). > > I''m glad I''m not the only person with this question. > > Whatever I decide, I will include in this thread. > > > > On 6/17/07, Richard Elling <Richard.Elling at sun.com > wrote: > > > > Joe S wrote: > > > I''m playing around with ZFS and want to figure out the best use of my > > 6x > > > 300GB SATA drives. The purpose of the drives is to store all of my > > data > > > at home (video, photos, music, etc). I''m debating between: > > > > > > 6x 300GB disks in a single raidz2 pool > > > > > > --or-- > > > > > > 2x (3x 300GB disks in a pool) mirrored > > > > > > > > > I''ve read up a lot on ZFS, but I can''t really figure out which is > > ideal. > > > I know that both ways will work but it seems like the 3 disk strip in > > a > > > 2 way mirror is the most flexible as far as adding drives. Any help is > > > appreciated. > > > > This is becoming one of the most frequently asked questions. > > The good news is that you have options. The bad news is that > > you have options. Deciding which option is best forces you to > > prioritize, and different people value different features. I''ve > > put together a series of blogs which you might find useful to > > help make such decisions (space vs. RAS vs. performance) > > > > http://blogs.sun.com/relling/entry/zfs_raid_recommendations_space_performance1 > > > > -- richard > > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20070617/0efc3cb0/attachment.html>
Darren Dunham
2007-Jun-17 20:22 UTC
[zfs-discuss] Re: 6 disk raidz2 or 3 stripe 2 way mirror
> After getting a few replies on this, I''ve realized the following: > > If I have 3 disks in a raidz vdev and I want to add more storage in the > future, I would need add another 3 disk raidz vdev.This is not a constraint that ZFS forces on you.> If I have 4 disks in a raidz2 vdev and I want to add more storage in the > future, I would need add another 4 disk raidz2 vdev. > If I have multiple 2 disk mirrors in a pool and I want to add more storage > in the future, I would need to add another 2 disk mirror.And neither are these. The configuration of any vdev that you create does not constrain you with any vdevs you want to add to the pool in the future. You can start with any of your three choices above and then add any of the other three to the same pool. Now some additions may not make sense for what you want to do, but they are available. -- Darren Dunham ddunham at taos.com Senior Technical Consultant TAOS http://www.taos.com/ Got some Dr Pepper? San Francisco, CA bay area < This line left intentionally blank to confuse you. >
Brian Hechinger
2007-Jun-17 23:30 UTC
[zfs-discuss] Re: 6 disk raidz2 or 3 stripe 2 way mirror
On Sun, Jun 17, 2007 at 01:22:34PM -0700, Darren Dunham wrote:> > The configuration of any vdev that you create does not constrain you > with any vdevs you want to add to the pool in the future. You can start > with any of your three choices above and then add any of the other three > to the same pool.''zpool add'' will complain if the number of disks aren''t equal, forcing you to use -f to get it to work. I wonder why this is? Does this have to do with the way zfs determines what goes where? Does whatever system that is used work better with similar vdevs?> Now some additions may not make sense for what you want to do, but they > are available.I setup a test zfs sytem at work for some of the other non-solaris savvy linux admins and it had a 6x36g raidz, a 6x73g raidz and a 143g mirror. It''s not something we pushed hard, so I don''t know what the performance impact would be, but it certainly did work. :) Which raises the question..... What are the performance implications of pairing different "types" of vdevs in a single pool? -brian -- "Perl can be fast and elegant as much as J2EE can be fast and elegant. In the hands of a skilled artisan, it can and does happen; it''s just that most of the shit out there is built by people who''d be better suited to making sure that my burger is cooked thoroughly." -- Jonathan Patschke
Darren Dunham
2007-Jun-18 15:57 UTC
[zfs-discuss] Re: 6 disk raidz2 or 3 stripe 2 way mirror
> > The configuration of any vdev that you create does not constrain you > > with any vdevs you want to add to the pool in the future. You can start > > with any of your three choices above and then add any of the other three > > to the same pool. > > ''zpool add'' will complain if the number of disks aren''t equal, forcing you > to use -f to get it to work. I wonder why this is? Does this have to do > with the way zfs determines what goes where? Does whatever system that is > used work better with similar vdevs?I think it''s mainly to keep you from doing something silly without meaning to. If you have the same type and columns, then you have the same availability expectations. If instead you take a 5 disk raidz and add a 2 disk stripe, you''re dramatically changing the availability expectations. So the force is required.> > Now some additions may not make sense for what you want to do, but they > > are available. > > I setup a test zfs sytem at work for some of the other non-solaris savvy > linux admins and it had a 6x36g raidz, a 6x73g raidz and a 143g mirror. > It''s not something we pushed hard, so I don''t know what the performance > impact would be, but it certainly did work. :) > > Which raises the question..... > > What are the performance implications of pairing different "types" of > vdevs in a single pool?It''s going to try to stripe across them without regard for performance today. So one especially slow vdev (maybe a single disk?) could slow the overall performance for a pool that has other faster components. -- Darren Dunham ddunham at taos.com Senior Technical Consultant TAOS http://www.taos.com/ Got some Dr Pepper? San Francisco, CA bay area < This line left intentionally blank to confuse you. >
Brian Hechinger
2007-Jun-18 16:37 UTC
[zfs-discuss] Re: 6 disk raidz2 or 3 stripe 2 way mirror
On Mon, Jun 18, 2007 at 08:57:29AM -0700, Darren Dunham wrote:> > I think it''s mainly to keep you from doing something silly without > meaning to.That''s certainly a good reason. :)> If you have the same type and columns, then you have the same > availability expectations. If instead you take a 5 disk raidz and add a > 2 disk stripe, you''re dramatically changing the availability > expectations. So the force is required.Certainly, but adding a mirror doesn''t degrade the availability expectations, so it''s a fairly safe addition.> It''s going to try to stripe across them without regard for performance > today. So one especially slow vdev (maybe a single disk?) could slow > the overall performance for a pool that has other faster components.Ok, so performance will be based on the slowest vdev for the most part. I can live with that. ;) I''m sure this is far from simple, but it would be interesting to see zfs able to figure out the relative performance characteristics of each of its vdevs and spread the load accordingly. :) -brian -- "Perl can be fast and elegant as much as J2EE can be fast and elegant. In the hands of a skilled artisan, it can and does happen; it''s just that most of the shit out there is built by people who''d be better suited to making sure that my burger is cooked thoroughly." -- Jonathan Patschke
Darren Dunham
2007-Jun-18 20:28 UTC
[zfs-discuss] Re: 6 disk raidz2 or 3 stripe 2 way mirror
> > If you have the same type and columns, then you have the same > > availability expectations. If instead you take a 5 disk raidz and add a > > 2 disk stripe, you''re dramatically changing the availability > > expectations. So the force is required. > > Certainly, but adding a mirror doesn''t degrade the availability expectations, > so it''s a fairly safe addition.Nonetheless, it''s different. It does change the cost expectations, so you could consider that it''s a warning for that reason then.> I''m sure this is far from simple, but it would be interesting to see zfs > able to figure out the relative performance characteristics of each of > its vdevs and spread the load accordingly. :)There are quite a few ideas around that. Simply the ability to recognize "slow" storage would be a good idea. Then you can make other decisions (lower it for write priority? temporarily not use it for writes? Decide it''s so bad that you should fail it?....) -- Darren Dunham ddunham at taos.com Senior Technical Consultant TAOS http://www.taos.com/ Got some Dr Pepper? San Francisco, CA bay area < This line left intentionally blank to confuse you. >
Thanks for all the comments. Very helpful. I have another question. The six disk raidz2 pool works, but I noticed in Richard Elling''s blog that a raidz/raidz2 pool has the read performance of a single drive (unless I misread something). What if I create 2x three disk raidz vdevs and put them in a pool? That would be like RAID 5-0 right? Any disadvantage to doing this? It looks like I''ll get the read performance of 2 disks. -joe On 6/17/07, Joe S <js.lists at gmail.com> wrote:> > I''m playing around with ZFS and want to figure out the best use of my 6x > 300GB SATA drives. The purpose of the drives is to store all of my data at > home (video, photos, music, etc). I''m debating between: > > 6x 300GB disks in a single raidz2 pool > > --or-- > > 2x (3x 300GB disks in a pool) mirrored > > > I''ve read up a lot on ZFS, but I can''t really figure out which is ideal. I > know that both ways will work but it seems like the 3 disk strip in a 2 way > mirror is the most flexible as far as adding drives. Any help is > appreciated. > > Running Solaris 10 with latest patches. > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20070618/6fc132b4/attachment.html>
Ian Collins
2007-Jun-19 04:21 UTC
[zfs-discuss] Re: 6 disk raidz2 or 3 stripe 2 way mirror
Joe S wrote:> Thanks for all the comments. Very helpful. > > I have another question. The six disk raidz2 pool works, but I noticed > in Richard Elling''s blog that a raidz/raidz2 pool has the read > performance of a single drive (unless I misread something). What if I > create 2x three disk raidz vdevs and put them in a pool? That would be > like RAID 5-0 right? Any disadvantage to doing this? It looks like > I''ll get the read performance of 2 disks. >Try it! Ian
Will Murnane
2007-Jun-19 04:29 UTC
[zfs-discuss] Re: 6 disk raidz2 or 3 stripe 2 way mirror
On 6/19/07, Joe S <js.lists at gmail.com> wrote:> Thanks for all the comments. Very helpful. > > I have another question. The six disk raidz2 pool works, but I noticed in > Richard Elling''s blog that a raidz/raidz2 pool has the read performance of a > single drive (unless I misread something). What if I create 2x three disk > raidz vdevs and put them in a pool? That would be like RAID 5-0 right? Any > disadvantage to doing this? It looks like I''ll get the read performance of 2 > disks.If you do small, random reads, then his numbers hold. If you do full-stripe reads and writes (which (for example) playing a movie does) then you need to look elsewhere for your numbers. The other thing to consider is that for most home uses, single-disk performance is plenty. Even if you only get performance like that of one disk, it''s overkill even for such demanding-sounding things as streaming four HD movies at once*. Sequential reads are plenty fast. It''s generally only when things get small and/or random that disk I/O becomes a bottleneck. Will *Blu-ray maximum mux rate is 54 megabits per second. That''s about 7 MB/s. HD-DVD is lower. x264 is lower still, generally. Disks are, generally speaking, fast enough to do a couple streaming reads at that speed.