Good day ZFS-Discuss, I am planning to build an array of 30 drives in a RaidZ2 configuration with two hot spares. However I read on the internet that this was not ideal. So I ask those who are more experianced then me, what configuration would you recommend with ZFS, I would like to have some redundancy but still keeping as much disk space open for my uses as possible. I don''t want to mirror 15 drives to 15 drives as that would drastically affect my storage capacity. Thank you for your time, Dan
Dan Saul wrote:> Good day ZFS-Discuss, > > I am planning to build an array of 30 drives in a RaidZ2 configuration > with two hot spares. However I read on the internet that this was not > ideal. > > So I ask those who are more experianced then me, what configuration > would you recommend with ZFS, I would like to have some redundancy but > still keeping as much disk space open for my uses as possible. > > I don''t want to mirror 15 drives to 15 drives as that would > drastically affect my storage capacity.There are hundreds of possible combinations of 30 drives. It really comes down to a trade-off of space vs performance vs RAS. http://blogs.sun.com/relling/entry/zfs_raid_recommendations_space_performance -- richard
I care more about data integrity then performance. Of course if performance is so bad that one would not be able to, say stream a video off of it that wouldn''t be acceptable. On 6/22/07, Richard Elling <Richard.Elling at sun.com> wrote:> Dan Saul wrote: > > Good day ZFS-Discuss, > > > > I am planning to build an array of 30 drives in a RaidZ2 configuration > > with two hot spares. However I read on the internet that this was not > > ideal. > > > > So I ask those who are more experianced then me, what configuration > > would you recommend with ZFS, I would like to have some redundancy but > > still keeping as much disk space open for my uses as possible. > > > > I don''t want to mirror 15 drives to 15 drives as that would > > drastically affect my storage capacity. > > There are hundreds of possible combinations of 30 drives. > It really comes down to a trade-off of space vs performance vs RAS. > http://blogs.sun.com/relling/entry/zfs_raid_recommendations_space_performance > -- richard >
Dan Saul wrote:> I care more about data integrity then performance. Of course if > performance is so bad that one would not be able to, say stream a > video off of it that wouldn''t be acceptable.The model I used in this blog deals with small, random reads, not streaming workloads. In part this is because the data needed to calculate small, random read performance is readily available on disk data sheets. For streaming workloads we can calculate the media bandwidth, which will be good when you have multiple drives. But there are other limitations in the system which will ultimately cap the bandwidth, and those limitations are not expressed in data sheets. The best way is to try it. Let us know how it works. -- richard> On 6/22/07, Richard Elling <Richard.Elling at sun.com> wrote: >> Dan Saul wrote: >> > Good day ZFS-Discuss, >> > >> > I am planning to build an array of 30 drives in a RaidZ2 configuration >> > with two hot spares. However I read on the internet that this was not >> > ideal. >> > >> > So I ask those who are more experianced then me, what configuration >> > would you recommend with ZFS, I would like to have some redundancy but >> > still keeping as much disk space open for my uses as possible. >> > >> > I don''t want to mirror 15 drives to 15 drives as that would >> > drastically affect my storage capacity. >> >> There are hundreds of possible combinations of 30 drives. >> It really comes down to a trade-off of space vs performance vs RAS. >> http://blogs.sun.com/relling/entry/zfs_raid_recommendations_space_performance >> >> -- richard >>
Dan Saul wrote:> I care more about data integrity then performance. Of course if > performance is so bad that one would not be able to, say stream a > video off of it that wouldn''t be acceptable.Any config I could imagine would be able to stream several videos at once (even 10Mbit/sec 1080p HD). For very high data integrity and plenty of performance for streaming a few videos, I''d try: 3 x 9-wide raidz-2 3 x hot spare For a total of 21 disks worth of usable space. If you need a bit more space, do: 2 x 14-wide raidz-2 2 x hot spare for a total of 24 disks worth of usable space. --matt> > On 6/22/07, Richard Elling <Richard.Elling at sun.com> wrote: >> Dan Saul wrote: >> > Good day ZFS-Discuss, >> > >> > I am planning to build an array of 30 drives in a RaidZ2 configuration >> > with two hot spares. However I read on the internet that this was not >> > ideal. >> > >> > So I ask those who are more experianced then me, what configuration >> > would you recommend with ZFS, I would like to have some redundancy but >> > still keeping as much disk space open for my uses as possible. >> > >> > I don''t want to mirror 15 drives to 15 drives as that would >> > drastically affect my storage capacity. >> >> There are hundreds of possible combinations of 30 drives. >> It really comes down to a trade-off of space vs performance vs RAS. >> http://blogs.sun.com/relling/entry/zfs_raid_recommendations_space_performance >> >> -- richard >> > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Bryan Wagoner
2007-Jun-25 16:28 UTC
[zfs-discuss] Re: Suggestions on 30 drive configuration?
What is the controller setup going to look like for the 30 drives? Is it going to be fibre channel, SAS, etc. and what will be the Controller-to-Disk ratio? ~Bryan This message posted from opensolaris.org
> an array of 30 drives in a RaidZ2 configuration with two hot spares> I don''t want to mirror 15 drives to 15 drives ok, so space over speed... and are willing to toss somewhere between 4 and 15 drives for protection. raidz splits the (up to 128k) write/read recordsize into each element of the raidz set.. (ie: all drives must be touched and all must finish before the block request is complete) so with a 9 disk raid1z set that''s (8 data + 1 parity (8+1)) or 16k per disk for a full 128k write. or for a smaller 4k block, that a single 512b sector per disk. on a 26+2 raid2z set that 4k block would still use 8 disks, with the other 18 disks unneeded but allocated. so perhaps three sets of 8+2 would let three blocks be read/written to at once with a total of 6 disks for protection. but for twice the speed, six sets of 4+1 would be the same size, (same number of disks for protection) but isn''t quite as safe for its 2x speed. Rob
Rob Logan wrote:> > an array of 30 drives in a RaidZ2 configuration with two hot spares > > I don''t want to mirror 15 drives to 15 drives > > ok, so space over speed... and are willing to toss somewhere between 4 > and 15 drives for protection. > > raidz splits the (up to 128k) write/read recordsize into each element of > the raidz set.. (ie: all drives must be touched and all must finish > before the block request is complete) so with a 9 disk raid1z set that''s > (8 data + 1 parity (8+1)) or 16k per disk for a full 128k write. or for > a smaller 4k block, that a single 512b sector per disk. on a 26+2 raid2z > set that 4k block would still use 8 disks, with the other 18 disks > unneeded but allocated.It is not so easy to predict. ZFS will coalesce writes. A single transaction group may have many different writes in it. Also, raidz[12] is dynamic, and will use what it needs, unlike separate volume managers who do not have any understanding of the context of the data.> so perhaps three sets of 8+2 would let three blocks be read/written to > at once with a total of 6 disks for protection. > > but for twice the speed, six sets of 4+1 would be the same size, (same > number of disks for protection) but isn''t quite as safe for its 2x speed.Yes, need to follow your priorities, there are just too many options otherwise. -- richard
Victor Latushkin
2007-Jun-27 14:29 UTC
[zfs-discuss] Suggestions on 30 drive configuration?
Richard Elling wrote:> Rob Logan wrote: >> > an array of 30 drives in a RaidZ2 configuration with two hot spares >> > I don''t want to mirror 15 drives to 15 drives >> >> ok, so space over speed... and are willing to toss somewhere between 4 >> and 15 drives for protection. >> >> raidz splits the (up to 128k) write/read recordsize into each element of >> the raidz set.. (ie: all drives must be touched and all must finish >> before the block request is complete) so with a 9 disk raid1z set that''s >> (8 data + 1 parity (8+1)) or 16k per disk for a full 128k write. or for >> a smaller 4k block, that a single 512b sector per disk. on a 26+2 raid2z >> set that 4k block would still use 8 disks, with the other 18 disks >> unneeded but allocated. > > It is not so easy to predict. ZFS will coalesce writes. A single > transaction > group may have many different writes in it. Also, raidz[12] is dynamic, > and > will use what it needs, unlike separate volume managers who do not have any > understanding of the context of the data.There is a good slide which illustrates how stripe width is selected dynamically in RAID-Z. Please see slide 13 in this slide deck: http://www.snia.org/events/past/sdc2006/zfs_File_Systems-bonwick-moore.pdf Yes, some space may be wasted (marked by X on the slide), but there are guidelines for the number of the devices in the RAID-Z(2) vdev which allows to avoid this waste. Btw, I believe there''s no link to this presentation on opensolaris.org. unfortunately... victor>> so perhaps three sets of 8+2 would let three blocks be read/written to >> at once with a total of 6 disks for protection. >> >> but for twice the speed, six sets of 4+1 would be the same size, (same >> number of disks for protection) but isn''t quite as safe for its 2x speed. > > Yes, need to follow your priorities, there are just too many options > otherwise. > -- richard > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
I have 8 SATA on the motherboard, 4 PCI cards with 4 SATA each, one PCIe 4x sata card with two, and one PCIe 1x with two. The operating system itself will be on a hard drive attached to one ATA 100 connector. Kind of like a "poor man''s" data centre, except not that cheap... It still is estimated to come out at around 6 thousand dollars, which in retrospect for that amount of sorage these days is actually relatively good. I''ve weighed my options and I am thinking that 3 raidz2 sets is the best balance of data safety to free space. Also, in case any of you are wondering why I would need space, most of it will be HDV footage and render files. Thank you everyone who contributed here you have been of great assistance. On 6/25/07, Bryan Wagoner <bryan at wagz.tv> wrote:> What is the controller setup going to look like for the 30 drives? Is it going to be fibre channel, SAS, etc. and what will be the Controller-to-Disk ratio? > > ~Bryan > > > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >
On 28/06/2007, at 12:29 AM, Victor Latushkin wrote:>> It is not so easy to predict. ZFS will coalesce writes. A single >> transaction >> group may have many different writes in it. Also, raidz[12] is >> dynamic, and >> will use what it needs, unlike separate volume managers who do not >> have any >> understanding of the context of the data. > > There is a good slide which illustrates how stripe width is > selected dynamically in RAID-Z. Please see slide 13 in this slide > deck: > http://www.snia.org/events/past/sdc2006/zfs_File_Systems-bonwick- > moore.pdf > [...] > Btw, I believe there''s no link to this presentation on > opensolaris.org. unfortunately...Indeed. Is there any reason that the presentation at http:// www.opensolaris.org/os/community/zfs/docs/zfs_last.pdf Couldn''t be updated to the one that Victor mentions?