I know this is potentially a loaded question, but what is generally considered the optimal disk configuration for ZFS. I have 48 disks on 2 RAID controllers (2x24). The RAID controller can do RAID 0/1/5/6/10/50/60 or JBOD. What is generally considered the optimal configuration for the disks. JBOD, RAID zvols on both controllers. -- John
The answer depends on your goals: space, performance, reliability To me, optimal is best performance and reliability so use: - JBOD - ZFS mirrored pool of 22x2 + 2 spares - Mirror the disk pairs across both controllers Let ZFS protect your data. Cindy On 07/21/10 15:10, John Andrunas wrote:> I know this is potentially a loaded question, but what is generally > considered the optimal disk configuration for ZFS. I have 48 disks on > 2 RAID controllers (2x24). The RAID controller can do RAID > 0/1/5/6/10/50/60 or JBOD. What is generally considered the optimal > configuration for the disks. JBOD, RAID zvols on both controllers. >
On 7/21/2010 2:38 PM, Cindy Swearingen wrote:> The answer depends on your goals: space, performance, reliability > > To me, optimal is best performance and reliability so use: > > - JBOD > - ZFS mirrored pool of 22x2 + 2 spares > - Mirror the disk pairs across both controllers > > Let ZFS protect your data. > > Cindy > > > > On 07/21/10 15:10, John Andrunas wrote: >> I know this is potentially a loaded question, but what is generally >> considered the optimal disk configuration for ZFS. I have 48 disks on >> 2 RAID controllers (2x24). The RAID controller can do RAID >> 0/1/5/6/10/50/60 or JBOD. What is generally considered the optimal >> configuration for the disks. JBOD, RAID zvols on both controllers. >> > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discussAs Cindy said. Also, consider that you don''t need to put everything into a single disk pool, so if you have different requirements for different data, remember you can break your disks into different configurations in different pools. And, of course, balance members of pools across the two controllers. For example: I need both performance for home directories, and lots of space for backups, so I''d go with: zpool1: 8-disk mirrored setup (4x2) zpool2: 2 x 7-disk RAIDZ2 (I get slightly paranoid, so if you are less so, you could do RAIDZ1) hot spares: 2 Result: 4-disk-space''s worth of high-performance, 10 disk-space''s worth of just storage. Oh, and just ignore the RAID controller''s configuration. Enable the NVRAM cache on the controllers, but otherwise, run the disks as either JBOD (if the controller allows use of the NVRAM for a JBOD config), or a 1-disk stripes (if it requires arrays to use the NVRAM). -- Erik Trimble Java System Support Mailstop: usca22-123 Phone: x17195 Santa Clara, CA
> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- > bounces at opensolaris.org] On Behalf Of John Andrunas > > I know this is potentially a loaded question, but what is generally > considered the optimal disk configuration for ZFS. I have 48 disks on > 2 RAID controllers (2x24). The RAID controller can do RAID > 0/1/5/6/10/50/60 or JBOD. What is generally considered the optimal > configuration for the disks. JBOD, RAID zvols on both controllers.No matter what your goals are, you should JBOD the disks, and let ZFS manage the raid. It not only performs better than hardware raid, it''s also more reliable. When you scrub, ZFS will be able to access all the bytes on all the disks. But if you had something like a hardware mirror, the hardware would only present a single device to the OS, and therefore the OS wouldn''t be able to scrub all the bits on both sides of the mirror. If there''s a data error encountered in hardware mirror, ZFS has no 2nd device to read from to correct the error. All it can do is blindly retry the hardware, and hope for a different answer. Which probably isn''t going to happen. Generally speaking, your choices are: Stripe, Mirror, Stripe & Mirror, or Raid(z,z2,z3) Generally speaking, a simple stripe gets you the highest performance and capacity per dollar. But no redundancy, so your whole pool is at risk. Very few scenarios can accept this sacrifice of reliability to gain the incremental performance. Generally speaking, a n-way mirror writes at the speed of 1 disk, reads at the speed of n-disks, and costs (n)x a single disk, for the capacity of a single disk. Generally speaking, a stripe of mirrors (suppose 3-way stripe of 2-way mirrors, totaling 6 disks) writes 3 times faster than a single disk, reads 6x faster than a single disk, costs 6 disks, and has capacity of 3 disks. Generally speaking, raid has the capacity of n-1, n-2, or n-3 disks depending on which config you choose. It performs just as fast (n-1, n-2, or n-3) for large sequential operations. But the performance for small random operations is poor. Maybe 1 disk or 2 disks performance. Raid is usually what you use when you need high capacity, and you need redundancy, and you''re budget constrained. You are forced to sacrifice some speed. You could have gotten better performance by striping mirrors, but then you''d have to spend more money on disks to have the same usable space.
Hi all, That''s what i have, so i''m probably on the good track :) Basically i have a Sun X4240 with 2 Sun HBA''s attached to 2 Sun J4400 , each of them with 12 SATA 1TB disks. The configuration is - ZFS mirrored pool with 22x2 +2 spares , with 1 disk on Jbod A attached to HBA A and the other disk attached to Jbod B and HBA B - rpool mirrored with internal 146GB 10k 2.5'''' SAS disks - 2 ZIL devices 32GB mirrored - 2 log cache devices with 64 GB each - multipath on all So far...? Just blazing performance for a workload that includes home directories hosting and Citrix XenServer VM''s hosting all over NFS with link-aggregation (1Gbx2). Bruno On 21-7-2010 23:38, Cindy Swearingen wrote:> The answer depends on your goals: space, performance, reliability > > To me, optimal is best performance and reliability so use: > > - JBOD > - ZFS mirrored pool of 22x2 + 2 spares > - Mirror the disk pairs across both controllers > > Let ZFS protect your data. > > Cindy > > > > On 07/21/10 15:10, John Andrunas wrote: >> I know this is potentially a loaded question, but what is generally >> considered the optimal disk configuration for ZFS. I have 48 disks on >> 2 RAID controllers (2x24). The RAID controller can do RAID >> 0/1/5/6/10/50/60 or JBOD. What is generally considered the optimal >> configuration for the disks. JBOD, RAID zvols on both controllers. >> > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >-- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
On Wed, Jul 21, 2010 at 7:43 PM, Edward Ned Harvey <shill at nedharvey.com> wrote:>> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- >> bounces at opensolaris.org] On Behalf Of John Andrunas >> >> I know this is potentially a loaded question, but what is generally >> considered the optimal disk configuration for ZFS. ?I have 48 disks on >> 2 RAID controllers (2x24). ?The RAID controller can do RAID >> 0/1/5/6/10/50/60 or JBOD. ?What is generally considered the optimal >> configuration for the disks. ?JBOD, RAID zvols on both controllers. > > No matter what your goals are, you should JBOD the disks, and let ZFS manage > the raid. ?It not only performs better than hardware raid, it''s also more > reliable. ?When you scrub, ZFS will be able to access all the bytes on all > the disks. ?But if you had something like a hardware mirror, the hardware > would only present a single device to the OS, and therefore the OS wouldn''t > be able to scrub all the bits on both sides of the mirror. ?If there''s a > data error encountered in hardware mirror, ZFS has no 2nd device to read > from to correct the error. ?All it can do is blindly retry the hardware, and > hope for a different answer. ?Which probably isn''t going to happen. > > Generally speaking, your choices are: > Stripe, Mirror, Stripe & Mirror, or Raid(z,z2,z3)You forgot "stripe & raidz", aka a pool with multiple raidz vdevs.> Generally speaking, raid has the capacity of n-1, n-2, or n-3 disks > depending on which config you choose. ?It performs just as fast (n-1, n-2, > or n-3) for large sequential operations. ?But the performance for small > random operations is poor. ?Maybe 1 disk or 2 disks performance. ?Raid is > usually what you use when you need high capacity, and you need redundancy, > and you''re budget constrained. ?You are forced to sacrifice some speed. ?You > could have gotten better performance by striping mirrors, but then you''d > have to spend more money on disks to have the same usable space.And, adding multiple raidz vdevs (each with under 10 disks) to a single pool (aka stripe of raidz) will give better performance than a single large raidz vdev. -- Freddie Cash fjwcash at gmail.com