Hello zfs-discuss,
While waiting for Thumpers to come I''m thinking how to configure
them. I would like to use raid-z. As thumper has 6 SATA controllers
each 8-port then maybe it would make sense to create raid-z groups
from 6 disks each from separate controller. Then combine 7 such
groups into one pool. Then there''re 6 disks remaining with two of
them designated for system (mirror) which leaves 4 disks probably
as hot-spares.
That way if one controller fails entire pool will still be ok.
What do you think?
ps. there still will be SPOF for boot disks and hot spares but it
looks like there''s no choice anyway.
--
Best regards,
Robert mailto:rmilkowski at task.gda.pl
http://milek.blogspot.com
Do you want data availability, data retention, space, or performance? -- richard Robert Milkowski wrote:> Hello zfs-discuss, > > While waiting for Thumpers to come I''m thinking how to configure > them. I would like to use raid-z. As thumper has 6 SATA controllers > each 8-port then maybe it would make sense to create raid-z groups > from 6 disks each from separate controller. Then combine 7 such > groups into one pool. Then there''re 6 disks remaining with two of > them designated for system (mirror) which leaves 4 disks probably > as hot-spares. > > That way if one controller fails entire pool will still be ok. > > What do you think? > > ps. there still will be SPOF for boot disks and hot spares but it > looks like there''s no choice anyway. >
Hello Richard,
Friday, October 13, 2006, 8:05:18 AM, you wrote:
REP> Do you want data availability, data retention, space, or performance?
data availability, space, performance
However we''re talking about quite a lot of small IOs (r+w).
The real question was what do you think about creating each raid group
only from disks from different controllers so controller failure won''t
affect data availability.
--
Best regards,
Robert mailto:rmilkowski at task.gda.pl
http://milek.blogspot.com
Robert Milkowski wrote:> Hello Richard, > > Friday, October 13, 2006, 8:05:18 AM, you wrote: > > REP> Do you want data availability, data retention, space, or performance? > > data availability, space, performance > > However we''re talking about quite a lot of small IOs (r+w).Then you should seriously consider using mirrors.> The real question was what do you think about creating each raid group > only from disks from different controllers so controller failure won''t > affect data availability.On thumper, where the controllers (and cables, etc) are integrated into the system board, controller failure is extremely unlikely. These controllers are much more reliable than your traditional SCSI card in a PCI slot. In fact, most controller "failures" are due to SCSI bus negotiation problems (confused devices, bad cables, etc), which simply don''t exist in the point-to-point (ie. SCSI, SAS) world. So I wouldn''t worry very much about spreading across controllers for the sake of controller failure. --matt
I don''t think controllers really fail that often, but anything that increases redundancy is likely to be an improvement. I would hope that the controllers used in Thumper mostly keep their channels independent from a PCI point of view, so that your pools don?t collide. (It does mean that each read or write is going to require using all PCI buses, but since the total available bandwidth is the same under any configuration, you should be OK.) Note that controllers have also been known to fail in such a way that they disable the attached PCI bus. Thumper has an independent bus for each controller (nice!), so as long as the bridges themselves can handle this failure mode (and they are supposed to), you should be OK even against this. This message posted from opensolaris.org
Hello Matthew, Friday, October 13, 2006, 5:37:45 PM, you wrote: MA> Robert Milkowski wrote:>> Hello Richard, >> >> Friday, October 13, 2006, 8:05:18 AM, you wrote: >> >> REP> Do you want data availability, data retention, space, or performance? >> >> data availability, space, performance >> >> However we''re talking about quite a lot of small IOs (r+w).MA> Then you should seriously consider using mirrors. ''coz of space requirements that''s not possible. I hope RAID-Z will do.>> The real question was what do you think about creating each raid group >> only from disks from different controllers so controller failure won''t >> affect data availability.MA> On thumper, where the controllers (and cables, etc) are integrated into MA> the system board, controller failure is extremely unlikely. These MA> controllers are much more reliable than your traditional SCSI card in a MA> PCI slot. In fact, most controller "failures" are due to SCSI bus MA> negotiation problems (confused devices, bad cables, etc), which simply MA> don''t exist in the point-to-point (ie. SCSI, SAS) world. So I wouldn''t MA> worry very much about spreading across controllers for the sake of MA> controller failure. That''s a good point. -- Best regards, Robert mailto:rmilkowski at task.gda.pl http://milek.blogspot.com
Robert Milkowski wrote:> Hello Richard, > > Friday, October 13, 2006, 8:05:18 AM, you wrote: > > REP> Do you want data availability, data retention, space, or performance? > > data availability, space, performanceThumper is not designed for high data availability, more traditional RAID arrays are fully redundant. Thumper is designed for long data retention, which is a slightly different design point. Thumper''s controllers and processors can be replaced without affecting the data, but the data will be inaccessible during such procedures.> However we''re talking about quite a lot of small IOs (r+w). > > The real question was what do you think about creating each raid group > only from disks from different controllers so controller failure won''t > affect data availability.I have modeled this and have the predicted reliability data. In a nutshell, don''t worry about the controllers. As Matt pointed out, each disk has a direct, unshared connection to the controller and each controller is on its own PCI bus. These are highly reliable components, much more reliable than the disks, power supplies, DIMMs, and fans. The choice of data redundancy is much more important for achieving a level of happiness. However, for lots of small iops, RAID-Z or RAID-Z2 will be a potential dissatisfier unless you use a large number of sets. To the first order, you can consider the performance of a RAID-Z or RAID-Z2 set for small, random reads to be equivalent to the performance of a single disk. Writes won''t be so bad under ZFS, but we don''t have a good performance model for small, random writes yet. -- richard
On Fri, 2006-10-13 at 01:05 +0200, Robert Milkowski wrote:> Hello zfs-discuss, > > While waiting for Thumpers to come I''m thinking how to configure > them. I would like to use raid-z. As thumper has 6 SATA controllers > each 8-port then maybe it would make sense to create raid-z groups > from 6 disks each from separate controller. Then combine 7 such > groups into one pool. Then there''re 6 disks remaining with two of > them designated for system (mirror) which leaves 4 disks probably > as hot-spares. > > That way if one controller fails entire pool will still be ok. > > What do you think? > > ps. there still will be SPOF for boot disks and hot spares but it > looks like there''s no choice anyway. >Base on what I have seen so far, when your thumper shows up it should have the following factory configuration, which is pretty close to your scenario above. c5t4 and c5t0 as boot disk mirror metadevices 1 pool with the following raidz groups: c0t0 c1t0 c4t0 c6t0 c7t0 c0t1 c1t1 c4t1 c5t1 c6t1 c7t1 c0t2 c1t2 c4t2 c5t2 c6t2 c7t2 c0t3 c1t3 c4t3 c5t3 c6t3 c7t3 c0t4 c1t4 c4t4 c6t4 c7t4 c0t5 c1t5 c4t5 c5t5 c6t5 c7t5 c0t6 c1t6 c4t6 c5t6 c6t6 c7t6 c0t7 c1t7 c4t7 c5t7 c6t7 c7t7