I'm setting up this huge RAID 6 box. I've always thought of hot spares,
but I'm reading things that are comparing RAID 5 with a hot spare to RAID
6, implying that the latter doesn't need one. I *certainly* have enough
drives to spare in this RAID box: 42 of 'em, so two questions: should I
assign one or more hot spares, and, if so, how many?
mark
>From: "m.roth at 5-cent.us" <m.roth at 5-cent.us>>To: CentOS mailing list <centos at centos.org> >Sent: Thursday, April 11, 2013 8:36 AM >Subject: [CentOS] RAID 6 - opinions> >I'm setting up this huge RAID 6 box. I've always thought of hot spares, >but I'm reading things that are comparing RAID 5 with a hot spare to RAID >6, implying that the latter doesn't need one. I *certainly* have enough >drives to spare in this RAID box: 42 of 'em, so two questions: should I >assign one or more hot spares, and, if so, how many?A RAID5 with a hot spare isn't really the same as a RAID6.? For those not familiar with this, a RAID5 in degraded mode (after it lost a disk) will suffer a performance hit, as well as while it rebuilds from a hot spare.? A RAID6 after losing a disk will not suffer.? So, depending on your need for performance, you'll need to decide. As far as having a spare disk on a RAID6, I'd say it's not necessary.? As long as you have some mechanism in place to inform you if/when a disk fails, you'll not suffer any performance hit.
On 04/11/2013 11:36 AM, m.roth at 5-cent.us wrote:> I'm setting up this huge RAID 6 box. I've always thought of hot spares, > but I'm reading things that are comparing RAID 5 with a hot spare to RAID > 6, implying that the latter doesn't need one. I *certainly* have enough > drives to spare in this RAID box: 42 of 'em, so two questions: should I > assign one or more hot spares, and, if so, how many? > > markI was building a home NAS over the holidays and had the same question (well, not hot spare, but 5 vs. 6). A good friend on mine pointed me to the following article; http://www.zdnet.com/blog/storage/why-raid-5-stops-working-in-2009/162 I was using 6x 3 TB drives, so I decided to opt for RAID 6. About a month ago, a drive cacked out and I was *very* relieved to know that I was covered until I replaced the disk and it finished rebuilding. If you have 42 disks, I'd not even think twice and I would use RAID level 6. If fact, with such a large number, I'd almost be tempted to break it into two separate RAID level 6 arrays and use something like LVM to pool their space, just to hedge my bets. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education?
On 4/11/2013 8:36 AM, m.roth at 5-cent.us wrote:> I'm setting up this huge RAID 6 box. I've always thought of hot spares, > but I'm reading things that are comparing RAID 5 with a hot spare to RAID > 6, implying that the latter doesn't need one. I*certainly* have enough > drives to spare in this RAID box: 42 of 'em, so two questions: should I > assign one or more hot spares, and, if so, how many?John's First Rule of Raid. when a drive fails 2-3 years downstream, replacements will be unavailable. If you had bought cold spares and stored them, odds are too high they will be lost when you need them. John's Second Rule of Raid. No single raid should be much over 10-12 disks, or the rebuild times become truly hellacious. John's Third Rule of Raid. allow 5-10% hot spares. so, with 42 disks, 10% would be ~4 spares, which leaves 38. 5% would be 2 spares, allowing 40 disks. 40 divided by 4 == 10. You could format that as 10 raid6's, and stripe those (aka raid6+0 or raid60), and use 2 hot spares. Alternately, 3*13 == 39, leaving three hotspares, so 3 stripes of 13 disks with 3 hot spares is an alternative. I did some testing of very large raids using LSI Logic 9261-8i MegaRAID SAS2 cards driving 36 3TB SATA3 disks. With 3 x 11 disk RAID6 (and 3 hot spares), a failed disk took about 12 hours to restripe with the rebuilding set to medium priority, and the raid essentially idle. if you're using XFS on this very large file system (which I *would* recommend), do be sure to use a LOT of ram, like 48GB... while regular operations might not need it, XFS's fsck process is fairly memory intensive on a very large volume with millions of files. -- john r pierce 37N 122W somewhere on the middle of the left coast
----- Original Message -----> From: "Reindl Harald" <h.reindl at thelounge.net> > To: "CentOS mailing list" <centos at centos.org> > Cc: "David C. Miller" <millerdc at fusion.gat.com> > Sent: Thursday, April 11, 2013 4:17:18 PM > Subject: Re: [CentOS] RAID 6 - opinions > > > > Am 12.04.2013 01:13, schrieb David C. Miller: > > dd if=/dev/zero of=/backup/5GB.img count=5000 bs=1M > > 5000+0 records in > > 5000+0 records out > > 5242880000 bytes (5.2 GB) copied, 10.8293 s, 484 MB/s > > aha, you have 16 GB RAM, write 5 GB to the disk which > is easily buffered into RAM and think this measures > anything? > > do the same with 32 GB instead 5 GB > >Good call, I did not even think about that. Here is a 31GB file write. dd if=/dev/zero of=/backup/30GB.img count=30000 bs=1M 30000+0 records in 30000+0 records out 31457280000 bytes (31 GB) copied, 78.041 s, 403 MB/s David.
On 04/11/2013 06:36 PM, m.roth at 5-cent.us wrote:> I'm setting up this huge RAID 6 box. I've always thought of hot spares, > but I'm reading things that are comparing RAID 5 with a hot spare to RAID > 6, implying that the latter doesn't need one. I *certainly* have enough > drives to spare in this RAID box: 42 of 'em, so two questions: should Iwe use several of this kind of boxes (but with 45 trays) and our experience was that the optimum volume size was 12 hdds (3 X 12 + 9) which will reduce the 45 disks to a actual size of 37 disks (a 12 disk volume is 40 TB size ... in event of a broken hdd it takes 1 day to recover.. more than 12 disks and i dont (want to) know how long it would take) and we don't use hot spares. HTH, Adrian
We build a storage unit that anyone using Centos can build. It is based on the 3ware 9750-16 controller. It has 16 x 2 TB Sata 6 gb/s disks. We always set it up as a 15 disk RAID 6 array and a hot spare. We have seen multiple instances were the A/C has gone off but the customer's UPS kept the systems running for an hour or two with no cooling. Once the ambient temperature goes above 40C you are stressing all the disks in the array. I ramble about this BECAUSE we have seen many RAID 5 arrays fail while rebuilding with a hot spare and then loose all the data. The chances of this happening on a R6 array are much much lower. Now for performance numbers: 550 MB/SEC on writes and 1150 MB/SEC on reads. A unit like this can support any type of I/O you want to put into it as Centos will usually either have thedrivers built in or you can add then easily. Also the rebuild time for a 50 TB array is about 12 hours while the array is in use and online. But during this time there is little or no degradation in performance. The down side is the cost of the controller. If anyone has questions about this I would be glad to answer them off line : at seth at integratedsolutions dot org. Adrian Sevcenco <Adrian.Sevcenco at cern.ch> wrote:>On 04/11/2013 06:36 PM, m.roth at 5-cent.us wrote: >> I'm setting up this huge RAID 6 box. I've always thought of hot spares, >> but I'm reading things that are comparing RAID 5 with a hot spare to RAID >> 6, implying that the latter doesn't need one. I *certainly* have enough >> drives to spare in this RAID box: 42 of 'em, so two questions: should I >we use several of this kind of boxes (but with 45 trays) and our >experience was that the optimum volume size was 12 hdds (3 X 12 + 9) >which will reduce the 45 disks to a actual size of 37 disks (a 12 disk >volume is 40 TB size ... in event of a broken hdd it takes 1 day to >recover.. more than 12 disks and i dont (want to) know how long it would >take) and we don't use hot spares. > >HTH, >Adrian > > >_______________________________________________ >CentOS mailing list >CentOS at centos.org >http://lists.centos.org/mailman/listinfo/centos