On 05/28/2015 11:13 AM, Valeri Galtsev wrote:> > So, I end up telling them: > before telling others that 3ware RAID cards are bad and let you down, > check that what you set up does not contain obvious blunders.OK. And I'll tell you that none of the failures that I've seen in the last 15 years were a result of user error or poor configuration.> 1. Bad choice of drives for RAID.None of our failures were due to drives.> 2. Bad configuration of RAID itself. You need to run "verification" of the > RAID every so often.The same is true of software RAID, and enabled by default on RHEL/CentOS.> 3. Smaller, yet still blunders: having card without memory battery backup > and running RAID with the cacheI don't know if that was ever allowed, but the last time I looked at a 3ware configuration, you cannot enable write caching without a BBU.> Of course, there are some restrictions, in particular, not always you can > attach drives to different card model and have RAID keep functioning.Yes, and that is one of the reasons I advocate software RAID. If I have a large data set on a 3ware controller, and that controller dies, then I can only move those disks to another system if I have a compatible controller. With software RAID, I don't have to hope the controller is still in production. I don't have to wait for delivery of a replacement. I don't have to keep a spare, expensive equipment on hand. I don't have to do a prolonged restore from backup. Any system running Linux will read the RAID set.> I do not want to start "software" vs "hardware" RAID wars hereI'm not saying that hardware RAID is bad, entirely. Software RAID does have some advantages over hardware RAID, but my point has been that 3ware cards, specifically, are less reliable than software RAID, in my experience.
On Thu, May 28, 2015 4:25 pm, Gordon Messmer wrote:> On 05/28/2015 11:13 AM, Valeri Galtsev wrote: >> >> I do not want to start "software" vs "hardware" RAID wars here > > I'm not saying that hardware RAID is bad, entirely. Software RAID does > have some advantages over hardware RAID, but my point has been that > 3ware cards, specifically, are less reliable than software RAID, in my > experience.If I get you correctly you are saying that 3ware RAID cards are prone to hardware failures - as opposed to software RAID which is not (as it does not include hardware, so never has a hardware failures), right? No this is a joke of course. But it's the one one asked for ;-) Now, seriously: of more than a couple of dozens of cards I used during last about 13 years not a single one died on me. Still I do not consider myself lucky. I also do not have specifically excellent conditions in a server room. Just ordinary ones. Sometimes during air conditioning maintenance temperature in the server room is higher than in our offices (once we has 96F for a couple of hours, of course this was a unique case. BTW during these 2 hours none of AMD based boxes got sick. But a few Intel ones did). So, I have no special conditions for our equipment. Boxes are behind APC UPSes, that's true, but that is sort of nothing special. I also probably should mention that all 3ware cards I used were bought new, never used by someone else. Couple of dozens is decently representative selection statistically, so I probably can't just claim myself lucky. I'm still convinced that 3ware RAID cards _ARE_ reliable. Again, I don't know your statistics: i.e. how many did you use, how many of them died (failed as hardware). How well surge-free the power is. How well the guys who installed your cards into your boxes followed static discharge precautions (yes, even though these cards are robust, one may fry them slightly by static discharge, and they may fail - as many fried ICs they not always fail right away, but some time later, say, in a year; still as a result of static discharge). To be fair, I'm often not wearing that anti-static bracelet, but I do touch metal tars of the box, anti-static bag the card ships in etc - which is sufficient IMHO. So, still not considering myself lucky, yet never having 3ware card die on me (out of over couple of dozens, during over 13 years, and most of the cards are in service for some 8-10 years since they originally were installed). In my book 3ware RAID cards are reliable hardware. Valeri ++++++++++++++++++++++++++++++++++++++++ Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247 ++++++++++++++++++++++++++++++++++++++++
On 05/28/2015 03:02 PM, Valeri Galtsev wrote:> > If I get you correctly you are saying that 3ware RAID cards are prone to > hardware failures - as opposed to software RAID which is not (as it does > not include hardware, so never has a hardware failures), right? No this is > a joke of course. But it's the one one asked for ;-)Software RAID uses whatever controller the disks are connected to. Often, that's the AHCI controller on the motherboard. And those controllers are almost always more reliable than 3ware. Seriously. No joke.> Again, I don't know your statistics: i.e. how many did you use,Hundreds of systems, both with 3ware and with software RAID.> how many > of them died (failed as hardware).Not many, but some. A handful of data corruption cases (maybe 5? I don't have logs). One BBU failure that resulted in a system that wouldn't boot until it was removed (and a couple of hours down time while 3ware techs worked that ticket). A couple of times when we really wanted to move an array of disks and couldn't. Vs zero reliability issues with software RAID.> How well surge-free the power is.We always used trusted UPS hardware.> How > well the guys who installed your cards into your boxes followed static > discharge precautionsOur systems were built by a professional VAR. Employees always wore ground straps. There were other anti-static measures in place as well. I had been to the facility. And I'll throw you a curveball: I've argued that software RAID is more reliable than 3ware cards, specifically. In the world of ZFS and btrfs, you absolutely should not use hardware RAID. As you mentioned earlier, hardware RAID volumes should scan disks regularly. However, scanning only tells you if disks have bad sectors. It can also detect data (parity) errors, but it can't repair them. Hardware RAID cards don't have information that can tell them which sectors are correct. ZFS and btrfs, on the other hand, checksum their data and metadata. They can tell which sectors are damaged in the case of corruption such as bit flips, and which sector should be used to repair the data. Now, btrfs and ZFS are completely different from software RAID. I'm not comparing the two. But hardware RAID simply has too many deficiencies to justify its continued use. It should go away.
On 2015-05-28, Valeri Galtsev <galtsev at kicp.uchicago.edu> wrote:> > Now, seriously: of more than a couple of dozens of cards I used during > last about 13 years not a single one died on me.I have had a couple dozen hardware RAID controllers over the years. I have not had the success you've had, but I've had very few hard failures. I have had one data loss event, where a bad BBU was causing problems with an old controller (of course I had backups, as should everyone). I've had two other controllers just die, but replacing it was easy, and the new controller recognized the arrays immediately. This includes moving two different disk arrays from two different 9650s to two different 9750s, so whoever wrote that arrays are not compatible across different models is at least partly incorrect. I do also have an LSI controller, which has been fine, but it's only one controller so it's not enough data points to draw any conclusions. I also have an md RAID array (on a very old 3ware controller which doesn't support RAID6), and it's also been fine. It hasn't suffered through any major catastrophes, though I do think it's had one or two fatal kernel panics, and once or twice had a hard reset done. It's still fine even with a small number of really crappy "green" drives still in the array (I learned that lesson the hard way--don't use green drives with a hardware RAID controller!). --keith -- kkeller at wombat.san-francisco.ca.us