Hello! Here is the newest story of mine about how one should never use raid5. Controller is 8xxx-4LP. I have a simple 360GB raid5 with 4 drives since 2004. Only about a year ago i realized how much speed i have wasted be saving lousy 120GB. I should have choosen bigger driver and setup two mirrors instead. But that's no the point. A week ago one driver just totally failed. It fell out of the unit and when i tried to rebuild the unit it failed. It seemed like the driver electronis failed. ANyhow, i have found newest 160gb seagate driver for replacement (twice as thin, very nicely done electornics on it). A day ago at 11 am i have turn off the server, pull out the old driver, installed a new one, turned of the server and started rebuild in an hour from remote location via web interface. After about 5 minuted the machine became unresponsive. Tried rebooting - nothing. I went to the machine and fingure out, that rebuild failed (0%) and some data cannot be read because of bad sectors. Well, hell, i thoght. Maybe i could tell teh controller to ignore all the errors and just some rebuilding and the figure out which driver failed, replace it, rebuild again and restore corrupted data from backup. Noway, controller said. - i cannot make it ignore read errors - i cannot figure out which driver has bad sectors (maybe someone know it?) But i don't understand how and why it happened. ONly 6 hours ago (a night before) all those files were backed up fine w/o any read error. And now, right after replacing the driver and starting rebuild it said that there are bad sectors all over those file. How come? Well. Since we have a buch of full and inceremnetal paraoid backups no data was lost and we are in process of recovering. However, i simply imaged what would happed if one more driver completelly failed. That would mean that we have lost all data, since any of the disk which left do not contain any readable copy of one data (unlink mirror, for example). So, we are migrating to mirror config with huge disks. I am thinking about raid10 for more perfomance. It seems a lot more safe, since if any pair of disks failed the data is still readable and even if all disks have bad block the data can be easily recovered by fairly simply script from the couterpart. But the problem, however, So, no raid5 or even raid 6 for me any more. Never! -- Regards, Artem
Artem Kuchin wrote:> > So, no raid5 or even raid 6 for me any more. Never! > >A better policy is to invest in a higher quality RAID controller. Also, always use a battery backup on the controller, and always have an extra disk configured as a hot spare. Data integrity is expensive, unfortunately. Scott
Artem Kuchin wrote:> A day ago at 11 am i have turn off the server, > pull out the old driver, installed a new one, turned of the server > and started rebuild in an hour from remote location via web interface. > After about 5 minuted the machine became unresponsive. Tried rebooting > - nothing. I went to the machine and fingure out, that rebuild failed (0%) > and some data cannot be read because of bad sectors.We had a similar failure a few years ago, in fact every drive in the array had bad sectors, but one failed completely. Rebuild would not work no matter what we did. Yet another reason to be sure your RAID disks come from different manufacturing batches!> - i cannot make it ignore read errors > - i cannot figure out which driver has bad sectors > (maybe someone know it?)We recovered from that situation like so: 1. Power off the server 2. Pull each drive, attach to another box w/o RAID adapter and use diagnostic tools to run a surface scan/remap on each disk. (These were SCSI disks, not sure if the same applies to PATA/SATA) 3. Put all drives back, including a replacement for the truly failed drive 4. Let the array rebuild Many crossed fingers/toes/eyes later, it came back to life. We replaced the whole box shortly thereafter. The downside was the entire server was offline for the duration of the process, instead of being online during a normal rebuild.> So, no raid5 or even raid 6 for me any more. Never!If it's done properly, with hot spares and other failsafe measures, it isn't too bad. Sometimes it's the best available option due to budget/hardware/etc constraints, especially on older systems. RAID can be a tough beast, though. We had one server that ran fine for nearly 5 years on a single PATA disk. Two months after I rebuild it with a proper SCSI RAID setup, it has a multi-drive failure and bombs. Sometimes all the safety measures in the world can't make up for what passes for hardware quality these days... Jim
> A day ago at 11 am i have turn off the server, > pull out the old driver, installed a new one, turned of the server > and started rebuild in an hour from remote location via web interface. > After about 5 minuted the machine became unresponsive. Tried rebooting > - nothing. I went to the machine and fingure out, that rebuild failed (0%) > and some data cannot be read because of bad sectors.Why would you power cycle a RAID 5 array with a failed drive? That's like the biggest no-no that there is. When you lose a drive on a RAID 5 array, you are vulnerable until a replacement drive is configured and the array is rebuilt. Any high risk operations during that time would be foolhardy.> So, no raid5 or even raid 6 for me any more. Never!Since RAID6 would have saved you from what presumably was a drive failure before a rebuild could be done, it's hard to understand why you would say this is a reason to avoid RAID 6. Perhaps you would do better to understand your failure and avoid the causes of the failure rather than avoiding the things you happened to be using at the time of the failure. If you get food poisoning while wearing a blue shirt, the solution is not to avoid blue shirts in the future. DS
Artem Kuchin wrote:> But i don't understand how and why it happened. ONly 6 hours ago (a > night before) > all those files were backed up fine w/o any read error. And now, right > after replacing > the driver and starting rebuild it said that there are bad sectors all > over those file. > How come?That is what patrol read is intended to detect before it is a problem. In a RAID5 array the checksums are only used when reconstructing data, if you have a bad block in a checksum sector it will not be detected until a drive have failed and you try to rebuild the array, unfortunately at that time it is too late... Beware that OS software solutions like diskcheckd will not find this as it only reads the data, not the checksums, it must be done on the controller. Regards, Martin
Artem Kuchin unleashed the infinite monkeys on 20/08/2007 23:38 producing: <---SNIP--->> But i don't understand how and why it happened. ONly 6 hours ago (a > night before) > all those files were backed up fine w/o any read error. And now, right > after replacing > the driver and starting rebuild it said that there are bad sectors all > over those file.I've had that happen on a RAID1 setup before. Because all the bad sectors were in different positions the array worked fine until one of the disks failed. As I'm fond of saying at work, RAID is no substitute for regular, tested, backups. Regarding your problem - you can download the (very detailed) manuals for the 3Ware cards and their software from the 3Ware website. A quick scan suggests you need the "mediascan" command (P48). Newer controllers (9000 series) support scheduling of their equivalent. -- Rob | Oh my God! They killed init! You bastards!
> While we are on the subject: > > What is the practical difference between VERIFY and REBUILD with regards > to a RAID-5 array?Verify should at a minimum read all the data. Ideally, it would read the checksum blocks too to make sure they are still valid, but it might not. Rebuild should read all the data and write out new checksum blocks. It might also validate that it can read back the checksum.> My Highpoint RocketRAID 2320 and 2340 cards can be scheduled to perform > either verify or rebuild. I currently have them set to verify the arrays > weekly. Is that reasonably often? Do I want to rebuild regularly also > (or instead of verify)?It really depends what these functions actually do. Ideally, verify would read the data and the checksum and so would ensure that all data can be read, even if one drive fails. If so, that would really be all you need to do periodically. DS
> On Tue, 21 Aug 2007 08:57:22 +0400 > "Artem Kuchin" <matrix@itlegion.ru> wrote: > >> Um.. it is because i did not have a map of hot swap baskets to >> conroller ports and i needed to check every driver basket to >> understand which port it sits on. I have no choise, i think. >> > > I'm just going to highlight the importance of knowing which > physical disk is which on your system. > > About a year ago I had to replace a hot-swappable disk from an > array, but then realised I had no idea which physical disk it > was as the map of the disks was rather helpfully *inside* the > case. Due to the physical setup, I had no way or removing the > cover without first powering down the server - which defeated > the whole point of paying extra for hot-swap disks. > > So yeah, be sure to label your disk bays, but be sure to put > those labels somewhere *useful*. > > -fr. >Useful like on the front of the drive bays ;-) The Areca cards have a nice function called drive identify that lights up the selected drive's LED. I think the 3ware cards have it too. -Clay
----- "Artem Kuchin" <matrix@itlegion.ru> wrote: ...> But i don't understand how and why it happened. ONly 6 hours ago (a > night before) > all those files were backed up fine w/o any read error. And now, right > after replacing > the driver and starting rebuild it said that there are bad sectors all > over those file. > How come?What happened to you was an extremely common occurrence. You had a disk develop a media failure sometime ago, but the controller never detected it, because that particular bad area was not read. Your backups worked because they never touched this portion of the disk (ex. empty space, meta data, etc). And then another drive developed a electronics failure, which is instantly detected, putting the array into a degraded mode. When you did a rebuild onto a replace drive, the controller discovered that there was a second failed disk, and this is unrecoverable. RAID, of any level, isn't magic. It is important to understand how it works, an realize that drives can passive fail. BTW, if you were using RAID1 or RAID10, you would likely have had the same problem (well, RAID10 can survive _some_ double-disk failures). RAID6 is the only RAID level that can survive failure of any two disks. The real solution is RAID scrubbing: a low level background process that reads every sector of every disk. All of the real RAID systems do this (usually scheduled weekly, or every other week). Most 3ware RAID card don't have this feature. So rather than not using RAID5 or RAID6 again, you should just not use 3ware anymore. Tom
Friday 24 August 2007 23:04:37 kirjutas Matthew Dillon:> A friend of mine once told me that the only worthwhile RAID systems are > the ones that email you a detailed message when something goes south. > > -Matt > _______________________________________________AFAIK all good raid hardware producers (3ware, LSI, Areca) got this functionality.