If you don't trust the hardware raid, then steer clear of raid-6 as mdadm raid 6 is stupidly slow. I don't completely trust hardware raid either, but rebuild times should be under a day and in order to lose a raid-6 array you have to lose 3 disks. My own systems are hardware raid-6. If you're not terribly worried about maximising usable storage, then mdadm raid-10 is your friend.> On 4 Jul 2016, at 18:15:26, Gandalf Corvotempesta <gandalf.corvotempesta at gmail.com> wrote: > > 2016-07-04 17:01 GMT+02:00 Matt Robinson <m.robinson at sheffield.ac.uk>: >> Hi Gandalf, >> >> Are you using hardware raid or mdadm? >> On high quality hardware raid, a 12 disk raid-6 is pretty solid. With mdadm any raid6 (especially with 12 disks) will be rubbish. > > I can use both. > I don't like very much hardware raid, even high quality. Recently i'm > having too many issue with hardware raid (like multiple disks kicked > out with no apparent reasons and virtual-disk failed with data loss) > > A RAID-6 with 12x2TB SATA disks would take days to rebuild, in the > meanwhile, multiple disks could fail resulting in data loss. > Yes, gluster is able to recover from this, but I prefere to avoid have > to resync 24TB of data via networks. > > What about a software RAID-1 ? 6 raid for each gluster nodes and 6 > disks wasted but SATA disks are cheaper.
2016-07-04 19:25 GMT+02:00 Matt Robinson <m.robinson at sheffield.ac.uk>:> If you don't trust the hardware raid, then steer clear of raid-6 as mdadm raid 6 is stupidly slow. > I don't completely trust hardware raid either, but rebuild times should be under a day and in order to lose a raid-6 array you have to lose 3 disks. > My own systems are hardware raid-6. > If you're not terribly worried about maximising usable storage, then mdadm raid-10 is your friend.All of my servers are hardware RAID-6 with 8x300GB SAS 15K (some servers with 600GB) A rebuild of a single disk in a 6x600GB SAS RAID-6 takes exactly 22 hours. This with 15K SAS disks. Now try with 2TB (more than twice the size) SATA 7200 (less than half speed)
For 3 servers with 12 disks each, I would do Hardware RAID0 (or madam if you don?t have a RAID card) of 3 disks. So four 3-disk RAID0?s per server. I would set them up as Replica 3 Arbiter 1 server1:/brickA server2:/brickC server3:/brickA server1:/brickB server2:/brickD server3:/brickB server2:/brickA server3:/brickC server1:/brickA server2:/brickB server3:/brickD server1:/brickB server3:/brickA server1:/brickC server2:/brickA server3:/brickB server1:/brickD server2:/brickB The benefit of this is that you can lose an entire server node (12 disks) and all of your data is still accessible. And you get the same space as if they were all in a RAID10. If you lose any disk, the entire 3 disk brick will need to be healed from the replica. I have 20GbE on each server so it doesn?t take long. It copied 20TB in about 18 hours once.