I am sure this is not the first discussion related to this... apologies for the duplication. What is the recommended way to make use of a Hardware RAID controller/HBA along with ZFS? Does it make sense to do RAID5 on the HW and then RAIDZ on the software? OR just stick to ZFS RAIDZ and connect the drives to the controller, w/o any HW RAID (to benefit from the batteries). What will give the most performance benefit? If there isn''t much difference, I just feel like I am spending too much money on a RAID controller but not making 100% use of it. :) .... perhaps, maybe the route I should take is to look for drives (SAS or SSD) with built-in batteries/capacitors - but perhaps then it becomes much more expensive then the HW RAID controller? Thanks -- This message posted from opensolaris.org
On Mon, 11 Jan 2010, Anil wrote:> > What is the recommended way to make use of a Hardware RAID > controller/HBA along with ZFS? > > Does it make sense to do RAID5 on the HW and then RAIDZ on the > software? OR just stick to ZFS RAIDZ and connect the drives to the > controller, w/o any HW RAID (to benefit from the batteries). What > will give the most performance benefit?ZFS will definitely benefit from battery backed RAM on the controller as long as the controller immediately acknowledges cache flushes (rather than waiting for battery-protected data to flush to the disks). There will be benefit as long as the size of the write data backlog does not exceed controller RAM size. The notion of "performance" is highly arbitrary since there are different types of performance. What sort of performance do you need to optimize for? I think that the best performance for most cases is to use mirroring. Use two controllers with battery-backed RAM and split the mirrors across the controllers so that a write to a mirror pair results in a write to each controller. Unfortunately, this is not nearly as space efficient as RAID5 or raidz. Many people will recommend against using RAID5 in "hardware" since then zfs is not as capable of repairing errors, and because most RAID5 controller cards use a particular format on the drives so that the drives become tied to the controller brand/model and it is not possible to move the pool to a different system without using an identical controller. If the controller fails and is no longer available for purchase, or the controller is found to have a design defect, then the pool may be toast. Bob -- Bob Friesenhahn bfriesen at simple.dallas.tx.us, simplesystems.org/users/bfriesen GraphicsMagick Maintainer, GraphicsMagick.org
> ZFS will definitely benefit from battery backed RAM > on the controller > as long as the controller immediately acknowledges > cache flushes > (rather than waiting for battery-protected data to > flush to theI am little confused with this. Do we not want the controller to ignore these cache flushes (since the cache is battery protected)? What did you mean by acknowledge? If it acknowledges and flushes the cache, then what is the benefit of the cache at all (if ZFS keeps telling to flush ever few seconds)?> The notion of "performance" is highly arbitrary since > there are > different types of performance. What sort of > performance do you need > to optimize for?It won''t be a big deal. Just general web/small databases. Just wondering how much of magnitude in performance difference there is.> > Many people will recommend against using RAID5 in > "hardware" since > then zfs is not as capable of repairing errors, and > because most RAID5 > controller cards use a particular format on the > drives so that the > drives become tied to the controller brand/model and > it is not > possible to move the pool to a different system > without using an > identical controller. If the controller fails and is > no longer > available for purchase, or the controller is found to > have a design > defect, then the pool may be toast. >I use mostly DAS for my servers. This is a x4170 with 8 drive bays. So, what''s the final recommendation? Should I just RAID 1 on the hardware and put ZFS on top of it? Got three options: Get the Storagetek Internal HBA (with batteries) and...: 7 disks 600gb usable ======no hardware raid 2 rpool 3 raidz 1 hot spare 1 ssd 8 disks 600gb usable ======2 rpool 2 mirror + 2 mirror (the mirrors would be in hardware with zfs doing striping across the mirrors) 1 hot spare 1 ssd 8 disks 1200gb usable ======no hardware raid 2 rpool 3 raidz + 3 raidz The hot spare and the SSD are optional and I can add them at a later point. -- This message posted from opensolaris.org
On 11-Jan-10, at 1:12 PM, Bob Friesenhahn wrote:> On Mon, 11 Jan 2010, Anil wrote: >> >> What is the recommended way to make use of a Hardware RAID >> controller/HBA along with ZFS? >> ... > > Many people will recommend against using RAID5 in "hardware" since > then zfs is not as capable of repairing errors, and because most > RAID5 controller cards use a particular format on the drives so > that the drives become tied to the controller brand/model and it is > not possible to move the pool to a different system without using > an identical controller. If the controller fails and is no longer > available for purchase, or the controller is found to have a design > defect, then the pool may be toast.+1 These drawbacks of proprietary RAID are frequently overlooked. Marty Scholes had a neat summary in a posting here, 21 October 2009: mail-archive.com/zfs-discuss at opensolaris.org/msg30452.html> Back when I did storage admin for a smaller company where > availability was hyper-critical (but we couldn''t afford EMC/ > Veritas), we had a hardware RAID5 array. After a few years of > service, we ran into some problems: > * Need to restripe the array? Screwed. > * Need to replace the array because current one is EOL? Screwed. > * Array controller barfed for whatever reason? Screwed. > * Need to flash the controller with latest firmware? Screwed. > * Need to replace a component on the array, e.g. NIC, controller or > power supply? Screwed. > * Need to relocate the array? Screwed. > > If we could stomach downtime or short-lived storage solutions, none > of this would have mattered. >
On Mon, 11 Jan 2010, Anil wrote:>> ZFS will definitely benefit from battery backed RAM on the >> controller as long as the controller immediately acknowledges cache >> flushes (rather than waiting for battery-protected data to flush to >> the > > I am little confused with this. Do we not want the controller to > ignore these cache flushes (since the cache is battery protected)? > What did you mean by acknowledge? If it acknowledges and flushes > the cache, then what is the benefit of the cache at all (if ZFS > keeps telling to flush ever few seconds)?We want the controller to flush unwritten data to disk as quickly as it can regardless of whether it receives a cache flush request. If the data is "safely" stored in battery backed RAM, then we would like the controller to acknowledge the flush request immediately. The primary benefit of the battery-protected cache is to reduce latency for small writes.> I use mostly DAS for my servers. This is a x4170 with 8 drive bays. > So, what''s the final recommendation? Should I just RAID 1 on the > hardware and put ZFS on top of it?Unless you have a severe I/O bottleneck through your controller, you should do the mirroring in zfs rather than in the controller. The reason for this is that zfs mirrors are highly resilient, intelligent, and resilver time is reduced. Zfs will be able to detect and correct errors that the controller might not be aware of, or be unable to correct. Bob -- Bob Friesenhahn bfriesen at simple.dallas.tx.us, simplesystems.org/users/bfriesen GraphicsMagick Maintainer, GraphicsMagick.org