Cooper Hubbell
2012-Mar-12 14:45 UTC
[zfs-discuss] Any recommendations on Perc H700 controller
Regarding the writeback cache on disks, what is the recommended method to disable the cache? Through HBA firmware, or via Solaris? On Sun, Mar 11, 2012 at 7:00 AM, <zfs-discuss-request at opensolaris.org>wrote:> > Message: 1 > Date: Sat, 10 Mar 2012 09:09:52 -0500 > From: John D Groenveld <jdg117 at elvis.arl.psu.edu> > To: zfs-discuss at opensolaris.org > Subject: Re: [zfs-discuss] Any recommendations on Perc H700 controller > on Dell Rx10 ? > Message-ID: <201203101409.q2AE9qW0021089 at elvis.arl.psu.edu> > Content-Type: text/plain; charset=us-ascii > > In message < > CANiY96a2t1n2m0KPMdx2RBODogjOFetOTH2qe4roT6x-Xn8e3A at mail.gmail.com> > , Sriram Narayanan writes: > >At work, I have an R510, and R610 and an R710 - all with the H700 PERC > >controller. > > BTW with some effort, Dell''s sales critter will sell you the > H200 LSI SAS HBA as a replacement for the H700 LSI MegaRAID > controller for those boxes. > > >Based on experiments, it seems like there is no way to bypass the PERC > >controller - it seems like one can only access the individual disks if > >they are set up in RAID0 each. > > Did you try deleting all of the RAID0 Virtual Disks and > then enabling JBOD? > # MegaCli -AdpSetProp -EnableJBOD -1 -aALL > # MegaCli -PDMakeJBOD -PhysDrv[E0:S0,E1:S1,...] -aALL > > John > groenveld at acm.org > > > > ------------------------------ > > Message: 2 > Date: Sat, 10 Mar 2012 09:10:06 -0500 > From: Edward Ned Harvey > <opensolarisisdeadlongliveopensolaris at nedharvey.com> > To: "''Sriram Narayanan''" <sriram at belenix.org>, > <zfs-discuss at opensolaris.org> > Subject: Re: [zfs-discuss] Any recommendations on Perc H700 controller > on Dell Rx10 ? > Message-ID: <000001ccfec7$8250c460$86f24d20$@nedharvey.com> > Content-Type: text/plain; charset="us-ascii" > > > From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- > > bounces at opensolaris.org] On Behalf Of Sriram Narayanan > > > > Based on experiments, it seems like there is no way to bypass the PERC > > controller - it seems like one can only access the individual disks if > > they are set up in RAID0 each. > > The first consideration to remember is: When your disks are set up by one > of these controllers, you won''t be able to move these disks to a generic > controller-less system and expect them to work. You are dependent on the > raid hardware (or compatible competitors) to make the disks accessible. > Just keep this in mind whenever your system warranty renewal period is > approaching. Also, if you need to move these disks to a new system that > has > a compatible controller, you must configure them exactly the same as they > are on this system - but don''t initialize. > > The second consideration you should think about is the writeback cache. If > you have any SSD dedicated log device, you actually get the best > performance > by DISABLING the writeback on all disks. The second best configuration is > without SSD, you enable writeback on all disks. And obviously the worst > configuration, you have no SSD and no writeback. > > > > ------------------------------ > > _______________________________________________ > zfs-discuss mailing list > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20120312/369199ca/attachment.html>
Edward Ned Harvey
2012-Mar-13 12:58 UTC
[zfs-discuss] Any recommendations on Perc H700 controller
> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- > bounces at opensolaris.org] On Behalf Of Cooper Hubbell > > Regarding the writeback cache on disks, what is the recommended method > to disable the cache? Through HBA firmware, or via Solaris?It''s the same either way - You press Ctrl-R (or whatever) during bootup and disable the writeback in the perc BIOS... Or you use MegaCLI, which does the same thing from within the OS. But again, I would only recommend disabling the writeback if you have either a dedicated SSD log device (or equivalent or faster), ... Or if you disable the ZIL completely (sync=disabled property). Because if you actually have ZIL and use spindle disks for ZIL, then the writeback cache is a big improvement over nothing.
Cooper Hubbell
2012-Mar-13 16:15 UTC
[zfs-discuss] Any recommendations on Perc H700 controller
My system is currently using LSI 9211 HBAs with Crucial M4 SSDs for ZIL/L2ARC. I have used LSIUTIL v1.63 to disable the write cache on the two controllers with only SATA HDDs, though my third controller has a combination of HDD and SSD as shown: SAS2008''s links are down, down, 6.0 G, 6.0 G, 3.0 G, down, 3.0 G, 3.0 G> > B___T___L Type Vendor Product Rev PhyNum > 0 9 0 Disk ATA M4-CT064M4SSD2 0309 2 > 0 10 0 Disk ATA M4-CT064M4SSD2 0309 3 > 0 11 0 Disk ATA ST3500320NS SN06 4 > 0 12 0 Disk ATA ST3500320NS SN06 6 > 0 13 0 Disk ATA ST3500320NS SN06 7 >Unfortunately, the SATA write cache can only be turned on or off at the controller level with LSIUTIL. In a case where the SSD ZIL and L2ARC are on the same controller as pool HDDs, is it still recommended that the cache be disabled? My other thought is that the cache to which LSIUTIL is referring is located on the controller and that the write cache on the individual disks is still active. This excerpt shows the interactive prompts of LSIUTIL during and after disabling the cache: Main menu, select an option: [1-99 or e/p/w or 0 to quit] 14> > Multi-pathing: [0=Disabled, 1=Enabled, default is 0] > SATA Native Command Queuing: [0=Disabled, 1=Enabled, default is 1] > SATA Write Caching: [0=Disabled, 1=Enabled, default is 1] 0 > > Main menu, select an option: [1-99 or e/p/w or 0 to quit] 68 > > Current Port State > ------------------ > SAS2008''s links are down, down, 6.0 G, 6.0 G, 3.0 G, down, 3.0 G, 3.0 G > > Software Version Information > ---------------------------- > Current active firmware version is 0b000000 (11.00.00) > Firmware image''s version is MPTFW-11.00.00.00-IT > LSI Logic > Not Packaged Yet > x86 BIOS image''s version is MPT2BIOS-7.21.00.00 (2011.08.11) > > Firmware Settings > ----------------- > SAS WWID: <stripped> > Multi-pathing: Disabled > SATA Native Command Queuing: Enabled > SATA Write Caching: Disabled > SATA Maximum Queue Depth: 32 > SAS Max Queue Depth, Narrow: 0 > SAS Max Queue Depth, Wide: 0 > Device Missing Report Delay: 0 seconds > Device Missing I/O Delay: 0 seconds > Phy Parameters for Phynum: 0 1 2 3 4 5 6 7 > Link Enabled: Yes Yes Yes Yes Yes Yes Yes Yes > Link Min Rate: 1.5 1.5 1.5 1.5 1.5 1.5 1.5 1.5 > Link Max Rate: 6.0 6.0 6.0 6.0 6.0 6.0 6.0 6.0 > SSP Initiator Enabled: Yes Yes Yes Yes Yes Yes Yes Yes > SSP Target Enabled: No No No No No No No No > Port Configuration: Auto Auto Auto Auto Auto Auto Auto Auto > Interrupt Coalescing: Enabled, timeout is 10 us, depth is 4 > >Thank you! On Tue, Mar 13, 2012 at 7:58 AM, Edward Ned Harvey < opensolarisisdeadlongliveopensolaris at nedharvey.com> wrote:> > From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- > > bounces at opensolaris.org] On Behalf Of Cooper Hubbell > > > > Regarding the writeback cache on disks, what is the recommended method > > to disable the cache? Through HBA firmware, or via Solaris? > > It''s the same either way - You press Ctrl-R (or whatever) during bootup > and disable the writeback in the perc BIOS... Or you use MegaCLI, which > does the same thing from within the OS. > > But again, I would only recommend disabling the writeback if you have > either a dedicated SSD log device (or equivalent or faster), ... Or if you > disable the ZIL completely (sync=disabled property). Because if you > actually have ZIL and use spindle disks for ZIL, then the writeback cache > is a big improvement over nothing. > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20120313/b1b42936/attachment.html>
Edward Ned Harvey
2012-Mar-16 11:31 UTC
[zfs-discuss] Any recommendations on Perc H700 controller
> From: Cooper Hubbell [mailto:cooper814 at gmail.com] > > My system is currently using LSI 9211 HBAs with Crucial M4 SSDs for > ZIL/L2ARC. I have used LSIUTIL v1.63 to disable the write cache on the two > > Unfortunately, the SATA write cache can only be turned on or off at the > controller level with LSIUTIL.You don''t want to disable the on-disk cache*. You only want to disable the on-controller writeback. Typically the on-disk is something like 16M, 32M or so. The on-controller is typically 256M, 512M, or greater. That might help you identify which cache you''re looking at, in any given situation. * Most disks are happy to buffer writes, and honor the OS request to flush. Assuming this is the case, it''s definitely what you want, as it significantly boosts the performance of any given disk - it''s able to internally optimize it''s head movements and so forth, it''s got the right data in the right buffer at the right time, so it doesn''t need to waste time on another rotation waiting for some buffer to be ready, etc. Unfortunately, not all disks are well behaved. I don''t know how to verify for certain that your disks are well behaved, but some drives acknowledge the flush request, and lie about having completed the request, and then your data is at risk. The easiest way to avoid that problem is to only buy sun(oracle) hardware that''s sold with solaris, because you know for sure they tested it. Anyone else, maybe. I dare say, probably you''re fine, but only "probably."> In a case where the SSD ZIL and L2ARC are on > the same controller as pool HDDs, is it still recommended that the cache be > disabled?Controller writeback. As long as you have SSD ZIL, then yes, disable controller writeback.> My other thought is that the cache to which LSIUTIL is referring is > located on the controller and that the write cache on the individual disks is > still active. This excerpt shows the interactive prompts of LSIUTIL during and > after disabling the cache:The controller certainly is smart enough to enable/disable on-controller writeback on a per-volume basis (and you''ve got one volume per disk). So you''re right, this certainly makes it confusing to know which cache you''re talking about in any given context. I suspect your controller is probably only letting you configure the controller writeback via LSIUTIL. The ironic thing is - With this controller in between your CPU and your storage, it''s only introducing additional buffers and chips that you''re going out of your way to disable. And it''s adding a layer of complexity, formatting the drive with proprietary headers, and reducing your access to the individual actual drives. The ironic thing is that you''re better off to have no controller, and instead use a simple, dumb, standard SATA/SAS bus. It''s both faster, more reliable, and more compatible. But maybe you don''t have that option.> SATA Write Caching: [0=Disabled, 1=Enabled, default is 1] 0Tough one. I don''t know. Generally I would say controllers only give you control over the controller writeback, but the way that''s written, it certainly sounds like it''s talking about the disk.