O. Hartmann
2016-Aug-01 06:45 UTC
mfi driver performance too bad on LSI MegaRAID SAS 9260-8i
On Wed, 22 Jun 2016 08:58:08 +0200 Borja Marcos <borjam at sarenet.es> wrote:> > On 22 Jun 2016, at 04:08, Jason Zhang <jasonzhang at cyphytech.com> wrote: > > > > Mark, > > > > Thanks > > > > We have same RAID setting both on FreeBSD and CentOS including cache > > setting. In FreeBSD, I enabled the write cache but the performance is the > > same. > > > > We don?t use ZFS or UFS, and test the performance on the RAW GEOM disk > > ?mfidx? exported by mfi driver. We observed the ?gstat? result and found > > that the write latency is too high. When we ?dd" the disk with 8k, it is > > lower than 1ms, but it is 6ms on 64kb write. It seems that each single > > write operation is very slow. But I don?t know whether it is a driver > > problem or not. > > There is an option you can use (I do it all the time!) to make the card > behave as a plain HBA so that the disks are handled by the ?da? driver. > > Add this to /boot/loader.conf > > hw.mfi.allow_cam_disk_passthrough=1 > mfip_load=?YES" > > And do the tests accessing the disks as ?da?. To avoid confusions, it?s > better to make sure the disks are not part of a ?jbod? or logical volume > configuration. > > > > > Borja.[...] How is this supposed to work when ALL disks (including boot device) are settled with the mfi (in our case, it is a Fujitsu CP400i, based upon LSI3008 and detected within FreeBSD 11-BETA and 12-CURRENT) controller itself? I did not find any solution to force the CP400i into a mode making itself acting as a HBA (we intend to use all drives with ZFS and let FreeBSD kernel/ZFS control everything). The boot device is a 256 GB Samsung SSD for enterprise use and putting the UEFI load onto a EFI partition from 11-CURRENT-ALPHA4 is worse: dd takes up to almost a minute to put the image onto the SSD. The SSD active LED is blinking alle the time indicating activity. Caches are off. I tried to enable the cache via the mfiutil command by 'mfiutil cache mfid0 enable', but it failed ... It failed also on all other attached drives. I didn't further go into more investigations right now, since the experience with the EFI boot loader makes me suspect bad performance and that is harsh so to speak. Glad to have found this thread anyway. I cross post this also to CURRENT as it might be an issue with CURRENT ... Kind regards, Oliver Hartmann
Borja Marcos
2016-Aug-01 09:48 UTC
mfi driver performance too bad on LSI MegaRAID SAS 9260-8i
> On 01 Aug 2016, at 08:45, O. Hartmann <ohartman at zedat.fu-berlin.de> wrote: > > On Wed, 22 Jun 2016 08:58:08 +0200 > Borja Marcos <borjam at sarenet.es> wrote: > >> There is an option you can use (I do it all the time!) to make the card >> behave as a plain HBA so that the disks are handled by the ?da? driver. >> >> Add this to /boot/loader.conf >> >> hw.mfi.allow_cam_disk_passthrough=1 >> mfip_load=?YES" >> >> And do the tests accessing the disks as ?da?. To avoid confusions, it?s >> better to make sure the disks are not part of a ?jbod? or logical volume >> configuration. >> >> >> >> >> Borja. > [...] > > How is this supposed to work when ALL disks (including boot device) are settled > with the mfi (in our case, it is a Fujitsu CP400i, based upon LSI3008 and > detected within FreeBSD 11-BETA and 12-CURRENT) controller itself? > > I did not find any solution to force the CP400i into a mode making itself > acting as a HBA (we intend to use all drives with ZFS and let FreeBSD > kernel/ZFS control everything).Have you tried that particular option? With kinda recent LSI based cards you have three options: - The most usual and definitely NOT RECOMMENDED option is to define a logical volume per disk which actually LSI Logic called before JBOD mode. It?s not recommended at all if you want to run ZFS. - Recent cards, I think I saw this first on the LSI3008, have a JBOD mode that exposes the drives as ?mfisyspd? devices. I don?t recommend it either, because the syspd drives are a sort of limited version of a disk device. With SSDs, especially, you don?t have access to the TRIM command. - The third option is to make the driver expose the SAS devices like a HBA would do, so that they are visible to the CAM layer, and disks are handled by the stock ?da? driver, which is the ideal solution. However, this third option might not be available in some custom firmware versions for certain manufacturers? I don?t know. And I would hesitate to make the conversion on a production machine unless you have a complete and reliable full backup of all the data in case you need to rebuild it. In order to do it you need a couple of things. You need to set the variable hw.mfi.allow_cam_disk_passthrough=1 and to load the mfip.ko module. When booting installation media, enter command mode and use these commands: ----- set hw.mfi.allow_cam_disk_passthrough=1 load mfip boot ??? Remember that after installation you need to update /boot/loader.conf in the system you just installed with the following contents: hw.mfi.allow_cam_disk_passthrough=1 mfip_load=?YES? A note regarding CAM and MFI visibility: On some old firmware versions for the LSI2008 I?ve even seen the disks available both as ?mfi? and ?da? drivers. If possible, you should try to set them up as ?unconfigured good? on the RAID firmware. Use the RAID firmware set up or maybe mfiutil(8) Also, make sure you don?t create any logical volumes on the disks you want exposed to CAM. You should delete the logical volumes so that the MFI firmware doesn?t do anything with them. AND BEWARE: Doing these changes to a system in production with valuable data is dangerous. Make sure you have a full and sound backup before making these changes. As a worst case, the card could expose the devices both as ?syspd? and CAM (i.e., ?da? drives) but as long as you don?t touch the syspd devices the card won?t do anything to them as far as I know. It could be a serious problem, however, if you access a drive part of a logical volume through CAM, as RAID cards tend do to ?patrol reads? and other stuff on them. Provided it?s safe to do what I recommended, try it and follow up by email. Borja.