O. Hartmann
2016-Aug-01 13:12 UTC
mfi driver performance too bad on LSI MegaRAID SAS 9260-8i
On Mon, 1 Aug 2016 11:48:30 +0200 Borja Marcos <borjam at sarenet.es> wrote: Hello. First, thanks for responding so quickly.> > On 01 Aug 2016, at 08:45, O. Hartmann <ohartman at zedat.fu-berlin.de> wrote: > > > > On Wed, 22 Jun 2016 08:58:08 +0200 > > Borja Marcos <borjam at sarenet.es> wrote: > > > >> There is an option you can use (I do it all the time!) to make the card > >> behave as a plain HBA so that the disks are handled by the ?da? driver. > >> > >> Add this to /boot/loader.conf > >> > >> hw.mfi.allow_cam_disk_passthrough=1 > >> mfip_load=?YES" > >> > >> And do the tests accessing the disks as ?da?. To avoid confusions, it?s > >> better to make sure the disks are not part of a ?jbod? or logical volume > >> configuration. > >> > >> > >> > >> > >> Borja. > > [...] > > > > How is this supposed to work when ALL disks (including boot device) are > > settled with the mfi (in our case, it is a Fujitsu CP400i, based upon > > LSI3008 and detected within FreeBSD 11-BETA and 12-CURRENT) controller > > itself? > > > > I did not find any solution to force the CP400i into a mode making itself > > acting as a HBA (we intend to use all drives with ZFS and let FreeBSD > > kernel/ZFS control everything). > > Have you tried that particular option?I have, indeed, used the "JBOD" function of the PRAID CP400i controller and the intention of my posting regards to the suspicion, that this is, as mentioned in many posts concerning RAID controllers and ZFS, the reason for the worse performance. And as I can see, it has been confirmed, sadly.> > With kinda recent LSI based cards you have three options: > > - The most usual and definitely NOT RECOMMENDED option is to define a logical > volume per disk which actually LSI Logic called before JBOD mode. It?s not > recommended at all if you want to run ZFS.This is the only way to expose each disk as it is to the OS with the PRAID CP400i built-in into our RX1330-M2 server (XEON Skylake based). I ordered that specific box with a HBA capable controller. Searching the net reveals that there is another one, called PSAS CP400i, which is also based on LSI/Avago SAS3008 and the possibility to expose drives as-is is explicitely mentioned. I do not know whether this is a software feature - as I suspect - or something which has been hardwired to the controller.> > - Recent cards, I think I saw this first on the LSI3008, have a JBOD mode > that exposes the drives as ?mfisyspd? devices. I don?t recommend it either, > because the syspd drives are a sort of limited version of a disk device. With > SSDs, especially, you don?t have access to the TRIM command.They expose the drives as "mfidX" if setup as JBOD.> > - The third option is to make the driver expose the SAS devices like a HBA > would do, so that they are visible to the CAM layer, and disks are handled by > the stock ?da? driver, which is the ideal solution.I didn't find any switch which offers me the opportunity to put the PRAID CP400i into a simple HBA mode.> > However, this third option might not be available in some custom firmware > versions for certain manufacturers? I don?t know. And I would hesitate to > make the conversion on a production machine unless you have a complete and > reliable full backup of all the data in case you need to rebuild it.The boxes are empty and ready-for-installation, so I do not worry. It is more worrying about this stupid software-based strangulations of options by Fujitsu - if any. i do not want to blame them before I haven't double-checked.> > In order to do it you need a couple of things. You need to set the variable > hw.mfi.allow_cam_disk_passthrough=1 and to load the mfip.ko module. > > When booting installation media, enter command mode and use these commands: > > ----- > set hw.mfi.allow_cam_disk_passthrough=1 > load mfip > boot > ???Well, I'm truly aware of this problemacy and solution (now), but I run into a henn-egg-problem, literally. As long as I can boot off of the installation medium, I have a kernel which deals with the setting. But the boot medium is supposed to be a SSD sitting with the PRAID CP400i controller itself! So, I never be able to boot off the system without crippling the ability to have a fullspeed ZFS configuration which I suppose to have with HBA mode, but not with any of the forced RAID modes offered by the controller. I will check with Fujitsu for a solution. Maybe the PRAID CP400i is capable somehow of being a PSAS CP400i also, even if not exposed by the recent/installed firmware. Kind regards, Oliver> > > Remember that after installation you need to update /boot/loader.conf in the > system you just installed with the following contents: > > hw.mfi.allow_cam_disk_passthrough=1 > mfip_load=?YES? > > > A note regarding CAM and MFI visibility: On some old firmware versions for > the LSI2008 I?ve even seen the disks available both as ?mfi? and ?da? > drivers. If possible, you should try to set them up as ?unconfigured good? on > the RAID firmware. Use the RAID firmware set up or maybe mfiutil(8) > > Also, make sure you don?t create any logical volumes on the disks you want > exposed to CAM. You should delete the logical volumes so that the MFI > firmware doesn?t do anything with them. > > AND BEWARE: Doing these changes to a system in production with valuable data > is dangerous. Make sure you have a full and sound backup before making these > changes. > > As a worst case, the card could expose the devices both as ?syspd? and CAM > (i.e., ?da? drives) but as long as you don?t touch the syspd devices the card > won?t do anything to them as far as I know. It could be a serious problem, > however, if you access a drive part of a logical volume through CAM, as RAID > cards tend do to ?patrol reads? and other stuff on them. > > Provided it?s safe to do what I recommended, try it and follow up by email. > > > > > > Borja. > > > > > _______________________________________________ > freebsd-performance at freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe at freebsd.org"
Borja Marcos
2016-Aug-01 13:29 UTC
mfi driver performance too bad on LSI MegaRAID SAS 9260-8i
> On 01 Aug 2016, at 15:12, O. Hartmann <ohartman at zedat.fu-berlin.de> wrote: > > First, thanks for responding so quickly. > >> - The third option is to make the driver expose the SAS devices like a HBA >> would do, so that they are visible to the CAM layer, and disks are handled by >> the stock ?da? driver, which is the ideal solution. > > I didn't find any switch which offers me the opportunity to put the PRAID > CP400i into a simple HBA mode.The switch is in the FreeBSD mfi driver, the loader tunable I mentioned, regardless of what the card firmware does or pretends to do. It?s not visible doing a "sysctl -a?, but it exists and it?s unique even. It?s defined here: https://svnweb.freebsd.org/base/stable/10/sys/dev/mfi/mfi_cam.c?revision=267084&view=markup (line 93)>> In order to do it you need a couple of things. You need to set the variable >> hw.mfi.allow_cam_disk_passthrough=1 and to load the mfip.ko module. >> >> When booting installation media, enter command mode and use these commands: >> >> ----- >> set hw.mfi.allow_cam_disk_passthrough=1 >> load mfip >> boot >> ??? > > Well, I'm truly aware of this problemacy and solution (now), but I run into a > henn-egg-problem, literally. As long as I can boot off of the installation > medium, I have a kernel which deals with the setting. But the boot medium is > supposed to be a SSD sitting with the PRAID CP400i controller itself! So, I > never be able to boot off the system without crippling the ability to have a > fullspeed ZFS configuration which I suppose to have with HBA mode, but not > with any of the forced RAID modes offered by the controller.Been there plenty of times, even argued quite strongly about the advantages of ZFS against hardware based RAID 5 cards. :) I remember when the Dell salesmen couldn?t possibly understand why I wanted a ?software based RAID rather than a robust, hardware based solution? :D At worst, you can set up a simple boot from a thumb drive or, even better, a SATADOM installed inside the server. I guess it will have SATA ports on the mainboard. That?s what I use to do. FreeNAS uses a similar approach as well. And some modern servers also can boot from a SD card which you can use just to load the kernel. Depending on the number of disks you have, you can also sacrifice two to set up a mirror with a ?nomal? boot system, and using the rest of the disks for ZFS. Actually I?ve got an old server I set up in 2012. It has 16 disks, and I created a logical volume (mirror) with 2 disks for boot, the other 14 disks for ZFS. If I installed this server now I would do it different, booting off a thumb drive. But I was younger and naiver :) Borja.