On 03/05/2019 08:09, Borja Marcos via freebsd-stable wrote:> The right way to use disks is to give ZFS access to the plain CAM devices, not thorugh some so-called JBOD on a RAID > controller which, at least for a long time, has been a *logical* ?RAID0? volume on a single disk. That additional layer can > completely break the semantics of transaction writes and cache flushes. > > With some older cards it can be tricky to achieve, from patching source drivers to enabling a sysctl tunable or even > flashing the card to turn it into a plain HBA with no RAID features (or minimal ones).Oddly enough I got bitten by something like this yesteray. I have a machine containing an HP P400 RAID controller, which is nice enough, but I run ZFS so I have made the drives all into RAID-0 as being as close as I can get to accessing the raw SAS drives. BSD seems them as da0, da1, da2, da3 - but the RAID controller oly presents one of them to the BIOS, so my booting has to be all from that drive. This has been da0 for as long as I can remember, but yesteday it decided to start using what BSD sees as da1. Of course this is very hard to recognise as da0 and da1 are pretty much mirrors of each other. Spent a long time trying to work out why the fixes I was applying to da0 were not being used at boot time. ( Having to use Windows XP to talk to the iLo due to browsers dropping support for the old ciphers doesnt help either, what do other people with oldish HP hardware do about this ? I know its off topic, but theres a lot of it out there... ) -pete.
> On 3 May 2019, at 11:55, Pete French <petefrench at ingresso.co.uk> wrote: > > > > On 03/05/2019 08:09, Borja Marcos via freebsd-stable wrote: > >> The right way to use disks is to give ZFS access to the plain CAM devices, not thorugh some so-called JBOD on a RAID >> controller which, at least for a long time, has been a *logical* ?RAID0? volume on a single disk. That additional layer can >> completely break the semantics of transaction writes and cache flushes. >> With some older cards it can be tricky to achieve, from patching source drivers to enabling a sysctl tunable or even >> flashing the card to turn it into a plain HBA with no RAID features (or minimal ones). > > Oddly enough I got bitten by something like this yesteray. I have a machine containing an HP P400 RAID controller, which is nice enough, but I run ZFS so I have made the drives all into RAID-0 as being as close as I can get to accessing the raw SAS drives. > > BSD seems them as da0, da1, da2, da3 - but the RAID controller oly presents one of them to the BIOS, so my booting has to be all from that drive. This has been da0 for as long as I can remember, but yesteday it decided to start using what BSD sees as da1. Of course this is very hard to recognise as da0 and da1 are pretty much mirrors of each other. Spent a long time trying to work out why the fixes I was applying to da0 were not being used at boot time.Hmm What happens when you do a ?camcontrol devlist?? Camcontrol tags da0 -v? How is the controller recognized by FreeBSD? For some of them it?s possible to instruct the controller to present the physical devices to CAM. Of course you need to be careful to avoid any logical volume configuration in that case. But I would only tinker with this at system installation time, making such a change on a running system with valid data can be disastrous. For mfi recognized cards there is a tunable: hw.mfi.allow_cam_disk_passthrough For aac cards it was a matter of commenting a couple of source code lines in the driver (at your own risk of course). I?ve been running a server for years doing that. Borja.
Pete French wrote on 2019/05/03 11:55:> Oddly enough I got bitten by something like this yesteray. I have a > machine containing an HP P400 RAID controller, which is nice enough, but > I run ZFS so I have made the drives all into RAID-0 as being as close as > I can get to accessing the raw SAS drives. > > BSD seems them as da0, da1, da2, da3 - but the RAID controller oly > presents one of them to the BIOS, so my booting has to be all from that > drive. This has been da0 for as long as I can remember, but yesteday it > decided to start using what BSD sees as da1. Of course this is very hard > to recognise as da0 and da1 are pretty much mirrors of each other. Spent > a long time trying to work out why the fixes I was applying to da0 were > not being used at boot time.I had this problem in the past too. I am not sure if it was on Dell or HP machine - controller presents first disk only in the boot time so I created small (10 - 15GB partition) on each disk and use them all in 4 way mirror. Cannot say if it was gmirror with UFS or ZFS mirroring. The rest of the each disk was used for ZFS RAIDZ.> ( Having to use Windows XP to talk to the iLo due to browsers dropping > support for the old ciphers doesnt help either, what do other people > with oldish HP hardware do about this ? I know its off topic, but theres > a lot of it out there... )I have WinXP guest in VirtualBox exactly for this situation. With old browser, with old Java, with Flash - believe it or not, Cisco UCS C200 has remote management created in Flash! Miroslav Lachman