freebsd stable - Feb 2014 - FreeBSD 10, ServeRAID M5210e, syspd corruption

(crossposting to -Stable just in case)

Hello,

I am configuring an IBM server with FreeBSD 10-RELEASE, a ServeRAID M5210e and
23 SSD disks.

uname -a
FreeBSD hostname 10.0-RELEASE FreeBSD 10.0-RELEASE #1: Fri Feb 14 09:35:12 CET
2014     toor@
hostname:/usr/obj/usr/src/sys/GENERIC  amd64


The server has a SAS backplane and a controller recognized by the mfi driver.

mfi0 Adapter:
    Product Name: ServeRAID M5210e
   Serial Number: 3CJ0SG          
        Firmware: 24.0.2-0013
     RAID Levels: JBOD, RAID0, RAID1, RAID10
  Battery Backup: not present
           NVRAM: 32K
  Onboard Memory: 0M
  Minimum Stripe: 64K
  Maximum Stripe: 64K


As I am intending to use ZFS, I need direct access to the disks, no need for
fancy RAID features.

I have seen that the newest cards support a so-called "syspd" mode
that gives direct acess to the disks.
However, in this configuration, syspd consistently corrupts data on the disks.

I have done tests with three models of disks:

- Samsung SSD 840 BB0Q (1 TB)
- OCZ-VERTEX4 1.5  (512 GB)
- SEAGATE ST9146803SS FS03 (136 GB)

In the three cases there is data corruption. Using FFS on the disks results in a
panic if I run a benchmark, for example, bonnie++.
Using ZFS (I've been creating one disk pools to test) I don't get panics
but the data  is consistently corrupted. The writes work, but whenever
there is read activity (either bonnie++ reaching the "rewrite" phase,
or a ZFS scrub), ZFS detects data corruption.

Trying the ?ber neat hw.mfi.allow_cam_disk_passthrough (which is great, because
ZFS can detect the SSDs and issue TRIM commands)
I get the same result: data corruption.

However, I have tried to create a one-disk raid0 volume, and in that case it
works like a charm, no corruption at all, so I can safely
assume that this is not a defective backplane, expander or cabling. 

So:

mfisyspd -> CORRUPT
da -> CORRUPT
mfid -> NOT CORRUPT

Any ideas? Could be a driver error or a firmware problem, I am clueless for now.

Anything I can test? The machine is not in production, I can try patches or
whatever.




Thanks!!






Borja.

On Fri, Feb 14, 2014 at 12:44 PM, Borja Marcos <borjam at sarenet.es>
wrote:
> (crossposting to -Stable just in case)
>
> Hello,
>
> I am configuring an IBM server with FreeBSD 10-RELEASE, a ServeRAID M5210e
and 23 SSD disks.
>
I'm afraid I have no solution to offer you for this issue, but with
this setup an mps(8) card (LSI SAS2008 and similar) in IT (passthru)
mode would work excellently.

Maybe easier to change the card than struggle to get it to do
something it doesn't want to?

Cheers

Tom

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 02/14/14 07:44, Borja Marcos wrote:
> I have seen that the newest cards support a so-called "syspd"
mode that gives direct acess to the disks.
> However, in this configuration, syspd consistently corrupts data on the
disks.
Just a silly "left field" idea .. can you try adding ..

vfs.unmapped_buf_allowed=0

 .. to /boot/loader.conf

and re-testing?

As I say, just an idea ..

	Michael

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iEYEARECAAYFAlL+IWAACgkQQv9rrgRC1JK/LwCfS8cO2BWAimSO/W7Y8E1VsjuK
L6wAnRgl46sXeO1f04Dn1oF+wwHK1zng
=k1oh
-----END PGP SIGNATURE-----