freebsd stable - Jun 2016 - mfi driver performance too bad on LSI MegaRAID SAS 9260-8i

Hi,

I am working on storage service based on FreeBSD.  I look forward to a good
result because many professional storage company use FreeBSD as its OS.  But I
am disappointed with the Bad performance.  I tested the the performance of LSI
MegaRAID 9260-8i and had the following bad result:

   1.  Test environment:
        (1) OS:   FreeBSD 10.0 release
        (2) Memory:  16G
        (3) RAID adapter:   LSI MegaRAID 9260-8i
        (4) Disks:  9 SAS hard drives (10000 rpm),  performance is expected for
each hard drive
        (5) Test tools:   fio with  io-depth=1, thread num is 32 and block size
is 64k or 1M
        (6)  RAID configuration:  RAID 5,   stripe size is 1M

  2.  Test result:
       (1)  write performance too bad:  20Mbytes/s throughput and 200 random
write IOPS
       (2)  read performance is expected:  700Mbytes/s throughput and 1500
random read IOPS


I tested the same hardware configuration with CentOS linux and Linux's write
performance is 5 times better than FreeBSD.


Anyone encountered the same performance problem?  Does the mfi driver have
performance issue or I should give up on FreeBSD?





???   Jason

??????????????              
Cyphy  Technology  (Xiamen)  Co.Ltd.
??????????????55?A?901-904??
?????????????????????2?
???4008798066
???0592-2936100
???jasonzhang at cyphytech.com
?????Http://www.cyphytech.com

On 17/06/2016 3:16 PM, Jason Zhang wrote:
> Hi,
>
> I am working on storage service based on FreeBSD.  I look forward to a good
result because many professional storage company use FreeBSD as its OS.  But I
am disappointed with the Bad performance.  I tested the the performance of LSI
MegaRAID 9260-8i and had the following bad result:
>
>     1.  Test environment:
>          (1) OS:   FreeBSD 10.0 release
>          (2) Memory:  16G
>          (3) RAID adapter:   LSI MegaRAID 9260-8i
>          (4) Disks:  9 SAS hard drives (10000 rpm),  performance is
expected for each hard drive
>          (5) Test tools:   fio with  io-depth=1, thread num is 32 and block
size is 64k or 1M
>          (6)  RAID configuration:  RAID 5,   stripe size is 1M
>
>    2.  Test result:
>         (1)  write performance too bad:  20Mbytes/s throughput and 200
random write IOPS
>         (2)  read performance is expected:  700Mbytes/s throughput and 1500
random read IOPS
>
>
> I tested the same hardware configuration with CentOS linux and Linux's
write performance is 5 times better than FreeBSD.
>
>
> Anyone encountered the same performance problem?  Does the mfi driver have
performance issue or I should give up on FreeBSD?
>
>
Unfortunatley issues related to performance can often be very specific.
We use the LSI cards with great success under FreeBSD 8 in our product 
at work but it is impossible to say what is specifically wrong in your 
setup.

Some years ago I did discover that fio needed to have completely 
different arguments to get good performance under FreeBSD, so please 
check that first.

What does performance look like with a single large write stream?

Also look at the handling of interrupts (systat -vmstat) to ensure 
that interrupts are being handled correctly.
that can vary greatly from motherboard to motherboard  and bios to 
bios. (even between revisions).
Sometimes Linux will cope differently  with these issues as they have 
better support from the motherboard makers themselves.
(sometimes we cope better too).

One final thought.. make sure you have partitioned your drives and 
filesyste,s so that all the block boundaries agree and line up.
At on place I worked we found we had accidentally  partitioned all our 
drives starting 63 sectors into the drive.
That did NOT work well. :-) 8k raid stripe writes were always 2 
writes  (and sometimes a read)
>
>
> ???   Jason
>
> ??????????????
> Cyphy  Technology  (Xiamen)  Co.Ltd.
> ??????????????55?A?901-904??
> ?????????????????????2?
> ???4008798066
> ???0592-2936100
> ???jasonzhang at cyphytech.com
> ?????Http://www.cyphytech.com
>
>
> _______________________________________________
> freebsd-performance at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-performance
> To unsubscribe, send any mail to "freebsd-performance-unsubscribe at
freebsd.org"
>
>

Bez?glich Jason Zhang's Nachricht vom 17.06.2016 09:16
(localtime):
> Hi,
>
> I am working on storage service based on FreeBSD.  I look forward to a good
result because many professional storage company use FreeBSD as its OS.  But I
am disappointed with the Bad performance.  I tested the the performance of LSI
MegaRAID 9260-8i and had the following bad result:
>
>    1.  Test environment:
>         (1) OS:   FreeBSD 10.0 release
>         (2) Memory:  16G
>         (3) RAID adapter:   LSI MegaRAID 9260-8i
>         (4) Disks:  9 SAS hard drives (10000 rpm),  performance is expected
for each hard drive
Were the drives completely initialized?
I remember that at least one vendor had implemented read-past-write for
every sector when written first.
It was with 15k 3.5" spindles and I'm really not sure which vendor it
was, so I won't name any. But "slow init" had solved a similir
problem
for me back then...

-Harry