CentOS - Nov 2016 - Avago (LSI) SAS-3 controller, poor performance on CentOS 7

I have a handful of new systems where I've seen unexpectedly low disk 
performance on an Avago SAS controller, when using CentOS 7. It looked 
like a regression, so I installed CentOS 6 on one of them and reloaded 
CentOS 7 on the other.  Immediately after install, a difference is 
apparent in the RAID rebuild speed.  The CentOS 6 system is initializing 
its software RAID5 array at somewhere around 120MB/s, while the CentOS 7 
system is rebuilding at around 55MB/s.

It took a while to get an older system installed, and I wasn't able to 
boot CentOS 6 under UEFI, to that's one difference between the systems.  
I also set the elevator on CentOS 7 to cfq, to match the CentOS 6 
system.  That didn't have any apparent effect.

Other than that, the kernel reports "MSI-X vectors supported: 96" on
C6,
but "MSI-X vectors supported: 8" on C7.  I don't know if
that's a
difference that would affect performance.

Does anyone have experience with these controllers?  Any pointers while 
I continue tracking this down?

Output from lspci follows, as well as dmesg output regarding the MPT 
controller and the RAID array status on C6 and C7.


01:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic 
SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
         Subsystem: LSI Logic / Symbios Logic Device 30e0
         Physical Slot: 1
         Flags: bus master, fast devsel, latency 0, IRQ 24
         I/O ports at e000 [size=256]
         Memory at fb240000 (64-bit, non-prefetchable) [size=64K]
         Memory at fb200000 (64-bit, non-prefetchable) [size=256K]
         Expansion ROM at fb100000 [disabled] [size=1M]
         Capabilities: [50] Power Management version 3
         Capabilities: [68] Express Endpoint, MSI 00
         Capabilities: [d0] Vital Product Data
         Capabilities: [a8] MSI: Enable- Count=1/1 Maskable+ 64bit+
         Capabilities: [c0] MSI-X: Enable+ Count=96 Masked-
         Capabilities: [100] Advanced Error Reporting
         Capabilities: [1e0] #19
         Capabilities: [1c0] Power Budgeting <?>
         Capabilities: [190] #16
         Capabilities: [148] Alternative Routing-ID Interpretation (ARI)
         Kernel driver in use: mpt3sas


CentOS 6:

# dmesg | grep -i mpt
mpt3sas version 09.102.00.00-rh loaded
mpt3sas 0000:01:00.0: PCI INT A -> GSI 26 (level, low) -> IRQ 26
mpt3sas 0000:01:00.0: setting latency timer to 64
mpt3sas_cm0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem 
(32772632 kB)
mpt3sas_cm0: MSI-X vectors supported: 96, no of cores: 12, 
max_msix_vectors: -1
mpt3sas 0000:01:00.0: irq 60 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 61 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 62 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 63 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 64 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 65 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 66 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 67 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 68 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 69 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 70 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 71 for MSI/MSI-X
mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 60
mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 61
mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 62
mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 63
mpt3sas0-msix4: PCI-MSI-X enabled: IRQ 64
mpt3sas0-msix5: PCI-MSI-X enabled: IRQ 65
mpt3sas0-msix6: PCI-MSI-X enabled: IRQ 66
mpt3sas0-msix7: PCI-MSI-X enabled: IRQ 67
mpt3sas0-msix8: PCI-MSI-X enabled: IRQ 68
mpt3sas0-msix9: PCI-MSI-X enabled: IRQ 69
mpt3sas0-msix10: PCI-MSI-X enabled: IRQ 70
mpt3sas0-msix11: PCI-MSI-X enabled: IRQ 71
mpt3sas_cm0: iomem(0x00000000fb240000), mapped(0xffffc90013ee0000), 
size(65536)
mpt3sas_cm0: ioport(0x000000000000e000), size(256)
mpt3sas_cm0: sending message unit reset !!
mpt3sas_cm0: message unit reset: SUCCESS
mpt3sas_cm0: Allocated physical memory: size(17971 kB)
mpt3sas_cm0: Current Controller Queue Depth(9979),Max Controller Queue 
Depth(10240)
mpt3sas_cm0: Scatter Gather Elements per IO(128)
mpt3sas_cm0: LSISAS3008: FWVersion(10.00.00.00), ChipRevision(0x02), 
BiosVersion(08.25.00.00)
mpt3sas_cm0: Protocol=(
scsi0 : Fusion MPT SAS Host
mpt3sas_cm0: sending port enable !!
mpt3sas_cm0: host_add: handle(0x0001), sas_addr(0x500605b00aee5510), phys(8)
mpt3sas_cm0: port enable: SUCCESS

#  cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [raid1]
md0 : active raid1 sda2[0] sdb2[1]
       999360 blocks super 1.0 [2/2] [UU]

md1 : active raid5 sdf3[6] sde3[4] sdd3[3] sdc3[2] sda3[0] sdb3[1]
       19528458240 blocks super 1.2 level 5, 64k chunk, algorithm 2 
[6/5] [UUUUU_]
       [====>................]  recovery = 22.1% (864374932/3905691648) 
finish=417.0min speed=121538K/sec
       bitmap: 1/30 pages [4KB], 65536KB chunk

unused devices: <none>


CentOS 7:

# dmesg | grep -i mpt
mpt3sas version 04.100.00.00 loaded
mpt3sas0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (32674176 kB)
mpt3sas0: MSI-X vectors supported: 8, no of cores: 12, max_msix_vectors: 8
mpt3sas 0000:01:00.0: irq 45 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 46 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 47 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 49 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 50 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 51 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 52 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 53 for MSI/MSI-X
mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 45
mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 46
mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 47
mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 49
mpt3sas0-msix4: PCI-MSI-X enabled: IRQ 50
mpt3sas0-msix5: PCI-MSI-X enabled: IRQ 51
mpt3sas0-msix6: PCI-MSI-X enabled: IRQ 52
mpt3sas0-msix7: PCI-MSI-X enabled: IRQ 53
mpt3sas0: iomem(0x00000000fb240000), mapped(0xffffc90013180000), size(65536)
mpt3sas0: ioport(0x000000000000e000), size(256)
mpt3sas0: sending message unit reset !!
mpt3sas0: message unit reset: SUCCESS
mpt3sas0: Allocated physical memory: size(17329 kB)
mpt3sas0: Current Controller Queue Depth(9979),Max Controller Queue 
Depth(10240)
mpt3sas0: Scatter Gather Elements per IO(128)
mpt3sas0: LSISAS3008: FWVersion(10.00.00.00), ChipRevision(0x02), 
BiosVersion(08.25.00.00)
mpt3sas0: Protocol=(
scsi host0: Fusion MPT SAS Host
mpt3sas0: sending port enable !!
mpt3sas0: host_add: handle(0x0001), sas_addr(0x500605b00aee5ca0), phys(8)
mpt3sas0: port enable: SUCCESS

#  cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md125 : active raid1 sda1[0] sdb1[1]
       194496 blocks super 1.0 [2/2] [UU]

md126 : active raid5 sdf3[6] sda3[0] sde3[4] sdd3[3] sdc3[2] sdb3[1]
       19528458240 blocks super 1.2 level 5, 64k chunk, algorithm 2 
[6/5] [UUUUU_]
       [=========>...........]  recovery = 47.6% (1862704880/3905691648) 
finish=569.6min speed=59772K/sec
       bitmap: 2/30 pages [8KB], 65536KB chunk

md127 : active raid1 sda2[0] sdb2[1]
       999360 blocks super 1.0 [2/2] [UU]

unused devices: <none>

> On Nov 4, 2016, at 11:08 PM, Gordon Messmer <gordon.messmer at
gmail.com> wrote:
> 
> LSI Logic / Symbios Logic SAS3008
What firmware is/are on the cards?
The driver version difference between C6 and C7 could interact badly with old
firmware.
What firmware does $vendor suggest?

Does C7 in BIOS mode work any better?

Run `lsiget` and compare lsi tech support bundles.

On 11/05/2016 06:40 AM, Steven Tardy wrote:
> What firmware is/are on the cards?
In the dmesg output I included:
mpt3sas0: LSISAS3008: FWVersion(10.00.00.00), ChipRevision(0x02), 
BiosVersion(08.25.00.00)
> The driver version difference between C6 and C7 could interact badly with
old firmware.
I suspected that was the case, and tried running elrepo's
"kmod-mpt3sas"
from their testing repo.  C7 has version 04.100.00.00, while elrepo's 
appears to be 13.100.00.00.  That didn't change anything.
> What firmware does $vendor suggest?
Haven't asked yet, since the same firmware produces different behavior 
on the two OSs.
> Does C7 in BIOS mode work any better?
That'll be one of my next tests.
> Run `lsiget` and compare lsi tech support bundles.
I'll look around for that, too.  No vendor tools are installed, currently.

For the archive's sake:

https://plus.google.com/+GordonMessmer/posts/H5DuyP1LHPU?sfc=false
https://plus.google.com/+GordonMessmer/posts/eSe6iNmk1Fs?sfc=false

During testing, I found that under CentOS 7.2, the default storage 
layout for software RAID5 was the worst possible configuration. 512k 
blocks perform noticeably worse than 64k blocks, and LVM had a 
tremendous impact as well.

After upgrading to CentOS 7.3, which features a newer mpt3sas driver, 
performance improved significantly.  LVM's overhead has decreased to an 
acceptable level (from my point of view).

Because chunk size has a significant impact on performance, I'm going to 
file an RFE to change the default chunk size for RAID5 and RAID6 volumes.