Gordon Messmer
2016-Nov-05 03:08 UTC
[CentOS] Avago (LSI) SAS-3 controller, poor performance on CentOS 7
I have a handful of new systems where I've seen unexpectedly low disk
performance on an Avago SAS controller, when using CentOS 7. It looked
like a regression, so I installed CentOS 6 on one of them and reloaded
CentOS 7 on the other. Immediately after install, a difference is
apparent in the RAID rebuild speed. The CentOS 6 system is initializing
its software RAID5 array at somewhere around 120MB/s, while the CentOS 7
system is rebuilding at around 55MB/s.
It took a while to get an older system installed, and I wasn't able to
boot CentOS 6 under UEFI, to that's one difference between the systems.
I also set the elevator on CentOS 7 to cfq, to match the CentOS 6
system. That didn't have any apparent effect.
Other than that, the kernel reports "MSI-X vectors supported: 96" on
C6,
but "MSI-X vectors supported: 8" on C7. I don't know if
that's a
difference that would affect performance.
Does anyone have experience with these controllers? Any pointers while
I continue tracking this down?
Output from lspci follows, as well as dmesg output regarding the MPT
controller and the RAID array status on C6 and C7.
01:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic
SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
Subsystem: LSI Logic / Symbios Logic Device 30e0
Physical Slot: 1
Flags: bus master, fast devsel, latency 0, IRQ 24
I/O ports at e000 [size=256]
Memory at fb240000 (64-bit, non-prefetchable) [size=64K]
Memory at fb200000 (64-bit, non-prefetchable) [size=256K]
Expansion ROM at fb100000 [disabled] [size=1M]
Capabilities: [50] Power Management version 3
Capabilities: [68] Express Endpoint, MSI 00
Capabilities: [d0] Vital Product Data
Capabilities: [a8] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [c0] MSI-X: Enable+ Count=96 Masked-
Capabilities: [100] Advanced Error Reporting
Capabilities: [1e0] #19
Capabilities: [1c0] Power Budgeting <?>
Capabilities: [190] #16
Capabilities: [148] Alternative Routing-ID Interpretation (ARI)
Kernel driver in use: mpt3sas
CentOS 6:
# dmesg | grep -i mpt
mpt3sas version 09.102.00.00-rh loaded
mpt3sas 0000:01:00.0: PCI INT A -> GSI 26 (level, low) -> IRQ 26
mpt3sas 0000:01:00.0: setting latency timer to 64
mpt3sas_cm0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem
(32772632 kB)
mpt3sas_cm0: MSI-X vectors supported: 96, no of cores: 12,
max_msix_vectors: -1
mpt3sas 0000:01:00.0: irq 60 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 61 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 62 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 63 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 64 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 65 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 66 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 67 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 68 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 69 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 70 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 71 for MSI/MSI-X
mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 60
mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 61
mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 62
mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 63
mpt3sas0-msix4: PCI-MSI-X enabled: IRQ 64
mpt3sas0-msix5: PCI-MSI-X enabled: IRQ 65
mpt3sas0-msix6: PCI-MSI-X enabled: IRQ 66
mpt3sas0-msix7: PCI-MSI-X enabled: IRQ 67
mpt3sas0-msix8: PCI-MSI-X enabled: IRQ 68
mpt3sas0-msix9: PCI-MSI-X enabled: IRQ 69
mpt3sas0-msix10: PCI-MSI-X enabled: IRQ 70
mpt3sas0-msix11: PCI-MSI-X enabled: IRQ 71
mpt3sas_cm0: iomem(0x00000000fb240000), mapped(0xffffc90013ee0000),
size(65536)
mpt3sas_cm0: ioport(0x000000000000e000), size(256)
mpt3sas_cm0: sending message unit reset !!
mpt3sas_cm0: message unit reset: SUCCESS
mpt3sas_cm0: Allocated physical memory: size(17971 kB)
mpt3sas_cm0: Current Controller Queue Depth(9979),Max Controller Queue
Depth(10240)
mpt3sas_cm0: Scatter Gather Elements per IO(128)
mpt3sas_cm0: LSISAS3008: FWVersion(10.00.00.00), ChipRevision(0x02),
BiosVersion(08.25.00.00)
mpt3sas_cm0: Protocol=(
scsi0 : Fusion MPT SAS Host
mpt3sas_cm0: sending port enable !!
mpt3sas_cm0: host_add: handle(0x0001), sas_addr(0x500605b00aee5510), phys(8)
mpt3sas_cm0: port enable: SUCCESS
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [raid1]
md0 : active raid1 sda2[0] sdb2[1]
999360 blocks super 1.0 [2/2] [UU]
md1 : active raid5 sdf3[6] sde3[4] sdd3[3] sdc3[2] sda3[0] sdb3[1]
19528458240 blocks super 1.2 level 5, 64k chunk, algorithm 2
[6/5] [UUUUU_]
[====>................] recovery = 22.1% (864374932/3905691648)
finish=417.0min speed=121538K/sec
bitmap: 1/30 pages [4KB], 65536KB chunk
unused devices: <none>
CentOS 7:
# dmesg | grep -i mpt
mpt3sas version 04.100.00.00 loaded
mpt3sas0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (32674176 kB)
mpt3sas0: MSI-X vectors supported: 8, no of cores: 12, max_msix_vectors: 8
mpt3sas 0000:01:00.0: irq 45 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 46 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 47 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 49 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 50 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 51 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 52 for MSI/MSI-X
mpt3sas 0000:01:00.0: irq 53 for MSI/MSI-X
mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 45
mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 46
mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 47
mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 49
mpt3sas0-msix4: PCI-MSI-X enabled: IRQ 50
mpt3sas0-msix5: PCI-MSI-X enabled: IRQ 51
mpt3sas0-msix6: PCI-MSI-X enabled: IRQ 52
mpt3sas0-msix7: PCI-MSI-X enabled: IRQ 53
mpt3sas0: iomem(0x00000000fb240000), mapped(0xffffc90013180000), size(65536)
mpt3sas0: ioport(0x000000000000e000), size(256)
mpt3sas0: sending message unit reset !!
mpt3sas0: message unit reset: SUCCESS
mpt3sas0: Allocated physical memory: size(17329 kB)
mpt3sas0: Current Controller Queue Depth(9979),Max Controller Queue
Depth(10240)
mpt3sas0: Scatter Gather Elements per IO(128)
mpt3sas0: LSISAS3008: FWVersion(10.00.00.00), ChipRevision(0x02),
BiosVersion(08.25.00.00)
mpt3sas0: Protocol=(
scsi host0: Fusion MPT SAS Host
mpt3sas0: sending port enable !!
mpt3sas0: host_add: handle(0x0001), sas_addr(0x500605b00aee5ca0), phys(8)
mpt3sas0: port enable: SUCCESS
# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md125 : active raid1 sda1[0] sdb1[1]
194496 blocks super 1.0 [2/2] [UU]
md126 : active raid5 sdf3[6] sda3[0] sde3[4] sdd3[3] sdc3[2] sdb3[1]
19528458240 blocks super 1.2 level 5, 64k chunk, algorithm 2
[6/5] [UUUUU_]
[=========>...........] recovery = 47.6% (1862704880/3905691648)
finish=569.6min speed=59772K/sec
bitmap: 2/30 pages [8KB], 65536KB chunk
md127 : active raid1 sda2[0] sdb2[1]
999360 blocks super 1.0 [2/2] [UU]
unused devices: <none>
Steven Tardy
2016-Nov-05 13:40 UTC
[CentOS] Avago (LSI) SAS-3 controller, poor performance on CentOS 7
> On Nov 4, 2016, at 11:08 PM, Gordon Messmer <gordon.messmer at gmail.com> wrote: > > LSI Logic / Symbios Logic SAS3008What firmware is/are on the cards? The driver version difference between C6 and C7 could interact badly with old firmware. What firmware does $vendor suggest? Does C7 in BIOS mode work any better? Run `lsiget` and compare lsi tech support bundles.
Gordon Messmer
2016-Nov-05 19:08 UTC
[CentOS] Avago (LSI) SAS-3 controller, poor performance on CentOS 7
On 11/05/2016 06:40 AM, Steven Tardy wrote:> What firmware is/are on the cards?In the dmesg output I included: mpt3sas0: LSISAS3008: FWVersion(10.00.00.00), ChipRevision(0x02), BiosVersion(08.25.00.00)> The driver version difference between C6 and C7 could interact badly with old firmware.I suspected that was the case, and tried running elrepo's "kmod-mpt3sas" from their testing repo. C7 has version 04.100.00.00, while elrepo's appears to be 13.100.00.00. That didn't change anything.> What firmware does $vendor suggest?Haven't asked yet, since the same firmware produces different behavior on the two OSs.> Does C7 in BIOS mode work any better?That'll be one of my next tests.> Run `lsiget` and compare lsi tech support bundles.I'll look around for that, too. No vendor tools are installed, currently.
Gordon Messmer
2017-Jan-18 22:04 UTC
[CentOS] Avago (LSI) SAS-3 controller, poor performance on CentOS 7
For the archive's sake: https://plus.google.com/+GordonMessmer/posts/H5DuyP1LHPU?sfc=false https://plus.google.com/+GordonMessmer/posts/eSe6iNmk1Fs?sfc=false During testing, I found that under CentOS 7.2, the default storage layout for software RAID5 was the worst possible configuration. 512k blocks perform noticeably worse than 64k blocks, and LVM had a tremendous impact as well. After upgrading to CentOS 7.3, which features a newer mpt3sas driver, performance improved significantly. LVM's overhead has decreased to an acceptable level (from my point of view). Because chunk size has a significant impact on performance, I'm going to file an RFE to change the default chunk size for RAID5 and RAID6 volumes.