John Baldwin
2011-Aug-31 17:34 UTC
mfi(4) patch to add MSI-X support, possibly address command timeouts
I'd like some folks to test a patch to the mfi(4) driver that may help to
address issues several folks have reported. The patch does two things, first
it adds some dummy reads of PCI registers when checking device status in the
interrupt handler to "flush" the writes to ACK interrupts. The Linux
megaraid-sas driver uses this approach and some folks have tested a patch from
Scott Long which had a somewhat similar effect. Second, it enables the use of
MSI-X interrupts for many newer devices.
The patch is available below and at www.freebsd.org/~jhb/patches/mfi.patch
Index: mfi_pci.c
==================================================================--- mfi_pci.c
(revision 224613)
+++ mfi_pci.c (working copy)
@@ -169,7 +169,7 @@
struct mfi_softc *sc;
struct mfi_ident *m;
uint32_t command;
- int error;
+ int count, error;
sc = device_get_softc(dev);
bzero(sc, sizeof(*sc));
@@ -226,6 +226,29 @@
goto out;
}
+ /* Allocate IRQ resource. */
+ sc->mfi_irq_rid = 0;
+ switch (pci_get_device(sc->mfi_dev)) {
+ case 0x0060: /* SAS1078R */
+ case 0x007c: /* SAS1078DE */
+ case 0x0413: /* Verde ZCR */
+ /* Do not use MSI-X for these systems. */
+ break;
+ default:
+ count = 1;
+ if (pci_alloc_msix(sc->mfi_dev, &count) == 0) {
+ device_printf(sc->mfi_dev, "Using MSI-X\n");
+ sc->mfi_irq_rid = 1;
+ }
+ break;
+ }
+ if ((sc->mfi_irq = bus_alloc_resource_any(sc->mfi_dev, SYS_RES_IRQ,
+ &sc->mfi_irq_rid, RF_SHAREABLE | RF_ACTIVE)) == NULL) {
+ device_printf(sc->mfi_dev, "Cannot allocate interrupt\n");
+ error = EINVAL;
+ goto out;
+ }
+
error = mfi_attach(sc);
out:
if (error) {
@@ -280,6 +303,8 @@
bus_release_resource(sc->mfi_dev, SYS_RES_MEMORY,
sc->mfi_regs_rid, sc->mfi_regs_resource);
}
+ if (sc->mfi_irq_rid != 0)
+ pci_release_msi(sc->mfi_dev);
return;
}
Index: mfi.c
==================================================================--- mfi.c
(revision 224613)
+++ mfi.c (working copy)
@@ -157,6 +157,9 @@
mfi_enable_intr_xscale(struct mfi_softc *sc)
{
MFI_WRITE4(sc, MFI_OMSK, 0x01);
+
+ /* Dummy read to force PCI flush. */
+ (void)MFI_READ4(sc, MFI_OMSK);
}
static void
@@ -168,6 +171,9 @@
} else if (sc->mfi_flags & MFI_FLAGS_GEN2) {
MFI_WRITE4(sc, MFI_OMSK, ~MFI_GEN2_EIM);
}
+
+ /* Dummy read to force PCI flush. */
+ (void)MFI_READ4(sc, MFI_OMSK);
}
static int32_t
@@ -192,6 +198,9 @@
return 1;
MFI_WRITE4(sc, MFI_OSTS, status);
+
+ /* Dummy read to force PCI flush. */
+ (void)MFI_READ4(sc, MFI_OSTS);
return 0;
}
@@ -212,6 +221,9 @@
}
MFI_WRITE4(sc, MFI_ODCR0, status);
+
+ /* Dummy read to force PCI flush. */
+ (void)MFI_READ4(sc, MFI_OSTS);
return 0;
}
@@ -484,15 +496,8 @@
mtx_unlock(&sc->mfi_io_lock);
/*
- * Set up the interrupt handler. XXX This should happen in
- * mfi_pci.c
+ * Set up the interrupt handler.
*/
- sc->mfi_irq_rid = 0;
- if ((sc->mfi_irq = bus_alloc_resource_any(sc->mfi_dev, SYS_RES_IRQ,
- &sc->mfi_irq_rid, RF_SHAREABLE | RF_ACTIVE)) == NULL) {
- device_printf(sc->mfi_dev, "Cannot allocate interrupt\n");
- return (EINVAL);
- }
if (bus_setup_intr(sc->mfi_dev, sc->mfi_irq, INTR_MPSAFE|INTR_TYPE_BIO,
NULL, mfi_intr, sc, &sc->mfi_intr)) {
device_printf(sc->mfi_dev, "Cannot set up interrupt\n");
--
John Baldwin
Sergey Kandaurov
2011-Aug-31 19:55 UTC
mfi(4) patch to add MSI-X support, possibly address command timeouts
On 31 August 2011 21:34, John Baldwin <jhb@freebsd.org> wrote:> I'd like some folks to test a patch to the mfi(4) driver that may help to > address issues several folks have reported. ?The patch does two things, first > it adds some dummy reads of PCI registers when checking device status in the > interrupt handler to "flush" the writes to ACK interrupts. ?The Linux > megaraid-sas driver uses this approach and some folks have tested a patch from > Scott Long which had a somewhat similar effect. ?Second, it enables the use of > MSI-X interrupts for many newer devices. > > The patch is available below and at www.freebsd.org/~jhb/patches/mfi.patchmfi0: <LSI MegaSAS Gen2> port 0x3000-0x30ff mem 0x9dd40000-0x9dd43fff,0x9dd00000-0x9dd3ffff irq 26 at device 0.0 on pci26 mfi0: Using MSI-X mfi0: Megaraid SAS driver Ver 3.00 However, booting never finishes ending up with: mfi0: COMMAND 0xffffff8000b3a550 TIMEOUT AFTER 58 SECONDS mfi0: COMMAND 0xffffff8000b3a550 TIMEOUT AFTER 88 SECONDS mfi0: COMMAND 0xffffff8000b3a550 TIMEOUT AFTER 118 SECONDS mfi0: COMMAND 0xffffff8000b3a550 TIMEOUT AFTER 148 SECONDS mfi0: COMMAND 0xffffff8000b3a550 TIMEOUT AFTER 179 SECONDS mfi0: COMMAND 0xffffff8000b3a550 TIMEOUT AFTER 209 SECONDS Patch applied and tested on RELENG_8_2. mfi0@pci0:26:0:0: class=0x010400 card=0x03b21014 chip=0x00791000 rev=0x03 hdr=0x00 vendor = 'LSI Logic (Was: Symbios Logic, NCR)' class = mass storage subclass = RAID bar [10] = type I/O Port, range 32, base 0x3000, size 256, enabled bar [14] = type Memory, range 64, base 0x9dd40000, size 16384, enabled bar [1c] = type Memory, range 64, base 0x9dd00000, size 262144, enabled cap 01[50] = powerspec 3 supports D0 D1 D2 D3 current D0 cap 10[68] = PCI-Express 2 endpoint max data 256(4096) link x8(x8) cap 03[d0] = VPD cap 05[a8] = MSI supports 1 message, 64 bit cap 11[c0] = MSI-X supports 15 messages in map 0x14 ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected ecap 0004[138] = unknown 1 -- wbr, pluknet
David Wood
2011-Sep-02 16:20 UTC
mfi(4) patch to add MSI-X support, possibly address command timeouts
Dear John (and freebsd-stable), In message <201108311334.10804.jhb@freebsd.org>, John Baldwin <jhb@freebsd.org> writes>I'd like some folks to test a patch to the mfi(4) driver that may help to >address issues several folks have reported. The patch does two things, first >it adds some dummy reads of PCI registers when checking device status in the >interrupt handler to "flush" the writes to ACK interrupts. The Linux >megaraid-sas driver uses this approach and some folks have tested a patch from >Scott Long which had a somewhat similar effect. Second, it enables the use of >MSI-X interrupts for many newer devices.I've installed this patch on my Dell PowerEdge 2950 III with a PERC 6/i controller, which is running 8.2-RELEASE. So far all is working well. Is there any hope of MSI-X support for this device? It's 0x0060 - a SAS1078R - so is deliberately excluded from MSI-X by this patch. The system appears capable of MSI-X operation, but there might be a silicon bug I'm not aware of or extra code needed that is not provided by this patch. With best wishes, David -- David Wood david@wood2.org.uk