On Thu, Jul 3, 2014 at 11:56 PM, Yijing Wang <wangyijing at huawei.com> wrote:> On 2014/7/4 11:30, Ilia Mirkin wrote: >> On Thu, Jul 3, 2014 at 11:09 PM, Yijing Wang <wangyijing at huawei.com> wrote: >>> On 2014/7/4 10:43, Ilia Mirkin wrote: >>>> On Thu, Jul 3, 2014 at 10:35 PM, Yijing Wang <wangyijing at huawei.com> wrote: >>>>> Hi Brian, >>>>> From your 01:00.0 VGA compatible controller PCI config register, it supports 1 MSI vector, so I think this >>>>> card has no problem. But you didn't answer what's the pci_enable_msi() return during it enable MSI fail. >>>>> >>>>> You can check PCI bus whether support MSI like: >>>>> >>>>> cat /sys/bus/pci/devices/00:0e.0/msi_bus >>>>> >>>>> Other, do you call pci_enable_device() before pci_enable_msi() ? >>>> >>>> Yes, the device is enabled. nouveau does generally work for most >>>> people :) And it works fine here if MSI is forced off. >>>> >>>> The problem here is that pci_enable_msi() succeeds, but it probably >>> >>> It's some strange, pci_enable_msi() succeeds, but there is nothing in MSI address and data register >>> >>> Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ >>> Address: 0000000000000000 Data: 0000 >>> Because the address and data is invalid, so MSI can not work. >>> >>> Because you said pci_enable_msi() return 0 which indicates success, so I guess the device >>> current power state is not D0, because checking code in __write_msi_msi. >>> >>> void __write_msi_msg(struct msi_desc *entry, struct msi_msg *msg) >>> { >>> if (entry->dev->current_state != PCI_D0) { >>> /* Don't touch the hardware now */ >>> } else if (entry->msi_attrib.is_msix) { >>> void __iomem *base; >>> base = entry->mask_base + >>> entry->msi_attrib.entry_nr * PCI_MSIX_ENTRY_SIZE; >>> >>> writel(msg->address_lo, base + PCI_MSIX_ENTRY_LOWER_ADDR); >>> writel(msg->address_hi, base + PCI_MSIX_ENTRY_UPPER_ADDR); >>> writel(msg->data, base + PCI_MSIX_ENTRY_DATA); >>> ...... >>> >>>> shouldn't. I don't know all the details about MSI, but doesn't the CPU >>>> or (L)APIC have to support it? In this case, it's a P200MMX on a 430FX >>>> chipset board. Both quite ancient... >>> >>> MSI in x86 always send to specific interrupt address(0xfeexxxx), most x86 CPU >>> should support MSI, but I am not sure, I don't know hardware much. >> >> To put things in perspective, P200MMX was released in early 1997. It >> looks like the PCI 2.2 spec, which defines MSI, was released in late >> 1998. The 430FX chipset only supports PCI 2.0. > > Maybe you are right, the chipset can not support MSI... > > But why after your call pci_enable_msi(), the MSI address and data register is nothing initialized.The lspci output you saw was not with pci_enable_msi having been run -- it was in the working configuration, where nouveau is told not to enable MSI.> > >> >>> >>>> >>>> So given that the PCI device itself supports MSI, how do we tell that >>>> it shouldn't actually get turned on? >>> >>> Generally, some message in dmesg like: >>> [ 17.322311] ahci 0000:00:1f.2: irq 119 for MSI/MSI-X >>> [ 17.322319] ahci 0000:00:1f.2: irq 120 for MSI/MSI-X >>> [ 17.322326] ahci 0000:00:1f.2: irq 121 for MSI/MSI-X >>> [ 17.322333] ahci 0000:00:1f.2: irq 122 for MSI/MSI-X >>> [ 17.322339] ahci 0000:00:1f.2: irq 123 for MSI/MSI-X >>> >>> Also you can check /proc/interrupts I think. >> >> I must not have made my question clear. Let me try again: >> >> There is a NVIDIA G96 GPU (which is PCIe only) hanging off of a PCIe >> <-> PCI bridge (all on one card), which is plugged into a motherboard >> with the 430FX chipset (PCI 2.0 supported). >> >> The GPU PCI device, of course, has full support for MSI. However my >> understanding is that MSI won't actually work here. This is confirmed >> by the fact that if we let nouveau enable MSI, the device doesn't work >> (presumably due to lack of interrupt delivery, although I admit to not >> having debugged it that far). How do I, as a nouveau driver developer, >> know not to call pci_enable_msi? Or alternatively how can >> pci_enable_msi be taught not to succeed in this case? > > You can set bus_flags or global pci_msi_enable flag by add quirk function. > You can refer to examples in drivers/pci/quirk.c > > Linux support some broken chipsets or devices to disable msi during device initialization by add quirk.So let me get this straight -- you're suggesting I add a quirk for every PCI chipset that doesn't support MSI? There are probably hundreds of these... anything made before 1999 or so, and probably a bunch since then too. There _has_ to be a way to do this generically. Is the PCI spec version anywhere in the root hub? Perhaps we can check if every bridge on the way to the CPU has the MSI capability (including the root hub)? (And naturally _that_ won't work... on my sandybridge laptop, the host bridge doesn't have the MSI cap but the system most definitely supports MSI.) Adding Bjorn... perhaps you know? Some of the info has been stripped out by now, you can see the full lspci -vvvxxx at http://marc.info/?l=linux-pci&m=140443441730503&w=2 -ilia
>>> There is a NVIDIA G96 GPU (which is PCIe only) hanging off of a PCIe >>> <-> PCI bridge (all on one card), which is plugged into a motherboard >>> with the 430FX chipset (PCI 2.0 supported). >>> >>> The GPU PCI device, of course, has full support for MSI. However my >>> understanding is that MSI won't actually work here. This is confirmed >>> by the fact that if we let nouveau enable MSI, the device doesn't work >>> (presumably due to lack of interrupt delivery, although I admit to not >>> having debugged it that far). How do I, as a nouveau driver developer, >>> know not to call pci_enable_msi? Or alternatively how can >>> pci_enable_msi be taught not to succeed in this case? >> >> You can set bus_flags or global pci_msi_enable flag by add quirk function. >> You can refer to examples in drivers/pci/quirk.c >> >> Linux support some broken chipsets or devices to disable msi during device initialization by add quirk. > > So let me get this straight -- you're suggesting I add a quirk for > every PCI chipset that doesn't support MSI? There are probably > hundreds of these... anything made before 1999 or so, and probably a > bunch since then too. There _has_ to be a way to do this generically. > Is the PCI spec version anywhere in the root hub?There is no register to identify PCI spec version in PCI config space registers. If your platform boot up with ACPI, you can setting ACPI FADT boot flag to disable MSI, you can refer to this in ACPI 5.2.9.3 chapter "Fixed ACPI Description Table Boot Architecture Flags"> > Perhaps we can check if every bridge on the way to the CPU has the MSI > capability (including the root hub)? (And naturally _that_ won't > work... on my sandybridge laptop, the host bridge doesn't have the MSI > cap but the system most definitely supports MSI.) > > Adding Bjorn... perhaps you know? Some of the info has been stripped > out by now, you can see the full lspci -vvvxxx at > http://marc.info/?l=linux-pci&m=140443441730503&w=2 > > -ilia > > . >-- Thanks! Yijing
I am booting a kernel with CONFIG_ACPI=n on a platform which does not support ACPI. -Brian On Fri, Jul 4, 2014 at 1:59 AM, Yijing Wang <wangyijing at huawei.com> wrote:>>>> There is a NVIDIA G96 GPU (which is PCIe only) hanging off of a PCIe >>>> <-> PCI bridge (all on one card), which is plugged into a motherboard >>>> with the 430FX chipset (PCI 2.0 supported). >>>> >>>> The GPU PCI device, of course, has full support for MSI. However my >>>> understanding is that MSI won't actually work here. This is confirmed >>>> by the fact that if we let nouveau enable MSI, the device doesn't work >>>> (presumably due to lack of interrupt delivery, although I admit to not >>>> having debugged it that far). How do I, as a nouveau driver developer, >>>> know not to call pci_enable_msi? Or alternatively how can >>>> pci_enable_msi be taught not to succeed in this case? >>> >>> You can set bus_flags or global pci_msi_enable flag by add quirk function. >>> You can refer to examples in drivers/pci/quirk.c >>> >>> Linux support some broken chipsets or devices to disable msi during device initialization by add quirk. >> >> So let me get this straight -- you're suggesting I add a quirk for >> every PCI chipset that doesn't support MSI? There are probably >> hundreds of these... anything made before 1999 or so, and probably a >> bunch since then too. There _has_ to be a way to do this generically. >> Is the PCI spec version anywhere in the root hub? > > There is no register to identify PCI spec version in PCI config space registers. > If your platform boot up with ACPI, you can setting ACPI FADT boot flag to disable MSI, > you can refer to this in ACPI 5.2.9.3 chapter "Fixed ACPI Description Table Boot Architecture Flags" > > >> >> Perhaps we can check if every bridge on the way to the CPU has the MSI >> capability (including the root hub)? (And naturally _that_ won't >> work... on my sandybridge laptop, the host bridge doesn't have the MSI >> cap but the system most definitely supports MSI.) >> >> Adding Bjorn... perhaps you know? Some of the info has been stripped >> out by now, you can see the full lspci -vvvxxx at >> http://marc.info/?l=linux-pci&m=140443441730503&w=2 >> >> -ilia >> >> . >> > > > -- > Thanks! > Yijing >
On Thu, Jul 3, 2014 at 10:32 PM, Ilia Mirkin <imirkin at alum.mit.edu> wrote:> So let me get this straight -- you're suggesting I add a quirk for > every PCI chipset that doesn't support MSI? There are probably > hundreds of these... anything made before 1999 or so, and probably a > bunch since then too. There _has_ to be a way to do this generically. > Is the PCI spec version anywhere in the root hub? > > Perhaps we can check if every bridge on the way to the CPU has the MSI > capability (including the root hub)? (And naturally _that_ won't > work... on my sandybridge laptop, the host bridge doesn't have the MSI > cap but the system most definitely supports MSI.) > > Adding Bjorn... perhaps you know? Some of the info has been stripped > out by now, you can see the full lspci -vvvxxx at > http://marc.info/?l=linux-pci&m=140443441730503&w=2Huh, this stinks. We don't really have a good way of figuring out whether the system chipset supports MSI. The ACPI FADT "MSI Not Supported" bit (ACPI_FADT_NO_MSI) was added to the ACPI v3.0b spec in October 2006, so that won't help the systems that predate that or don't have ACPI. We have quirks for some Serverworks, ATI, and VIA chipsets that basically do the same as booting with "pci=nomsi". But as you say, it's unreasonable to add quirks for all old systems. Brian, can you open a report at http://bugzilla.kernel.org and attach a complete dmesg log, /proc/cpuinfo contents, and "lspci -vvv" output? If I understand correctly, you have a P200MMX with a 430FX chipset. I'm not a hardware guy, but sounds like that might be a 200MHz Pentium with MMX (P54CS), which does have an integrated LAPIC, according to wikipedia.>From the PCI host bridge's perspective, an incoming MSI just lookslike a normal DMA write. As long as that write reaches the CPU LAPIC, it should work fine. There's not really any specific MSI support required, except to route the incoming write to the CPU LAPIC. So it's possible that a bridge designed before MSI was added to the PCI spec might be able to support MSI. But I don't know how much value there is in MSI on such old systems. Maybe we could default to disabling MSI on BIOS dates before 1998 or something. Bjorn
Bjorn, I went ahead and sent a report to https://bugzilla.kernel.org/show_bug.cgi?id=79531. It does not appear that my system supports APIC/LAPIC, from the dmesg logs which claim that APIC is hardware disabled or not available. Perhaps the BIOS is allowed to disable it, or is required to enable it, and the BIOS does indeed predate the installed CPU. Thanks, Brian Becker On Sat, Jul 5, 2014 at 6:21 PM, Bjorn Helgaas <bhelgaas at google.com> wrote:> On Thu, Jul 3, 2014 at 10:32 PM, Ilia Mirkin <imirkin at alum.mit.edu> wrote: >> So let me get this straight -- you're suggesting I add a quirk for >> every PCI chipset that doesn't support MSI? There are probably >> hundreds of these... anything made before 1999 or so, and probably a >> bunch since then too. There _has_ to be a way to do this generically. >> Is the PCI spec version anywhere in the root hub? >> >> Perhaps we can check if every bridge on the way to the CPU has the MSI >> capability (including the root hub)? (And naturally _that_ won't >> work... on my sandybridge laptop, the host bridge doesn't have the MSI >> cap but the system most definitely supports MSI.) >> >> Adding Bjorn... perhaps you know? Some of the info has been stripped >> out by now, you can see the full lspci -vvvxxx at >> http://marc.info/?l=linux-pci&m=140443441730503&w=2 > > Huh, this stinks. We don't really have a good way of figuring out > whether the system chipset supports MSI. The ACPI FADT "MSI Not > Supported" bit (ACPI_FADT_NO_MSI) was added to the ACPI v3.0b spec in > October 2006, so that won't help the systems that predate that or > don't have ACPI. > > We have quirks for some Serverworks, ATI, and VIA chipsets that > basically do the same as booting with "pci=nomsi". But as you say, > it's unreasonable to add quirks for all old systems. > > Brian, can you open a report at http://bugzilla.kernel.org and attach > a complete dmesg log, /proc/cpuinfo contents, and "lspci -vvv" output? > > If I understand correctly, you have a P200MMX with a 430FX chipset. > I'm not a hardware guy, but sounds like that might be a 200MHz Pentium > with MMX (P54CS), which does have an integrated LAPIC, according to > wikipedia. > > From the PCI host bridge's perspective, an incoming MSI just looks > like a normal DMA write. As long as that write reaches the CPU LAPIC, > it should work fine. There's not really any specific MSI support > required, except to route the incoming write to the CPU LAPIC. So > it's possible that a bridge designed before MSI was added to the PCI > spec might be able to support MSI. > > But I don't know how much value there is in MSI on such old systems. > Maybe we could default to disabling MSI on BIOS dates before 1998 or > something. > > Bjorn