On Fri, Oct 25, 2013 at 7:43 AM, Robert Morell <rmorell at nvidia.com> wrote:> On Mon, Sep 30, 2013 at 10:44:12AM -0700, Lucas Stach wrote: >> Hi, >> >> recently we tried to enable MSI interrupts with nouveau. Unfortunately >> there have been some reports of things failing with certain cards, where >> it isn't entirely clear if this is a GPU errata or some other component >> in the PCIe chain failing. >> >> Could you perhaps investigate if there are any known Nvidia GPU erratas >> with regard to MSI interrupts, or maybe tell us the generations of cards >> that are generally safe to enable MSIs with? > > Sorry for the slow reply, Lucas. > > We enabled MSI interrupts by default relatively recently in the proprietary > driver (version 325.08, released 2013-07-01 according to our public > changelog). This was enabled across the board for all of the GPUs supported > by the latest release series, so NV50 and up. We believe it should be safe to > enable MSI on those GPUs. We never enabled MSI by default on earlier GPUs so > I can't comment there. > > I investigated our internal documentation and source code, and found a couple > of things that are probably interesting to you: > - For all pre-Fermi GPUs, we use a write through PCI config space to the "EOI" > register to rearm the MSI interrupt after servicing it, rather than a write > through the MMIO pcicfg shadow region in the GPU's PCI BAR0 window at offset > 0x88000. (This was actually originally implemented for NV4x, so you'll > probably want to do that there as well.)Hm, I recently discovered this while looking for the errata myself, but it seems you guys only do it on G80/G84/G86, and use 0x88068 at some point after that, before Fermi (G96/GT200, for example).> It seems that this was done to > avoid a hardware bug that may cause the EOI write to be dropped, blocking > all further interrupts from being serviced.That was our theory also. Thanks for clarifying.> - For GF100 and GF104, an alternate MSI EOI/rearm mechanism is used: instead > of writing to the MSI rearm register at offset 0x68 in the pcicfg window, > we write to offset 0x704.Thanks for this. Maarten discovered this for GF100, I'll modify our patch to cover GF104 also.> > > I hope this information helps. > > - Robert > _______________________________________________ > Nouveau mailing list > Nouveau at lists.freedesktop.org > http://lists.freedesktop.org/mailman/listinfo/nouveau
On Thu, Oct 24, 2013 at 04:03:12PM -0700, Ben Skeggs wrote:> On Fri, Oct 25, 2013 at 7:43 AM, Robert Morell <rmorell at nvidia.com> wrote: > > On Mon, Sep 30, 2013 at 10:44:12AM -0700, Lucas Stach wrote: > >> Hi, > >> > >> recently we tried to enable MSI interrupts with nouveau. Unfortunately > >> there have been some reports of things failing with certain cards, where > >> it isn't entirely clear if this is a GPU errata or some other component > >> in the PCIe chain failing. > >> > >> Could you perhaps investigate if there are any known Nvidia GPU erratas > >> with regard to MSI interrupts, or maybe tell us the generations of cards > >> that are generally safe to enable MSIs with? > > > > Sorry for the slow reply, Lucas. > > > > We enabled MSI interrupts by default relatively recently in the proprietary > > driver (version 325.08, released 2013-07-01 according to our public > > changelog). This was enabled across the board for all of the GPUs supported > > by the latest release series, so NV50 and up. We believe it should be safe to > > enable MSI on those GPUs. We never enabled MSI by default on earlier GPUs so > > I can't comment there. > > > > I investigated our internal documentation and source code, and found a couple > > of things that are probably interesting to you: > > - For all pre-Fermi GPUs, we use a write through PCI config space to the "EOI" > > register to rearm the MSI interrupt after servicing it, rather than a write > > through the MMIO pcicfg shadow region in the GPU's PCI BAR0 window at offset > > 0x88000. (This was actually originally implemented for NV4x, so you'll > > probably want to do that there as well.) > Hm, I recently discovered this while looking for the errata myself, > but it seems you guys only do it on G80/G84/G86, and use 0x88068 at > some point after that, before Fermi (G96/GT200, for example).Yes, you're right; I was reading this wrong. The config space path described above is used on G8x and G92. Newer GPUs (other than GF100 and GF104 as mentioned previously) use the "normal" path. - Robert
On Fri, Oct 25, 2013 at 9:10 AM, Robert Morell <rmorell at nvidia.com> wrote:> On Thu, Oct 24, 2013 at 04:03:12PM -0700, Ben Skeggs wrote: >> On Fri, Oct 25, 2013 at 7:43 AM, Robert Morell <rmorell at nvidia.com> wrote: >> > On Mon, Sep 30, 2013 at 10:44:12AM -0700, Lucas Stach wrote: >> >> Hi, >> >> >> >> recently we tried to enable MSI interrupts with nouveau. Unfortunately >> >> there have been some reports of things failing with certain cards, where >> >> it isn't entirely clear if this is a GPU errata or some other component >> >> in the PCIe chain failing. >> >> >> >> Could you perhaps investigate if there are any known Nvidia GPU erratas >> >> with regard to MSI interrupts, or maybe tell us the generations of cards >> >> that are generally safe to enable MSIs with? >> > >> > Sorry for the slow reply, Lucas. >> > >> > We enabled MSI interrupts by default relatively recently in the proprietary >> > driver (version 325.08, released 2013-07-01 according to our public >> > changelog). This was enabled across the board for all of the GPUs supported >> > by the latest release series, so NV50 and up. We believe it should be safe to >> > enable MSI on those GPUs. We never enabled MSI by default on earlier GPUs so >> > I can't comment there. >> > >> > I investigated our internal documentation and source code, and found a couple >> > of things that are probably interesting to you: >> > - For all pre-Fermi GPUs, we use a write through PCI config space to the "EOI" >> > register to rearm the MSI interrupt after servicing it, rather than a write >> > through the MMIO pcicfg shadow region in the GPU's PCI BAR0 window at offset >> > 0x88000. (This was actually originally implemented for NV4x, so you'll >> > probably want to do that there as well.) >> Hm, I recently discovered this while looking for the errata myself, >> but it seems you guys only do it on G80/G84/G86, and use 0x88068 at >> some point after that, before Fermi (G96/GT200, for example). > > Yes, you're right; I was reading this wrong. The config space path > described above is used on G8x and G92. Newer GPUs (other than GF100 > and GF104 as mentioned previously) use the "normal" path.Ah, excellent. I probably should've guessed G92 would be involved here too. Thanks again, Ben.> > - Robert