Wei, Quick question: Am I reading the code correctly, that even with per-device interrupt remap tables, that GSIs are accounted to the intremap table of the corresponding IOAPIC, presumably because the IOMMU sees interrupts generated as GSIs as coming from the IOAPIC? In that case, then we need all devices sharing the same IOAPIC must not have any vector collisions. Is that correct? -George _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 06/09/11 16:47, George Dunlap wrote:> Wei, > > Quick question: Am I reading the code correctly, that even with > per-device interrupt remap tables, that GSIs are accounted to the > intremap table of the corresponding IOAPIC, presumably because the > IOMMU sees interrupts generated as GSIs as coming from the IOAPIC? In > that case, then we need all devices sharing the same IOAPIC must not > have any vector collisions. Is that correct?Based on the ICH10 IO-APIC documentation with respect to auto EOIs, we cant have any two IRQs across any IO-APICs sharing a vector, irrespective of IOMMU or not. (Because the EOI''ing an IO-APIC entry only takes account of vector and not destination) If we were to disable the auto EOI broadcast and do manual EOI''ing (only available on newer versions of the local apic) then we could reduce that restriction to "no two IRQs in the same IO-APIC may share a vector". ~Andrew> -George > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel-- Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer T: +44 (0)1223 225 900, http://www.citrix.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, Sep 6, 2011 at 4:54 PM, Andrew Cooper <andrew.cooper3@citrix.com> wrote:> On 06/09/11 16:47, George Dunlap wrote: >> Wei, >> >> Quick question: Am I reading the code correctly, that even with >> per-device interrupt remap tables, that GSIs are accounted to the >> intremap table of the corresponding IOAPIC, presumably because the >> IOMMU sees interrupts generated as GSIs as coming from the IOAPIC? In >> that case, then we need all devices sharing the same IOAPIC must not >> have any vector collisions. Is that correct? > > Based on the ICH10 IO-APIC documentation with respect to auto EOIs, we > cant have any two IRQs across any IO-APICs sharing a vector, > irrespective of IOMMU or not. (Because the EOI''ing an IO-APIC entry > only takes account of vector and not destination) > > If we were to disable the auto EOI broadcast and do manual EOI''ing (only > available on newer versions of the local apic) then we could reduce that > restriction to "no two IRQs in the same IO-APIC may share a vector".Hmm, so it sounds like enforcing non-sharing of vectors within a single IOAPIC is something we probably want to do even when we''re not using an AMD IOMMU? -George _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, 2011-09-06 at 17:03 +0100, Andrew Cooper wrote:> On 06/09/11 16:57, George Dunlap wrote: > > On Tue, Sep 6, 2011 at 4:54 PM, Andrew Cooper <andrew.cooper3@citrix.com> wrote: > >> On 06/09/11 16:47, George Dunlap wrote: > >>> Wei, > >>> > >>> Quick question: Am I reading the code correctly, that even with > >>> per-device interrupt remap tables, that GSIs are accounted to the > >>> intremap table of the corresponding IOAPIC, presumably because the > >>> IOMMU sees interrupts generated as GSIs as coming from the IOAPIC? In > >>> that case, then we need all devices sharing the same IOAPIC must not > >>> have any vector collisions. Is that correct? > >> Based on the ICH10 IO-APIC documentation with respect to auto EOIs, we > >> cant have any two IRQs across any IO-APICs sharing a vector, > >> irrespective of IOMMU or not. (Because the EOI''ing an IO-APIC entry > >> only takes account of vector and not destination) > >> > >> If we were to disable the auto EOI broadcast and do manual EOI''ing (only > >> available on newer versions of the local apic) then we could reduce that > >> restriction to "no two IRQs in the same IO-APIC may share a vector". > > Hmm, so it sounds like enforcing non-sharing of vectors within a > > single IOAPIC is something we probably want to do even when we''re not > > using an AMD IOMMU? > > > > -George > > Currently there is no code to disable auto EOI and do manual EOI > instead. As a result, we should enforce non-sharing of vectors across > all IO-APICs. It was on my irq cleanup list but seems to be a specific > problem for you at the moment.My problem doesn''t have anything to do with EOIs, but with the AMD IOMMU interrupt remapping table, and a device driver which simply dies if it misses any interrupts. If we can kill two birds with one stone, all the better. -George _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 06/09/11 16:57, George Dunlap wrote:> On Tue, Sep 6, 2011 at 4:54 PM, Andrew Cooper <andrew.cooper3@citrix.com> wrote: >> On 06/09/11 16:47, George Dunlap wrote: >>> Wei, >>> >>> Quick question: Am I reading the code correctly, that even with >>> per-device interrupt remap tables, that GSIs are accounted to the >>> intremap table of the corresponding IOAPIC, presumably because the >>> IOMMU sees interrupts generated as GSIs as coming from the IOAPIC? In >>> that case, then we need all devices sharing the same IOAPIC must not >>> have any vector collisions. Is that correct? >> Based on the ICH10 IO-APIC documentation with respect to auto EOIs, we >> cant have any two IRQs across any IO-APICs sharing a vector, >> irrespective of IOMMU or not. (Because the EOI''ing an IO-APIC entry >> only takes account of vector and not destination) >> >> If we were to disable the auto EOI broadcast and do manual EOI''ing (only >> available on newer versions of the local apic) then we could reduce that >> restriction to "no two IRQs in the same IO-APIC may share a vector". > Hmm, so it sounds like enforcing non-sharing of vectors within a > single IOAPIC is something we probably want to do even when we''re not > using an AMD IOMMU? > > -GeorgeCurrently there is no code to disable auto EOI and do manual EOI instead. As a result, we should enforce non-sharing of vectors across all IO-APICs. It was on my irq cleanup list but seems to be a specific problem for you at the moment. -- Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer T: +44 (0)1223 225 900, http://www.citrix.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, 2011-09-06 at 17:06 +0100, Tim Deegan wrote:> At 16:54 +0100 on 06 Sep (1315328056), Andrew Cooper wrote: > > On 06/09/11 16:47, George Dunlap wrote: > > > Wei, > > > > > > Quick question: Am I reading the code correctly, that even with > > > per-device interrupt remap tables, that GSIs are accounted to the > > > intremap table of the corresponding IOAPIC, presumably because the > > > IOMMU sees interrupts generated as GSIs as coming from the IOAPIC? In > > > that case, then we need all devices sharing the same IOAPIC must not > > > have any vector collisions. Is that correct? > > > > Based on the ICH10 IO-APIC documentation with respect to auto EOIs, we > > cant have any two IRQs across any IO-APICs sharing a vector, > > irrespective of IOMMU or not. (Because the EOI''ing an IO-APIC entry > > only takes account of vector and not destination) > > If this is the case, is there any point in having per-CPU IDTs? > Or per-device remapping tables?Yes, for devices that use MSIs, at least. -George _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
At 16:54 +0100 on 06 Sep (1315328056), Andrew Cooper wrote:> On 06/09/11 16:47, George Dunlap wrote: > > Wei, > > > > Quick question: Am I reading the code correctly, that even with > > per-device interrupt remap tables, that GSIs are accounted to the > > intremap table of the corresponding IOAPIC, presumably because the > > IOMMU sees interrupts generated as GSIs as coming from the IOAPIC? In > > that case, then we need all devices sharing the same IOAPIC must not > > have any vector collisions. Is that correct? > > Based on the ICH10 IO-APIC documentation with respect to auto EOIs, we > cant have any two IRQs across any IO-APICs sharing a vector, > irrespective of IOMMU or not. (Because the EOI''ing an IO-APIC entry > only takes account of vector and not destination)If this is the case, is there any point in having per-CPU IDTs? Or per-device remapping tables? Tim. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 06/09/2011 17:06, "Tim Deegan" <tim@xen.org> wrote:> At 16:54 +0100 on 06 Sep (1315328056), Andrew Cooper wrote: >> On 06/09/11 16:47, George Dunlap wrote: >>> Wei, >>> >>> Quick question: Am I reading the code correctly, that even with >>> per-device interrupt remap tables, that GSIs are accounted to the >>> intremap table of the corresponding IOAPIC, presumably because the >>> IOMMU sees interrupts generated as GSIs as coming from the IOAPIC? In >>> that case, then we need all devices sharing the same IOAPIC must not >>> have any vector collisions. Is that correct? >> >> Based on the ICH10 IO-APIC documentation with respect to auto EOIs, we >> cant have any two IRQs across any IO-APICs sharing a vector, >> irrespective of IOMMU or not. (Because the EOI''ing an IO-APIC entry >> only takes account of vector and not destination) > > If this is the case, is there any point in having per-CPU IDTs? > Or per-device remapping tables?It still makes sense for MSIs, which are the most common interrupt type these days. -- Keir> Tim. > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 06/09/11 17:06, Tim Deegan wrote:> At 16:54 +0100 on 06 Sep (1315328056), Andrew Cooper wrote: >> On 06/09/11 16:47, George Dunlap wrote: >>> Wei, >>> >>> Quick question: Am I reading the code correctly, that even with >>> per-device interrupt remap tables, that GSIs are accounted to the >>> intremap table of the corresponding IOAPIC, presumably because the >>> IOMMU sees interrupts generated as GSIs as coming from the IOAPIC? In >>> that case, then we need all devices sharing the same IOAPIC must not >>> have any vector collisions. Is that correct? >> Based on the ICH10 IO-APIC documentation with respect to auto EOIs, we >> cant have any two IRQs across any IO-APICs sharing a vector, >> irrespective of IOMMU or not. (Because the EOI''ing an IO-APIC entry >> only takes account of vector and not destination) > If this is the case, is there any point in having per-CPU IDTs? > Or per-device remapping tables? > > Tim.My understanding is: GSIs should always have unique vectors (due to the auto EOI broadcast) AMD IOMMU with per-device remapping tables means that devices cant ever share interrupts when migrating. AMD IOMMU with a global remapping table negates the benefit of per-CPU IDTs Therefore per-CPU IDTs are useful, but there is juggling required when choosing a new vector on migrate (With the possibility of having free vectors, but unable to assign any because of other restrictions) -- Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer T: +44 (0)1223 225 900, http://www.citrix.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tuesday 06 September 2011 17:47:59 George Dunlap wrote:> Wei, > > Quick question: Am I reading the code correctly, that even with > per-device interrupt remap tables, that GSIs are accounted to the > intremap table of the corresponding IOAPIC, presumably because the > IOMMU sees interrupts generated as GSIs as coming from the IOAPIC? In > that case, then we need all devices sharing the same IOAPIC must not > have any vector collisions. Is that correct? > > -GeorgeThat is true. All legacy devices have to send interrupts to IOMMU via IOAPIC. So even we use per-device table. All devices attached to the same IOAPIC will use the same interrupt table. There should be no vector collisions. Per-device table only makes sense for MSI devices or systems with multiple IOAPICs. Thanks Wei _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel