Hello, since applying the patches related to XSA-36 Xen recognizes a broken IVRS ACPI table and disables I/O virtualisation. I contacted the manufacturer of the mainboard/BIOS and they want to help me by providing a patched BIOS - so far so good. However, they need details about what to fix, which I don''t know either. Could you pls. give me some hints which I can forward to the manufacturer support? Thanks a lot & best regards Hans PS: Hardware is a Gigabyte GA-970A-UD3(rev. 1.0), BIOS F7 (tested F8a, too). PPS: I know about ''no-amd-iommu-perdev-intremap'' - this does not really help as e.g. heavy i/o on a usb device in one domain causes other domains to disable the related irq etc. ... From the support mail: "Our hardware team replied: Please inform us what settings or specific detail he needs to modify on IVRS ACPI Table. They will try to patch it in a special BIOS and ask customer to check. Or maybe you can ask Xen for a proper IVRS ACPI Table form/example and send to us. We can study on this and provide a special BIOS." From ''xl dmesg'': (XEN) AMD-Vi: Found MSI capability block at 0x54 (XEN) AMD-Vi: ACPI Table: (XEN) AMD-Vi: Signature IVRS (XEN) AMD-Vi: Length 0xd0 (XEN) AMD-Vi: Revision 0x1 (XEN) AMD-Vi: CheckSum 0x9b (XEN) AMD-Vi: OEM_Id AMD (XEN) AMD-Vi: OEM_Table_Id RD890S (XEN) AMD-Vi: OEM_Revision 0x202031 (XEN) AMD-Vi: Creator_Id AMD (XEN) AMD-Vi: Creator_Revision 0x0 (XEN) AMD-Vi: IVRS Block: type 0x10 flags 0x3e len 0xa0 id 0x2 (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x0 flags 0x0 (XEN) AMD-Vi: Dev_Id Range: 0x0 -> 0x2 (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x10 flags 0x0 (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x100 flags 0x0 (XEN) AMD-Vi: Dev_Id Range: 0x100 -> 0x101 (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x38 flags 0x0 (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x200 flags 0x0 (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x48 flags 0x0 (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x300 flags 0x0 (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x88 flags 0x0 (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x90 flags 0x0 (XEN) AMD-Vi: Dev_Id Range: 0x90 -> 0x92 (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x98 flags 0x0 (XEN) AMD-Vi: Dev_Id Range: 0x98 -> 0x9a (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa0 flags 0xd7 (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa3 flags 0x0 (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa4 flags 0x0 (XEN) AMD-Vi: IVHD Device Entry: type 0x0 id 0x0 flags 0x0 (XEN) AMD-Vi: IVHD Device Entry: type 0x43 id 0x400 flags 0x0 (XEN) AMD-Vi: Dev_Id Range: 0x400 -> 0x4ff alias 0xa4 (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa5 flags 0x0 (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa8 flags 0x0 (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x500 flags 0x0 (XEN) AMD-Vi: Dev_Id Range: 0x500 -> 0x5ff (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0xb0 flags 0x0 (XEN) AMD-Vi: Dev_Id Range: 0xb0 -> 0xb2 (XEN) AMD-Vi: IVHD Device Entry: type 0x0 id 0x0 flags 0x0 (XEN) AMD-Vi: IVHD Device Entry: type 0x48 id 0x0 flags 0xd7 (XEN) AMD-Vi: IVHD Special: 0000:00:14.0 variety 0x1 handle 0x8 (XEN) AMD-Vi: IVHD Device Entry: type 0x48 id 0x0 flags 0xd7 (XEN) AMD-Vi: IVHD Special: 0000:00:14.0 variety 0x2 handle 0x0 (XEN) AMD-Vi: IVHD Device Entry: type 0x48 id 0x0 flags 0x0 (XEN) AMD-Vi: IVHD Special: 0000:00:00.1 variety 0x1 handle 0x8 (XEN) IVHD Error: Conflicting IO-APIC 0x8 entries (XEN) AMD-Vi: Error initialization (XEN) I/O virtualisation disabled _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On 03/12/2013 03:04 PM, Hans Mueller wrote:> Hello, > > since applying the patches related to XSA-36 Xen recognizes a broken IVRS ACPI > table and disables I/O virtualisation. > > I contacted the manufacturer of the mainboard/BIOS and they want to help me by > providing a patched BIOS - so far so good. > > However, they need details about what to fix, which I don''t know either. > > Could you pls. give me some hints which I can forward to the manufacturer > support?They should look at AMD IOMMU spec. For example, support.amd.com/us/Processor_TechDocs/48882.pdf. Tables 77 and 79. More specifically, the problem is these two entries: (XEN) AMD-Vi: IVHD Device Entry: type 0x48 id 0x0 flags 0xd7 (XEN) AMD-Vi: IVHD Special: 0000:00:14.0 variety 0x1 handle 0x8 .. (XEN) AMD-Vi: IVHD Device Entry: type 0x48 id 0x0 flags 0x0 (XEN) AMD-Vi: IVHD Special: 0000:00:00.1 variety 0x1 handle 0x8 which tell IOMMU driver that there are two IOAPICs, both with APICID 8. I believe the second one is wrong. -boris> > Thanks a lot & best regards > Hans > > > > PS: Hardware is a Gigabyte GA-970A-UD3(rev. 1.0), BIOS F7 (tested F8a, too). > > PPS: I know about ''no-amd-iommu-perdev-intremap'' - this does not really help > as e.g. heavy i/o on a usb device in one domain causes other domains to > disable the related irq etc. ... > > > > From the support mail: > "Our hardware team replied: > > Please inform us what settings or specific detail he needs to modify on IVRS > ACPI Table. > They will try to patch it in a special BIOS and ask customer to check. > Or maybe you can ask Xen for a proper IVRS ACPI Table form/example and send to > us. > We can study on this and provide a special BIOS." > > > > From ''xl dmesg'': > (XEN) AMD-Vi: Found MSI capability block at 0x54 > (XEN) AMD-Vi: ACPI Table: > (XEN) AMD-Vi: Signature IVRS > (XEN) AMD-Vi: Length 0xd0 > (XEN) AMD-Vi: Revision 0x1 > (XEN) AMD-Vi: CheckSum 0x9b > (XEN) AMD-Vi: OEM_Id AMD > (XEN) AMD-Vi: OEM_Table_Id RD890S > (XEN) AMD-Vi: OEM_Revision 0x202031 > (XEN) AMD-Vi: Creator_Id AMD > (XEN) AMD-Vi: Creator_Revision 0x0 > (XEN) AMD-Vi: IVRS Block: type 0x10 flags 0x3e len 0xa0 id 0x2 > (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x0 flags 0x0 > (XEN) AMD-Vi: Dev_Id Range: 0x0 -> 0x2 > (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x10 flags 0x0 > (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x100 flags 0x0 > (XEN) AMD-Vi: Dev_Id Range: 0x100 -> 0x101 > (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x38 flags 0x0 > (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x200 flags 0x0 > (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x48 flags 0x0 > (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x300 flags 0x0 > (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0x88 flags 0x0 > (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x90 flags 0x0 > (XEN) AMD-Vi: Dev_Id Range: 0x90 -> 0x92 > (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x98 flags 0x0 > (XEN) AMD-Vi: Dev_Id Range: 0x98 -> 0x9a > (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa0 flags 0xd7 > (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa3 flags 0x0 > (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa4 flags 0x0 > (XEN) AMD-Vi: IVHD Device Entry: type 0x0 id 0x0 flags 0x0 > (XEN) AMD-Vi: IVHD Device Entry: type 0x43 id 0x400 flags 0x0 > (XEN) AMD-Vi: Dev_Id Range: 0x400 -> 0x4ff alias 0xa4 > (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa5 flags 0x0 > (XEN) AMD-Vi: IVHD Device Entry: type 0x2 id 0xa8 flags 0x0 > (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0x500 flags 0x0 > (XEN) AMD-Vi: Dev_Id Range: 0x500 -> 0x5ff > (XEN) AMD-Vi: IVHD Device Entry: type 0x3 id 0xb0 flags 0x0 > (XEN) AMD-Vi: Dev_Id Range: 0xb0 -> 0xb2 > (XEN) AMD-Vi: IVHD Device Entry: type 0x0 id 0x0 flags 0x0 > (XEN) AMD-Vi: IVHD Device Entry: type 0x48 id 0x0 flags 0xd7 > (XEN) AMD-Vi: IVHD Special: 0000:00:14.0 variety 0x1 handle 0x8 > (XEN) AMD-Vi: IVHD Device Entry: type 0x48 id 0x0 flags 0xd7 > (XEN) AMD-Vi: IVHD Special: 0000:00:14.0 variety 0x2 handle 0x0 > (XEN) AMD-Vi: IVHD Device Entry: type 0x48 id 0x0 flags 0x0 > (XEN) AMD-Vi: IVHD Special: 0000:00:00.1 variety 0x1 handle 0x8 > (XEN) IVHD Error: Conflicting IO-APIC 0x8 entries > (XEN) AMD-Vi: Error initialization > (XEN) I/O virtualisation disabled > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Hello, I got a patched BIOS (version F8c) from Gigabyte which removes the 2nd IOAPIC entry (for device 0000:00:00.1) from the IVRS table. This causes Xen to enable per-device vector maps: (XEN) AMD-Vi: IOMMU 0 Enabled. (XEN) AMD-Vi: Enabling per-device vector maps (XEN) I/O virtualisation enabled (XEN) - Dom0 mode: Relaxed (XEN) Interrupt remapping enabled However the problem seems not really to be fixed: Interrupts generated within one domain can still harm other domains which at least causes the kernel within these other domains to disable interrupts. Before going to investigate/debug this problem, I want to know if one IOAPIC is sufficient as the AMD 970 chipset seems to have one IOAPIC related to the northbridge and one for the southbridge. The one currently enabled is the southbridge one (0000:00:14.0). Does it also support devices connected to the northbridge or needs the northbridge one to be enabled, too? Are there other limitations/problems to expect regarding the disabled northbridge IOAPIC? Thanks & best regards Hans _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
>>> On 23.04.13 at 19:11, Hans Mueller <mcbeagle@gmx.de> wrote: > However the problem seems not really to be fixed: > Interrupts generated within one domain can still harm other domains which at > least causes the kernel within these other domains to disable interrupts."The problem" being which problem? Even after looking back through the list archives, I haven''t been able to spot a corresponding problem description. And if you resume a thread after several weeks without sufficiently quoting the original, it would be rather desirable for you to at least briefly summarize your original problem. The same physical IRQ being shared by multiple devices is entirely independent of XSA-36, and if your Dom0/DomU kernels can''t cope with that they are what need fixing. Jan
On Wednesday, 24. April 2013 07:47:25 Jan Beulich wrote:> >>> On 23.04.13 at 19:11, Hans Mueller <mcbeagle@gmx.de> wrote: > > However the problem seems not really to be fixed: > > Interrupts generated within one domain can still harm other domains which > > at least causes the kernel within these other domains to disable > > interrupts. > "The problem" being which problem? Even after looking back through > the list archives, I haven''t been able to spot a corresponding problem > description. And if you resume a thread after several weeks without > sufficiently quoting the original, it would be rather desirable for you > to at least briefly summarize your original problem.Perhaps it was unclear - better than ''the'' problem might be ''a'' problem - as briefly described regarding interrupts becoming disabled across domain borders. However, I don''t want to address that problem in this thread in detail, I just worry that the fixes applied to the BIOS are insufficient and want to check this before starting a perhaps unnecessary discussion about the ''interrupt problem''. Since XSA-36 Xen disabled the I/O virtualisation because there was a conflicting IOAPIC entry within the IVRS table (probably for the disabled northbridge IOAPIC & using the handle/id for the enabled southbridge IOAPIC). The manufacturer just removed this entry from the IVRS table which causes Xen to no longer complain about conflicting IOAPIC entries and enabling per-device vector maps. So currently only the southbridge IOAPIC is enabled. The question is whether the missing/disabled northbridge IOAPIC is a valid setup or might/will it raise any problems.> The same physical IRQ being shared by multiple devices is entirely > independent of XSA-36, and if your Dom0/DomU kernels can''t cope > with that they are what need fixing.Is it obvious that such problems are not caused by the missing northbridge IOAPIC? I would then provide related details in a new thread. Best Regards Hans
>>> On 24.04.13 at 22:03, Hans Mueller <mcbeagle@gmx.de> wrote: > The question is whether the missing/disabled northbridge IOAPIC is a valid > setup or might/will it raise any problems.The setup is certainly valid, but of course not optimal (just because interrupt sharing is generally not optimal).>> The same physical IRQ being shared by multiple devices is entirely >> independent of XSA-36, and if your Dom0/DomU kernels can''t cope >> with that they are what need fixing. > Is it obvious that such problems are not caused by the missing northbridge > IOAPIC? I would then provide related details in a new thread.First of all, I would assume that even before that change the NB IO-APIC wasn''t actually used either. But if it was, then suppressing its use of course would increase IRQ sharing and hence more readily expose eventual bugs in the kernel(s) you use. Jan
Apparently Analagous Threads
- IOMMU/AMD-Vi not working after XSA-36 with 970A-UD3
- AMD IOMMU disabled - No Perdev Intremap
- Re: XSA-36 / howto fix broken IVRS ACPI table
- [PATCH 1/1 V3] x86/AMD-Vi: Add additional check for invalid special->handle
- [PATCH V2] x86/AMD-Vi: Add additional check for invalid special->handle