I have the same motherboard. The chipset has 2 IOAPICs: one in the NB
(00:00.1), and another in the SB (00:14.0).
As currently configured by the BIOS, the NB IOAPIC is _entirely_
disabled, with only the SB IOAPIC enabled and configured. From
48693.pdf, it would seem that the NB is improperly configured, as
6.4.1 recommends that
"The IOAPIC should be enabled by the system BIOS and the southbridge
routing feature should be switched on. This will allow the interrupts
to be forwarded to the PIC in the southbridge. This way, if the system
boots in DOS, interrupts will still work. Once the system boots an
ACPI OS (where IOAPIC is assumed to be supported), the system BIOS
will disable the southbridge routing feature so that the masked
interrupts are masked out."
It would seem that no NB-side interrupts are going through an IOAPIC.
If so, I do not know what the impact is - somehow NB-side interrupts
are getting to the CPUs. However, given that very few devices have
MSI interrupts enabled by the BIOS, it may be a little more of an
issue.
It seems there are two fixes available, which would be nice to see
from the BIOS vendor:
1) "the bandaid" - just remove the second IVHD IOAPIC entry. Then,
Xen will do interrupt remapping at least for the SB IOAPIC. I
modified the Xen source to ignore this second entry, and get the dmesg
shown below. My system seems to run fine after the change.
2) Properly configure the NB IOMMU, even if just as set out in 6.4.1.
http://review.coreboot.org/gitweb?p=coreboot.git;a=blob;f=src/vendorcode/amd/cimx/rd890/nbIoApic.c
which I understand to be from CIMX, has the necessary code.
I suppose if it came down to it, you could add code to Xen to bang in
the necessary configuration via the 10 indirect registers for the NB
IOMMU to match up with Table 6-6, which is what the above source file
seems to be doing (although in a more generalized and therefore
complex way). However, I do not know what, if any, possible
consequences there would be if this does not match what is being
done/reported by ACPI.
- Eric
From "xl dmesg":
. . .
(XEN) ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 8, version 33, address 0xfec00000, GSI 0-23
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode: Flat. Using 1 I/O APICs
. . .
(XEN) AMD-Vi: IVHD Special: 0000:00:14.0 variety 0x1 handle 0x8
(XEN) AMD-Vi: IVHD Device Entry: type 0x48 id 0 flags 0xd7
(XEN) AMD-Vi: IVHD Special: 0000:00:14.0 variety 0x2 handle 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x48 id 0 flags 0
(XEN) AMD-Vi: IVHD Special: 0000:00:00.1 variety 0x1 handle 0x8
(XEN) ignored - IVHD Error: Conflicting IO-APIC 0x8 entries
(XEN) AMD-Vi: IOMMU 0 Enabled.
(XEN) AMD-Vi: Enabling per-device vector maps
(XEN) I/O virtualisation enabled
(XEN) - Dom0 mode: Relaxed
(XEN) Interrupt remapping enabled
. . .
(XEN) ENABLING IO-APIC IRQs
(XEN) -> Using new ACK method
(XEN) init IO_APIC IRQs
(XEN) IO-APIC (apicid-pin) 8-0, 8-16, 8-17, 8-18, 8-19, 8-20, 8-21,
8-22, 8-23 not connected.
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) number of MP IRQ sources: 15.
(XEN) number of IO-APIC #8 registers: 24.
(XEN) testing the IO APIC.......................
(XEN) IO APIC #8......
(XEN) .... register #00: 00000000
(XEN) ....... : physical APIC id: 00
(XEN) ....... : Delivery Type: 0
(XEN) ....... : LTS : 0
(XEN) .... register #01: 00178021
(XEN) ....... : max redirection entries: 0017
(XEN) ....... : PRQ implemented: 1
(XEN) ....... : IO APIC version: 0021
(XEN) .... register #02: 00000000
(XEN) ....... : arbitration: 00
(XEN) .... register #03: 01000000
(XEN) ....... : Boot DT : 0
(XEN) .... IRQ redirection table:
(XEN) NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:
(XEN) 00 000 00 1 0 0 0 0 0 0 00
(XEN) 01 001 01 0 0 0 0 0 1 1 30
(XEN) 02 001 01 0 0 0 0 0 1 1 F0
(XEN) 03 001 01 0 0 0 0 0 1 1 38
(XEN) 04 001 01 0 0 0 0 0 1 1 40
(XEN) 05 001 01 0 0 0 0 0 1 1 48
(XEN) 06 001 01 0 0 0 0 0 1 1 50
(XEN) 07 001 01 0 0 0 0 0 1 1 58
(XEN) 08 001 01 0 0 0 0 0 1 1 60
(XEN) 09 001 01 1 1 0 1 0 1 1 68
(XEN) 0a 001 01 0 0 0 0 0 1 1 70
(XEN) 0b 001 01 0 0 0 0 0 1 1 78
(XEN) 0c 001 01 0 0 0 0 0 1 1 88
(XEN) 0d 001 01 0 0 0 0 0 1 1 90
(XEN) 0e 001 01 0 0 0 0 0 1 1 98
(XEN) 0f 001 01 0 0 0 0 0 1 1 A0
(XEN) 10 000 00 1 0 0 0 0 0 0 00
(XEN) 11 000 00 1 0 0 0 0 0 0 00
(XEN) 12 000 00 1 0 0 0 0 0 0 00
(XEN) 13 000 00 1 0 0 0 0 0 0 00
(XEN) 14 000 00 1 0 0 0 0 0 0 00
(XEN) 15 000 00 1 0 0 0 0 0 0 00
(XEN) 16 000 00 1 0 0 0 0 0 0 00
(XEN) 17 000 00 1 0 0 0 0 0 0 00
(XEN) Using vector-based indexing
. . .
(XEN) *** LOADING DOMAIN 0 ***
. . .
(XEN) PCI add device 0000:02:00.0
(XEN) PCI add device 0000:04:00.0
(XEN) PCI add device 0000:04:00.1
(XEN) IOAPIC[0]: Set PCI routing entry (8-8 -> 0x60 -> IRQ 8 Mode:0
Active:0)
(XEN) IOAPIC[0]: Set PCI routing entry (8-13 -> 0x90 -> IRQ 13 Mode:0
Active:0)
(XEN) IOAPIC[0]: Set PCI routing entry (8-16 -> 0xb0 -> IRQ 16 Mode:1
Active:1)
(XEN) IOAPIC[0]: Set PCI routing entry (8-17 -> 0xb8 -> IRQ 17 Mode:1
Active:1)
(XEN) IOAPIC[0]: Set PCI routing entry (8-18 -> 0xc0 -> IRQ 18 Mode:1
Active:1)
(XEN) IOAPIC[0]: Set PCI routing entry (8-19 -> 0xc8 -> IRQ 19 Mode:1
Active:1)
> > On 03/12/2013 03:04 PM, Hans Mueller wrote:
> > Hello,
> >
> > since applying the patches related to XSA-36 Xen recognizes a broken
IVRS ACPI
> > table and disables I/O virtualisation.
> >
> > I contacted the manufacturer of the mainboard/BIOS and they want to
help me by
> > providing a patched BIOS - so far so good.
> >
> > However, they need details about what to fix, which I don''t
know either.
> >
> > Could you pls. give me some hints which I can forward to the
manufacturer
> > support?
>
> They should look at AMD IOMMU spec. For example,
> support.amd.com/us/Processor_TechDocs/48882.pdf. Tables 77 and 79.
>
> More specifically, the problem is these two entries:
>
> (XEN) AMD-Vi: IVHD Device Entry: type 0x48 id 0x0 flags 0xd7
> (XEN) AMD-Vi: IVHD Special: 0000:00:14.0 variety 0x1 handle 0x8
> ..
> (XEN) AMD-Vi: IVHD Device Entry: type 0x48 id 0x0 flags 0x0
> (XEN) AMD-Vi: IVHD Special: 0000:00:00.1 variety 0x1 handle 0x8
>
> which tell IOMMU driver that there are two IOAPICs, both with APICID 8.
>
> I believe the second one is wrong.
>
> -boris
> >
> > Thanks a lot & best regards
> > Hans
> >
> >
> >
> > PS: Hardware is a Gigabyte GA-970A-UD3(rev. 1.0), BIOS F7 (tested F8a,
too).
> >
> > PPS: I know about ''no-amd-iommu-perdev-intremap'' -
this does not really help
> > as e.g. heavy i/o on a usb device in one domain causes other domains
to
> > disable the related irq etc. ...
