Huang2, Wei
2010-Jul-13 23:13 UTC
[Xen-devel] RE: MSI 890FXA-GD70: AMD IOMMU, VmError: Failed to assign device to IOMMU, (XEN) AMD_IOV: IO_PAGE_FALT: domain:10, device id:0x900, fault address:0x0
[cc Wei Wang here]
In the function reassign_device() in pci_amd_iommu.c, it seems
pci_get_pdev_by_domain() fail to return a correct pci_dev. Here is the decode of
IVRS table. It looks like 03:xx.x is behind a bridge 00:14.4 and uses this value
as an alias ID. That might be a cause. From IVRS, I don''t see anything
wrong with 00:14.2 though. Maybe you can add more debug code to
reassign_device().
-Wei
02 10 00 00 DEV=00:2.0
03 00 0f 00 04 01 0f 00 DEV=[0f:1.0, 0f:1.0]
02 18 00 00 DEV=00:2.0
02 00 0e 00 DEV=0e:0.0
02 20 00 00 DEV=00:4.0
02 00 0d 00 DEV=0d:0.0
02 28 00 00 DEV=00:5.0
02 00 0c 00 DEV=0c:0.0
02 30 00 00 DEV=00:6.0
02 00 0b 00 DEV=0b:0.0
02 48 00 00 DEV=00:9.0
02 00 0a 00 DEV=0a:0.0
02 50 00 00 DEV=00:a.0
02 00 08 00 00 00 00 00 DEV=08:0.0
43 08 09 00 00 00 09 00 04 ff 09 00 DEV=[09:1.0, 09:1f.7]
02 58 00 00 DEV=00:b.0
02 00 04 00 DEV=04:0.0
02 08 05 00 DEV=05:1.0
02 00 07 00 DEV=07:0.0
02 10 05 00 DEV=05:2.0
03 00 06 00 04 00 06 00 DEV=06:0.0
02 88 00 00 DEV=00:11.0
03 90 00 00 04 92 00 00 DEV=[00:12.0, 00:12.2]
03 98 00 00 04 9a 00 00 DEV=[00:13.0, 00:13.2]
02 a0 00 d7 DEV=00:14.0
02 a1 00 00 DEV=00:14.1
02 a2 00 00 DEV=00:14.2
02 a3 00 00 DEV=00:14.3
02 a4 00 00 DEV=00:14.4
43 00 03 00 00 a4 00 00 04 ff 03 00 DEV=[03:0.0, 03:1f.7]
02 a5 00 00 DEV=00:14.5
02 a8 00 00 DEV=00:16.0
02 a9 00 00 DEV=00:16.1
02 00 01 00 DEV=01:0.0
03 b0 00 00 04 b2 00 00 00 00 00 00 DEV=[00:16.0, 00:16.2]
48 00 00 d7 00 a0 00 02
48 00 00 00 07 01 00 01
-----Original Message-----
From: Sander Eikelenboom [mailto:linux@eikelenboom.it]
Sent: Tuesday, July 13, 2010 5:07 PM
To: xen-devel@lists.xensource.com
Cc: Przywara, Andre; Joerg Roedel; Huang2, Wei; Konrad Rzeszutek Wilk; Jeremy
Fitzhardinge
Subject: MSI 890FXA-GD70: AMD IOMMU, VmError: Failed to assign device to IOMMU,
(XEN) AMD_IOV: IO_PAGE_FALT: domain:10, device id:0x900, fault address:0x0
Hi All,
I have bought a MSI 890FXA-GD70, and have been able to enable the IOMMU with a
beta-bios from MSI.
I succeed in passing through some pci-e devices to domU''s (PV), but
some other devices fail with both PV and HVM domains.
The message is allways the same "Failed to assign device to IOMMU",
with some added printk''s the return code from
"iommu.c/assign_device" seems to be ffffffed instead of the desired 0.
Devices 00:14.2 and 03:06.* cause the domain(PV and HVM) to fail on creation
with: Failed to assign device to IOMMU (BDF) Devices 09:01.* and 07:00.0
passthrough fine ( visible and working in pv domU).
Since it''s a beta bios, could it be the IVRS table supplied by the bios
is not complete and therefore the iommu functions only for some devices ?
With amd_iommu_debug it sometimes also prints an "IO_PAGE_FALT":
(XEN) [VT-D]iommu.c:144: SEIK: assign_device: domain 14 (0:14.2)!
(XEN) [VT-D]iommu.c:162: SEIK: assign_device rc: ffffffed !
(XEN) domctl.c:853:d0 XEN_DOMCTL_assign_device: assign device (0:14.2)
failed
(XEN) AMD_IOV: IO_PAGE_FALT: domain:10, device id:0x900, fault
address:0x0
--
Sander
Xen version: xen-4.0-testing-pre4 (changeset: 21263:57859775f88f)
Dom0 kernel: 2.6.32.16 pvops (commit 006bfe8f594370f36f09d8e8ab13692d34e88533)
Attached:
- xm info
- xm dmesg
- dmesg
- lspci -vvvknn
- Dump of ACPI IVRS table
- Dump of ACPI DSDT table
serveerstertje:~/iommu-report# xm pci-list-assignable-devices 0000:0e:00.0
0000:09:01.0
0000:09:01.1
0000:09:01.2
0000:03:06.0
0000:03:06.1
0000:03:06.2
0000:00:14.2
grub menu.lst:
title xen-4.0.1-rc4-pre.gz / Debian GNU/Linux, kernel
2.6.32.16+xen-2.6.32.x-20100713-a
root (hd0,0)
kernel /xen-4.0.1-rc4-pre.gz dom0_mem=768M loglvl=all loglvl_guest=all
com1=115200,8n1 console=com1,vga iommu=1,pv,verbose,amd_iommu_debug
module /vmlinuz-2.6.32.16+xen-2.6.32.x-20100713-a
root=/dev/mapper/serveerstertje-root ro earlyprintk=xen max_loop=255
loop_max_part=63
xen-pciback.hide=(00:14.2)(03:06.0)(03:06.1)(03:06.2)(07:00.0)(09:01.0)(09:01.1)(09:01.2)(0e:00.0)
pci=resource_alignment=00:14.2;03:06.0;03:06.1;03:06.2;09:01.0;09:01.1;09:01.2
module /initrd.img-2.6.32.16+xen-2.6.32.x-20100713-a
lspci:
00:00.0 Host bridge: ATI Technologies Inc RD890 Northbridge only single slot
PCI-e GFX Hydra part (rev 02)
00:00.2 Generic system peripheral [0806]: ATI Technologies Inc Device 5a23
00:02.0 PCI bridge: ATI Technologies Inc RD890 PCI to PCI bridge (PCI express
gpp port B) 00:03.0 PCI bridge: ATI Technologies Inc RD890 PCI to PCI bridge
(PCI express gpp port C) 00:04.0 PCI bridge: ATI Technologies Inc RD890 PCI to
PCI bridge (PCI express gpp port D) 00:05.0 PCI bridge: ATI Technologies Inc
RD890 PCI to PCI bridge (PCI express gpp port E) 00:06.0 PCI bridge: ATI
Technologies Inc RD890 PCI to PCI bridge (PCI express gpp port F) 00:09.0 PCI
bridge: ATI Technologies Inc RD890 PCI to PCI bridge (PCI express gpp port H)
00:0a.0 PCI bridge: ATI Technologies Inc RD890 PCI to PCI bridge (external gfx1
port A) 00:0b.0 PCI bridge: ATI Technologies Inc RD890 PCI to PCI bridge (NB-SB
link) 00:11.0 SATA controller: ATI Technologies Inc SB700/SB800 SATA Controller
[IDE mode] (rev 40) 00:12.0 USB Controller: ATI Technologies Inc SB700/SB800 USB
OHCI0 Controller
00:12.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
00:13.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
00:13.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
00:14.0 SMBus: ATI Technologies Inc SBx00 SMBus Controller (rev 41)
00:14.1 IDE interface: ATI Technologies Inc SB700/SB800 IDE Controller (rev 40)
00:14.2 Audio device: ATI Technologies Inc SBx00 Azalia (Intel HDA) (rev 40)
00:14.3 ISA bridge: ATI Technologies Inc SB700/SB800 LPC host controller (rev
40)
00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (rev 40)
00:14.5 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI2 Controller
00:15.0 PCI bridge: ATI Technologies Inc Device 43a0 00:16.0 USB Controller: ATI
Technologies Inc SB700/SB800 USB OHCI0 Controller
00:16.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
00:18.0 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64,
Sempron] HyperTransport Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64,
Sempron] Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64,
Sempron] DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64,
Sempron] Miscellaneous Control
00:18.4 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64,
Sempron] Link Control 03:06.0 USB Controller: VIA Technologies, Inc. VT82xxxxx
UHCI USB 1.1 Controller (rev 62)
03:06.1 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller
(rev 62)
03:06.2 USB Controller: VIA Technologies, Inc. USB 2.0 (rev 65) 04:00.0 PCI
bridge: PLX Technology, Inc. Device 8613 (rev ba) 05:01.0 PCI bridge: PLX
Technology, Inc. Device 8613 (rev ba) 05:02.0 PCI bridge: PLX Technology, Inc.
Device 8613 (rev ba) 06:00.0 SATA controller: Device 1b4b:9123 (rev 10) 07:00.0
USB Controller: NEC Corporation Device 0194 (rev 03) 08:00.0 PCI bridge: Texas
Instruments XIO2000(A)/XIO2200(A) PCI Express-to-PCI Bridge (rev 03) 09:01.0 USB
Controller: NEC Corporation USB (rev 43)
09:01.1 USB Controller: NEC Corporation USB (rev 43)
09:01.2 USB Controller: NEC Corporation USB 2.0 (rev 04) 0a:00.0 USB Controller:
NEC Corporation Device 0194 (rev 03) 0b:00.0 Ethernet controller: Realtek
Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller
(rev 03) 0c:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 03) 0d:00.0 FireWire
(IEEE 1394): VIA Technologies, Inc. Device 3403 0e:00.0 VGA compatible
controller: nVidia Corporation GeForce 8400 GS (rev a1) 0f:00.0 VGA compatible
controller: ATI Technologies Inc Mobility Radeon HD 3450
0f:00.1 Audio device: ATI Technologies Inc RV620 Audio device [Radeon HD 34xx
Series]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Wei Wang2
2010-Jul-14 14:31 UTC
[Xen-devel] Re: MSI 890FXA-GD70: AMD IOMMU, VmError: Failed to assign device to IOMMU, (XEN) AMD_IOV: IO_PAGE_FALT: domain:10, device id:0x900, fault address:0x0
Hi, AFAIK, Xen does not allow devices which share a same device id to be assigned into different domains. According to the IVRS table, device 03:xx.x uses the device id of 00:14.4, which means, both 03:xx.x and 00:14.4 should be hidden from dom0 and them can only be assign to a same domain. Device 00:14.2 might be an different story. Probably, current xen had added some checks to prevent devices on the SAME device number from been spread among different domains. I suppose you did not hide all functions on bus 0 device 14? Thanks, Wei On Wednesday 14 July 2010 01:13:40 Huang2, Wei wrote:> [cc Wei Wang here] > > In the function reassign_device() in pci_amd_iommu.c, it seems > pci_get_pdev_by_domain() fail to return a correct pci_dev. Here is the > decode of IVRS table. It looks like 03:xx.x is behind a bridge 00:14.4 and > uses this value as an alias ID. That might be a cause. From IVRS, I don''t > see anything wrong with 00:14.2 though. Maybe you can add more debug code > to reassign_device(). > > -Wei > > 02 10 00 00 DEV=00:2.0 > 03 00 0f 00 04 01 0f 00 DEV=[0f:1.0, 0f:1.0] > 02 18 00 00 DEV=00:2.0 > 02 00 0e 00 DEV=0e:0.0 > 02 20 00 00 DEV=00:4.0 > 02 00 0d 00 DEV=0d:0.0 > 02 28 00 00 DEV=00:5.0 > 02 00 0c 00 DEV=0c:0.0 > 02 30 00 00 DEV=00:6.0 > 02 00 0b 00 DEV=0b:0.0 > 02 48 00 00 DEV=00:9.0 > 02 00 0a 00 DEV=0a:0.0 > 02 50 00 00 DEV=00:a.0 > 02 00 08 00 00 00 00 00 DEV=08:0.0 > 43 08 09 00 00 00 09 00 04 ff 09 00 DEV=[09:1.0, 09:1f.7] > 02 58 00 00 DEV=00:b.0 > 02 00 04 00 DEV=04:0.0 > 02 08 05 00 DEV=05:1.0 > 02 00 07 00 DEV=07:0.0 > 02 10 05 00 DEV=05:2.0 > 03 00 06 00 04 00 06 00 DEV=06:0.0 > 02 88 00 00 DEV=00:11.0 > 03 90 00 00 04 92 00 00 DEV=[00:12.0, 00:12.2] > 03 98 00 00 04 9a 00 00 DEV=[00:13.0, 00:13.2] > 02 a0 00 d7 DEV=00:14.0 > 02 a1 00 00 DEV=00:14.1 > 02 a2 00 00 DEV=00:14.2 > 02 a3 00 00 DEV=00:14.3 > 02 a4 00 00 DEV=00:14.4 > 43 00 03 00 00 a4 00 00 04 ff 03 00 DEV=[03:0.0, 03:1f.7] > 02 a5 00 00 DEV=00:14.5 > 02 a8 00 00 DEV=00:16.0 > 02 a9 00 00 DEV=00:16.1 > 02 00 01 00 DEV=01:0.0 > 03 b0 00 00 04 b2 00 00 00 00 00 00 DEV=[00:16.0, 00:16.2] > 48 00 00 d7 00 a0 00 02 > 48 00 00 00 07 01 00 01 > > > > -----Original Message----- > From: Sander Eikelenboom [mailto:linux@eikelenboom.it] > Sent: Tuesday, July 13, 2010 5:07 PM > To: xen-devel@lists.xensource.com > Cc: Przywara, Andre; Joerg Roedel; Huang2, Wei; Konrad Rzeszutek Wilk; > Jeremy Fitzhardinge Subject: MSI 890FXA-GD70: AMD IOMMU, VmError: Failed to > assign device to IOMMU, (XEN) AMD_IOV: IO_PAGE_FALT: domain:10, device > id:0x900, fault address:0x0 > > Hi All, > > I have bought a MSI 890FXA-GD70, and have been able to enable the IOMMU > with a beta-bios from MSI. I succeed in passing through some pci-e devices > to domU''s (PV), but some other devices fail with both PV and HVM domains. > > The message is allways the same "Failed to assign device to IOMMU", with > some added printk''s the return code from "iommu.c/assign_device" seems to > be ffffffed instead of the desired 0. > > Devices 00:14.2 and 03:06.* cause the domain(PV and HVM) to fail on > creation with: Failed to assign device to IOMMU (BDF) Devices 09:01.* and > 07:00.0 passthrough fine ( visible and working in pv domU). > > Since it''s a beta bios, could it be the IVRS table supplied by the bios is > not complete and therefore the iommu functions only for some devices ? With > amd_iommu_debug it sometimes also prints an "IO_PAGE_FALT": > (XEN) [VT-D]iommu.c:144: SEIK: assign_device: domain 14 (0:14.2)! > (XEN) [VT-D]iommu.c:162: SEIK: assign_device rc: ffffffed ! > (XEN) domctl.c:853:d0 XEN_DOMCTL_assign_device: assign device > (0:14.2) failed (XEN) AMD_IOV: IO_PAGE_FALT: domain:10, device id:0x900, > fault address:0x0 > > > -- > Sander > > > > > > > > Xen version: xen-4.0-testing-pre4 (changeset: 21263:57859775f88f) > Dom0 kernel: 2.6.32.16 pvops (commit > 006bfe8f594370f36f09d8e8ab13692d34e88533) > > Attached: > - xm info > - xm dmesg > - dmesg > - lspci -vvvknn > - Dump of ACPI IVRS table > - Dump of ACPI DSDT table > > > serveerstertje:~/iommu-report# xm pci-list-assignable-devices 0000:0e:00.0 > 0000:09:01.0 0000:09:01.1 > 0000:09:01.2 > 0000:03:06.0 > 0000:03:06.1 > 0000:03:06.2 > 0000:00:14.2 > > > grub menu.lst: > title xen-4.0.1-rc4-pre.gz / Debian GNU/Linux, kernel > 2.6.32.16+xen-2.6.32.x-20100713-a root (hd0,0) > kernel /xen-4.0.1-rc4-pre.gz dom0_mem=768M loglvl=all > loglvl_guest=all com1=115200,8n1 console=com1,vga > iommu=1,pv,verbose,amd_iommu_debug module > /vmlinuz-2.6.32.16+xen-2.6.32.x-20100713-a > root=/dev/mapper/serveerstertje-root ro earlyprintk=xen max_loop=255 > loop_max_part=63 > xen-pciback.hide=(00:14.2)(03:06.0)(03:06.1)(03:06.2)(07:00.0)(09:01.0)(09: >01.1)(09:01.2)(0e:00.0) > pci=resource_alignment=00:14.2;03:06.0;03:06.1;03:06.2;09:01.0;09:01.1;09:0 >1.2 module /initrd.img-2.6.32.16+xen-2.6.32.x-20100713-a > > > > > lspci: > > 00:00.0 Host bridge: ATI Technologies Inc RD890 Northbridge only single > slot PCI-e GFX Hydra part (rev 02) 00:00.2 Generic system peripheral > [0806]: ATI Technologies Inc Device 5a23 00:02.0 PCI bridge: ATI > Technologies Inc RD890 PCI to PCI bridge (PCI express gpp port B) 00:03.0 > PCI bridge: ATI Technologies Inc RD890 PCI to PCI bridge (PCI express gpp > port C) 00:04.0 PCI bridge: ATI Technologies Inc RD890 PCI to PCI bridge > (PCI express gpp port D) 00:05.0 PCI bridge: ATI Technologies Inc RD890 PCI > to PCI bridge (PCI express gpp port E) 00:06.0 PCI bridge: ATI Technologies > Inc RD890 PCI to PCI bridge (PCI express gpp port F) 00:09.0 PCI bridge: > ATI Technologies Inc RD890 PCI to PCI bridge (PCI express gpp port H) > 00:0a.0 PCI bridge: ATI Technologies Inc RD890 PCI to PCI bridge (external > gfx1 port A) 00:0b.0 PCI bridge: ATI Technologies Inc RD890 PCI to PCI > bridge (NB-SB link) 00:11.0 SATA controller: ATI Technologies Inc > SB700/SB800 SATA Controller [IDE mode] (rev 40) 00:12.0 USB Controller: ATI > Technologies Inc SB700/SB800 USB OHCI0 Controller 00:12.2 USB Controller: > ATI Technologies Inc SB700/SB800 USB EHCI Controller 00:13.0 USB > Controller: ATI Technologies Inc SB700/SB800 USB OHCI0 Controller 00:13.2 > USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller > 00:14.0 SMBus: ATI Technologies Inc SBx00 SMBus Controller (rev 41) 00:14.1 > IDE interface: ATI Technologies Inc SB700/SB800 IDE Controller (rev 40) > 00:14.2 Audio device: ATI Technologies Inc SBx00 Azalia (Intel HDA) (rev > 40) 00:14.3 ISA bridge: ATI Technologies Inc SB700/SB800 LPC host > controller (rev 40) 00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to > PCI Bridge (rev 40) 00:14.5 USB Controller: ATI Technologies Inc > SB700/SB800 USB OHCI2 Controller 00:15.0 PCI bridge: ATI Technologies Inc > Device 43a0 00:16.0 USB Controller: ATI Technologies Inc SB700/SB800 USB > OHCI0 Controller 00:16.2 USB Controller: ATI Technologies Inc SB700/SB800 > USB EHCI Controller 00:18.0 Host bridge: Advanced Micro Devices [AMD] > Family 10h [Opteron, Athlon64, Sempron] HyperTransport Configuration > 00:18.1 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron, > Athlon64, Sempron] Address Map 00:18.2 Host bridge: Advanced Micro Devices > [AMD] Family 10h [Opteron, Athlon64, Sempron] DRAM Controller 00:18.3 Host > bridge: Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, > Sempron] Miscellaneous Control 00:18.4 Host bridge: Advanced Micro Devices > [AMD] Family 10h [Opteron, Athlon64, Sempron] Link Control 03:06.0 USB > Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev > 62) 03:06.1 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 > Controller (rev 62) 03:06.2 USB Controller: VIA Technologies, Inc. USB 2.0 > (rev 65) 04:00.0 PCI bridge: PLX Technology, Inc. Device 8613 (rev ba) > 05:01.0 PCI bridge: PLX Technology, Inc. Device 8613 (rev ba) 05:02.0 PCI > bridge: PLX Technology, Inc. Device 8613 (rev ba) 06:00.0 SATA controller: > Device 1b4b:9123 (rev 10) 07:00.0 USB Controller: NEC Corporation Device > 0194 (rev 03) 08:00.0 PCI bridge: Texas Instruments XIO2000(A)/XIO2200(A) > PCI Express-to-PCI Bridge (rev 03) 09:01.0 USB Controller: NEC Corporation > USB (rev 43) 09:01.1 USB Controller: NEC Corporation USB (rev 43) > 09:01.2 USB Controller: NEC Corporation USB 2.0 (rev 04) 0a:00.0 USB > Controller: NEC Corporation Device 0194 (rev 03) 0b:00.0 Ethernet > controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express > Gigabit Ethernet controller (rev 03) 0c:00.0 Ethernet controller: Realtek > Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet > controller (rev 03) 0d:00.0 FireWire (IEEE 1394): VIA Technologies, Inc. > Device 3403 0e:00.0 VGA compatible controller: nVidia Corporation GeForce > 8400 GS (rev a1) 0f:00.0 VGA compatible controller: ATI Technologies Inc > Mobility Radeon HD 3450 0f:00.1 Audio device: ATI Technologies Inc RV620 > Audio device [Radeon HD 34xx Series]_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel