Xen devel - Aug 2010 - AMD iommu , pci passthrough problems assigning devices to HVM

Hi,

I''m running a machine with:
- Hypervisor: xen-unstable
- Dom0: xen/next-2.6.32.17  commit c3f5aba942c7516a5c76c47345ff0a473daf2542

Hardware:
-MSI 890FXA-GD70 with AMD IOMMU
-Phenom X6

I''m trying to do PCI passthrough to a HVM, this fails with:

Error: Failed to assign device to IOMMU
(0000:04:00.0@100,msitranslate=1,power_mgmt=0)
Error: pci: to avoid potential security issue, 0000:08:00.0 is not allowed to be
assigned to guest since it is behind PCIe switch that does not support or enable
ACS.


I have found this thread https://bugzilla.redhat.com/show_bug.cgi?id=526713
regarding the ACS problem, which says the pci bridges need these capabilities:
        ScrValid+, ReqRedir+, CmpltRedir+, and UpstreamFwd+  

When i do a lspci -vvv on my system i see these capabilities, but i do see some
disturbing debug message on boot when amd_iommu_debug is on.

Attached:
- xm info
- xm dmesg (booted with amd_iommu_debug on)
- dmesg
- lspci -vvv

--
Sander





_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

In your device 0b.0 I see this:

		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl-
DirectTrans+
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl-
DirectTrans-

So it looks to me as if the device is ACS capable but that the capability is not
turned on?

I think every device in the path has to be ACS capable and the capability
enabled.

- Greg Rose
LAD
Intel Corp.
>-----Original Message-----
>From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
>bounces@lists.xensource.com] On Behalf Of Sander Eikelenboom
>Sent: Tuesday, August 10, 2010 5:26 AM
>To: Huang2, Wei
>Cc: Wang2, Wei; Przywara, Andre; Joerg Roedel; xen-
>devel@lists.xensource.com
>Subject: [Xen-devel] AMD iommu , pci passthrough problems assigning
>devices to HVM
>
>Hi,
>
>I''m running a machine with:
>- Hypervisor: xen-unstable
>- Dom0: xen/next-2.6.32.17  commit
>c3f5aba942c7516a5c76c47345ff0a473daf2542
>
>Hardware:
>-MSI 890FXA-GD70 with AMD IOMMU
>-Phenom X6
>
>I''m trying to do PCI passthrough to a HVM, this fails with:
>
>Error: Failed to assign device to IOMMU
>(0000:04:00.0@100,msitranslate=1,power_mgmt=0)
>Error: pci: to avoid potential security issue, 0000:08:00.0 is not
>allowed to be assigned to guest since it is behind PCIe switch that does
>not support or enable ACS.
>
>
>I have found this thread
>https://bugzilla.redhat.com/show_bug.cgi?id=526713 regarding the ACS
>problem, which says the pci bridges need these capabilities:
>        ScrValid+, ReqRedir+, CmpltRedir+, and UpstreamFwd+
>
>When i do a lspci -vvv on my system i see these capabilities, but i do
>see some disturbing debug message on boot when amd_iommu_debug is on.
>
>Attached:
>- xm info
>- xm dmesg (booted with amd_iommu_debug on)
>- dmesg
>- lspci -vvv
>
>--
>Sander
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi,
In your debug output I saw:
(XEN) AMD-Vi: Fail to find iommu for device04:00.0
This means no iommu attached to 04:00.0, so it cannot be assigned to any hvm 
guest. What is this device in your system? An add-on device or on board 
device? It looks like BIOS failed to create an IVRS entry for this device.

Wei

On Tuesday 10 August 2010 14:26:14 Sander Eikelenboom
wrote:
> Hi,
>
> I''m running a machine with:
> - Hypervisor: xen-unstable
> - Dom0: xen/next-2.6.32.17  commit c3f5aba942c7516a5c76c47345ff0a473daf2542
>
> Hardware:
> -MSI 890FXA-GD70 with AMD IOMMU
> -Phenom X6
>
> I''m trying to do PCI passthrough to a HVM, this fails with:
>
> Error: Failed to assign device to IOMMU
> (0000:04:00.0@100,msitranslate=1,power_mgmt=0) Error: pci: to avoid
> potential security issue, 0000:08:00.0 is not allowed to be assigned to
> guest since it is behind PCIe switch that does not support or enable ACS.
>
>
> I have found this thread https://bugzilla.redhat.com/show_bug.cgi?id=526713
> regarding the ACS problem, which says the pci bridges need these
> capabilities: ScrValid+, ReqRedir+, CmpltRedir+, and UpstreamFwd+
>
> When i do a lspci -vvv on my system i see these capabilities, but i do see
> some disturbing debug message on boot when amd_iommu_debug is on.
>
> Attached:
> - xm info
> - xm dmesg (booted with amd_iommu_debug on)
> - dmesg
> - lspci -vvv
>
> --
> Sander



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hello Wei,

It''s an pci express usb3/xhci add-on card, there also seemed to be an
alias problem with another device, see this thread on LKML, it also contains the
result of amd_iommu_dump on baremetal 2.6.35 kernel.
At first the kernel didn''t boot at all, though xen-unstable did load,
so i guess there are a few differences in the amd iommu implementation and
parsing between the linux kernel and xen.

http://lkml.indiana.edu/hypermail/linux/kernel/1008.1/00782.html

Would be nice to have some good explanation of the problem, so i could contact
MSI to get the bios fixed.

--
Sander



Wednesday, August 11, 2010, 12:11:58 PM, you wrote:
> Hi,
> In your debug output I saw:
> (XEN) AMD-Vi: Fail to find iommu for device04:00.0
> This means no iommu attached to 04:00.0, so it cannot be assigned to any
hvm
> guest. What is this device in your system? An add-on device or on board 
> device? It looks like BIOS failed to create an IVRS entry for this device.
> Wei
> On Tuesday 10 August 2010 14:26:14 Sander Eikelenboom wrote:
>> Hi,
>>
>> I''m running a machine with:
>> - Hypervisor: xen-unstable
>> - Dom0: xen/next-2.6.32.17  commit
c3f5aba942c7516a5c76c47345ff0a473daf2542
>>
>> Hardware:
>> -MSI 890FXA-GD70 with AMD IOMMU
>> -Phenom X6
>>
>> I''m trying to do PCI passthrough to a HVM, this fails with:
>>
>> Error: Failed to assign device to IOMMU
>> (0000:04:00.0@100,msitranslate=1,power_mgmt=0) Error: pci: to avoid
>> potential security issue, 0000:08:00.0 is not allowed to be assigned to
>> guest since it is behind PCIe switch that does not support or enable
ACS.
>>
>>
>> I have found this thread
https://bugzilla.redhat.com/show_bug.cgi?id=526713
>> regarding the ACS problem, which says the pci bridges need these
>> capabilities: ScrValid+, ReqRedir+, CmpltRedir+, and UpstreamFwd+
>>
>> When i do a lspci -vvv on my system i see these capabilities, but i do
see
>> some disturbing debug message on boot when amd_iommu_debug is on.
>>
>> Attached:
>> - xm info
>> - xm dmesg (booted with amd_iommu_debug on)
>> - dmesg
>> - lspci -vvv
>>
>> --
>> Sander





-- 
Best regards,
 Sander                            mailto:linux@eikelenboom.it


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel