Xen devel - Jan 2012 - [PATCH 0 of 7 V4] amd iommu: support ATS/GPGPU passthru on IOMMUv2 systems

Sorry forgot to CC xen-devel
Thanks,
Wei

Hi all, this is patch v4.
ATS devices with PRI and PASID capabilities can communicate with iommuv2 
to perform two level (nested) address translation and demand paging for 
DMA. To passthru such devices, iommu driver has to beenenabled in guest 
OS. This patch set adds initial
iommu emulation for hvm guests to support ATS device passthru.

This patch set should work together with following hw & sw systems:

* Host system with AMD trinity APU which has enhanced iommu(iommuv2).
* AMD Radeon HD 7900 series graphic card.
* Linux 3.3 or later which has included iommuv2 kernel driver.
* AMD catalyst or open source Radeon driver in guest that support HD7900.

Most of the works is done by hypervisor. a new guest config parameter
guest_iommu = {1,0} is introduced to en/disable iommu emulation.
3 hyper-calls are added to support communications among different xen 
components.

* iommu_set_msi: used by qemu to inform hypervisor iommu vector number 
in guest space. Hypervisor needs this vector to inject msi into guest 
after write PPR log entry into guest buffer.

* iommu_bind_bdf: used by xl to bind virtual bdf of passthru device to 
machine bdf. IOMMU emulator receives iommu cmd from guest OS and then 
forwards them  to host iommu. But virtual device id in cmds from guest 
should be converted into physical id before sending them to real hardware.

* guest_iommu_set_base: IOMMU MMIO base address is dynamically allocated 
by firmware. This hypercall allows hvmloader to notify hypervisor where 
the iommu mmio pages are.

I had a picture to explain this better, pls refer to the link.

http://www.amd64.org/pub/iommuv2.png

Thanks,
Wei

=====================================================================
changes in v4:
* Only tools part in this version, since hypervisor patches have already 
been committed.
* rename guest config option from "iommu = {0,1}" to "guest_iommu
= {0,1}"
* add description into docs/man/xl.cfg.pod.5


changes in v3:
* Use xenstore to receive guest iommu configuration instead of adding in 
a new field in hvm_info_table.
* Support pci segment in vbdf to mbdf bind.
* Make hypercalls visible for non-x86 platforms.
* A few code cleanups according to comments from Jan and Ian.

Changes in v2:
* Do not use linked list to access guest iommu tables.
* Do not parse iommu parameter in libxl_device_model_info again.
* Fix incorrect logical calculation in patch 11.
* Fix hypercall definition for non-x86 systems.

Thanks,
Wei
_______________________________________________
osrc-patches mailing list
osrc-patches@elbe.amd.com
https://elbe.amd.com/mailman/listinfo/osrc-patches

>>> On 20.01.12 at 16:44, Wei Wang <wei.wang2@amd.com> wrote:
> Hi all, this is patch v4.
With the current apparent regression from the previous patch series,
can we ask you to first look into that problem before pushing in more
changes?

Thanks, Jan
> ATS devices with PRI and PASID capabilities can communicate with iommuv2 to
> perform 
> two level (nested) address translation and demand paging for DMA. To 
> passthru such 
> devices, iommu driver has to beenenabled in guest OS. This patch set adds 
> initial 
> iommu emulation for hvm guests to support ATS device passthru. 
> 
> This patch set should work together with following hw & sw systems:
> 
> * Host system with AMD trinity APU which has enhanced iommu(iommuv2). 
> * AMD Radeon HD 7900 series graphic card.  
> * Linux 3.3 or later which has included iommuv2 kernel driver.
> * AMD catalyst or open source Radeon driver in guest that support HD7900. 
> 
> Most of the works is done by hypervisor. a new guest config parameter
> guest_iommu = {1,0} is introduced to en/disable iommu emulation. 
> 3 hyper-calls are added to support communications among different xen 
> components.
>  
> * iommu_set_msi: used by qemu to inform hypervisor iommu vector number in 
> guest 
> space. Hypervisor needs this vector to inject msi into guest after write
PPR
> log
> entry into guest buffer.
> 
> * iommu_bind_bdf: used by xl to bind virtual bdf of passthru device to 
> machine bdf. 
> IOMMU emulator receives iommu cmd from guest OS and then forwards them  to 
> host 
> iommu. But virtual device id in cmds from guest should be converted into 
> physical 
> id before sending them to real hardware.
> 
> * guest_iommu_set_base: IOMMU MMIO base address is dynamically allocated by
> firmware. This hypercall allows hvmloader to notify hypervisor where the 
> iommu
> mmio pages are.
> 
> I had a picture to explain this better, pls refer to the link.
> 
> http://www.amd64.org/pub/iommuv2.png 
> 
> Thanks,
> Wei
> 
> =====================================================================> 
> changes in v4:
> * Only tool part in this version, since hypervisor patches have already
been
> committed.
> * rename guest config option from "iommu = {0,1}" to
"guest_iommu = {0,1}"
> * add description into docs/man/xl.cfg.pod.5
> 
> 
> changes in v3:
> * Use xenstore to receive guest iommu configuration instead of adding in a 
> new field in hvm_info_table.
> * Support pci segment in vbdf to mbdf bind.
> * Make hypercalls visible for non-x86 platforms.
> * A few code cleanups according to comments from Jan and Ian.
> 
> Changes in v2:
> * Do not use linked list to access guest iommu tables.
> * Do not parse iommu parameter in libxl_device_model_info again.
> * Fix incorrect logical calculation in patch 11.
> * Fix hypercall definition for non-x86 systems.
> 
> Thanks,
> Wei

>>> On 20.01.12 at 17:21, Wei Wang <wei.wang2@amd.com> wrote:
> could you provide more info about your push test on amd machines? This 
> patch queue should not affect any amd machines that do not have iommuv2 
> hardware in it.
My suspicion is that the previous set still has something in it that breaks
if a system does not have a v2 IOMMU (or maybe no IOMMU at all). Did
you cross check the previous series this way (apparently you didn''t
check on Intel systems, otherwise 24520:9a967990b4d2 wouldn''t have
been necessary, which allowed for the fuzzy push Ian referred to)?

Jan

On 01/20/2012 05:26 PM, Jan Beulich wrote:
>>>> On 20.01.12 at 17:21, Wei Wang<wei.wang2@amd.com>  wrote:
>> could you provide more info about your push test on amd machines? This
>> patch queue should not affect any amd machines that do not have iommuv2
>> hardware in it.
>
> My suspicion is that the previous set still has something in it that breaks
> if a system does not have a v2 IOMMU (or maybe no IOMMU at all). Did
> you cross check the previous series this way (apparently you
didn''t
> check on Intel systems, otherwise 24520:9a967990b4d2 wouldn''t have
> been necessary, which allowed for the fuzzy push Ian referred to)?
>
> Jan
>
>
Hi I had tested this on old iommu systems before submission. But I can 
re-test it again. Non-iommu system will also be tested. It might take 
some in setting up an Intel machine on my side. Could Ian confirm that 
it works on Intel or not?

Thanks,
Wei

>>> On 20.01.12 at 17:37, Wei Wang <wei.wang2@amd.com> wrote:
> Hi I had tested this on old iommu systems before submission. But I can 
> re-test it again. Non-iommu system will also be tested. It might take 
> some in setting up an Intel machine on my side. Could Ian confirm that 
> it works on Intel or not?
According to the failure pattern of the stage tester this is now
apparently working okay on Intel, but failing on AMD.

Jan