Xen devel - Nov 2011 - [PATCH 0/6] IOMMU, vtd and iotlb flush rework (v4)

In one of my previous email I detailed a bug I was seeing when passing
through a Intel GPU on a guest that has more that 4G or RAM.

Allen suggested that I go for the Plan B but after a discussion with Tim
we agreed that Plan B was way to disruptive in term of code change.

This patch series implements Plan A.

http://xen.1045712.n5.nabble.com/VTD-Intel-iommu-IOTLB-flush-really-slow-td4952866.html

Changes between v3 and v4:
        - Move the loop for gmfn_range from arch_memory_op to
xenmem_add_to_physmap.
        - Add a comment to comment to explain the purpose of
iommu_dont_flush_iotlb.

Changes between v2 and v3:
       - Check for the presence iotlb_flush_all callback before calling it.

Changes between v1 and v2:
      - Move size in struct xen_add_to_physmap in padding between .domid and
.space.
      - Store iommu_dont_flush per cpu
      - Change the code in hvmloader to relocate by batch of 64K, .size is now
16 bits.


 tools/firmware/hvmloader/pci.c      |   20 +++++--
 xen/arch/x86/mm.c                   |   82 ++++++++++++++++++++++------
 xen/arch/x86/x86_64/compat/mm.c     |    4 ++
 xen/drivers/passthrough/iommu.c     |   25 +++++++++
 xen/drivers/passthrough/vtd/iommu.c |  100 ++++++++++++++++++++---------------
 xen/include/public/memory.h         |    4 ++
 xen/include/xen/iommu.h             |   17 ++++++
 7 files changed, 186 insertions(+), 66 deletions(-)

Jean Guyader (6):
      vtd: Refactor iotlb flush code
      iommu: Introduce iommu_flush and iommu_flush_all.
      add_to_physmap: Move the code for XENMEM_add_to_physmap.
      mm: New XENMEM space, XENMAPSPACE_gmfn_range
      hvmloader: Change memory relocation loop when overlap with PCI hole.
      Introduce per cpu flag (iommu_dont_flush_iotlb) to avoid unnecessary iotlb
flush



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Factorize the iotlb flush code from map_page and unmap_page into
it''s own function.

Signed-off-by: Jean Guyader <jean.guyader@eu.citrix.com>
---
 xen/drivers/passthrough/vtd/iommu.c |   86 +++++++++++++++++-----------------
 1 files changed, 43 insertions(+), 43 deletions(-)



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Signed-off-by: Jean Guyader <jean.guyader@eu.citrix.com>
---
 xen/drivers/passthrough/iommu.c     |   20 ++++++++++++++++++++
 xen/drivers/passthrough/vtd/iommu.c |   12 ++++++++++++
 xen/include/xen/iommu.h             |    5 +++++
 3 files changed, 37 insertions(+), 0 deletions(-)



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Move the code for the XENMEM_add_to_physmap case into it''s own
function (xenmem_add_to_physmap).

Signed-off-by: Jean Guyader <jean.guyader@eu.citrix.com>
---
 xen/arch/x86/mm.c |  188 ++++++++++++++++++++++++++++-------------------------
 1 files changed, 99 insertions(+), 89 deletions(-)



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

XENMAPSPACE_gmfn_range is like XENMAPSPACE_gmfn but it runs on
a range of pages. The size of the range is defined in a new field.

This new field .size is located in the 16 bits padding between .domid
and .space in struct xen_add_to_physmap to stay compatible with older
versions.

Signed-off-by: Jean Guyader <jean.guyader@eu.citrix.com>
---
 xen/arch/x86/mm.c               |  164 ++++++++++++++++++++++-----------------
 xen/arch/x86/x86_64/compat/mm.c |    4 +
 xen/include/public/memory.h     |    4 +
 3 files changed, 102 insertions(+), 70 deletions(-)



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Change the way we relocate the memory page if they overlap with pci hole.
Use new map space (XENMAPSPACE_gmfn_range) to move the loop into xen.

This code usually get triggered when a device is pass through to a guest
and the PCI hole has to be extended to have enough room to map the device BARs.
The PCI hole will starts lower and it might overlap with some RAM that has been
alocated for the guest. That usually happen if the guest has more than 4G of
RAM.
We have to relocate those pages in high mem otherwise they won''t be
accessible.

Signed-off-by: Jean Guyader <jean.guyader@eu.citrix.com>
---
 tools/firmware/hvmloader/pci.c |   20 ++++++++++++++------
 1 files changed, 14 insertions(+), 6 deletions(-)



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Add cpu flag that will be checked by the iommu low level code
to skip iotlb flushes. iommu_iotlb_flush shall be called explicitly.

Signed-off-by: Jean Guyader <jean.guyader@eu.citrix.com>
---
 xen/arch/x86/mm.c                   |   14 ++++++++++++++
 xen/drivers/passthrough/iommu.c     |    5 +++++
 xen/drivers/passthrough/vtd/iommu.c |    6 ++++--
 xen/include/xen/iommu.h             |   12 ++++++++++++
 4 files changed, 35 insertions(+), 2 deletions(-)



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi, 

diff --git a/xen/arch/x86/x86_64/compat/mm.c b/xen/arch/x86/x86_64/compat/mm.c
index 2c05099..05c3098 100644
--- a/xen/arch/x86/x86_64/compat/mm.c
+++ b/xen/arch/x86/x86_64/compat/mm.c
@@ -63,6 +63,10 @@ int compat_arch_memory_op(int op, XEN_GUEST_HANDLE(void) arg)
 
         XLAT_add_to_physmap(nat, &cmp);
         rc = arch_memory_op(op, guest_handle_from_ptr(nat, void));
+        XLAT_add_to_physmap(&cmp, nat);
+
+        if ( copy_to_guest(arg, &cmp, 1) )
+            return -EFAULT;
 
         break;
     }

I think you might need to do a bit more than this - have a look at how
XENMEM_get_pod_target is handled (in the same file), including
re-translating the hypercall continuation.

Please explicitly test the case where the hypercall gets interrupted and
the caller is a 32-bit compat guest, and make sure it does what you want.

(Apart from that the x86/mm parts of the series are now fine by me)

Cheers,

Tim.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel