Kay, Allen M
2007-Apr-06 18:21 UTC
[Xen-devel] [VTD][patch 0/6] HVM device assignment using vt-d
Keir,
Following 6 patches are for enabling PCI device assignment to a HVM
domain
using vt-d. We have done no-harm-done testing for PV and non-vtd HVM
guests on
32PAE, x86_64 and IPF. See below for details of the patch.
Allen
----------------------
1) patch description: (applies cleanly to cs #14753)
vtd1.patch:
- vt-d specific code + header file changes
- not much common code interaction
vtd2.patch:
- PCI config virtualization in QEMU + control panel/libxc changes
- pass pci parameter to qemu, pci config change, make hypercalls
described in (5)
- not much common code interaction
vtd3.patch:
- domctl hypercall changes
- impelments hypercalls described in (5)
- some common code interaction
vtd4.patch:
- io port handling
- some common code interaction
vtd5.patch:
- interrupt handling
- heavy common code interaction
vtd6.patch:
- mmio handling
- heavy common code interaction in multi.c and page_alloc.c
2) environment tested:
Assigned PCIe E1000 add-on card to 32-bit FC5 on 64-bit Xen. Informal
"scp" test
shows 200+Mbps - similar to native performance on my system.
3) how to run
- Use same syntax as PV driver domain method to "hide" and assign PCI
device
- use pciback.hid=(02:00.0) to "hide" device from dom0
- use pci = [ ''02:00.00'' ] in /etc/xen/hvm.conf to assign
device to
HVM domain
- set acpi and apic to 0 in hvm.conf as current patch only works
with PIC
- grub.conf: use "ioapic_ack=old" for /boot/xen.gz
(io_apic.c contains code for avoiding global interrupt problem)
4) description of hvm PCI device assignment design:
- pci config virtualization
- Control panel and qemu changed to pass assigned PCI devices to qemu.
- A new file ioemu/hw/dpci.c reads assigned devices PCI conf and
constructs a
new virtual device and attaches to the guest PCI bus.
- PCI read/write functions are similar to other virtual devices.
Except
write function intercepts writes to COMMAND register and do actual
hardware writes.
- interrupt virtualization
- Currently only works for ACPI/APIC mode
- dpci.c makes a hypercall to tell xen device/intx on vPCI
- In do_IRQ_guest(), when Xen determines a interrupt belongs to a
device
owned by HVM domain, it injects guest IRQ to the domain
- Revert back to ioapic_ack=old to allow for IRQ sharing amongst
guests.
- Implemented new method in io_apic.c to avoid global interrupt issue.
- mmio
- When guest BIOS (i.e hvmloader) or OS changes PCI BAR, PCI config
write
function in qemu makes a hypercall to instruct Xen to construct p2m
mapping.
- shadow page table fault handler have been modified to allow memory
above
max_pages to be mapped.
- ioport
- Xen intercepts guest io port accesses
- translates guest io port to machine io port
- does machine port access on behalf of guest
5) new hypercalls
int xc_assign_device(int xc_handle,
uint32_t domain_id,
uint32_t machine_bdf);
int xc_domain_ioport_mapping(int xc_handle,
uint32_t domid,
uint32_t first_gport,
uint32_t first_mport,
uint32_t nr_ports,
uint32_t add_mapping);
int xc_irq_mapping(int xc_handle,
uint32_t domain_id,
uint32_t method,
uint32_t machine_irq,
uint32_t device,
uint32_t intx,
uint32_t add_mapping);
int xc_domain_memory_mapping(int xc_handle,
uint32_t domid,
unsigned long first_gfn,
unsigned long first_mfn,
unsigned long nr_mfns,
uint32_t add_mapping);
6) interface to common code:
Calls to following common code interfaces are qualified by
iommu_found() and device_assigned() macros.
int iommu_setup(void);
int iommu_domain_init(struct domain *d);
int assign_device(struct domain *d, u8 bus, u8 devfn);
int release_devices(struct vcpu *v);
int hvm_do_IRQ_dpci(struct domain *d, unsigned int irq);
int dpci_ioport_intercept(ioreq_t *p, int type);
int iommu_page_mapping(
struct domain *domain, dma_addr_t iova,
void *hpa, size_t size, int prot);
int iommu_page_unmapping(
struct domain *domain, dma_addr_t iova, size_t size);
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Muli Ben-Yehuda
2007-Apr-06 19:52 UTC
Re: [Xen-devel] [VTD][patch 0/6] HVM device assignment using vt-d
On Fri, Apr 06, 2007 at 11:21:29AM -0700, Kay, Allen M wrote:> Keir, > > Following 6 patches are for enabling PCI device assignment to a HVM > domain using vt-d. We have done no-harm-done testing for PV and > non-vtd HVM guests on 32PAE, x86_64 and IPF. See below for details > of the patch.Cool! I have a couple of questions: 1. When is VT-d hardware expected to be available? 2. We need a way for this to work with multiple IOMMUs - VT-d, AMD''s upcoming IOMMU and IBM''s Calgary/CalIOC2 family of IOMMUs are the ones that are either available or will be soon. From a quick look at the code your patch is VT-d specific? What I''ve done so far in the xen-iommu tree is to add a new platform op by which dom0 tells Xen about the presence of a new IOMMU and then add a common layer in Xen for different IOMMUs to hook into using an iommu_ops structure. Will this work for you, or do you have something else in mind? More to come as I digest your patches. It''s great to finally see VT-d support out-there! Cheers, Muli _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Kay, Allen M
2007-Apr-06 20:28 UTC
RE: [Xen-devel] [VTD][patch 0/6] HVM device assignment using vt-d
Hi Muli, I will need to check with people on HW availability outside of Intel and get back to you. I just checked out your iommu patch. Yes, this is a framework we can look into using. It is just a matter of mapping hypercalls and iommu_ops. It is possible we may need to add some operations. I would be happy to work with you on this. I see that your hypercall definitions do not contain interrupt/ioport/mmio related stuff. I''m wondering how do you handle these on Calgary implementation. Allen>-----Original Message----- >From: Muli Ben-Yehuda [mailto:muli@il.ibm.com] >Sent: Friday, April 06, 2007 12:53 PM >To: Kay, Allen M >Cc: xen-devel@lists.xensource.com >Subject: Re: [Xen-devel] [VTD][patch 0/6] HVM device >assignment using vt-d > >On Fri, Apr 06, 2007 at 11:21:29AM -0700, Kay, Allen M wrote: >> Keir, >> >> Following 6 patches are for enabling PCI device assignment to a HVM >> domain using vt-d. We have done no-harm-done testing for PV and >> non-vtd HVM guests on 32PAE, x86_64 and IPF. See below for details >> of the patch. > >Cool! I have a couple of questions: > >1. When is VT-d hardware expected to be available? >2. We need a way for this to work with multiple IOMMUs - VT-d, AMD''s > upcoming IOMMU and IBM''s Calgary/CalIOC2 family of IOMMUs are the > ones that are either available or will be soon. From a quick look > at the code your patch is VT-d specific? What I''ve done so far in > the xen-iommu tree is to add a new platform op by which dom0 tells > Xen about the presence of a new IOMMU and then add a common layer > in Xen for different IOMMUs to hook into using an iommu_ops > structure. Will this work for you, or do you have something else in > mind? > >More to come as I digest your patches. It''s great to finally see VT-d >support out-there! > >Cheers, >Muli >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Muli Ben-Yehuda
2007-Apr-06 20:32 UTC
Re: [Xen-devel] [VTD][patch 0/6] HVM device assignment using vt-d
On Fri, Apr 06, 2007 at 01:28:36PM -0700, Kay, Allen M wrote:> Hi Muli, > > I will need to check with people on HW availability outside of Intel > and get back to you.Thanks.> I just checked out your iommu patch. Yes, this is a framework we > can look into using. It is just a matter of mapping hypercalls and > iommu_ops. It is possible we may need to add some operations. I > would be happy to work with you on this.Great!> I see that your hypercall definitions do not contain > interrupt/ioport/mmio related stuff. I''m wondering how do you > handle these on Calgary implementationThe Calgary bits are implemented at the moment for PV domains only, so we blissfully ignore all that stuff. The IOMMU framework should definitely handle interrupt/ioport/mmio''s however, so let''s work together on adding them. Cheers, Muli _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel