Xen devel - Nov 2007 - [VTD][PATCH] Support intra-domain shared interrupt

Inter-domain shared interrupt has been supported by timeout method, but
it still doesn''t support intra-domain shared interrupt, that is
assigning multiple devices which share a physical irq to the same
domain. This patch implements intra-domain shared interrupt support. In
addition, this patch maps link to guest device/intx instead of directly
mapping isairq in pt_irq_create_bind_vtd(), because at this point the
isairqs got from pci_link are always 0. 

Note that assigning multiple devices to guests which uses PIC to handle
interrupts may be failed, because different links possibly connect to
same irq.


Signed-off-by: Weidong Han <weidong.han@intel.com>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

The two cases of hvm_dpci_eoi() (timeout versus non-timeout) have very
little code in common. Should be two different functions instead of adding a
cryptic final bool argument.

xmalloc() and list_add() should usually be accompanied with a destructing
list_del() and xfree(). So this patch leaks memory.

And why is intra-domain sharing harder (or at least different) to deal with
than inter-domain? Is this real physical IRQ sharing that''s a problem,
or
sharing of the virtual GSI?

 -- Keir

On 5/11/07 05:35, "Han, Weidong" <weidong.han@intel.com> wrote:
> Inter-domain shared interrupt has been supported by timeout method, but
> it still doesn''t support intra-domain shared interrupt, that is
> assigning multiple devices which share a physical irq to the same
> domain. This patch implements intra-domain shared interrupt support. In
> addition, this patch maps link to guest device/intx instead of directly
> mapping isairq in pt_irq_create_bind_vtd(), because at this point the
> isairqs got from pci_link are always 0.
> 
> Note that assigning multiple devices to guests which uses PIC to handle
> interrupts may be failed, because different links possibly connect to
> same irq.
> 
> 
> Signed-off-by: Weidong Han <weidong.han@intel.com>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
> The two cases of hvm_dpci_eoi() (timeout versus non-timeout) have very
> little code in common. Should be two different functions instead of
> adding a cryptic final bool argument.
> 
will cleanup it.
> xmalloc() and list_add() should usually be accompanied with a
> destructing list_del() and xfree(). So this patch leaks memory.
> 
will fix it.
> And why is intra-domain sharing harder (or at least different) to
> deal with than inter-domain? Is this real physical IRQ sharing
that''s
> a problem, or sharing of the virtual GSI?
> 
Real physical sharing is a problem. Guest irq is generated from
device/intx, so two shared physical interrupt devices will be mapped to
two different guest irqs. Current data structure hvm_irq_dpci_mapping
doesn''t consider this situation, mirq[i] contains only one device and
corresponding guest irq, need to add a list to contain multiple devices
whice share physical interrupt. What''s more, we need to deal with
intra-domain shared interrupt injection and eoi specially.

> 
> On 5/11/07 05:35, "Han, Weidong" <weidong.han@intel.com>
wrote:
> 
>> Inter-domain shared interrupt has been supported by timeout method,
>> but it still doesn''t support intra-domain shared interrupt,
that is
>> assigning multiple devices which share a physical irq to the same
>> domain. This patch implements intra-domain shared interrupt support.
>> In addition, this patch maps link to guest device/intx instead of
>> directly mapping isairq in pt_irq_create_bind_vtd(), because at this
>> point the isairqs got from pci_link are always 0.
>> 
>> Note that assigning multiple devices to guests which uses PIC to
>> handle interrupts may be failed, because different links possibly
>> connect to same irq. 
>> 
>> 
>> Signed-off-by: Weidong Han <weidong.han@intel.com>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 5/11/07 08:16, "Han, Weidong" <weidong.han@intel.com> wrote:
> Current data structure hvm_irq_dpci_mapping
> doesn''t consider this situation, mirq[i] contains only one device
and
> corresponding guest irq, need to add a list to contain multiple devices
> whice share physical interrupt.
The reverse is also true I believe: you can have two devices with different
physical IRQs getting aliased to the same guest GSI. Is that a problem?
e.g., do you need the reverse mapping from guest GSI to physical IRQ or
physical device (I''d guess you do, to work out what to do when the
guest GSI
is EOIed)?
> What''s more, we need to deal with
> intra-domain shared interrupt injection and eoi specially.
How?

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
> On 5/11/07 08:16, "Han, Weidong" <weidong.han@intel.com>
wrote:
> 
>> Current data structure hvm_irq_dpci_mapping
>> doesn''t consider this situation, mirq[i] contains only one
device and
>> corresponding guest irq, need to add a list to contain multiple
>> devices whice share physical interrupt.
> 
> The reverse is also true I believe: you can have two devices with
> different physical IRQs getting aliased to the same guest GSI. Is
> that a problem? e.g., do you need the reverse mapping from guest GSI
> to physical IRQ or physical device (I''d guess you do, to work out
> what to do when the guest GSI is EOIed)?
> 
Yes, we need the reverse mapping from guest GSI to physical IRQ. 
According to the following guest GSI computing formula, 
   #define hvm_pci_intx_gsi(dev, intx)  \
     (((((dev) << 2) + ((dev) >> 3) + (intx)) & 31) + 16)

that situation two devices with different pyhsical IRQs getting aliased
to the same guest GSI is hard to occur. And we need one guest GSI mapped
to only one physical IRQ, or we can''t know which device issues the
guest
GSI. Ideally, if we can make 1:1 mapping between guest GSI and physcial
IRQ, the intra-domain shared interrupt can naturally handled by guest.

>> What''s more, we need to deal with
>> intra-domain shared interrupt injection and eoi specially.
> 
> How?
> 
We need to inject guest GSI for each device, and really do eoi when all
devices with same physical IRQ in the domain finished eoi.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 5/11/07 08:52, "Han, Weidong" <weidong.han@intel.com> wrote:
> that situation two devices with different pyhsical IRQs getting aliased
> to the same guest GSI is hard to occur. And we need one guest GSI mapped
> to only one physical IRQ, or we can''t know which device issues the
guest
> GSI. Ideally, if we can make 1:1 mapping between guest GSI and physcial
> IRQ, the intra-domain shared interrupt can naturally handled by guest.
Ah, that''s true. I suppose you can assign virtual devfn locations for
devices to purposely avoid aliasing in GSI space. What do you do about
aliasing into the ISA IRQ space?

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>From: Keir Fraser
>Sent: 2007年11月5日 17:45
>On 5/11/07 08:52, "Han, Weidong" <weidong.han@intel.com>
wrote:
>
>> that situation two devices with different pyhsical IRQs 
>getting aliased
>> to the same guest GSI is hard to occur. And we need one 
>guest GSI mapped
>> to only one physical IRQ, or we can''t know which device 
>issues the guest
>> GSI. Ideally, if we can make 1:1 mapping between guest GSI 
>and physcial
>> IRQ, the intra-domain shared interrupt can naturally handled 
>by guest.
>
>Ah, that''s true. I suppose you can assign virtual devfn locations
for
>devices to purposely avoid aliasing in GSI space. What do you do about
>aliasing into the ISA IRQ space?
>
> -- Keir
>
ISA IRQ alias is less likely to be avoided, since guest OS itself may
program the vPIRQ links. Since it''s guest itself to choose using IOAPIC
or PIC, reverse alias (guest GSI -> physical irq) is inevitable. But we may
not need add extra structures. Just do bit search for all assigned mirqs 
and check whether corresponding girqs matching the EOI gsi. Normally
one domain is only assigned few devices and sharing case should be fewer.
Above walk just adds negligible overhead. :-)


Thanks,
Kevin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
> On 5/11/07 08:52, "Han, Weidong" <weidong.han@intel.com>
wrote:
> 
>> that situation two devices with different pyhsical IRQs getting
>> aliased to the same guest GSI is hard to occur. And we need one
>> guest GSI mapped to only one physical IRQ, or we can''t know
which
>> device issues the guest GSI. Ideally, if we can make 1:1 mapping
>> between guest GSI and physcial IRQ, the intra-domain shared
>> interrupt can naturally handled by guest. 
> 
> Ah, that''s true. I suppose you can assign virtual devfn locations
for
> devices to purposely avoid aliasing in GSI space. What do you do about
> aliasing into the ISA IRQ space?
> 
I map device/intx to link, and then map to the isa irq of the link. When
link route is updated, device/intx will map to the new isa irq of the
link. But due to different links may be set the same  isa irq, it can''t
guarantee that one guest isa irq maps to only one device/intx. So now I
don''t have a good solution to avoid aliasing into the ISA IRQ space.

-- Weidong

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir, how about this intra-domain shared interrupt patch? Attached patch
fixes memory leak issue and updates timeout function. Thanks.

-- Weidong 

Keir Fraser wrote:
> On 5/11/07 08:52, "Han, Weidong" <weidong.han@intel.com>
wrote:
> 
>> that situation two devices with different pyhsical IRQs getting
>> aliased to the same guest GSI is hard to occur. And we need one
>> guest GSI mapped to only one physical IRQ, or we can''t know
which
>> device issues the guest GSI. Ideally, if we can make 1:1 mapping
>> between guest GSI and physcial IRQ, the intra-domain shared
>> interrupt can naturally handled by guest. 
> 
> Ah, that''s true. I suppose you can assign virtual devfn locations
for
> devices to purposely avoid aliasing in GSI space. What do you do about
> aliasing into the ISA IRQ space?
> 
>  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Yes, I''ll slip this one in.

 K.

On 7/11/07 15:02, "Han, Weidong" <weidong.han@intel.com> wrote:
> Keir, how about this intra-domain shared interrupt patch? Attached patch
> fixes memory leak issue and updates timeout function. Thanks.
> 
> -- Weidong 
> 
> Keir Fraser wrote:
>> On 5/11/07 08:52, "Han, Weidong"
<weidong.han@intel.com> wrote:
>> 
>>> that situation two devices with different pyhsical IRQs getting
>>> aliased to the same guest GSI is hard to occur. And we need one
>>> guest GSI mapped to only one physical IRQ, or we can''t
know which
>>> device issues the guest GSI. Ideally, if we can make 1:1 mapping
>>> between guest GSI and physcial IRQ, the intra-domain shared
>>> interrupt can naturally handled by guest.
>> 
>> Ah, that''s true. I suppose you can assign virtual devfn
locations for
>> devices to purposely avoid aliasing in GSI space. What do you do about
>> aliasing into the ISA IRQ space?
>> 
>>  -- Keir
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir,

I found changeset 16331 breaks VT-d. When create guest with VT-d, qemu
window is white, and we see following messages on serial console:

(XEN) intel-iommu.c:1374:d0 reassign_device-1:0:0- source = 0 target = 1
(XEN) mm.c:684:d2 Attempt to change cache attributes of Xen heap page
(XEN) mm.c:684:d2 Attempt to change cache attributes of Xen heap page
(XEN) mm.c:684:d2 Attempt to change cache attributes of Xen heap page
(XEN) mm.c:684:d2 Attempt to change cache attributes of Xen heap page
(XEN) mm.c:684:d2 Attempt to change cache attributes of Xen heap page
(XEN) mm.c:684:d2 Attempt to change cache attributes of Xen heap page
(XEN) mm.c:684:d2 Attempt to change cache attributes of Xen heap page
(XEN) mm.c:684:d2 Attempt to change cache attributes of Xen heap page
(XEN) mm.c:684:d2 Attempt to change cache attributes of Xen heap page
(XEN) printk: 2214196 messages suppressed.
(XEN) mm.c:684:d2 Attempt to change cache attributes of Xen heap page
(XEN) printk: 2510520 messages suppressed.
(XEN) mm.c:684:d2 Attempt to change cache attributes of Xen heap page
(XEN) printk: 2515634 messages suppressed.
(XEN) mm.c:684:d2 Attempt to change cache attributes of Xen heap page
(XEN) printk: 2510520 messages suppressed.
(XEN) mm.c:684:d2 Attempt to change cache attributes of Xen heap page
(XEN) printk: 2515634 messages suppressed.
(XEN) mm.c:684:d2 Attempt to change cache attributes of Xen heap page
(XEN) printk: 2525556 messages suppressed.
(XEN) mm.c:684:d2 Attempt to change cache attributes of Xen heap page
(XEN) printk: 2513768 messages suppressed.
(XEN) mm.c:684:d2 Attempt to change cache attributes of Xen heap page
(XEN) printk: 2510754 messages suppressed.

Keir Fraser wrote:
> Yes, I''ll slip this one in.
> 
>  K.
> 
> On 7/11/07 15:02, "Han, Weidong" <weidong.han@intel.com>
wrote:
> 
>> Keir, how about this intra-domain shared interrupt patch? Attached
>> patch fixes memory leak issue and updates timeout function. Thanks.
>> 
>> -- Weidong
>> 
>> Keir Fraser wrote:
>>> On 5/11/07 08:52, "Han, Weidong"
<weidong.han@intel.com> wrote:
>>> 
>>>> that situation two devices with different pyhsical IRQs getting
>>>> aliased to the same guest GSI is hard to occur. And we need one
>>>> guest GSI mapped to only one physical IRQ, or we can''t
know which
>>>> device issues the guest GSI. Ideally, if we can make 1:1
mapping
>>>> between guest GSI and physcial IRQ, the intra-domain shared
>>>> interrupt can naturally handled by guest.
>>> 
>>> Ah, that''s true. I suppose you can assign virtual devfn
locations
>>> for devices to purposely avoid aliasing in GSI space. What do you
>>> do about aliasing into the ISA IRQ space?
>>> 
>>>  -- Keir
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Can you extend the test at line 750 in arch/x86/mm/shadow/multi.c with
... &&
!(mfn_valid(target_mfn)&&is_xen_heap_frame(mfn_to_page(target_mfn)))

And see if that fixes it? I suspect that the VLAPIC access page is mapped
uncacheable by HVM guests, and that uncacheability is getting passed through
to the shadow pte if the guest has pass-thru devices. We should not pass
thru cache attribute flags for xen heap pages as they should never be
involved in real I/O.

 -- Keir

On 7/11/07 15:16, "Han, Weidong" <weidong.han@intel.com> wrote:
> Keir,
> 
> I found changeset 16331 breaks VT-d. When create guest with VT-d, qemu
> window is white, and we see following messages on serial console:
> 
> (XEN) intel-iommu.c:1374:d0 reassign_device-1:0:0- source = 0 target = 1
> (XEN) mm.c:684:d2 Attempt to change cache attributes of Xen heap page
> (XEN) mm.c:684:d2 Attempt to change cache attributes of Xen heap page
> (XEN) mm.c:684:d2 Attempt to change cache attributes of Xen heap page
> (XEN) mm.c:684:d2 Attempt to change cache attributes of Xen heap page
> (XEN) mm.c:684:d2 Attempt to change cache attributes of Xen heap page
> (XEN) mm.c:684:d2 Attempt to change cache attributes of Xen heap page
> (XEN) mm.c:684:d2 Attempt to change cache attributes of Xen heap page
> (XEN) mm.c:684:d2 Attempt to change cache attributes of Xen heap page
> (XEN) mm.c:684:d2 Attempt to change cache attributes of Xen heap page
> (XEN) printk: 2214196 messages suppressed.
> (XEN) mm.c:684:d2 Attempt to change cache attributes of Xen heap page
> (XEN) printk: 2510520 messages suppressed.
> (XEN) mm.c:684:d2 Attempt to change cache attributes of Xen heap page
> (XEN) printk: 2515634 messages suppressed.
> (XEN) mm.c:684:d2 Attempt to change cache attributes of Xen heap page
> (XEN) printk: 2510520 messages suppressed.
> (XEN) mm.c:684:d2 Attempt to change cache attributes of Xen heap page
> (XEN) printk: 2515634 messages suppressed.
> (XEN) mm.c:684:d2 Attempt to change cache attributes of Xen heap page
> (XEN) printk: 2525556 messages suppressed.
> (XEN) mm.c:684:d2 Attempt to change cache attributes of Xen heap page
> (XEN) printk: 2513768 messages suppressed.
> (XEN) mm.c:684:d2 Attempt to change cache attributes of Xen heap page
> (XEN) printk: 2510754 messages suppressed.
> 
> Keir Fraser wrote:
>> Yes, I''ll slip this one in.
>> 
>>  K.
>> 
>> On 7/11/07 15:02, "Han, Weidong"
<weidong.han@intel.com> wrote:
>> 
>>> Keir, how about this intra-domain shared interrupt patch? Attached
>>> patch fixes memory leak issue and updates timeout function. Thanks.
>>> 
>>> -- Weidong
>>> 
>>> Keir Fraser wrote:
>>>> On 5/11/07 08:52, "Han, Weidong"
<weidong.han@intel.com> wrote:
>>>> 
>>>>> that situation two devices with different pyhsical IRQs
getting
>>>>> aliased to the same guest GSI is hard to occur. And we need
one
>>>>> guest GSI mapped to only one physical IRQ, or we
can''t know which
>>>>> device issues the guest GSI. Ideally, if we can make 1:1
mapping
>>>>> between guest GSI and physcial IRQ, the intra-domain shared
>>>>> interrupt can naturally handled by guest.
>>>> 
>>>> Ah, that''s true. I suppose you can assign virtual
devfn locations
>>>> for devices to purposely avoid aliasing in GSI space. What do
you
>>>> do about aliasing into the ISA IRQ space?
>>>> 
>>>>  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
> Can you extend the test at line 750 in arch/x86/mm/shadow/multi.c with
> ... &&
>
!(mfn_valid(target_mfn)&&is_xen_heap_frame(mfn_to_page(target_mfn)))
> 
> And see if that fixes it? I suspect that the VLAPIC access page is
> mapped uncacheable by HVM guests, and that uncacheability is getting
> passed through to the shadow pte if the guest has pass-thru devices.
> We should not pass thru cache attribute flags for xen heap pages as
> they should never be involved in real I/O.
> 
Yes, that fixes it. My modification is as follows:

diff -r d4c5a1cdcf2e xen/arch/x86/mm/shadow/multi.c
--- a/xen/arch/x86/mm/shadow/multi.c	Wed Nov 07 16:55:13 2007 +0000
+++ b/xen/arch/x86/mm/shadow/multi.c	Thu Nov 08 09:17:50 2007 +0800
@@ -748,7 +748,9 @@ _sh_propagate(struct vcpu *v,
      * caching attributes in the shadows to match what was asked for
      */
     if ( (level == 1) && is_hvm_domain(d) &&
-         !list_empty(&(domain_hvm_iommu(d)->pdev_list)) )
+         !list_empty(&(domain_hvm_iommu(d)->pdev_list)) &&
+         !(mfn_valid(target_mfn) &&
+           is_xen_heap_frame(mfn_to_page(target_mfn))) )
     {
         unsigned int type;
         if ( hvm_get_mem_pinned_cacheattr(d, gfn_x(target_gfn), &type)
)
 

Thanks
Weidong

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel