Hi maintainer,
Reprodue an bug on xen-unstable, it''s an irq affinity issue for
passthroughed msix device to uek1 pvhvm(2.6.32 stable).
I passthrough two mptsas devices(0000:0d:0.0, 0000:1f:0.0) to a pvhvm,
irq affinity can''t be changed.
Step to reproduce is as below:
1. xl -f pci-assignable-add 0000:0d:0.0; xl -f pci-assignable-add
0000:1f:0.0
2.xm cr -c vm.cfg
[root@rac10box2 ~]# cat /proc/interrupts |grep mpt2sas0
48: 340449 0 0 0 0 0
0 0 0 0 0 0 PCI-MSI-edge
mpt2sas0-msix0
[root@rac10box2 ~]# cat /proc/irq/48/smp_affinity
0fff
[root@rac10box2 ~]# echo 2>/proc/irq/48/smp_affinity
[root@rac10box2 ~]# cat /proc/irq/48/smp_affinity
0002
[root@rac10box2 ~]# cat /proc/interrupts |grep mpt
48: 342051 0 0 0 0 0
0 0 0 0 0 0 PCI-MSI-edge
mpt2sas0-msix0
After change affinity to vcpu1, interrupt still happen on vcpu0. And
sometimes we got "/No irq handler for vector (irq -1)/" and panic.
qemu-dm.log shows error:
pt_msixctrl_reg_write: guest enabling MSI-X, disable MSI-INTx translation
pci_intx: intx=1
pt_msi_disable: Unmap msi with pirq 4f
pt_msix_update_one: Update msix entry 0 with pirq 4f gvec 59
pt_msixctrl_reg_write: guest enabling MSI-X, disable MSI-INTx translation
pci_intx: intx=1
pt_msi_disable: Unmap msi with pirq 4e
pt_msix_update_one: Update msix entry 0 with pirq 4e gvec 69
pci_msix_writel: Can''t update entry 0 since MSI-X is already enabled
(fee00000 -> fee02000)
pci_msix_writel: Can''t update entry 0 since MSI-X is already enabled
(00004059 -> 00004071)
Can''t reproduce with uek2(3.1 stable), but if I disable hvm_pirqs
support in uek2, reproduce the same.
Test patch below:
diff --git a/arch/x86/pci/xen.c b/arch/x86/pci/xen.c
index a22ad4b..2d795d1 100644
--- a/arch/x86/pci/xen.c
+++ b/arch/x86/pci/xen.c
@-400,7 +400,6 @int __init pci_xen_init(void)
.
int __init pci_xen_hvm_init(void)
{
- if (!xen_have_vector_callback || !xen_feature(XENFEAT_hvm_pirqs))
return 0;
.
#ifdef CONFIG_ACPI
diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
index bf28d69..64f6c6c 100644
--- a/arch/x86/xen/enlighten.c
+++ b/arch/x86/xen/enlighten.c
@-1536,8 +1536,6 @bool xen_hvm_need_lapic(void)
return false;
if (!xen_hvm_domain())
return false;
- if (xen_feature(XENFEAT_hvm_pirqs) && xen_have_vector_callback)
- return false;
return true;
}
EXPORT_SYMBOL_GPL(xen_hvm_need_lapic);
btw: can''t reproduce too if disable msi support.
I''ll upload what you need, just tell me.
thanks
zduan
>>> On 26.02.13 at 09:56, DuanZhenzhong <zhenzhong.duan@oracle.com> wrote: > pci_msix_writel: Can''t update entry 0 since MSI-X is already enabled > (fee00000 -> fee02000) > pci_msix_writel: Can''t update entry 0 since MSI-X is already enabled > (00004059 -> 00004071)If you look at the code issuing this message, the situation is pretty clear (and I think it as described already in the past, albeit I have no link at hand): qemu lacks proper emulation of the mask bit. pci_msix_write() looks at the physical one, yet when the guest sets the virtual mask bit, nothing is being done at all to make the hypervisor also mask the physical entry: if (offset == PCI_MSIX_ENTRY_VECTOR_CTRL) { if (msix->enabled && !(val & PCI_MSIX_ENTRY_CTRL_MASKBIT)) { xen_pt_msix_update_one(s, entry_nr); } } There''s probably quite a bit of code to be written to make this work. Jan
On 2013-02-26 18:08, Jan Beulich wrote:>>>> On 26.02.13 at 09:56, DuanZhenzhong <zhenzhong.duan@oracle.com> wrote: >> pci_msix_writel: Can''t update entry 0 since MSI-X is already enabled >> (fee00000 -> fee02000) >> pci_msix_writel: Can''t update entry 0 since MSI-X is already enabled >> (00004059 -> 00004071) > If you look at the code issuing this message, the situation is > pretty clear (and I think it as described already in the past, > albeit I have no link at hand): qemu lacks proper emulation of > the mask bit. pci_msix_write() looks at the physical one, yet > when the guest sets the virtual mask bit, nothing is being > done at all to make the hypervisor also mask the physical > entry: > > if (offset == PCI_MSIX_ENTRY_VECTOR_CTRL) { > if (msix->enabled && !(val & PCI_MSIX_ENTRY_CTRL_MASKBIT)) { > xen_pt_msix_update_one(s, entry_nr); > } > } > > There''s probably quite a bit of code to be written to make this > work.Is there plan of fixing it? thanks zduan
>>> On 27.02.13 at 06:49, Zhenzhong Duan <zhenzhong.duan@oracle.com> wrote:> On 2013-02-26 18:08, Jan Beulich wrote: >>>>> On 26.02.13 at 09:56, DuanZhenzhong <zhenzhong.duan@oracle.com> wrote: >>> pci_msix_writel: Can''t update entry 0 since MSI-X is already enabled >>> (fee00000 -> fee02000) >>> pci_msix_writel: Can''t update entry 0 since MSI-X is already enabled >>> (00004059 -> 00004071) >> If you look at the code issuing this message, the situation is >> pretty clear (and I think it as described already in the past, >> albeit I have no link at hand): qemu lacks proper emulation of >> the mask bit. pci_msix_write() looks at the physical one, yet >> when the guest sets the virtual mask bit, nothing is being >> done at all to make the hypervisor also mask the physical >> entry: >> >> if (offset == PCI_MSIX_ENTRY_VECTOR_CTRL) { >> if (msix->enabled && !(val & PCI_MSIX_ENTRY_CTRL_MASKBIT)) { >> xen_pt_msix_update_one(s, entry_nr); >> } >> } >> >> There''s probably quite a bit of code to be written to make this >> work. > Is there plan of fixing it?I''m not aware of anyone working on this, or having planned to. Want to take a shot? Jan
On 2013-02-27 16:36, Jan Beulich wrote:>>>> On 27.02.13 at 06:49, Zhenzhong Duan <zhenzhong.duan@oracle.com> wrote: >> On 2013-02-26 18:08, Jan Beulich wrote: >>>>>> On 26.02.13 at 09:56, DuanZhenzhong <zhenzhong.duan@oracle.com> wrote: >>>> pci_msix_writel: Can''t update entry 0 since MSI-X is already enabled >>>> (fee00000 -> fee02000) >>>> pci_msix_writel: Can''t update entry 0 since MSI-X is already enabled >>>> (00004059 -> 00004071) >>> If you look at the code issuing this message, the situation is >>> pretty clear (and I think it as described already in the past, >>> albeit I have no link at hand): qemu lacks proper emulation of >>> the mask bit. pci_msix_write() looks at the physical one, yet >>> when the guest sets the virtual mask bit, nothing is being >>> done at all to make the hypervisor also mask the physical >>> entry: >>> >>> if (offset == PCI_MSIX_ENTRY_VECTOR_CTRL) { >>> if (msix->enabled && !(val & PCI_MSIX_ENTRY_CTRL_MASKBIT)) { >>> xen_pt_msix_update_one(s, entry_nr); >>> } >>> } >>> >>> There''s probably quite a bit of code to be written to make this >>> work. >> Is there plan of fixing it? > I''m not aware of anyone working on this, or having planned to. > Want to take a shot?I''m looking at it. zduan