Hi maintainer, Reprodue an bug on xen-unstable, it''s an irq affinity issue for passthroughed msix device to uek1 pvhvm(2.6.32 stable). I passthrough two mptsas devices(0000:0d:0.0, 0000:1f:0.0) to a pvhvm, irq affinity can''t be changed. Step to reproduce is as below: 1. xl -f pci-assignable-add 0000:0d:0.0; xl -f pci-assignable-add 0000:1f:0.0 2.xm cr -c vm.cfg [root@rac10box2 ~]# cat /proc/interrupts |grep mpt2sas0 48: 340449 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge mpt2sas0-msix0 [root@rac10box2 ~]# cat /proc/irq/48/smp_affinity 0fff [root@rac10box2 ~]# echo 2>/proc/irq/48/smp_affinity [root@rac10box2 ~]# cat /proc/irq/48/smp_affinity 0002 [root@rac10box2 ~]# cat /proc/interrupts |grep mpt 48: 342051 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge mpt2sas0-msix0 After change affinity to vcpu1, interrupt still happen on vcpu0. And sometimes we got "/No irq handler for vector (irq -1)/" and panic. qemu-dm.log shows error: pt_msixctrl_reg_write: guest enabling MSI-X, disable MSI-INTx translation pci_intx: intx=1 pt_msi_disable: Unmap msi with pirq 4f pt_msix_update_one: Update msix entry 0 with pirq 4f gvec 59 pt_msixctrl_reg_write: guest enabling MSI-X, disable MSI-INTx translation pci_intx: intx=1 pt_msi_disable: Unmap msi with pirq 4e pt_msix_update_one: Update msix entry 0 with pirq 4e gvec 69 pci_msix_writel: Can''t update entry 0 since MSI-X is already enabled (fee00000 -> fee02000) pci_msix_writel: Can''t update entry 0 since MSI-X is already enabled (00004059 -> 00004071) Can''t reproduce with uek2(3.1 stable), but if I disable hvm_pirqs support in uek2, reproduce the same. Test patch below: diff --git a/arch/x86/pci/xen.c b/arch/x86/pci/xen.c index a22ad4b..2d795d1 100644 --- a/arch/x86/pci/xen.c +++ b/arch/x86/pci/xen.c @-400,7 +400,6 @int __init pci_xen_init(void) . int __init pci_xen_hvm_init(void) { - if (!xen_have_vector_callback || !xen_feature(XENFEAT_hvm_pirqs)) return 0; . #ifdef CONFIG_ACPI diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c index bf28d69..64f6c6c 100644 --- a/arch/x86/xen/enlighten.c +++ b/arch/x86/xen/enlighten.c @-1536,8 +1536,6 @bool xen_hvm_need_lapic(void) return false; if (!xen_hvm_domain()) return false; - if (xen_feature(XENFEAT_hvm_pirqs) && xen_have_vector_callback) - return false; return true; } EXPORT_SYMBOL_GPL(xen_hvm_need_lapic); btw: can''t reproduce too if disable msi support. I''ll upload what you need, just tell me. thanks zduan
>>> On 26.02.13 at 09:56, DuanZhenzhong <zhenzhong.duan@oracle.com> wrote: > pci_msix_writel: Can''t update entry 0 since MSI-X is already enabled > (fee00000 -> fee02000) > pci_msix_writel: Can''t update entry 0 since MSI-X is already enabled > (00004059 -> 00004071)If you look at the code issuing this message, the situation is pretty clear (and I think it as described already in the past, albeit I have no link at hand): qemu lacks proper emulation of the mask bit. pci_msix_write() looks at the physical one, yet when the guest sets the virtual mask bit, nothing is being done at all to make the hypervisor also mask the physical entry: if (offset == PCI_MSIX_ENTRY_VECTOR_CTRL) { if (msix->enabled && !(val & PCI_MSIX_ENTRY_CTRL_MASKBIT)) { xen_pt_msix_update_one(s, entry_nr); } } There''s probably quite a bit of code to be written to make this work. Jan
On 2013-02-26 18:08, Jan Beulich wrote:>>>> On 26.02.13 at 09:56, DuanZhenzhong <zhenzhong.duan@oracle.com> wrote: >> pci_msix_writel: Can''t update entry 0 since MSI-X is already enabled >> (fee00000 -> fee02000) >> pci_msix_writel: Can''t update entry 0 since MSI-X is already enabled >> (00004059 -> 00004071) > If you look at the code issuing this message, the situation is > pretty clear (and I think it as described already in the past, > albeit I have no link at hand): qemu lacks proper emulation of > the mask bit. pci_msix_write() looks at the physical one, yet > when the guest sets the virtual mask bit, nothing is being > done at all to make the hypervisor also mask the physical > entry: > > if (offset == PCI_MSIX_ENTRY_VECTOR_CTRL) { > if (msix->enabled && !(val & PCI_MSIX_ENTRY_CTRL_MASKBIT)) { > xen_pt_msix_update_one(s, entry_nr); > } > } > > There''s probably quite a bit of code to be written to make this > work.Is there plan of fixing it? thanks zduan
>>> On 27.02.13 at 06:49, Zhenzhong Duan <zhenzhong.duan@oracle.com> wrote:> On 2013-02-26 18:08, Jan Beulich wrote: >>>>> On 26.02.13 at 09:56, DuanZhenzhong <zhenzhong.duan@oracle.com> wrote: >>> pci_msix_writel: Can''t update entry 0 since MSI-X is already enabled >>> (fee00000 -> fee02000) >>> pci_msix_writel: Can''t update entry 0 since MSI-X is already enabled >>> (00004059 -> 00004071) >> If you look at the code issuing this message, the situation is >> pretty clear (and I think it as described already in the past, >> albeit I have no link at hand): qemu lacks proper emulation of >> the mask bit. pci_msix_write() looks at the physical one, yet >> when the guest sets the virtual mask bit, nothing is being >> done at all to make the hypervisor also mask the physical >> entry: >> >> if (offset == PCI_MSIX_ENTRY_VECTOR_CTRL) { >> if (msix->enabled && !(val & PCI_MSIX_ENTRY_CTRL_MASKBIT)) { >> xen_pt_msix_update_one(s, entry_nr); >> } >> } >> >> There''s probably quite a bit of code to be written to make this >> work. > Is there plan of fixing it?I''m not aware of anyone working on this, or having planned to. Want to take a shot? Jan
On 2013-02-27 16:36, Jan Beulich wrote:>>>> On 27.02.13 at 06:49, Zhenzhong Duan <zhenzhong.duan@oracle.com> wrote: >> On 2013-02-26 18:08, Jan Beulich wrote: >>>>>> On 26.02.13 at 09:56, DuanZhenzhong <zhenzhong.duan@oracle.com> wrote: >>>> pci_msix_writel: Can''t update entry 0 since MSI-X is already enabled >>>> (fee00000 -> fee02000) >>>> pci_msix_writel: Can''t update entry 0 since MSI-X is already enabled >>>> (00004059 -> 00004071) >>> If you look at the code issuing this message, the situation is >>> pretty clear (and I think it as described already in the past, >>> albeit I have no link at hand): qemu lacks proper emulation of >>> the mask bit. pci_msix_write() looks at the physical one, yet >>> when the guest sets the virtual mask bit, nothing is being >>> done at all to make the hypervisor also mask the physical >>> entry: >>> >>> if (offset == PCI_MSIX_ENTRY_VECTOR_CTRL) { >>> if (msix->enabled && !(val & PCI_MSIX_ENTRY_CTRL_MASKBIT)) { >>> xen_pt_msix_update_one(s, entry_nr); >>> } >>> } >>> >>> There''s probably quite a bit of code to be written to make this >>> work. >> Is there plan of fixing it? > I''m not aware of anyone working on this, or having planned to. > Want to take a shot?I''m looking at it. zduan