Hi, I am trying to debug a problem where a guest does not receive the correct interrupt. It''s a passtrough system where the guest is given a virtual function and is using MSIX to recieve the interrupts. I am trying to understand how is the guest supposed to recieve the interrupt: Is it handled by dom0 which is using event channels to pass the interrupt to domU, or is handled directly by domU. Please send pointers to the code. Thanks in advance, eli; _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 04/10/11 17:29, Eli Cohen wrote:> Hi, > I am trying to debug a problem where a guest does not receive the > correct interrupt. It''s a passtrough system where the guest is given a > virtual function and is using MSIX to recieve the interrupts.Do you mean that it is receiving interrupts for another device, or are you trying to pass through multiple interrupts for a device and the interrupt is appearing in the wrong event channel? Could you describe your hardware setup in detail please, especially AMD/Intel, IOMMU features in use according to Xen, and the PCI topology of the box.> I am > trying to understand how is the guest supposed to recieve the > interrupt: Is it handled by dom0 which is using event channels to pass > the interrupt to domU, or is handled directly by domU. Please send > pointers to the code.(I am not very familiar with guest IRQs so the following might not be accurate) For a PCI passthrough system, the interrupts should be bound directly to domU, with no dom0 interaction. A PIRQ is bound to an event channel using the EVTCHNOP_bind_pirq in common/event_channel.c, which should be set up by qemu for the relevant domU, when it is interpreting the PCI config space reads/writes.> Thanks in advance, > eli; > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel-- Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer T: +44 (0)1223 225 900, http://www.citrix.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Andrew, thanks for your response. On Tue, Oct 04, 2011 at 05:50:17PM +0100, Andrew Cooper wrote:> > Do you mean that it is receiving interrupts for another device, or are > you trying to pass through multiple interrupts for a device and the > interrupt is appearing in the wrong event channel? >I am not sure I understand your question.> Could you describe your hardware setup in detail please, especially > AMD/Intel, IOMMU features in use according to Xen, and the PCI topology > of the box.Here''s a description. I have a PCI device which supports SRIOV, so there are several virtual functions besides the primary function. The device generates interrupts using MSI-X. I bind one of the virtual functions to the guest. The PCI device driver at the guest requests msix vectors from kernel and gets two vectors. Then it calls request_irq for those vectors. When the device generates msix interrupt using the address/data in it''s msi-x table. However, the guest kernel complains the no handler is registered for this vector. Specifically, I request_irq for vectors 48 and 49. When the device issues interrupt for vector 49, the guest complains that it received interrupt for vector 89 which has no handler routine.> > > I am > > trying to understand how is the guest supposed to recieve the > > interrupt: Is it handled by dom0 which is using event channels to pass > > the interrupt to domU, or is handled directly by domU. Please send > > pointers to the code. > > (I am not very familiar with guest IRQs so the following might not be > accurate) > > For a PCI passthrough system, the interrupts should be bound directly to > domU, with no dom0 interaction. A PIRQ is bound to an event channel > using the EVTCHNOP_bind_pirq in common/event_channel.c, which should be > set up by qemu for the relevant domU, when it is interpreting the PCI > config space reads/writes.My geust kernel is 2.6.32 based and I can''t see any file with the name event_channel.c. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>>> On 04.10.11 at 21:00, Eli Cohen <eli@dev.mellanox.co.il> wrote: > Here''s a description. I have a PCI device which supports SRIOV, so > there are several virtual functions besides the primary function. The > device generates interrupts using MSI-X. I bind one of the virtual > functions to the guest. The PCI device driver at the guest requests > msix vectors from kernel and gets two vectors. Then it calls > request_irq for those vectors. > When the device generates msix interrupt using the address/data in > it''s msi-x table. However, the guest kernel complains the no handler > is registered for this vector. Specifically, I request_irq for vectors > 48 and 49. When the device issues interrupt for vector 49, the guest > complains that it received interrupt for vector 89 which has no > handler routine.Quite possibly a mixup of Xen and Linux IRQ numbers. You may want to check Xen''s ''i'' and ''e'' debug key output for whether this might be the case.> My geust kernel is 2.6.32 basedUpstream (pv-ops), forward-port (most likely SUSE-derived), HVM, or yet something else?> and I can''t see any file with the name > event_channel.c.That''s a hypervisor source file, not a Linux one. Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, Oct 05, 2011 at 08:27:07AM +0100, Jan Beulich wrote:> > Quite possibly a mixup of Xen and Linux IRQ numbers. You may want > to check Xen''s ''i'' and ''e'' debug key output for whether this might be > the case.Can you tell how this is done - sorry, I''m not a xen expert.> > > My geust kernel is 2.6.32 based > > Upstream (pv-ops), forward-port (most likely SUSE-derived), HVM, or > yet something else? > > > and I can''t see any file with the name > > event_channel.c. > > That''s a hypervisor source file, not a Linux one. >So how do you think interrupts should reach the guest. Is it through event channels (i.e dom0 receives them first) or do they go directly to the guest with dom0 unaware that there was an interrupt? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>>> On 05.10.11 at 10:26, Eli Cohen <eli@dev.mellanox.co.il> wrote: > On Wed, Oct 05, 2011 at 08:27:07AM +0100, Jan Beulich wrote: >> >> Quite possibly a mixup of Xen and Linux IRQ numbers. You may want >> to check Xen''s ''i'' and ''e'' debug key output for whether this might be >> the case. > > Can you tell how this is done - sorry, I''m not a xen expert.Either by sending the characters from the serial console (after switching input to Xen), or by using "{xl,xm] debug-key" followed by "{xm,xl} dmesg" to view the output.>> >> > My geust kernel is 2.6.32 based >> >> Upstream (pv-ops), forward-port (most likely SUSE-derived), HVM, or >> yet something else? >> >> > and I can''t see any file with the name >> > event_channel.c. >> >> That''s a hypervisor source file, not a Linux one. >> > > So how do you think interrupts should reach the guest. Is it through > event channels (i.e dom0 receives them first) or do they go directly > to the guest with dom0 unaware that there was an interrupt?Dom0 shouldn''t get involved at all - Xen does the necessary conversion. Dom0 may only be involved in setting things up. Whether they get received via event channels or virtual interrupts depends on the kind of guest. Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, Oct 05, 2011 at 09:51:10AM +0100, Jan Beulich wrote:> > Either by sending the characters from the serial console (after switching > input to Xen), or by using "{xl,xm] debug-key" followed by > "{xm,xl} dmesg" to view the output. >Thanks.> > Dom0 shouldn''t get involved at all - Xen does the necessary conversion. > Dom0 may only be involved in setting things up. Whether they get > received via event channels or virtual interrupts depends on the kind > of guest. >I think I used the wrong terminology not distinguishing between the hypervisor and dom0 so I will ask again just to make sure. Is the hypervisor involved in passing the interrupts? Does it depend on whether the guest is paravirtualized or not? The guest I am using is paravirtualized (i.e uses xen hypercalls), so does it mean that it uses event channels or is there a possibilty that it does not use event channels. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>>> On 05.10.11 at 11:00, Eli Cohen <eli@dev.mellanox.co.il> wrote: > On Wed, Oct 05, 2011 at 09:51:10AM +0100, Jan Beulich wrote: >> Dom0 shouldn''t get involved at all - Xen does the necessary conversion. >> Dom0 may only be involved in setting things up. Whether they get >> received via event channels or virtual interrupts depends on the kind >> of guest. >> > I think I used the wrong terminology not distinguishing between the > hypervisor and dom0 so I will ask again just to make sure. Is the > hypervisor involved in passing the interrupts?Yes, of course. The hypervisor receives all physical interrupts.> Does it depend on whether the guest is paravirtualized or not?Yes, at least to some degree.> The guest I am using is > paravirtualized (i.e uses xen hypercalls), so does it mean that it > uses event channels or is there a possibilty that it does not use > event channels.Yes and no: PV guests can receive interrupts only via the event channel mechanism. Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Thanks for the information. On Wed, Oct 05, 2011 at 10:09:01AM +0100, Jan Beulich wrote:> >>> On 05.10.11 at 11:00, Eli Cohen <eli@dev.mellanox.co.il> wrote: > > On Wed, Oct 05, 2011 at 09:51:10AM +0100, Jan Beulich wrote: > >> Dom0 shouldn''t get involved at all - Xen does the necessary conversion. > >> Dom0 may only be involved in setting things up. Whether they get > >> received via event channels or virtual interrupts depends on the kind > >> of guest. > >> > > I think I used the wrong terminology not distinguishing between the > > hypervisor and dom0 so I will ask again just to make sure. Is the > > hypervisor involved in passing the interrupts? > > Yes, of course. The hypervisor receives all physical interrupts. > > > Does it depend on whether the guest is paravirtualized or not? > > Yes, at least to some degree. > > > The guest I am using is > > paravirtualized (i.e uses xen hypercalls), so does it mean that it > > uses event channels or is there a possibilty that it does not use > > event channels. > > Yes and no: PV guests can receive interrupts only via the event > channel mechanism. > > Jan_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, 2011-10-05 at 10:09 +0100, Jan Beulich wrote:> >>> On 05.10.11 at 11:00, Eli Cohen <eli@dev.mellanox.co.il> wrote: > > On Wed, Oct 05, 2011 at 09:51:10AM +0100, Jan Beulich wrote: > >> Dom0 shouldn''t get involved at all - Xen does the necessary conversion. > >> Dom0 may only be involved in setting things up. Whether they get > >> received via event channels or virtual interrupts depends on the kind > >> of guest. > >> > > I think I used the wrong terminology not distinguishing between the > > hypervisor and dom0 so I will ask again just to make sure. Is the > > hypervisor involved in passing the interrupts? > > Yes, of course. The hypervisor receives all physical interrupts. > > > Does it depend on whether the guest is paravirtualized or not? > > Yes, at least to some degree. > > > The guest I am using is > > paravirtualized (i.e uses xen hypercalls), so does it mean that it > > uses event channels or is there a possibilty that it does not use > > event channels. > > Yes and no: PV guests can receive interrupts only via the event > channel mechanism.While an HVM guest can receive interrupts via any of: * Emulated interrupt injection (i.e. from the emulated APIC) * Event channels, injected via the INTx interrupt on the PCI platform device (itself injected by the previous option) * Event channels, injected directly to a vector. The first is available to any HVM guest, the middle one is the older style PV Driver for HVM guest mechanism while the third is an additional feature of more recent Linux kernels using the PVHVM mechanism. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel