I really need to have the ability to tie event channel port''s to interrupts for my gplpv drivers under Windows. Is anyone working on anything like this? Does MSI allow more than one interrupt per PCI device? Thanks James _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 26/12/2008 03:29, "James Harper" <james.harper@bendigoit.com.au> wrote:> I really need to have the ability to tie event channel port''s to > interrupts for my gplpv drivers under Windows. Is anyone working on > anything like this? Does MSI allow more than one interrupt per PCI > device?What kind of interface do you want? -- Keir> Thanks > > James > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> On 26/12/2008 03:29, "James Harper" <james.harper@bendigoit.com.au>wrote:> > > I really need to have the ability to tie event channel port''s to > > interrupts for my gplpv drivers under Windows. Is anyone working on > > anything like this? Does MSI allow more than one interrupt per PCI > > device? > > What kind of interface do you want? >I''m not entirely sure. Windows makes it hard. Essentially, I need to be able to tie an event channel to a PIC or APIC interrupt, and have a way of checking the event channel to see if it was responsible for the interrupt and then clearing the pending flag for only that event channel. I don''t think that in itself is that hard - what will be hard is making Windows happy to have those interrupt lines as resources and to program the APIC correctly to allow them to be delivered. The above probably shows a lack of understanding on my part about how events are delivered into hvm domains as interrupts - can you point me to what files I need to look at to understand this better? Then maybe I''ll have a better idea about what I''m asking for (or I''ll understand why it won''t work :) Thanks James _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 26/12/2008 10:15, "James Harper" <james.harper@bendigoit.com.au> wrote:> The above probably shows a lack of understanding on my part about how > events are delivered into hvm domains as interrupts - can you point me > to what files I need to look at to understand this better? Then maybe > I''ll have a better idea about what I''m asking for (or I''ll understand > why it won''t work :)Currently all event channels are tied together as a single interrupt line, which can be configured to appear to come from the xen-platform fake PCI device. We could duplicate that device so you could have a PCI INTx line per CPU. Or duplicate it so that you could have a PCI INTx line per event channel (or at least barber-pole onto such a set of INTx lines). Or we could bypass the INTx emulation entirely, and deliver per-cpu event_pending status to that HVM VCPU''s local APIC, on a pre-registered interrupt vector (this is most like MSI, except the interrupt wouldn''t be coming from a PCI device, although we could perhaps even fake that out too). There''s quite a few options. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Or we could bypass the INTx emulation entirely, and deliver per-cpu > event_pending status to that HVM VCPU''s local APIC, on apre-registered> interrupt vector (this is most like MSI, except the interrupt wouldn''tbe> coming from a PCI device, although we could perhaps even fake that out > too). > > There''s quite a few options. >How many interrupts do we have to choose from here? I was able to get Windows to use up to (I think) IRQ31. As I understand it, most of the ''protocol'' we are talking about is based around the shared_info_t structure, which appears to make the assumption that there is a single point of entry for event delivery into a domain. To make use of multiple IRQ''s we''d have to check every single event bit right? Is that a performance problem. The reason all of this comes up btw, is that Windows is a PITA. What I do in my PV drivers is attach each device (xen platform pci driver, vbd driver, net driver, scsi driver, etc) to the same interrupt. The platform driver attaches first and gets called first. It clears the pending flag etc in the normal way, but where the event channel is registered as wanting delivery via an ISR, it sets a corresponding event channel bit in a private area, and then tells Windows that the interrupt was not actually for that device afterall, which prompts Windows to call the next ISR in the chain. That ISR (eg the vbd driver) check''s its bit in the private area and does it''s ISR code if the bit is set, and then tells Windows that the interrupt was not for that device, and so on. When Windows is using one of the ACPI HAL''s all this works fine. However when Windows is using the ''Standard PC'' HAL, I stop seeing interrupts for the network device very soon after boot. I think that because of all this ''interrupt not for me''-ness going on, Windows thinks that spurious interrupts are occurring and stops calling the network ISR except for every 3-20 interrupts. I can work around it, but that kind of behaviour makes me think that maybe I''m going around things the wrong way... For the network device I have gone back to using a direct callback, but if I ever want to get these drivers WHQL certified (and I do eventually), that sort of thing won''t fly without a whole lot of explaining. And for storage devices using the scsiport framework, an interrupt is the only way to do it. The storport framework is a possibility but currently it seems too broken and isn''t available under XP. Thanks James _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 26/12/2008 10:54, "James Harper" <james.harper@bendigoit.com.au> wrote:> How many interrupts do we have to choose from here? I was able to get > Windows to use up to (I think) IRQ31.If going via the virtual PIC, then there are as many interrupts as there are non-legacy IO-APIC pins. I think currently we have 32 of them.> As I understand it, most of the ''protocol'' we are talking about is based > around the shared_info_t structure, which appears to make the assumption > that there is a single point of entry for event delivery into a domain. > To make use of multiple IRQ''s we''d have to check every single event bit > right? Is that a performance problem.No, currently we call into HVM interrupt emulation code to assert an IO-APIC pin when CPU0''s event_pending flag is asserted. We could do that also when other CPU''s event_pending flags become asserted, or when individual event channels become asserted. We just have to hook into event-channel code in a different place. Or indeed we can just send ''messages'' to HVM virtual local APICs to trigger interrupts on the virtual CPUs directly, without any integration with the virtual PCI or PIC subsystems. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> On 26/12/2008 10:54, "James Harper" <james.harper@bendigoit.com.au>wrote:> > > How many interrupts do we have to choose from here? I was able toget> > Windows to use up to (I think) IRQ31. > > If going via the virtual PIC, then there are as many interrupts asthere> are > non-legacy IO-APIC pins. I think currently we have 32 of them. > > > As I understand it, most of the ''protocol'' we are talking about isbased> > around the shared_info_t structure, which appears to make theassumption> > that there is a single point of entry for event delivery into adomain.> > To make use of multiple IRQ''s we''d have to check every single eventbit> > right? Is that a performance problem. > > No, currently we call into HVM interrupt emulation code to assert anIO-> APIC > pin when CPU0''s event_pending flag is asserted. We could do that alsowhen> other CPU''s event_pending flags become asserted, or when individualevent> channels become asserted. We just have to hook into event-channel codein> a > different place. Or indeed we can just send ''messages'' to HVM virtual > local > APICs to trigger interrupts on the virtual CPUs directly, without any > integration with the virtual PCI or PIC subsystems. >What about when we are running on a non-APIC HAL? People keep telling me that the ''Standard PC'' HAL (all IRQ''s < 16) under Windows 2003 is higher performance under Xen, but I''ve only done a quick test via iperf and maybe there is 10% higher throughput with ''Standard PC'', but that could easily be within the level of background noise on that machine. The original problem I am having is that Windows (NDIS actually) doesn''t always call my ISR when an interrupt is shared if it suspects that the interrupts might be spurious. To work around that I would need for the interrupt to be triggered if the ''pending'' flag for that event wasn''t cleared, which is the ''check each event bit'' that I was talking about. This would also happen if two events are bound to the same IRQ and are both triggered at the same time, eg: . Event channels 2, 5, and 8 are bound to IRQ 29 . Event channels 2 and 8 are both triggered simultaneously (or close enough that by the time the ISR happens they are both set) . Windows will call the ISR for the device using event channel 2 . The device will see that its event is responsible for the interrupt, clear the ''pending'' flag _for that port only_ and then schedule a DPC to do the work. It will return TRUE from its ISR and windows will not call any other ISR for that interrupt . Xen/qemu will need to notice that there is still an outstanding interrupt for that event and will trigger the ISRs again . ISR for the device using event channel 2 will get called again, but will return FALSE this time meaning ''interrupt was not for me'' . ditto for the device using event channel 5 . Finally, the ISR for the device using event channel 8 will be called and will do its thing So instead of checking for one ''pending'' flag, I think the pending flags for each port will need to be called. If that''s not a problem for performance (only need to do it when returning from the interrupt I think...) then I think we can proceed... I can think of the following pieces of the puzzle: . A mechanism to assign an event channel to an IRQ - a new type of parameter for HYPERVISOR_hvm_op? . At the end of each interrupt, re-trigger the interrupt if the event bits are set rather than if the ''evtchn_upcall_pending'' flag is set (where is this done currently? I couldn''t find it) . Some way to make this backwards compatible... maybe assume the old behaviour until the new HYPERVISOR_hvm_op parameter is set? What else would be required? Thanks James _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 27/12/2008 09:19, "James Harper" <james.harper@bendigoit.com.au> wrote:> I can think of the following pieces of the puzzle: > . A mechanism to assign an event channel to an IRQ - a new type of > parameter for HYPERVISOR_hvm_op?That''s part of the problem. What do you mean by ''IRQ'' here? What''s the IRQ namespace that you''re indexing into? -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> On 27/12/2008 09:19, "James Harper" <james.harper@bendigoit.com.au>wrote:> > > I can think of the following pieces of the puzzle: > > . A mechanism to assign an event channel to an IRQ - a new type of > > parameter for HYPERVISOR_hvm_op? > > That''s part of the problem. What do you mean by ''IRQ'' here? What''s theIRQ> namespace that you''re indexing into? >I''m not sure if I understand the question. Under my drivers I just do: hvm_set_parameter(HVM_PARAM_CALLBACK_IRQ, irq_number); where irq_number is what Windows assigns the platform PCI device, and is normally 28 when the HAL is APIC, otherwise it is 5, assuming no qemu network adapters or passed through PCI devices. I think there is another way of setting the IRQ using the PCI address, but that doesn''t appear to be required here. James _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 27/12/2008 09:35, "James Harper" <james.harper@bendigoit.com.au> wrote:> I''m not sure if I understand the question. Under my drivers I just do: > > hvm_set_parameter(HVM_PARAM_CALLBACK_IRQ, irq_number); > > where irq_number is what Windows assigns the platform PCI device, and is > normally 28 when the HAL is APIC, otherwise it is 5, assuming no qemu > network adapters or passed through PCI devices. > > I think there is another way of setting the IRQ using the PCI address, > but that doesn''t appear to be required here.So where will irq_number come from when every event channel needs one? -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> On 27/12/2008 09:35, "James Harper" <james.harper@bendigoit.com.au>wrote:> > > I''m not sure if I understand the question. Under my drivers I justdo:> > > > hvm_set_parameter(HVM_PARAM_CALLBACK_IRQ, irq_number); > > > > where irq_number is what Windows assigns the platform PCI device,and is> > normally 28 when the HAL is APIC, otherwise it is 5, assuming noqemu> > network adapters or passed through PCI devices. > > > > I think there is another way of setting the IRQ using the PCIaddress,> > but that doesn''t appear to be required here. > > So where will irq_number come from when every event channel needs one? >The driver for the xen platform PCI device is a ''bus driver'' under windows, and enumerates child devices. When it enumerates a child device, I can say ''and allocate me an interrupt line''. During past experimentation I did this with the criteria ''allocate me an interrupt line somewhere between 16 and 31'', and then I would call that interrupt via a software interrupt mechanism (asm {int 0xXX} ), so I know that the allocation and setup side of things work. What I don''t know is if Windows is programming the APIC correctly to match that physical IRQ number with the vector... the ''int 0xXX'' instruction just calls the vector. With enough devices, event channels are going to end up having to share interrupts, but the number of event channels would be around 15 before this would be required I think (windows would never allocate me an IRQ > 31), which I don''t think is an unreasonable limitation. James _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 27/12/2008 10:03, "James Harper" <james.harper@bendigoit.com.au> wrote:> The driver for the xen platform PCI device is a ''bus driver'' under > windows, and enumerates child devices. When it enumerates a child > device, I can say ''and allocate me an interrupt line''.So these child devices don''t have to have a physical manifestation in PCI space? And you can really request an arbitrary IRQ and then you are expected to plumb it through? That sounds weird, but actually quite helpful for us. Probably we''d implement it with an hvm_op to associate an event channel with an IO-APIC pin or a local APIC vector. If implemented as wire-OR into a set of IO-APIC pins, we''d need logic to deassert wires when event channels become not-pending, before the wire gets resampled by the PIC/IO-APIC. It''s all easier if we can directly deliver to a LAPIC vector because those are inherently edge-triggered / message-based (which I think is what we really want; although it''s more complicated if we need to be able to share a LAPIC vector among several event channels without ''losing'' edges). -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> On 27/12/2008 10:03, "James Harper" <james.harper@bendigoit.com.au>wrote:> > > The driver for the xen platform PCI device is a ''bus driver'' under > > windows, and enumerates child devices. When it enumerates a child > > device, I can say ''and allocate me an interrupt line''. > > So these child devices don''t have to have a physical manifestation inPCI> space? And you can really request an arbitrary IRQ and then you are > expected > to plumb it through? That sounds weird, but actually quite helpful forus.> > Probably we''d implement it with an hvm_op to associate an eventchannel> with > an IO-APIC pin or a local APIC vector. If implemented as wire-OR intoa> set > of IO-APIC pins, we''d need logic to deassert wires when event channels > become not-pending, before the wire gets resampled by the PIC/IO-APIC. > It''s > all easier if we can directly deliver to a LAPIC vector because thoseare> inherently edge-triggered / message-based (which I think is what wereally> want; although it''s more complicated if we need to be able to share a > LAPIC vector among several event channels without ''losing'' edges). >Well... the ''old'' way would probably still have to work (or would it?), so we could just keep allocating IRQ''s until we run out and any leftover devices just have to use the old way. I''ve mentioned the possibility of using MSI before... would that work? I''m not yet sure if they are supported across all windows versions, but we get lots more ''interrupt channels''... James _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 27/12/2008 10:28, "James Harper" <james.harper@bendigoit.com.au> wrote:> Well... the ''old'' way would probably still have to work (or would it?), > so we could just keep allocating IRQ''s until we run out and any leftover > devices just have to use the old way.Yes, that did occur to me. Might be a nice fallback while still allowing up to 16 or whatever devices to have their interrupts distributed across VCPUs. The old mechanism does still need to work, so making it a fallback in this new mechanism would be probably not too difficult.> I''ve mentioned the possibility of using MSI before... would that work? > I''m not yet sure if they are supported across all windows versions, but > we get lots more ''interrupt channels''...Well, would Windows need to see more fake PCI devices (where these MSIs would emanate from) for this to work? It would be nice, though perhaps not essential, to avoid this since it needs backwards-compatible changes to qemu-dm, and also possibly our vBIOS. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> On 27/12/2008 10:28, "James Harper" <james.harper@bendigoit.com.au>wrote:> > > Well... the ''old'' way would probably still have to work (or wouldit?),> > so we could just keep allocating IRQ''s until we run out and anyleftover> > devices just have to use the old way. > > Yes, that did occur to me. Might be a nice fallback while stillallowing> up to 16 or whatever devices to have their interrupts distributedacross> VCPUs. > The old mechanism does still need to work, so making it a fallback inthis> new mechanism would be probably not too difficult.Is Windows limited to 31 IRQ''s? Linux appears to allow much more than that, but maybe I''m not comparing apples with apples here. Still... an average Windows system under Xen would have a few disks, a few network adapters, and maybe a scsi passthrough. I think it would be unusual to see a requirement for more than 6 IRQ''s, so I don''t think that a limit of 16 is going to cripple anyone. So are you suggesting that these interrupts be non-shareable? Would that remove the need to check for a pending interrupt status in Xen, and for the virtual device driver to clear any flags?> > > I''ve mentioned the possibility of using MSI before... would thatwork?> > I''m not yet sure if they are supported across all windows versions,but> > we get lots more ''interrupt channels''... > > Well, would Windows need to see more fake PCI devices (where theseMSIs> would emanate from) for this to work? It would be nice, though perhapsnot> essential, to avoid this since it needs backwards-compatible changesto> qemu-dm, and also possibly our vBIOS. >Unfortunately I don''t know enough about MSI and how Windows handles MSI to be able to answer that. Lets give it a miss :) James _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> On 27/12/2008 10:28, "James Harper" <james.harper@bendigoit.com.au>wrote:> > > Well... the ''old'' way would probably still have to work (or wouldit?),> > so we could just keep allocating IRQ''s until we run out and anyleftover> > devices just have to use the old way. > > Yes, that did occur to me. Might be a nice fallback while stillallowing> up > to 16 or whatever devices to have their interrupts distributed across > VCPUs. > The old mechanism does still need to work, so making it a fallback inthis> new mechanism would be probably not too difficult.I''ve been reading up on APIC''s, and it appears that (depending on the model) they can support up to 64 interrupts. Is the emulated APIC under Xen of the 16 interrupt variety? James _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 28/12/2008 03:22, "James Harper" <james.harper@bendigoit.com.au> wrote:>> Yes, that did occur to me. Might be a nice fallback while still > allowing >> up >> to 16 or whatever devices to have their interrupts distributed across >> VCPUs. >> The old mechanism does still need to work, so making it a fallback in > this >> new mechanism would be probably not too difficult. > > I''ve been reading up on APIC''s, and it appears that (depending on the > model) they can support up to 64 interrupts. Is the emulated APIC under > Xen of the 16 interrupt variety?The IO-APIC has 48 pins, 16 of which are mapped for legacy ISA, so there are 32 non-legacy pins. Actually we could have more, but that creates backward-compatibility issues now. 32 should be enough for most uses! -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> I''m not yet sure if they are supported across all windows versionsI believe MSI is supported on Vista and later for workstation and server 2003 and later. -----Original Message----- From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of James Harper Sent: Saturday, December 27, 2008 5:28 AM To: Keir Fraser; xen-devel@lists.xensource.com Subject: RE: [Xen-devel] Multiple IRQ''s in HVM for Windows> On 27/12/2008 10:03, "James Harper" <james.harper@bendigoit.com.au>wrote:> > > The driver for the xen platform PCI device is a ''bus driver'' under > > windows, and enumerates child devices. When it enumerates a child > > device, I can say ''and allocate me an interrupt line''. > > So these child devices don''t have to have a physical manifestation inPCI> space? And you can really request an arbitrary IRQ and then you are > expected > to plumb it through? That sounds weird, but actually quite helpful forus.> > Probably we''d implement it with an hvm_op to associate an eventchannel> with > an IO-APIC pin or a local APIC vector. If implemented as wire-OR intoa> set > of IO-APIC pins, we''d need logic to deassert wires when event channels > become not-pending, before the wire gets resampled by the PIC/IO-APIC. > It''s > all easier if we can directly deliver to a LAPIC vector because thoseare> inherently edge-triggered / message-based (which I think is what wereally> want; although it''s more complicated if we need to be able to share a > LAPIC vector among several event channels without ''losing'' edges). >Well... the ''old'' way would probably still have to work (or would it?), so we could just keep allocating IRQ''s until we run out and any leftover devices just have to use the old way. I''ve mentioned the possibility of using MSI before... would that work? I''m not yet sure if they are supported across all windows versions, but we get lots more ''interrupt channels''... James _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel