Hi, on xen-ia64-devel, we are currently discussing on IRQ delivery. Xen/x86 delivers IRQs using the event channel. Xen/ia64 currently delivers IRQs using the virtualized hardware way. Is there major drawbacks about not using event channel (from a XEN POV) ? How xen/ppc delivers interrupts ? Thanks, Tristan. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Friday 10 March 2006 04:07, Tristan Gingold wrote:> > How xen/ppc delivers interrupts ?Ignoring hypervisors for a moment, the PPC model for external interrupts is this: the hardware resets the PC to a fixed entry point (0x500) and changes the MSR to put the processor into real (untranslated) mode. (The old PC and MSR are saved into a pair of supervisor-only registers, SRR0 and SRR1.) The exception handler is then responsible for querying the interrupt controller(s) to get the interrupt vector, and then call the appropriate driver. The IBM Research hypervisor (rhype) virtualized the PIC, so that instead of querying hardware, the kernel makes an hcall to find the incoming vector. If the interrupt is not in fact for this domain, the hypervisor queues the interrupt for later and tells the current domain "nothing was pending after all". Note that in this model, interrupts are delivered by the hardware directly to the current domain without hypervisor involvement. With Xen, the hypervisor simulates hardware delivering an interrupt (jumping to 0x500), and then the kernel looks at the event channel structures (instead of making an hcall) to figure out what happened. Important note: in our current implementation, interrupts are always delivered by hardware to the hypervisor, and then we simulate the interrupt delivery to the domain. Because of the increased latency, it''s possible that in the future we''ll want to adopt a more rhypish model. However, since the hypervisor itself takes interrupts, we need to be able to simulate domain interrupt delivery anyways.> on xen-ia64-devel, we are currently discussing on IRQ delivery. > Xen/x86 delivers IRQs using the event channel. > Xen/ia64 currently delivers IRQs using the virtualized hardware way. > > Is there major drawbacks about not using event channel (from a XEN POV) ?I''m not sure what "the virtualized hardware way" is. You probably need to virtualize the PIC *somehow*, and the Xen event channels are as good a way as any to do that. If IA64 already has an unrelated way of virtualizing the PIC (as PPC did with the rhype model), then you get to choose. If that''s the case, I would recommend adopting the event channels to share more Xen code and design... -- Hollis Blanchard IBM Linux Technology Center _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Le Vendredi 10 Mars 2006 16:30, Hollis Blanchard a écrit :> On Friday 10 March 2006 04:07, Tristan Gingold wrote: > > How xen/ppc delivers interrupts ?[...] Thank you for the vast amount of details.> > on xen-ia64-devel, we are currently discussing on IRQ delivery. > > Xen/x86 delivers IRQs using the event channel. > > Xen/ia64 currently delivers IRQs using the virtualized hardware way. > > > > Is there major drawbacks about not using event channel (from a XEN POV) ? > > I''m not sure what "the virtualized hardware way" is. You probably need to > virtualize the PIC *somehow*, and the Xen event channels are as good a way > as any to do that. If IA64 already has an unrelated way of virtualizing the > PIC (as PPC did with the rhype model), then you get to choose. If that''s > the case, I would recommend adopting the event channels to share more Xen > code and design...On ia64, PIC is part of the processor and is already fully-virtualized. We are considering to switch to event channel. Thanks, Tristan. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, Mar 10, 2006 at 09:30:38AM -0600, Hollis Blanchard wrote:> On Friday 10 March 2006 04:07, Tristan Gingold wrote: > > > > How xen/ppc delivers interrupts ? > > Ignoring hypervisors for a moment, the PPC model for external interrupts is > this: the hardware resets the PC to a fixed entry point (0x500) and changes > the MSR to put the processor into real (untranslated) mode. (The old PC and > MSR are saved into a pair of supervisor-only registers, SRR0 and SRR1.) The > exception handler is then responsible for querying the interrupt > controller(s) to get the interrupt vector, and then call the appropriate > driver. > > The IBM Research hypervisor (rhype) virtualized the PIC, so that instead of > querying hardware, the kernel makes an hcall to find the incoming vector. If > the interrupt is not in fact for this domain, the hypervisor queues the > interrupt for later and tells the current domain "nothing was pending after > all". Note that in this model, interrupts are delivered by the hardware > directly to the current domain without hypervisor involvement.Not that you have a choice on PPC970, but there''s a drawback to this: If you let all interrupts be delivered directly to the domain, then it can hold the PIC "hostage" by disabling the delivery (keeping MSR_EE off), causing interrupts to other domains to be delayed. Some PPC processors have a feature called "mediated external interrupts", where they will be delivered to the hypervisor instead, and if the domain can''t service it then (MSR_EE off), the HV can request to be notified when the domain can take it. The extra code path for re-delivering to a domain can be made short. rHype has some support for this already, but it''s unclear which hardware has it. If I had to guess I would say at least Cell, since those parts of rhype are ripped out in the sources but some of the infrastructure is still in there. I''m just mentioning this in case the IA64 guys have something like mediated interrupts as an option, they might want to go with that for the fairness/latency reasons. -Olof _______________________________________________ Xen-ppc-devel mailing list Xen-ppc-devel@lists.xensource.com http://lists.xensource.com/xen-ppc-devel
On Fri, 2006-03-10 at 11:00 -0600, Olof Johansson wrote:> On Fri, Mar 10, 2006 at 09:30:38AM -0600, Hollis Blanchard wrote: > > On Friday 10 March 2006 04:07, Tristan Gingold wrote: > > > > > > How xen/ppc delivers interrupts ? > > > > Ignoring hypervisors for a moment, the PPC model for external interrupts is > > this: the hardware resets the PC to a fixed entry point (0x500) and changes > > the MSR to put the processor into real (untranslated) mode. (The old PC and > > MSR are saved into a pair of supervisor-only registers, SRR0 and SRR1.) The > > exception handler is then responsible for querying the interrupt > > controller(s) to get the interrupt vector, and then call the appropriate > > driver. > > > > The IBM Research hypervisor (rhype) virtualized the PIC, so that instead of > > querying hardware, the kernel makes an hcall to find the incoming vector. If > > the interrupt is not in fact for this domain, the hypervisor queues the > > interrupt for later and tells the current domain "nothing was pending after > > all". Note that in this model, interrupts are delivered by the hardware > > directly to the current domain without hypervisor involvement. >Acutally, rhype handles all 0x500 exceptions directly and then simulates 0x500 exceptions to partitions when they are to actually see an interrupt. Thus if an interrupt occurs for a partition that is not currently running, the partition does not see the exception. PHYP implements it differently; 0x500 exceptions are always seen by the currently running partitions even if the interrupt is not for them.> Not that you have a choice on PPC970, but there''s a drawback to this: If > you let all interrupts be delivered directly to the domain, then it can > hold the PIC "hostage" by disabling the delivery (keeping MSR_EE off), > causing interrupts to other domains to be delayed. >PPC970 gives you the choice of where external exceptions are directed to; hypervisor or partition (e.g. do external interrupts turn on MSR.HV?).> Some PPC processors have a feature called "mediated external interrupts", > where they will be delivered to the hypervisor instead, and if the domain > can''t service it then (MSR_EE off), the HV can request to be notified > when the domain can take it. The extra code path for re-delivering to a > domain can be made short. > > rHype has some support for this already, but it''s unclear which hardware > has it. If I had to guess I would say at least Cell, since those parts > of rhype are ripped out in the sources but some of the infrastructure > is still in there. >rhype interrupt handling code has been design with mediated interrupts in mind. If there are no mediated interrupts certain branches are never taken and certain things become no-ops. -- Michal Ostrowski <mostrows@watson.ibm.com> _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Tristan Gingold wrote:>On ia64, PIC is part of the processor and is already fully-virtualized. >We are considering to switch to event channel. > >What do you mean by "already fully virtualized" -- you mean before Xen, this was already part of the architecture? Could you elaborate on that? -- Hollis Blanchard IBM Linux Technology Center _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Mar 10, 2006, at 12:00 PM, Olof Johansson wrote:> On Fri, Mar 10, 2006 at 09:30:38AM -0600, Hollis Blanchard wrote: >> >> The IBM Research hypervisor (rhype) virtualized the PIC, so that >> instead of >> querying hardware, the kernel makes an hcall to find the incoming >> vector. If >> the interrupt is not in fact for this domain, the hypervisor >> queues the >> interrupt for later and tells the current domain "nothing was >> pending after >> all". Note that in this model, interrupts are delivered by the >> hardware >> directly to the current domain without hypervisor involvement.Actually rHype and xen configure the processor to deliver external interrupts directly to the hypervisor.> Not that you have a choice on PPC970,This _is_ the case for 970 as well (it was not the case for POWER4).> but there''s a drawback to this: If > you let all interrupts be delivered directly to the domain, then it > can > hold the PIC "hostage" by disabling the delivery (keeping MSR_EE off), > causing interrupts to other domains to be delayed.The Hostage (MSR_EE) bit problem exists does exist with 970.> > Some PPC processors have a feature called "mediated external > interrupts", > where they will be delivered to the hypervisor instead,MEI does not make this descision.. the LPES does. The problem MEI solves (as you mention below) is the case that the HV wants to deliver an interrupt but MSR_EE for the domain is off. MEI allows the hypervisor to force the processor a special form of external when the domain sets MSR_EE.> and if the domain > can''t service it then (MSR_EE off), the HV can request to be notified > when the domain can take it. The extra code path for re-delivering > to a > domain can be made short. > > rHype has some support for this already, but it''s unclear which > hardware > has it. If I had to guess I would say at least Cell, since those parts > of rhype are ripped out in the sources but some of the infrastructure > is still in there.Yes it is for Cell and other forth-comming processors. It is really for latency issues of RTOS (and games), server workloads are not expected to ever need this.> > > I''m just mentioning this in case the IA64 guys have something like > mediated interrupts as an option, they might want to go with that for > the fairness/latency reasons.Does IA64 indeed have MEI like abilities or the ability to route the interrupt? -JX _______________________________________________ Xen-ppc-devel mailing list Xen-ppc-devel@lists.xensource.com http://lists.xensource.com/xen-ppc-devel
On Fri, Mar 10, 2006 at 09:27:50PM -0500, Jimi Xenidis wrote:> > >Not that you have a choice on PPC970, > > This _is_ the case for 970 as well (it was not the case for POWER4).Since POWER4 didn''t have a hypervisor that allowed shared processors, I guess this wasn''t much of a problem. (Well, maybe seen the other way the lack of the processor feature kept them from adding shared processor support).> >Some PPC processors have a feature called "mediated external > >interrupts", > >where they will be delivered to the hypervisor instead, > > MEI does not make this descision.. the LPES does.Right. Hollis threw me off by saying that interrupts go to the domain, leading me to believe the two features were introduced at the same time. Especially since both MEI and HV/SV delivery are configured via LPES.> Yes it is for Cell and other forth-comming processors. It is really > for latency issues of RTOS (and games), server workloads are not > expected to ever need this.Yep, you''d definitely want it on a desktop running a hypervisor too. Classic server workloads are obviously not nearly as latency sensitive. -Olof _______________________________________________ Xen-ppc-devel mailing list Xen-ppc-devel@lists.xensource.com http://lists.xensource.com/xen-ppc-devel
>From: Hollis Blanchard >Sent: 2006年3月11日 1:27 > >Tristan Gingold wrote: > >>On ia64, PIC is part of the processor and is already fully-virtualized. >>We are considering to switch to event channel. >> >> >What do you mean by "already fully virtualized" -- you mean before Xen, >this was already part of the architecture? Could you elaborate on that? >I may put some word on this, since simple "PIC" is not clear term here. The interrupt architecture on IA64 is called SAPIC (Steamlined Advanced Programmable Interrupt Controller), which is mostly like APIC architecture on X86. For example, there''re also two parts: IOSAPIC and Local SAPIC, with IOSAPIC sitting in the chipset, while Local SAPIC part of processor. Part of Local SAPIC is virtualized due to processor virtualization, like registers. MMIO of Local SAPIC like PIB/XTP and IOSAPIC are handled differently, and that''s the part we''re proposing strongly to reuse same xen/x86 event channel mechanism on xen-ia64 mailing list now. Hope helps. Thanks, Kevin _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Le Vendredi 10 Mars 2006 18:27, Hollis Blanchard a écrit :> Tristan Gingold wrote: > >On ia64, PIC is part of the processor and is already fully-virtualized. > >We are considering to switch to event channel. > > What do you mean by "already fully virtualized" -- you mean before Xen, > this was already part of the architecture? Could you elaborate on that?I think so. Before working on Xen, Dan Magenheimer worked on an HP project, vBlades. It seems he has imported ideas and sources files from vBlades to Xen/ia64. Tristan. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Le Samedi 11 Mars 2006 03:27, Jimi Xenidis a écrit : [...]> > I''m just mentioning this in case the IA64 guys have something like > > mediated interrupts as an option, they might want to go with that for > > the fairness/latency reasons. > > Does IA64 indeed have MEI like abilities or the ability to route the > interrupt?Well, I am a little bit lost with these ppc features. Currently on ia64, interrupts are caught by Xen and then injected into dom0. I don''t see any fairness or latency problems [except when we are dealing with event channel]. On ia64, there are two PIC: one on each processor (LSAPIC ~= LAPIC) and one (or several) for IO (IOSAPIC ~= IO_APIC). Tristan. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Mar 13, 2006, at 3:41 AM, Tristan Gingold wrote:> I don''t see any fairness or latency problems [except when we are > dealing with > event channel].Problem exists: If you defer the delivery of an IO interrupt because the VCPU is masking them --and-- If the domain can unmask interrupts without the hypervisor knowing -JX _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Le Lundi 13 Mars 2006 21:44, Jimi Xenidis a écrit :> On Mar 13, 2006, at 3:41 AM, Tristan Gingold wrote: > > I don''t see any fairness or latency problems [except when we are > > dealing with > > event channel]. > > Problem exists: > If you defer the delivery of an IO interrupt because the VCPU is > masking them > --and-- > If the domain can unmask interrupts without the hypervisor knowingSure, but this cannot happen on ia64 since masking/unmasking interrupts is a privilegied operation which is always trapped. Tristan. _______________________________________________ Xen-ppc-devel mailing list Xen-ppc-devel@lists.xensource.com http://lists.xensource.com/xen-ppc-devel