Hi Keir, I have some questions about evtchn_callback, As we know all external interrupts and events are deliver to dom0 by evtchn_callback, Are evtchn_callback reentered? I mean when evtchn_callbacak is executing on dom0, Another external interrupt or event may happen, can this event be deliverred immediately? I found __sti will set vcpu_info-> evtchn_upcall_mask=0, which permits event deliverring. and __sti may be called at any device driver. If evtchn_callback can be reentered, how does it hanlde external interrupt privilege? If there is no special handling, low privileged interrupt may "interrupt" high privileged interrupt. BTW, I do see evtchn_callback reentry happen on IPF dom0, and that incur about 5% performance degradation with KB on VTI-domain compared to non_reentry. Thanks, Anthony _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 18/12/06 5:28 am, "Xu, Anthony" <anthony.xu@intel.com> wrote:> If there is no special handling, low privileged interrupt may > "interrupt" high privileged interrupt.There''s no concept of privilege among event channels. Since Xen simply allocates ExtInts to vectors in a round-robin manner, there is no meaningful priority comparsion to be made between vectors anyway.> BTW, > I do see evtchn_callback reentry happen on IPF dom0, and that incur > about 5% performance degradation > with KB on VTI-domain compared to non_reentry.KB == Keyboard?! Reentry is allowed -- it shouldn''t happen often unless the ISRs are really long (bad) or we are hammering the interrupt line for some reason (bad). -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser write on 2006年12月18日 17:32:> On 18/12/06 5:28 am, "Xu, Anthony" <anthony.xu@intel.com> wrote: >Thanks for your quick response>> If there is no special handling, low privileged interrupt may >> "interrupt" high privileged interrupt. > > There''s no concept of privilege among event channels. Since Xen simply > allocates ExtInts to vectors in a round-robin manner, there is no > meaningful priority comparsion to be made between vectors anyway.Yes there''s no concept of privilege among event channel, But all device interrupts have privilege and device interrupts are delivered through event channel. That means ide interrupt may interrupt time interrup in dom0. It maybe not reasonable.> >> BTW, >> I do see evtchn_callback reentry happen on IPF dom0, and that incur >> about 5% performance degradation >> with KB on VTI-domain compared to non_reentry. > > KB == Keyboard?!Sorry for confusion, KB means Kernal Build.> > Reentry is allowed -- it shouldn''t happen often unless the ISRs are > really long (bad) or we are hammering the interrupt line for some > reason (bad).If evtchn_callback reentry is allowed. There are 1024 event channels. In theory, dom0 kernal stack may be overflowed. Is there any mechanism to prevent this? Since event channels have no concept of privilege. Why we allow evtchn_callback reentered ? I think evtchn_callback can handle all events in batch. This is good for performance. -- Anthony> > -- Keir_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> But all device interrupts have privilege and device interrupts are delivered > through event channel.Do they? Architecturally defined? I don''t think so.> That means ide interrupt may interrupt time interrup in dom0. > It maybe not reasonable.Linux doesn''t care. If it did we could apply some form of prioritisation in Linux''s event-channel subsystem.> If evtchn_callback reentry is allowed. > There are 1024 event channels. > In theory, dom0 kernal stack may be overflowed. > Is there any mechanism to prevent this?When running natively there are over 200 vectors available for allocation to interrupt sources. In theory the kernel stack may be overflowed, and there is no mechanism in Linux to prevent this.> Since event channels have no concept of privilege. > Why we allow evtchn_callback reentered ?Mainly because it follows the model of interrupt delivery on native Linux. We could deviate this default behaviour when running on Xen -- interrupt handlers should be quick to run so really I would expect no performance difference one way or the other. Do you understand why you are getting so much reentrancy, and hence such a horrible performance degradation, on ia64? -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 18/12/06 11:24, "Keir Fraser" <keir@xensource.com> wrote:>> If evtchn_callback reentry is allowed. >> There are 1024 event channels. >> In theory, dom0 kernal stack may be overflowed. >> Is there any mechanism to prevent this? > > When running natively there are over 200 vectors available for allocation to > interrupt sources. In theory the kernel stack may be overflowed, and there > is no mechanism in Linux to prevent this.Okay, so I remember now that of course the APIC divides these into 16 priority levels so that does rather restrict the maximum nesting. I''ll be happy to consider adding prioritisation (or just complete disallow of nesting) for 3.0.5. But I think you should also work to understand the root cause of the performance drop on ia64. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser write on 2006年12月18日 21:42:> On 18/12/06 11:24, "Keir Fraser" <keir@xensource.com> wrote: > > Okay, so I remember now that of course the APIC divides these into 16 > priority levels so that does rather restrict the maximum nesting. > I''ll be happy to consider adding prioritisation (or just complete > disallow of nesting) for 3.0.5.Hi Keir, I did a experiment, If we completely disallow of nesting. Even dom0 can''t boot. I suspected timer interrupt is blocked by other events. --Anthony> > -- Keir_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 20/12/06 3:03 am, "Xu, Anthony" <anthony.xu@intel.com> wrote:>> Okay, so I remember now that of course the APIC divides these into 16 >> priority levels so that does rather restrict the maximum nesting. >> I''ll be happy to consider adding prioritisation (or just complete >> disallow of nesting) for 3.0.5. > > Hi Keir, > > I did a experiment, > If we completely disallow of nesting. > Even dom0 can''t boot. > > I suspected timer interrupt is blocked by other events.How did you implement nesting avoidance? It''ll be a bit subtle to get right. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser write on 2006年12月20日 16:31:> On 20/12/06 3:03 am, "Xu, Anthony" <anthony.xu@intel.com> wrote: > > How did you implement nesting avoidance? It''ll be a bit subtle to get > right.I added another global_event_mask in share page. 1. before injecting to event_callback_handler, set global_event_mask to 1 to disable event inject.. 2. after event_callback_handler is executed, it will return to ia64_leave_kernel assembly code in dom0, in this assembly code, it sets global_event_mask to 0 to enable event inject. There is no other code operating the global_event_mask. Global_event_mask is initiated to 0(unmask). Only global_event_mask is 0 and vcpu->vcpu_info->evtchn_upcall_mask is 0, Can XEN injects to event_callback_handler of dom0. --Anthony> > -- Keir_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 20/12/06 10:08, "Xu, Anthony" <anthony.xu@intel.com> wrote:> There is no other code operating the global_event_mask. > Global_event_mask is initiated to 0(unmask). > > Only global_event_mask is 0 and vcpu->vcpu_info->evtchn_upcall_mask is 0, > Can XEN injects to event_callback_handler of dom0.How much does injecting a stack frame on ia64 actually cost? It sounds like it must be quite hideously expensive... I would be inclined to implement this for x86 by adding the nesting detection into the callback handler (i.e., the extra callback injection would not be avoided). Keeps the hypervisor interface clean. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel