Jan Beulich
2010-Jun-29 09:53 UTC
[Xen-devel] APIC errors resulting from too early set_timer()
Keir, in c/s 17422 you moved the alloc_pdata scheduler callout into the CPU_UP_PREPARE notifier handler, but the credit scheduler uses set_timer() for a timer bound to the to be brought up CPU from sched_alloc_pdata(), with the resulting send_IPI_mask() targeting the not yet started CPU, thus (on my AMD systems at least) resulting in APIC send accept errors. I see three possible fixes, neither really nice imo: Either a second callout (from CPU_ONLINE), or a credit scheduler specific notifier (handling CPU_ONLINE), or binding the timer initially to the current CPU, and migrate it to the target CPU as soon as that one''s online. Do you favor any one of these, or do you see any alternative? Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2010-Jun-29 10:32 UTC
[Xen-devel] Re: APIC errors resulting from too early set_timer()
On 29/06/2010 10:53, "Jan Beulich" <JBeulich@novell.com> wrote:> in c/s 17422 you moved the alloc_pdata scheduler callout into the > CPU_UP_PREPARE notifier handler, but the credit scheduler uses > set_timer() for a timer bound to the to be brought up CPU from > sched_alloc_pdata(), with the resulting send_IPI_mask() targeting > the not yet started CPU, thus (on my AMD systems at least) > resulting in APIC send accept errors. > > I see three possible fixes, neither really nice imo: Either a second > callout (from CPU_ONLINE), or a credit scheduler specific notifier > (handling CPU_ONLINE), or binding the timer initially to the current > CPU, and migrate it to the target CPU as soon as that one''s online. > > Do you favor any one of these, or do you see any alternative?The IPI is needless since the CPU will check for pending softirqs when it is brought up, from its idle loop or before entering guest context for the first time. I suggest that we mask the given cpumap against cpu_online_map, and smp_processor_id() while we''re at it, in arch/x86/smp.c:smp_send_event_check_mask(). If you agree that this makes sense I will make the patch myself. Thanks, Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jan Beulich
2010-Jun-29 11:52 UTC
[Xen-devel] Re: APIC errors resulting from too early set_timer()
>>> On 29.06.10 at 12:32, Keir Fraser <keir.fraser@eu.citrix.com> wrote: > I suggest that we mask the given cpumap against cpu_online_map, and > smp_processor_id() while we''re at it, in > arch/x86/smp.c:smp_send_event_check_mask(). If you agree that this makes > sense I will make the patch myself.Yes, this sounds appropriate to me. Thanks, Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2010-Jun-29 12:17 UTC
[Xen-devel] Re: APIC errors resulting from too early set_timer()
On 29/06/2010 12:52, "Jan Beulich" <JBeulich@novell.com> wrote:>>>> On 29.06.10 at 12:32, Keir Fraser <keir.fraser@eu.citrix.com> wrote: >> I suggest that we mask the given cpumap against cpu_online_map, and >> smp_processor_id() while we''re at it, in >> arch/x86/smp.c:smp_send_event_check_mask(). If you agree that this makes >> sense I will make the patch myself. > > Yes, this sounds appropriate to me.xen-unstable:21690, although I ended up pushing the logic into the send_IPI_mask variants. -- Keir> Thanks, Jan >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel