Keir, I made a cpu status notifier for sched_credit2() to actually read an arrange the runqueue information, and found the next niggle: the callbacks are not guaranteed to finish before the cpu tried to go through the scheduler. The callback notifiers are handled on the cpu that issues the boot command (i.e., cpu 0 during boot), and there''s no interlock to prevent the booted cpu from continuing until the notifiers have completed execution. Making a simple interlock (similar to the one in __cpu_up()) allows the system to boot properly. Another possibility would be to run the notifiers on the freshly booted cpu before calling into the scheduler, rather than on the cpu that issued the cpu boot sequence. Thoughts? -George _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 09/12/2010 12:49, "George Dunlap" <dunlapg@umich.edu> wrote:> Keir, > > I made a cpu status notifier for sched_credit2() to actually read an > arrange the runqueue information, and found the next niggle: the > callbacks are not guaranteed to finish before the cpu tried to go > through the scheduler. The callback notifiers are handled on the cpu > that issues the boot command (i.e., cpu 0 during boot), and there''s no > interlock to prevent the booted cpu from continuing until the > notifiers have completed execution. > > Making a simple interlock (similar to the one in __cpu_up()) allows > the system to boot properly. Another possibility would be to run the > notifiers on the freshly booted cpu before calling into the scheduler, > rather than on the cpu that issued the cpu boot sequence.I could bring Linux''s CPU_STARTING notifier over into Xen. Runs in context of new CPU before it is fully online (e.g., before interrupts are enabled). So you couldn''t do any allocations there, or anything else that can fail. This might require some juggling to pre-allocate memory (e.g., for possibly-required new runqueue) on CPU_UP_PREPARE/alloc_pdata, and potentially free that memory if unused on CPU_ONLINE. Or not, if actually you require no dynamic memory allocation. This might be the best solution overall I think? I can knock up a patch for CPU_STARTING if that sounds good. -- Keir> Thoughts? > > -George > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
That could work, if you want. ATM I don''t allocate anything; if I need to in the future, I should be able to do it allocation in alloc_pdata(). I don''t strictly need it to run on the processor that''s coming up; I just need: * The function to happen after the cpu ID stuff, so that (for example) cpu_to_socket() returns a reasonable value * The function to finish before the cpu tries to run the scheduler But if you''d rather add CPU_STARTING than an interlock for CPU_ONLINE for technical reasons, that''s fine. Thanks, -George On Thu, Dec 9, 2010 at 2:16 PM, Keir Fraser <keir@xen.org> wrote:> On 09/12/2010 12:49, "George Dunlap" <dunlapg@umich.edu> wrote: > >> Keir, >> >> I made a cpu status notifier for sched_credit2() to actually read an >> arrange the runqueue information, and found the next niggle: the >> callbacks are not guaranteed to finish before the cpu tried to go >> through the scheduler. The callback notifiers are handled on the cpu >> that issues the boot command (i.e., cpu 0 during boot), and there''s no >> interlock to prevent the booted cpu from continuing until the >> notifiers have completed execution. >> >> Making a simple interlock (similar to the one in __cpu_up()) allows >> the system to boot properly. Another possibility would be to run the >> notifiers on the freshly booted cpu before calling into the scheduler, >> rather than on the cpu that issued the cpu boot sequence. > > I could bring Linux''s CPU_STARTING notifier over into Xen. Runs in context > of new CPU before it is fully online (e.g., before interrupts are enabled). > So you couldn''t do any allocations there, or anything else that can fail. > This might require some juggling to pre-allocate memory (e.g., for > possibly-required new runqueue) on CPU_UP_PREPARE/alloc_pdata, and > potentially free that memory if unused on CPU_ONLINE. Or not, if actually > you require no dynamic memory allocation. > > This might be the best solution overall I think? I can knock up a patch for > CPU_STARTING if that sounds good. > > -- Keir > >> Thoughts? >> >> -George >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 09/12/2010 14:35, "George Dunlap" <George.Dunlap@eu.citrix.com> wrote:> That could work, if you want. ATM I don''t allocate anything; if I > need to in the future, I should be able to do it allocation in > alloc_pdata(). > > I don''t strictly need it to run on the processor that''s coming up; I just > need: > * The function to happen after the cpu ID stuff, so that (for example) > cpu_to_socket() returns a reasonable value > * The function to finish before the cpu tries to run the scheduler > > But if you''d rather add CPU_STARTING than an interlock for CPU_ONLINE > for technical reasons, that''s fine.CPU_ONLINE means that cpu is already in cpu_online_map. I doubt you''d want to delay this scheduler stuff until then -- at that point other cpus can see you and poke your runqueue. So I think a new notifier type is required, and we may as well stick with Linux-ish semantics with CPU_STARTING. -- Keir> Thanks, > -George > > > On Thu, Dec 9, 2010 at 2:16 PM, Keir Fraser <keir@xen.org> wrote: >> On 09/12/2010 12:49, "George Dunlap" <dunlapg@umich.edu> wrote: >> >>> Keir, >>> >>> I made a cpu status notifier for sched_credit2() to actually read an >>> arrange the runqueue information, and found the next niggle: the >>> callbacks are not guaranteed to finish before the cpu tried to go >>> through the scheduler. The callback notifiers are handled on the cpu >>> that issues the boot command (i.e., cpu 0 during boot), and there''s no >>> interlock to prevent the booted cpu from continuing until the >>> notifiers have completed execution. >>> >>> Making a simple interlock (similar to the one in __cpu_up()) allows >>> the system to boot properly. Another possibility would be to run the >>> notifiers on the freshly booted cpu before calling into the scheduler, >>> rather than on the cpu that issued the cpu boot sequence. >> >> I could bring Linux''s CPU_STARTING notifier over into Xen. Runs in context >> of new CPU before it is fully online (e.g., before interrupts are enabled). >> So you couldn''t do any allocations there, or anything else that can fail. >> This might require some juggling to pre-allocate memory (e.g., for >> possibly-required new runqueue) on CPU_UP_PREPARE/alloc_pdata, and >> potentially free that memory if unused on CPU_ONLINE. Or not, if actually >> you require no dynamic memory allocation. >> >> This might be the best solution overall I think? I can knock up a patch for >> CPU_STARTING if that sounds good. >> >> -- Keir >> >>> Thoughts? >>> >>> -George >>> >>> _______________________________________________ >>> Xen-devel mailing list >>> Xen-devel@lists.xensource.com >>> http://lists.xensource.com/xen-devel >> >> >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel >>_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
I implemented CPU_STARTING as xen-unstable:22474. -- Keir On 09/12/2010 14:35, "George Dunlap" <George.Dunlap@eu.citrix.com> wrote:> That could work, if you want. ATM I don''t allocate anything; if I > need to in the future, I should be able to do it allocation in > alloc_pdata(). > > I don''t strictly need it to run on the processor that''s coming up; I just > need: > * The function to happen after the cpu ID stuff, so that (for example) > cpu_to_socket() returns a reasonable value > * The function to finish before the cpu tries to run the scheduler > > But if you''d rather add CPU_STARTING than an interlock for CPU_ONLINE > for technical reasons, that''s fine. > > Thanks, > -George > > > On Thu, Dec 9, 2010 at 2:16 PM, Keir Fraser <keir@xen.org> wrote: >> On 09/12/2010 12:49, "George Dunlap" <dunlapg@umich.edu> wrote: >> >>> Keir, >>> >>> I made a cpu status notifier for sched_credit2() to actually read an >>> arrange the runqueue information, and found the next niggle: the >>> callbacks are not guaranteed to finish before the cpu tried to go >>> through the scheduler. The callback notifiers are handled on the cpu >>> that issues the boot command (i.e., cpu 0 during boot), and there''s no >>> interlock to prevent the booted cpu from continuing until the >>> notifiers have completed execution. >>> >>> Making a simple interlock (similar to the one in __cpu_up()) allows >>> the system to boot properly. Another possibility would be to run the >>> notifiers on the freshly booted cpu before calling into the scheduler, >>> rather than on the cpu that issued the cpu boot sequence. >> >> I could bring Linux''s CPU_STARTING notifier over into Xen. Runs in context >> of new CPU before it is fully online (e.g., before interrupts are enabled). >> So you couldn''t do any allocations there, or anything else that can fail. >> This might require some juggling to pre-allocate memory (e.g., for >> possibly-required new runqueue) on CPU_UP_PREPARE/alloc_pdata, and >> potentially free that memory if unused on CPU_ONLINE. Or not, if actually >> you require no dynamic memory allocation. >> >> This might be the best solution overall I think? I can knock up a patch for >> CPU_STARTING if that sounds good. >> >> -- Keir >> >>> Thoughts? >>> >>> -George >>> >>> _______________________________________________ >>> Xen-devel mailing list >>> Xen-devel@lists.xensource.com >>> http://lists.xensource.com/xen-devel >> >> >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel >>_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
That works great, thanks! -George On Thu, Dec 9, 2010 at 4:20 PM, Keir Fraser <keir@xen.org> wrote:> I implemented CPU_STARTING as xen-unstable:22474. > > -- Keir > > On 09/12/2010 14:35, "George Dunlap" <George.Dunlap@eu.citrix.com> wrote: > >> That could work, if you want. ATM I don''t allocate anything; if I >> need to in the future, I should be able to do it allocation in >> alloc_pdata(). >> >> I don''t strictly need it to run on the processor that''s coming up; I just >> need: >> * The function to happen after the cpu ID stuff, so that (for example) >> cpu_to_socket() returns a reasonable value >> * The function to finish before the cpu tries to run the scheduler >> >> But if you''d rather add CPU_STARTING than an interlock for CPU_ONLINE >> for technical reasons, that''s fine. >> >> Thanks, >> -George >> >> >> On Thu, Dec 9, 2010 at 2:16 PM, Keir Fraser <keir@xen.org> wrote: >>> On 09/12/2010 12:49, "George Dunlap" <dunlapg@umich.edu> wrote: >>> >>>> Keir, >>>> >>>> I made a cpu status notifier for sched_credit2() to actually read an >>>> arrange the runqueue information, and found the next niggle: the >>>> callbacks are not guaranteed to finish before the cpu tried to go >>>> through the scheduler. The callback notifiers are handled on the cpu >>>> that issues the boot command (i.e., cpu 0 during boot), and there''s no >>>> interlock to prevent the booted cpu from continuing until the >>>> notifiers have completed execution. >>>> >>>> Making a simple interlock (similar to the one in __cpu_up()) allows >>>> the system to boot properly. Another possibility would be to run the >>>> notifiers on the freshly booted cpu before calling into the scheduler, >>>> rather than on the cpu that issued the cpu boot sequence. >>> >>> I could bring Linux''s CPU_STARTING notifier over into Xen. Runs in context >>> of new CPU before it is fully online (e.g., before interrupts are enabled). >>> So you couldn''t do any allocations there, or anything else that can fail. >>> This might require some juggling to pre-allocate memory (e.g., for >>> possibly-required new runqueue) on CPU_UP_PREPARE/alloc_pdata, and >>> potentially free that memory if unused on CPU_ONLINE. Or not, if actually >>> you require no dynamic memory allocation. >>> >>> This might be the best solution overall I think? I can knock up a patch for >>> CPU_STARTING if that sounds good. >>> >>> -- Keir >>> >>>> Thoughts? >>>> >>>> -George >>>> >>>> _______________________________________________ >>>> Xen-devel mailing list >>>> Xen-devel@lists.xensource.com >>>> http://lists.xensource.com/xen-devel >>> >>> >>> >>> _______________________________________________ >>> Xen-devel mailing list >>> Xen-devel@lists.xensource.com >>> http://lists.xensource.com/xen-devel >>> > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel