Xen devel - Feb 2010 - [RFC][PATCH] Scheduler interface changes for credit2

The two attached patches change the scheduler interface to allow
credit2 to have several cpus share the same runqueue.  The patches
should have almost no impact on the current schedulers.  The patches
and their reasonings are below.  I''ve also attached the patches for
the prototype credit2 scheduler, for reference.

* Add a context swich callback (sched-context_switch-callback.diff)

Add a callback to tell a scheduler that a vcpu has been completely
context-switched off a cpu.

When sharing a runqueue, we can''t put a scheduled-out vcpu back on the
runqueue until it''s been completely de-scheduled, because it may be
grabbed by another processor before it''s ready.  This callback allows
a scheduler to detect when a vcpu on its way out is completely off the
processor, so that it can put the vcpu on the runqueue.

* Allow sharing of locks between cpus (sched-spin_lock-pointers.diff)

Have per-cpu pointers, initialized to per-cpu locks, which the
scheduler may change during its init to reconfigure locking
granularity.

There are a number of race conditions having to do with updating of
v->is_running and v->processor, all having to do with the fact that
vcpus may change cpus without an explicit migrate.  Furthermore, the
scheduler needs runqueues to be covered by a lock as well.  The
cleanest way to solve all of these is to have the scheduler lock and
the runqueue lock coincide.

* Add a "scheduler" trace class (trace-sched-class.diff)
Uses defined on a per-scheduler basis

I''ve been running parallel kernel compiles on a 16-way box (2x4x2) for
several hours now without deadlocks or BUG()s.  As far as I''m
concerned, with these changes, credit2 is now ready to be checked in,
as long as it''s not set to the default scheduler.

All of the above:
Signed-off-by: George Dunlap <george.dunlap@eu.citrix.com>






_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir, any thoughts?  Does this seem like a reasonable approach?

Thanks,
 -George

On Mon, Feb 15, 2010 at 5:20 PM, George Dunlap
<George.Dunlap@eu.citrix.com> wrote:
> The two attached patches change the scheduler interface to allow
> credit2 to have several cpus share the same runqueue.  The patches
> should have almost no impact on the current schedulers.  The patches
> and their reasonings are below.  I''ve also attached the patches
for
> the prototype credit2 scheduler, for reference.
>
> * Add a context swich callback (sched-context_switch-callback.diff)
>
> Add a callback to tell a scheduler that a vcpu has been completely
> context-switched off a cpu.
>
> When sharing a runqueue, we can''t put a scheduled-out vcpu back on
the
> runqueue until it''s been completely de-scheduled, because it may
be
> grabbed by another processor before it''s ready.  This callback
allows
> a scheduler to detect when a vcpu on its way out is completely off the
> processor, so that it can put the vcpu on the runqueue.
>
> * Allow sharing of locks between cpus (sched-spin_lock-pointers.diff)
>
> Have per-cpu pointers, initialized to per-cpu locks, which the
> scheduler may change during its init to reconfigure locking
> granularity.
>
> There are a number of race conditions having to do with updating of
> v->is_running and v->processor, all having to do with the fact that
> vcpus may change cpus without an explicit migrate.  Furthermore, the
> scheduler needs runqueues to be covered by a lock as well.  The
> cleanest way to solve all of these is to have the scheduler lock and
> the runqueue lock coincide.
>
> * Add a "scheduler" trace class (trace-sched-class.diff)
> Uses defined on a per-scheduler basis
>
> I''ve been running parallel kernel compiles on a 16-way box (2x4x2)
for
> several hours now without deadlocks or BUG()s.  As far as I''m
> concerned, with these changes, credit2 is now ready to be checked in,
> as long as it''s not set to the default scheduler.
>
> All of the above:
> Signed-off-by: George Dunlap <george.dunlap@eu.citrix.com>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

They''re all fine, even probably sched-context_switch-callback.diff,
which I
suppose is your new alternative to having vcpus-which-arent-yet-schedulable
left on the shared runqueue? I suppose, although I reckon it could still be
done another way and using the vcpu_migrate logic, that this is a smaller
and neater way to do it really.

All this are apply-able after 4.0.

 -- Keir

On 22/02/2010 15:22, "George Dunlap"
<George.Dunlap@eu.citrix.com> wrote:
> Keir, any thoughts?  Does this seem like a reasonable approach?
> 
> Thanks,
>  -George
> 
> On Mon, Feb 15, 2010 at 5:20 PM, George Dunlap
> <George.Dunlap@eu.citrix.com> wrote:
>> The two attached patches change the scheduler interface to allow
>> credit2 to have several cpus share the same runqueue.  The patches
>> should have almost no impact on the current schedulers.  The patches
>> and their reasonings are below.  I''ve also attached the
patches for
>> the prototype credit2 scheduler, for reference.
>> 
>> * Add a context swich callback (sched-context_switch-callback.diff)
>> 
>> Add a callback to tell a scheduler that a vcpu has been completely
>> context-switched off a cpu.
>> 
>> When sharing a runqueue, we can''t put a scheduled-out vcpu
back on the
>> runqueue until it''s been completely de-scheduled, because it
may be
>> grabbed by another processor before it''s ready.  This callback
allows
>> a scheduler to detect when a vcpu on its way out is completely off the
>> processor, so that it can put the vcpu on the runqueue.
>> 
>> * Allow sharing of locks between cpus (sched-spin_lock-pointers.diff)
>> 
>> Have per-cpu pointers, initialized to per-cpu locks, which the
>> scheduler may change during its init to reconfigure locking
>> granularity.
>> 
>> There are a number of race conditions having to do with updating of
>> v->is_running and v->processor, all having to do with the fact
that
>> vcpus may change cpus without an explicit migrate.  Furthermore, the
>> scheduler needs runqueues to be covered by a lock as well.  The
>> cleanest way to solve all of these is to have the scheduler lock and
>> the runqueue lock coincide.
>> 
>> * Add a "scheduler" trace class (trace-sched-class.diff)
>> Uses defined on a per-scheduler basis
>> 
>> I''ve been running parallel kernel compiles on a 16-way box
(2x4x2) for
>> several hours now without deadlocks or BUG()s.  As far as I''m
>> concerned, with these changes, credit2 is now ready to be checked in,
>> as long as it''s not set to the default scheduler.
>> 
>> All of the above:
>> Signed-off-by: George Dunlap <george.dunlap@eu.citrix.com>
>> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
> They''re all fine, even probably
sched-context_switch-callback.diff, which I
> suppose is your new alternative to having vcpus-which-arent-yet-schedulable
> left on the shared runqueue? I suppose, although I reckon it could still be
> done another way and using the vcpu_migrate logic, that this is a smaller
> and neater way to do it really.
>   
Exactly.  The migrate logic calls cpu_pick, which is also called from 
other contexts.  Having the same function try to do both, and setting 
the "migrating" flag when the vcpu isn''t migrating, just
seems hackish
and likely to be confusing to me.  It does add another "Check to see if 
this function exists" on the hot path for the other schedulers.

I suppose we could think about changing things around, so that there''s
a
generic "callback on context switch" flag.  Then we''d only
have one
callback in the context switch code, and the check would only happen if 
the flag was set (reducing cache footprint).  It would be a slightly 
larger patch, and make changes to the other schedulers, but those would 
be pretty minimal.

I''ll play around with it and see how it looks.

-George
> All this are apply-able after 4.0.
>
>  -- Keir
>
> On 22/02/2010 15:22, "George Dunlap"
<George.Dunlap@eu.citrix.com> wrote:
>
>   
>> Keir, any thoughts?  Does this seem like a reasonable approach?
>>
>> Thanks,
>>  -George
>>
>> On Mon, Feb 15, 2010 at 5:20 PM, George Dunlap
>> <George.Dunlap@eu.citrix.com> wrote:
>>     
>>> The two attached patches change the scheduler interface to allow
>>> credit2 to have several cpus share the same runqueue.  The patches
>>> should have almost no impact on the current schedulers.  The
patches
>>> and their reasonings are below.  I''ve also attached the
patches for
>>> the prototype credit2 scheduler, for reference.
>>>
>>> * Add a context swich callback (sched-context_switch-callback.diff)
>>>
>>> Add a callback to tell a scheduler that a vcpu has been completely
>>> context-switched off a cpu.
>>>
>>> When sharing a runqueue, we can''t put a scheduled-out vcpu
back on the
>>> runqueue until it''s been completely de-scheduled, because
it may be
>>> grabbed by another processor before it''s ready.  This
callback allows
>>> a scheduler to detect when a vcpu on its way out is completely off
the
>>> processor, so that it can put the vcpu on the runqueue.
>>>
>>> * Allow sharing of locks between cpus
(sched-spin_lock-pointers.diff)
>>>
>>> Have per-cpu pointers, initialized to per-cpu locks, which the
>>> scheduler may change during its init to reconfigure locking
>>> granularity.
>>>
>>> There are a number of race conditions having to do with updating of
>>> v->is_running and v->processor, all having to do with the
fact that
>>> vcpus may change cpus without an explicit migrate.  Furthermore,
the
>>> scheduler needs runqueues to be covered by a lock as well.  The
>>> cleanest way to solve all of these is to have the scheduler lock
and
>>> the runqueue lock coincide.
>>>
>>> * Add a "scheduler" trace class (trace-sched-class.diff)
>>> Uses defined on a per-scheduler basis
>>>
>>> I''ve been running parallel kernel compiles on a 16-way box
(2x4x2) for
>>> several hours now without deadlocks or BUG()s.  As far as
I''m
>>> concerned, with these changes, credit2 is now ready to be checked
in,
>>> as long as it''s not set to the default scheduler.
>>>
>>> All of the above:
>>> Signed-off-by: George Dunlap <george.dunlap@eu.citrix.com>
>>>
>>>       
>
>
>   

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 22/02/2010 16:16, "George Dunlap"
<George.Dunlap@eu.citrix.com> wrote:
> I suppose we could think about changing things around, so that
there''s a
> generic "callback on context switch" flag.  Then we''d
only have one
> callback in the context switch code, and the check would only happen if
> the flag was set (reducing cache footprint).  It would be a slightly
> larger patch, and make changes to the other schedulers, but those would
> be pretty minimal.
> 
> I''ll play around with it and see how it looks.
Nah, I think we''ll just go with it as it is. It''s clean and
the path isn''t
*that* hot.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel