Xen devel - Mar 2013 - [PATCH v4 2/2] arm: prefer PSCI for SMP bringup

If PSCI initializes correctly and PSCI SMP operations are available, use them.
This is required for SMP support in Dom0 on Xen.

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
CC: will.deacon@arm.com
CC: arnd@arndb.de
CC: marc.zyngier@arm.com
CC: linux@arm.linux.org.uk
CC: nico@linaro.org
---
 arch/arm/kernel/setup.c |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/arch/arm/kernel/setup.c b/arch/arm/kernel/setup.c
index 341efaa..c7e50dd 100644
--- a/arch/arm/kernel/setup.c
+++ b/arch/arm/kernel/setup.c
@@ -770,10 +770,10 @@ void __init setup_arch(char **cmdline_p)
 	psci_init();
 #ifdef CONFIG_SMP
 	if (is_smp()) {
-		if (mdesc->smp)
-			smp_set_ops(mdesc->smp);
-		else if (psci_smp_available())
+		if (psci_smp_available())
 			smp_set_ops(&psci_smp_ops);
+		else
+			smp_set_ops(mdesc->smp);
 		smp_init_cpus();
 	}
 #endif
-- 
1.7.2.5

On Fri, 29 Mar 2013, Stefano Stabellini wrote:
> If PSCI initializes correctly and PSCI SMP operations are available, use
them.
> This is required for SMP support in Dom0 on Xen.
> 
> Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
> CC: will.deacon@arm.com
> CC: arnd@arndb.de
> CC: marc.zyngier@arm.com
> CC: linux@arm.linux.org.uk
> CC: nico@linaro.org
I''d suggest you also include in your series the patch I posted earlier 
providing a runtime mdesc->smp_init method as well.  This way the 
priority order would be:

- If mdesc->smp_init is non null then use that.

- Otherwise, if PSCI is available then use that.

- Otherwise use mdesc->smp.

This way, if the PSCI default has to be overriden (like in the MCPM case 
because it needs to wrap PSCI itself, or to cover Rob''s concern) then 
this can be achieved at run time on a per mdesc basis.


Nicolas

On Fri, 29 Mar 2013, Nicolas Pitre wrote:
> On Fri, 29 Mar 2013, Stefano Stabellini wrote:
> 
> > If PSCI initializes correctly and PSCI SMP operations are available,
use them.
> > This is required for SMP support in Dom0 on Xen.
> > 
> > Signed-off-by: Stefano Stabellini
<stefano.stabellini@eu.citrix.com>
> > CC: will.deacon@arm.com
> > CC: arnd@arndb.de
> > CC: marc.zyngier@arm.com
> > CC: linux@arm.linux.org.uk
> > CC: nico@linaro.org
> 
> I''d suggest you also include in your series the patch I posted
earlier
> providing a runtime mdesc->smp_init method as well.
OK.

> This way the 
> priority order would be:
> 
> - If mdesc->smp_init is non null then use that.
> 
> - Otherwise, if PSCI is available then use that.
> 
> - Otherwise use mdesc->smp.
> 
> This way, if the PSCI default has to be overriden (like in the MCPM case 
> because it needs to wrap PSCI itself, or to cover Rob''s concern)
then
> this can be achieved at run time on a per mdesc basis.
Actually that''s not a bad idea, it could make everybody happy.
What about the following, in this precise order:

- if a xen hypervisor node is present on device tree, use PSCI;
- otherwise if mdesc->smp_init is non null then use it;
- otherwise if PSCI is available then use it;
- otherwise use mdesc->smp.

It''s the most practical solution to satisfy everybody''s needs.

On Fri, 29 Mar 2013, Stefano Stabellini wrote:
> On Fri, 29 Mar 2013, Nicolas Pitre wrote:
> > On Fri, 29 Mar 2013, Stefano Stabellini wrote:
> > 
> > > If PSCI initializes correctly and PSCI SMP operations are
available, use them.
> > > This is required for SMP support in Dom0 on Xen.
> > > 
> > > Signed-off-by: Stefano Stabellini
<stefano.stabellini@eu.citrix.com>
> > > CC: will.deacon@arm.com
> > > CC: arnd@arndb.de
> > > CC: marc.zyngier@arm.com
> > > CC: linux@arm.linux.org.uk
> > > CC: nico@linaro.org
> > 
> > I''d suggest you also include in your series the patch I
posted earlier
> > providing a runtime mdesc->smp_init method as well.
> 
> OK.
> 
> 
> > This way the 
> > priority order would be:
> > 
> > - If mdesc->smp_init is non null then use that.
> > 
> > - Otherwise, if PSCI is available then use that.
> > 
> > - Otherwise use mdesc->smp.
> > 
> > This way, if the PSCI default has to be overriden (like in the MCPM
case
> > because it needs to wrap PSCI itself, or to cover Rob''s
concern) then
> > this can be achieved at run time on a per mdesc basis.
> 
> Actually that''s not a bad idea, it could make everybody happy.
> What about the following, in this precise order:
> 
> - if a xen hypervisor node is present on device tree, use PSCI;
> - otherwise if mdesc->smp_init is non null then use it;
> - otherwise if PSCI is available then use it;
> - otherwise use mdesc->smp.
> 
> It''s the most practical solution to satisfy everybody''s
needs.
Maybe I''m missing something obvious, but why can''t xen declare
a mdesc
of its own?  Given it is going to tweak the DT passed to the kernel 
anyway that shouldn''t be a problem.

That would be more eleguant than adding xen exception hooks in generic 
code.


Nicolas

On Fri, 29 Mar 2013, Stefano Stabellini wrote:
> On Fri, 29 Mar 2013, Nicolas Pitre wrote:
> > This way the 
> > priority order would be:
> > 
> > - If mdesc->smp_init is non null then use that.
> > 
> > - Otherwise, if PSCI is available then use that.
> > 
> > - Otherwise use mdesc->smp.
> > 
> > This way, if the PSCI default has to be overriden (like in the MCPM
case
> > because it needs to wrap PSCI itself, or to cover Rob''s
concern) then
> > this can be achieved at run time on a per mdesc basis.
> 
> Actually that''s not a bad idea, it could make everybody happy.
> What about the following, in this precise order:
> 
> - if a xen hypervisor node is present on device tree, use PSCI;
> - otherwise if mdesc->smp_init is non null then use it;
> - otherwise if PSCI is available then use it;
> - otherwise use mdesc->smp.
> 
> It''s the most practical solution to satisfy everybody''s
needs.
Regardless of my previous email suggesting a mdesc for xen, I still 
don''t understand why you need this absolute priority for Xen. 
Isn''t my
original suggestion sufficient?

The likely reason why mdesc->smp_init might be needed is to provide an 
extra encapsulation layer before actually using PSCI instead of using it 
directly.  Why would you need to bypass that?


Nicolas

On Fri, 29 Mar 2013, Nicolas Pitre wrote:
> On Fri, 29 Mar 2013, Stefano Stabellini wrote:
> 
> > On Fri, 29 Mar 2013, Nicolas Pitre wrote:
> > > On Fri, 29 Mar 2013, Stefano Stabellini wrote:
> > > 
> > > > If PSCI initializes correctly and PSCI SMP operations are
available, use them.
> > > > This is required for SMP support in Dom0 on Xen.
> > > > 
> > > > Signed-off-by: Stefano Stabellini
<stefano.stabellini@eu.citrix.com>
> > > > CC: will.deacon@arm.com
> > > > CC: arnd@arndb.de
> > > > CC: marc.zyngier@arm.com
> > > > CC: linux@arm.linux.org.uk
> > > > CC: nico@linaro.org
> > > 
> > > I''d suggest you also include in your series the patch I
posted earlier
> > > providing a runtime mdesc->smp_init method as well.
> > 
> > OK.
> > 
> > 
> > > This way the 
> > > priority order would be:
> > > 
> > > - If mdesc->smp_init is non null then use that.
> > > 
> > > - Otherwise, if PSCI is available then use that.
> > > 
> > > - Otherwise use mdesc->smp.
> > > 
> > > This way, if the PSCI default has to be overriden (like in the
MCPM case
> > > because it needs to wrap PSCI itself, or to cover Rob''s
concern) then
> > > this can be achieved at run time on a per mdesc basis.
> > 
> > Actually that''s not a bad idea, it could make everybody
happy.
> > What about the following, in this precise order:
> > 
> > - if a xen hypervisor node is present on device tree, use PSCI;
> > - otherwise if mdesc->smp_init is non null then use it;
> > - otherwise if PSCI is available then use it;
> > - otherwise use mdesc->smp.
> > 
> > It''s the most practical solution to satisfy
everybody''s needs.
> 
> Maybe I''m missing something obvious, but why can''t xen
declare a mdesc
> of its own?  Given it is going to tweak the DT passed to the kernel 
> anyway that shouldn''t be a problem.
> 
> That would be more eleguant than adding xen exception hooks in generic 
> code.
I guess I could introduce a Xen-enabled mdesc for all the platforms we intend to
support, but as the list of boards with virtualization extensions grow,
also the list of Xen-enabled mdescs would grow accordingly.
We would end up with dozen (more?) of Xen mdescs.

On Fri, 29 Mar 2013, Nicolas Pitre wrote:
> On Fri, 29 Mar 2013, Stefano Stabellini wrote:
> 
> > On Fri, 29 Mar 2013, Nicolas Pitre wrote:
> > > This way the 
> > > priority order would be:
> > > 
> > > - If mdesc->smp_init is non null then use that.
> > > 
> > > - Otherwise, if PSCI is available then use that.
> > > 
> > > - Otherwise use mdesc->smp.
> > > 
> > > This way, if the PSCI default has to be overriden (like in the
MCPM case
> > > because it needs to wrap PSCI itself, or to cover Rob''s
concern) then
> > > this can be achieved at run time on a per mdesc basis.
> > 
> > Actually that''s not a bad idea, it could make everybody
happy.
> > What about the following, in this precise order:
> > 
> > - if a xen hypervisor node is present on device tree, use PSCI;
> > - otherwise if mdesc->smp_init is non null then use it;
> > - otherwise if PSCI is available then use it;
> > - otherwise use mdesc->smp.
> > 
> > It''s the most practical solution to satisfy
everybody''s needs.
> 
> Regardless of my previous email suggesting a mdesc for xen, I still 
> don''t understand why you need this absolute priority for Xen. 
Isn''t my
> original suggestion sufficient?
> 
> The likely reason why mdesc->smp_init might be needed is to provide an 
> extra encapsulation layer before actually using PSCI instead of using it 
> directly.  Why would you need to bypass that?
Uhm.. maybe I wouldn''t, I am not 100% sure TBH.

I am expecting that this "extra encapsulation layer" would read/write
some platform specific registers that Xen doesn''t export.
If this is the case, then we would need to bypass it.

But if it''s just to mangle parameters, then it should be OK.

On 03/29/2013 12:53 PM, Nicolas Pitre wrote:
> On Fri, 29 Mar 2013, Stefano Stabellini wrote:
> 
>> On Fri, 29 Mar 2013, Nicolas Pitre wrote:
>>> On Fri, 29 Mar 2013, Stefano Stabellini wrote:
>>>
>>>> If PSCI initializes correctly and PSCI SMP operations are
available, use them.
>>>> This is required for SMP support in Dom0 on Xen.
>>>>
>>>> Signed-off-by: Stefano Stabellini
<stefano.stabellini@eu.citrix.com>
>>>> CC: will.deacon@arm.com
>>>> CC: arnd@arndb.de
>>>> CC: marc.zyngier@arm.com
>>>> CC: linux@arm.linux.org.uk
>>>> CC: nico@linaro.org
>>>
>>> I''d suggest you also include in your series the patch I
posted earlier
>>> providing a runtime mdesc->smp_init method as well.
>>
>> OK.
>>
>>
>>> This way the 
>>> priority order would be:
>>>
>>> - If mdesc->smp_init is non null then use that.
>>>
>>> - Otherwise, if PSCI is available then use that.
>>>
>>> - Otherwise use mdesc->smp.
>>>
>>> This way, if the PSCI default has to be overriden (like in the MCPM
case
>>> because it needs to wrap PSCI itself, or to cover Rob''s
concern) then
>>> this can be achieved at run time on a per mdesc basis.
>>
>> Actually that''s not a bad idea, it could make everybody happy.
>> What about the following, in this precise order:
>>
>> - if a xen hypervisor node is present on device tree, use PSCI;
>> - otherwise if mdesc->smp_init is non null then use it;
>> - otherwise if PSCI is available then use it;
>> - otherwise use mdesc->smp.
>>
>> It''s the most practical solution to satisfy
everybody''s needs.
> 
> Maybe I''m missing something obvious, but why can''t xen
declare a mdesc
> of its own?  Given it is going to tweak the DT passed to the kernel 
> anyway that shouldn''t be a problem.
Xen does have it''s own mdesc. It is (or will be) mach-virt, but that is
only for DomU guests. For Dom0, you still need all the platform specific
code except smp_ops. However, I''m doubtful this would work without
other
changes on more complicated platforms like OMAP.

I would say wait to add this until you have platforms that actually need
the first case.

Rob
> 
> That would be more eleguant than adding xen exception hooks in generic 
> code.
> 
> 
> Nicolas
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>

On Fri, 29 Mar 2013, Rob Herring wrote:
> On 03/29/2013 12:53 PM, Nicolas Pitre wrote:
> > On Fri, 29 Mar 2013, Stefano Stabellini wrote:
> > 
> >> On Fri, 29 Mar 2013, Nicolas Pitre wrote:
> >>> On Fri, 29 Mar 2013, Stefano Stabellini wrote:
> >>>
> >>>> If PSCI initializes correctly and PSCI SMP operations are
available, use them.
> >>>> This is required for SMP support in Dom0 on Xen.
> >>>>
> >>>> Signed-off-by: Stefano Stabellini
<stefano.stabellini@eu.citrix.com>
> >>>> CC: will.deacon@arm.com
> >>>> CC: arnd@arndb.de
> >>>> CC: marc.zyngier@arm.com
> >>>> CC: linux@arm.linux.org.uk
> >>>> CC: nico@linaro.org
> >>>
> >>> I''d suggest you also include in your series the patch
I posted earlier
> >>> providing a runtime mdesc->smp_init method as well.
> >>
> >> OK.
> >>
> >>
> >>> This way the 
> >>> priority order would be:
> >>>
> >>> - If mdesc->smp_init is non null then use that.
> >>>
> >>> - Otherwise, if PSCI is available then use that.
> >>>
> >>> - Otherwise use mdesc->smp.
> >>>
> >>> This way, if the PSCI default has to be overriden (like in the
MCPM case
> >>> because it needs to wrap PSCI itself, or to cover
Rob''s concern) then
> >>> this can be achieved at run time on a per mdesc basis.
> >>
> >> Actually that''s not a bad idea, it could make everybody
happy.
> >> What about the following, in this precise order:
> >>
> >> - if a xen hypervisor node is present on device tree, use PSCI;
> >> - otherwise if mdesc->smp_init is non null then use it;
> >> - otherwise if PSCI is available then use it;
> >> - otherwise use mdesc->smp.
> >>
> >> It''s the most practical solution to satisfy
everybody''s needs.
> > 
> > Maybe I''m missing something obvious, but why can''t
xen declare a mdesc
> > of its own?  Given it is going to tweak the DT passed to the kernel 
> > anyway that shouldn''t be a problem.
> 
> Xen does have it''s own mdesc. It is (or will be) mach-virt, but
that is
> only for DomU guests. For Dom0, you still need all the platform specific
> code except smp_ops. However, I''m doubtful this would work without
other
> changes on more complicated platforms like OMAP.
> 
> I would say wait to add this until you have platforms that actually need
> the first case.
OK, that is not unreasonable.

What are the platforms that are going to use smp_init? Do we know how do
they intend to use it?

On Mon, 1 Apr 2013, Stefano Stabellini wrote:
> On Fri, 29 Mar 2013, Rob Herring wrote:
> > On 03/29/2013 12:53 PM, Nicolas Pitre wrote:
> > > On Fri, 29 Mar 2013, Stefano Stabellini wrote:
> > > 
> > >> On Fri, 29 Mar 2013, Nicolas Pitre wrote:
> > >>> On Fri, 29 Mar 2013, Stefano Stabellini wrote:
> > >>>
> > >>>> If PSCI initializes correctly and PSCI SMP operations
are available, use them.
> > >>>> This is required for SMP support in Dom0 on Xen.
> > >>>>
> > >>>> Signed-off-by: Stefano Stabellini
<stefano.stabellini@eu.citrix.com>
> > >>>> CC: will.deacon@arm.com
> > >>>> CC: arnd@arndb.de
> > >>>> CC: marc.zyngier@arm.com
> > >>>> CC: linux@arm.linux.org.uk
> > >>>> CC: nico@linaro.org
> > >>>
> > >>> I''d suggest you also include in your series the
patch I posted earlier
> > >>> providing a runtime mdesc->smp_init method as well.
> > >>
> > >> OK.
> > >>
> > >>
> > >>> This way the 
> > >>> priority order would be:
> > >>>
> > >>> - If mdesc->smp_init is non null then use that.
> > >>>
> > >>> - Otherwise, if PSCI is available then use that.
> > >>>
> > >>> - Otherwise use mdesc->smp.
> > >>>
> > >>> This way, if the PSCI default has to be overriden (like
in the MCPM case
> > >>> because it needs to wrap PSCI itself, or to cover
Rob''s concern) then
> > >>> this can be achieved at run time on a per mdesc basis.
> > >>
> > >> Actually that''s not a bad idea, it could make
everybody happy.
> > >> What about the following, in this precise order:
> > >>
> > >> - if a xen hypervisor node is present on device tree, use
PSCI;
> > >> - otherwise if mdesc->smp_init is non null then use it;
> > >> - otherwise if PSCI is available then use it;
> > >> - otherwise use mdesc->smp.
> > >>
> > >> It''s the most practical solution to satisfy
everybody''s needs.
> > > 
> > > Maybe I''m missing something obvious, but why
can''t xen declare a mdesc
> > > of its own?  Given it is going to tweak the DT passed to the
kernel
> > > anyway that shouldn''t be a problem.
> > 
> > Xen does have it''s own mdesc. It is (or will be) mach-virt,
but that is
> > only for DomU guests. For Dom0, you still need all the platform
specific
> > code except smp_ops. However, I''m doubtful this would work
without other
> > changes on more complicated platforms like OMAP.
> > 
> > I would say wait to add this until you have platforms that actually
need
> > the first case.
> 
> OK, that is not unreasonable.
> 
> What are the platforms that are going to use smp_init? Do we know how do
> they intend to use it?
VExpress for one.  When booting on a big.LITTLE system such as TC2 on 
VExpress, the MCPM layer needs to arbitrate power management operations 
on a per cluster basis.  In that case there is a MCPM specific set of 
SMP ops to be used, even if it may end up calling into PSCI.

But the important point is that we don''t know beforehand what to use, 
especially with a kernel that can boot on multiple different VExpress 
configurations.  The decision has to be made at run time, and therefore 
a static default or mdesc->smp ops doesn''t cut it.


Nicolas

On Mon, 1 Apr 2013, Nicolas Pitre wrote:
> On Mon, 1 Apr 2013, Stefano Stabellini wrote:
> 
> > On Fri, 29 Mar 2013, Rob Herring wrote:
> > > On 03/29/2013 12:53 PM, Nicolas Pitre wrote:
> > > > On Fri, 29 Mar 2013, Stefano Stabellini wrote:
> > > > 
> > > >> On Fri, 29 Mar 2013, Nicolas Pitre wrote:
> > > >>> On Fri, 29 Mar 2013, Stefano Stabellini wrote:
> > > >>>
> > > >>>> If PSCI initializes correctly and PSCI SMP
operations are available, use them.
> > > >>>> This is required for SMP support in Dom0 on Xen.
> > > >>>>
> > > >>>> Signed-off-by: Stefano Stabellini
<stefano.stabellini@eu.citrix.com>
> > > >>>> CC: will.deacon@arm.com
> > > >>>> CC: arnd@arndb.de
> > > >>>> CC: marc.zyngier@arm.com
> > > >>>> CC: linux@arm.linux.org.uk
> > > >>>> CC: nico@linaro.org
> > > >>>
> > > >>> I''d suggest you also include in your series
the patch I posted earlier
> > > >>> providing a runtime mdesc->smp_init method as
well.
> > > >>
> > > >> OK.
> > > >>
> > > >>
> > > >>> This way the 
> > > >>> priority order would be:
> > > >>>
> > > >>> - If mdesc->smp_init is non null then use that.
> > > >>>
> > > >>> - Otherwise, if PSCI is available then use that.
> > > >>>
> > > >>> - Otherwise use mdesc->smp.
> > > >>>
> > > >>> This way, if the PSCI default has to be overriden
(like in the MCPM case
> > > >>> because it needs to wrap PSCI itself, or to cover
Rob''s concern) then
> > > >>> this can be achieved at run time on a per mdesc
basis.
> > > >>
> > > >> Actually that''s not a bad idea, it could make
everybody happy.
> > > >> What about the following, in this precise order:
> > > >>
> > > >> - if a xen hypervisor node is present on device tree,
use PSCI;
> > > >> - otherwise if mdesc->smp_init is non null then use
it;
> > > >> - otherwise if PSCI is available then use it;
> > > >> - otherwise use mdesc->smp.
> > > >>
> > > >> It''s the most practical solution to satisfy
everybody''s needs.
> > > > 
> > > > Maybe I''m missing something obvious, but why
can''t xen declare a mdesc
> > > > of its own?  Given it is going to tweak the DT passed to the
kernel
> > > > anyway that shouldn''t be a problem.
> > > 
> > > Xen does have it''s own mdesc. It is (or will be)
mach-virt, but that is
> > > only for DomU guests. For Dom0, you still need all the platform
specific
> > > code except smp_ops. However, I''m doubtful this would
work without other
> > > changes on more complicated platforms like OMAP.
> > > 
> > > I would say wait to add this until you have platforms that
actually need
> > > the first case.
> > 
> > OK, that is not unreasonable.
> > 
> > What are the platforms that are going to use smp_init? Do we know how
do
> > they intend to use it?
> 
> VExpress for one.  When booting on a big.LITTLE system such as TC2 on 
> VExpress, the MCPM layer needs to arbitrate power management operations 
> on a per cluster basis.  In that case there is a MCPM specific set of 
> SMP ops to be used, even if it may end up calling into PSCI.
> 
> But the important point is that we don''t know beforehand what to
use,
> especially with a kernel that can boot on multiple different VExpress 
> configurations.  The decision has to be made at run time, and therefore 
> a static default or mdesc->smp ops doesn''t cut it.
I certainly like the principle and I am in favor of anything that moves
the decisions at runtime. I have pulled the patch in the series, it''s
going to be in the next version.

However I am concerned that these platform specific operations won''t
work with Xen at all.
I am getting increasingly certain that we need a Xen specific check in
setup_arch to bump up of the priority of PSCI over anything else if Xen
is running.

On Tue, 2 Apr 2013, Stefano Stabellini wrote:
> On Mon, 1 Apr 2013, Nicolas Pitre wrote:
> > On Mon, 1 Apr 2013, Stefano Stabellini wrote:
> > > What are the platforms that are going to use smp_init? Do we know
how do
> > > they intend to use it?
> > 
> > VExpress for one.  When booting on a big.LITTLE system such as TC2 on 
> > VExpress, the MCPM layer needs to arbitrate power management
operations
> > on a per cluster basis.  In that case there is a MCPM specific set of 
> > SMP ops to be used, even if it may end up calling into PSCI.
> > 
> > But the important point is that we don''t know beforehand what
to use,
> > especially with a kernel that can boot on multiple different VExpress 
> > configurations.  The decision has to be made at run time, and
therefore
> > a static default or mdesc->smp ops doesn''t cut it.
> 
> I certainly like the principle and I am in favor of anything that moves
> the decisions at runtime. I have pulled the patch in the series,
it''s
> going to be in the next version.
> 
> However I am concerned that these platform specific operations
won''t
> work with Xen at all.
> I am getting increasingly certain that we need a Xen specific check in
> setup_arch to bump up of the priority of PSCI over anything else if Xen
> is running. 
I''m concerned about mixing big.LITTLE and Xen as well.  I
don''t think
this is going to make an easy match.  KVM might have an easier fit here.

But, in any case, even if the MCPM layer gets involved, if Xen is there 
then PSCI will end up being the ultimate interface anyway.

But let''s cross that bridge when we get to it.  For now this is still a
non existing problem.


Nicolas

On Tue, Apr 02, 2013 at 12:11:25PM -0400, Nicolas Pitre
wrote:
> On Tue, 2 Apr 2013, Stefano Stabellini wrote:
> 
> > On Mon, 1 Apr 2013, Nicolas Pitre wrote:
> > > On Mon, 1 Apr 2013, Stefano Stabellini wrote:
> > > > What are the platforms that are going to use smp_init? Do we
know how do
> > > > they intend to use it?
> > > 
> > > VExpress for one.  When booting on a big.LITTLE system such as
TC2 on
> > > VExpress, the MCPM layer needs to arbitrate power management
operations
> > > on a per cluster basis.  In that case there is a MCPM specific
set of
> > > SMP ops to be used, even if it may end up calling into PSCI.
> > > 
> > > But the important point is that we don''t know beforehand
what to use,
> > > especially with a kernel that can boot on multiple different
VExpress
> > > configurations.  The decision has to be made at run time, and
therefore
> > > a static default or mdesc->smp ops doesn''t cut it.
> > 
> > I certainly like the principle and I am in favor of anything that
moves
> > the decisions at runtime. I have pulled the patch in the series,
it''s
> > going to be in the next version.
> > 
> > However I am concerned that these platform specific operations
won''t
> > work with Xen at all.
> > I am getting increasingly certain that we need a Xen specific check in
> > setup_arch to bump up of the priority of PSCI over anything else if
Xen
> > is running. 
> 
> I''m concerned about mixing big.LITTLE and Xen as well.  I
don''t think
> this is going to make an easy match.  KVM might have an easier fit here.
> 
> But, in any case, even if the MCPM layer gets involved, if Xen is there 
> then PSCI will end up being the ultimate interface anyway.
Note that big.LITTLE != MCPM.  Virtualisation hosts might be large multi-
cluster systems, but the CPUs might be all of the same type.  MCPM or
similar would me needed for the multi-cluster power management even
though there is no big.LITTLE mix of CPUs.
> But let''s cross that bridge when we get to it.  For now this is
still a
> non existing problem.
That''s a big open question.  Either the host or hypervisor needs to be
very clever about scheduling guests, or you need to bind each guest virtual
CPU to a specific class of physical CPUs -- so, for example you provide
a guest with an explicit mix of bigs and littles.

All we can say about that for now is that it''s a potential research
area...

Cheers
---Dave

On Tue, 9 Apr 2013, Dave Martin wrote:
> On Tue, Apr 02, 2013 at 12:11:25PM -0400, Nicolas Pitre wrote:
> > I''m concerned about mixing big.LITTLE and Xen as well.  I
don''t think
> > this is going to make an easy match.  KVM might have an easier fit
here.
> > 
> > But, in any case, even if the MCPM layer gets involved, if Xen is
there
> > then PSCI will end up being the ultimate interface anyway.
> 
> Note that big.LITTLE != MCPM.  Virtualisation hosts might be large multi-
> cluster systems, but the CPUs might be all of the same type.  MCPM or
> similar would me needed for the multi-cluster power management even
> though there is no big.LITTLE mix of CPUs.
Absolutely!  But in this case, there is no need for Xen to learn about 
the computing capacity differences between different CPU sets.

What I wanted to emphasize is the fact that, if Xen decides to expose a 
b.L topology to guests, then those guests must be b.L aware to make good 
scheduling decisions, etc.  Initially I suspect that guests will be 
confined to the same sets of CPUs to simplify things.  Or it could even 
migrate a guest between little and big CPUs like the switcher does.  
But a single guest probably won''t span different CPU classes 
simultaneously initially.  And therefore it is unlikely that MCPM will 
be active inside a Xen guest for quite a while.
> > But let''s cross that bridge when we get to it.  For now this
is still a
> > non existing problem.
> 
> That''s a big open question.  Either the host or hypervisor needs
to be
> very clever about scheduling guests, or you need to bind each guest virtual
> CPU to a specific class of physical CPUs -- so, for example you provide
> a guest with an explicit mix of bigs and littles.
> 
> All we can say about that for now is that it''s a potential
research area...
Absolutely.


Nicolas