Hi Ian, I try compile the bootwrapper for cubieboard2 like this: joshzhao@joshzhao-ThinkCentre-M58p:~/project/Xen/A20/boot-wrapper$ make cubieboard2_defconfig CROSS_COMPILE=arm-linux-gnueabihf- # # configuration written to .config # joshzhao@joshzhao-ThinkCentre-M58p:~/project/Xen/A20/boot-wrapper$ make CROSS_COMPILE=arm-linux-gnueabihf- make -C scripts/kconfig -f Makefile.bootwrapper silentoldconfig cd ../../ && scripts/kconfig/conf --silentoldconfig /home/joshzhao/project/Xen/A20/boot-wrapper/Kconfig make: Nothing to be done for `_all''. joshzhao@joshzhao-ThinkCentre-M58p:~/project/Xen/A20/boot-wrapper$
On Fri, 2013-08-30 at 15:12 +0800, Josh Zhao wrote:> Hi Ian, > I try compile the bootwrapper for cubieboard2 like this:Hi Josh, cubieboards2 support is not really in place yet, so I''m afraid there will currently be rough edges like this. That said I think this worked for me, but based on the discussions I think the better approach is to merge Andre''s u-boot patches to support entry in HYP mode into the cubieboard branch. I''ll only work on bootwrapper any further if some interesting platform comes along for which there is no possibility of updating the bootloader, so for now you should consider it dead/dormant. Anyhow, I''ve tried inserting Andre''s patches into the cubieboard2 uboot and stumbled across a problem -- it turns out that u-boot has enabled the MMU (SCTLR.M), apparently because it is required for the caches to be on (see arch/arm/lib/cache-cp15.c:cache_enable). unfortunately SCTLR is banked so when nonsec-virt.S:_secure_monitor enabled SCR.NS everything falls apart. Naïvely enabling SCTLR.M while in monitor mode didn''t seem to help. Calling cleanup_before_linux() early in armv7_switch_nonsec() seemed to do the job -- Andre, do you think that is the right fix? Not sure why this didn''t affect vexpress etc -- perhaps caches aren''t enabled for that platform? Even with this fixed things don''t quite work for me, I don''t quite remember how/why (I last looked at this 7 days ago). I''ve been trying to set aside Friday as "cubieboard hacking day", but I''ve ended up with a bunch of meetings today, hopefully I''ll still manage to investigate a bit. Ian.
On 08/30/2013 10:06 AM, Ian Campbell wrote:> On Fri, 2013-08-30 at 15:12 +0800, Josh Zhao wrote: >> Hi Ian, >> I try compile the bootwrapper for cubieboard2 like this: > > Hi Josh, > > cubieboards2 support is not really in place yet, so I''m afraid there > will currently be rough edges like this. > > That said I think this worked for me, but based on the discussions I > think the better approach is to merge Andre''s u-boot patches to support > entry in HYP mode into the cubieboard branch. I''ll only work on > bootwrapper any further if some interesting platform comes along for > which there is no possibility of updating the bootloader, so for now you > should consider it dead/dormant. > > Anyhow, I''ve tried inserting Andre''s patches into the cubieboard2 uboot > and stumbled across a problem -- it turns out that u-boot has enabled > the MMU (SCTLR.M), apparently because it is required for the caches to > be on (see arch/arm/lib/cache-cp15.c:cache_enable). unfortunately SCTLR > is banked so when nonsec-virt.S:_secure_monitor enabled SCR.NS > everything falls apart. Naïvely enabling SCTLR.M while in monitor mode > didn''t seem to help.Ah, good point. My whole code approach was relying on the fact that u-boot runs with MMU _off_. With the MMU differences in HYP mode we may run into troubles here, it may work if we switch late enough, though.> Calling cleanup_before_linux() early in armv7_switch_nonsec() seemed to > do the job -- Andre, do you think that is the right fix?Do you have a pointer to a u-boot repository with the cubieboard patches? Then I''d take a look.> Not sure why this didn''t affect vexpress etc -- perhaps caches aren''t > enabled for that platform?AFAIK it''s rather normal for u-boot to run with at least some caches off. These only get turned on if really necessary. So I''d refrain from enabling the MMU just to support some cache levels on a bootloader. Thanks for looking at this! Andre.> Even with this fixed things don''t quite work for me, I don''t quite > remember how/why (I last looked at this 7 days ago). I''ve been trying to > set aside Friday as "cubieboard hacking day", but I''ve ended up with a > bunch of meetings today, hopefully I''ll still manage to investigate a > bit. > > Ian. > >
On Fri, 2013-08-30 at 10:27 +0200, Andre Przywara wrote:> On 08/30/2013 10:06 AM, Ian Campbell wrote: > > On Fri, 2013-08-30 at 15:12 +0800, Josh Zhao wrote: > >> Hi Ian, > >> I try compile the bootwrapper for cubieboard2 like this: > > > > Hi Josh, > > > > cubieboards2 support is not really in place yet, so I'm afraid there > > will currently be rough edges like this. > > > > That said I think this worked for me, but based on the discussions I > > think the better approach is to merge Andre's u-boot patches to support > > entry in HYP mode into the cubieboard branch. I'll only work on > > bootwrapper any further if some interesting platform comes along for > > which there is no possibility of updating the bootloader, so for now you > > should consider it dead/dormant. > > > > Anyhow, I've tried inserting Andre's patches into the cubieboard2 uboot > > and stumbled across a problem -- it turns out that u-boot has enabled > > the MMU (SCTLR.M), apparently because it is required for the caches to > > be on (see arch/arm/lib/cache-cp15.c:cache_enable). unfortunately SCTLR > > is banked so when nonsec-virt.S:_secure_monitor enabled SCR.NS > > everything falls apart. Naïvely enabling SCTLR.M while in monitor mode > > didn't seem to help. > > Ah, good point. My whole code approach was relying on the fact that > u-boot runs with MMU _off_. With the MMU differences in HYP mode we may > run into troubles here, it may work if we switch late enough, though.This new "Falcon mode" thing seems to be driving some changes here, in particular it seems that turning on the caches can dramatically speed up boot in some circumstance.> > > Calling cleanup_before_linux() early in armv7_switch_nonsec() seemed to > > do the job -- Andre, do you think that is the right fix? > > Do you have a pointer to a u-boot repository with the cubieboard > patches? Then I'd take a look.https://github.com/hno/u-boot.git branch wip/a20 specifically 80fd9a5c5b87ba2f48f4a71b666839870e780be6 and your hypmode v4 5068b337518617586f2e51b6d616c54dbec4fa62 It looks like I should switch to the sunxi branch of https://github.com/linux-sunxi/u-boot-sunxi.git though, it includes the above and is more up to date. I'll try that shortly and let you know how I get on. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Fri, 2013-08-30 at 10:51 +0100, Ian Campbell wrote:> I''ll try that shortly and let you know how I get on.Success! My failure previously was not getting the kernel command line in the right place, so no printk... Starting from https://github.com/linux-sunxi/u-boot-sunxi.git sunxi I merged your stuff: git pull git://git.linaro.org/people/aprzywara/u-boot.git hypmode_v4 and applied the patch below (NB: u-boot doesn''t support SMP on this platform yet, hence stubbing out the kicking of secondaries etc) and https://groups.google.com/forum/#!topic/linux-sunxi/7oS0CM2WSy4. Then built with: $ make Cubieboard2_FEL CROSS_COMPILE=arm-linux-gnueabihf- Then I could boot via FEL (http://linux-sunxi.org/FEL/USBBoot) using usb-boot from git://github.com/linux-sunxi/sunxi-tools.git: usb-boot ~/devel/u-boot.git/spl/u-boot-spl.bin ~/devel/u-boot.git/u-boot.bin uImage where uImage is the zImage+dtb (cat into tmp) and mkimage -A arm -O linux -T kernel -a 0x40008000 -e 0x40008000 -C none -d tmp uImage Linux says: [ 0.000000] CPU: All CPU(s) started in HYP mode. [ 0.000000] CPU: Virtualization extensions available. (All == all 1) Kernel is https://github.com/mripard/linux.git sunxi-next-a20-cubie2 multi_v7_defconfig + CONFIG_DEBUG_LL w/ CONFIG_DEBUG_SUNXI_UART0 and CONFIG_EARLY_PRINTK. I used appended dtb and set /chosen/bootargs = "console=ttyS0,115200 earlyprintk" in arch/arm/boot/dts/sun7i-a20-cubieboard2.dts (only because usb-boot doesn''t happen to load dtb and pass it to bootm, will fix that up...) Now to try Xen! Ian. commit 9267730ab1c3b6be5cf9118b2ce4e708e9d4ba66 Author: Ian Campbell <ijc@hellion.org.uk> Date: Fri Aug 30 13:52:07 2013 +0100 sunxi: Enable hyp mode boot on sunxi platforms Required a call to cleanup_before_linux in order to disable caches before switching to NS world, otherwise we switch to an incorrecly configured banked version of SCTLR. Signed-off-by: Ian Campbell <ijc@hellion.org.uk> diff --git a/arch/arm/cpu/armv7/virt-v7.c b/arch/arm/cpu/armv7/virt-v7.c index 4dc8c45..a76dcf7 100644 --- a/arch/arm/cpu/armv7/virt-v7.c +++ b/arch/arm/cpu/armv7/virt-v7.c @@ -132,6 +132,8 @@ int armv7_switch_nonsec(void) unsigned int reg; unsigned itlinesnr, i; + cleanup_before_linux(); + /* check whether the CPU supports the security extensions */ reg = read_id_pfr1(); if ((reg & 0xF0) == 0) { diff --git a/board/sunxi/Makefile b/board/sunxi/Makefile index 4780c99..d756091 100644 --- a/board/sunxi/Makefile +++ b/board/sunxi/Makefile @@ -76,9 +76,13 @@ COBJS-$(CONFIG_UHOST_U1A) += dram_sun4i_360_1024_iow8.o COBJS-$(CONFIG_WOBO_I5) += dram_wobo_i5.o COBJS-$(CONFIG_XZPAD700) += dram_xzpad700.o +ifneq ($(CONFIG_ARMV7_NONSEC)$(CONFIG_ARMV7_VIRT),) +SOBJS := smp.o +endif + COBJS := $(COBJS-y) -SRCS := $(COBJS:.o=.c) -OBJS := $(addprefix $(obj),$(COBJS)) +SRCS := $(COBJS:.o=.c) $(SOBJS:.o=.S) +OBJS := $(addprefix $(obj),$(COBJS) $(SOBJS)) all: $(LIB) diff --git a/board/sunxi/board.c b/board/sunxi/board.c index 50c07e2..4b8c732 100755 --- a/board/sunxi/board.c +++ b/board/sunxi/board.c @@ -40,6 +40,20 @@ DECLARE_GLOBAL_DATA_PTR; +/* Setting the address at which secondary cores start from. + * SMP currently unsupported. + */ +void smp_set_core_boot_addr(unsigned long addr, int corenr) +{ + printf("WARNING: ignoring attempt to set core boot address %lx on core %d\n", + addr, corenr); +} + +void smp_kick_all_cpus(void) +{ + printf("WARNING: Not kicking secondary CPUs\n"); +} + /* add board specific code here */ int board_init(void) { diff --git a/board/sunxi/smp.S b/board/sunxi/smp.S new file mode 100644 index 0000000..6a6e08c --- /dev/null +++ b/board/sunxi/smp.S @@ -0,0 +1,33 @@ +/* + * code for redirecting secondary cores to their start address + * + * Copyright (c) 2013 Andre Przywara <andre.przywara@linaro.org> + * + * See file CREDITS for list of people who contributed to this + * project. + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License as + * published by the Free Software Foundation; either version 2 of + * the License, or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place, Suite 330, Boston, + * MA 02111-1307 USA + */ + +#include <config.h> +#include <linux/linkage.h> + +/* void _smp_waitloop(unsigned previous_address); */ +ENTRY(smp_waitloop) + /* SMP currently unsupported */ +1: wfi + b 1b +ENDPROC(smp_waitloop) diff --git a/include/configs/sun7i.h b/include/configs/sun7i.h index a6ede2a..f0d9222 100644 --- a/include/configs/sun7i.h +++ b/include/configs/sun7i.h @@ -34,6 +34,8 @@ #define CONFIG_SYS_PROMPT "sun7i# " #define CONFIG_MACH_TYPE 4283 +#define CONFIG_ARMV7_VIRT + #if defined(CONFIG_SYS_SECONDARY_ON) #define CONFIG_BOARD_POSTCLK_INIT 1 #endif
On 08/30/2013 03:12 PM, Ian Campbell wrote:> On Fri, 2013-08-30 at 10:51 +0100, Ian Campbell wrote: >> I''ll try that shortly and let you know how I get on. > > Success! My failure previously was not getting the kernel command line > in the right place, so no printk...Excellent! Great news! Seems I need to shell out those 70 EUR now ;-) Thanks! Andre. P.S. Keep my fingers cross for the Xen boot.> Starting from > https://github.com/linux-sunxi/u-boot-sunxi.git sunxi > I merged your stuff: > git pull git://git.linaro.org/people/aprzywara/u-boot.git hypmode_v4 > and applied the patch below (NB: u-boot doesn''t support SMP on this > platform yet, hence stubbing out the kicking of secondaries etc) and > https://groups.google.com/forum/#!topic/linux-sunxi/7oS0CM2WSy4. > > Then built with: > > $ make Cubieboard2_FEL CROSS_COMPILE=arm-linux-gnueabihf- > > Then I could boot via FEL (http://linux-sunxi.org/FEL/USBBoot) using > usb-boot from git://github.com/linux-sunxi/sunxi-tools.git: > usb-boot ~/devel/u-boot.git/spl/u-boot-spl.bin ~/devel/u-boot.git/u-boot.bin uImage > > where uImage is the zImage+dtb (cat into tmp) and mkimage -A arm -O > linux -T kernel -a 0x40008000 -e 0x40008000 -C none -d tmp uImage > > Linux says: > [ 0.000000] CPU: All CPU(s) started in HYP mode. > [ 0.000000] CPU: Virtualization extensions available. > > (All == all 1) > > Kernel is https://github.com/mripard/linux.git sunxi-next-a20-cubie2 > multi_v7_defconfig + CONFIG_DEBUG_LL w/ CONFIG_DEBUG_SUNXI_UART0 and > CONFIG_EARLY_PRINTK. > I used appended dtb and set /chosen/bootargs = "console=ttyS0,115200 > earlyprintk" in arch/arm/boot/dts/sun7i-a20-cubieboard2.dts (only > because usb-boot doesn''t happen to load dtb and pass it to bootm, will > fix that up...) > > Now to try Xen! > > Ian. > > commit 9267730ab1c3b6be5cf9118b2ce4e708e9d4ba66 > Author: Ian Campbell <ijc@hellion.org.uk> > Date: Fri Aug 30 13:52:07 2013 +0100 > > sunxi: Enable hyp mode boot on sunxi platforms > > Required a call to cleanup_before_linux in order to disable caches before > switching to NS world, otherwise we switch to an incorrecly configured banked > version of SCTLR. > > Signed-off-by: Ian Campbell <ijc@hellion.org.uk> > > diff --git a/arch/arm/cpu/armv7/virt-v7.c b/arch/arm/cpu/armv7/virt-v7.c > index 4dc8c45..a76dcf7 100644 > --- a/arch/arm/cpu/armv7/virt-v7.c > +++ b/arch/arm/cpu/armv7/virt-v7.c > @@ -132,6 +132,8 @@ int armv7_switch_nonsec(void) > unsigned int reg; > unsigned itlinesnr, i; > > + cleanup_before_linux(); > + > /* check whether the CPU supports the security extensions */ > reg = read_id_pfr1(); > if ((reg & 0xF0) == 0) { > diff --git a/board/sunxi/Makefile b/board/sunxi/Makefile > index 4780c99..d756091 100644 > --- a/board/sunxi/Makefile > +++ b/board/sunxi/Makefile > @@ -76,9 +76,13 @@ COBJS-$(CONFIG_UHOST_U1A) += dram_sun4i_360_1024_iow8.o > COBJS-$(CONFIG_WOBO_I5) += dram_wobo_i5.o > COBJS-$(CONFIG_XZPAD700) += dram_xzpad700.o > > +ifneq ($(CONFIG_ARMV7_NONSEC)$(CONFIG_ARMV7_VIRT),) > +SOBJS := smp.o > +endif > + > COBJS := $(COBJS-y) > -SRCS := $(COBJS:.o=.c) > -OBJS := $(addprefix $(obj),$(COBJS)) > +SRCS := $(COBJS:.o=.c) $(SOBJS:.o=.S) > +OBJS := $(addprefix $(obj),$(COBJS) $(SOBJS)) > > all: $(LIB) > > diff --git a/board/sunxi/board.c b/board/sunxi/board.c > index 50c07e2..4b8c732 100755 > --- a/board/sunxi/board.c > +++ b/board/sunxi/board.c > @@ -40,6 +40,20 @@ > > DECLARE_GLOBAL_DATA_PTR; > > +/* Setting the address at which secondary cores start from. > + * SMP currently unsupported. > + */ > +void smp_set_core_boot_addr(unsigned long addr, int corenr) > +{ > + printf("WARNING: ignoring attempt to set core boot address %lx on core %d\n", > + addr, corenr); > +} > + > +void smp_kick_all_cpus(void) > +{ > + printf("WARNING: Not kicking secondary CPUs\n"); > +} > + > /* add board specific code here */ > int board_init(void) > { > diff --git a/board/sunxi/smp.S b/board/sunxi/smp.S > new file mode 100644 > index 0000000..6a6e08c > --- /dev/null > +++ b/board/sunxi/smp.S > @@ -0,0 +1,33 @@ > +/* > + * code for redirecting secondary cores to their start address > + * > + * Copyright (c) 2013 Andre Przywara <andre.przywara@linaro.org> > + * > + * See file CREDITS for list of people who contributed to this > + * project. > + * > + * This program is free software; you can redistribute it and/or > + * modify it under the terms of the GNU General Public License as > + * published by the Free Software Foundation; either version 2 of > + * the License, or (at your option) any later version. > + * > + * This program is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + * GNU General Public License for more details. > + * > + * You should have received a copy of the GNU General Public License > + * along with this program; if not, write to the Free Software > + * Foundation, Inc., 59 Temple Place, Suite 330, Boston, > + * MA 02111-1307 USA > + */ > + > +#include <config.h> > +#include <linux/linkage.h> > + > +/* void _smp_waitloop(unsigned previous_address); */ > +ENTRY(smp_waitloop) > + /* SMP currently unsupported */ > +1: wfi > + b 1b > +ENDPROC(smp_waitloop) > diff --git a/include/configs/sun7i.h b/include/configs/sun7i.h > index a6ede2a..f0d9222 100644 > --- a/include/configs/sun7i.h > +++ b/include/configs/sun7i.h > @@ -34,6 +34,8 @@ > #define CONFIG_SYS_PROMPT "sun7i# " > #define CONFIG_MACH_TYPE 4283 > > +#define CONFIG_ARMV7_VIRT > + > #if defined(CONFIG_SYS_SECONDARY_ON) > #define CONFIG_BOARD_POSTCLK_INIT 1 > #endif > >
2013/8/30 Ian Campbell <ian.campbell@citrix.com>:> On Fri, 2013-08-30 at 10:51 +0100, Ian Campbell wrote: >> I''ll try that shortly and let you know how I get on. > > Success! My failure previously was not getting the kernel command line > in the right place, so no printk... > > Starting from > https://github.com/linux-sunxi/u-boot-sunxi.git sunxi > I merged your stuff: > git pull git://git.linaro.org/people/aprzywara/u-boot.git hypmode_v4 > and applied the patch below (NB: u-boot doesn''t support SMP on this > platform yet, hence stubbing out the kicking of secondaries etc) and > https://groups.google.com/forum/#!topic/linux-sunxi/7oS0CM2WSy4. > > Then built with: > > $ make Cubieboard2_FEL CROSS_COMPILE=arm-linux-gnueabihf- > > Then I could boot via FEL (http://linux-sunxi.org/FEL/USBBoot) using > usb-boot from git://github.com/linux-sunxi/sunxi-tools.git: > usb-boot ~/devel/u-boot.git/spl/u-boot-spl.bin ~/devel/u-boot.git/u-boot.bin uImage > > where uImage is the zImage+dtb (cat into tmp) and mkimage -A arm -O > linux -T kernel -a 0x40008000 -e 0x40008000 -C none -d tmp uImage > > Linux says: > [ 0.000000] CPU: All CPU(s) started in HYP mode. > [ 0.000000] CPU: Virtualization extensions available. > > (All == all 1) > > Kernel is https://github.com/mripard/linux.git sunxi-next-a20-cubie2 > multi_v7_defconfig + CONFIG_DEBUG_LL w/ CONFIG_DEBUG_SUNXI_UART0 and > CONFIG_EARLY_PRINTK. > I used appended dtb and set /chosen/bootargs = "console=ttyS0,115200 > earlyprintk" in arch/arm/boot/dts/sun7i-a20-cubieboard2.dts (only > because usb-boot doesn''t happen to load dtb and pass it to bootm, will > fix that up...) >That''s cool. Following your guide, the u-boot can enter into hyper mode now.
On Mon, 2013-09-02 at 11:59 +0800, Josh Zhao wrote:> That''s cool. Following your guide, the u-boot can enter into hyper mode now.Great! I''ve got it to the point of dom0 failing to find a root filesystem, which is because there is no MMC, SATA or USB driver in mainline yet -- so no storage. I''m rather busy at the moment, so I kind of hoping that by the time I get back to this someone will have picked up the allwinner sata driver and made it fit upstream. Ian.