Hi, I want to try Xen on ARMv8 Simulator. Can you please provide guidance?. I am looking for information about sources, build and procedure to launch Xen on ARMv8 Thanks & Regards Vijay
On Fri, 2013-11-29 at 13:20 +0530, Vijay Kilari wrote:> Hi, > > I want to try Xen on ARMv8 Simulator. > Can you please provide guidance?. I am looking for > information about sources, build and procedure to > launch Xen on ARMv8http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions is the place to start from. Ian.
(Adding Xen ML) On 12/02/2013 07:52 AM, Vijay Kilari wrote:> Hi Julien,Hello Vijay,> I am trying to boot Xen on ARMv8 simulation model. > Can you please let me know which linaro kernel version > that I can use to boot Xen? > Also please provide details about the kernel configuration & DTS file changes > that needs to be enabled to build boot wrapper, Dom0 & DomU kernel?We have a wiki page to explain how to boot Xen on ARMv8 simulation model: http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions/FastModels#arm64 I think the page is up-to-date. The best person to contact for Armv8 is Ian Campbell (in CC). Cheers, -- Julien Grall
On Mon, 2013-12-02 at 15:48 +0000, Julien Grall wrote:> (Adding Xen ML) > > On 12/02/2013 07:52 AM, Vijay Kilari wrote: > > Hi Julien, > > Hello Vijay, > > > I am trying to boot Xen on ARMv8 simulation model. > > Can you please let me know which linaro kernel version > > that I can use to boot Xen? > > Also please provide details about the kernel configuration & DTS file changes > > that needs to be enabled to build boot wrapper, Dom0 & DomU kernel? > > We have a wiki page to explain how to boot Xen on ARMv8 simulation model: > http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions/FastModels#arm64 > > I think the page is up-to-date. The best person to contact for Armv8 is > Ian Campbell (in CC).This question was already asked and answered on xen-devel on Friday. Ian.
Hi Ian,
Thanks for the information
I was successful to boot Xen on our ARMv8 simulator to some extent,
but facing following issues:
1) With Stable 4.3 version, Xen boots, but fails in ctxt_switch_from()
call from schedule_tail().
This happens to be because it does not support 64-bit guest. I was
using 64-bit guest
2) I see that staging branch has support for 64-bit guest, but it
fails to boot completely.
I see continuous IRQ 37 which is not seen with stable version
Any comments?
Regards
Vijay
On Fri, Nov 29, 2013 at 4:19 PM, Ian Campbell <Ian.Campbell@citrix.com>
wrote:> On Fri, 2013-11-29 at 13:20 +0530, Vijay Kilari wrote:
>> Hi,
>>
>> I want to try Xen on ARMv8 Simulator.
>> Can you please provide guidance?. I am looking for
>> information about sources, build and procedure to
>> launch Xen on ARMv8
>
> http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions is the
> place to start from.
>
> Ian.
>
On Thu, 2013-12-05 at 20:21 +0530, Vijay Kilari wrote:> Hi Ian, > > Thanks for the information > > I was successful to boot Xen on our ARMv8 simulator to some extent, > but facing following issues: > > 1) With Stable 4.3 version, Xen boots, but fails in ctxt_switch_from() > call from schedule_tail(). > This happens to be because it does not support 64-bit guest. I was > using 64-bit guest4.3 is probably not the best option for bringing up a new platform TBH.> 2) I see that staging branch has support for 64-bit guest, but it > fails to boot completely. > I see continuous IRQ 37 which is not seen with stable versionThat''s pretty strange. What is IRQ37 in your system? I know that Julien and Stefano are looking at some interrupt stuff, in particular there was a patch series way back in June which is currently been reworked titled "Fix multiple issues with the interrupts on ARM", of which "Only enable physical IRQs when the guest asks" sounds like it might be relevant given the symptom you describe. Ian.> > Any comments? > > Regards > Vijay > > > On Fri, Nov 29, 2013 at 4:19 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote: > > On Fri, 2013-11-29 at 13:20 +0530, Vijay Kilari wrote: > >> Hi, > >> > >> I want to try Xen on ARMv8 Simulator. > >> Can you please provide guidance?. I am looking for > >> information about sources, build and procedure to > >> launch Xen on ARMv8 > > > > http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions is the > > place to start from. > > > > Ian. > >
On Thu, Dec 5, 2013 at 8:27 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:> On Thu, 2013-12-05 at 20:21 +0530, Vijay Kilari wrote: >> Hi Ian, >> >> Thanks for the information >> >> I was successful to boot Xen on our ARMv8 simulator to some extent, >> but facing following issues: >> >> 1) With Stable 4.3 version, Xen boots, but fails in ctxt_switch_from() >> call from schedule_tail(). >> This happens to be because it does not support 64-bit guest. I was >> using 64-bit guest > > 4.3 is probably not the best option for bringing up a new platform TBH. > >> 2) I see that staging branch has support for 64-bit guest, but it >> fails to boot completely. >> I see continuous IRQ 37 which is not seen with stable version > > That''s pretty strange. What is IRQ37 in your system? > > I know that Julien and Stefano are looking at some interrupt stuff, in > particular there was a patch series way back in June which is currently > been reworked titled "Fix multiple issues with the interrupts on ARM", > of which "Only enable physical IRQs when the guest asks" sounds like it > might be relevant given the symptom you describe. >IRQ 37 is uart interrupt. The problem is with the pl011 driver changes which are not compatible with our ARMv8 simulator. After reverting changes to pl011 to inline with stable branch, issue is not seen. I will identify the exact patch that is causing this problem later. Now, I am facing problem with loading dom0 & dtb, the following is the console output. I have changed dom0_mem to 512MB instead of 128 MB Can you please provide your quick comments on whats going wrong? Looks like the Dom0 load address is wrong. Do I need to regenerate Dom0 kernel with different address? I am using Xen staging tag (XEN) Using PSCI for SMP bringup (XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 (XEN) Using generic timer at 100000 KHz (XEN) GIC initialization: (XEN) gic_dist_addr=000000002c001000 (XEN) gic_cpu_addr=000000002c002000 (XEN) gic_hyp_addr=000000002c004000 (XEN) gic_vcpu_addr=000000002c006000 (XEN) gic_maintenance_irq=25 (XEN) GIC: 160 lines, 8 cpus, secure (IID a002104c). (XEN) Using scheduler: SMP Credit Scheduler (credit) (XEN) Allocated console ring of 16 KiB. (XEN) Brought up 1 CPUs (XEN) *** LOADING DOMAIN 0 *** (XEN) Populate P2M 0xa0000000->0xc0000000 (1:1 mapping for dom0) (XEN) Loading kernel from boot module 2 (XEN) elf_parse_binary: phdr: paddr=0xffffffc000080000 memsz=0x702518 (XEN) elf_parse_binary: memory: 0xffffffc000080000 -> 0xffffffc000782518 (XEN) elf_xen_addr_calc_check: VIRT_BASE unset, using 0x0 (XEN) elf_xen_addr_calc_check: ELF_PADDR_OFFSET unset, using 0x0 (XEN) elf_xen_addr_calc_check: addresses: (XEN) virt_base = 0x0 (XEN) elf_paddr_offset = 0x0 (XEN) virt_offset = 0x0 (XEN) virt_kstart = 0xffffffc000080000 (XEN) virt_kend = 0xffffffc000782518 (XEN) virt_entry = 0xffffffc000080020 (XEN) p2m_base = 0xffffffffffffffff (XEN) Loading ELF image into guest memory (XEN) elf_load_binary: phdr 0 at 0xffffffc000080000 -> 0xffffffc0006b0024 (XEN) Free temporary kernel buffer (XEN) Loading dom0 DTB to 0xffffffc000800000-0xffffffc0008017f7 (XEN) (XEN) **************************************** (XEN) Panic on CPU 0: (XEN) Unable to copy the DTB to dom0 memory (rc = 18446744073709551602) (XEN) ****************************************> Ian. > >> >> Any comments? >> >> Regards >> Vijay >> >> >> On Fri, Nov 29, 2013 at 4:19 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote: >> > On Fri, 2013-11-29 at 13:20 +0530, Vijay Kilari wrote: >> >> Hi, >> >> >> >> I want to try Xen on ARMv8 Simulator. >> >> Can you please provide guidance?. I am looking for >> >> information about sources, build and procedure to >> >> launch Xen on ARMv8 >> > >> > http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions is the >> > place to start from. >> > >> > Ian. >> > > >
On Fri, 2013-12-06 at 19:48 +0530, Vijay Kilari wrote:> On Thu, Dec 5, 2013 at 8:27 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote: > > On Thu, 2013-12-05 at 20:21 +0530, Vijay Kilari wrote: > >> Hi Ian, > >> > >> Thanks for the information > >> > >> I was successful to boot Xen on our ARMv8 simulator to some extent, > >> but facing following issues: > >> > >> 1) With Stable 4.3 version, Xen boots, but fails in ctxt_switch_from() > >> call from schedule_tail(). > >> This happens to be because it does not support 64-bit guest. I was > >> using 64-bit guest > > > > 4.3 is probably not the best option for bringing up a new platform TBH. > > > >> 2) I see that staging branch has support for 64-bit guest, but it > >> fails to boot completely. > >> I see continuous IRQ 37 which is not seen with stable version > > > > That''s pretty strange. What is IRQ37 in your system? > > > > I know that Julien and Stefano are looking at some interrupt stuff, in > > particular there was a patch series way back in June which is currently > > been reworked titled "Fix multiple issues with the interrupts on ARM", > > of which "Only enable physical IRQs when the guest asks" sounds like it > > might be relevant given the symptom you describe. > > > > IRQ 37 is uart interrupt. The problem is with the pl011 driver changes which > are not compatible with our ARMv8 simulator. After reverting changes to pl011 > to inline with stable branch, issue is not seen. I will identify the exact patch > that is causing this problem later.Please do, there are a few interrupt logic changes since 4.3 in that driver, if there is a problem there it would be good to get to the bottom of it before Xen 4.4 comes out (in Jan/Feb next year). That said it works on at least midway, vexpress and the arm fast models. Did you work around this with the patch from Julien? Or some other method?> (XEN) Using PSCI for SMP bringup > (XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 > (XEN) Using generic timer at 100000 KHz > (XEN) GIC initialization: > (XEN) gic_dist_addr=000000002c001000 > (XEN) gic_cpu_addr=000000002c002000 > (XEN) gic_hyp_addr=000000002c004000 > (XEN) gic_vcpu_addr=000000002c006000 > (XEN) gic_maintenance_irq=25 > (XEN) GIC: 160 lines, 8 cpus, secure (IID a002104c). > (XEN) Using scheduler: SMP Credit Scheduler (credit) > (XEN) Allocated console ring of 16 KiB. > (XEN) Brought up 1 CPUs > (XEN) *** LOADING DOMAIN 0 *** > (XEN) Populate P2M 0xa0000000->0xc0000000 (1:1 mapping for dom0) > (XEN) Loading kernel from boot module 2 > (XEN) elf_parse_binary: phdr: paddr=0xffffffc000080000 memsz=0x702518 > (XEN) elf_parse_binary: memory: 0xffffffc000080000 -> 0xffffffc000782518 > (XEN) elf_xen_addr_calc_check: VIRT_BASE unset, using 0x0 > (XEN) elf_xen_addr_calc_check: ELF_PADDR_OFFSET unset, using 0x0 > (XEN) elf_xen_addr_calc_check: addresses: > (XEN) virt_base = 0x0 > (XEN) elf_paddr_offset = 0x0 > (XEN) virt_offset = 0x0 > (XEN) virt_kstart = 0xffffffc000080000 > (XEN) virt_kend = 0xffffffc000782518 > (XEN) virt_entry = 0xffffffc000080020 > (XEN) p2m_base = 0xffffffffffffffff > (XEN) Loading ELF image into guest memory > (XEN) elf_load_binary: phdr 0 at 0xffffffc000080000 -> 0xffffffc0006b0024 > (XEN) Free temporary kernel buffer > (XEN) Loading dom0 DTB to 0xffffffc000800000-0xffffffc0008017f7 > (XEN) > (XEN) **************************************** > (XEN) Panic on CPU 0: > (XEN) Unable to copy the DTB to dom0 memory (rc = 18446744073709551602) > (XEN) ****************************************It looks like you are trying to load an ELF format kernel, which probably doesn''t work for Linux. Try passing it the arch/arm64/boot/Image and it should work. (ELF is useful for *BSD I think, which is why we don''t just nuke this support right now) Ian.
On Fri, Dec 6, 2013 at 8:55 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:> On Fri, 2013-12-06 at 19:48 +0530, Vijay Kilari wrote: >> On Thu, Dec 5, 2013 at 8:27 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote: >> > On Thu, 2013-12-05 at 20:21 +0530, Vijay Kilari wrote: >> >> Hi Ian, >> >> >> >> Thanks for the information >> >> >> >> I was successful to boot Xen on our ARMv8 simulator to some extent, >> >> but facing following issues: >> >> >> >> 1) With Stable 4.3 version, Xen boots, but fails in ctxt_switch_from() >> >> call from schedule_tail(). >> >> This happens to be because it does not support 64-bit guest. I was >> >> using 64-bit guest >> > >> > 4.3 is probably not the best option for bringing up a new platform TBH. >> > >> >> 2) I see that staging branch has support for 64-bit guest, but it >> >> fails to boot completely. >> >> I see continuous IRQ 37 which is not seen with stable version >> > >> > That''s pretty strange. What is IRQ37 in your system? >> > >> > I know that Julien and Stefano are looking at some interrupt stuff, in >> > particular there was a patch series way back in June which is currently >> > been reworked titled "Fix multiple issues with the interrupts on ARM", >> > of which "Only enable physical IRQs when the guest asks" sounds like it >> > might be relevant given the symptom you describe. >> > >> >> IRQ 37 is uart interrupt. The problem is with the pl011 driver changes which >> are not compatible with our ARMv8 simulator. After reverting changes to pl011 >> to inline with stable branch, issue is not seen. I will identify the exact patch >> that is causing this problem later. > > Please do, there are a few interrupt logic changes since 4.3 in that > driver, if there is a problem there it would be good to get to the > bottom of it before Xen 4.4 comes out (in Jan/Feb next year). That said > it works on at least midway, vexpress and the arm fast models. >> Did you work around this with the patch from Julien? Or some other > method? >I don''t have any work around as of now. I just reverted changes made to pl011 driver in staging branch on top of stable 4.3. I see couple of patches that has changed pl011 registers initialization. I will check once I get the my setup is up & running.>> (XEN) Using PSCI for SMP bringup[..]>> (XEN) Panic on CPU 0: >> (XEN) Unable to copy the DTB to dom0 memory (rc = 18446744073709551602) >> (XEN) **************************************** > > It looks like you are trying to load an ELF format kernel, which > probably doesn''t work for Linux. Try passing it the > arch/arm64/boot/Image and it should work. > > (ELF is useful for *BSD I think, which is why we don''t just nuke this > support right now) >Yes, I have set my Image to vmlinux (stripped) for 4.3 stable as it was failing to load plain arch/arm64/boot/Image I forget to revert back this change with staging branch. Now I could boot Xen hypervisor. Thanks for this help. But my console on dom0 is not showing any logs. However ARMv8 simulator shows that dom0 is booted and cpu has entered idle loop.> Ian. >
On Mon, 2013-12-09 at 19:22 +0530, Vijay Kilari wrote:> On Fri, Dec 6, 2013 at 8:55 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote: > I don''t have any work around as of now. I just reverted changes made > to pl011 driver in staging branch on top of stable 4.3. I see couple of > patches that has changed pl011 registers initialization. > > I will check once I get the my setup is up & running.OK, thanks.> > >> (XEN) Using PSCI for SMP bringup > [..] > >> (XEN) Panic on CPU 0: > >> (XEN) Unable to copy the DTB to dom0 memory (rc = 18446744073709551602) > >> (XEN) **************************************** > > > > It looks like you are trying to load an ELF format kernel, which > > probably doesn''t work for Linux. Try passing it the > > arch/arm64/boot/Image and it should work. > > > > (ELF is useful for *BSD I think, which is why we don''t just nuke this > > support right now) > > > Yes, I have set my Image to vmlinux (stripped) for 4.3 stable as it was failing > to load plain arch/arm64/boot/Image > I forget to revert back this change with staging branch. Now I could > boot Xen hypervisor.Excellent!> Thanks for this help. > > But my console on dom0 is not showing any logs. > However ARMv8 simulator shows that dom0 is booted and > cpu has entered idle loop.Do you have console=hvc0 on your dom0 command line and CONFIG_HVC_XEN in your kernel build? You might also want to try enabling earlyprintk, either via a second UART not used by Xen, or the vuart which Xen replaces its console with for dom0 (assuming your Xen console driver exports the right hooks) or with something like: diff --git a/arch/arm64/kernel/early_printk.c b/arch/arm64/kernel/early_printk.c index fbb6e18..9302d7a 100644 --- a/arch/arm64/kernel/early_printk.c +++ b/arch/arm64/kernel/early_printk.c @@ -26,6 +26,8 @@ #include <linux/amba/serial.h> #include <linux/serial_reg.h> +#include <asm/xen/hypercall.h> + static void __iomem *early_base; static void (*printch)(char ch); @@ -52,6 +54,11 @@ static void smh_printch(char ch) : : "r" (&ch) : "x0", "x1", "memory"); } +static void xen_printch(char ch) +{ + HYPERVISOR_console_io(CONSOLEIO_write, 1, &ch); +} + /* * 8250/16550 (8-bit aligned registers) single character TX. */ @@ -80,6 +87,7 @@ struct earlycon_match { static const struct earlycon_match earlycon_match[] __initconst = { { .name = "pl011", .printch = pl011_printch, }, { .name = "smh", .printch = smh_printch, }, + { .name = "xen", .printch = xen_printch, }, { .name = "uart8250-8bit", .printch = uart8250_8bit_printch, }, { .name = "uart8250-32bit", .printch = uart8250_32bit_printch, }, {}
On Mon, Dec 9, 2013 at 7:34 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:> On Mon, 2013-12-09 at 19:22 +0530, Vijay Kilari wrote: >> On Fri, Dec 6, 2013 at 8:55 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote: >> I don''t have any work around as of now. I just reverted changes made >> to pl011 driver in staging branch on top of stable 4.3. I see couple of >> patches that has changed pl011 registers initialization. >> >> I will check once I get the my setup is up & running. > > OK, thanks. > >> >> >> (XEN) Using PSCI for SMP bringup >> [..] >> >> (XEN) Panic on CPU 0: >> >> (XEN) Unable to copy the DTB to dom0 memory (rc = 18446744073709551602) >> >> (XEN) **************************************** >> > >> > It looks like you are trying to load an ELF format kernel, which >> > probably doesn''t work for Linux. Try passing it the >> > arch/arm64/boot/Image and it should work. >> > >> > (ELF is useful for *BSD I think, which is why we don''t just nuke this >> > support right now) >> > >> Yes, I have set my Image to vmlinux (stripped) for 4.3 stable as it was failing >> to load plain arch/arm64/boot/Image >> I forget to revert back this change with staging branch. Now I could >> boot Xen hypervisor. > > Excellent! > >> Thanks for this help. >> >> But my console on dom0 is not showing any logs. >> However ARMv8 simulator shows that dom0 is booted and >> cpu has entered idle loop. > > Do you have console=hvc0 on your dom0 command line and CONFIG_HVC_XEN in > your kernel build? >Changing console=hvc0 works. Kernel boot still fails. I have my rootfs of IDE(ATA) on AHCI. Because there is no link up interrupt from IDE and hence it fails to boot and mount rootfs. Where as the same kernel boots fine without Xen From HW traces, I see that GIC does not raise IRQ. I am using kernel 3.10.14+. Can you please let me know Dom0 kernel configuration. I have enabled following configuration in dom0 kernel CONFIG_XEN_DOM0=y CONFIG_XEN=y CONFIG_XEN_NETDEV_FRONTEND=y CONFIG_XEN_NETDEV_BACKEND=y CONFIG_HVC_DRIVER=y CONFIG_HVC_IRQ=y CONFIG_HVC_XEN=y CONFIG_HVC_XEN_FRONTEND=y CONFIG_XEN_DEV_EVTCHN=y CONFIG_XEN_BACKEND=y CONFIG_XENFS=y CONFIG_XEN_COMPAT_XENFS=y CONFIG_XEN_XENBUS_FRONTEND=y CONFIG_XEN_GNTDEV=y CONFIG_XEN_PRIVCMD=y> You might also want to try enabling earlyprintk, either via a second > UART not used by Xen, or the vuart which Xen replaces its console with > for dom0 (assuming your Xen console driver exports the right hooks) or > with something like: > > diff --git a/arch/arm64/kernel/early_printk.c b/arch/arm64/kernel/early_printk.c > index fbb6e18..9302d7a 100644 > --- a/arch/arm64/kernel/early_printk.c > +++ b/arch/arm64/kernel/early_printk.c > @@ -26,6 +26,8 @@ > #include <linux/amba/serial.h> > #include <linux/serial_reg.h> > > +#include <asm/xen/hypercall.h> > + > static void __iomem *early_base; > static void (*printch)(char ch); > > @@ -52,6 +54,11 @@ static void smh_printch(char ch) > : : "r" (&ch) : "x0", "x1", "memory"); > } > > +static void xen_printch(char ch) > +{ > + HYPERVISOR_console_io(CONSOLEIO_write, 1, &ch); > +} > + > /* > * 8250/16550 (8-bit aligned registers) single character TX. > */ > @@ -80,6 +87,7 @@ struct earlycon_match { > static const struct earlycon_match earlycon_match[] __initconst = { > { .name = "pl011", .printch = pl011_printch, }, > { .name = "smh", .printch = smh_printch, }, > + { .name = "xen", .printch = xen_printch, }, > { .name = "uart8250-8bit", .printch = uart8250_8bit_printch, }, > { .name = "uart8250-32bit", .printch = uart8250_32bit_printch, }, > {} > >
On Tue, Dec 10, 2013 at 5:59 PM, Vijay Kilari <vijay.kilari@gmail.com> wrote:> On Mon, Dec 9, 2013 at 7:34 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote: >> On Mon, 2013-12-09 at 19:22 +0530, Vijay Kilari wrote: >>> On Fri, Dec 6, 2013 at 8:55 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote: >>> I don''t have any work around as of now. I just reverted changes made >>> to pl011 driver in staging branch on top of stable 4.3. I see couple of >>> patches that has changed pl011 registers initialization. >>> >>> I will check once I get the my setup is up & running. >> >> OK, thanks. >> >>> >>> >> (XEN) Using PSCI for SMP bringup >>> [..] >>> >> (XEN) Panic on CPU 0: >>> >> (XEN) Unable to copy the DTB to dom0 memory (rc = 18446744073709551602) >>> >> (XEN) **************************************** >>> > >>> > It looks like you are trying to load an ELF format kernel, which >>> > probably doesn''t work for Linux. Try passing it the >>> > arch/arm64/boot/Image and it should work. >>> > >>> > (ELF is useful for *BSD I think, which is why we don''t just nuke this >>> > support right now) >>> > >>> Yes, I have set my Image to vmlinux (stripped) for 4.3 stable as it was failing >>> to load plain arch/arm64/boot/Image >>> I forget to revert back this change with staging branch. Now I could >>> boot Xen hypervisor. >> >> Excellent! >> >>> Thanks for this help. >>> >>> But my console on dom0 is not showing any logs. >>> However ARMv8 simulator shows that dom0 is booted and >>> cpu has entered idle loop. >> >> Do you have console=hvc0 on your dom0 command line and CONFIG_HVC_XEN in >> your kernel build? >> > > Changing console=hvc0 works. > > Kernel boot still fails. I have my rootfs of IDE(ATA) on AHCI. Because there is > no link up interrupt from IDE and hence it fails to boot and mount rootfs. > Where as the same kernel boots fine without Xen > > From HW traces, I see that GIC does not raise IRQ. > > I am using kernel 3.10.14+. Can you please let me know Dom0 kernel > configuration.Missed to mention. On top of 3.10.14 I added following patches for arm64 1e33368 arm/xen: define xen_remap as ioremap_cached 71750f7 arm64/xen: introduce asm/xen header files on arm64 a554a92 xen/arm and xen/arm64: implement HYPERVISOR_tmem_op b80ef86 arm64/xen: introduce CONFIG_XEN and hypercall.S on ARM64 7605d0c xen/arm: disable cpuidle and cpufreq when linux is running as dom0> I have enabled following configuration in dom0 kernel > > CONFIG_XEN_DOM0=y > CONFIG_XEN=y > CONFIG_XEN_NETDEV_FRONTEND=y > CONFIG_XEN_NETDEV_BACKEND=y > CONFIG_HVC_DRIVER=y > CONFIG_HVC_IRQ=y > CONFIG_HVC_XEN=y > CONFIG_HVC_XEN_FRONTEND=y > CONFIG_XEN_DEV_EVTCHN=y > CONFIG_XEN_BACKEND=y > CONFIG_XENFS=y > CONFIG_XEN_COMPAT_XENFS=y > CONFIG_XEN_XENBUS_FRONTEND=y > CONFIG_XEN_GNTDEV=y > CONFIG_XEN_PRIVCMD=y > > >> You might also want to try enabling earlyprintk, either via a second >> UART not used by Xen, or the vuart which Xen replaces its console with >> for dom0 (assuming your Xen console driver exports the right hooks) or >> with something like: >> >> diff --git a/arch/arm64/kernel/early_printk.c b/arch/arm64/kernel/early_printk.c >> index fbb6e18..9302d7a 100644 >> --- a/arch/arm64/kernel/early_printk.c >> +++ b/arch/arm64/kernel/early_printk.c >> @@ -26,6 +26,8 @@ >> #include <linux/amba/serial.h> >> #include <linux/serial_reg.h> >> >> +#include <asm/xen/hypercall.h> >> + >> static void __iomem *early_base; >> static void (*printch)(char ch); >> >> @@ -52,6 +54,11 @@ static void smh_printch(char ch) >> : : "r" (&ch) : "x0", "x1", "memory"); >> } >> >> +static void xen_printch(char ch) >> +{ >> + HYPERVISOR_console_io(CONSOLEIO_write, 1, &ch); >> +} >> + >> /* >> * 8250/16550 (8-bit aligned registers) single character TX. >> */ >> @@ -80,6 +87,7 @@ struct earlycon_match { >> static const struct earlycon_match earlycon_match[] __initconst = { >> { .name = "pl011", .printch = pl011_printch, }, >> { .name = "smh", .printch = smh_printch, }, >> + { .name = "xen", .printch = xen_printch, }, >> { .name = "uart8250-8bit", .printch = uart8250_8bit_printch, }, >> { .name = "uart8250-32bit", .printch = uart8250_32bit_printch, }, >> {} >> >>
On Tue, 2013-12-10 at 18:18 +0530, Vijay Kilari wrote:> On Tue, Dec 10, 2013 at 5:59 PM, Vijay Kilari <vijay.kilari@gmail.com> wrote: > > On Mon, Dec 9, 2013 at 7:34 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote: > >> On Mon, 2013-12-09 at 19:22 +0530, Vijay Kilari wrote: > >>> On Fri, Dec 6, 2013 at 8:55 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote: > >>> I don''t have any work around as of now. I just reverted changes made > >>> to pl011 driver in staging branch on top of stable 4.3. I see couple of > >>> patches that has changed pl011 registers initialization. > >>> > >>> I will check once I get the my setup is up & running. > >> > >> OK, thanks. > >> > >>> > >>> >> (XEN) Using PSCI for SMP bringup > >>> [..] > >>> >> (XEN) Panic on CPU 0: > >>> >> (XEN) Unable to copy the DTB to dom0 memory (rc = 18446744073709551602) > >>> >> (XEN) **************************************** > >>> > > >>> > It looks like you are trying to load an ELF format kernel, which > >>> > probably doesn''t work for Linux. Try passing it the > >>> > arch/arm64/boot/Image and it should work. > >>> > > >>> > (ELF is useful for *BSD I think, which is why we don''t just nuke this > >>> > support right now) > >>> > > >>> Yes, I have set my Image to vmlinux (stripped) for 4.3 stable as it was failing > >>> to load plain arch/arm64/boot/Image > >>> I forget to revert back this change with staging branch. Now I could > >>> boot Xen hypervisor. > >> > >> Excellent! > >> > >>> Thanks for this help. > >>> > >>> But my console on dom0 is not showing any logs. > >>> However ARMv8 simulator shows that dom0 is booted and > >>> cpu has entered idle loop. > >> > >> Do you have console=hvc0 on your dom0 command line and CONFIG_HVC_XEN in > >> your kernel build? > >> > > > > Changing console=hvc0 works. > > > > Kernel boot still fails. I have my rootfs of IDE(ATA) on AHCI. Because there is > > no link up interrupt from IDE and hence it fails to boot and mount rootfs. > > Where as the same kernel boots fine without Xen > > > > From HW traces, I see that GIC does not raise IRQ. > > > > I am using kernel 3.10.14+. Can you please let me know Dom0 kernel > > configuration.> Missed to mention. On top of 3.10.14 I added following patches for arm64 > > 1e33368 arm/xen: define xen_remap as ioremap_cached > 71750f7 arm64/xen: introduce asm/xen header files on arm64 > a554a92 xen/arm and xen/arm64: implement HYPERVISOR_tmem_op > b80ef86 arm64/xen: introduce CONFIG_XEN and hypercall.S on ARM64 > 7605d0c xen/arm: disable cpuidle and cpufreq when linux is running as dom03.10.x plus those might be ok. The big thing which comes to mind that you will be missing here is the swiotlb stuff which Stefano did, which went into v3.13-rc1 (with some subsequent fixups). Stefano probably remembers better than I what went into the kernel since 3.10. I''m mostly tracking mainline myself and you might find it easier to move up to something more recent rather than backport the swiotlb stuff. You might also want to investigate Stefano''s "interrupt handling fixes" to the hypervisor side -- IIRC they are related to failures similar to the one you are seeing.> > I have enabled following configuration in dom0 kernelYour config looks fine.> > > > CONFIG_XEN_DOM0=y > > CONFIG_XEN=y > > CONFIG_XEN_NETDEV_FRONTEND=y > > CONFIG_XEN_NETDEV_BACKEND=y > > CONFIG_HVC_DRIVER=y > > CONFIG_HVC_IRQ=y > > CONFIG_HVC_XEN=y > > CONFIG_HVC_XEN_FRONTEND=y > > CONFIG_XEN_DEV_EVTCHN=y > > CONFIG_XEN_BACKEND=y > > CONFIG_XENFS=y > > CONFIG_XEN_COMPAT_XENFS=y > > CONFIG_XEN_XENBUS_FRONTEND=y > > CONFIG_XEN_GNTDEV=y > > CONFIG_XEN_PRIVCMD=y > > > > > >> You might also want to try enabling earlyprintk, either via a second > >> UART not used by Xen, or the vuart which Xen replaces its console with > >> for dom0 (assuming your Xen console driver exports the right hooks) or > >> with something like: > >> > >> diff --git a/arch/arm64/kernel/early_printk.c b/arch/arm64/kernel/early_printk.c > >> index fbb6e18..9302d7a 100644 > >> --- a/arch/arm64/kernel/early_printk.c > >> +++ b/arch/arm64/kernel/early_printk.c > >> @@ -26,6 +26,8 @@ > >> #include <linux/amba/serial.h> > >> #include <linux/serial_reg.h> > >> > >> +#include <asm/xen/hypercall.h> > >> + > >> static void __iomem *early_base; > >> static void (*printch)(char ch); > >> > >> @@ -52,6 +54,11 @@ static void smh_printch(char ch) > >> : : "r" (&ch) : "x0", "x1", "memory"); > >> } > >> > >> +static void xen_printch(char ch) > >> +{ > >> + HYPERVISOR_console_io(CONSOLEIO_write, 1, &ch); > >> +} > >> + > >> /* > >> * 8250/16550 (8-bit aligned registers) single character TX. > >> */ > >> @@ -80,6 +87,7 @@ struct earlycon_match { > >> static const struct earlycon_match earlycon_match[] __initconst = { > >> { .name = "pl011", .printch = pl011_printch, }, > >> { .name = "smh", .printch = smh_printch, }, > >> + { .name = "xen", .printch = xen_printch, }, > >> { .name = "uart8250-8bit", .printch = uart8250_8bit_printch, }, > >> { .name = "uart8250-32bit", .printch = uart8250_32bit_printch, }, > >> {} > >> > >>
On Tue, Dec 10, 2013 at 6:29 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:> On Tue, 2013-12-10 at 18:18 +0530, Vijay Kilari wrote: >> On Tue, Dec 10, 2013 at 5:59 PM, Vijay Kilari <vijay.kilari@gmail.com> wrote: >> > On Mon, Dec 9, 2013 at 7:34 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote: >> >> On Mon, 2013-12-09 at 19:22 +0530, Vijay Kilari wrote: >> >>> On Fri, Dec 6, 2013 at 8:55 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote: >> >>> I don''t have any work around as of now. I just reverted changes made >> >>> to pl011 driver in staging branch on top of stable 4.3. I see couple of >> >>> patches that has changed pl011 registers initialization. >> >>> >> >>> I will check once I get the my setup is up & running. >> >> >> >> OK, thanks. >> >> >> >>> >> >>> >> (XEN) Using PSCI for SMP bringup >> >>> [..] >> >>> >> (XEN) Panic on CPU 0: >> >>> >> (XEN) Unable to copy the DTB to dom0 memory (rc = 18446744073709551602) >> >>> >> (XEN) **************************************** >> >>> > >> >>> > It looks like you are trying to load an ELF format kernel, which >> >>> > probably doesn''t work for Linux. Try passing it the >> >>> > arch/arm64/boot/Image and it should work. >> >>> > >> >>> > (ELF is useful for *BSD I think, which is why we don''t just nuke this >> >>> > support right now) >> >>> > >> >>> Yes, I have set my Image to vmlinux (stripped) for 4.3 stable as it was failing >> >>> to load plain arch/arm64/boot/Image >> >>> I forget to revert back this change with staging branch. Now I could >> >>> boot Xen hypervisor. >> >> >> >> Excellent! >> >> >> >>> Thanks for this help. >> >>> >> >>> But my console on dom0 is not showing any logs. >> >>> However ARMv8 simulator shows that dom0 is booted and >> >>> cpu has entered idle loop. >> >> >> >> Do you have console=hvc0 on your dom0 command line and CONFIG_HVC_XEN in >> >> your kernel build? >> >> >> > >> > Changing console=hvc0 works. >> > >> > Kernel boot still fails. I have my rootfs of IDE(ATA) on AHCI. Because there is >> > no link up interrupt from IDE and hence it fails to boot and mount rootfs. >> > Where as the same kernel boots fine without Xen >> > >> > From HW traces, I see that GIC does not raise IRQ. >> > >> > I am using kernel 3.10.14+. Can you please let me know Dom0 kernel >> > configuration. > >> Missed to mention. On top of 3.10.14 I added following patches for arm64 >> >> 1e33368 arm/xen: define xen_remap as ioremap_cached >> 71750f7 arm64/xen: introduce asm/xen header files on arm64 >> a554a92 xen/arm and xen/arm64: implement HYPERVISOR_tmem_op >> b80ef86 arm64/xen: introduce CONFIG_XEN and hypercall.S on ARM64 >> 7605d0c xen/arm: disable cpuidle and cpufreq when linux is running as dom0 > > 3.10.x plus those might be ok. > > The big thing which comes to mind that you will be missing here is the > swiotlb stuff which Stefano did, which went into v3.13-rc1 (with some > subsequent fixups). Stefano probably remembers better than I what went > into the kernel since 3.10. > > I''m mostly tracking mainline myself and you might find it easier to move > up to something more recent rather than backport the swiotlb stuff. > > You might also want to investigate Stefano''s "interrupt handling fixes" > to the hypervisor side -- IIRC they are related to failures similar to > the one you are seeing. >I have taken v3 version of Stefano''s interrupt patches for Xen and 3.13.rc2 kernel and the issue is same as 3.10.14+ The issue seems to be with timer. The ATA driver calls for hardware reset and calls msleep(1) in kernel thread context before it checks for link status. The msleep(1) never schedules this kernel thread back and hence Dom0 boot is not complete. From the GIC traces, I see that there is IRQ 26 (virt timer interrupt) coming to xen but kernel''s process_timeout is never called to schedule kernel thread.>> > I have enabled following configuration in dom0 kernel > > Your config looks fine. > >> > >> > CONFIG_XEN_DOM0=y >> > CONFIG_XEN=y >> > CONFIG_XEN_NETDEV_FRONTEND=y >> > CONFIG_XEN_NETDEV_BACKEND=y >> > CONFIG_HVC_DRIVER=y >> > CONFIG_HVC_IRQ=y >> > CONFIG_HVC_XEN=y >> > CONFIG_HVC_XEN_FRONTEND=y >> > CONFIG_XEN_DEV_EVTCHN=y >> > CONFIG_XEN_BACKEND=y >> > CONFIG_XENFS=y >> > CONFIG_XEN_COMPAT_XENFS=y >> > CONFIG_XEN_XENBUS_FRONTEND=y >> > CONFIG_XEN_GNTDEV=y >> > CONFIG_XEN_PRIVCMD=y >> > >> > >> >> You might also want to try enabling earlyprintk, either via a second >> >> UART not used by Xen, or the vuart which Xen replaces its console with >> >> for dom0 (assuming your Xen console driver exports the right hooks) or >> >> with something like:[..]>> >> > >
On Wed, 11 Dec 2013, Vijay Kilari wrote:> On Tue, Dec 10, 2013 at 6:29 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote: > > On Tue, 2013-12-10 at 18:18 +0530, Vijay Kilari wrote: > >> On Tue, Dec 10, 2013 at 5:59 PM, Vijay Kilari <vijay.kilari@gmail.com> wrote: > >> > On Mon, Dec 9, 2013 at 7:34 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote: > >> >> On Mon, 2013-12-09 at 19:22 +0530, Vijay Kilari wrote: > >> >>> On Fri, Dec 6, 2013 at 8:55 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote: > >> >>> I don''t have any work around as of now. I just reverted changes made > >> >>> to pl011 driver in staging branch on top of stable 4.3. I see couple of > >> >>> patches that has changed pl011 registers initialization. > >> >>> > >> >>> I will check once I get the my setup is up & running. > >> >> > >> >> OK, thanks. > >> >> > >> >>> > >> >>> >> (XEN) Using PSCI for SMP bringup > >> >>> [..] > >> >>> >> (XEN) Panic on CPU 0: > >> >>> >> (XEN) Unable to copy the DTB to dom0 memory (rc = 18446744073709551602) > >> >>> >> (XEN) **************************************** > >> >>> > > >> >>> > It looks like you are trying to load an ELF format kernel, which > >> >>> > probably doesn''t work for Linux. Try passing it the > >> >>> > arch/arm64/boot/Image and it should work. > >> >>> > > >> >>> > (ELF is useful for *BSD I think, which is why we don''t just nuke this > >> >>> > support right now) > >> >>> > > >> >>> Yes, I have set my Image to vmlinux (stripped) for 4.3 stable as it was failing > >> >>> to load plain arch/arm64/boot/Image > >> >>> I forget to revert back this change with staging branch. Now I could > >> >>> boot Xen hypervisor. > >> >> > >> >> Excellent! > >> >> > >> >>> Thanks for this help. > >> >>> > >> >>> But my console on dom0 is not showing any logs. > >> >>> However ARMv8 simulator shows that dom0 is booted and > >> >>> cpu has entered idle loop. > >> >> > >> >> Do you have console=hvc0 on your dom0 command line and CONFIG_HVC_XEN in > >> >> your kernel build? > >> >> > >> > > >> > Changing console=hvc0 works. > >> > > >> > Kernel boot still fails. I have my rootfs of IDE(ATA) on AHCI. Because there is > >> > no link up interrupt from IDE and hence it fails to boot and mount rootfs. > >> > Where as the same kernel boots fine without Xen > >> > > >> > From HW traces, I see that GIC does not raise IRQ. > >> > > >> > I am using kernel 3.10.14+. Can you please let me know Dom0 kernel > >> > configuration. > > > >> Missed to mention. On top of 3.10.14 I added following patches for arm64 > >> > >> 1e33368 arm/xen: define xen_remap as ioremap_cached > >> 71750f7 arm64/xen: introduce asm/xen header files on arm64 > >> a554a92 xen/arm and xen/arm64: implement HYPERVISOR_tmem_op > >> b80ef86 arm64/xen: introduce CONFIG_XEN and hypercall.S on ARM64 > >> 7605d0c xen/arm: disable cpuidle and cpufreq when linux is running as dom0 > > > > 3.10.x plus those might be ok. > > > > The big thing which comes to mind that you will be missing here is the > > swiotlb stuff which Stefano did, which went into v3.13-rc1 (with some > > subsequent fixups). Stefano probably remembers better than I what went > > into the kernel since 3.10. > > > > I''m mostly tracking mainline myself and you might find it easier to move > > up to something more recent rather than backport the swiotlb stuff. > > > > You might also want to investigate Stefano''s "interrupt handling fixes" > > to the hypervisor side -- IIRC they are related to failures similar to > > the one you are seeing. > > > > I have taken v3 version of Stefano''s interrupt patches for Xen and > 3.13.rc2 kernel > and the issue is same as 3.10.14+ > > The issue seems to be with timer. The ATA driver calls for hardware reset > and calls msleep(1) in kernel thread context before it checks for link status. > The msleep(1) never schedules this kernel thread back and hence Dom0 > boot is not complete.Are you getting any interrupts in dom0? Arch_timer interrupts and interrupts for the ATA controller at all?
On Wed, 2013-12-11 at 16:16 +0000, Stefano Stabellini wrote:> On Wed, 11 Dec 2013, Vijay Kilari wrote: > > On Tue, Dec 10, 2013 at 6:29 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote: > > > On Tue, 2013-12-10 at 18:18 +0530, Vijay Kilari wrote: > > >> On Tue, Dec 10, 2013 at 5:59 PM, Vijay Kilari <vijay.kilari@gmail.com> wrote: > > >> > On Mon, Dec 9, 2013 at 7:34 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote: > > >> >> On Mon, 2013-12-09 at 19:22 +0530, Vijay Kilari wrote: > > >> >>> On Fri, Dec 6, 2013 at 8:55 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote: > > >> >>> I don''t have any work around as of now. I just reverted changes made > > >> >>> to pl011 driver in staging branch on top of stable 4.3. I see couple of > > >> >>> patches that has changed pl011 registers initialization. > > >> >>> > > >> >>> I will check once I get the my setup is up & running. > > >> >> > > >> >> OK, thanks. > > >> >> > > >> >>> > > >> >>> >> (XEN) Using PSCI for SMP bringup > > >> >>> [..] > > >> >>> >> (XEN) Panic on CPU 0: > > >> >>> >> (XEN) Unable to copy the DTB to dom0 memory (rc = 18446744073709551602) > > >> >>> >> (XEN) **************************************** > > >> >>> > > > >> >>> > It looks like you are trying to load an ELF format kernel, which > > >> >>> > probably doesn''t work for Linux. Try passing it the > > >> >>> > arch/arm64/boot/Image and it should work. > > >> >>> > > > >> >>> > (ELF is useful for *BSD I think, which is why we don''t just nuke this > > >> >>> > support right now) > > >> >>> > > > >> >>> Yes, I have set my Image to vmlinux (stripped) for 4.3 stable as it was failing > > >> >>> to load plain arch/arm64/boot/Image > > >> >>> I forget to revert back this change with staging branch. Now I could > > >> >>> boot Xen hypervisor. > > >> >> > > >> >> Excellent! > > >> >> > > >> >>> Thanks for this help. > > >> >>> > > >> >>> But my console on dom0 is not showing any logs. > > >> >>> However ARMv8 simulator shows that dom0 is booted and > > >> >>> cpu has entered idle loop. > > >> >> > > >> >> Do you have console=hvc0 on your dom0 command line and CONFIG_HVC_XEN in > > >> >> your kernel build? > > >> >> > > >> > > > >> > Changing console=hvc0 works. > > >> > > > >> > Kernel boot still fails. I have my rootfs of IDE(ATA) on AHCI. Because there is > > >> > no link up interrupt from IDE and hence it fails to boot and mount rootfs. > > >> > Where as the same kernel boots fine without Xen > > >> > > > >> > From HW traces, I see that GIC does not raise IRQ. > > >> > > > >> > I am using kernel 3.10.14+. Can you please let me know Dom0 kernel > > >> > configuration. > > > > > >> Missed to mention. On top of 3.10.14 I added following patches for arm64 > > >> > > >> 1e33368 arm/xen: define xen_remap as ioremap_cached > > >> 71750f7 arm64/xen: introduce asm/xen header files on arm64 > > >> a554a92 xen/arm and xen/arm64: implement HYPERVISOR_tmem_op > > >> b80ef86 arm64/xen: introduce CONFIG_XEN and hypercall.S on ARM64 > > >> 7605d0c xen/arm: disable cpuidle and cpufreq when linux is running as dom0 > > > > > > 3.10.x plus those might be ok. > > > > > > The big thing which comes to mind that you will be missing here is the > > > swiotlb stuff which Stefano did, which went into v3.13-rc1 (with some > > > subsequent fixups). Stefano probably remembers better than I what went > > > into the kernel since 3.10. > > > > > > I''m mostly tracking mainline myself and you might find it easier to move > > > up to something more recent rather than backport the swiotlb stuff. > > > > > > You might also want to investigate Stefano''s "interrupt handling fixes" > > > to the hypervisor side -- IIRC they are related to failures similar to > > > the one you are seeing. > > > > > > > I have taken v3 version of Stefano''s interrupt patches for Xen and > > 3.13.rc2 kernel > > and the issue is same as 3.10.14+ > > > > The issue seems to be with timer. The ATA driver calls for hardware reset > > and calls msleep(1) in kernel thread context before it checks for link status. > > The msleep(1) never schedules this kernel thread back and hence Dom0 > > boot is not complete. > > Are you getting any interrupts in dom0? > Arch_timer interrupts and interrupts for the ATA controller at all?The other thing to check is that platform->dom0_evtchn_ppi (default ==irq 31, ppi 15) doesn''t conflict with a real interrupt number. Ian.
Reasonably Related Threads
- [PATCH 0/4] ARM/early-printk: Improve reusability and add Calxeda support
- [RFC PATCHv1] cover: celt_pitch_xcorr: Introduce ARM neon intrinsics
- [RFC PATCHv1] cover: celt_pitch_xcorr: Introduce ARM neon intrinsics
- [LLVMdev] Proposal: AArch64/ARM64 merge from EuroLLVM
- [CentOS-announce] CentOS-7 disk images for AArch64 Platforms