Alex Bennée
2021-May-28 12:49 UTC
[Pkg-xen-devel] What's missing for arm64 Xen boot with FDT via Grub in Debian Bullseye?
Hi,
I'm currently trying to pull together the threads for booting Xen on
Debian. I'm currently doing this within QEMU's TCG emulation and the
"virt" machine model:
-machine type=virt,virtualization=on,gic-version=3 \
-cpu max,pauth-impdef=on
with the firmware on my Ubuntu machine:
-drive if=pflash,file=/usr/share/AAVMF/AAVMF_CODE.fd,format=raw,readonly=on
-drive if=pflash,file=$HOME/images/AAVMF_VARS.fd,format=raw
(qemu-efi-aarch64 Version: 0~20180205.c0d9813c-2ubuntu0.3)
When booting this way I get the Grub menu and Xen is loaded by Grub but
falls over later:
(XEN) MODULE[0]: 00000000f5869000 - 00000000f59b60c8 Xen
(XEN) MODULE[1]: 000000013857d000 - 0000000138580000 Device Tree
(XEN) MODULE[2]: 00000000f73a1000 - 00000000f8da0780 Kernel
(XEN) MODULE[3]: 00000000f59b7000 - 00000000f739f99b Ramdisk
(XEN)
(XEN) CMDLINE[00000000f73a1000]:chosen placeholder
root=UUID=435201aa-c5cf-4e7a-8107-5eef28844188 ro console=hvc0
(XEN)
(XEN) Command line: placeholder dom0_mem=2G loglvl=all guest_loglvl=all
no-real-mode edd=off
(XEN) parameter "placeholder" unknown!
(XEN) parameter "no-real-mode" unknown!
(XEN) parameter "edd" unknown!
(XEN) Domain heap initialised
(XEN) Booting using Device Tree
(XEN) Platform: Generic System
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Unable to find a compatible timer in the device tree
(XEN) ****************************************
It seems like there are bits of the DT missing. I can however
successfully boot Xen with the Linux guest using the guest-loader device
and bypassing the firmware/boot code step. This gives:
(XEN) MODULE[0]: 0000000040200000 - 000000004034d0c8 Xen
(XEN) MODULE[1]: 0000000048000000 - 0000000048100000 Device Tree
(XEN) MODULE[2]: 0000000046000000 - 0000000046eb2200 Kernel
(XEN)
(XEN) CMDLINE[0000000046000000]:chosen root=/dev/sda2 console=hvc0
earlyprintk=xen
(XEN)
(XEN) Command line: dom0_mem=4G dom0_max_vcpus=4
(XEN) Domain heap initialised
(XEN) Booting using Device Tree
(XEN) Platform: Generic System
(XEN) Taking dtuart configuration from /chosen/stdout-path
(XEN) Looking for dtuart at "/pl011 at 9000000", options
""
Xen 4.15.1-pre
(XEN) Xen version 4.15.1-pre (alex.bennee@) (aarch64-linux-gnu-gcc (Debian
8.3.0-2) 8.3.0) debug=y Tue May 18 09:34:55 UTC 2021
(XEN) Latest ChangeSet:
(XEN) build-id: a50d8f03a1a15662ac7c4e5f73f2f544a6739df2
(XEN) Processor: 411fd070: "ARM Limited", variant: 0x1, part 0xd07,
rev 0x0
(XEN) 64-bit Execution:
(XEN) Processor Features: 0000000001000222 0000000000000000
(XEN) Exception Levels: EL3:No EL2:64+32 EL1:64+32 EL0:64+32
(XEN) Extensions: FloatingPoint AdvancedSIMD GICv3-SysReg
(XEN) Debug Features: 0000000010305106 0000000000000000
(XEN) Auxiliary Features: 0000000000000000 0000000000000000
(XEN) Memory Model Features: 0000000000001124 0000000000000000
(XEN) ISA Features: 0000000000011120 0000000000000000
(XEN) 32-bit Execution:
(XEN) Processor Features: 00000131:10011001
(XEN) Instruction Sets: AArch32 A32 Thumb Thumb-2 Jazelle
(XEN) Extensions: GenericTimer
(XEN) Debug Features: 03010066
(XEN) Auxiliary Features: 00000000
(XEN) Memory Model Features: 10101105 40000000 01260000 02102211
(XEN) ISA Features: 02101110 13112111 21232042 01112131 00011142 00011121
(XEN) Using SMC Calling Convention v1.0
(XEN) Using PSCI v0.2
(XEN) SMP: Allowing 8 CPUs
(XEN) enabled workaround for: ARM erratum 832075
(XEN) enabled workaround for: ARM erratum 834220
(XEN) enabled workaround for: ARM erratum 1319367
(XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 62500 KHz
(XEN) GICv3 initialization:
(XEN) gic_dist_addr=0x00000008000000
(XEN) gic_maintenance_irq=25
(XEN) gic_rdist_stride=0
(XEN) gic_rdist_regions=1
(XEN) redistributor regions:
(XEN) - region 0: 0x000000080a0000 - 0x00000009000000
(XEN) GICv3: 256 lines, (IID 0000043b).
(XEN) GICv3: CPU0: Found redistributor in region 0 @000000004001c000
Attempting to boot with acpi=on still sees Grub attempt to use DT to
boot the hypervisor. However selecting the kernel directly boots with
ACPI (which is a shame as I'd like to see what FDT it gets presented
with).
The full command line for booting via the guest-loader is:
./qemu-system-aarch64 \
-machine virt,virtualization=on,gic-version=3 \
-cpu max,pauth-impdef=on \
-serial mon:stdio \
-netdev user,id=net1,hostfwd=tcp::2222-:22 \
-device virtio-net-pci,netdev=net1 \
-device virtio-scsi-pci \
-drive
file=/dev/zvol/hackpool-0/debian-buster-arm64,id=hd0,index=0,if=none,format=raw
\
-device scsi-hd,drive=hd0 \
-display none \
-m 16384 \
-kernel ~/lsrc/xen/xen.build.arm64-xen/xen/xen \
-append "dom0_mem=4G dom0_max_vcpus=4" \
-device
guest-loader,addr=0x46000000,kernel=$HOME/lsrc/linux.git/builds/arm64/arch/arm64/boot/Image,bootargs="root=/dev/sda2
console=hvc0 earlyprintk=xen" \
-smp 8
So some questions:
- is Xen on arm64 tested on Debian Bullseye? If so what platform?
- how do I tell Grub to do a straight FDT boot with the DT from the firmware?
- are there any missing pieces I should be aware of?
I appreciate that ACPI is the preferred enterprise way of booting but at
the moment I think FDT is probably preferred because:
- lack of real HW with decent ACPI (my MachiatoBin only boots with DT)
- I want to try additional hypervisors who don't have ACPI aware
implementations
That said if I can get an ACPI version of Xen booting via Grub that
would be an improvement.
--
Alex Benn?e
Julien Grall
2021-May-28 15:28 UTC
[Pkg-xen-devel] What's missing for arm64 Xen boot with FDT via Grub in Debian Bullseye?
On 28/05/2021 13:49, Alex Benn?e wrote:> > Hi,Hi Alex,> I'm currently trying to pull together the threads for booting Xen on > Debian. I'm currently doing this within QEMU's TCG emulation and the > "virt" machine model: > > -machine type=virt,virtualization=on,gic-version=3 \ > -cpu max,pauth-impdef=on > > with the firmware on my Ubuntu machine: > > -drive if=pflash,file=/usr/share/AAVMF/AAVMF_CODE.fd,format=raw,readonly=on -drive if=pflash,file=$HOME/images/AAVMF_VARS.fd,format=raw > > (qemu-efi-aarch64 Version: 0~20180205.c0d9813c-2ubuntu0.3) > > When booting this way I get the Grub menu and Xen is loaded by Grub but > falls over later: > > (XEN) MODULE[0]: 00000000f5869000 - 00000000f59b60c8 Xen > (XEN) MODULE[1]: 000000013857d000 - 0000000138580000 Device Tree > (XEN) MODULE[2]: 00000000f73a1000 - 00000000f8da0780 Kernel > (XEN) MODULE[3]: 00000000f59b7000 - 00000000f739f99b Ramdisk > (XEN) > (XEN) CMDLINE[00000000f73a1000]:chosen placeholder root=UUID=435201aa-c5cf-4e7a-8107-5eef28844188 ro console=hvc0 > (XEN) > (XEN) Command line: placeholder dom0_mem=2G loglvl=all guest_loglvl=all no-real-mode edd=off > (XEN) parameter "placeholder" unknown! > (XEN) parameter "no-real-mode" unknown! > (XEN) parameter "edd" unknown! > (XEN) Domain heap initialised > (XEN) Booting using Device Tree > (XEN) Platform: Generic System > (XEN) > (XEN) **************************************** > (XEN) Panic on CPU 0: > (XEN) Unable to find a compatible timer in the device tree > (XEN) **************************************** > > It seems like there are bits of the DT missing. I can however > successfully boot Xen with the Linux guest using the guest-loader device > and bypassing the firmware/boot code step. This gives: > > (XEN) MODULE[0]: 0000000040200000 - 000000004034d0c8 Xen > (XEN) MODULE[1]: 0000000048000000 - 0000000048100000 Device Tree > (XEN) MODULE[2]: 0000000046000000 - 0000000046eb2200 Kernel > (XEN) > (XEN) CMDLINE[0000000046000000]:chosen root=/dev/sda2 console=hvc0 earlyprintk=xen > (XEN) > (XEN) Command line: dom0_mem=4G dom0_max_vcpus=4 > (XEN) Domain heap initialised > (XEN) Booting using Device Tree > (XEN) Platform: Generic System > (XEN) Taking dtuart configuration from /chosen/stdout-path > (XEN) Looking for dtuart at "/pl011 at 9000000", options "" > Xen 4.15.1-pre > (XEN) Xen version 4.15.1-pre (alex.bennee@) (aarch64-linux-gnu-gcc (Debian 8.3.0-2) 8.3.0) debug=y Tue May 18 09:34:55 UTC 2021 > (XEN) Latest ChangeSet: > (XEN) build-id: a50d8f03a1a15662ac7c4e5f73f2f544a6739df2 > (XEN) Processor: 411fd070: "ARM Limited", variant: 0x1, part 0xd07, rev 0x0 > (XEN) 64-bit Execution: > (XEN) Processor Features: 0000000001000222 0000000000000000 > (XEN) Exception Levels: EL3:No EL2:64+32 EL1:64+32 EL0:64+32 > (XEN) Extensions: FloatingPoint AdvancedSIMD GICv3-SysReg > (XEN) Debug Features: 0000000010305106 0000000000000000 > (XEN) Auxiliary Features: 0000000000000000 0000000000000000 > (XEN) Memory Model Features: 0000000000001124 0000000000000000 > (XEN) ISA Features: 0000000000011120 0000000000000000 > (XEN) 32-bit Execution: > (XEN) Processor Features: 00000131:10011001 > (XEN) Instruction Sets: AArch32 A32 Thumb Thumb-2 Jazelle > (XEN) Extensions: GenericTimer > (XEN) Debug Features: 03010066 > (XEN) Auxiliary Features: 00000000 > (XEN) Memory Model Features: 10101105 40000000 01260000 02102211 > (XEN) ISA Features: 02101110 13112111 21232042 01112131 00011142 00011121 > (XEN) Using SMC Calling Convention v1.0 > (XEN) Using PSCI v0.2 > (XEN) SMP: Allowing 8 CPUs > (XEN) enabled workaround for: ARM erratum 832075 > (XEN) enabled workaround for: ARM erratum 834220 > (XEN) enabled workaround for: ARM erratum 1319367 > (XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 62500 KHz > (XEN) GICv3 initialization: > (XEN) gic_dist_addr=0x00000008000000 > (XEN) gic_maintenance_irq=25 > (XEN) gic_rdist_stride=0 > (XEN) gic_rdist_regions=1 > (XEN) redistributor regions: > (XEN) - region 0: 0x000000080a0000 - 0x00000009000000 > (XEN) GICv3: 256 lines, (IID 0000043b). > (XEN) GICv3: CPU0: Found redistributor in region 0 @000000004001c000 > > Attempting to boot with acpi=on still sees Grub attempt to use DT to > boot the hypervisor. However selecting the kernel directly boots with > ACPI (which is a shame as I'd like to see what FDT it gets presented > with).ACPI is not built by default in Xen on Arm. You will need to select it from Kconfig and rebuild the hypervisor.> > The full command line for booting via the guest-loader is: > > ./qemu-system-aarch64 \ > -machine virt,virtualization=on,gic-version=3 \ > -cpu max,pauth-impdef=on \ > -serial mon:stdio \ > -netdev user,id=net1,hostfwd=tcp::2222-:22 \ > -device virtio-net-pci,netdev=net1 \ > -device virtio-scsi-pci \ > -drive file=/dev/zvol/hackpool-0/debian-buster-arm64,id=hd0,index=0,if=none,format=raw \ > -device scsi-hd,drive=hd0 \ > -display none \ > -m 16384 \ > -kernel ~/lsrc/xen/xen.build.arm64-xen/xen/xen \ > -append "dom0_mem=4G dom0_max_vcpus=4" \ > -device guest-loader,addr=0x46000000,kernel=$HOME/lsrc/linux.git/builds/arm64/arch/arm64/boot/Image,bootargs="root=/dev/sda2 console=hvc0 earlyprintk=xen" \ > -smp 8 > > So some questions: > > - is Xen on arm64 tested on Debian Bullseye? If so what platform?I am using Debian Bullseye on QEMU and also the FVP. We are also using Debian in Osstest for all the testing (it is possible an older version of Debian).> - how do I tell Grub to do a straight FDT boot with the DT from the firmware?Is the firmware actually providing a DT? You could try to boot Xen from UEFI directly to confirm that. However, I vaguely recall that GRUB may only pass ACPI if it is provided.> - are there any missing pieces I should be aware of?Other than re-building Xen with ACPI=y, I am not aware of any issue to use Xen with Debian bullseye. Cheers, -- Julien Grall