Hans van Kranenburg
2019-Jan-03 00:02 UTC
[Pkg-xen-devel] Xen PVH support for grub-xen in Buster
Hi Debian Grub maintainer, In december, Xen PVH support has been committed in grub master: https://www.mail-archive.com/grub-devel at gnu.org/msg28125.html The last pieces needed in the Linux kernel to boot PVH with grub2 landed in Linux 4.20. I asked our kernel team to carry those patches on top of the Linux 4.19 kernel that is going to ship in Buster, and that wish was granted: https://salsa.debian.org/kernel-team/linux/commit/4d63e6ccbbd6081068633b1147e0f77a59379795 Please see the commit message in there for an explanation why having the possibility to boot PVH+grub in Debian Buster out of the box would be great. So here's the question: Do you want to help completing the puzzle and getting a PVH capable grub2 boot image in the grub-xen-* packages in Buster? ---- >8 ---- I'm currently running self-built boot images, and use the following recipe to create those: git clone https://git.savannah.gnu.org/git/grub.git cd grub ./autogen.sh rm -rf foo/build mkdir -p foo/build cd foo/build ../../configure TARGET_LDFLAGS=-static --target=i386 --with-platform=xen_pvh make Now, I create a grub.cfg file and then... ./grub-mkstandalone --grub-mkimage=./grub-mkimage -o grub-i386-xen_pvh.bin -O i386-xen_pvh -d grub-core/ boot/grub/grub.cfg=grub.cfg ...I end up with something that I can use as kernel image for the Xen PVH virtual machine. ---- >8 ---- I tried a bit to find out how to change debian/rules in the grub packaging to make this happen in a similar way as the current support for PV mode is done, but I have to admit that that's a bit above my "paygrade". So, I can't attach some patch here yet that already makes it happen. However, I'm available to help with testing etc. any time. ---- >8 ---- Slightly OT, but may be interesting for others reading this: At work, I'm using this in a bit different way than the debian grub-xen-host/bin package does, since I avoid having anything grub related in virtual machines. An example config file is: root='(xen/xvda)' insmod xzio insmod gzio insmod zstd insmod btrfs insmod ext2 echo 'Loading Linux ...' linux /vmlinuz root=/dev/xvda ro console=hvc0 elevator=noop echo 'Loading initial ramdisk ...' initrd /initrd.img echo 'There we go! ...' boot I have a bunch of those with different options etc, and we create the images and package those up and install them on all the Xen hosts. Since the disk is always xvda and the right kernel image is always symlinked, this works for us. ---- >8 ---- Thanks, Hans van Kranenburg
Colin Watson
2019-Jan-05 19:04 UTC
[Pkg-xen-devel] Bug#776450: Xen PVH support for grub-xen in Buster
On Thu, Jan 03, 2019 at 01:02:47AM +0100, Hans van Kranenburg wrote:> In december, Xen PVH support has been committed in grub master: > https://www.mail-archive.com/grub-devel at gnu.org/msg28125.html > > The last pieces needed in the Linux kernel to boot PVH with grub2 landed > in Linux 4.20. I asked our kernel team to carry those patches on top of > the Linux 4.19 kernel that is going to ship in Buster, and that wish was > granted: > > https://salsa.debian.org/kernel-team/linux/commit/4d63e6ccbbd6081068633b1147e0f77a59379795 > > Please see the commit message in there for an explanation why having the > possibility to boot PVH+grub in Debian Buster out of the box would be great. > > So here's the question: Do you want to help completing the puzzle and > getting a PVH capable grub2 boot image in the grub-xen-* packages in Buster?I think I'm OK with cherry-picking the relevant patch stack. Presumably this would need to be in new grub-xen-pvh-bin / grub-xen-pvh binary packages, as is usual for separate platform builds?> Now, I create a grub.cfg file and then... > > ./grub-mkstandalone --grub-mkimage=./grub-mkimage -o > grub-i386-xen_pvh.bin -O i386-xen_pvh -d grub-core/ > boot/grub/grub.cfg=grub.cfg > > ...I end up with something that I can use as kernel image for the Xen > PVH virtual machine.Can you elaborate on what you mean by "use as kernel image for the Xen PVH virtual machine"? In other words, what does the process of installing a boot loader in a PVH guest look like? The current code in upstream grub-install essentially just does a grub-mkimage but doesn't have any platform-dependent install steps. If you run upstream grub-install, are you then just supposed to write 'kernel = "/boot/grub/i386-xen_pvh/core.elf"' in your guest configuration file? In PVH mode, is the file name referenced via "kernel" loaded from the dom0's filesystem or the domU's? (Compare https://salsa.debian.org/grub-team/grub/blob/master/debian/patches/grub-install-pvxen-paths.patch which we carry for PV guests, although I gather PVH is sufficiently different that the same approach probably doesn't work.) -- Colin Watson [cjwatson at debian.org]
Hans van Kranenburg
2019-Jan-05 21:51 UTC
[Pkg-xen-devel] Bug#776450: Xen PVH support for grub-xen in Buster
Hi Colin, Thanks for the quick reply. On 1/5/19 8:04 PM, Colin Watson wrote:> On Thu, Jan 03, 2019 at 01:02:47AM +0100, Hans van Kranenburg wrote: >> In december, Xen PVH support has been committed in grub master: >> https://www.mail-archive.com/grub-devel at gnu.org/msg28125.html >> >> The last pieces needed in the Linux kernel to boot PVH with grub2 landed >> in Linux 4.20. I asked our kernel team to carry those patches on top of >> the Linux 4.19 kernel that is going to ship in Buster, and that wish was >> granted: >> >> https://salsa.debian.org/kernel-team/linux/commit/4d63e6ccbbd6081068633b1147e0f77a59379795 >> >> Please see the commit message in there for an explanation why having the >> possibility to boot PVH+grub in Debian Buster out of the box would be great. >> >> So here's the question: Do you want to help completing the puzzle and >> getting a PVH capable grub2 boot image in the grub-xen-* packages in Buster? > > I think I'm OK with cherry-picking the relevant patch stack. Presumably > this would need to be in new grub-xen-pvh-bin / grub-xen-pvh binary > packages, as is usual for separate platform builds?I was thinking about having an additional... /usr/lib/grub-xen/grub-i386-xen_pvh.bin ...in the grub-xen-host package, built with the same config file (debian/grub-xen-host_grub.cfg in the packaging) and then... /usr/lib/grub/i386-xen_pvh/*.mod ...in grub-xen-bin, just as an additional third option besides i386-xen and x86_64-xen? This would be the most convenient "upgrade path". If you want to switch to PVH, just change kernel line in the xen guest config file, and set type='pvh'.>> Now, I create a grub.cfg file and then... >> >> ./grub-mkstandalone --grub-mkimage=./grub-mkimage -o >> grub-i386-xen_pvh.bin -O i386-xen_pvh -d grub-core/ >> boot/grub/grub.cfg=grub.cfg >> >> ...I end up with something that I can use as kernel image for the Xen >> PVH virtual machine. > > Can you elaborate on what you mean by "use as kernel image for the Xen > PVH virtual machine"? In other words, what does the process of > installing a boot loader in a PVH guest look like?You don't. There is no BIOS, no boot loader, it is like booting a PV guest with grub. The whole 'trick' about PVH is that it's HVM, but then without the qemu etc part. https://wiki.xen.org/wiki/Virtualization_Spectrum#Almost_fully_PV:_PVH_mode So, if I have this grub.cfg... (as example, not what we want for the package) ---- >8 ---- root='(xen/xvda)' insmod xzio insmod gzio insmod btrfs insmod ext2 echo 'Loading Linux ...' linux /vmlinuz root=/dev/xvda ro console=hvc0 elevator=noop echo 'Loading initial ramdisk ...' initrd /initrd.img echo 'There we go! ...' boot ---- >8 ---- ...and then use grub-mkstandalone like above, and name my output file flapsie.bin, and put it in / on the dom0, then I can do this in my xen guest config file: ---- >8 ---- kernel = "/flapsie.bin" type = "pvh" ---- >8 ---- Now if I start the Xen domU, it will load the kernel and initrd from the filesystem inside the domU. No boot sector, no partition table, no boot loader, no grub config file or anything grub related on the filesystem of the virtual machine. So this is what I'm used to [0], but the way in which the Debian grub-xen packages work is a bit different of course, since it tries to do most of the process inside the virtual machine, and has a very generic config file that delegates as much as possible to additional grub config with access to modules inside the virtual machine itself. So the thought process I'm going through right now is how to add this third option for the debian package.> The current code in > upstream grub-install essentially just does a grub-mkimage but doesn't > have any platform-dependent install steps. > > If you run upstream grub-install, are you then just supposed to write > 'kernel = "/boot/grub/i386-xen_pvh/core.elf"' in your guest > configuration file? In PVH mode, is the file name referenced via > "kernel" loaded from the dom0's filesystem or the domU's?Ok, does the above make this more clear? It's the same way of working as for PV.> (Compare > https://salsa.debian.org/grub-team/grub/blob/master/debian/patches/grub-install-pvxen-paths.patch > which we carry for PV guests, although I gather PVH is sufficiently > different that the same approach probably doesn't work.)Hm, I have never seen this /boot/xen/pvboot-*.elf stuff before, and I also don't see any of those files in the grub-xen-* packages? I'll try to see if I can get things working when creating a grub image using the grub-xen-host_grub.cfg like we have already, and then put module stuff and a grub.cfg on the domU filesystem. If that works, then it's a matter of figuring out how to get that grub-i386-xen_pvh.bin and the modules built in the packaging? Thanks, Hans [0] Background of the whole story how we ended up doing so for PV: https://tech.mendix.com/linux/2014/09/06/pvgrub2/ https://tech.mendix.com/linux/2014/09/16/pvgrub2-live/