Pkg xen devel - Jan 2019 - Xen PVH support for grub-xen in Buster

Hi Debian Grub maintainer,

In december, Xen PVH support has been committed in grub master:
https://www.mail-archive.com/grub-devel at gnu.org/msg28125.html

The last pieces needed in the Linux kernel to boot PVH with grub2 landed
in Linux 4.20. I asked our kernel team to carry those patches on top of
the Linux 4.19 kernel that is going to ship in Buster, and that wish was
granted:

https://salsa.debian.org/kernel-team/linux/commit/4d63e6ccbbd6081068633b1147e0f77a59379795

Please see the commit message in there for an explanation why having the
possibility to boot PVH+grub in Debian Buster out of the box would be great.

So here's the question: Do you want to help completing the puzzle and
getting a PVH capable grub2 boot image in the grub-xen-* packages in Buster?

---- >8 ----

I'm currently running self-built boot images, and use the following
recipe to create those:

git clone https://git.savannah.gnu.org/git/grub.git
cd grub
./autogen.sh
rm -rf foo/build
mkdir -p foo/build
cd foo/build
../../configure TARGET_LDFLAGS=-static --target=i386 --with-platform=xen_pvh
make

Now, I create a grub.cfg file and then...

./grub-mkstandalone --grub-mkimage=./grub-mkimage -o
grub-i386-xen_pvh.bin -O i386-xen_pvh -d grub-core/
boot/grub/grub.cfg=grub.cfg

...I end up with something that I can use as kernel image for the Xen
PVH virtual machine.

---- >8 ----

I tried a bit to find out how to change debian/rules in the grub
packaging to make this happen in a similar way as the current support
for PV mode is done, but I have to admit that that's a bit above my
"paygrade". So, I can't attach some patch here yet that already
makes it
happen.

However, I'm available to help with testing etc. any time.

---- >8 ----

Slightly OT, but may be interesting for others reading this:

At work, I'm using this in a bit different way than the debian
grub-xen-host/bin package does, since I avoid having anything grub
related in virtual machines.

An example config file is:

root='(xen/xvda)'
insmod xzio
insmod gzio
insmod zstd
insmod btrfs
insmod ext2
echo 'Loading Linux ...'
linux /vmlinuz root=/dev/xvda ro console=hvc0 elevator=noop
echo 'Loading initial ramdisk ...'
initrd /initrd.img
echo 'There we go! ...'
boot

I have a bunch of those with different options etc, and we create the
images and package those up and install them on all the Xen hosts. Since
the disk is always xvda and the right kernel image is always symlinked,
this works for us.

---- >8 ----

Thanks,
Hans van Kranenburg

On Thu, Jan 03, 2019 at 01:02:47AM +0100, Hans van Kranenburg
wrote:
> In december, Xen PVH support has been committed in grub master:
> https://www.mail-archive.com/grub-devel at gnu.org/msg28125.html
> 
> The last pieces needed in the Linux kernel to boot PVH with grub2 landed
> in Linux 4.20. I asked our kernel team to carry those patches on top of
> the Linux 4.19 kernel that is going to ship in Buster, and that wish was
> granted:
> 
>
https://salsa.debian.org/kernel-team/linux/commit/4d63e6ccbbd6081068633b1147e0f77a59379795
> 
> Please see the commit message in there for an explanation why having the
> possibility to boot PVH+grub in Debian Buster out of the box would be
great.
> 
> So here's the question: Do you want to help completing the puzzle and
> getting a PVH capable grub2 boot image in the grub-xen-* packages in
Buster?
I think I'm OK with cherry-picking the relevant patch stack.  Presumably
this would need to be in new grub-xen-pvh-bin / grub-xen-pvh binary
packages, as is usual for separate platform builds?
> Now, I create a grub.cfg file and then...
> 
> ./grub-mkstandalone --grub-mkimage=./grub-mkimage -o
> grub-i386-xen_pvh.bin -O i386-xen_pvh -d grub-core/
> boot/grub/grub.cfg=grub.cfg
> 
> ...I end up with something that I can use as kernel image for the Xen
> PVH virtual machine.
Can you elaborate on what you mean by "use as kernel image for the Xen
PVH virtual machine"?  In other words, what does the process of
installing a boot loader in a PVH guest look like?  The current code in
upstream grub-install essentially just does a grub-mkimage but doesn't
have any platform-dependent install steps.

If you run upstream grub-install, are you then just supposed to write
'kernel = "/boot/grub/i386-xen_pvh/core.elf"' in your guest
configuration file?  In PVH mode, is the file name referenced via
"kernel" loaded from the dom0's filesystem or the domU's?

(Compare
https://salsa.debian.org/grub-team/grub/blob/master/debian/patches/grub-install-pvxen-paths.patch
which we carry for PV guests, although I gather PVH is sufficiently
different that the same approach probably doesn't work.)

-- 
Colin Watson                                       [cjwatson at debian.org]

Hi Colin,

Thanks for the quick reply.

On 1/5/19 8:04 PM, Colin Watson wrote:
> On Thu, Jan 03, 2019 at 01:02:47AM +0100, Hans van Kranenburg wrote:
>> In december, Xen PVH support has been committed in grub master:
>> https://www.mail-archive.com/grub-devel at gnu.org/msg28125.html
>>
>> The last pieces needed in the Linux kernel to boot PVH with grub2
landed
>> in Linux 4.20. I asked our kernel team to carry those patches on top of
>> the Linux 4.19 kernel that is going to ship in Buster, and that wish
was
>> granted:
>>
>>
https://salsa.debian.org/kernel-team/linux/commit/4d63e6ccbbd6081068633b1147e0f77a59379795
>>
>> Please see the commit message in there for an explanation why having
the
>> possibility to boot PVH+grub in Debian Buster out of the box would be
great.
>>
>> So here's the question: Do you want to help completing the puzzle
and
>> getting a PVH capable grub2 boot image in the grub-xen-* packages in
Buster?
> 
> I think I'm OK with cherry-picking the relevant patch stack. 
Presumably
> this would need to be in new grub-xen-pvh-bin / grub-xen-pvh binary
> packages, as is usual for separate platform builds?
I was thinking about having an additional...

    /usr/lib/grub-xen/grub-i386-xen_pvh.bin

...in the grub-xen-host package, built with the same config file
(debian/grub-xen-host_grub.cfg in the packaging) and then...

    /usr/lib/grub/i386-xen_pvh/*.mod

...in grub-xen-bin, just as an additional third option besides i386-xen
and x86_64-xen? This would be the most convenient "upgrade path". If
you
want to switch to PVH, just change kernel line in the xen guest config
file, and set type='pvh'.
>> Now, I create a grub.cfg file and then...
>>
>> ./grub-mkstandalone --grub-mkimage=./grub-mkimage -o
>> grub-i386-xen_pvh.bin -O i386-xen_pvh -d grub-core/
>> boot/grub/grub.cfg=grub.cfg
>>
>> ...I end up with something that I can use as kernel image for the Xen
>> PVH virtual machine.
> 
> Can you elaborate on what you mean by "use as kernel image for the Xen
> PVH virtual machine"?  In other words, what does the process of
> installing a boot loader in a PVH guest look like?
You don't. There is no BIOS, no boot loader, it is like booting a PV
guest with grub. The whole 'trick' about PVH is that it's HVM, but
then
without the qemu etc part.

https://wiki.xen.org/wiki/Virtualization_Spectrum#Almost_fully_PV:_PVH_mode

So, if I have this grub.cfg... (as example, not what we want for the
package)

---- >8 ----
root='(xen/xvda)'
insmod xzio
insmod gzio
insmod btrfs
insmod ext2
echo 'Loading Linux ...'
linux /vmlinuz root=/dev/xvda ro console=hvc0 elevator=noop
echo 'Loading initial ramdisk ...'
initrd /initrd.img
echo 'There we go! ...'
boot
---- >8 ----

...and then use grub-mkstandalone like above, and name my output file
flapsie.bin, and put it in / on the dom0, then I can do this in my xen
guest config file:

---- >8 ----
kernel = "/flapsie.bin"
type = "pvh"
---- >8 ----

Now if I start the Xen domU, it will load the kernel and initrd from the
filesystem inside the domU. No boot sector, no partition table, no boot
loader, no grub config file or anything grub related on the filesystem
of the virtual machine.

So this is what I'm used to [0], but the way in which the Debian
grub-xen packages work is a bit different of course, since it tries to
do most of the process inside the virtual machine, and has a very
generic config file that delegates as much as possible to additional
grub config with access to modules inside the virtual machine itself.

So the thought process I'm going through right now is how to add this
third option for the debian package.
> The current code in
> upstream grub-install essentially just does a grub-mkimage but doesn't
> have any platform-dependent install steps.
> 
> If you run upstream grub-install, are you then just supposed to write
> 'kernel = "/boot/grub/i386-xen_pvh/core.elf"' in your
guest
> configuration file?  In PVH mode, is the file name referenced via
> "kernel" loaded from the dom0's filesystem or the domU's?
Ok, does the above make this more clear? It's the same way of working as
for PV.
> (Compare
>
https://salsa.debian.org/grub-team/grub/blob/master/debian/patches/grub-install-pvxen-paths.patch
> which we carry for PV guests, although I gather PVH is sufficiently
> different that the same approach probably doesn't work.)
Hm, I have never seen this /boot/xen/pvboot-*.elf stuff before, and I
also don't see any of those files in the grub-xen-* packages?

I'll try to see if I can get things working when creating a grub image
using the grub-xen-host_grub.cfg like we have already, and then put
module stuff and a grub.cfg on the domU filesystem.

If that works, then it's a matter of figuring out how to get that
grub-i386-xen_pvh.bin and the modules built in the packaging?

Thanks,
Hans

[0] Background of the whole story how we ended up doing so for PV:
https://tech.mendix.com/linux/2014/09/06/pvgrub2/
https://tech.mendix.com/linux/2014/09/16/pvgrub2-live/