Pkg xen devel - Jan 2019 - Bug#776450: Xen PVH support for grub-xen in Buster

Hi Colin,

Thanks for the quick reply.

On 1/5/19 8:04 PM, Colin Watson wrote:
> On Thu, Jan 03, 2019 at 01:02:47AM +0100, Hans van Kranenburg wrote:
>> In december, Xen PVH support has been committed in grub master:
>> https://www.mail-archive.com/grub-devel at gnu.org/msg28125.html
>>
>> The last pieces needed in the Linux kernel to boot PVH with grub2
landed
>> in Linux 4.20. I asked our kernel team to carry those patches on top of
>> the Linux 4.19 kernel that is going to ship in Buster, and that wish
was
>> granted:
>>
>>
https://salsa.debian.org/kernel-team/linux/commit/4d63e6ccbbd6081068633b1147e0f77a59379795
>>
>> Please see the commit message in there for an explanation why having
the
>> possibility to boot PVH+grub in Debian Buster out of the box would be
great.
>>
>> So here's the question: Do you want to help completing the puzzle
and
>> getting a PVH capable grub2 boot image in the grub-xen-* packages in
Buster?
> 
> I think I'm OK with cherry-picking the relevant patch stack. 
Presumably
> this would need to be in new grub-xen-pvh-bin / grub-xen-pvh binary
> packages, as is usual for separate platform builds?
I was thinking about having an additional...

    /usr/lib/grub-xen/grub-i386-xen_pvh.bin

...in the grub-xen-host package, built with the same config file
(debian/grub-xen-host_grub.cfg in the packaging) and then...

    /usr/lib/grub/i386-xen_pvh/*.mod

...in grub-xen-bin, just as an additional third option besides i386-xen
and x86_64-xen? This would be the most convenient "upgrade path". If
you
want to switch to PVH, just change kernel line in the xen guest config
file, and set type='pvh'.
>> Now, I create a grub.cfg file and then...
>>
>> ./grub-mkstandalone --grub-mkimage=./grub-mkimage -o
>> grub-i386-xen_pvh.bin -O i386-xen_pvh -d grub-core/
>> boot/grub/grub.cfg=grub.cfg
>>
>> ...I end up with something that I can use as kernel image for the Xen
>> PVH virtual machine.
> 
> Can you elaborate on what you mean by "use as kernel image for the Xen
> PVH virtual machine"?  In other words, what does the process of
> installing a boot loader in a PVH guest look like?
You don't. There is no BIOS, no boot loader, it is like booting a PV
guest with grub. The whole 'trick' about PVH is that it's HVM, but
then
without the qemu etc part.

https://wiki.xen.org/wiki/Virtualization_Spectrum#Almost_fully_PV:_PVH_mode

So, if I have this grub.cfg... (as example, not what we want for the
package)

---- >8 ----
root='(xen/xvda)'
insmod xzio
insmod gzio
insmod btrfs
insmod ext2
echo 'Loading Linux ...'
linux /vmlinuz root=/dev/xvda ro console=hvc0 elevator=noop
echo 'Loading initial ramdisk ...'
initrd /initrd.img
echo 'There we go! ...'
boot
---- >8 ----

...and then use grub-mkstandalone like above, and name my output file
flapsie.bin, and put it in / on the dom0, then I can do this in my xen
guest config file:

---- >8 ----
kernel = "/flapsie.bin"
type = "pvh"
---- >8 ----

Now if I start the Xen domU, it will load the kernel and initrd from the
filesystem inside the domU. No boot sector, no partition table, no boot
loader, no grub config file or anything grub related on the filesystem
of the virtual machine.

So this is what I'm used to [0], but the way in which the Debian
grub-xen packages work is a bit different of course, since it tries to
do most of the process inside the virtual machine, and has a very
generic config file that delegates as much as possible to additional
grub config with access to modules inside the virtual machine itself.

So the thought process I'm going through right now is how to add this
third option for the debian package.
> The current code in
> upstream grub-install essentially just does a grub-mkimage but doesn't
> have any platform-dependent install steps.
> 
> If you run upstream grub-install, are you then just supposed to write
> 'kernel = "/boot/grub/i386-xen_pvh/core.elf"' in your
guest
> configuration file?  In PVH mode, is the file name referenced via
> "kernel" loaded from the dom0's filesystem or the domU's?
Ok, does the above make this more clear? It's the same way of working as
for PV.
> (Compare
>
https://salsa.debian.org/grub-team/grub/blob/master/debian/patches/grub-install-pvxen-paths.patch
> which we carry for PV guests, although I gather PVH is sufficiently
> different that the same approach probably doesn't work.)
Hm, I have never seen this /boot/xen/pvboot-*.elf stuff before, and I
also don't see any of those files in the grub-xen-* packages?

I'll try to see if I can get things working when creating a grub image
using the grub-xen-host_grub.cfg like we have already, and then put
module stuff and a grub.cfg on the domU filesystem.

If that works, then it's a matter of figuring out how to get that
grub-i386-xen_pvh.bin and the modules built in the packaging?

Thanks,
Hans

[0] Background of the whole story how we ended up doing so for PV:
https://tech.mendix.com/linux/2014/09/06/pvgrub2/
https://tech.mendix.com/linux/2014/09/16/pvgrub2-live/

On 1/5/19 10:51 PM, Hans van Kranenburg wrote:
> [...]
> 
> I'll try to see if I can get things working when creating a grub image
> using the grub-xen-host_grub.cfg like we have already, and then put
> module stuff and a grub.cfg on the domU filesystem.
Hm, so I have a PV (or PVH) domU in a test environment here, and I tried
to use it in the Debian way of booting it.

If I install grub-xen-bin inside the domU then I get some stuff, but not
update-grub. When I just also install grub2 it drags in more stuff, and
then I end up with update-grub and a /boot/grub/grub.cfg file as result.

If I install grub-xen-host in the dom0 and use kernel
"/usr/lib/grub-xen/grub-x86_64-xen.bin" in the guest config, then I
get
a nice ascii art blue grub menu on the serial console when starting the
domU with xen create -c.

So, that's good, that's the already working PV scenario.

Now I want to have the same for PVH.

The config file I'm using is (debian.cfg):

---- >8 ----
if search -s -f /boot/grub/grub.cfg ; then
	echo "Reading (${root})/boot/grub/grub.cfg"
	configfile /boot/grub/grub.cfg
fi

if search -s -f /grub/grub.cfg ; then
	echo "Reading (${root})/grub/grub.cfg"
	configfile /grub/grub.cfg
fi
---- >8 ----

Now I do (with my custom compile blah as shown before):

-$ ./grub-mkstandalone --grub-mkimage=./grub-mkimage -o
out/grub-i386-xen_pvh-debian -O i386-xen_pvh -d grub-core/
boot/grub/grub.cfg=debian.cfg

-$ scp out/grub-i386-xen_pvh-debian root at dom0:

In the guest config file on the dom0, I've set:

---- >8 ----
kernel = "/root/grub-i386-xen_pvh-debian"
type = "pvh"
---- >8 ----

I just copied *.mod that I got from my grub build (with the make blah
shown before) into /usr/lib/grub/i386-xen_pvh of the domU, and that's
probably a very naive thing to do, since I mostly don't know what I'm
doing at this point. I especially have no idea how the booting grub
image from the dom0 would know to do anything with a file from
/usr/lib/grub/<whatever> inside the domU filesystem, but apparently the
current PV things do that.

When trying to start that, I get:

Reading (memdisk)/boot/grub/grub.cfg

and then it exits back to the dom0 prompt and the domU is gone.

This is where my been-there-done-that ends. It still feels like it's
very close, but I have no clue.

Hopefully this will ring some bells. :)

If needed, we can ask some help from the author of the xen_pvh grub
patchset.

Thanks,
Hans

On Sat, Jan 05, 2019 at 10:51:13PM +0100, Hans van Kranenburg
wrote:
> On 1/5/19 8:04 PM, Colin Watson wrote:
> > I think I'm OK with cherry-picking the relevant patch stack. 
Presumably
> > this would need to be in new grub-xen-pvh-bin / grub-xen-pvh binary
> > packages, as is usual for separate platform builds?
> 
> I was thinking about having an additional...
> 
>     /usr/lib/grub-xen/grub-i386-xen_pvh.bin
> 
> ...in the grub-xen-host package, built with the same config file
> (debian/grub-xen-host_grub.cfg in the packaging) and then...
> 
>     /usr/lib/grub/i386-xen_pvh/*.mod
> 
> ...in grub-xen-bin, just as an additional third option besides i386-xen
> and x86_64-xen? This would be the most convenient "upgrade path".
If you
> want to switch to PVH, just change kernel line in the xen guest config
> file, and set type='pvh'.
Ah yes, I'd forgotten that i386-xen and x86_64-xen were both in
grub-xen-bin.  I agree it seems reasonable enough to combine them, which
simplifies things, and given that the boot loader (at least the first
stage - see below) is read from the dom0 filesystem, it makes sense to
include an image in grub-xen-host, although we may or may not be able to
use the exact same config file.
> > Can you elaborate on what you mean by "use as kernel image for
the Xen
> > PVH virtual machine"?  In other words, what does the process of
> > installing a boot loader in a PVH guest look like?
> 
> You don't. There is no BIOS, no boot loader, it is like booting a PV
> guest with grub. The whole 'trick' about PVH is that it's HVM,
but then
> without the qemu etc part.
> 
> https://wiki.xen.org/wiki/Virtualization_Spectrum#Almost_fully_PV:_PVH_mode
> 
> So, if I have this grub.cfg... (as example, not what we want for the
> package)
[...]

Thanks for clarifying all that.
> So this is what I'm used to [0], but the way in which the Debian
> grub-xen packages work is a bit different of course, since it tries to
> do most of the process inside the virtual machine, and has a very
> generic config file that delegates as much as possible to additional
> grub config with access to modules inside the virtual machine itself.
[...]
> > (Compare
> >
https://salsa.debian.org/grub-team/grub/blob/master/debian/patches/grub-install-pvxen-paths.patch
> > which we carry for PV guests, although I gather PVH is sufficiently
> > different that the same approach probably doesn't work.)
> 
> Hm, I have never seen this /boot/xen/pvboot-*.elf stuff before, and I
> also don't see any of those files in the grub-xen-* packages?
Anything in /boot is copied/generated by grub-install rather than being
shipped in the packages themselves.

Let me give some background here.  The Xen PV boot protocol was the
result of a series of discussions between me, Ian Jackson (Xen), Ian
Campbell (Xen), Vladimir Serbinenko (GRUB), and possibly a few others
I've forgotten, and I don't recall to what extent we ever wrote down the
entire rationale in one place.  In fact
https://wiki.xen.org/wiki/PvGrub2 seems to have most of it now that I
look, but I'll try to summarise it all here.

Historically (before GRUB 2.02), there were two main choices for booting
a PV guest using GRUB:

 * Use pygrub.  This was very flexible, because it parsed boot loader
   configuration files from the guest's filesystem and decided what
   kernel and initramfs to boot based on that.  However, getting the
   configuration file parsing right was a difficult and moving target,
   and this system was inherently somewhat insecure because it required
   complex parsing of untrusted guest inputs in the dom0.

 * Use the PV port of GRUB Legacy (pvgrub).  This generally worked well
   and fixed the inherent security problems of pygrub because all the
   parsing work was done in the guest context, but it was a very old
   fork of unmaintained code and expected the GRUB Legacy configuration
   file syntax.  Guests running on a host using this generally had to
   carry around otherwise-obsolete code to generate appropriate menu.lst
   files so that they could declare how they expected to be booted.

Of course it was also possible to use pvgrub with the sort of scheme you
describe, where the host declares how the guest kernel should boot.
This works well enough in a small environment where there's a
substantial amount of cooperation between the people running the host
and the people running the guests, or where all the guests are
sufficiently similar.  But it's not great for a larger or more generic
environment, and we didn't want to rely on that degree of cooperation.
(However, if you're operating a Xen host and that approach works for
you, by all means go for it!  I understand that there exist environments
where that approach is more convenient.)

So, around the time Vladimir was doing the core PV porting work for GRUB
2, we were working on a better design for this.  The first pass was just
to provide the host with a pvgrub2 image which it would boot in the
guest context, configured to look for /boot/grub/grub.cfg in the guest.
As long as the guest's needs are simple enough, this works OK.  However,
it relies on certain compatibility assumptions: specifically that the
guest doesn't use any configuration file features from versions of GRUB
more recent than the version that the host has, and that the guest's
grub.cfg doesn't need to load any modules not present in the host's GRUB
image (since modules in the guest's filesystem may very well not be
compatible with the host's image).

So, the second pass was the two-stage arrangement that you see in the
current packaging.  The host has a fairly simple generic PV GRUB image
which it boots in the guest context; the first thing that that image
does is look for a second-stage loader in a defined place in the guest
filesystem and chainload it (with fallbacks in case it isn't present).
This means that the guest can take full control of its own booting
process: it can use any features of whatever version of GRUB it likes,
as long as it supports the correct entry point, and needs no changes to
the host in order to do so.  This is particularly good if you're working
on GRUB itself, of course, but the looser coupling also generally makes
the process more robust.

Given what you've described, I see no intrinsic reason we couldn't do
the same thing with PVH, but I'm pretty sure that GRUB's Xen multiboot
support doesn't yet support using the PVH entry point, so I don't think
it will work with the code as it stands.  It probably makes more sense
to package what we have today (which I can do) and then work on the
two-stage system as a further refinement.

-- 
Colin Watson                                       [cjwatson at debian.org]

On Sun, Jan 06, 2019 at 01:45:53AM +0100, Hans van Kranenburg
wrote:
> Hm, so I have a PV (or PVH) domU in a test environment here, and I tried
> to use it in the Debian way of booting it.
> 
> If I install grub-xen-bin inside the domU then I get some stuff, but not
> update-grub. When I just also install grub2 it drags in more stuff, and
> then I end up with update-grub and a /boot/grub/grub.cfg file as result.
For the record, you wanted grub-xen rather than grub-xen-bin.
> I just copied *.mod that I got from my grub build (with the make blah
> shown before) into /usr/lib/grub/i386-xen_pvh of the domU,
I'd expect them to need to be in /boot/grub/i386-xen_pvh/ in order for
the boot loader to read them; grub-install copies modules there.
(However, a standalone image normally shouldn't need to read modules
from the guest anyway.)
> When trying to start that, I get:
> 
> Reading (memdisk)/boot/grub/grub.cfg
> 
> and then it exits back to the dom0 prompt and the domU is gone.
Seeing (memdisk) there suggests that $root is wrong.  I think that
putting your bootstrap configuration file in /boot/grub/grub.cfg in the
memdisk is a recipe for confusion; grub-xen-host puts it in /grub.cfg
instead, and generally hooks things up a bit differently.  You can
inspect debian/rules to see what it does.

Anyway, this sounds close enough that we can probably go for it and see
where it lands :-)

-- 
Colin Watson                                       [cjwatson at debian.org]

On 1/6/19 11:21 AM, Colin Watson wrote:
> On Sat, Jan 05, 2019 at 10:51:13PM +0100, Hans van Kranenburg wrote:
>> On 1/5/19 8:04 PM, Colin Watson wrote:
>>> I think I'm OK with cherry-picking the relevant patch stack. 
Presumably
>>> this would need to be in new grub-xen-pvh-bin / grub-xen-pvh binary
>>> packages, as is usual for separate platform builds?
>>
>> I was thinking about having an additional...
>>
>>     /usr/lib/grub-xen/grub-i386-xen_pvh.bin
>>
>> ...in the grub-xen-host package, built with the same config file
>> (debian/grub-xen-host_grub.cfg in the packaging) and then...
>>
>>     /usr/lib/grub/i386-xen_pvh/*.mod
>>
>> ...in grub-xen-bin, just as an additional third option besides i386-xen
>> and x86_64-xen? This would be the most convenient "upgrade
path". If you
>> want to switch to PVH, just change kernel line in the xen guest config
>> file, and set type='pvh'.
> 
> Ah yes, I'd forgotten that i386-xen and x86_64-xen were both in
> grub-xen-bin.  I agree it seems reasonable enough to combine them, which
> simplifies things, and given that the boot loader (at least the first
> stage - see below) is read from the dom0 filesystem, it makes sense to
> include an image in grub-xen-host, although we may or may not be able to
> use the exact same config file.
For the test I just did, I just reused the same config files, as you can
see. I don't know if the first half of it makes sense, with the
@@PVBOOT_ARCH@@ things, but I'm going to leave that alone for now, after
reading your comments below.
>>> Can you elaborate on what you mean by "use as kernel image for
the Xen
>>> PVH virtual machine"?  In other words, what does the process
of
>>> installing a boot loader in a PVH guest look like?
>>
>> You don't. There is no BIOS, no boot loader, it is like booting a
PV
>> guest with grub. The whole 'trick' about PVH is that it's
HVM, but then
>> without the qemu etc part.
>>
>>
https://wiki.xen.org/wiki/Virtualization_Spectrum#Almost_fully_PV:_PVH_mode
>>
>> So, if I have this grub.cfg... (as example, not what we want for the
>> package)
> [...]
> 
> Thanks for clarifying all that.
Yes. So if the i386-xen_pvh stuff could be shipped in the same packages
(grub-xen-host) that users already have installed, then the user will
end up with a situation (without realizing) when upgrading a dom0 to
Buster (with Xen 4.11), in which the only thing (tm) that is needed to
do to convert a domU from PV to PVH is changing the kernel line in the
guest config file to the new grub-i386-xen_pvh.bin and adding a
type='pvh' line.

Well, for now that means that the only domU for which this will work is
one that has the debian buster 4.19 kernel, or the stretch-backports one
of course, but the future has to start somewhere. :)
>> So this is what I'm used to [0], but the way in which the Debian
>> grub-xen packages work is a bit different of course, since it tries to
>> do most of the process inside the virtual machine, and has a very
>> generic config file that delegates as much as possible to additional
>> grub config with access to modules inside the virtual machine itself.
> [...]
>>> (Compare
>>>
https://salsa.debian.org/grub-team/grub/blob/master/debian/patches/grub-install-pvxen-paths.patch
>>> which we carry for PV guests, although I gather PVH is sufficiently
>>> different that the same approach probably doesn't work.)
>>
>> Hm, I have never seen this /boot/xen/pvboot-*.elf stuff before, and I
>> also don't see any of those files in the grub-xen-* packages?
> 
> Anything in /boot is copied/generated by grub-install rather than being
> shipped in the packages themselves.
Yes, I see now.
> Let me give some background here.  The Xen PV boot protocol was the
> result of a series of discussions between me, Ian Jackson (Xen), Ian
> Campbell (Xen), Vladimir Serbinenko (GRUB), and possibly a few others
> I've forgotten, and I don't recall to what extent we ever wrote
down the
> entire rationale in one place.  In fact
> https://wiki.xen.org/wiki/PvGrub2 seems to have most of it now that I
> look, but I'll try to summarise it all here.
> 
> [...]
Thanks a lot for the explanation again, that really helps.
> Of course it was also possible to use pvgrub with the sort of scheme you
> describe, where the host declares how the guest kernel should boot.
> This works well enough in a small environment where there's a
> substantial amount of cooperation between the people running the host
> and the people running the guests, or where all the guests are
> sufficiently similar.  But it's not great for a larger or more generic
> environment, and we didn't want to rely on that degree of cooperation.
> (However, if you're operating a Xen host and that approach works for
> you, by all means go for it!  I understand that there exist environments
> where that approach is more convenient.)
Yup, that applies to our case at Mendix. We have roughly 7k domUs in
total (almost 15TiB of physical ram) running customer production
environments and they are all very much alike. Control that the customer
has is on a way higher level than shell access, so just choosing the
right pre-built standalone image with the right linux boot command line
works, since we make sure the to-be-booted linux kernel is always
symlinked as /vmlinuz etc using debian-style kernel packages.

And, all of it is going to be rebooted as PVH on Xen 4.11 during the
maintenance project that we're preparing now. \o/
> Given what you've described, I see no intrinsic reason we couldn't
do
> the same thing with PVH, but I'm pretty sure that GRUB's Xen
multiboot
> support doesn't yet support using the PVH entry point, so I don't
think
> it will work with the code as it stands.  It probably makes more sense
> to package what we have today (which I can do) and then work on the
> two-stage system as a further refinement.
Ok great. Since I'm not using (and probably not going to use) the
multi-stage multiboot things, this is harder for me to help with.
Anyhow, I like learning new things, and I liked figuring out things
while following your hints.

---- >8 ----

I have some additional questions about the way in which I'm creating my
own boot images now.

I just created a mirror of the pvgrub2 and pvhgrub2 repositories from
Mendix into my salsa account:

https://salsa.debian.org/knorrie-guest/pvgrub2
https://salsa.debian.org/knorrie-guest/pvhgrub2

The first one, pvgrub2 is what we've been using since 2014. It's a
simple debian package, which just build-depends on grub-xen...

https://salsa.debian.org/knorrie-guest/pvgrub2/blob/debian/master/debian/control

...which provides a grub-mkstandalone that has -O x86_64-xen...

https://salsa.debian.org/knorrie-guest/pvgrub2/blob/debian/master/Makefile

...so that the resulting package ends up containing a bunch of boot
images that end up on the dom0s.

Now, while trying to start doing the same for PVH, I just started by
copying the build output of grub master branch into the source package,
#yolo, and then I have a grub-mkstandalone that can do -O i386-xen_pvh...

https://salsa.debian.org/knorrie-guest/pvhgrub2/blob/master/Makefile

So my question is... Will the new grub-mkstandalone that I drag in as
build dependency be able to do -O x86_64-xen as well as -O i386-xen_pvh?

I see that /usr/bin/grub-mkstandalone is in grub-common, and the one I
get is "of architecture amd64". However, I don't understand enough
yet
to figure this out myself now.

Thanks,
Hans