Pkg xen devel - Jan 2019 - Bug#776450: Xen PVH support for grub-xen in Buster

On 1/6/19 11:21 AM, Colin Watson wrote:
> On Sat, Jan 05, 2019 at 10:51:13PM +0100, Hans van Kranenburg wrote:
>> On 1/5/19 8:04 PM, Colin Watson wrote:
>>> I think I'm OK with cherry-picking the relevant patch stack. 
Presumably
>>> this would need to be in new grub-xen-pvh-bin / grub-xen-pvh binary
>>> packages, as is usual for separate platform builds?
>>
>> I was thinking about having an additional...
>>
>>     /usr/lib/grub-xen/grub-i386-xen_pvh.bin
>>
>> ...in the grub-xen-host package, built with the same config file
>> (debian/grub-xen-host_grub.cfg in the packaging) and then...
>>
>>     /usr/lib/grub/i386-xen_pvh/*.mod
>>
>> ...in grub-xen-bin, just as an additional third option besides i386-xen
>> and x86_64-xen? This would be the most convenient "upgrade
path". If you
>> want to switch to PVH, just change kernel line in the xen guest config
>> file, and set type='pvh'.
> 
> Ah yes, I'd forgotten that i386-xen and x86_64-xen were both in
> grub-xen-bin.  I agree it seems reasonable enough to combine them, which
> simplifies things, and given that the boot loader (at least the first
> stage - see below) is read from the dom0 filesystem, it makes sense to
> include an image in grub-xen-host, although we may or may not be able to
> use the exact same config file.
For the test I just did, I just reused the same config files, as you can
see. I don't know if the first half of it makes sense, with the
@@PVBOOT_ARCH@@ things, but I'm going to leave that alone for now, after
reading your comments below.
>>> Can you elaborate on what you mean by "use as kernel image for
the Xen
>>> PVH virtual machine"?  In other words, what does the process
of
>>> installing a boot loader in a PVH guest look like?
>>
>> You don't. There is no BIOS, no boot loader, it is like booting a
PV
>> guest with grub. The whole 'trick' about PVH is that it's
HVM, but then
>> without the qemu etc part.
>>
>>
https://wiki.xen.org/wiki/Virtualization_Spectrum#Almost_fully_PV:_PVH_mode
>>
>> So, if I have this grub.cfg... (as example, not what we want for the
>> package)
> [...]
> 
> Thanks for clarifying all that.
Yes. So if the i386-xen_pvh stuff could be shipped in the same packages
(grub-xen-host) that users already have installed, then the user will
end up with a situation (without realizing) when upgrading a dom0 to
Buster (with Xen 4.11), in which the only thing (tm) that is needed to
do to convert a domU from PV to PVH is changing the kernel line in the
guest config file to the new grub-i386-xen_pvh.bin and adding a
type='pvh' line.

Well, for now that means that the only domU for which this will work is
one that has the debian buster 4.19 kernel, or the stretch-backports one
of course, but the future has to start somewhere. :)
>> So this is what I'm used to [0], but the way in which the Debian
>> grub-xen packages work is a bit different of course, since it tries to
>> do most of the process inside the virtual machine, and has a very
>> generic config file that delegates as much as possible to additional
>> grub config with access to modules inside the virtual machine itself.
> [...]
>>> (Compare
>>>
https://salsa.debian.org/grub-team/grub/blob/master/debian/patches/grub-install-pvxen-paths.patch
>>> which we carry for PV guests, although I gather PVH is sufficiently
>>> different that the same approach probably doesn't work.)
>>
>> Hm, I have never seen this /boot/xen/pvboot-*.elf stuff before, and I
>> also don't see any of those files in the grub-xen-* packages?
> 
> Anything in /boot is copied/generated by grub-install rather than being
> shipped in the packages themselves.
Yes, I see now.
> Let me give some background here.  The Xen PV boot protocol was the
> result of a series of discussions between me, Ian Jackson (Xen), Ian
> Campbell (Xen), Vladimir Serbinenko (GRUB), and possibly a few others
> I've forgotten, and I don't recall to what extent we ever wrote
down the
> entire rationale in one place.  In fact
> https://wiki.xen.org/wiki/PvGrub2 seems to have most of it now that I
> look, but I'll try to summarise it all here.
> 
> [...]
Thanks a lot for the explanation again, that really helps.
> Of course it was also possible to use pvgrub with the sort of scheme you
> describe, where the host declares how the guest kernel should boot.
> This works well enough in a small environment where there's a
> substantial amount of cooperation between the people running the host
> and the people running the guests, or where all the guests are
> sufficiently similar.  But it's not great for a larger or more generic
> environment, and we didn't want to rely on that degree of cooperation.
> (However, if you're operating a Xen host and that approach works for
> you, by all means go for it!  I understand that there exist environments
> where that approach is more convenient.)
Yup, that applies to our case at Mendix. We have roughly 7k domUs in
total (almost 15TiB of physical ram) running customer production
environments and they are all very much alike. Control that the customer
has is on a way higher level than shell access, so just choosing the
right pre-built standalone image with the right linux boot command line
works, since we make sure the to-be-booted linux kernel is always
symlinked as /vmlinuz etc using debian-style kernel packages.

And, all of it is going to be rebooted as PVH on Xen 4.11 during the
maintenance project that we're preparing now. \o/
> Given what you've described, I see no intrinsic reason we couldn't
do
> the same thing with PVH, but I'm pretty sure that GRUB's Xen
multiboot
> support doesn't yet support using the PVH entry point, so I don't
think
> it will work with the code as it stands.  It probably makes more sense
> to package what we have today (which I can do) and then work on the
> two-stage system as a further refinement.
Ok great. Since I'm not using (and probably not going to use) the
multi-stage multiboot things, this is harder for me to help with.
Anyhow, I like learning new things, and I liked figuring out things
while following your hints.

---- >8 ----

I have some additional questions about the way in which I'm creating my
own boot images now.

I just created a mirror of the pvgrub2 and pvhgrub2 repositories from
Mendix into my salsa account:

https://salsa.debian.org/knorrie-guest/pvgrub2
https://salsa.debian.org/knorrie-guest/pvhgrub2

The first one, pvgrub2 is what we've been using since 2014. It's a
simple debian package, which just build-depends on grub-xen...

https://salsa.debian.org/knorrie-guest/pvgrub2/blob/debian/master/debian/control

...which provides a grub-mkstandalone that has -O x86_64-xen...

https://salsa.debian.org/knorrie-guest/pvgrub2/blob/debian/master/Makefile

...so that the resulting package ends up containing a bunch of boot
images that end up on the dom0s.

Now, while trying to start doing the same for PVH, I just started by
copying the build output of grub master branch into the source package,
#yolo, and then I have a grub-mkstandalone that can do -O i386-xen_pvh...

https://salsa.debian.org/knorrie-guest/pvhgrub2/blob/master/Makefile

So my question is... Will the new grub-mkstandalone that I drag in as
build dependency be able to do -O x86_64-xen as well as -O i386-xen_pvh?

I see that /usr/bin/grub-mkstandalone is in grub-common, and the one I
get is "of architecture amd64". However, I don't understand enough
yet
to figure this out myself now.

Thanks,
Hans

On Mon, Jan 07, 2019 at 01:58:19AM +0100, Hans van Kranenburg
wrote:
> On 1/6/19 11:21 AM, Colin Watson wrote:
> > Given what you've described, I see no intrinsic reason we
couldn't do
> > the same thing with PVH, but I'm pretty sure that GRUB's Xen
multiboot
> > support doesn't yet support using the PVH entry point, so I
don't think
> > it will work with the code as it stands.  It probably makes more sense
> > to package what we have today (which I can do) and then work on the
> > two-stage system as a further refinement.
> 
> Ok great. Since I'm not using (and probably not going to use) the
> multi-stage multiboot things, this is harder for me to help with.
With any luck I should be able to work that bit out myself, since I
think stable's Xen has backported PVH support.
> Now, while trying to start doing the same for PVH, I just started by
> copying the build output of grub master branch into the source package,
> #yolo, and then I have a grub-mkstandalone that can do -O i386-xen_pvh...
> 
> https://salsa.debian.org/knorrie-guest/pvhgrub2/blob/master/Makefile
> 
> So my question is... Will the new grub-mkstandalone that I drag in as
> build dependency be able to do -O x86_64-xen as well as -O i386-xen_pvh?
Yes, that should work.  The tools are intentionally multi-target; in
order to support a given target they just need to have its modules etc.
available.

-- 
Colin Watson                                       [cjwatson at debian.org]

On Mon, Jan 07, 2019 at 08:44:50AM +0000, Colin Watson
wrote:
> On Mon, Jan 07, 2019 at 01:58:19AM +0100, Hans van Kranenburg wrote:
> > Ok great. Since I'm not using (and probably not going to use) the
> > multi-stage multiboot things, this is harder for me to help with.
> 
> With any luck I should be able to work that bit out myself, since I
> think stable's Xen has backported PVH support.
So it seems it doesn't quite have enough PVH support.  I get:

  xc: error: panic: xc_dom_core.c:702: xc_dom_find_loader: no loader found:
Invalid kernel
  libxl: error: libxl_dom.c:638:libxl__build_dom: xc_dom_parse_image failed: No
such file or directory
  libxl: error: libxl_create.c:1223:domcreate_rebuild_done: cannot (re-)build
domain: -3
  libxl: error: libxl.c:1575:libxl__destroy_domid: non-existant domain 8
  libxl: error: libxl.c:1534:domain_destroy_callback: unable to destroy guest
with domid 8
  libxl: error: libxl.c:1463:domain_destroy_cb: destruction of domain 8 failed

(I get the same with your recipe based on upstream master, so I don't
think it's a bug in my backport.)

I can upgrade this host to buster at some point in the nearish future,
but not quite right now.

Would you mind trying out the temporary pvh branch of
https://salsa.debian.org/grub-team/grub ?  I'd like to know whether the
resulting grub-xen-host binary package does the right thing, when built
in the ordinary way (I've test-built this branch using sbuild).

-- 
Colin Watson                                       [cjwatson at debian.org]