Pkg xen devel - Mar 2017 - Bug#858962: Request: enable OVMF at build time in 4.8 as it does not require non-free anymore

Package: src:xen
Version: 4.8.1~pre.2017.01.23-1
Severity: wishlist

Dear Maintainer,

In Xen 4.8 it is possible to enable OVMF without the need for any OVMF code or
binary to be on the system.
Currently, OVMF is not enabled, probably because it used to require OVMF at
compile time which would
make for a hard dependency on non-free code. Since this is no longer the case,
you could make it a run-time
option by enabling OVMF for this package, and when a user would want to actually
use it, they would only need
to add a OVMF binary to a preset path themselves (i.e. by installing the
non-free ovmf package).

I tested this on a strech machine, by getting the sources using apt-get source
and editing debian/rules.real.
Around line 74 the compile-time options for Xen are listed, and enabling OVMF is
as simple as adding one line:

                        --enable-ovmf --with-system-ovmf=/usr/share/ovmf/OVMF.fd

The path specified does not need to exist at compile time. In this case I chose
to use the path where the OVMF package
installs the binary firmware file so it can be automatically used.

I'm not sure how to create a .patch file for this, but since it's one
line with very clear results, I hope it's sufficient.

Regards,
John Keates

-- System Information:
Debian Release: 9.0
  APT prefers testing
  APT policy: (650, 'testing')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.8.0-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

xen-hypervisor-4.8-amd64 depends on no packages.

Versions of packages xen-hypervisor-4.8-amd64 recommends:
ii  xen-utils-4.8  4.8.1~pre.2017.01.23-1

xen-hypervisor-4.8-amd64 suggests no packages.

-- Configuration Files:
/etc/default/grub.d/xen.cfg changed [not included]

-- no debconf information

Control: tags -1 patch

John Keates writes ("Bug#858962: Request: enable OVMF at build time in 4.8
as it does not require non-free anymore"):
> Package: src:xen
> Version: 4.8.1~pre.2017.01.23-1
> Severity: wishlist
Hi.  Thanks for the report.
> Currently, OVMF is not enabled, probably because it used to require
> OVMF at compile time which would make for a hard dependency on
> non-free code. Since this is no longer the case, you could make it a
> run-time option by enabling OVMF for this package, and when a user
> would want to actually use it, they would only need to add a OVMF
> binary to a preset path themselves (i.e. by installing the non-free
> ovmf package).
I looked into this OVMF nonfreeness and it seems to be fixed:
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=815618
> Around line 74 the compile-time options for Xen are listed, and enabling
OVMF is as simple as adding one line:
> 
>                --enable-ovmf --with-system-ovmf=/usr/share/ovmf/OVMF.fd
> 
> The path specified does not need to exist at compile time. In this
> case I chose to use the path where the OVMF package installs the
> binary firmware file so it can be automatically used.
Thanks for this testing.

I think that, given that UEFI is becoming quite common, it might be
worth adding a dependency of some kind to the Debian Xen packages, as
well as simply enabling ovmf support.  But that's probably not a
blocker for fixing this.
> I'm not sure how to create a .patch file for this, but since it's
one line with very clear results, I hope it's sufficient.
Indeed.

However: I do not intend to make this change at this stage of the
stretch release freeze.

If you (or someone else) gets preapproval from the release team, then
I would be prepared to do.  But I think such approval would probably
be refused for good reasons.  If someone wants to make such an
approval request, please let me know and I may be able to help
pre-review it etc. - on the condition that this isn't taken as
endorsement :-).

Thanks,
Ian.

Processing control commands:
> tags -1 patch
Bug #858962 [src:xen] Request: enable OVMF at build time in 4.8 as it does not
require non-free anymore
Added tag(s) patch.

-- 
858962: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=858962
Debian Bug Tracking System
Contact owner at bugs.debian.org with problems

>> I'm not sure how to create a .patch file for this, but since
it's one line with very clear results, I hope it's sufficient.
> 
> Indeed.
> 
> However: I do not intend to make this change at this stage of the
> stretch release freeze.
I suppose that makes sense since in a freeze you wouldn?t want ?new? features
added.
> If you (or someone else) gets preapproval from the release team, then
> I would be prepared to do.  But I think such approval would probably
> be refused for good reasons.  If someone wants to make such an
> approval request, please let me know and I may be able to help
> pre-review it etc. - on the condition that this isn't taken as
> endorsement :-).
> 
> Thanks,
> Ian.
I really think this should be enabled, it?s side-effects are nil by default as
far as I know, and it?s been in there for a long time, but simply not enabled.
Enabling this will allow virtualisation of operating systems that don?t support
BIOS-style machine firmware in HVM mode. On top of that, it allows
virtualising Apple?s macOS since that requires EFI. Legally, that requires
Debian to be installed in Apple hardware, which happens to be what I?m doing, so
that?s fine too.

Some Windows variants will only boot on EFI enabled machines too, so there is a
need for OVMF there as well. Currently, some people resort to using KVM since it
works there,
but I?d rather see it working with Xen. Same goes for some ARM setups where you
can?t boot HVM machines without EFI.

Where would I go to make a request for approval from the release team?

Thanks,
John

> On 29 Mar 2017, at 12:51, Ian Jackson <ian.jackson at eu.citrix.com>
wrote:
> 
> Control: tags -1 patch
> 
> John Keates writes ("Bug#858962: Request: enable OVMF at build time in
4.8 as it does not require non-free anymore"):
>> Package: src:xen
>> Version: 4.8.1~pre.2017.01.23-1
>> Severity: wishlist
> 
> Hi.  Thanks for the report.
> 
>> Currently, OVMF is not enabled, probably because it used to require
>> OVMF at compile time which would make for a hard dependency on
>> non-free code. Since this is no longer the case, you could make it a
>> run-time option by enabling OVMF for this package, and when a user
>> would want to actually use it, they would only need to add a OVMF
>> binary to a preset path themselves (i.e. by installing the non-free
>> ovmf package).
> 
> I looked into this OVMF nonfreeness and it seems to be fixed:
>  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=815618
> 
>> Around line 74 the compile-time options for Xen are listed, and
enabling OVMF is as simple as adding one line:
>> 
>>               --enable-ovmf --with-system-ovmf=/usr/share/ovmf/OVMF.fd
>> 
>> The path specified does not need to exist at compile time. In this
>> case I chose to use the path where the OVMF package installs the
>> binary firmware file so it can be automatically used.
> 
> Thanks for this testing.
> 
> I think that, given that UEFI is becoming quite common, it might be
> worth adding a dependency of some kind to the Debian Xen packages, as
> well as simply enabling ovmf support.  But that's probably not a
> blocker for fixing this.
> 
>> I'm not sure how to create a .patch file for this, but since
it's one line with very clear results, I hope it's sufficient.
> 
> Indeed.
> 
> However: I do not intend to make this change at this stage of the
> stretch release freeze.
> 
> If you (or someone else) gets preapproval from the release team, then
> I would be prepared to do.  But I think such approval would probably
> be refused for good reasons.  If someone wants to make such an
> approval request, please let me know and I may be able to help
> pre-review it etc. - on the condition that this isn't taken as
> endorsement :-).
> 
> Thanks,
> Ian.
> 
> _______________________________________________
> Pkg-xen-devel mailing list
> Pkg-xen-devel at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-xen-devel

Hi John,

I don't think this will ever hit 4.8 in stable, but I want to have this
in the new 4.10+ packages for unstable because you've been waiting long
enough now.

Thanks for sending the Merge Request about this. Now the change will go
in under your name. :)

I don't think that a discussion in gitlab on some commit that got
rebased later is the best way, so let's continue here.

I know --enable-ovmf builds ok, now I need instructions to test this. I
suspect you're on 4.8 and unless you have a spare test system already
you might not want to be running random experimental test builds for
4.10. It also needs rebuilt qemu packages against xen 4.10.

So, what is the minimal set of things I can do to make sure the
resulting build does something sensible? I have to be honest I'm not a
qemu user myself, so I might not be totally familiar with all steps.

I see https://wiki.xenproject.org/wiki/OVMF which has an example, which
also mentions use of a live CD.

Can you point me so something I can provide as a disk, and tell me if
this is the right path to a minimal test to check that everything is
working right?

Thanks,
Hans

HI Hans,

We probably should update the mailing list anyway since not everybody will be on
GitLab anyway. Here is how you can test it:

1. Install Xen with OVMF support
2. Install OVMF (which basically just gets you the binary file, package is
called ovmf)
3. Get an EFI-bootable image, the Debian Netinstall image will probably do fine
3. Create a domU config like this:

name = 'uefi-domu-thingy'
bios = "ovmf"
builder = 'hvm'
memory = '512'
vcpus = 1

Then sudo xl create <whateveryoucalledit.cfg>

This should start the domu, and since there is no disk, just get to the UEFI
shell and idle around and not do much else. You can connect to it over VNC, but
I?m sure it can be started in UEFI text mode too so you get UEFI access via the
serial console.

If you want to test booting an OS, a simple way is starting debian. Grabbing the
netinstall image works, add a disk stanza and there you go:

disk = ['file:/tmp/debian-9.3.0-amd64-netinst.iso,xvda?]

If you want to control the virtual frame buffer VNC setup:

vfb = ["type=vnc, vnclisten=0.0.0.0, vncpasswd=1337-haxx0r"]

If you also want to have some networking fun, you could add a virtual interface
as normal:

vif = [ 'mac=00:16:3e:12:34:56, bridge=testlan0, script=vif-openvswitch,
type=vif?]

By the way, if OVMF isn?t enabled or the ovmf.fd file doesn?t exist in the
expected or configured path, it will error out.

- John
> On 28 Feb 2018, at 00:18, Hans van Kranenburg <hans at knorrie.org>
wrote:
> 
> Hi John,
> 
> I don't think this will ever hit 4.8 in stable, but I want to have this
> in the new 4.10+ packages for unstable because you've been waiting long
> enough now.
> 
> Thanks for sending the Merge Request about this. Now the change will go
> in under your name. :)
> 
> I don't think that a discussion in gitlab on some commit that got
> rebased later is the best way, so let's continue here.
> 
> I know --enable-ovmf builds ok, now I need instructions to test this. I
> suspect you're on 4.8 and unless you have a spare test system already
> you might not want to be running random experimental test builds for
> 4.10. It also needs rebuilt qemu packages against xen 4.10.
> 
> So, what is the minimal set of things I can do to make sure the
> resulting build does something sensible? I have to be honest I'm not a
> qemu user myself, so I might not be totally familiar with all steps.
> 
> I see https://wiki.xenproject.org/wiki/OVMF which has an example, which
> also mentions use of a live CD.
> 
> Can you point me so something I can provide as a disk, and tell me if
> this is the right path to a minimal test to check that everything is
> working right?
> 
> Thanks,
> Hans
> 
> _______________________________________________
> Pkg-xen-devel mailing list
> Pkg-xen-devel at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-xen-devel

On 28/02/2018 00:59, John Keates wrote:
> Here is how you can test it:
> 
> [...]
Thanks! This is really helpful. I'm going to try this tomorrow.

Hans

Hey,

On 02/28/2018 12:59 AM, John Keates wrote:
> 
> [...]
> 
> 1. Install Xen with OVMF support
> 2. Install OVMF (which basically just gets you the binary file, package is
called ovmf)
> 3. Get an EFI-bootable image, the Debian Netinstall image will probably do
fine
> 3. Create a domU config like this:
> 
> name = 'uefi-domu-thingy'
> bios = "ovmf"
> builder = 'hvm'
> memory = '512'
> vcpus = 1
> 
> Then sudo xl create <whateveryoucalledit.cfg>
> 
> This should start the domu, and since there is no disk, just get to the
UEFI shell and idle around and not do much else. You can connect to it over VNC,
but I?m sure it can be started in UEFI text mode too so you get UEFI access via
the serial console.
Ok, as promised, I tried.

For Xen 4.10, I ended up with:

name = 'uefi-domu-thingy'
bios = "ovmf"
type = 'hvm'
memory = '512'
vcpus = 1
vnc=1
vnclisten='0.0.0.0'
serial='pty'

I installed the qemu packages that Mark Pryor already rebuilt as test
(which use libxen 4.10).

When doing xen create -c on that, I get a serial console to it:

UEFI Interactive Shell v2.2
EDK II
UEFI v2.70 (EDK II, 0x00010000)
Mapping table
     BLK0: Alias(s):
          PciRoot(0x0)/Pci(0x1,0x0)/Floppy(0x0)
     BLK1: Alias(s):
          PciRoot(0x0)/Pci(0x1,0x0)/Floppy(0x1)
Press ESC in 1 seconds to skip startup.nsh or any other key to continue.
Shell>

A vnc connection also shows an ugly screen which a copy of the output.

Step 2: Use a disk

I simply added...
disk = ['file:/yolo/ovmf/debian-9.3.0-amd64-netinst.iso,xvda']

and now the serial console shows a purple blinking cursor, and the vnc
connection shows a graphical Debian GNU/Linux UEFI Installer menu.

Great success!

Hans

Excellent! Good to see you got the serial console working as well. 
There are a bunch of extra options like allowing custom nvram per-vm (normally
it?s just one read-only nvram for all instances),
but those aren?t important right now, basic EFI domu booting is pretty fair to
get in Debian first.

John
> On 1 Mar 2018, at 00:14, Hans van Kranenburg <hans at knorrie.org>
wrote:
> 
> Hey,
> 
> On 02/28/2018 12:59 AM, John Keates wrote:
>> 
>> [...]
>> 
>> 1. Install Xen with OVMF support
>> 2. Install OVMF (which basically just gets you the binary file, package
is called ovmf)
>> 3. Get an EFI-bootable image, the Debian Netinstall image will probably
do fine
>> 3. Create a domU config like this:
>> 
>> name = 'uefi-domu-thingy'
>> bios = "ovmf"
>> builder = 'hvm'
>> memory = '512'
>> vcpus = 1
>> 
>> Then sudo xl create <whateveryoucalledit.cfg>
>> 
>> This should start the domu, and since there is no disk, just get to the
UEFI shell and idle around and not do much else. You can connect to it over VNC,
but I?m sure it can be started in UEFI text mode too so you get UEFI access via
the serial console.
> 
> Ok, as promised, I tried.
> 
> For Xen 4.10, I ended up with:
> 
> name = 'uefi-domu-thingy'
> bios = "ovmf"
> type = 'hvm'
> memory = '512'
> vcpus = 1
> vnc=1
> vnclisten='0.0.0.0'
> serial='pty'
> 
> I installed the qemu packages that Mark Pryor already rebuilt as test
> (which use libxen 4.10).
> 
> When doing xen create -c on that, I get a serial console to it:
> 
> UEFI Interactive Shell v2.2
> EDK II
> UEFI v2.70 (EDK II, 0x00010000)
> Mapping table
>     BLK0: Alias(s):
>          PciRoot(0x0)/Pci(0x1,0x0)/Floppy(0x0)
>     BLK1: Alias(s):
>          PciRoot(0x0)/Pci(0x1,0x0)/Floppy(0x1)
> Press ESC in 1 seconds to skip startup.nsh or any other key to continue.
> Shell>
> 
> A vnc connection also shows an ugly screen which a copy of the output.
> 
> Step 2: Use a disk
> 
> I simply added...
> disk = ['file:/yolo/ovmf/debian-9.3.0-amd64-netinst.iso,xvda']
> 
> and now the serial console shows a purple blinking cursor, and the vnc
> connection shows a graphical Debian GNU/Linux UEFI Installer menu.
> 
> Great success!
> 
> Hans

Your message dated Thu, 11 Oct 2018 11:00:42 +0000
with message-id <E1gAYhu-000HaP-J8 at fasolo.debian.org>
and subject line Bug#858962: fixed in xen 4.11.1~pre.20180911.5acdd26fdc+dfsg-2
has caused the Debian Bug report #858962,
regarding Request: enable OVMF at build time in 4.8 as it does not require
non-free anymore
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner at bugs.debian.org
immediately.)


-- 
858962: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=858962
Debian Bug Tracking System
Contact owner at bugs.debian.org with problems
-------------- next part --------------
An embedded message was scrubbed...
From: John Keates <john at keates.nl>
Subject: Request: enable OVMF at build time in 4.8 as it does not require
non-free anymore
Date: Wed, 29 Mar 2017 02:26:34 +0200
Size: 3488
URL:
<http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20181011/7ba38ed6/attachment-0002.mht>
-------------- next part --------------
An embedded message was scrubbed...
From: Ian Jackson <ijackson at chiark.greenend.org.uk>
Subject: Bug#858962: fixed in xen 4.11.1~pre.20180911.5acdd26fdc+dfsg-2
Date: Thu, 11 Oct 2018 11:00:42 +0000
Size: 25594
URL:
<http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20181011/7ba38ed6/attachment-0003.mht>