libvirt users - Mar 2016 - uefi built from tiancore via edk2 can't persist boot changes

Apologies if this has been gone over, but I believe I have checked the
intertubes more than a bit.....

I am using libvirt and have vms booting under an OVMF.fd to use an efi
firmware. I can create vms, linux ubuntu, and they will boot up. However,
everytime I reboot am I dropped into the default efi shell provide by the
tianocore build.
Then I must walk the FS to the booting efi app and run, in this case
grubx64.efi, to actually finish booting the host.
I had tried adding boot entries with efibootmgr within the OS and also bcfg
with the efi shell. I get no errors when adding an entry, and the new entry
shows up and I can manipulate the entry i.e. set it to next boot and the
like. Soon as I reboot I get dumped back to efi shell.

Am I missing something? Are the var changes not being stored? Is there
someplace to look for an error perhaps?

I also can use the same efi bootloader under libvirt to boot a hybrid iso.
location of the efi application on the hybrid iso is /EFI/boot/bootx64.efi.
Drops me right into my grub menu, after falling thru from EFI FLOPPY1 and
EFI FLOPPY2 to EFI DVD
So since that was/is working,  tried adding that to my efi partition so it
has /EFI/boot/bootx64.efi. The thinking being that since I can't add an
entry, I will setup for the "known" efi boot path. Of course that
didn't
work either

I am on arch linux:

Linux X 4.4.5-1-ARCH #1 SMP PREEMPT Thu Mar 10 07:38:19 CET 2016 x86_64
GNU/Linux
local/libvirt 1.3.2-3
    API for controlling virtualization engines
(openvz,kvm,qemu,virtualbox,xen,etc)
local/libvirt-python 1.3.1-1
    libvirt python binding

UEFI:
Shell> ver
UEFI Interactive Shell v2.1
EDK II
UEFI v2.50 (EDK II, 0x00010000)


for this host....my nvram setting look like
  <os>
    <type arch='x86_64'
machine='pc-i440fx-2.5'>hvm</type>
    <loader readonly='yes'
type='pflash'>/home/xyz/OVMF.fd</loader>
    <nvram
template='/usr/share/OVMF/OVMF_VARS.fd'>/var/lib/libvirt/qemu/nvram/X_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>

On Mon, 2016-03-21 at 13:53 -0500, jsl6uy js16uy wrote:
> for this host....my nvram setting look like
>   <os>
>     <type arch='x86_64'
machine='pc-i440fx-2.5'>hvm</type>
>     <loader readonly='yes'
type='pflash'>/home/xyz/OVMF.fd</loader>
>     <nvram
template='/usr/share/OVMF/OVMF_VARS.fd'>/var/lib/libvirt/qemu/nvram/X_VARS.fd</nvram>
>     <boot dev='hd'/>
>   </os>
Have you tried deleting the *VARS.fd in /var/lib/libvirt/qemu/nvram
and changing the 'template' attribute so that it points to the
*VARS.fd file you should have in your edk2 source tree, eg. the one
matching the OVMF.fd you're using to boot?

Not sure if that would help, but it's definitely worth a shot :)

Cheers.

-- 
Andrea Bolognani
Software Engineer - Virtualization Team

Thanks very much for the help!
Unfortunately I did try that previously and was still dumped into the efi
shell.

Looking round again, I didn't build my OVMF with secure boot enabled. Going
to see if I get any joy rebuilding with secure boot support.

thanks again

On Tue, Mar 22, 2016 at 3:35 AM, Andrea Bolognani <abologna@redhat.com>
wrote:
> On Mon, 2016-03-21 at 13:53 -0500, jsl6uy js16uy wrote:
> > for this host....my nvram setting look like
> >   <os>
> >     <type arch='x86_64'
machine='pc-i440fx-2.5'>hvm</type>
> >     <loader readonly='yes'
type='pflash'>/home/xyz/OVMF.fd</loader>
> >     <nvram
>
template='/usr/share/OVMF/OVMF_VARS.fd'>/var/lib/libvirt/qemu/nvram/X_VARS.fd</nvram>
> >     <boot dev='hd'/>
> >   </os>
>
> Have you tried deleting the *VARS.fd in /var/lib/libvirt/qemu/nvram
> and changing the 'template' attribute so that it points to the
> *VARS.fd file you should have in your edk2 source tree, eg. the one
> matching the OVMF.fd you're using to boot?
>
> Not sure if that would help, but it's definitely worth a shot :)
>
> Cheers.
>
> --
> Andrea Bolognani
> Software Engineer - Virtualization Team
>
>

On 03/21/16 19:53, jsl6uy js16uy wrote:
> Apologies if this has been gone over, but I believe I have checked the
> intertubes more than a bit.....
> 
> I am using libvirt and have vms booting under an OVMF.fd to use an efi
> firmware. I can create vms, linux ubuntu, and they will boot up.
> However, everytime I reboot am I dropped into the default efi shell
> provide by the tianocore build.
> Then I must walk the FS to the booting efi app and run, in this case
> grubx64.efi, to actually finish booting the host.
> I had tried adding boot entries with efibootmgr within the OS and also
> bcfg with the efi shell. I get no errors when adding an entry, and the
> new entry shows up and I can manipulate the entry i.e. set it to next
> boot and the like. Soon as I reboot I get dumped back to efi shell. 
> 
> Am I missing something? Are the var changes not being stored? Is there
> someplace to look for an error perhaps?
> 
> I also can use the same efi bootloader under libvirt to boot a hybrid
> iso. location of the efi application on the hybrid iso is
> /EFI/boot/bootx64.efi. Drops me right into my grub menu, after falling
> thru from EFI FLOPPY1 and EFI FLOPPY2 to EFI DVD
> So since that was/is working,  tried adding that to my efi partition so
> it has /EFI/boot/bootx64.efi. The thinking being that since I can't add
> an entry, I will setup for the "known" efi boot path. Of course
that
> didn't work either
> 
> I am on arch linux:
> 
> Linux X 4.4.5-1-ARCH #1 SMP PREEMPT Thu Mar 10 07:38:19 CET 2016 x86_64
> GNU/Linux
> local/libvirt 1.3.2-3
>     API for controlling virtualization engines
> (openvz,kvm,qemu,virtualbox,xen,etc)
> local/libvirt-python 1.3.1-1
>     libvirt python binding
> 
> UEFI:
> Shell> ver
> UEFI Interactive Shell v2.1
> EDK II
> UEFI v2.50 (EDK II, 0x00010000)
> 
> 
> for this host....my nvram setting look like
>   <os>
>     <type arch='x86_64'
machine='pc-i440fx-2.5'>hvm</type>
>     <loader readonly='yes'
type='pflash'>/home/xyz/OVMF.fd</loader>
>     <nvram
>
template='/usr/share/OVMF/OVMF_VARS.fd'>/var/lib/libvirt/qemu/nvram/X_VARS.fd</nvram>
>     <boot dev='hd'/>
>   </os>
Your domain XML is wrong. You can use OVMF with libvirt in three setups.

(1) worst setup: map the unified OVMF.fd file (which includes variable
store and fimrware executable) as ROM. I don't want people to use this,
ever, so although libvirt does support it, I'm not going to share the
XML snippet that implements this.

(2) a bit less wrong setup (but still pretty wrong): map the unified
OVMF.fd file as writeable pflash. This is bad because (a) you can't
share the unified OVMF.fd file between guests, and (b) whenever you
rebuild the OVMF.fd file, the guest that is pointed to it will lose all
of its persistent variables.

(3) the right setup: you almost got it right, but the <loader> element
in this setup must not point to the unified OVMF.fd file. It must point
to the executable-only part, the OVMF_CODE.fd file. So here's what you need:

  <os>
    <loader
     readonly='yes'
     type='pflash'
    >/your/build/output/OVMF_CODE.fd</loader>
    <nvram
     template='/your/build/output/OVMF_VARS.fd'
    >/var/lib/libvirt/qemu/nvram/guest_name_VARS.fd</nvram>
  </os>

This allows the firmware binary to be upgraded centrally, while every VM
can keep its private varstore.

The virt-install(1) manual describes this, under the example

--boot
loader=/.../OVMF_CODE.fd,loader_ro=yes,loader_type=pflash,nvram_template=/.../OVMF_VARS.fd

The way you specified it implies that the varstore range will be mapped
as read-only pflash directly from the unified OVMF.fd file, which
happens to make the varstore non-writeable. In addition, the real
varstore file will be mapped below *that*, which is completely bogus.

(4) Instead of <boot> elements in the <os> element, <boot
order='...'/>
elements are preferred under <disk> and friends.

(5) You can (and are encouraged to) look at the OVMF log. If you want to
capture it, you should implement the following changes:

- add the following attribute to the root element of the domain XML:

  <domain
   type='kvm'
   xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'  <-- this
one
  >

- add the following XML fragment near the end of the domain XML, as a
direct child of the <domain> element:

    <qemu:arg value='-global'/>
    <qemu:arg value='isa-debugcon.iobase=0x402'/>
    <qemu:arg value='-debugcon'/>
    <qemu:arg value='file:/tmp/guest_name.log'/>

This will send the OVMF log to /tmp/guest_name.log.

(6) If you are already missing a boot option to boot your installed OS,
you might have to reestablish that boot option manually (after the above
changes) -- add the option, and place it first in the boot order. You
can use efibootmgr for this, or the BCFG command, or simply the firmware
TUI.

I could go into why, but this time I won't. If you are interested, you
can mostly find the reason in the OVMF whitepaper
<http://people.redhat.com/~lersek/ovmf-whitepaper-c770f8c.txt>, section
"Platform-specific boot policy", bullet (b), "Auto-generation of
boot
options". The point is, boot options pointing to arbitrary boot loader
UEFI applications on non-removable media cannot be autogenerated, hence
OVMF cannot filter them in. If you install the guest OS with the correct
domain XML in the first place, then things will work.

Some UEFI OSes also install "system rescue" utilities to the target
non-removable media, under the default removable media pathname. The
idea is that if you lose your "main" boot option (pointing to an
arbitrary OS boot loader), then the firmware can still auto-generate a
boot option for this fallback utility, and the fallback utility can
restore the OS-specific boot loader's boot option. You can read more
about this at
<https://blog.uncooperative.org/blog/2014/02/06/the-efi-system-partition/>.

Summary:
- use a domain XML like described in (3) and (4)
- when in doubt, capture the OVMF debug log as described in (5)

Thanks
Laszlo

Thanks VERY MUCH for all the info and help!
Apologies for the extreme delay. I got distracted by other threads that
forced out this work to later date. Also some frustration as well. ;)
I completely missed this update. Apologies and thanks
Will be diving back into this shortly!

On Thu, Mar 24, 2016 at 3:57 PM, Laszlo Ersek <lersek@redhat.com> wrote:
> On 03/21/16 19:53, jsl6uy js16uy wrote:
> > Apologies if this has been gone over, but I believe I have checked the
> > intertubes more than a bit.....
> >
> > I am using libvirt and have vms booting under an OVMF.fd to use an efi
> > firmware. I can create vms, linux ubuntu, and they will boot up.
> > However, everytime I reboot am I dropped into the default efi shell
> > provide by the tianocore build.
> > Then I must walk the FS to the booting efi app and run, in this case
> > grubx64.efi, to actually finish booting the host.
> > I had tried adding boot entries with efibootmgr within the OS and also
> > bcfg with the efi shell. I get no errors when adding an entry, and the
> > new entry shows up and I can manipulate the entry i.e. set it to next
> > boot and the like. Soon as I reboot I get dumped back to efi shell.
> >
> > Am I missing something? Are the var changes not being stored? Is there
> > someplace to look for an error perhaps?
> >
> > I also can use the same efi bootloader under libvirt to boot a hybrid
> > iso. location of the efi application on the hybrid iso is
> > /EFI/boot/bootx64.efi. Drops me right into my grub menu, after falling
> > thru from EFI FLOPPY1 and EFI FLOPPY2 to EFI DVD
> > So since that was/is working,  tried adding that to my efi partition
so
> > it has /EFI/boot/bootx64.efi. The thinking being that since I
can't add
> > an entry, I will setup for the "known" efi boot path. Of
course that
> > didn't work either
> >
> > I am on arch linux:
> >
> > Linux X 4.4.5-1-ARCH #1 SMP PREEMPT Thu Mar 10 07:38:19 CET 2016
x86_64
> > GNU/Linux
> > local/libvirt 1.3.2-3
> >     API for controlling virtualization engines
> > (openvz,kvm,qemu,virtualbox,xen,etc)
> > local/libvirt-python 1.3.1-1
> >     libvirt python binding
> >
> > UEFI:
> > Shell> ver
> > UEFI Interactive Shell v2.1
> > EDK II
> > UEFI v2.50 (EDK II, 0x00010000)
> >
> >
> > for this host....my nvram setting look like
> >   <os>
> >     <type arch='x86_64'
machine='pc-i440fx-2.5'>hvm</type>
> >     <loader readonly='yes'
type='pflash'>/home/xyz/OVMF.fd</loader>
> >     <nvram
> >
>
template='/usr/share/OVMF/OVMF_VARS.fd'>/var/lib/libvirt/qemu/nvram/X_VARS.fd</nvram>
> >     <boot dev='hd'/>
> >   </os>
>
> Your domain XML is wrong. You can use OVMF with libvirt in three setups.
>
> (1) worst setup: map the unified OVMF.fd file (which includes variable
> store and fimrware executable) as ROM. I don't want people to use this,
> ever, so although libvirt does support it, I'm not going to share the
> XML snippet that implements this.
>
> (2) a bit less wrong setup (but still pretty wrong): map the unified
> OVMF.fd file as writeable pflash. This is bad because (a) you can't
> share the unified OVMF.fd file between guests, and (b) whenever you
> rebuild the OVMF.fd file, the guest that is pointed to it will lose all
> of its persistent variables.
>
> (3) the right setup: you almost got it right, but the <loader>
element
> in this setup must not point to the unified OVMF.fd file. It must point
> to the executable-only part, the OVMF_CODE.fd file. So here's what you
> need:
>
>   <os>
>     <loader
>      readonly='yes'
>      type='pflash'
>     >/your/build/output/OVMF_CODE.fd</loader>
>     <nvram
>      template='/your/build/output/OVMF_VARS.fd'
>     >/var/lib/libvirt/qemu/nvram/guest_name_VARS.fd</nvram>
>   </os>
>
> This allows the firmware binary to be upgraded centrally, while every VM
> can keep its private varstore.
>
> The virt-install(1) manual describes this, under the example
>
> --boot
>
>
loader=/.../OVMF_CODE.fd,loader_ro=yes,loader_type=pflash,nvram_template=/.../OVMF_VARS.fd
>
> The way you specified it implies that the varstore range will be mapped
> as read-only pflash directly from the unified OVMF.fd file, which
> happens to make the varstore non-writeable. In addition, the real
> varstore file will be mapped below *that*, which is completely bogus.
>
> (4) Instead of <boot> elements in the <os> element, <boot
order='...'/>
> elements are preferred under <disk> and friends.
>
> (5) You can (and are encouraged to) look at the OVMF log. If you want to
> capture it, you should implement the following changes:
>
> - add the following attribute to the root element of the domain XML:
>
>   <domain
>    type='kvm'
>    xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'  <--
this one
>   >
>
> - add the following XML fragment near the end of the domain XML, as a
> direct child of the <domain> element:
>
>     <qemu:arg value='-global'/>
>     <qemu:arg value='isa-debugcon.iobase=0x402'/>
>     <qemu:arg value='-debugcon'/>
>     <qemu:arg value='file:/tmp/guest_name.log'/>
>
> This will send the OVMF log to /tmp/guest_name.log.
>
> (6) If you are already missing a boot option to boot your installed OS,
> you might have to reestablish that boot option manually (after the above
> changes) -- add the option, and place it first in the boot order. You
> can use efibootmgr for this, or the BCFG command, or simply the firmware
> TUI.
>
> I could go into why, but this time I won't. If you are interested, you
> can mostly find the reason in the OVMF whitepaper
> <http://people.redhat.com/~lersek/ovmf-whitepaper-c770f8c.txt>,
section
> "Platform-specific boot policy", bullet (b),
"Auto-generation of boot
> options". The point is, boot options pointing to arbitrary boot loader
> UEFI applications on non-removable media cannot be autogenerated, hence
> OVMF cannot filter them in. If you install the guest OS with the correct
> domain XML in the first place, then things will work.
>
> Some UEFI OSes also install "system rescue" utilities to the
target
> non-removable media, under the default removable media pathname. The
> idea is that if you lose your "main" boot option (pointing to an
> arbitrary OS boot loader), then the firmware can still auto-generate a
> boot option for this fallback utility, and the fallback utility can
> restore the OS-specific boot loader's boot option. You can read more
> about this at
>
<https://blog.uncooperative.org/blog/2014/02/06/the-efi-system-partition/
> >.
>
> Summary:
> - use a domain XML like described in (3) and (4)
> - when in doubt, capture the OVMF debug log as described in (5)
>
> Thanks
> Laszlo
>
>