libvirt users - Oct 2020 - Re: Encrypting boot partition Libvirt not showing the OS booting up

On 10/12/20 2:14 PM, john doe wrote:
>> <snip/>
> 
> I sent privately the requested xml file to 'Peter Krempa
> <pkrempa@redhat.com>'.
> Peter Krempa 's privately answered me back suggesting to add the
> following in the domain xml file:
Solving things privately doesn't help the community.
> 
> 
> <bios useserial='yes'/> under <os>
> 
> such as ...
> 
>     <os>
>       <type arch='x86_64'
machine='pc-q35-3.1'>hvm</type>
>       <boot dev='hd'/>
>       <bios useserial='yes'/>
>     </os>
> 
Try adding:

     <loader type='rom'>/usr/share/seabios/bios.bin</loader>

to <os/> so that qemu actually uses SGA bios.

Michal

On Mon, Oct 12, 2020 at 16:05:43 +0200, Michal Privoznik
wrote:
> On 10/12/20 2:14 PM, john doe wrote:
> > > <snip/>
> > 
> > I sent privately the requested xml file to 'Peter Krempa
> > <pkrempa@redhat.com>'.
> > Peter Krempa 's privately answered me back suggesting to add the
> > following in the domain xml file:
> 
> Solving things privately doesn't help the community.
Additionally it doesn't help solving the problem, since it's now opaque
to others what the problem might be.
> > 
> > <bios useserial='yes'/> under <os>
I've suggested this as the outputs I've got privately hinted that the
console (as in virsh console) didn't get to asking for the password,
while the manually-started-qemu did.

Thus the problem actually doesn't have to do with encryption or
wahatver, but the console doesn't plainly work.
> > 
> > such as ...
> > 
> >     <os>
> >       <type arch='x86_64'
machine='pc-q35-3.1'>hvm</type>
> >       <boot dev='hd'/>
> >       <bios useserial='yes'/>
> >     </os>
> > 
> 
> Try adding:
> 
>     <loader
type='rom'>/usr/share/seabios/bios.bin</loader>
> 
> to <os/> so that qemu actually uses SGA bios.
> 
> Michal

On 10/12/20 4:27 PM, john doe wrote:
> On 10/12/2020 4:09 PM, Peter Krempa wrote:
>> On Mon, Oct 12, 2020 at 16:05:43 +0200, Michal Privoznik wrote:
>>> On 10/12/20 2:14 PM, john doe wrote:
>>>>> <snip/>
>>>>
>>>> I sent privately the requested xml file to 'Peter Krempa
>>>> <pkrempa@redhat.com>'.
>>>> Peter Krempa 's privately answered me back suggesting to
add the
>>>> following in the domain xml file:
>>>
>>> Solving things privately doesn't help the community.
>>
>> Additionally it doesn't help solving the problem, since it's
now opaque
>> to others what the problem might be.
>>
>>>>
>>>> <bios useserial='yes'/> under <os>
>>
>> I've suggested this as the outputs I've got privately hinted
that the
>> console (as in virsh console) didn't get to asking for the
password,
>> while the manually-started-qemu did.
>>
>> Thus the problem actually doesn't have to do with encryption or
>> wahatver, but the console doesn't plainly work.
>>
>>>>
>>>> such as ...
>>>>
>>>>      <os>
>>>>        <type arch='x86_64'
machine='pc-q35-3.1'>hvm</type>
>>>>        <boot dev='hd'/>
>>>>        <bios useserial='yes'/>
>>>>      </os>
>>>>
>>>
>>> Try adding:
>>>
>>>      <loader
type='rom'>/usr/share/seabios/bios.bin</loader>
Darn, this should have been sgabios: /usr/share/sgabios/sgabios.bin
but if your seabios is new enough (v1.11.0 and newer) then this is not 
needed as seabios itself is capable of serial interface. And looking at 
earlier e-mails in the thread you have v1.12.0-1 you you're good and 
don't need to add <loader/> at all.

But honestly, I don't know why you are not getting the console. Could it 
be that you are getting the console and the qemu is waiting for your 
input, i.e. what happens if you type in the password?

Michal

On 10/12/20 1:10 PM, john doe wrote:
> On 10/12/2020 5:14 PM, Michal Privoznik wrote:
>> On 10/12/20 4:27 PM, john doe wrote:
>>> On 10/12/2020 4:09 PM, Peter Krempa wrote:
>>>> On Mon, Oct 12, 2020 at 16:05:43 +0200, Michal Privoznik wrote:
>>>>> On 10/12/20 2:14 PM, john doe wrote:
>>>>>>> <snip/>
>>>>>>
>>>>>> I sent privately the requested xml file to 'Peter
Krempa
>>>>>> <pkrempa@redhat.com>'.
>>>>>> Peter Krempa 's privately answered me back
suggesting to add the
>>>>>> following in the domain xml file:
>>>>>
>>>>> Solving things privately doesn't help the community.
>>>>
>>>> Additionally it doesn't help solving the problem, since
it's now opaque
>>>> to others what the problem might be.
>>>>
>>>>>>
>>>>>> <bios useserial='yes'/> under <os>
>>>>
>>>> I've suggested this as the outputs I've got privately
hinted that the
>>>> console (as in virsh console) didn't get to asking for the
password,
>>>> while the manually-started-qemu did.
>>>>
>>>> Thus the problem actually doesn't have to do with
encryption or
>>>> wahatver, but the console doesn't plainly work.
>>>>
>>>>>>
>>>>>> such as ...
>>>>>>
>>>>>>      <os>
>>>>>>        <type arch='x86_64'
machine='pc-q35-3.1'>hvm</type>
>>>>>>        <boot dev='hd'/>
>>>>>>        <bios useserial='yes'/>
>>>>>>      </os>
>>>>>>
>>>>>
>>>>> Try adding:
>>>>>
>>>>>      <loader
type='rom'>/usr/share/seabios/bios.bin</loader>
>>
>> Darn, this should have been sgabios: /usr/share/sgabios/sgabios.bin
>> but if your seabios is new enough (v1.11.0 and newer) then this is not
>> needed as seabios itself is capable of serial interface. And looking at
>> earlier e-mails in the thread you have v1.12.0-1 you you're good
and
>> don't need to add <loader/> at all.
>>
>> But honestly, I don't know why you are not getting the console.
Could it
>> be that you are getting the console and the qemu is waiting for your
>> input, i.e. what happens if you type in the password?
>>
> 
> Nothing happened at all if I try to type the password.
> Yes, so am I , I'm totaly lost on why it does not work.
> 
> How can I find the command libvirt is passing to qemu?
The qemu command issued by libvirt can be found at the end of 
/etc/libvirt/qemu/${guestname}.log
> 
> Are you at least able to reproduce the issue (Debian Buster for host and
> guest)?
> 
> -- 
> John Doe
> 
>

On Tue, Oct 13, 2020 at 10:40:11AM +0200, john doe
wrote:
> On 10/13/2020 8:50 AM, john doe wrote:
> > On 10/13/2020 3:30 AM, Laine Stump wrote:
> > > On 10/12/20 1:10 PM, john doe wrote:
> > > > On 10/12/2020 5:14 PM, Michal Privoznik wrote:
> > > > > On 10/12/20 4:27 PM, john doe wrote:
> > > > > > On 10/12/2020 4:09 PM, Peter Krempa wrote:
> > > > > > > On Mon, Oct 12, 2020 at 16:05:43 +0200,
Michal Privoznik wrote:
> > > > > > > > On 10/12/20 2:14 PM, john doe wrote:
> > > > > > > > > > <snip/>
> > > > > > > > > 
> > > > > > > > > I sent privately the requested xml
file to 'Peter Krempa
> > > > > > > > > <pkrempa@redhat.com>'.
> > > > > > > > > Peter Krempa 's privately
answered me back suggesting to add the
> > > > > > > > > following in the domain xml file:
> > > > > > > > 
> > > > > > > > Solving things privately doesn't
help the community.
> > > > > > > 
> > > > > > > Additionally it doesn't help solving the
problem, since it's now
> > > > > > > opaque
> > > > > > > to others what the problem might be.
> > > > > > > 
> > > > > > > > > 
> > > > > > > > > <bios
useserial='yes'/> under <os>
> > > > > > > 
> > > > > > > I've suggested this as the outputs
I've got privately hinted that the
> > > > > > > console (as in virsh console) didn't get
to asking for the password,
> > > > > > > while the manually-started-qemu did.
> > > > > > > 
> > > > > > > Thus the problem actually doesn't have to
do with encryption or
> > > > > > > wahatver, but the console doesn't plainly
work.
> > > > > > > 
> > > > > > > > > 
> > > > > > > > > such as ...
> > > > > > > > > 
> > > > > > > > >      <os>
> > > > > > > > >        <type
arch='x86_64' machine='pc-q35-3.1'>hvm</type>
> > > > > > > > >        <boot
dev='hd'/>
> > > > > > > > >        <bios
useserial='yes'/>
> > > > > > > > >      </os>
> > > > > > > > > 
> > > > > > > > 
> > > > > > > > Try adding:
> > > > > > > > 
> > > > > > > >      <loader
type='rom'>/usr/share/seabios/bios.bin</loader>
> > > > > 
> > > > > Darn, this should have been sgabios:
/usr/share/sgabios/sgabios.bin
> > > > > but if your seabios is new enough (v1.11.0 and newer)
then this is not
> > > > > needed as seabios itself is capable of serial
interface. And looking at
> > > > > earlier e-mails in the thread you have v1.12.0-1 you
you're good and
> > > > > don't need to add <loader/> at all.
> > > > > 
> > > > > But honestly, I don't know why you are not getting
the console.
> > > > > Could it
> > > > > be that you are getting the console and the qemu is
waiting for your
> > > > > input, i.e. what happens if you type in the password?
> > > > > 
> > > > 
> > > > Nothing happened at all if I try to type the password.
> > > > Yes, so am I , I'm totaly lost on why it does not work.
> > > > 
> > > > How can I find the command libvirt is passing to qemu?
> > > 
> > > The qemu command issued by libvirt can be found at the end of
> > > /etc/libvirt/qemu/${guestname}.log
> > > 
> > 
> > Thank you, I have now isolated the command generated by libvirt.
> > Starting this command from a script, a vnc server is started.
> > 
> > Is libvirt internally using vnc connection?
> > 
> 
> It looks like the issue is that the libvirt command pass to qemu is
> using '-display none' where it should be '-nographic'.
No, using "-display none" is correct to disable any graphical display
in QEMU.

-nographic is an old discouraged syntax which has many side effects,
beyond disabling graphical display it mess with serial port config.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On Tue, Oct 13, 2020 at 10:40:11 +0200, john doe wrote:
> On 10/13/2020 8:50 AM, john doe wrote:
[...]
> > Thank you, I have now isolated the command generated by libvirt.
> > Starting this command from a script, a vnc server is started.
> > 
> > Is libvirt internally using vnc connection?
> > 
> 
> It looks like the issue is that the libvirt command pass to qemu is
> using '-display none' where it should be '-nographic'.
That is deliberate:

commit 7a97676b968bcdf60fb987afe6308bc0b4ab9cd3
Author: Paolo Bonzini <pbonzini@redhat.com>
Date:   Tue Jul 5 14:35:28 2016 +0200

    qemu: generate -display none

    This is preferrable to -nographic which (in addition to disabling
    graphics output) redirects the serial port to stdio and on OpenBIOS
    enables the firmware's serial console.


We don't want to use stdio serial port at all, but rather explicitly
defined ones exposed internally such as:

    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>


I've verified that once I add the suggested useserial attribute:

  <os>
    <type arch='x86_64'
machine='pc-i440fx-2.9'>hvm</type>
    <bootmenu enable='yes'/>
    <bios useserial='yes'/>
  </os>

to one of my VMs I can see the console when started via virsh start
--console:

$ virsh start --console upstream
Domain upstream started
Connected to domain upstream
Escape character is ^] (Ctrl + ])


iPXE (http://ipxe.org) 00:03.0 C100 PCI2.10 PnP PMM+3E7835A0+3E6E35A0 C100





iPXE (http://ipxe.org) 00:08.0 C200 PCI2.10 PnP PMM+3E6D25A0 3E6E35A0 C200




Press ESC for boot menu.

On 10/13/2020 10:58 AM, Peter Krempa wrote:
> On Tue, Oct 13, 2020 at 10:40:11 +0200, john doe wrote:
>> On 10/13/2020 8:50 AM, john doe wrote:
>
> [...]
>
>>> Thank you, I have now isolated the command generated by libvirt.
>>> Starting this command from a script, a vnc server is started.
>>>
>>> Is libvirt internally using vnc connection?
>>>
>>
>> It looks like the issue is that the libvirt command pass to qemu is
>> using '-display none' where it should be '-nographic'.
>
> That is deliberate:
>
> commit 7a97676b968bcdf60fb987afe6308bc0b4ab9cd3
> Author: Paolo Bonzini <pbonzini@redhat.com>
> Date:   Tue Jul 5 14:35:28 2016 +0200
>
>      qemu: generate -display none
>
>      This is preferrable to -nographic which (in addition to disabling
>      graphics output) redirects the serial port to stdio and on OpenBIOS
>      enables the firmware's serial console.
>
>
> We don't want to use stdio serial port at all, but rather explicitly
> defined ones exposed internally such as:
>
>      <serial type='pty'>
>        <target type='isa-serial' port='0'>
>          <model name='isa-serial'/>
>        </target>
>      </serial>
>      <console type='pty'>
>        <target type='serial' port='0'/>
>      </console>
>
>
> I've verified that once I add the suggested useserial attribute:
>
>    <os>
>      <type arch='x86_64'
machine='pc-i440fx-2.9'>hvm</type>
>      <bootmenu enable='yes'/>
>      <bios useserial='yes'/>
>    </os>
>
> to one of my VMs I can see the console when started via virsh start
> --console:
>
> $ virsh start --console upstream
> Domain upstream started
> Connected to domain upstream
> Escape character is ^] (Ctrl + ])
>
>
> iPXE (http://ipxe.org) 00:03.0 C100 PCI2.10 PnP PMM+3E7835A0+3E6E35A0 C100
>
>
>
>
>
> iPXE (http://ipxe.org) 00:08.0 C200 PCI2.10 PnP PMM+3E6D25A0 3E6E35A0 C200
>
>
>
>
> Press ESC for boot menu.
>
>
To summerise:

After having installed the vm with virt-install, if you encrypt the boot
partition, you may need to add the below two lines in the os section of
the domain xml:

     <bootmenu enable='yes'/>
     <bios useserial='yes'/>

Then start the guest with the below command:

virsh start $vm --console


I was able to find the log file in the directory
'/var/log/libvirt/qemu'.


A big thank you for all the help:
- Peter Krempa
- Daniel P. Berrangé
- Laine Stump
- Michal Privoznik


Much appriciated.

--
John Doe