libvirt users - Dec 2019 - Re: Change in treatment of qcow2 with chained backing files in v5.10.0?

Hello list,

upon upgrading from libvirt v5.9.0 -> v5.10.0, libvirt started calling
qemu differently causing the VM operating system to not be found. I'm
wondering whether I'm chasing a bug, or have to otherwise fix my domain
config (created by virt-manager).

To start with, I have a Windows 7 domain based on the following images:

  windows7-base.qcow2:       QEMU QCOW2 Image (v3), 53687091200 bytes
  windows7-Office2013.qcow2: QEMU QCOW2 Image (v3), has backing file (path
windows7-taintable.qcow2), 80530636800 bytes
  windows7-taintable.qcow2:  QEMU QCOW2 Image (v3), has backing file (path
windows7-base.qcow2), 53687091200 bytes

The VM must be launched with image windows7-Office2013.qcow2.

Using the domain config [1] with a <disk> section

  <disk type='file' device='disk'>
    <driver name='qemu' type='qcow2'/>
    <source
file='/home/joj/lib/vm/windows/windows7-Office2013.qcow2'/>
    <target dev='vda' bus='virtio'/>
    <address type='pci' domain='0x0000' bus='0x00'
slot='0x07' function='0x0'/>
  </disk>

libvirt 5.9.0 is calling qemu 4.2.0 like [2], and libvirt 5.10.0 like [3], the
differences being (5.9.0 1st, 5.10.0 2nd lines)

 [--drive
file=/home/joj/lib/vm/windows/windows7-Office2013.qcow2,format=qcow2,if=none,id=drive-virtio-disk0-]
 {+-blockdev
{"driver":"file","filename":"/home/joj/lib/vm/windows/windows7-taintable.qcow2","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}
-blockdev
{"node-name":"libvirt-2-format","read-only":true,"driver":"raw","file":"libvirt-2-storage"}
-blockdev
{"driver":"file","filename":"/home/joj/lib/vm/windows/windows7-Office2013.qcow2","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}
-blockdev
{"node-name":"libvirt-1-format","read-only":false,"driver":"qcow2","file":"libvirt-1-storage","backing":"libvirt-2-format"}+}

 -device
[-virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1-]
        
{+virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=libvirt-1-format,id=virtio-disk0,bootindex=1+}

As you can see, 5.10.0 is expanding the backing file chain, not leaving
that to qemu. Under 5.10.0, I get an Operating System not Found error,
under 5.9.0, with the shorter -drive argument, everything works as
desired.

Can you perhaps advise me how I either fix the domain spec [1] to get
libvirt to use a true equivalent to the old qemu invocation as I don't
think this is a qemu problem, or how I can turn the new-style JSON
parametrization off?

Other suggestions how to tackle this would also be much appreciated.

---
[1] https://0x0.st/z0KM.xml
[2] https://0x0.st/z0K1.txt
[3] https://0x0.st/z0Kj.txt

On Mon, Dec 16, 2019 at 12:49:40 +0100, Jens John wrote:
> Hello list,
Hi Jens,
> 
> upon upgrading from libvirt v5.9.0 -> v5.10.0, libvirt started calling
> qemu differently causing the VM operating system to not be found. I'm
> wondering whether I'm chasing a bug, or have to otherwise fix my domain
> config (created by virt-manager).
Yes, this change was done by me when enabling the new way to configure
block devices in qemu.

I think I know what's going on:
> 
> To start with, I have a Windows 7 domain based on the following images:
> 
>   windows7-base.qcow2:       QEMU QCOW2 Image (v3), 53687091200 bytes
>   windows7-Office2013.qcow2: QEMU QCOW2 Image (v3), has backing file (path
windows7-taintable.qcow2), 80530636800 bytes
>   windows7-taintable.qcow2:  QEMU QCOW2 Image (v3), has backing file (path
windows7-base.qcow2), 53687091200 bytes
So this output was obtained by 'file'. That unfortunately doesn't
give
enough information. Please run 'qemu-img info' on
those files and look for the 'backing file format:'field.

That field is important, because otherwise we'd fall back to image
format detection which was deemed unsafe and libvirt no longer allows
it.


$ qemu-img info pull4.qcow2
image: pull4.qcow2
file format: qcow2
virtual size: 10 MiB (10485760 bytes)
disk size: 196 KiB
cluster_size: 65536
backing file: pull3.qcow2
backing file format: qcow2

^^^^^^^^^^^^^^^^^^^^^^^^^^

Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false


The root of the problem is that the images mentioned above will not have
the format section because -F argument for qemu-img was not used when
creating them. This would force libvirt to use format detection which is
insecure and thus libvirt does not do that and assumes the image is raw.

This in normal circumstances causes problems right away, because when
running any more advanced security model than unix permissions (selinux
apparmor) libvirt would not allow qemu to access the backing file. You
don't seem to be running it so it was accidentally working for you until
now.
> The VM must be launched with image windows7-Office2013.qcow2.
> 
> Using the domain config [1] with a <disk> section
> 
>   <disk type='file' device='disk'>
>     <driver name='qemu' type='qcow2'/>
>     <source
file='/home/joj/lib/vm/windows/windows7-Office2013.qcow2'/>
>     <target dev='vda' bus='virtio'/>
>     <address type='pci' domain='0x0000'
bus='0x00' slot='0x07' function='0x0'/>
>   </disk>
> 
> libvirt 5.9.0 is calling qemu 4.2.0 like [2], and libvirt 5.10.0 like [3],
the
> differences being (5.9.0 1st, 5.10.0 2nd lines)
> 
>  [--drive
file=/home/joj/lib/vm/windows/windows7-Office2013.qcow2,format=qcow2,if=none,id=drive-virtio-disk0-]
>  {+-blockdev
{"driver":"file","filename":"/home/joj/lib/vm/windows/windows7-taintable.qcow2","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}
-blockdev
{"node-name":"libvirt-2-format","read-only":true,"driver":"raw","file":"libvirt-2-storage"}
-blockdev
{"driver":"file","filename":"/home/joj/lib/vm/windows/windows7-Office2013.qcow2","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}
-blockdev
{"node-name":"libvirt-1-format","read-only":false,"driver":"qcow2","file":"libvirt-1-storage","backing":"libvirt-2-format"}+}
 -blockdev
{"driver":"file","filename":"/home/joj/lib/vm/windows/windows7-taintable.qcow2","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}
 -blockdev
{"node-name":"libvirt-2-format","read-only":true,"driver":"raw","file":"libvirt-2-storage"}

Here you can see that the
/home/joj/lib/vm/windows/windows7-taintable.qcow2 image was considered
being 'raw' this is usually because you didn't record the approprate
format. This means that qemu will no longer open the backing image.

This is the manifestation of the above problem.

 -blockdev
{"driver":"file","filename":"/home/joj/lib/vm/windows/windows7-Office2013.qcow2","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}
 -blockdev
{"node-name":"libvirt-1-format","read-only":false,"driver":"qcow2","file":"libvirt-1-storage","backing":"libvirt-2-format"}+}

> 
>  -device
[-virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1-]
>         
{+virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=libvirt-1-format,id=virtio-disk0,bootindex=1+}
> 
> As you can see, 5.10.0 is expanding the backing file chain, not leaving
> that to qemu. Under 5.10.0, I get an Operating System not Found error,
> under 5.9.0, with the shorter -drive argument, everything works as
> desired.
> 
> Can you perhaps advise me how I either fix the domain spec [1] to get
> libvirt to use a true equivalent to the old qemu invocation as I don't
> think this is a qemu problem, or how I can turn the new-style JSON
> parametrization off?
So as a first thing you should create the images properly. This means
that when you are creating the overlay image, you have to use the '-F'
parameter of qemu-img create. (Note that snapshots/overlays created by
libvirt always do this). The -F parameter specifies the format of the
backing image specified by -b:

qemu-img create -f qcow2 -F qcow2 -b windows7-taintable.qcow2
windows7-Office2013.qcow2

For fixing the image you can use the qemu-img rebase:

qemu-img rebase -f qcow2 -F qcow2 -b
/home/joj/lib/vm/windows/windows7-taintable.qcow2  windows7-Office2013.qcow2

Please use full paths for the -b argument unless you want relative
relationships between images.

A second option is to specify the image chain in the XML. This works
starting from libvirt 5.10 :

   <disk type='file' device='disk'>
     <driver name='qemu' type='qcow2'/>
     <source
file='/home/joj/lib/vm/windows/windows7-Office2013.qcow2'/>
     <target dev='vda' bus='virtio'/>
     <address type='pci' domain='0x0000' bus='0x00'
slot='0x07' function='0x0'/>
     <backingStore type='file'>
       <format type='qcow2'/>
       <source
file='/home/joj/lib/vm/windows/windows7-taintable.qcow2'/>
       <backingStore type='file'>
         <format type='qcow2'/>
         <source
file='/home/joj/lib/vm/windows/windows7-base.qcow2'/>
         <backingStore/>
       </backingStore>
     </backingStore>
   </disk>

Note that this does not fix the images so they will work only with the
above XML.

As a last possibility you can disable the 'blockdev' feature in
/etc/libvirt/qemu.conf. I will not be more specific on how to do it
as it's mostly for debugging purposes and could break stuff in the
future.

Lastly this is more-or-less tracked in:
https://bugzilla.redhat.com/show_bug.cgi?id=1588373

I think I should finally fix it and report a proper error rather than
silently assuming it might work. With -blockdev it will never work
properly anyways.
> 
> Other suggestions how to tackle this would also be much appreciated.
> 
> ---
> [1] https://0x0.st/z0KM.xml
> [2] https://0x0.st/z0K1.txt
> [3] https://0x0.st/z0Kj.txt
> 
> 
> _______________________________________________
> libvirt-users mailing list
> libvirt-users@redhat.com
> https://www.redhat.com/mailman/listinfo/libvirt-users

Hello Peter,

thank you for replying in such detail.

On Tue, Dec 17, 2019 at 11:22:52AM +0100, Peter Krempa
wrote:
> So this output was obtained by 'file'. That unfortunately
doesn't give
> enough information. Please run 'qemu-img info' on those files and
look
> for the 'backing file format:'field.
>
> That field is important, because otherwise we'd fall back to image
> format detection which was deemed unsafe and libvirt no longer allows
> it.
You're right. My backing files are missing the 'backing file format'
field:

  image: windows7-base.qcow2
  file format: qcow2
  virtual size: 50 GiB (53687091200 bytes)
  disk size: 7.37 GiB
  cluster_size: 65536
  Format specific information:
      compat: 1.1
      lazy refcounts: false
      refcount bits: 16
      corrupt: false

  image: windows7-Office2013.qcow2
  file format: qcow2
  virtual size: 75 GiB (80530636800 bytes)
  disk size: 50.4 GiB
  cluster_size: 65536
  backing file: windows7-taintable.qcow2
  Format specific information:
      compat: 1.1
      lazy refcounts: false
      refcount bits: 16
      corrupt: false

  image: windows7-taintable.qcow2
  file format: qcow2
  virtual size: 50 GiB (53687091200 bytes)
  disk size: 7.88 GiB
  cluster_size: 65536
  backing file: windows7-base.qcow2
  Format specific information:
      compat: 1.1
      lazy refcounts: false
      refcount bits: 16
      corrupt: false
> The root of the problem is that the images mentioned above will not
> have the format section because -F argument for qemu-img was not used
> when creating them. This would force libvirt to use format detection
> which is insecure and thus libvirt does not do that and assumes the
> image is raw.
Understood. So in order to load an image or backing file, qemu/libvirt
expects the format to be explicitly defined whenever a 'next image' is
being referenced.
> This in normal circumstances causes problems right away, because when
> running any more advanced security model than unix permissions (selinux
> apparmor) libvirt would not allow qemu to access the backing file. You
> don't seem to be running it so it was accidentally working for you
until
> now.
Correct. I'm using neither selinux nor apparmor.
> (...) So as a first thing you should create the images properly.  that when
> you are creating the overlay image, you have to use the '-F'
parameter
> of qemu-img create. (...)
Fixed the issue successfully using qemu-img rebase.
> (...) (Note that snapshots/overlays created by libvirt always do
> this). (...)
It's probably clear by now but these images were created manually, many
years ago.
> Lastly this is more-or-less tracked in:
> https://bugzilla.redhat.com/show_bug.cgi?id=1588373
>
> I think I should finally fix it and report a proper error rather than
> silently assuming it might work. With -blockdev it will never work
> properly anyways.
I agree that failing in a clear way would indeed help the user, though
now that I know what I'm looking for, it's also clear from the -blockdev
argument what's going on.

Thanks, again, for your time,
Jens.