libvirt users - Aug 2020 - unable to migrate non shared storage in tunneled mode

Hey all,
With libvirt 6.5.0 and qemu 5.1.0 migration of non shared disks in
tunneled mode does not work for me:

virsh # migrate alpinelinux3.8 qemu+tls://ratchet.lan/system --live
--persistent --undefinesource --copy-storage-all --tunneled --p2p
error: internal error: qemu unexpectedly closed the monitor: Receiving
block device images
Error unknown block device
2020-08-15T21:21:48.995016Z qemu-system-x86_64: error while loading
state section id 1(block)
2020-08-15T21:21:48.995193Z qemu-system-x86_64: load of migration
failed: Invalid argument

This is both with UEFI and BIOS guests.

I understand that newer ways of migrating non shared disks is via NBD
directly with QEMU, however I am certain
that this used to work before libvirt 6.0. There is a series of
commits to /src/qemu/qemu_migration.c on Dec 8, 2019,
could they have something to do with this?

Is migration of non shared disks supported and supposed to work in
tunneled mode or is it not a supported configuration
and native NBD directly with QEMU should be used in all cases?

Thanks in advance!

Full qemu log on receiving host:

2020-08-15 21:23:38.917+0000: starting up libvirt version: 6.5.0, qemu
version: 5.1.0, kernel: 5.4.57-gentoo, hostname: ratchet.lan
LC_ALL=C \
PATH=/bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/bin
\
HOME=/var/lib/libvirt/qemu/domain-4-alpinelinux3.8 \
USER=root \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-4-alpinelinux3.8/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-4-alpinelinux3.8/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-4-alpinelinux3.8/.config \
QEMU_AUDIO_DRV=none \
/usr/bin/qemu-system-x86_64 \
-name guest=alpinelinux3.8,debug-threads=on \
-S \
-object
secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-4-alpinelinux3.8/master-key.aes
\
-blockdev
'{"driver":"file","filename":"/var/lib/libvirt/uefi/OVMF_CODE.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}'
\
-blockdev
'{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}'
\
-blockdev
'{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/alpinelinux3.8_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}'
\
-blockdev
'{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}'
\
-machine
pc-q35-5.1,accel=kvm,usb=off,dump-guest-core=off,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format
\
-cpu
kvm64,ibpb=on,md-clear=on,spec-ctrl=on,ssbd=on,vme=on,x2apic=on,hypervisor=on
\
-m 2048 \
-overcommit mem-lock=off \
-smp 2,sockets=2,cores=1,threads=1 \
-uuid 95286971-32fa-4138-be0e-519ec21af800 \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=35,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot strict=on \
-device
pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2
\
-device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \
-device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \
-device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \
-device qemu-xhci,p2=15,p3=15,id=usb,bus=pci.1,addr=0x0 \
-blockdev
'{"driver":"host_device","filename":"/dev/vg0/test","aio":"native","node-name":"libvirt-2-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}'
\
-blockdev
'{"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-2-storage"}'
\
-device
virtio-blk-pci,bus=pci.2,addr=0x0,drive=libvirt-2-format,id=virtio-disk0,bootindex=1,write-cache=on
\
-device ide-cd,bus=ide.0,id=sata0-0-0 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-vnc 127.0.0.1:2 \
-device VGA,id=video0,vgamem_mb=16,bus=pcie.0,addr=0x1 \
-incoming defer \
-device virtio-balloon-pci,id=balloon0,bus=pci.3,addr=0x0 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny
\
-msg timestamp=on
char device redirected to /dev/pts/5 (label charserial0)
Receiving block device images
Error unknown block device
2020-08-15T21:23:39.278287Z qemu-system-x86_64: error while loading
state section id 1(block)
2020-08-15T21:23:39.278422Z qemu-system-x86_64: load of migration
failed: Invalid argument
2020-08-15 21:23:39.344+0000: shutting down, reason=failed

On Sat, Aug 15, 2020 at 15:38:19 -0700, Vjaceslavs Klimovs
wrote:
> Hey all,
> With libvirt 6.5.0 and qemu 5.1.0 migration of non shared disks in
> tunneled mode does not work for me:
> 
> virsh # migrate alpinelinux3.8 qemu+tls://ratchet.lan/system --live
> --persistent --undefinesource --copy-storage-all --tunneled --p2p
> error: internal error: qemu unexpectedly closed the monitor: Receiving
> block device images
> Error unknown block device
> 2020-08-15T21:21:48.995016Z qemu-system-x86_64: error while loading
> state section id 1(block)
> 2020-08-15T21:21:48.995193Z qemu-system-x86_64: load of migration
> failed: Invalid argument
> 
> This is both with UEFI and BIOS guests.
The migration of storage using the qemu migration stream is not
supported by qemu when -blockdev is used. Since we can only transport
one stream when migrating with '-tunelled', it's no longer possible
to
combine those two. Unfortunately automagic block migration was deemed a
legacy feature.

> I understand that newer ways of migrating non shared disks is via NBD
> directly with QEMU, however I am certain
> that this used to work before libvirt 6.0. There is a series of
> commits to /src/qemu/qemu_migration.c on Dec 8, 2019,
> could they have something to do with this?
I presume those are my commits which fix the NBD migration though.
Unfortunately they can't fix the old one.
> Is migration of non shared disks supported and supposed to work in
> tunneled mode or is it not a supported configuration
> and native NBD directly with QEMU should be used in all cases?
NBD should be used nowadays. The old style storage migration was
neglected for a very long time now.

On Mon, Aug 17, 2020 at 3:24 AM Peter Krempa <pkrempa@redhat.com>
wrote:
>
> On Sat, Aug 15, 2020 at 15:38:19 -0700, Vjaceslavs Klimovs wrote:
> > Hey all,
> > With libvirt 6.5.0 and qemu 5.1.0 migration of non shared disks in
> > tunneled mode does not work for me:
> >
> > virsh # migrate alpinelinux3.8 qemu+tls://ratchet.lan/system --live
> > --persistent --undefinesource --copy-storage-all --tunneled --p2p
> > error: internal error: qemu unexpectedly closed the monitor: Receiving
> > block device images
> > Error unknown block device
> > 2020-08-15T21:21:48.995016Z qemu-system-x86_64: error while loading
> > state section id 1(block)
> > 2020-08-15T21:21:48.995193Z qemu-system-x86_64: load of migration
> > failed: Invalid argument
> >
> > This is both with UEFI and BIOS guests.
>
> The migration of storage using the qemu migration stream is not
> supported by qemu when -blockdev is used. Since we can only transport
> one stream when migrating with '-tunelled', it's no longer
possible to
> combine those two. Unfortunately automagic block migration was deemed a
> legacy feature.
>
>
> > I understand that newer ways of migrating non shared disks is via NBD
> > directly with QEMU, however I am certain
> > that this used to work before libvirt 6.0. There is a series of
> > commits to /src/qemu/qemu_migration.c on Dec 8, 2019,
> > could they have something to do with this?
>
> I presume those are my commits which fix the NBD migration though.
> Unfortunately they can't fix the old one.
>
> > Is migration of non shared disks supported and supposed to work in
> > tunneled mode or is it not a supported configuration
> > and native NBD directly with QEMU should be used in all cases?
>
> NBD should be used nowadays. The old style storage migration was
> neglected for a very long time now.
>
Thank you for the clarification, very helpful.

Local libvirtd of course knows about --tunneled and -blockdev being
used - would it be possible for it to provide a clear error message
that this combination is not supported when such migration is
attempted?