libvirt users - May 2018 - virsh migration job unexpectedly failed

Hi ,

    when i have run virsh migrate command , the migration progress fiailed
at [61 %].
    command :
    virsh migrate --live --undefinesource --p2p --copy-storage-inc
--verbose --persistent instance-00001959 qemu+tcp://host292/system

    result :
    Migration: [ 61 %]error: operation failed: migration job: unexpectedly
failed

    os/kernel version: centos 6.7 / kernel 2.6.32
    qemu-kvm version : 0.12.1.2
    libvirt version : 0.10.2-54

    the libvirt.log on target host:
    2018-05-22 11:45:38.603+0000: 32314: warning : qemuDomainObjTaint:1459
: Domain id=30 name='instance-00001959'
uuid=41975d30-2490-473e-be7b-9d2d08bf66d2 is tainted: high-privileges
    2018-05-22 11:45:38.883+0000: 32314: warning :
qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async
job owner; entering monitor without asking for a nested job is dangerous
2018-05-22 11:45:38.902+0000: 32314: warning :
qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async
job owner; entering monitor without asking for a nested job is dangerous
2018-05-22 11:45:38.903+0000: 32314: warning :
qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async
job owner; entering monitor without asking for a nested job is dangerous
2018-05-22 11:45:39.090+0000: 32314: warning :
qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async
job owner; entering monitor without asking for a nested job is dangerous
2018-05-22 11:45:39.090+0000: 32314: warning :
qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async
job owner; entering monitor without asking for a nested job is dangerous
2018-05-22 11:45:39.091+0000: 32314: warning :
qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async
job owner; entering monitor without asking for a nested job is dangerous
2018-05-22 11:45:41.841+0000: 32310: error : qemuMonitorIO:615 : internal
error End of file from monitor

    the qmeu-kvminstance-00001959.log:
2018-05-22 11:45:38.603+0000: starting up
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin HOME=/home/ning.zhao USER=root
LOGNAME=root QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name
instance-00001959 -S -M rhel6.6.0 -cpu
Broadwell,+abm,+pdpe1gb,+rdrand,+f16c,+osxsave,+dca,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+m$
nitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -enable-kvm -m 4096 -realtime
mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid
41975d30-2490-473e-be7b-9d2d08bf66d2 -smbios type=1,manufacturer=Red Hat
Inc.,product=OpenStack Nova,version=2013.1.2-4.el6,serial=fcbe6c27-c9$
d-e811-ac36-a281f9ebc503,uuid=41975d30-2490-473e-be7b-9d2d08bf66d2
-nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00001959.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc
base=utc,driftfix=slew -n$
-kvm-pit-reinjection -no-shutdown -device
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=/var/lib/nova/instances/41975d30-2490-473e-be7b-9d2d08bf66d2/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4$
drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev
tap,fd=28,id=hostnet0,vhost=on,vhostfd=29 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:c0:8f:6b,bus=pci.0,addr=0x3
-chardev
file,id=charserial0,path=/var/lib/nova/instances/41975d30-2490-473e-b$
7b-9d2d08bf66d2/console.log -device
isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1
-device isa-serial,chardev=charserial1,id=serial1 -device
usb-tablet,id=input0 -vnc 127.0.0.1:2 -k en-us -vga cirrus -incoming tcp:
0.0.0.0:49152 -device virtio-bal$
oon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on
Domain id=30 is tainted: high-privileges
char device redirected to /dev/pts/2
Receiving block device images
qemu: warning: error while loading state for instance 0x0 of device
'block'
load of migration failed
2018-05-22 11:45:41.841+0000: shutting down

i checked the log and disk file and they have right permissions,  has
anyone know about this ,i have google many times ,but i cant find the right
debug way to resolve it.

thanks

On 05/23/2018 01:26 PM, cokabug wrote:
> Hi ,
> 
>     when i have run virsh migrate command , the migration progress fiailed
> at [61 %].
>     command :
>     virsh migrate --live --undefinesource --p2p --copy-storage-inc
> --verbose --persistent instance-00001959 qemu+tcp://host292/system
> 
>     result :
>     Migration: [ 61 %]error: operation failed: migration job: unexpectedly
> failed
> 
>     os/kernel version: centos 6.7 / kernel 2.6.32
>     qemu-kvm version : 0.12.1.2
>     libvirt version : 0.10.2-54
These are very ancient versions. I wouldn't be surprised if you upgrade
and find out the problem is fixed.
> 
>     the libvirt.log on target host:
>     2018-05-22 11:45:38.603+0000: 32314: warning : qemuDomainObjTaint:1459
> : Domain id=30 name='instance-00001959'
> uuid=41975d30-2490-473e-be7b-9d2d08bf66d2 is tainted: high-privileges
>     2018-05-22 11:45:38.883+0000: 32314: warning :
> qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async
> job owner; entering monitor without asking for a nested job is dangerous
> 2018-05-22 11:45:38.902+0000: 32314: warning :
> qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async
> job owner; entering monitor without asking for a nested job is dangerous
> 2018-05-22 11:45:38.903+0000: 32314: warning :
> qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async
> job owner; entering monitor without asking for a nested job is dangerous
> 2018-05-22 11:45:39.090+0000: 32314: warning :
> qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async
> job owner; entering monitor without asking for a nested job is dangerous
> 2018-05-22 11:45:39.090+0000: 32314: warning :
> qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async
> job owner; entering monitor without asking for a nested job is dangerous
> 2018-05-22 11:45:39.091+0000: 32314: warning :
> qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async
> job owner; entering monitor without asking for a nested job is dangerous
> 2018-05-22 11:45:41.841+0000: 32310: error : qemuMonitorIO:615 : internal
> error End of file from monitor
> 
This says that qemu died. You can try to look into domain log to see why.

Michal

finally, i found the problem, qemu-img create command not specific vm disk
size on target host, when i use the size same as source vm, the migration
finish successfully.

On Wed, May 23, 2018 at 7:56 PM, Michal Privoznik <mprivozn@redhat.com>
wrote:
> On 05/23/2018 01:26 PM, cokabug wrote:
> > Hi ,
> >
> >     when i have run virsh migrate command , the migration progress
> fiailed
> > at [61 %].
> >     command :
> >     virsh migrate --live --undefinesource --p2p --copy-storage-inc
> > --verbose --persistent instance-00001959 qemu+tcp://host292/system
> >
> >     result :
> >     Migration: [ 61 %]error: operation failed: migration job:
> unexpectedly
> > failed
> >
> >     os/kernel version: centos 6.7 / kernel 2.6.32
> >     qemu-kvm version : 0.12.1.2
> >     libvirt version : 0.10.2-54
>
> These are very ancient versions. I wouldn't be surprised if you upgrade
> and find out the problem is fixed.
>
> >
> >     the libvirt.log on target host:
> >     2018-05-22 11:45:38.603+0000: 32314: warning :
> qemuDomainObjTaint:1459
> > : Domain id=30 name='instance-00001959'
> > uuid=41975d30-2490-473e-be7b-9d2d08bf66d2 is tainted: high-privileges
> >     2018-05-22 11:45:38.883+0000: 32314: warning :
> > qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the
> async
> > job owner; entering monitor without asking for a nested job is
dangerous
> > 2018-05-22 11:45:38.902+0000: 32314: warning :
> > qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the
> async
> > job owner; entering monitor without asking for a nested job is
dangerous
> > 2018-05-22 11:45:38.903+0000: 32314: warning :
> > qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the
> async
> > job owner; entering monitor without asking for a nested job is
dangerous
> > 2018-05-22 11:45:39.090+0000: 32314: warning :
> > qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the
> async
> > job owner; entering monitor without asking for a nested job is
dangerous
> > 2018-05-22 11:45:39.090+0000: 32314: warning :
> > qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the
> async
> > job owner; entering monitor without asking for a nested job is
dangerous
> > 2018-05-22 11:45:39.091+0000: 32314: warning :
> > qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the
> async
> > job owner; entering monitor without asking for a nested job is
dangerous
> > 2018-05-22 11:45:41.841+0000: 32310: error : qemuMonitorIO:615 :
internal
> > error End of file from monitor
> >
>
> This says that qemu died. You can try to look into domain log to see why.
>
> Michal
>