Pkg xen devel - Jan 2019 - Bug#833153: pv guest live migration fails

Package: xen-system-amd64
Version: 4.6.0-1+nmu2
Severity: important

Dear Maintainer,


New install on identical dell servers.

1. Migration of pv guest from server1 to server2 works
Attempt to migrate back to server1 fails.

2. Migration of pv guest from server2 to server1 works
Attempt to migrate back to server2 fails.

Below are test results & configs

----------------
test1

root at c1-sb-vs1:~# xl create /etc/xen/vmcfg/c1-sb-test1.cfg
Parsing config from /etc/xen/vmcfg/c1-sb-test1.cfg


root at c1-sb-vs1:~# xl migrate c1-sb-test1 192.168.0.2
migration target: Ready to receive domain.
Saving to migration stream new xl format (info 0x3/0x0/1559)
Loading new save file <incoming migration stream> (new xl fmt info 
0x3/0x0/1559)
  Savefile contains xl domain config in JSON format
Parsing config from <saved>
xc: info: Saving domain 10, type x86 PV
xc: info: Found x86 PV domain from Xen 4.6
xc: info: Restoring domain
xc: info: Restore successful
xc: info: XenStore: mfn 0x1f6f61, dom 0, evt 1
xc: info: Console: mfn 0x1f6f60, dom 0, evt 2
migration target: Transfer complete, requesting permission to start domain.
migration sender: Target has acknowledged transfer.
migration sender: Giving target permission to start.
migration target: Got permission, starting domain.
migration target: Domain started successsfully.
migration sender: Target reports successful startup.
Migration successful.

root at c1-sb-vs2:~# xl migrate c1-sb-test1 192.168.0.1
migration target: Ready to receive domain.
Saving to migration stream new xl format (info 0x3/0x0/1559)
Loading new save file <incoming migration stream> (new xl fmt info 
0x3/0x0/1559)
  Savefile contains xl domain config in JSON format
Parsing config from <saved>
xc: info: Saving domain 3, type x86 PV
xc: info: Found x86 PV domain from Xen 4.6
xc: info: Restoring domain
libxl: error: libxl_dom_suspend.c:318:suspend_common_wait_guest_timeout: 
guest did not suspend, timed out
xc: error: save callback suspend() failed: 0: Internal error
xc: error: Save failed (0 = Success): Internal error
libxl: error: libxl_stream_write.c:329:libxl__xc_domain_save_done: 
saving domain: domain responded to suspend request: Success
migration sender: libxl_domain_suspend failed (rc=-8)
xc: error: Failed to read Record Header from stream (0 = Success): 
Internal error
xc: error: Restore failed (0 = Success): Internal error
libxl: error: libxl_stream_read.c:749:libxl__xc_domain_restore_done: 
restoring domain: Success
libxl: error: libxl_create.c:1142:domcreate_rebuild_done: cannot 
(re-)build domain: -3
libxl: error: libxl.c:1591:libxl__destroy_domid: non-existant domain 11
libxl: error: libxl.c:1549:domain_destroy_callback: unable to destroy 
guest with domid 11
libxl: error: libxl.c:1476:domain_destroy_cb: destruction of domain 11 
failed
migration target: Domain creation failed (code -3).
libxl: info: libxl_exec.c:118:libxl_report_child_exitstatus: migration 
transport process [13580] exited with error status 3
Migration failed, failed to suspend at sender.


----------------
test2

root at c1-sb-vs2:~# xl create /etc/xen/vmcfg/c1-sb-test1.cfg
Parsing config from /etc/xen/vmcfg/c1-sb-test1.cfg

root at c1-sb-vs2:~# xl migrate c1-sb-test1 192.168.0.1
migration target: Ready to receive domain.
Saving to migration stream new xl format (info 0x3/0x0/1559)
Loading new save file <incoming migration stream> (new xl fmt info 
0x3/0x0/1559)
  Savefile contains xl domain config in JSON format
Parsing config from <saved>
xc: info: Saving domain 3, type x86 PV
xc: info: Found x86 PV domain from Xen 4.6
xc: info: Restoring domain
xc: info: Restore successful
xc: info: XenStore: mfn 0x1daaa3, dom 0, evt 1
xc: info: Console: mfn 0x1daaa2, dom 0, evt 2
migration target: Transfer complete, requesting permission to start domain.
migration sender: Target has acknowledged transfer.
migration sender: Giving target permission to start.
migration target: Got permission, starting domain.
migration target: Domain started successsfully.
migration sender: Target reports successful startup.
Migration successful.


root at c1-sb-vs1:~# xl migrate c1-sb-test1 192.168.0.2
migration target: Ready to receive domain.
Saving to migration stream new xl format (info 0x3/0x0/1559)
Loading new save file <incoming migration stream> (new xl fmt info 
0x3/0x0/1559)
  Savefile contains xl domain config in JSON format
Parsing config from <saved>
xc: info: Saving domain 4, type x86 PV
xc: info: Found x86 PV domain from Xen 4.6
xc: info: Restoring domain
libxl: error: libxl_dom_suspend.c:318:suspend_common_wait_guest_timeout: 
guest did not suspend, timed out
xc: error: save callback suspend() failed: 0: Internal error
xc: error: Save failed (0 = Success): Internal error
libxl: error: libxl_stream_write.c:329:libxl__xc_domain_save_done: 
saving domain: domain responded to suspend request: Success
migration sender: libxl_domain_suspend failed (rc=-8)
xc: error: Failed to read Record Header from stream (0 = Success): 
Internal error
xc: error: Restore failed (0 = Success): Internal error
libxl: error: libxl_stream_read.c:749:libxl__xc_domain_restore_done: 
restoring domain: Success
libxl: error: libxl_create.c:1142:domcreate_rebuild_done: cannot 
(re-)build domain: -3
libxl: error: libxl.c:1591:libxl__destroy_domid: non-existant domain 4
libxl: error: libxl.c:1549:domain_destroy_callback: unable to destroy 
guest with domid 4
libxl: error: libxl.c:1476:domain_destroy_cb: destruction of domain 4 failed
migration target: Domain creation failed (code -3).
libxl: info: libxl_exec.c:118:libxl_report_child_exitstatus: migration 
transport process [29045] exited with error status 3
Migration failed, failed to suspend at sender.


----------------
Config files

  /etc/xen/vmcfg/c1-sb-test1.cfg
#
# Configuration file for the Xen instance c1-sb-test1, created
# by xen-tools 4.6.2 on Mon Aug  1 10:07:16 2016.
#

#
#  Kernel + memory size
#
kernel      = '/boot/vmlinuz-4.6.0-1-amd64'
extra       = 'elevator=noop'
ramdisk     = '/boot/initrd.img-4.6.0-1-amd64'

vcpus       = '1'
memory      = '512'


#
#  Disk device(s).
#
root        = '/dev/xvda2 ro'
disk        = [
                   'phy:/dev/vg1/c1-sb-test1-disk,xvda2,w',
                   'phy:/dev/vg1/c1-sb-test1-swap,xvda1,w',
               ]


#
#  Physical volumes
#


#
#  Hostname
#
name        = 'c1-sb-test1'

#
#  Networking
#
vif         = [ 'ip=10.0.0.127 ,mac=00:16:3E:0C:44:90,bridge=br1' ]

#
#  Behaviour
#
on_poweroff = 'destroy'
on_reboot   = 'restart'
on_crash    = 'restart'


----------------
defautls
TOOLSTACK=xl
XENCONSOLED_ARGSXENSTORED_ARGSXENDOMAINS_SAVE=""
XENDOMAINS_RESTORE=false
XENDOMAINS_AUTOXENDOMAINS_STOP_MAXWAIT=300

----------------
/etc/xen/xend-config.sxp

(xend-relocation-server yes)
(xend-relocation-port 8002)
(xend-relocation-address 'bond0')
(xend-relocation-hosts-allow '')
(vif-script vif-bridge)
(dom0-min-mem 196)
(enable-dom0-ballooning yes)
(total_available_memory 0)
(dom0-cpus 0)






-- System Information:
Debian Release: stretch/sid
   APT prefers stable-updates
   APT policy: (500, 'stable-updates'), (500, 'testing'), (500,
'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 4.6.0-1-amd64 (SMP w/8 CPU cores)
Locale: LANG=en_PH.UTF-8, LC_CTYPE=en_PH.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages xen-system-amd64 depends on:
ii  xen-hypervisor-4.6-amd64  4.6.0-1+nmu2
ii  xen-utils-4.6             4.6.0-1+nmu2

xen-system-amd64 recommends no packages.

xen-system-amd64 suggests no packages.

-- no debconf information

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.alioth.debian.org/pipermail/pkg-xen-devel/attachments/20160801/947de5c7/attachment.html>

reassign 833153 src:xen 4.6.0-1+nmu2
tags 833153 + moreinfo
thanks

Hi,

This is about the bug report about live migration issues that you filed
before in the Debian bug tracker, against the Xen packages.

Your bug report was targeted at a Xen package in a Debian distribution
older than the current stable (Stretch).

Can you please help us by confirming that any of the following scenarios
does apply to your situation?

* I had this problem a long time ago. It was never solved, but I found a
workaround, which is ...
* I had this problem a long time ago, and I solved it by not using Xen
any more, but by doing ...
* I still experience this problem, and I'm still using Xen
3.2/4.1/4.4/etc. I cannot upgrade to Debian Stretch or Buster because ...
* I had this problem, and since upgrading to Stretch / Buster / ? it
seems it was solved, and I forgot to report it again. Please close it,
thanks.
* Other: ...

Note that even if you found a solution, it's still very useful to report
it back to our bug tracker. There might be someone else running into the
same problem, who can be helped with your information.

Please note that unless there's a response within a month from now, we
will close the bug report. If you discover this message later, and this
case is important to you, then you can try unarchiving the bug and
replying to it, or reach out to the maintainers email list at
pkg-xen-devel at lists.alioth.debian.org (no subscription required) and
post a message.

Thanks,
Hans van Kranenburg

Processing commands for control at bugs.debian.org:
> reassign 833153 src:xen 4.6.0-1+nmu2
Bug #833153 [xen-system-amd64] pv guest live migration fails,
(libxl_dom_suspend.c:318:suspend_common_wait_guest_timeout)
Bug reassigned from package 'xen-system-amd64' to 'src:xen'.
No longer marked as found in versions xen/4.6.0-1+nmu2.
Ignoring request to alter fixed versions of bug #833153 to the same values
previously set
Bug #833153 [src:xen] pv guest live migration fails,
(libxl_dom_suspend.c:318:suspend_common_wait_guest_timeout)
Marked as found in versions xen/4.6.0-1+nmu2.
> tags 833153 + moreinfo
Bug #833153 [src:xen] pv guest live migration fails,
(libxl_dom_suspend.c:318:suspend_common_wait_guest_timeout)
Added tag(s) moreinfo.
> thanks
Stopping processing here.

Please contact me if you need assistance.
-- 
833153: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=833153
Debian Bug Tracking System
Contact owner at bugs.debian.org with problems

Your message dated Fri, 22 Feb 2019 16:23:56 +0100
with message-id <33b0a856-fd90-9bf9-4235-7ca3ebd8a2d4 at knorrie.org>
and subject line Re: pv guest live migration fails
has caused the Debian Bug report #833153,
regarding pv guest live migration fails,
(libxl_dom_suspend.c:318:suspend_common_wait_guest_timeout)
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner at bugs.debian.org
immediately.)


-- 
833153: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=833153
Debian Bug Tracking System
Contact owner at bugs.debian.org with problems
-------------- next part --------------
An embedded message was scrubbed...
From: Systems Administrator at Cyberone <sysadm at cyberone.ph>
Subject: pv guest live migration fails,
(libxl_dom_suspend.c:318:suspend_common_wait_guest_timeout)
Date: Mon, 1 Aug 2016 21:35:22 +0800
Size: 21070
URL:
<http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20190222/cabb2212/attachment-0002.mht>
-------------- next part --------------
An embedded message was scrubbed...
From: Hans van Kranenburg <hans at knorrie.org>
Subject: Re: pv guest live migration fails
Date: Fri, 22 Feb 2019 16:23:56 +0100
Size: 7464
URL:
<http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20190222/cabb2212/attachment-0003.mht>