thr3ads.net - Xen users - [Xen-users] Live migration fails (ERROR Restore failed) [Jan 2011]

If this information is useful, please help other people find it:
Share via:
Hitesh Khandelwal
2011-Jan-31 19:28 UTC
[Xen-users] Live migration fails (ERROR Restore failed)

I compiled xen-4.0.1 (kernel 2.6.32.27) on Ubuntu 10.10 as Dom0. Ubuntu
10.10 with stock kernel is used as DomU. I tried live migration by following
instructions on
http://www.virtuatopia.com/index.php/Migrating_Xen_domainU_Guests_Between_Host_Systems
,
but it failed with the following logs.

Both the machines have same configuration and same amount of memory, and
there is plenty of memory available on target host. VM guest runs
successfully on both the source and target hosts individually. Relevant bug
report: http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1653

Thanks,
Hitesh


Log on source Dom0:

[2011-01-31 14:20:24 1008] DEBUG (XendCheckpoint:124) [xc_save]:
/usr/lib/xen/bin/xc_save 46 10 0 0 1
[2011-01-31 14:20:24 1008] INFO (XendCheckpoint:423)
/usr/lib/xen/bin/xc_save: symbol lookup error: /usr/lib/xen/bin/xc_save:
undefined symbol: xs_suspend_evtchn_port
[2011-01-31 14:20:24 1008] ERROR (XendCheckpoint:178) Save failed on domain
cloud-vm01 (10) - resuming.
Traceback (most recent call last):
  File
"/usr/local/lib/python2.6/dist-packages/xen/xend/XendCheckpoint.py",
line 146, in save
    forkHelper(cmd, fd, saveInputHandler, False)
  File
"/usr/local/lib/python2.6/dist-packages/xen/xend/XendCheckpoint.py",
line 409, in forkHelper
    raise XendError("%s failed: popen failed" % string.join(cmd))
XendError: /usr/lib/xen/bin/xc_save 46 10 0 0 1 failed: popen failed
[2011-01-31 14:20:24 1008] DEBUG (XendDomainInfo:3117)
XendDomainInfo.resumeDomain(10)
[2011-01-31 14:20:27 1008] ERROR (xmlrpclib2:178) Internal error handling
xend.domain.migrate
Traceback (most recent call last):
  File
"/usr/local/lib/python2.6/dist-packages/xen/util/xmlrpclib2.py", line
131, in _marshaled_dispatch
    response = self._dispatch(method, params)
  File "/usr/lib/python2.6/SimpleXMLRPCServer.py", line 418, in
_dispatch
    return func(*params)
  File
"/usr/local/lib/python2.6/dist-packages/xen/xend/XendDomain.py", line
1366, in domain_migrate
    self._domain_migrate(dominfo, dst, live, port, node)
  File
"/usr/local/lib/python2.6/dist-packages/xen/xend/XendDomain.py", line
1458, in _domain_migrate
    dsterr = sock.recv(1024)
timeout: timed out

Log on target Dom0:

[2011-01-31 14:20:23 12024] DEBUG (XendDomainInfo:235)
XendDomainInfo.restore([\047domain\047, [\047domid\047, \04710\047],
[\047cpu_weight\047, \047256\047], [\047cpu_cap\047, \0470\047],
[\047bootloader\047, \047/usr/bin/pygrub\047], [\047on_crash\047,
\047restart\047], [\047uuid\047,
\047f32d195c-1fed-a9ad-5d7e-ff0456f79505\047], [\047bootloader_args\047],
[\047vcpus\047, \0471\047], [\047description\047], [\047name\047,
\047cloud-vm01\047], [\047cpus\047, [[]]], [\047on_reboot\047,
\047restart\047], [\047on_poweroff\047, \047destroy\047], [\047maxmem\047,
\047512\047], [\047memory\047, \047512\047], [\047shadow_memory\047,
\0470\047], [\047vcpu_avail\047, \0471\047], [\047features\047],
[\047on_xend_start\047, \047ignore\047], [\047on_xend_stop\047,
\047ignore\047], [\047start_time\047, \0471296428689.01\047],
[\047cpu_time\047, \04711.77212655\047], [\047online_vcpus\047, \0471\047],
[\047image\047, [\047linux\047, [\047kernel\047], [\047args\047,
\047root=/dev/sda2 ro \047], [\047superpages\047, \0470\047],
[\047videoram\047, \0474\047], [\047pci\047, []], [\047nomigrate\047,
\0470\047], [\047tsc_mode\047, \0470\047], [\047notes\047,
[\047HV_START_LOW\047, \0474118806528\047], [\047FEATURES\047,
\047!writable_page_tables|pae_pgdir_above_4gb\047], [\047VIRT_BASE\047,
\0473221225472\047], [\047GUEST_VERSION\047, \0472.6\047],
[\047PADDR_OFFSET\047, \0470\047], [\047GUEST_OS\047, \047linux\047],
[\047HYPERCALL_PAGE\047, \0473222282240\047], [\047LOADER\047,
\047generic\047], [\047SUSPEND_CANCEL\047, \0471\047], [\047PAE_MODE\047,
\047yes\047], [\047ENTRY\047, \0473229974528\047], [\047XEN_VERSION\047,
\047xen-3.0\047]]]], [\047status\047, \0472\047], [\047state\047,
\047-b----\047], [\047store_mfn\047, \0472262985\047], [\047console_mfn\047,
\0472262984\047], [\047device\047, [\047vif\047, [\047bridge\047,
\047eth2\047], [\047uuid\047, \0477c22b250-2997-9b91-e6f0-2ad17a7e00db\047],
[\047script\047, \047/etc/xen/scripts/vif-bridge\047], [\047ip\047,
\047128.10.130.186\047], [\047mac\047, \04700:16:3E:C3:D2:A8\047],
[\047backend\047, \0470\047]]], [\047device\047, [\047console\047,
[\047protocol\047, \047vt100\047], [\047location\047, \0472\047],
[\047uuid\047, \047b250cf24-1551-eaa0-cfdf-ea14e0d47934\047]]],
[\047device\047, [\047vbd\047, [\047protocol\047, \047x86_32-abi\047],
[\047uuid\047, \047f522f311-68bf-4358-216d-b8f81488a31b\047],
[\047bootable\047, \0471\047], [\047dev\047, \047sda2:disk\047],
[\047uname\047,
\047file:/home/hkhande/xen-image/domains/cloud-vm01/disk.img\047],
[\047mode\047, \047w\047], [\047backend\047, \0470\047], [\047VDI\047]]],
[\047device\047, [\047vbd\047, [\047protocol\047, \047x86_32-abi\047],
[\047uuid\047, \047a7f4ba1e-84e7-f104-a5f3-cd607b83a09f\047],
[\047bootable\047, \0470\047], [\047dev\047, \047sda1:disk\047],
[\047uname\047,
\047file:/home/hkhande/xen-image/domains/cloud-vm01/swap.img\047],
[\047mode\047, \047w\047], [\047backend\047, \0470\047], [\047VDI\047]]],
[\047change_home_server\047, \047False\047]])
[2011-01-31 14:20:23 12024] DEBUG (XendDomainInfo:2508)
XendDomainInfo.constructDomain
[2011-01-31 14:20:23 12024] DEBUG (balloon:220) Balloon: 1125176 KiB free;
need 16384; done.
[2011-01-31 14:20:23 12024] DEBUG (XendDomain:464) Adding Domain: 8
[2011-01-31 14:20:23 12024] DEBUG (XendDomainInfo:3400) Storing VM details:
{\047on_xend_stop\047: \047ignore\047, \047shadow_memory\047: \0470\047,
\047uuid\047: \047f32d195c-1fed-a9ad-5d7e-ff0456f79505\047,
\047on_reboot\047: \047restart\047, \047start_time\047:
\0471296428689.01\047, \047on_poweroff\047: \047destroy\047,
\047bootloader_args\047: \047\047, \047on_xend_start\047: \047ignore\047,
\047on_crash\047: \047restart\047, \047xend/restart_count\047: \0470\047,
\047vcpus\047: \0471\047, \047vcpu_avail\047: \0471\047, \047bootloader\047:
\047/usr/bin/pygrub\047, \047image\047: "(linux (kernel ) (args
\047root=/dev/sda2 ro \047) (superpages 0) (videoram 4) (pci ()) (nomigrate
0) (tsc_mode 0) (notes (HV_START_LOW 4118806528) (FEATURES
\047!writable_page_tables|pae_pgdir_above_4gb\047) (VIRT_BASE 3221225472)
(GUEST_VERSION 2.6) (PADDR_OFFSET 0) (GUEST_OS linux) (HYPERCALL_PAGE
3222282240) (LOADER generic) (SUSPEND_CANCEL 1) (PAE_MODE yes) (ENTRY
3229974528) (XEN_VERSION xen-3.0)))", \047name\047: \047cloud-vm01\047}
[2011-01-31 14:20:23 12024] INFO (XendDomainInfo:2367) createDevice: console
: {\047protocol\047: \047vt100\047, \047location\047: \0472\047,
\047uuid\047: \047b250cf24-1551-eaa0-cfdf-ea14e0d47934\047}
[2011-01-31 14:20:23 12024] DEBUG (DevController:95) DevController: writing
{\047state\047: \0471\047, \047backend-id\047: \0470\047, \047backend\047:
\047/local/domain/0/backend/console/8/0\047} to
/local/domain/8/device/console/0.
[2011-01-31 14:20:23 12024] DEBUG (DevController:97) DevController: writing
{\047domain\047: \047cloud-vm01\047, \047frontend\047:
\047/local/domain/8/device/console/0\047, \047uuid\047:
\047b250cf24-1551-eaa0-cfdf-ea14e0d47934\047, \047frontend-id\047:
\0478\047, \047state\047: \0471\047, \047location\047: \0472\047,
\047online\047: \0471\047, \047protocol\047: \047vt100\047} to
/local/domain/0/backend/console/8/0.
[2011-01-31 14:20:23 12024] INFO (XendDomainInfo:2367) createDevice: vbd :
{\047protocol\047: \047x86_32-abi\047, \047uuid\047:
\047f522f311-68bf-4358-216d-b8f81488a31b\047, \047bootable\047: 1,
\047driver\047: \047paravirtualised\047, \047dev\047: \047sda2:disk\047,
\047uname\047:
\047file:/home/hkhande/xen-image/domains/cloud-vm01/disk.img\047,
\047mode\047: \047w\047, \047backend\047: \0470\047}
[2011-01-31 14:20:23 12024] DEBUG (DevController:95) DevController: writing
{\047virtual-device\047: \0472050\047, \047protocol\047: \047x86_32-abi\047,
\047device-type\047: \047disk\047, \047backend-id\047: \0470\047,
\047state\047: \0471\047, \047backend\047:
\047/local/domain/0/backend/vbd/8/2050\047} to
/local/domain/8/device/vbd/2050.
[2011-01-31 14:20:23 12024] DEBUG (DevController:97) DevController: writing
{\047domain\047: \047cloud-vm01\047, \047frontend\047:
\047/local/domain/8/device/vbd/2050\047, \047uuid\047:
\047f522f311-68bf-4358-216d-b8f81488a31b\047, \047bootable\047: \0471\047,
\047dev\047: \047sda2\047, \047state\047: \0471\047, \047params\047:
\047/home/hkhande/xen-image/domains/cloud-vm01/disk.img\047, \047mode\047:
\047w\047, \047online\047: \0471\047, \047frontend-id\047: \0478\047,
\047type\047: \047file\047} to /local/domain/0/backend/vbd/8/2050.
[2011-01-31 14:20:24 12024] INFO (XendDomainInfo:2367) createDevice: vbd :
{\047protocol\047: \047x86_32-abi\047, \047uuid\047:
\047a7f4ba1e-84e7-f104-a5f3-cd607b83a09f\047, \047bootable\047: 0,
\047driver\047: \047paravirtualised\047, \047dev\047: \047sda1:disk\047,
\047uname\047:
\047file:/home/hkhande/xen-image/domains/cloud-vm01/swap.img\047,
\047mode\047: \047w\047, \047backend\047: \0470\047}
[2011-01-31 14:20:24 12024] DEBUG (DevController:95) DevController: writing
{\047virtual-device\047: \0472049\047, \047protocol\047: \047x86_32-abi\047,
\047device-type\047: \047disk\047, \047backend-id\047: \0470\047,
\047state\047: \0471\047, \047backend\047:
\047/local/domain/0/backend/vbd/8/2049\047} to
/local/domain/8/device/vbd/2049.
[2011-01-31 14:20:24 12024] DEBUG (DevController:97) DevController: writing
{\047domain\047: \047cloud-vm01\047, \047frontend\047:
\047/local/domain/8/device/vbd/2049\047, \047uuid\047:
\047a7f4ba1e-84e7-f104-a5f3-cd607b83a09f\047, \047bootable\047: \0470\047,
\047dev\047: \047sda1\047, \047state\047: \0471\047, \047params\047:
\047/home/hkhande/xen-image/domains/cloud-vm01/swap.img\047, \047mode\047:
\047w\047, \047online\047: \0471\047, \047frontend-id\047: \0478\047,
\047type\047: \047file\047} to /local/domain/0/backend/vbd/8/2049.
[2011-01-31 14:20:24 12024] INFO (XendDomainInfo:2367) createDevice: vif :
{\047bridge\047: \047eth2\047, \047uuid\047:
\0477c22b250-2997-9b91-e6f0-2ad17a7e00db\047, \047script\047:
\047/etc/xen/scripts/vif-bridge\047, \047ip\047: \047128.10.130.186\047,
\047mac\047: \04700:16:3E:C3:D2:A8\047, \047backend\047: \0470\047}
[2011-01-31 14:20:24 12024] DEBUG (DevController:95) DevController: writing
{\047backend-id\047: \0470\047, \047mac\047: \04700:16:3E:C3:D2:A8\047,
\047handle\047: \0470\047, \047state\047: \0471\047, \047backend\047:
\047/local/domain/0/backend/vif/8/0\047} to /local/domain/8/device/vif/0.
[2011-01-31 14:20:24 12024] DEBUG (DevController:97) DevController: writing
{\047bridge\047: \047eth2\047, \047domain\047: \047cloud-vm01\047,
\047handle\047: \0470\047, \047uuid\047:
\0477c22b250-2997-9b91-e6f0-2ad17a7e00db\047, \047script\047:
\047/etc/xen/scripts/vif-bridge\047, \047ip\047: \047128.10.130.186\047,
\047mac\047: \04700:16:3E:C3:D2:A8\047, \047frontend-id\047: \0478\047,
\047state\047: \0471\047, \047online\047: \0471\047, \047frontend\047:
\047/local/domain/8/device/vif/0\047} to /local/domain/0/backend/vif/8/0.
[2011-01-31 14:20:24 12024] DEBUG (XendDomainInfo:1804) Storing domain
details: {\047image/entry\047: \0473229974528\047, \047console/port\047:
\0472\047, \047image/loader\047: \047generic\047, \047vm\047:
\047/vm/f32d195c-1fed-a9ad-5d7e-ff0456f79505\047,
\047control/platform-feature-multiprocessor-suspend\047: \0471\047,
\047image/hv-start-low\047: \0474118806528\047, \047image/guest-os\047:
\047linux\047, \047image/virt-base\047: \0473221225472\047,
\047memory/target\047: \047524288\047, \047image/guest-version\047:
\0472.6\047, \047image/pae-mode\047: \047yes\047, \047description\047:
\047\047, \047console/limit\047: \0471048576\047,
\047image/paddr-offset\047: \0470\047, \047image/hypercall-page\047:
\0473222282240\047, \047image/suspend-cancel\047: \0471\047,
\047cpu/0/availability\047: \047online\047,
\047image/features/pae-pgdir-above-4gb\047: \0471\047,
\047image/features/writable-page-tables\047: \0470\047,
\047console/type\047: \047xenconsoled\047, \047name\047: \047cloud-vm01\047,
\047domid\047: \0478\047, \047image/xen-version\047: \047xen-3.0\047,
\047store/port\047: \0471\047}
[2011-01-31 14:20:24 12024] DEBUG (XendCheckpoint:286) restore:shadow=0x0,
_static_max=0x20000000, _static_min=0x0,
[2011-01-31 14:20:24 12024] DEBUG (XendCheckpoint:305) [xc_restore]:
/usr/lib/xen/bin/xc_restore 16 8 1 2 0 0 0 0
[2011-01-31 14:20:26 12024] INFO (XendCheckpoint:423) ERROR Internal error:
read: p2m_size
[2011-01-31 14:20:26 12024] INFO (XendCheckpoint:423) Restore exit with rc=1
[2011-01-31 14:20:26 12024] DEBUG (XendDomainInfo:3053)
XendDomainInfo.destroy: domid=8
[2011-01-31 14:20:26 12024] ERROR (XendDomainInfo:3067)
XendDomainInfo.destroy: domain destruction failed.
Traceback (most recent call last):
  File
"/usr/local/lib/python2.6/dist-packages/xen/xend/XendDomainInfo.py",
line 3060, in destroy
    xc.domain_pause(self.domid)
Error: (3, \047No such process\047)
[2011-01-31 14:20:26 12024] DEBUG (XendDomainInfo:2416) No device model
[2011-01-31 14:20:26 12024] DEBUG (XendDomainInfo:2418) Releasing devices
[2011-01-31 14:20:26 12024] DEBUG (XendDomainInfo:2424) Removing vif/0
[2011-01-31 14:20:26 12024] DEBUG (XendDomainInfo:1286)
XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/0
[2011-01-31 14:20:26 12024] DEBUG (XendDomainInfo:2424) Removing console/0
[2011-01-31 14:20:26 12024] DEBUG (XendDomainInfo:1286)
XendDomainInfo.destroyDevice: deviceClass = console, device = console/0
[2011-01-31 14:20:27 12024] DEBUG (XendDomainInfo:2424) Removing vbd/2050
[2011-01-31 14:20:27 12024] DEBUG (XendDomainInfo:1286)
XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/2050
[2011-01-31 14:20:27 12024] DEBUG (XendDomainInfo:2424) Removing vbd/2049
[2011-01-31 14:20:27 12024] DEBUG (XendDomainInfo:1286)
XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/2049
[2011-01-31 14:20:27 12024] INFO (XendDomain:1107) Domain cloud-vm01
(f32d195c-1fed-a9ad-5d7e-ff0456f79505) deleted.
[2011-01-31 14:20:27 12024] ERROR (XendCheckpoint:357)
/usr/lib/xen/bin/xc_restore 16 8 1 2 0 0 0 0 failed
Traceback (most recent call last):
  File
"/usr/local/lib/python2.6/dist-packages/xen/xend/XendCheckpoint.py",
line 309, in restore
    forkHelper(cmd, fd, handler.handler, True)
  File
"/usr/local/lib/python2.6/dist-packages/xen/xend/XendCheckpoint.py",
line 411, in forkHelper
    raise XendError("%s failed" % string.join(cmd))
XendError: /usr/lib/xen/bin/xc_restore 16 8 1 2 0 0 0 0 failed
[2011-01-31 14:20:27 12024] ERROR (XendDomain:1175) Restore failed
Traceback (most recent call last):
  File
"/usr/local/lib/python2.6/dist-packages/xen/xend/XendDomain.py", line
1159, in domain_restore_fd
    dominfo = XendCheckpoint.restore(self, fd, paused=paused,
relocating=relocating)
  File
"/usr/local/lib/python2.6/dist-packages/xen/xend/XendCheckpoint.py",
line 358, in restore
    raise exn
XendError: /usr/lib/xen/bin/xc_restore 16 8 1 2 0 0 0 0 failed


Hitesh Khandelwal | Graduate Student | Purdue University |
http://www.hiteshkhandelwal.com <http://hiteshkhandelwal.com/>


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Xen users - Jan 2011 - Live migration fails (ERROR Restore failed)

[Xen-users] Live migration fails (ERROR Restore failed)
