Hitesh Khandelwal
2011-Jan-31 19:28 UTC
[Xen-users] Live migration fails (ERROR Restore failed)
I compiled xen-4.0.1 (kernel 2.6.32.27) on Ubuntu 10.10 as Dom0. Ubuntu 10.10 with stock kernel is used as DomU. I tried live migration by following instructions on http://www.virtuatopia.com/index.php/Migrating_Xen_domainU_Guests_Between_Host_Systems , but it failed with the following logs. Both the machines have same configuration and same amount of memory, and there is plenty of memory available on target host. VM guest runs successfully on both the source and target hosts individually. Relevant bug report: http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1653 Thanks, Hitesh Log on source Dom0: [2011-01-31 14:20:24 1008] DEBUG (XendCheckpoint:124) [xc_save]: /usr/lib/xen/bin/xc_save 46 10 0 0 1 [2011-01-31 14:20:24 1008] INFO (XendCheckpoint:423) /usr/lib/xen/bin/xc_save: symbol lookup error: /usr/lib/xen/bin/xc_save: undefined symbol: xs_suspend_evtchn_port [2011-01-31 14:20:24 1008] ERROR (XendCheckpoint:178) Save failed on domain cloud-vm01 (10) - resuming. Traceback (most recent call last): File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendCheckpoint.py", line 146, in save forkHelper(cmd, fd, saveInputHandler, False) File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendCheckpoint.py", line 409, in forkHelper raise XendError("%s failed: popen failed" % string.join(cmd)) XendError: /usr/lib/xen/bin/xc_save 46 10 0 0 1 failed: popen failed [2011-01-31 14:20:24 1008] DEBUG (XendDomainInfo:3117) XendDomainInfo.resumeDomain(10) [2011-01-31 14:20:27 1008] ERROR (xmlrpclib2:178) Internal error handling xend.domain.migrate Traceback (most recent call last): File "/usr/local/lib/python2.6/dist-packages/xen/util/xmlrpclib2.py", line 131, in _marshaled_dispatch response = self._dispatch(method, params) File "/usr/lib/python2.6/SimpleXMLRPCServer.py", line 418, in _dispatch return func(*params) File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendDomain.py", line 1366, in domain_migrate self._domain_migrate(dominfo, dst, live, port, node) File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendDomain.py", line 1458, in _domain_migrate dsterr = sock.recv(1024) timeout: timed out Log on target Dom0: [2011-01-31 14:20:23 12024] DEBUG (XendDomainInfo:235) XendDomainInfo.restore([\047domain\047, [\047domid\047, \04710\047], [\047cpu_weight\047, \047256\047], [\047cpu_cap\047, \0470\047], [\047bootloader\047, \047/usr/bin/pygrub\047], [\047on_crash\047, \047restart\047], [\047uuid\047, \047f32d195c-1fed-a9ad-5d7e-ff0456f79505\047], [\047bootloader_args\047], [\047vcpus\047, \0471\047], [\047description\047], [\047name\047, \047cloud-vm01\047], [\047cpus\047, [[]]], [\047on_reboot\047, \047restart\047], [\047on_poweroff\047, \047destroy\047], [\047maxmem\047, \047512\047], [\047memory\047, \047512\047], [\047shadow_memory\047, \0470\047], [\047vcpu_avail\047, \0471\047], [\047features\047], [\047on_xend_start\047, \047ignore\047], [\047on_xend_stop\047, \047ignore\047], [\047start_time\047, \0471296428689.01\047], [\047cpu_time\047, \04711.77212655\047], [\047online_vcpus\047, \0471\047], [\047image\047, [\047linux\047, [\047kernel\047], [\047args\047, \047root=/dev/sda2 ro \047], [\047superpages\047, \0470\047], [\047videoram\047, \0474\047], [\047pci\047, []], [\047nomigrate\047, \0470\047], [\047tsc_mode\047, \0470\047], [\047notes\047, [\047HV_START_LOW\047, \0474118806528\047], [\047FEATURES\047, \047!writable_page_tables|pae_pgdir_above_4gb\047], [\047VIRT_BASE\047, \0473221225472\047], [\047GUEST_VERSION\047, \0472.6\047], [\047PADDR_OFFSET\047, \0470\047], [\047GUEST_OS\047, \047linux\047], [\047HYPERCALL_PAGE\047, \0473222282240\047], [\047LOADER\047, \047generic\047], [\047SUSPEND_CANCEL\047, \0471\047], [\047PAE_MODE\047, \047yes\047], [\047ENTRY\047, \0473229974528\047], [\047XEN_VERSION\047, \047xen-3.0\047]]]], [\047status\047, \0472\047], [\047state\047, \047-b----\047], [\047store_mfn\047, \0472262985\047], [\047console_mfn\047, \0472262984\047], [\047device\047, [\047vif\047, [\047bridge\047, \047eth2\047], [\047uuid\047, \0477c22b250-2997-9b91-e6f0-2ad17a7e00db\047], [\047script\047, \047/etc/xen/scripts/vif-bridge\047], [\047ip\047, \047128.10.130.186\047], [\047mac\047, \04700:16:3E:C3:D2:A8\047], [\047backend\047, \0470\047]]], [\047device\047, [\047console\047, [\047protocol\047, \047vt100\047], [\047location\047, \0472\047], [\047uuid\047, \047b250cf24-1551-eaa0-cfdf-ea14e0d47934\047]]], [\047device\047, [\047vbd\047, [\047protocol\047, \047x86_32-abi\047], [\047uuid\047, \047f522f311-68bf-4358-216d-b8f81488a31b\047], [\047bootable\047, \0471\047], [\047dev\047, \047sda2:disk\047], [\047uname\047, \047file:/home/hkhande/xen-image/domains/cloud-vm01/disk.img\047], [\047mode\047, \047w\047], [\047backend\047, \0470\047], [\047VDI\047]]], [\047device\047, [\047vbd\047, [\047protocol\047, \047x86_32-abi\047], [\047uuid\047, \047a7f4ba1e-84e7-f104-a5f3-cd607b83a09f\047], [\047bootable\047, \0470\047], [\047dev\047, \047sda1:disk\047], [\047uname\047, \047file:/home/hkhande/xen-image/domains/cloud-vm01/swap.img\047], [\047mode\047, \047w\047], [\047backend\047, \0470\047], [\047VDI\047]]], [\047change_home_server\047, \047False\047]]) [2011-01-31 14:20:23 12024] DEBUG (XendDomainInfo:2508) XendDomainInfo.constructDomain [2011-01-31 14:20:23 12024] DEBUG (balloon:220) Balloon: 1125176 KiB free; need 16384; done. [2011-01-31 14:20:23 12024] DEBUG (XendDomain:464) Adding Domain: 8 [2011-01-31 14:20:23 12024] DEBUG (XendDomainInfo:3400) Storing VM details: {\047on_xend_stop\047: \047ignore\047, \047shadow_memory\047: \0470\047, \047uuid\047: \047f32d195c-1fed-a9ad-5d7e-ff0456f79505\047, \047on_reboot\047: \047restart\047, \047start_time\047: \0471296428689.01\047, \047on_poweroff\047: \047destroy\047, \047bootloader_args\047: \047\047, \047on_xend_start\047: \047ignore\047, \047on_crash\047: \047restart\047, \047xend/restart_count\047: \0470\047, \047vcpus\047: \0471\047, \047vcpu_avail\047: \0471\047, \047bootloader\047: \047/usr/bin/pygrub\047, \047image\047: "(linux (kernel ) (args \047root=/dev/sda2 ro \047) (superpages 0) (videoram 4) (pci ()) (nomigrate 0) (tsc_mode 0) (notes (HV_START_LOW 4118806528) (FEATURES \047!writable_page_tables|pae_pgdir_above_4gb\047) (VIRT_BASE 3221225472) (GUEST_VERSION 2.6) (PADDR_OFFSET 0) (GUEST_OS linux) (HYPERCALL_PAGE 3222282240) (LOADER generic) (SUSPEND_CANCEL 1) (PAE_MODE yes) (ENTRY 3229974528) (XEN_VERSION xen-3.0)))", \047name\047: \047cloud-vm01\047} [2011-01-31 14:20:23 12024] INFO (XendDomainInfo:2367) createDevice: console : {\047protocol\047: \047vt100\047, \047location\047: \0472\047, \047uuid\047: \047b250cf24-1551-eaa0-cfdf-ea14e0d47934\047} [2011-01-31 14:20:23 12024] DEBUG (DevController:95) DevController: writing {\047state\047: \0471\047, \047backend-id\047: \0470\047, \047backend\047: \047/local/domain/0/backend/console/8/0\047} to /local/domain/8/device/console/0. [2011-01-31 14:20:23 12024] DEBUG (DevController:97) DevController: writing {\047domain\047: \047cloud-vm01\047, \047frontend\047: \047/local/domain/8/device/console/0\047, \047uuid\047: \047b250cf24-1551-eaa0-cfdf-ea14e0d47934\047, \047frontend-id\047: \0478\047, \047state\047: \0471\047, \047location\047: \0472\047, \047online\047: \0471\047, \047protocol\047: \047vt100\047} to /local/domain/0/backend/console/8/0. [2011-01-31 14:20:23 12024] INFO (XendDomainInfo:2367) createDevice: vbd : {\047protocol\047: \047x86_32-abi\047, \047uuid\047: \047f522f311-68bf-4358-216d-b8f81488a31b\047, \047bootable\047: 1, \047driver\047: \047paravirtualised\047, \047dev\047: \047sda2:disk\047, \047uname\047: \047file:/home/hkhande/xen-image/domains/cloud-vm01/disk.img\047, \047mode\047: \047w\047, \047backend\047: \0470\047} [2011-01-31 14:20:23 12024] DEBUG (DevController:95) DevController: writing {\047virtual-device\047: \0472050\047, \047protocol\047: \047x86_32-abi\047, \047device-type\047: \047disk\047, \047backend-id\047: \0470\047, \047state\047: \0471\047, \047backend\047: \047/local/domain/0/backend/vbd/8/2050\047} to /local/domain/8/device/vbd/2050. [2011-01-31 14:20:23 12024] DEBUG (DevController:97) DevController: writing {\047domain\047: \047cloud-vm01\047, \047frontend\047: \047/local/domain/8/device/vbd/2050\047, \047uuid\047: \047f522f311-68bf-4358-216d-b8f81488a31b\047, \047bootable\047: \0471\047, \047dev\047: \047sda2\047, \047state\047: \0471\047, \047params\047: \047/home/hkhande/xen-image/domains/cloud-vm01/disk.img\047, \047mode\047: \047w\047, \047online\047: \0471\047, \047frontend-id\047: \0478\047, \047type\047: \047file\047} to /local/domain/0/backend/vbd/8/2050. [2011-01-31 14:20:24 12024] INFO (XendDomainInfo:2367) createDevice: vbd : {\047protocol\047: \047x86_32-abi\047, \047uuid\047: \047a7f4ba1e-84e7-f104-a5f3-cd607b83a09f\047, \047bootable\047: 0, \047driver\047: \047paravirtualised\047, \047dev\047: \047sda1:disk\047, \047uname\047: \047file:/home/hkhande/xen-image/domains/cloud-vm01/swap.img\047, \047mode\047: \047w\047, \047backend\047: \0470\047} [2011-01-31 14:20:24 12024] DEBUG (DevController:95) DevController: writing {\047virtual-device\047: \0472049\047, \047protocol\047: \047x86_32-abi\047, \047device-type\047: \047disk\047, \047backend-id\047: \0470\047, \047state\047: \0471\047, \047backend\047: \047/local/domain/0/backend/vbd/8/2049\047} to /local/domain/8/device/vbd/2049. [2011-01-31 14:20:24 12024] DEBUG (DevController:97) DevController: writing {\047domain\047: \047cloud-vm01\047, \047frontend\047: \047/local/domain/8/device/vbd/2049\047, \047uuid\047: \047a7f4ba1e-84e7-f104-a5f3-cd607b83a09f\047, \047bootable\047: \0470\047, \047dev\047: \047sda1\047, \047state\047: \0471\047, \047params\047: \047/home/hkhande/xen-image/domains/cloud-vm01/swap.img\047, \047mode\047: \047w\047, \047online\047: \0471\047, \047frontend-id\047: \0478\047, \047type\047: \047file\047} to /local/domain/0/backend/vbd/8/2049. [2011-01-31 14:20:24 12024] INFO (XendDomainInfo:2367) createDevice: vif : {\047bridge\047: \047eth2\047, \047uuid\047: \0477c22b250-2997-9b91-e6f0-2ad17a7e00db\047, \047script\047: \047/etc/xen/scripts/vif-bridge\047, \047ip\047: \047128.10.130.186\047, \047mac\047: \04700:16:3E:C3:D2:A8\047, \047backend\047: \0470\047} [2011-01-31 14:20:24 12024] DEBUG (DevController:95) DevController: writing {\047backend-id\047: \0470\047, \047mac\047: \04700:16:3E:C3:D2:A8\047, \047handle\047: \0470\047, \047state\047: \0471\047, \047backend\047: \047/local/domain/0/backend/vif/8/0\047} to /local/domain/8/device/vif/0. [2011-01-31 14:20:24 12024] DEBUG (DevController:97) DevController: writing {\047bridge\047: \047eth2\047, \047domain\047: \047cloud-vm01\047, \047handle\047: \0470\047, \047uuid\047: \0477c22b250-2997-9b91-e6f0-2ad17a7e00db\047, \047script\047: \047/etc/xen/scripts/vif-bridge\047, \047ip\047: \047128.10.130.186\047, \047mac\047: \04700:16:3E:C3:D2:A8\047, \047frontend-id\047: \0478\047, \047state\047: \0471\047, \047online\047: \0471\047, \047frontend\047: \047/local/domain/8/device/vif/0\047} to /local/domain/0/backend/vif/8/0. [2011-01-31 14:20:24 12024] DEBUG (XendDomainInfo:1804) Storing domain details: {\047image/entry\047: \0473229974528\047, \047console/port\047: \0472\047, \047image/loader\047: \047generic\047, \047vm\047: \047/vm/f32d195c-1fed-a9ad-5d7e-ff0456f79505\047, \047control/platform-feature-multiprocessor-suspend\047: \0471\047, \047image/hv-start-low\047: \0474118806528\047, \047image/guest-os\047: \047linux\047, \047image/virt-base\047: \0473221225472\047, \047memory/target\047: \047524288\047, \047image/guest-version\047: \0472.6\047, \047image/pae-mode\047: \047yes\047, \047description\047: \047\047, \047console/limit\047: \0471048576\047, \047image/paddr-offset\047: \0470\047, \047image/hypercall-page\047: \0473222282240\047, \047image/suspend-cancel\047: \0471\047, \047cpu/0/availability\047: \047online\047, \047image/features/pae-pgdir-above-4gb\047: \0471\047, \047image/features/writable-page-tables\047: \0470\047, \047console/type\047: \047xenconsoled\047, \047name\047: \047cloud-vm01\047, \047domid\047: \0478\047, \047image/xen-version\047: \047xen-3.0\047, \047store/port\047: \0471\047} [2011-01-31 14:20:24 12024] DEBUG (XendCheckpoint:286) restore:shadow=0x0, _static_max=0x20000000, _static_min=0x0, [2011-01-31 14:20:24 12024] DEBUG (XendCheckpoint:305) [xc_restore]: /usr/lib/xen/bin/xc_restore 16 8 1 2 0 0 0 0 [2011-01-31 14:20:26 12024] INFO (XendCheckpoint:423) ERROR Internal error: read: p2m_size [2011-01-31 14:20:26 12024] INFO (XendCheckpoint:423) Restore exit with rc=1 [2011-01-31 14:20:26 12024] DEBUG (XendDomainInfo:3053) XendDomainInfo.destroy: domid=8 [2011-01-31 14:20:26 12024] ERROR (XendDomainInfo:3067) XendDomainInfo.destroy: domain destruction failed. Traceback (most recent call last): File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendDomainInfo.py", line 3060, in destroy xc.domain_pause(self.domid) Error: (3, \047No such process\047) [2011-01-31 14:20:26 12024] DEBUG (XendDomainInfo:2416) No device model [2011-01-31 14:20:26 12024] DEBUG (XendDomainInfo:2418) Releasing devices [2011-01-31 14:20:26 12024] DEBUG (XendDomainInfo:2424) Removing vif/0 [2011-01-31 14:20:26 12024] DEBUG (XendDomainInfo:1286) XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/0 [2011-01-31 14:20:26 12024] DEBUG (XendDomainInfo:2424) Removing console/0 [2011-01-31 14:20:26 12024] DEBUG (XendDomainInfo:1286) XendDomainInfo.destroyDevice: deviceClass = console, device = console/0 [2011-01-31 14:20:27 12024] DEBUG (XendDomainInfo:2424) Removing vbd/2050 [2011-01-31 14:20:27 12024] DEBUG (XendDomainInfo:1286) XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/2050 [2011-01-31 14:20:27 12024] DEBUG (XendDomainInfo:2424) Removing vbd/2049 [2011-01-31 14:20:27 12024] DEBUG (XendDomainInfo:1286) XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/2049 [2011-01-31 14:20:27 12024] INFO (XendDomain:1107) Domain cloud-vm01 (f32d195c-1fed-a9ad-5d7e-ff0456f79505) deleted. [2011-01-31 14:20:27 12024] ERROR (XendCheckpoint:357) /usr/lib/xen/bin/xc_restore 16 8 1 2 0 0 0 0 failed Traceback (most recent call last): File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendCheckpoint.py", line 309, in restore forkHelper(cmd, fd, handler.handler, True) File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendCheckpoint.py", line 411, in forkHelper raise XendError("%s failed" % string.join(cmd)) XendError: /usr/lib/xen/bin/xc_restore 16 8 1 2 0 0 0 0 failed [2011-01-31 14:20:27 12024] ERROR (XendDomain:1175) Restore failed Traceback (most recent call last): File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendDomain.py", line 1159, in domain_restore_fd dominfo = XendCheckpoint.restore(self, fd, paused=paused, relocating=relocating) File "/usr/local/lib/python2.6/dist-packages/xen/xend/XendCheckpoint.py", line 358, in restore raise exn XendError: /usr/lib/xen/bin/xc_restore 16 8 1 2 0 0 0 0 failed Hitesh Khandelwal | Graduate Student | Purdue University | http://www.hiteshkhandelwal.com <http://hiteshkhandelwal.com/> _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users