I have been having a issue recently with the xm migrate feature. It seems to me that the migration begins to take place however somewhere in the middle of the migration things seem to fail. I run this command.. root@xen900:/xen-images# xm migrate --live feisty_base 192.168.0.1 and to check and see if the system has made the connection I did root@xen901:/var/log/xen# netstat -an|grep 8002 tcp 0 0 0.0.0.0:8002 0.0.0.0:* LISTEN tcp 0 0 192.168.0.1:8002 192.168.0.2:60671 ESTABLISHED this is from the other xen host. after a few moments the xm migrate command completes. Reviewing the logs it seems to be a issue on the receiving host. I am running the host machines with ubuntu 7.04. there are a few logs on both machines.. from xen900 /var/log/xen/xen-hotplug.log reads> Nothing to flush. > Nothing to flush. > Nothing to flush. > Nothing to flush. > SIOCGIFFLAGS: No such device > xenstore-read: couldn''t read path backend/vbd/5/769/node > xenstore-read: couldn''t read path backend/vbd/5/770/node > Nothing to flush. > Nothing to flush. > xenstore-read: couldn''t read path backend/vbd/7/769/node > xenstore-read: couldn''t read path backend/vbd/7/770/node > Nothing to flush. > Nothing to flush. > xenstore-read: couldn''t read path backend/vbd/9/770/node > xenstore-read: couldn''t read path backend/vbd/9/769/node > Nothing to flush. > Nothing to flush.(xen900)/var/xen/xend.log reads [2007-09-20 10:16:32 xend 15789] INFO (XendCheckpoint:239) 4: sent 81, skipped 0, delta 9ms, dom0 100%, target 0%, sent 294Mb/s, dirtied 294Mb/s $ [2007-09-20 10:16:32 xend 15789] INFO (XendCheckpoint:239) Total pages sent= 130884 (0.98x) [2007-09-20 10:16:32 xend 15789] INFO (XendCheckpoint:239) (of which 0 were fixups) [2007-09-20 10:16:32 xend 15789] INFO (XendCheckpoint:239) All memory is saved [2007-09-20 10:16:32 xend 15789] INFO (XendCheckpoint:239) Save exit rc=0 [2007-09-20 10:16:32 xend.XendDomainInfo 15789] DEBUG (XendDomainInfo:1466) XendDomainInfo.destroyDomain(11) [2007-09-20 10:16:32 xend.XendDomainInfo 15789] INFO (XendDomainInfo:1564) Dev 769 still active, looping... [2007-09-20 10:16:32 xend.XendDomainInfo 15789] INFO (XendDomainInfo:1564) Dev 770 still active, looping... [2007-09-20 10:16:32 xend.XendDomainInfo 15789] INFO (XendDomainInfo:1564) Dev 769 still active, looping... [2007-09-20 10:16:32 xend.XendDomainInfo 15789] INFO (XendDomainInfo:1564) Dev 770 still active, looping... [2007-09-20 10:16:32 xend.XendDomainInfo 15789] INFO (XendDomainInfo:1564) Dev 769 still active, looping... [2007-09-20 10:16:32 xend.XendDomainInfo 15789] INFO (XendDomainInfo:1564) Dev 770 still active, looping... (xen900)/var/xen/xend-debug.log reads(these logs are not timestamped so I am not sure if they are from this run) Traceback (most recent call last): File "SocketServer.py", line 464, in process_request_thread self.finish_request(request, client_address) File "SocketServer.py", line 254, in finish_request self.RequestHandlerClass(request, client_address, self) File "SocketServer.py", line 522, in __init__ self.handle() File "BaseHTTPServer.py", line 316, in handle self.handle_one_request() File "BaseHTTPServer.py", line 310, in handle_one_request method() File "/usr/lib/python2.5/site-packages/xen/util/xmlrpclib2.py", line 66, in do_POST self.send_response(200) File "BaseHTTPServer.py", line 368, in send_response (self.protocol_version, code, message)) File "socket.py", line 262, in write self.flush() File "socket.py", line 249, in flush self._sock.sendall(buffer) error: (32, ''Broken pipe'') on xen901 /var/xen/xend.log reads [2007-09-20 10:15:45 xend.XendDomainInfo 6492] DEBUG (XendDomainInfo:265) XendDomainInfo.restore([''domain'', [''domid'', ''11''], [''uuid'', ''9e934f3b-1$ [2007-09-20 10:15:45 xend.XendDomainInfo 6492] DEBUG (XendDomainInfo:296) parseConfig: config is [''domain'', [''domid'', ''11''], [''uuid'', ''9e934f3b-1$ [2007-09-20 10:15:45 xend.XendDomainInfo 6492] DEBUG (XendDomainInfo:395) parseConfig: result is {''shadow_memory'': 0, ''uuid'': ''9e934f3b-10b2-a40f$ [2007-09-20 10:15:45 xend.XendDomainInfo 6492] DEBUG (XendDomainInfo:1262) XendDomainInfo.construct: None [2007-09-20 10:15:45 xend.XendDomainInfo 6492] DEBUG (XendDomainInfo:713) Storing VM details: {''shadow_memory'': ''0'', ''uuid'': ''9e934f3b-10b2-a40f-$ [2007-09-20 10:15:45 xend 6492] DEBUG (DevController:110) DevController: writing {''backend-id'': ''0'', ''mac'': ''00:16:3e:5b:e6:66'', ''handle'': ''0'', ''$ [2007-09-20 10:15:45 xend 6492] DEBUG (DevController:112) DevController: writing {''bridge'': ''xenbr0'', ''domain'': ''feisty_base'', ''handle'': ''0'', ''sc$ [2007-09-20 10:15:45 xend 6492] DEBUG (blkif:24) exception looking up device number for hda1: [Errno 2] No such file or directory: ''/dev/hda1'' [2007-09-20 10:15:45 xend 6492] DEBUG (DevController:110) DevController: writing {''backend-id'': ''0'', ''virtual-device'': ''769'', ''device-type'': ''dis$ [2007-09-20 10:15:45 xend 6492] DEBUG (DevController:112) DevController: writing {''domain'': ''feisty_base'', ''frontend'': ''/local/domain/6/device/vb$ [2007-09-20 10:15:45 xend 6492] DEBUG (blkif:24) exception looking up device number for hda2: [Errno 2] No such file or directory: ''/dev/hda2'' [2007-09-20 10:15:45 xend 6492] DEBUG (DevController:110) DevController: writing {''backend-id'': ''0'', ''virtual-device'': ''770'', ''device-type'': ''dis$ [2007-09-20 10:15:45 xend 6492] DEBUG (DevController:112) DevController: writing {''domain'': ''feisty_base'', ''frontend'': ''/local/domain/6/device/vb$ [2007-09-20 10:15:45 xend.XendDomainInfo 6492] DEBUG (XendDomainInfo:748) Storing domain details: {''console/port'': ''2'', ''name'': ''feisty_base'', ''c$ [2007-09-20 10:15:45 xend 6492] DEBUG (balloon:127) Balloon: 533004 KiB free; need 532480; done. [2007-09-20 10:15:45 xend 6492] DEBUG (XendCheckpoint:155) [xc_restore]: /usr/lib/xen/bin/xc_restore 18 6 133120 1 2 [2007-09-20 10:15:45 xend 6492] INFO (XendCheckpoint:239) xc_linux_restore start: max_pfn = 20800 [2007-09-20 10:15:45 xend 6492] INFO (XendCheckpoint:239) Increased domain reservation by 82000 KB [2007-09-20 10:15:45 xend 6492] INFO (XendCheckpoint:239) Reloading memory pages: 0% [2007-09-20 10:16:30 xend 6492] INFO (XendCheckpoint:239) Received all pages (0 races) [2007-09-20 10:16:30 xend 6492] INFO (XendCheckpoint:239) ^H^H^H^H100% [2007-09-20 10:16:30 xend 6492] INFO (XendCheckpoint:239) Memory reloaded. [2007-09-20 10:16:30 xend 6492] INFO (XendCheckpoint:239) Decreased reservation by 2409 pages [2007-09-20 10:16:30 xend 6492] INFO (XendCheckpoint:239) Domain ready to be built. [2007-09-20 10:16:30 xend 6492] INFO (XendCheckpoint:239) Restore exit with rc=0 [2007-09-20 10:16:30 xend 6492] DEBUG (XendCheckpoint:209) store-mfn 565003 [2007-09-20 10:16:30 xend 6492] DEBUG (XendCheckpoint:209) console-mfn 561602 [2007-09-20 10:16:31 xend 6492] DEBUG (DevController:143) Waiting for devices vif. [2007-09-20 10:16:31 xend 6492] DEBUG (DevController:149) Waiting for 0. [2007-09-20 10:16:31 xend 6492] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vif/6/0/hotplug-status. [2007-09-20 10:16:31 xend 6492] DEBUG (DevController:478) hotplugStatusCallback 1. [2007-09-20 10:16:31 xend 6492] DEBUG (DevController:143) Waiting for devices usb. [2007-09-20 10:16:31 xend 6492] DEBUG (DevController:143) Waiting for devices vbd. [2007-09-20 10:16:31 xend 6492] DEBUG (DevController:149) Waiting for 769. [2007-09-20 10:16:31 xend 6492] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vbd/6/769/hotplug-status. [2007-09-20 10:16:31 xend 6492] DEBUG (DevController:478) hotplugStatusCallback 2. [2007-09-20 10:16:31 xend.XendDomainInfo 6492] DEBUG (XendDomainInfo:1458) XendDomainInfo.destroy: domid=6 [2007-09-20 10:16:31 xend.XendDomainInfo 6492] DEBUG (XendDomainInfo:1466) XendDomainInfo.destroyDomain(6) [2007-09-20 10:16:31 xend 6492] ERROR (__init__:1021) Restore failed Traceback (most recent call last): File "/usr/lib/python2.5/site-packages/xen/xend/XendDomain.py", line 263, in domain_restore_fd return XendCheckpoint.restore(self, fd) File "/usr/lib/python2.5/site-packages/xen/xend/XendCheckpoint.py", line 165, in restore dominfo.waitForDevices() # Wait for backends to set up File "/usr/lib/python2.5/site-packages/xen/xend/XendDomainInfo.py", line 1605, in waitForDevices self.waitForDevices_(c) File "/usr/lib/python2.5/site-packages/xen/xend/XendDomainInfo.py", line 1099, in waitForDevices_ return self.getDeviceController(deviceClass).waitForDevices() File "/usr/lib/python2.5/site-packages/xen/xend/server/DevController.py", line 145, in waitForDevices return map(self.waitForDevice, self.deviceIDs()) File "/usr/lib/python2.5/site-packages/xen/xend/server/DevController.py", line 163, in waitForDevice (devid, self.deviceClass)) VmError: Device 769 (vbd) could not be connected. Backend device not found. there is nothing notable in the other logs on xen901. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
well I think I figured it out. It was a matter of having the xen images in different directories on the different hosts. for some reason it did not like it if the directories were: /xen-images/xen-image.img on xen900 and /var/xen/xen-image.img on xen901 once I created similar directories on both machines live migration worked like a charm. IE /var/xen/xen-image.img on both xen901 and xen900 thanks for xen - austin Mark Nielsen wrote:> looks to me like the destination can''t see the guest''s storage device > > [2007-09-20 10:15:45 xend 6492] DEBUG (blkif:24) exception looking up > device number for hda1: [Errno 2] No such file or directory: ''/dev/hda1'' > > Mark > > > > austin trask wrote: >> I have been having a issue recently with the xm migrate feature. It >> seems to me that the migration begins to take place however somewhere >> in the middle of the migration things seem to fail. >> I run this command.. >> root@xen900:/xen-images# xm migrate --live feisty_base 192.168.0.1 >> >> >> and to check and see if the system has made the connection I did >> root@xen901:/var/log/xen# netstat -an|grep 8002 >> tcp 0 0 0.0.0.0:8002 0.0.0.0:* >> LISTEN tcp 0 0 192.168.0.1:8002 192.168.0.2:60671 >> ESTABLISHED >> this is from the other xen host. >> >> after a few moments the xm migrate command completes. Reviewing the >> logs it seems to be a issue on the receiving host. I am running the >> host machines with ubuntu 7.04. >> there are a few logs on both machines.. from xen900 >> /var/log/xen/xen-hotplug.log reads >>> Nothing to flush. >>> Nothing to flush. >>> Nothing to flush. >>> Nothing to flush. >>> SIOCGIFFLAGS: No such device >>> xenstore-read: couldn''t read path backend/vbd/5/769/node >>> xenstore-read: couldn''t read path backend/vbd/5/770/node >>> Nothing to flush. >>> Nothing to flush. >>> xenstore-read: couldn''t read path backend/vbd/7/769/node >>> xenstore-read: couldn''t read path backend/vbd/7/770/node >>> Nothing to flush. >>> Nothing to flush. >>> xenstore-read: couldn''t read path backend/vbd/9/770/node >>> xenstore-read: couldn''t read path backend/vbd/9/769/node >>> Nothing to flush. >>> Nothing to flush. >> (xen900)/var/xen/xend.log reads >> >> [2007-09-20 10:16:32 xend 15789] INFO (XendCheckpoint:239) 4: sent >> 81, skipped 0, delta 9ms, dom0 100%, target 0%, sent 294Mb/s, dirtied >> 294Mb/s $ >> [2007-09-20 10:16:32 xend 15789] INFO (XendCheckpoint:239) Total >> pages sent= 130884 (0.98x) >> [2007-09-20 10:16:32 xend 15789] INFO (XendCheckpoint:239) (of which >> 0 were fixups) >> [2007-09-20 10:16:32 xend 15789] INFO (XendCheckpoint:239) All memory >> is saved >> [2007-09-20 10:16:32 xend 15789] INFO (XendCheckpoint:239) Save exit >> rc=0 >> [2007-09-20 10:16:32 xend.XendDomainInfo 15789] DEBUG >> (XendDomainInfo:1466) XendDomainInfo.destroyDomain(11) >> [2007-09-20 10:16:32 xend.XendDomainInfo 15789] INFO >> (XendDomainInfo:1564) Dev 769 still active, looping... >> [2007-09-20 10:16:32 xend.XendDomainInfo 15789] INFO >> (XendDomainInfo:1564) Dev 770 still active, looping... >> [2007-09-20 10:16:32 xend.XendDomainInfo 15789] INFO >> (XendDomainInfo:1564) Dev 769 still active, looping... >> [2007-09-20 10:16:32 xend.XendDomainInfo 15789] INFO >> (XendDomainInfo:1564) Dev 770 still active, looping... >> [2007-09-20 10:16:32 xend.XendDomainInfo 15789] INFO >> (XendDomainInfo:1564) Dev 769 still active, looping... >> [2007-09-20 10:16:32 xend.XendDomainInfo 15789] INFO >> (XendDomainInfo:1564) Dev 770 still active, looping... >> >> >> (xen900)/var/xen/xend-debug.log reads(these logs are not timestamped >> so I am not sure if they are from this run) >> >> Traceback (most recent call last): >> File "SocketServer.py", line 464, in process_request_thread >> self.finish_request(request, client_address) >> File "SocketServer.py", line 254, in finish_request >> self.RequestHandlerClass(request, client_address, self) >> File "SocketServer.py", line 522, in __init__ >> self.handle() >> File "BaseHTTPServer.py", line 316, in handle >> self.handle_one_request() >> File "BaseHTTPServer.py", line 310, in handle_one_request >> method() >> File "/usr/lib/python2.5/site-packages/xen/util/xmlrpclib2.py", line >> 66, in do_POST >> self.send_response(200) >> File "BaseHTTPServer.py", line 368, in send_response >> (self.protocol_version, code, message)) >> File "socket.py", line 262, in write >> self.flush() >> File "socket.py", line 249, in flush >> self._sock.sendall(buffer) >> error: (32, ''Broken pipe'') >> >> >> >> on xen901 /var/xen/xend.log reads >> >> [2007-09-20 10:15:45 xend.XendDomainInfo 6492] DEBUG >> (XendDomainInfo:265) XendDomainInfo.restore([''domain'', [''domid'', >> ''11''], [''uuid'', ''9e934f3b-1$ >> [2007-09-20 10:15:45 xend.XendDomainInfo 6492] DEBUG >> (XendDomainInfo:296) parseConfig: config is [''domain'', [''domid'', >> ''11''], [''uuid'', ''9e934f3b-1$ >> [2007-09-20 10:15:45 xend.XendDomainInfo 6492] DEBUG >> (XendDomainInfo:395) parseConfig: result is {''shadow_memory'': 0, >> ''uuid'': ''9e934f3b-10b2-a40f$ >> [2007-09-20 10:15:45 xend.XendDomainInfo 6492] DEBUG >> (XendDomainInfo:1262) XendDomainInfo.construct: None >> [2007-09-20 10:15:45 xend.XendDomainInfo 6492] DEBUG >> (XendDomainInfo:713) Storing VM details: {''shadow_memory'': ''0'', >> ''uuid'': ''9e934f3b-10b2-a40f-$ >> [2007-09-20 10:15:45 xend 6492] DEBUG (DevController:110) >> DevController: writing {''backend-id'': ''0'', ''mac'': >> ''00:16:3e:5b:e6:66'', ''handle'': ''0'', ''$ >> [2007-09-20 10:15:45 xend 6492] DEBUG (DevController:112) >> DevController: writing {''bridge'': ''xenbr0'', ''domain'': ''feisty_base'', >> ''handle'': ''0'', ''sc$ >> [2007-09-20 10:15:45 xend 6492] DEBUG (blkif:24) exception looking up >> device number for hda1: [Errno 2] No such file or directory: ''/dev/hda1'' >> [2007-09-20 10:15:45 xend 6492] DEBUG (DevController:110) >> DevController: writing {''backend-id'': ''0'', ''virtual-device'': ''769'', >> ''device-type'': ''dis$ >> [2007-09-20 10:15:45 xend 6492] DEBUG (DevController:112) >> DevController: writing {''domain'': ''feisty_base'', ''frontend'': >> ''/local/domain/6/device/vb$ >> [2007-09-20 10:15:45 xend 6492] DEBUG (blkif:24) exception looking up >> device number for hda2: [Errno 2] No such file or directory: ''/dev/hda2'' >> [2007-09-20 10:15:45 xend 6492] DEBUG (DevController:110) >> DevController: writing {''backend-id'': ''0'', ''virtual-device'': ''770'', >> ''device-type'': ''dis$ >> [2007-09-20 10:15:45 xend 6492] DEBUG (DevController:112) >> DevController: writing {''domain'': ''feisty_base'', ''frontend'': >> ''/local/domain/6/device/vb$ >> [2007-09-20 10:15:45 xend.XendDomainInfo 6492] DEBUG >> (XendDomainInfo:748) Storing domain details: {''console/port'': ''2'', >> ''name'': ''feisty_base'', ''c$ >> [2007-09-20 10:15:45 xend 6492] DEBUG (balloon:127) Balloon: 533004 >> KiB free; need 532480; done. >> [2007-09-20 10:15:45 xend 6492] DEBUG (XendCheckpoint:155) >> [xc_restore]: /usr/lib/xen/bin/xc_restore 18 6 133120 1 2 >> [2007-09-20 10:15:45 xend 6492] INFO (XendCheckpoint:239) >> xc_linux_restore start: max_pfn = 20800 >> [2007-09-20 10:15:45 xend 6492] INFO (XendCheckpoint:239) Increased >> domain reservation by 82000 KB >> [2007-09-20 10:15:45 xend 6492] INFO (XendCheckpoint:239) Reloading >> memory pages: 0% >> [2007-09-20 10:16:30 xend 6492] INFO (XendCheckpoint:239) Received >> all pages (0 races) >> [2007-09-20 10:16:30 xend 6492] INFO (XendCheckpoint:239) ^H^H^H^H100% >> [2007-09-20 10:16:30 xend 6492] INFO (XendCheckpoint:239) Memory >> reloaded. >> [2007-09-20 10:16:30 xend 6492] INFO (XendCheckpoint:239) Decreased >> reservation by 2409 pages >> [2007-09-20 10:16:30 xend 6492] INFO (XendCheckpoint:239) Domain >> ready to be built. >> [2007-09-20 10:16:30 xend 6492] INFO (XendCheckpoint:239) Restore >> exit with rc=0 >> [2007-09-20 10:16:30 xend 6492] DEBUG (XendCheckpoint:209) store-mfn >> 565003 >> [2007-09-20 10:16:30 xend 6492] DEBUG (XendCheckpoint:209) >> console-mfn 561602 >> [2007-09-20 10:16:31 xend 6492] DEBUG (DevController:143) Waiting for >> devices vif. >> [2007-09-20 10:16:31 xend 6492] DEBUG (DevController:149) Waiting for 0. >> [2007-09-20 10:16:31 xend 6492] DEBUG (DevController:464) >> hotplugStatusCallback /local/domain/0/backend/vif/6/0/hotplug-status. >> [2007-09-20 10:16:31 xend 6492] DEBUG (DevController:478) >> hotplugStatusCallback 1. >> [2007-09-20 10:16:31 xend 6492] DEBUG (DevController:143) Waiting for >> devices usb. >> [2007-09-20 10:16:31 xend 6492] DEBUG (DevController:143) Waiting for >> devices vbd. >> [2007-09-20 10:16:31 xend 6492] DEBUG (DevController:149) Waiting for >> 769. >> [2007-09-20 10:16:31 xend 6492] DEBUG (DevController:464) >> hotplugStatusCallback /local/domain/0/backend/vbd/6/769/hotplug-status. >> [2007-09-20 10:16:31 xend 6492] DEBUG (DevController:478) >> hotplugStatusCallback 2. >> [2007-09-20 10:16:31 xend.XendDomainInfo 6492] DEBUG >> (XendDomainInfo:1458) XendDomainInfo.destroy: domid=6 >> [2007-09-20 10:16:31 xend.XendDomainInfo 6492] DEBUG >> (XendDomainInfo:1466) XendDomainInfo.destroyDomain(6) >> [2007-09-20 10:16:31 xend 6492] ERROR (__init__:1021) Restore failed >> Traceback (most recent call last): >> File "/usr/lib/python2.5/site-packages/xen/xend/XendDomain.py", line >> 263, in domain_restore_fd >> return XendCheckpoint.restore(self, fd) >> File "/usr/lib/python2.5/site-packages/xen/xend/XendCheckpoint.py", >> line 165, in restore >> dominfo.waitForDevices() # Wait for backends to set up >> File "/usr/lib/python2.5/site-packages/xen/xend/XendDomainInfo.py", >> line 1605, in waitForDevices >> self.waitForDevices_(c) >> File "/usr/lib/python2.5/site-packages/xen/xend/XendDomainInfo.py", >> line 1099, in waitForDevices_ >> return self.getDeviceController(deviceClass).waitForDevices() >> File >> "/usr/lib/python2.5/site-packages/xen/xend/server/DevController.py", >> line 145, in waitForDevices >> return map(self.waitForDevice, self.deviceIDs()) >> File >> "/usr/lib/python2.5/site-packages/xen/xend/server/DevController.py", >> line 163, in waitForDevice >> (devid, self.deviceClass)) >> VmError: Device 769 (vbd) could not be connected. Backend device not >> found. >> >> >> there is nothing notable in the other logs on xen901. >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xensource.com >> http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users