Borja Sotomayor
2005-Dec-15 00:00 UTC
[Xen-users] Device 2049 (vbd) could not be connected. Backend device not found.
Hi everyone, I''m running into a weird problem when running Xen 3.0. If I try to create new domains that use a single file-backed VBD (e.g. ttylinux-xen or a custom Debian image), everything works fine. However, if I try to create a domain that requires several VBDs, I get the following error message: tp-c120:/vm/vm_images/vosg/GK # xm create -c osg-gk.config Using config file "osg-gk.config". Error: Device 2049 (vbd) could not be connected. Backend device not found. [The config file osg-gk.config is included at the end of this e-mail] Note that I get the same error message if I try to create a new domain with a custom Debian image (which works with a single VBD) and simply add a swap partition. Also, once I get this error, I can''t create more domains (single or multiple VBD). Along with the config file, I''m attaching the output from syslog, xend.log, and xm list --long. I noticed that the domain _does_ get created, but is placed in paused state. However, if I try to unpause it, the root partition misteriously disappears and the DomU kernel crashes (I''m also attaching the output of xm list --long after unpausing the domain) Finally, here''s some information on the machine and it''s software: * Machine: 1U IBM e1350 eServer cluster node * IBM e325 Opteron Model 248 (2.2 GHz/1MB L2), 2x512 MB * Operating system: Enterprise Suse Linux 9 (SLES9) Service Pack 2 tp-c120:/vm/vm_images/vosg/GK # rpm -q udev hotplug udev-021-36.49 hotplug-0.44-32.35 Note that I have only been able to reproduce this error on the above machine. I can run all my domains (ttylinux, Debian, osg-gk) fine in a different computer (a Thinkpad with Ubuntu Breezy Badger) Any help or suggestion would be appreciated :-) Cheers! -- Borja ========================= CONFIG FILE ================================= kernel = "/boot/vmlinuz-2.6.12-xenU" name = "osg-gk" memory = 396 cpu=0 disk = [ "file:/vm/vm_images/vosg/GK/osg-gk-base.img,sda1,w" , "file:/vm/vm_images/vosg/GK/swap1.img,sda2,w", "file:/vm/vm_images/vosg/GK/osg-gk-grid.img,sda3,w", "file:/vm/vm_images/vosg/GK/osg-gk-app.img,sda4,w"] nics=2 # public ip ip="XXX.YYY.67.232" netmask="255.255.254.0" network="XXX.YYY.66.0" broadcast="XXX.YYY.67.255" gateway="XXX.YYY.67.253" vif = [''mac=00:06:AA:F6:BB:B3'', ''mac=00:06:AA:F7:BB:B3''] # private ip # can''t set through the VM config file #ip="192.168.255.1" #netmask="255.255.255.0" #network="192.168.255.0" #broadcast="192.168.255.255" ##gateway="192.168.255.1" root = "/dev/sda1 ro" #extra = "4" ============================ SYSLOG ===================================Dec 14 15:44:34 tp-c120 kernel: <6>device vif0.0 entered promiscuous mode Dec 14 15:44:34 tp-c120 kernel: klogd 1.4.1, ---------- state change ---------- Dec 14 15:44:34 tp-c120 kernel: bridge: can''t decode speed from peth0: 0 Dec 14 15:44:34 tp-c120 kernel: device peth0 entered promiscuous mode Dec 14 15:44:34 tp-c120 kernel: xenbr0: port 1(vif0.0) entering learning state Dec 14 15:44:35 tp-c120 kernel: xenbr0: topology change detected, propagating Dec 14 15:44:35 tp-c120 kernel: xenbr0: port 1(vif0.0) entering forwarding state Dec 14 15:44:38 tp-c120 kernel: tg3: peth0: Link is up at 1000 Mbps, full duplex. Dec 14 15:44:38 tp-c120 kernel: tg3: peth0: Flow control is on for TX and on for RX. Dec 14 15:44:38 tp-c120 kernel: xenbr0: port 2(peth0) entering learning state Dec 14 15:44:38 tp-c120 kernel: xenbr0: topology change detected, propagating Dec 14 15:44:38 tp-c120 kernel: xenbr0: port 2(peth0) entering forwarding state Dec 14 15:44:39 tp-c120 ifup: No configuration found for xenbr0 Dec 14 15:44:54 tp-c120 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/1/2052 Dec 14 15:44:54 tp-c120 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/1/2051 Dec 14 15:44:54 tp-c120 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/1/2049 Dec 14 15:44:54 tp-c120 logger: /etc/xen/scripts/vif-bridge: online XENBUS_PATH=backend/vif/1/1 Dec 14 15:44:55 tp-c120 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/1/2050 Dec 14 15:44:55 tp-c120 logger: /etc/xen/scripts/vif-bridge: online XENBUS_PATH=backend/vif/1/0 Dec 14 15:44:55 tp-c120 kernel: device vif1.1 entered promiscuous mode Dec 14 15:44:55 tp-c120 kernel: xenbr0: port 3(vif1.1) entering learning state Dec 14 15:44:55 tp-c120 kernel: xenbr0: topology change detected, propagating Dec 14 15:44:55 tp-c120 kernel: xenbr0: port 3(vif1.1) entering forwarding state Dec 14 15:44:55 tp-c120 kernel: ip_tables: (C) 2000-2002 Netfilter core team Dec 14 15:44:56 tp-c120 logger: /etc/xen/scripts/vif-bridge: Successful vif-bridge online for vif1.1, bridge xenbr0. Dec 14 15:44:56 tp-c120 logger: /etc/xen/scripts/vif-bridge: Writing backend/vif/1/1/hotplug-status connected to xenstore. Dec 14 15:44:56 tp-c120 logger: /etc/xen/scripts/vif-bridge: Successful vif-bridge online for vif1.0, bridge xenbr0. Dec 14 15:44:56 tp-c120 logger: /etc/xen/scripts/vif-bridge: Writing backend/vif/1/0/hotplug-status connected to xenstore. Dec 14 15:44:56 tp-c120 kernel: device vif1.0 entered promiscuous mode Dec 14 15:44:56 tp-c120 kernel: xenbr0: port 4(vif1.0) entering learning state Dec 14 15:44:56 tp-c120 kernel: xenbr0: topology change detected, propagating Dec 14 15:44:56 tp-c120 kernel: xenbr0: port 4(vif1.0) entering forwarding state Dec 14 15:44:59 tp-c120 logger: /etc/xen/scripts/block: Forced to steal lock on /var/run/xen-hotplug/block from unknown! Dec 14 15:44:59 tp-c120 logger: /etc/xen/scripts/block: Forced to steal lock on /var/run/xen-hotplug/block from 12554: /etc/xen/scripts/block! Dec 14 15:44:59 tp-c120 logger: /etc/xen/scripts/block: Writing backend/vbd/1/2051/node /dev/loop0 to xenstore. Dec 14 15:44:59 tp-c120 ifup: No configuration found for vif1.0 Dec 14 15:44:59 tp-c120 logger: /etc/xen/scripts/block: Writing backend/vbd/1/2051/physical-device 7:0 to xenstore. Dec 14 15:44:59 tp-c120 logger: /etc/xen/scripts/block: Writing backend/vbd/1/2051/hotplug-status connected to xenstore. Dec 14 15:44:59 tp-c120 logger: /etc/xen/scripts/block: Writing backend/vbd/1/2052/hotplug-status error to xenstore. Dec 14 15:44:59 tp-c120 logger: /etc/xen/scripts/block: losetup /dev/loop0 /vm/vm_images/vosg/GK/osg-gk-app.img failed Dec 14 15:45:00 tp-c120 ifup: No configuration found for vif1.1 Dec 14 15:45:00 tp-c120 logger: /etc/xen/scripts/block: Forced to steal lock on /var/run/xen-hotplug/block from unknown! Dec 14 15:45:00 tp-c120 logger: /etc/xen/scripts/block: Forced to steal lock on /var/run/xen-hotplug/block from 12453: /etc/xen/scripts/block! Dec 14 15:45:00 tp-c120 logger: /etc/xen/scripts/block: Writing backend/vbd/1/2050/node /dev/loop1 to xenstore. Dec 14 15:45:00 tp-c120 logger: /etc/xen/scripts/block: Writing backend/vbd/1/2050/physical-device 7:1 to xenstore. Dec 14 15:45:00 tp-c120 logger: /etc/xen/scripts/block: Writing backend/vbd/1/2049/hotplug-status error to xenstore. Dec 14 15:45:00 tp-c120 logger: /etc/xen/scripts/block: Writing backend/vbd/1/2050/hotplug-status connected to xenstore. Dec 14 15:45:00 tp-c120 logger: /etc/xen/scripts/block: losetup /dev/loop1 /vm/vm_images/vosg/GK/osg-gk-base.img failed ========================= XEND.LOG ==================================== [2005-12-14 15:44:53 xend.XendDomainInfo] DEBUG (XendDomainInfo:176) XendDomainInfo.create([''vm'', [''name'', ''osg-gk''], [''memory'', ''396''], [''cpu'', ''0''], [''image'', [''linux'', [''kernel'', ''/boot/vmlinuz-2.6.12-xenU''], [''ip'', ''140.221.67.232:1.2.3.4:140.221.67.253:255.255.254.0::eth0:off''], [''root'', ''/dev/sda1 ro''], [''vcpus'', ''1''], [''vcpus'', ''1''], [''boot'', ''c''], [''nics'', ''2'']]], [''device'', [''vbd'', [''uname'', ''file:/vm/vm_images/vosg/GK/osg-gk-base.img''], [''dev'', ''sda1''], [''mode'', ''w'']]], [''device'', [''vbd'', [''uname'', ''file:/vm/vm_images/vosg/GK/swap1.img''], [''dev'', ''sda2''], [''mode'', ''w'']]], [''device'', [''vbd'', [''uname'', ''file:/vm/vm_images/vosg/GK/osg-gk-grid.img''], [''dev'', ''sda3''], [''mode'', ''w'']]], [''device'', [''vbd'', [''uname'', ''file:/vm/vm_images/vosg/GK/osg-gk-app.img''], [''dev'', ''sda4''], [''mode'', ''w'']]], [''device'', [''vif'', [''mac'', ''00:06:AA:F6:BB:B3'']]], [''device'', [''vif'', [''mac'', ''00:06:AA:F7:BB:B3'']]]]) [2005-12-14 15:44:53 xend.XendDomainInfo] DEBUG (XendDomainInfo:281) parseConfig: config is [''vm'', [''name'', ''osg-gk''], [''memory'', ''396''], [''cpu'', ''0''], [''image'', [''linux'', [''kernel'', ''/boot/vmlinuz-2.6.12-xenU''], [''ip'', ''140.221.67.232:1.2.3.4:140.221.67.253:255.255.254.0::eth0:off''], [''root'', ''/dev/sda1 ro''], [''vcpus'', ''1''], [''vcpus'', ''1''], [''boot'', ''c''], [''nics'', ''2'']]], [''device'', [''vbd'', [''uname'', ''file:/vm/vm_images/vosg/GK/osg-gk-base.img''], [''dev'', ''sda1''], [''mode'', ''w'']]], [''device'', [''vbd'', [''uname'', ''file:/vm/vm_images/vosg/GK/swap1.img''], [''dev'', ''sda2''], [''mode'', ''w'']]], [''device'', [''vbd'', [''uname'', ''file:/vm/vm_images/vosg/GK/osg-gk-grid.img''], [''dev'', ''sda3''], [''mode'', ''w'']]], [''device'', [''vbd'', [''uname'', ''file:/vm/vm_images/vosg/GK/osg-gk-app.img''], [''dev'', ''sda4''], [''mode'', ''w'']]], [''device'', [''vif'', [''mac'', ''00:06:AA:F6:BB:B3'']]], [''device'', [''vif'', [''mac'', ''00:06:AA:F7:BB:B3'']]]] [2005-12-14 15:44:53 xend.XendDomainInfo] DEBUG (XendDomainInfo:373) parseConfig: result is {''ssidref'': None, ''uuid'': None, ''on_crash'': None, ''on_reboot'': None, ''image'': [''linux'', [''kernel'', ''/boot/vmlinuz-2.6.12-xenU''], [''ip'', ''140.221.67.232:1.2.3.4:140.221.67.253:255.255.254.0::eth0:off''], [''root'', ''/dev/sda1 ro''], [''vcpus'', ''1''], [''vcpus'', ''1''], [''boot'', ''c''], [''nics'', ''2'']], ''on_poweroff'': None, ''cpus'': None, ''name'': ''osg-gk'', ''backend'': [], ''vcpus'': 1, ''cpu_weight'': None, ''vcpu_avail'': None, ''memory'': 396, ''device'': [(''vbd'', [''vbd'', [''uname'', ''file:/vm/vm_images/vosg/GK/osg-gk-base.img''], [''dev'', ''sda1''], [''mode'', ''w'']]), (''vbd'', [''vbd'', [''uname'', ''file:/vm/vm_images/vosg/GK/swap1.img''], [''dev'', ''sda2''], [''mode'', ''w'']]), (''vbd'', [''vbd'', [''uname'', ''file:/vm/vm_images/vosg/GK/osg-gk-grid.img''], [''dev'', ''sda3''], [''mode'', ''w'']]), (''vbd'', [''vbd'', [''uname'', ''file:/vm/vm_images/vosg/GK/osg-gk-app.img''], [''dev'', ''sda4''], [''mode'', ''w'']]), (''vif'', [''vif'', [''mac'', ''00:06:AA:F6:BB:B3'']]), (''vif'', [''vif'', [''mac'', ''00:06:AA:F7:BB:B3'']])], ''bootloader'': None, ''cpu'': 0, ''maxmem'': None} [2005-12-14 15:44:53 xend.XendDomainInfo] DEBUG (XendDomainInfo:1123) XendDomainInfo.construct: None 0 [2005-12-14 15:44:53 xend.XendDomainInfo] DEBUG (XendDomainInfo:1155) XendDomainInfo.initDomain: 1 1.0 [2005-12-14 15:44:53 xend] DEBUG (balloon:47) Balloon: free 2950; need 397. [2005-12-14 15:44:53 xend] INFO (image:132) buildDomain os=linux dom=1 vcpus=1 [2005-12-14 15:44:53 xend] DEBUG (image:170) dom = 1 [2005-12-14 15:44:53 xend] DEBUG (image:171) image /boot/vmlinuz-2.6.12-xenU [2005-12-14 15:44:53 xend] DEBUG (image:172) store_evtchn = 1 [2005-12-14 15:44:53 xend] DEBUG (image:173) console_evtchn = 2 [2005-12-14 15:44:53 xend] DEBUG (image:174) cmdline ip=140.221.67.232:1.2.3.4:140.221.67.253:255.255.254.0::eth0:off root=/dev/sda1 ro [2005-12-14 15:44:53 xend] DEBUG (image:175) ramdisk [2005-12-14 15:44:53 xend] DEBUG (image:176) vcpus = 1 [2005-12-14 15:44:54 xend] DEBUG (DevController:104) DevController: writing {''virtual-device'': ''2049'', ''backend-id'': ''0'', ''state'': ''1'', ''backend'': ''/local/domain/0/backend/vbd/1/2049''} to /local/domain/1/device/vbd/2049. [2005-12-14 15:44:54 xend] DEBUG (DevController:106) DevController: writing {''domain'': ''osg-gk'', ''frontend'': ''/local/domain/1/device/vbd/2049'', ''dev'': ''sda1'', ''state'': ''1'', ''params'': ''/vm/vm_images/vosg/GK/osg-gk-base.img'', ''mode'': ''w'', ''frontend-id'': ''1'', ''type'': ''file''} to /local/domain/0/backend/vbd/1/2049. [2005-12-14 15:44:54 xend] DEBUG (DevController:104) DevController: writing {''virtual-device'': ''2050'', ''backend-id'': ''0'', ''state'': ''1'', ''backend'': ''/local/domain/0/backend/vbd/1/2050''} to /local/domain/1/device/vbd/2050. [2005-12-14 15:44:54 xend] DEBUG (DevController:104) DevController: writing {''virtual-device'': ''2051'', ''backend-id'': ''0'', ''state'': ''1'', ''backend'': ''/local/domain/0/backend/vbd/1/2051''} to /local/domain/1/device/vbd/2051. [2005-12-14 15:44:54 xend] DEBUG (DevController:106) DevController: writing {''domain'': ''osg-gk'', ''frontend'': ''/local/domain/1/device/vbd/2051'', ''dev'': ''sda3'', ''state'': ''1'', ''params'': ''/vm/vm_images/vosg/GK/osg-gk-grid.img'', ''mode'': ''w'', ''frontend-id'': ''1'', ''type'': ''file''} to /local/domain/0/backend/vbd/1/2051. [2005-12-14 15:44:54 xend] DEBUG (DevController:104) DevController: writing {''virtual-device'': ''2052'', ''backend-id'': ''0'', ''state'': ''1'', ''backend'': ''/local/domain/0/backend/vbd/1/2052''} to /local/domain/1/device/vbd/2052. [2005-12-14 15:44:54 xend] DEBUG (DevController:106) DevController: writing {''domain'': ''osg-gk'', ''frontend'': ''/local/domain/1/device/vbd/2052'', ''dev'': ''sda4'', ''state'': ''1'', ''params'': ''/vm/vm_images/vosg/GK/osg-gk-app.img'', ''mode'': ''w'', ''frontend-id'': ''1'', ''type'': ''file''} to /local/domain/0/backend/vbd/1/2052. [2005-12-14 15:44:54 xend] DEBUG (DevController:104) DevController: writing {''backend-id'': ''0'', ''mac'': ''00:06:AA:F6:BB:B3'', ''handle'': ''0'', ''state'': ''1'', ''backend'': ''/local/domain/0/backend/vif/1/0''} to /local/domain/1/device/vif/0. [2005-12-14 15:44:54 xend] DEBUG (DevController:106) DevController: writing {''mac'': ''00:06:AA:F6:BB:B3'', ''state'': ''1'', ''handle'': ''0'', ''script'': ''/etc/xen/scripts/vif-bridge'', ''frontend-id'': ''1'', ''domain'': ''osg-gk'', ''frontend'': ''/local/domain/1/device/vif/0''} to /local/domain/0/backend/vif/1/0. [2005-12-14 15:44:54 xend] DEBUG (DevController:104) DevController: writing {''backend-id'': ''0'', ''mac'': ''00:06:AA:F7:BB:B3'', ''handle'': ''1'', ''state'': ''1'', ''backend'': ''/local/domain/0/backend/vif/1/1''} to /local/domain/1/device/vif/1. [2005-12-14 15:44:54 xend] DEBUG (DevController:106) DevController: writing {''mac'': ''00:06:AA:F7:BB:B3'', ''state'': ''1'', ''handle'': ''1'', ''script'': ''/etc/xen/scripts/vif-bridge'', ''frontend-id'': ''1'', ''domain'': ''osg-gk'', ''frontend'': ''/local/domain/1/device/vif/1''} to /local/domain/0/backend/vif/1/1. [2005-12-14 15:44:54 xend.XendDomainInfo] DEBUG (XendDomainInfo:655) Storing VM details: {''ssidref'': ''0'', ''uuid'': ''fdd737fd-63b3b254-562d4b14-94c311e7'', ''on_reboot'': ''restart'', ''image'': "(linux (kernel /boot/vmlinuz-2.6.12-xenU) (ip 140.221.67.232:1.2.3.4:140.221.67.253:255.255.254.0::eth0:off) (root ''/dev/sda1 ro'') (vcpus 1) (vcpus 1) (boot c) (nics 2))", ''on_poweroff'': ''destroy'', ''name'': ''osg-gk'', ''vcpus'': ''1'', ''vcpu_avail'': ''1'', ''memory'': ''396'', ''on_crash'': ''restart'', ''start_time'': ''1134596694.65'', ''maxmem'': ''396''} [2005-12-14 15:44:54 xend.XendDomainInfo] DEBUG (XendDomainInfo:680) Storing domain details: {''console/ring-ref'': ''773181'', ''console/port'': ''2'', ''name'': ''osg-gk'', ''console/limit'': ''1048576'', ''vm'': ''/vm/fdd737fd-63b3b254-562d4b14-94c311e7'', ''domid'': ''1'', ''cpu/0/availability'': ''online'', ''memory/target'': ''405504'', ''store/ring-ref'': ''773182'', ''store/port'': ''1''} [2005-12-14 15:44:54 xend] DEBUG (DevController:133) Waiting for devices vif. [2005-12-14 15:44:54 xend] DEBUG (DevController:139) Waiting for 0. [2005-12-14 15:44:54 xend] DEBUG (DevController:409) hotplugStatusCallback /local/domain/0/backend/vif/1/0/hotplug-status. [2005-12-14 15:44:56 xend] DEBUG (DevController:409) hotplugStatusCallback /local/domain/0/backend/vif/1/0/hotplug-status. [2005-12-14 15:44:56 xend] DEBUG (DevController:426) hotplugStatusCallback 1. [2005-12-14 15:44:56 xend] DEBUG (DevController:139) Waiting for 1. [2005-12-14 15:44:56 xend] DEBUG (DevController:409) hotplugStatusCallback /local/domain/0/backend/vif/1/1/hotplug-status. [2005-12-14 15:44:56 xend] DEBUG (DevController:426) hotplugStatusCallback 1. [2005-12-14 15:44:56 xend] DEBUG (DevController:133) Waiting for devices usb. [2005-12-14 15:44:56 xend] DEBUG (DevController:133) Waiting for devices vbd. [2005-12-14 15:44:56 xend] DEBUG (DevController:139) Waiting for 2049. [2005-12-14 15:44:56 xend] DEBUG (DevController:409) hotplugStatusCallback /local/domain/0/backend/vbd/1/2049/hotplug-status. [2005-12-14 15:45:00 xend] DEBUG (DevController:409) hotplugStatusCallback /local/domain/0/backend/vbd/1/2049/hotplug-status. [2005-12-14 15:45:00 xend] DEBUG (DevController:426) hotplugStatusCallback 3. [2005-12-14 15:45:00 xend] ERROR (SrvBase:87) Request wait_for_devices failed. Traceback (most recent call last): File "/usr/lib64/python/xen/web/SrvBase.py", line 85, in perform return op_method(op, req) File "/usr/lib64/python/xen/xend/server/SrvDomain.py", line 72, in op_wait_for_devices return self.dom.waitForDevices() File "/usr/lib64/python/xen/xend/XendDomainInfo.py", line 1343, in waitForDevices self.waitForDevices_(c) File "/usr/lib64/python/xen/xend/XendDomainInfo.py", line 971, in waitForDevices_ return self.getDeviceController(deviceClass).waitForDevices() File "/usr/lib64/python/xen/xend/server/DevController.py", line 135, in waitForDevices return map(self.waitForDevice, self.deviceIDs()) File "/usr/lib64/python/xen/xend/server/DevController.py", line 151, in waitForDevice raise VmError("Device %s (%s) could not be connected. " VmError: Device 2049 (vbd) could not be connected. Backend device not found. ================ XM LIST --LONG (BEFORE UNPAUSE) ====================== (domain (domid 0) (uuid 00000000-00000000-00000000-00000000) (ssidref 0) (vcpus 2) (vcpu_avail 3) (cpu_weight 1.0) (memory 1019) (maxmem 1019) (name Domain-0) (on_poweroff destroy) (on_reboot restart) (on_crash restart) (state r-----) (shutdown_reason poweroff) (cpu_time 111.513931896) (vcpus 2) (online_vcpus 1) ) (domain (domid 1) (uuid fdd737fd-63b3b254-562d4b14-94c311e7) (ssidref 0) (vcpus 1) (vcpu_avail 1) (cpu_weight 1.0) (memory 396) (maxmem 396) (name osg-gk) (on_poweroff destroy) (on_reboot restart) (on_crash restart) (image (linux (kernel /boot/vmlinuz-2.6.12-xenU) (ip XXX.YYY.67.232:1.2.3.4:XXX.YYY.67.253:255.255.254.0::eth0:off) (root ''/dev/sda1 ro'') (vcpus 1) (vcpus 1) (boot c) (nics 2) ) ) (device (vif (backend 0) (script vif-bridge) (mac 00:06:AA:F6:BB:B3))) (device (vif (backend 0) (script vif-bridge) (mac 00:06:AA:F7:BB:B3))) (device (vbd (backend 0) (dev sda1) (uname file:/vm/vm_images/vosg/GK/osg-gk-base.img) (mode w) ) ) (device (vbd (backend 0) (dev sda2) (uname file:/vm/vm_images/vosg/GK/swap1.img) (mode w) ) ) (device (vbd (backend 0) (dev sda3) (uname file:/vm/vm_images/vosg/GK/osg-gk-grid.img) (mode w) ) ) (device (vbd (backend 0) (dev sda4) (uname file:/vm/vm_images/vosg/GK/osg-gk-app.img) (mode w) ) ) (state --p---) (shutdown_reason poweroff) (cpu_time 0.0) (vcpus 1) (online_vcpus 1) (up_time 397.679515123) (start_time 1134596694.65) (store_mfn 773182) (console_mfn 773181) ) ================ XM LIST --LONG (AFTER UNPAUSE) ====================== (domain (domid 0) (uuid 00000000-00000000-00000000-00000000) (ssidref 0) (vcpus 2) (vcpu_avail 3) (cpu_weight 1.0) (memory 1019) (maxmem 1019) (name Domain-0) (on_poweroff destroy) (on_reboot restart) (on_crash restart) (state r-----) (shutdown_reason poweroff) (cpu_time 113.247016938) (vcpus 2) (online_vcpus 1) ) (domain (domid 1) (uuid fdd737fd-63b3b254-562d4b14-94c311e7) (ssidref 0) (vcpus 1) (vcpu_avail 1) (cpu_weight 1.0) (memory 396) (maxmem 396) (name osg-gk) (on_poweroff destroy) (on_reboot restart) (on_crash restart) (image (linux (kernel /boot/vmlinuz-2.6.12-xenU) (ip XXX.YYY.67.232:1.2.3.4:XXX.YYY.67.253:255.255.254.0::eth0:off) (root ''/dev/sda1 ro'') (vcpus 1) (vcpus 1) (boot c) (nics 2) ) ) (device (vif (backend 0) (script vif-bridge) (mac 00:06:AA:F6:BB:B3))) (device (vif (backend 0) (script vif-bridge) (mac 00:06:AA:F7:BB:B3))) (device (vbd (backend 0) (dev sda2) (uname file:/vm/vm_images/vosg/GK/swap1.img) (mode w) ) ) (device (vbd (backend 0) (dev sda3) (uname file:/vm/vm_images/vosg/GK/osg-gk-grid.img) (mode w) ) ) (device (vbd (backend 0) (dev sda4) (uname file:/vm/vm_images/vosg/GK/osg-gk-app.img) (mode w) ) ) (state r-----) (shutdown_reason poweroff) (cpu_time 6.079421914) (vcpus 1) (online_vcpus 1) (up_time 473.439181089) (start_time 1134596694.65) (store_mfn 773182) (console_mfn 773181) ) -- ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: Borja Sotomayor, University of Chicago Ph.D. Student, Department of Computer Science Hinds 016, 5734 South Ellis Avenue, 60637 Chicago, IL GT4 Tutorial: http://gdp.globus.org/gt4-tutorial/ BorjaNet: http://www.borjanet.com/ borja@borjanet.com ····························································· "Dis maschine vill run und run!" -- Kurt Gödel (on the Turing Machine) ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Satoshi Uchida
2005-Dec-15 01:32 UTC
RE: [Xen-users] Device 2049 (vbd) could not be connected. Backenddevice not found.
Hi. Borja What version do you use? CD images? Source download? In Xen 3.0 CD which is download in 25 Nov., if one domain is constructed using two and more than images, these VBDs conflict at same loop device. This problem is fixed at lock mechanism before two weeks. If would you like to download new patches ? Satoshi UCHIDA -----Original Message----- From: Borja Sotomayor [mailto:borja@borjanet.com] Sent: Thursday, December 15, 2005 9:00 AM To: xen-users@lists.xensource.com Subject: [Xen-users] Device 2049 (vbd) could not be connected. Backenddevice not found. Hi everyone, I''m running into a weird problem when running Xen 3.0. If I try to create new domains that use a single file-backed VBD (e.g. ttylinux-xen or a custom Debian image), everything works fine. However, if I try to create a domain that requires several VBDs, I get the following error message: tp-c120:/vm/vm_images/vosg/GK # xm create -c osg-gk.config Using config file "osg-gk.config". Error: Device 2049 (vbd) could not be connected. Backend device not found. [The config file osg-gk.config is included at the end of this e-mail] Note that I get the same error message if I try to create a new domain with a custom Debian image (which works with a single VBD) and simply add a swap partition. Also, once I get this error, I can''t create more domains (single or multiple VBD). Along with the config file, I''m attaching the output from syslog, xend.log, and xm list --long. I noticed that the domain _does_ get created, but is placed in paused state. However, if I try to unpause it, the root partition misteriously disappears and the DomU kernel crashes (I''m also attaching the output of xm list --long after unpausing the domain) Finally, here''s some information on the machine and it''s software: * Machine: 1U IBM e1350 eServer cluster node * IBM e325 Opteron Model 248 (2.2 GHz/1MB L2), 2x512 MB * Operating system: Enterprise Suse Linux 9 (SLES9) Service Pack 2 tp-c120:/vm/vm_images/vosg/GK # rpm -q udev hotplug udev-021-36.49 hotplug-0.44-32.35 Note that I have only been able to reproduce this error on the above machine. I can run all my domains (ttylinux, Debian, osg-gk) fine in a different computer (a Thinkpad with Ubuntu Breezy Badger) Any help or suggestion would be appreciated :-) Cheers! -- Borja ========================= CONFIG FILE ================================= kernel = "/boot/vmlinuz-2.6.12-xenU" name = "osg-gk" memory = 396 cpu=0 disk = [ "file:/vm/vm_images/vosg/GK/osg-gk-base.img,sda1,w" , "file:/vm/vm_images/vosg/GK/swap1.img,sda2,w", "file:/vm/vm_images/vosg/GK/osg-gk-grid.img,sda3,w", "file:/vm/vm_images/vosg/GK/osg-gk-app.img,sda4,w"] nics=2 # public ip ip="XXX.YYY.67.232" netmask="255.255.254.0" network="XXX.YYY.66.0" broadcast="XXX.YYY.67.255" gateway="XXX.YYY.67.253" vif = [''mac=00:06:AA:F6:BB:B3'', ''mac=00:06:AA:F7:BB:B3''] # private ip # can''t set through the VM config file #ip="192.168.255.1" #netmask="255.255.255.0" #network="192.168.255.0" #broadcast="192.168.255.255" ##gateway="192.168.255.1" root = "/dev/sda1 ro" #extra = "4" ============================ SYSLOG ===================================Dec 14 15:44:34 tp-c120 kernel: <6>device vif0.0 entered promiscuous mode Dec 14 15:44:34 tp-c120 kernel: klogd 1.4.1, ---------- state change ---------- Dec 14 15:44:34 tp-c120 kernel: bridge: can''t decode speed from peth0: 0 Dec 14 15:44:34 tp-c120 kernel: device peth0 entered promiscuous mode Dec 14 15:44:34 tp-c120 kernel: xenbr0: port 1(vif0.0) entering learning state Dec 14 15:44:35 tp-c120 kernel: xenbr0: topology change detected, propagating Dec 14 15:44:35 tp-c120 kernel: xenbr0: port 1(vif0.0) entering forwarding state Dec 14 15:44:38 tp-c120 kernel: tg3: peth0: Link is up at 1000 Mbps, full duplex. Dec 14 15:44:38 tp-c120 kernel: tg3: peth0: Flow control is on for TX and on for RX. Dec 14 15:44:38 tp-c120 kernel: xenbr0: port 2(peth0) entering learning state Dec 14 15:44:38 tp-c120 kernel: xenbr0: topology change detected, propagating Dec 14 15:44:38 tp-c120 kernel: xenbr0: port 2(peth0) entering forwarding state Dec 14 15:44:39 tp-c120 ifup: No configuration found for xenbr0 Dec 14 15:44:54 tp-c120 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/1/2052 Dec 14 15:44:54 tp-c120 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/1/2051 Dec 14 15:44:54 tp-c120 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/1/2049 Dec 14 15:44:54 tp-c120 logger: /etc/xen/scripts/vif-bridge: online XENBUS_PATH=backend/vif/1/1 Dec 14 15:44:55 tp-c120 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/1/2050 Dec 14 15:44:55 tp-c120 logger: /etc/xen/scripts/vif-bridge: online XENBUS_PATH=backend/vif/1/0 Dec 14 15:44:55 tp-c120 kernel: device vif1.1 entered promiscuous mode Dec 14 15:44:55 tp-c120 kernel: xenbr0: port 3(vif1.1) entering learning state Dec 14 15:44:55 tp-c120 kernel: xenbr0: topology change detected, propagating Dec 14 15:44:55 tp-c120 kernel: xenbr0: port 3(vif1.1) entering forwarding state Dec 14 15:44:55 tp-c120 kernel: ip_tables: (C) 2000-2002 Netfilter core team Dec 14 15:44:56 tp-c120 logger: /etc/xen/scripts/vif-bridge: Successful vif-bridge online for vif1.1, bridge xenbr0. Dec 14 15:44:56 tp-c120 logger: /etc/xen/scripts/vif-bridge: Writing backend/vif/1/1/hotplug-status connected to xenstore. Dec 14 15:44:56 tp-c120 logger: /etc/xen/scripts/vif-bridge: Successful vif-bridge online for vif1.0, bridge xenbr0. Dec 14 15:44:56 tp-c120 logger: /etc/xen/scripts/vif-bridge: Writing backend/vif/1/0/hotplug-status connected to xenstore. Dec 14 15:44:56 tp-c120 kernel: device vif1.0 entered promiscuous mode Dec 14 15:44:56 tp-c120 kernel: xenbr0: port 4(vif1.0) entering learning state Dec 14 15:44:56 tp-c120 kernel: xenbr0: topology change detected, propagating Dec 14 15:44:56 tp-c120 kernel: xenbr0: port 4(vif1.0) entering forwarding state Dec 14 15:44:59 tp-c120 logger: /etc/xen/scripts/block: Forced to steal lock on /var/run/xen-hotplug/block from unknown! Dec 14 15:44:59 tp-c120 logger: /etc/xen/scripts/block: Forced to steal lock on /var/run/xen-hotplug/block from 12554: /etc/xen/scripts/block! Dec 14 15:44:59 tp-c120 logger: /etc/xen/scripts/block: Writing backend/vbd/1/2051/node /dev/loop0 to xenstore. Dec 14 15:44:59 tp-c120 ifup: No configuration found for vif1.0 Dec 14 15:44:59 tp-c120 logger: /etc/xen/scripts/block: Writing backend/vbd/1/2051/physical-device 7:0 to xenstore. Dec 14 15:44:59 tp-c120 logger: /etc/xen/scripts/block: Writing backend/vbd/1/2051/hotplug-status connected to xenstore. Dec 14 15:44:59 tp-c120 logger: /etc/xen/scripts/block: Writing backend/vbd/1/2052/hotplug-status error to xenstore. Dec 14 15:44:59 tp-c120 logger: /etc/xen/scripts/block: losetup /dev/loop0 /vm/vm_images/vosg/GK/osg-gk-app.img failed Dec 14 15:45:00 tp-c120 ifup: No configuration found for vif1.1 Dec 14 15:45:00 tp-c120 logger: /etc/xen/scripts/block: Forced to steal lock on /var/run/xen-hotplug/block from unknown! Dec 14 15:45:00 tp-c120 logger: /etc/xen/scripts/block: Forced to steal lock on /var/run/xen-hotplug/block from 12453: /etc/xen/scripts/block! Dec 14 15:45:00 tp-c120 logger: /etc/xen/scripts/block: Writing backend/vbd/1/2050/node /dev/loop1 to xenstore. Dec 14 15:45:00 tp-c120 logger: /etc/xen/scripts/block: Writing backend/vbd/1/2050/physical-device 7:1 to xenstore. Dec 14 15:45:00 tp-c120 logger: /etc/xen/scripts/block: Writing backend/vbd/1/2049/hotplug-status error to xenstore. Dec 14 15:45:00 tp-c120 logger: /etc/xen/scripts/block: Writing backend/vbd/1/2050/hotplug-status connected to xenstore. Dec 14 15:45:00 tp-c120 logger: /etc/xen/scripts/block: losetup /dev/loop1 /vm/vm_images/vosg/GK/osg-gk-base.img failed ========================= XEND.LOG ==================================== [2005-12-14 15:44:53 xend.XendDomainInfo] DEBUG (XendDomainInfo:176) XendDomainInfo.create([''vm'', [''name'', ''osg-gk''], [''memory'', ''396''], [''cpu'', ''0''], [''image'', [''linux'', [''kernel'', ''/boot/vmlinuz-2.6.12-xenU''], [''ip'', ''140.221.67.232:1.2.3.4:140.221.67.253:255.255.254.0::eth0:off''], [''root'', ''/dev/sda1 ro''], [''vcpus'', ''1''], [''vcpus'', ''1''], [''boot'', ''c''], [''nics'', ''2'']]], [''device'', [''vbd'', [''uname'', ''file:/vm/vm_images/vosg/GK/osg-gk-base.img''], [''dev'', ''sda1''], [''mode'', ''w'']]], [''device'', [''vbd'', [''uname'', ''file:/vm/vm_images/vosg/GK/swap1.img''], [''dev'', ''sda2''], [''mode'', ''w'']]], [''device'', [''vbd'', [''uname'', ''file:/vm/vm_images/vosg/GK/osg-gk-grid.img''], [''dev'', ''sda3''], [''mode'', ''w'']]], [''device'', [''vbd'', [''uname'', ''file:/vm/vm_images/vosg/GK/osg-gk-app.img''], [''dev'', ''sda4''], [''mode'', ''w'']]], [''device'', [''vif'', [''mac'', ''00:06:AA:F6:BB:B3'']]], [''device'', [''vif'', [''mac'', ''00:06:AA:F7:BB:B3'']]]]) [2005-12-14 15:44:53 xend.XendDomainInfo] DEBUG (XendDomainInfo:281) parseConfig: config is [''vm'', [''name'', ''osg-gk''], [''memory'', ''396''], [''cpu'', ''0''], [''image'', [''linux'', [''kernel'', ''/boot/vmlinuz-2.6.12-xenU''], [''ip'', ''140.221.67.232:1.2.3.4:140.221.67.253:255.255.254.0::eth0:off''], [''root'', ''/dev/sda1 ro''], [''vcpus'', ''1''], [''vcpus'', ''1''], [''boot'', ''c''], [''nics'', ''2'']]], [''device'', [''vbd'', [''uname'', ''file:/vm/vm_images/vosg/GK/osg-gk-base.img''], [''dev'', ''sda1''], [''mode'', ''w'']]], [''device'', [''vbd'', [''uname'', ''file:/vm/vm_images/vosg/GK/swap1.img''], [''dev'', ''sda2''], [''mode'', ''w'']]], [''device'', [''vbd'', [''uname'', ''file:/vm/vm_images/vosg/GK/osg-gk-grid.img''], [''dev'', ''sda3''], [''mode'', ''w'']]], [''device'', [''vbd'', [''uname'', ''file:/vm/vm_images/vosg/GK/osg-gk-app.img''], [''dev'', ''sda4''], [''mode'', ''w'']]], [''device'', [''vif'', [''mac'', ''00:06:AA:F6:BB:B3'']]], [''device'', [''vif'', [''mac'', ''00:06:AA:F7:BB:B3'']]]] [2005-12-14 15:44:53 xend.XendDomainInfo] DEBUG (XendDomainInfo:373) parseConfig: result is {''ssidref'': None, ''uuid'': None, ''on_crash'': None, ''on_reboot'': None, ''image'': [''linux'', [''kernel'', ''/boot/vmlinuz-2.6.12-xenU''], [''ip'', ''140.221.67.232:1.2.3.4:140.221.67.253:255.255.254.0::eth0:off''], [''root'', ''/dev/sda1 ro''], [''vcpus'', ''1''], [''vcpus'', ''1''], [''boot'', ''c''], [''nics'', ''2'']], ''on_poweroff'': None, ''cpus'': None, ''name'': ''osg-gk'', ''backend'': [], ''vcpus'': 1, ''cpu_weight'': None, ''vcpu_avail'': None, ''memory'': 396, ''device'': [(''vbd'', [''vbd'', [''uname'', ''file:/vm/vm_images/vosg/GK/osg-gk-base.img''], [''dev'', ''sda1''], [''mode'', ''w'']]), (''vbd'', [''vbd'', [''uname'', ''file:/vm/vm_images/vosg/GK/swap1.img''], [''dev'', ''sda2''], [''mode'', ''w'']]), (''vbd'', [''vbd'', [''uname'', ''file:/vm/vm_images/vosg/GK/osg-gk-grid.img''], [''dev'', ''sda3''], [''mode'', ''w'']]), (''vbd'', [''vbd'', [''uname'', ''file:/vm/vm_images/vosg/GK/osg-gk-app.img''], [''dev'', ''sda4''], [''mode'', ''w'']]), (''vif'', [''vif'', [''mac'', ''00:06:AA:F6:BB:B3'']]), (''vif'', [''vif'', [''mac'', ''00:06:AA:F7:BB:B3'']])], ''bootloader'': None, ''cpu'': 0, ''maxmem'': None} [2005-12-14 15:44:53 xend.XendDomainInfo] DEBUG (XendDomainInfo:1123) XendDomainInfo.construct: None 0 [2005-12-14 15:44:53 xend.XendDomainInfo] DEBUG (XendDomainInfo:1155) XendDomainInfo.initDomain: 1 1.0 [2005-12-14 15:44:53 xend] DEBUG (balloon:47) Balloon: free 2950; need 397. [2005-12-14 15:44:53 xend] INFO (image:132) buildDomain os=linux dom=1 vcpus=1 [2005-12-14 15:44:53 xend] DEBUG (image:170) dom = 1 [2005-12-14 15:44:53 xend] DEBUG (image:171) image /boot/vmlinuz-2.6.12-xenU [2005-12-14 15:44:53 xend] DEBUG (image:172) store_evtchn = 1 [2005-12-14 15:44:53 xend] DEBUG (image:173) console_evtchn = 2 [2005-12-14 15:44:53 xend] DEBUG (image:174) cmdline ip=140.221.67.232:1.2.3.4:140.221.67.253:255.255.254.0::eth0:off root=/dev/sda1 ro [2005-12-14 15:44:53 xend] DEBUG (image:175) ramdisk [2005-12-14 15:44:53 xend] DEBUG (image:176) vcpus = 1 [2005-12-14 15:44:54 xend] DEBUG (DevController:104) DevController: writing {''virtual-device'': ''2049'', ''backend-id'': ''0'', ''state'': ''1'', ''backend'': ''/local/domain/0/backend/vbd/1/2049''} to /local/domain/1/device/vbd/2049. [2005-12-14 15:44:54 xend] DEBUG (DevController:106) DevController: writing {''domain'': ''osg-gk'', ''frontend'': ''/local/domain/1/device/vbd/2049'', ''dev'': ''sda1'', ''state'': ''1'', ''params'': ''/vm/vm_images/vosg/GK/osg-gk-base.img'', ''mode'': ''w'', ''frontend-id'': ''1'', ''type'': ''file''} to /local/domain/0/backend/vbd/1/2049. [2005-12-14 15:44:54 xend] DEBUG (DevController:104) DevController: writing {''virtual-device'': ''2050'', ''backend-id'': ''0'', ''state'': ''1'', ''backend'': ''/local/domain/0/backend/vbd/1/2050''} to /local/domain/1/device/vbd/2050. [2005-12-14 15:44:54 xend] DEBUG (DevController:104) DevController: writing {''virtual-device'': ''2051'', ''backend-id'': ''0'', ''state'': ''1'', ''backend'': ''/local/domain/0/backend/vbd/1/2051''} to /local/domain/1/device/vbd/2051. [2005-12-14 15:44:54 xend] DEBUG (DevController:106) DevController: writing {''domain'': ''osg-gk'', ''frontend'': ''/local/domain/1/device/vbd/2051'', ''dev'': ''sda3'', ''state'': ''1'', ''params'': ''/vm/vm_images/vosg/GK/osg-gk-grid.img'', ''mode'': ''w'', ''frontend-id'': ''1'', ''type'': ''file''} to /local/domain/0/backend/vbd/1/2051. [2005-12-14 15:44:54 xend] DEBUG (DevController:104) DevController: writing {''virtual-device'': ''2052'', ''backend-id'': ''0'', ''state'': ''1'', ''backend'': ''/local/domain/0/backend/vbd/1/2052''} to /local/domain/1/device/vbd/2052. [2005-12-14 15:44:54 xend] DEBUG (DevController:106) DevController: writing {''domain'': ''osg-gk'', ''frontend'': ''/local/domain/1/device/vbd/2052'', ''dev'': ''sda4'', ''state'': ''1'', ''params'': ''/vm/vm_images/vosg/GK/osg-gk-app.img'', ''mode'': ''w'', ''frontend-id'': ''1'', ''type'': ''file''} to /local/domain/0/backend/vbd/1/2052. [2005-12-14 15:44:54 xend] DEBUG (DevController:104) DevController: writing {''backend-id'': ''0'', ''mac'': ''00:06:AA:F6:BB:B3'', ''handle'': ''0'', ''state'': ''1'', ''backend'': ''/local/domain/0/backend/vif/1/0''} to /local/domain/1/device/vif/0. [2005-12-14 15:44:54 xend] DEBUG (DevController:106) DevController: writing {''mac'': ''00:06:AA:F6:BB:B3'', ''state'': ''1'', ''handle'': ''0'', ''script'': ''/etc/xen/scripts/vif-bridge'', ''frontend-id'': ''1'', ''domain'': ''osg-gk'', ''frontend'': ''/local/domain/1/device/vif/0''} to /local/domain/0/backend/vif/1/0. [2005-12-14 15:44:54 xend] DEBUG (DevController:104) DevController: writing {''backend-id'': ''0'', ''mac'': ''00:06:AA:F7:BB:B3'', ''handle'': ''1'', ''state'': ''1'', ''backend'': ''/local/domain/0/backend/vif/1/1''} to /local/domain/1/device/vif/1. [2005-12-14 15:44:54 xend] DEBUG (DevController:106) DevController: writing {''mac'': ''00:06:AA:F7:BB:B3'', ''state'': ''1'', ''handle'': ''1'', ''script'': ''/etc/xen/scripts/vif-bridge'', ''frontend-id'': ''1'', ''domain'': ''osg-gk'', ''frontend'': ''/local/domain/1/device/vif/1''} to /local/domain/0/backend/vif/1/1. [2005-12-14 15:44:54 xend.XendDomainInfo] DEBUG (XendDomainInfo:655) Storing VM details: {''ssidref'': ''0'', ''uuid'': ''fdd737fd-63b3b254-562d4b14-94c311e7'', ''on_reboot'': ''restart'', ''image'': "(linux (kernel /boot/vmlinuz-2.6.12-xenU) (ip 140.221.67.232:1.2.3.4:140.221.67.253:255.255.254.0::eth0:off) (root ''/dev/sda1 ro'') (vcpus 1) (vcpus 1) (boot c) (nics 2))", ''on_poweroff'': ''destroy'', ''name'': ''osg-gk'', ''vcpus'': ''1'', ''vcpu_avail'': ''1'', ''memory'': ''396'', ''on_crash'': ''restart'', ''start_time'': ''1134596694.65'', ''maxmem'': ''396''} [2005-12-14 15:44:54 xend.XendDomainInfo] DEBUG (XendDomainInfo:680) Storing domain details: {''console/ring-ref'': ''773181'', ''console/port'': ''2'', ''name'': ''osg-gk'', ''console/limit'': ''1048576'', ''vm'': ''/vm/fdd737fd-63b3b254-562d4b14-94c311e7'', ''domid'': ''1'', ''cpu/0/availability'': ''online'', ''memory/target'': ''405504'', ''store/ring-ref'': ''773182'', ''store/port'': ''1''} [2005-12-14 15:44:54 xend] DEBUG (DevController:133) Waiting for devices vif. [2005-12-14 15:44:54 xend] DEBUG (DevController:139) Waiting for 0. [2005-12-14 15:44:54 xend] DEBUG (DevController:409) hotplugStatusCallback /local/domain/0/backend/vif/1/0/hotplug-status. [2005-12-14 15:44:56 xend] DEBUG (DevController:409) hotplugStatusCallback /local/domain/0/backend/vif/1/0/hotplug-status. [2005-12-14 15:44:56 xend] DEBUG (DevController:426) hotplugStatusCallback 1. [2005-12-14 15:44:56 xend] DEBUG (DevController:139) Waiting for 1. [2005-12-14 15:44:56 xend] DEBUG (DevController:409) hotplugStatusCallback /local/domain/0/backend/vif/1/1/hotplug-status. [2005-12-14 15:44:56 xend] DEBUG (DevController:426) hotplugStatusCallback 1. [2005-12-14 15:44:56 xend] DEBUG (DevController:133) Waiting for devices usb. [2005-12-14 15:44:56 xend] DEBUG (DevController:133) Waiting for devices vbd. [2005-12-14 15:44:56 xend] DEBUG (DevController:139) Waiting for 2049. [2005-12-14 15:44:56 xend] DEBUG (DevController:409) hotplugStatusCallback /local/domain/0/backend/vbd/1/2049/hotplug-status. [2005-12-14 15:45:00 xend] DEBUG (DevController:409) hotplugStatusCallback /local/domain/0/backend/vbd/1/2049/hotplug-status. [2005-12-14 15:45:00 xend] DEBUG (DevController:426) hotplugStatusCallback 3. [2005-12-14 15:45:00 xend] ERROR (SrvBase:87) Request wait_for_devices failed. Traceback (most recent call last): File "/usr/lib64/python/xen/web/SrvBase.py", line 85, in perform return op_method(op, req) File "/usr/lib64/python/xen/xend/server/SrvDomain.py", line 72, in op_wait_for_devices return self.dom.waitForDevices() File "/usr/lib64/python/xen/xend/XendDomainInfo.py", line 1343, in waitForDevices self.waitForDevices_(c) File "/usr/lib64/python/xen/xend/XendDomainInfo.py", line 971, in waitForDevices_ return self.getDeviceController(deviceClass).waitForDevices() File "/usr/lib64/python/xen/xend/server/DevController.py", line 135, in waitForDevices return map(self.waitForDevice, self.deviceIDs()) File "/usr/lib64/python/xen/xend/server/DevController.py", line 151, in waitForDevice raise VmError("Device %s (%s) could not be connected. " VmError: Device 2049 (vbd) could not be connected. Backend device not found. ================ XM LIST --LONG (BEFORE UNPAUSE) ====================== (domain (domid 0) (uuid 00000000-00000000-00000000-00000000) (ssidref 0) (vcpus 2) (vcpu_avail 3) (cpu_weight 1.0) (memory 1019) (maxmem 1019) (name Domain-0) (on_poweroff destroy) (on_reboot restart) (on_crash restart) (state r-----) (shutdown_reason poweroff) (cpu_time 111.513931896) (vcpus 2) (online_vcpus 1) ) (domain (domid 1) (uuid fdd737fd-63b3b254-562d4b14-94c311e7) (ssidref 0) (vcpus 1) (vcpu_avail 1) (cpu_weight 1.0) (memory 396) (maxmem 396) (name osg-gk) (on_poweroff destroy) (on_reboot restart) (on_crash restart) (image (linux (kernel /boot/vmlinuz-2.6.12-xenU) (ip XXX.YYY.67.232:1.2.3.4:XXX.YYY.67.253:255.255.254.0::eth0:off) (root ''/dev/sda1 ro'') (vcpus 1) (vcpus 1) (boot c) (nics 2) ) ) (device (vif (backend 0) (script vif-bridge) (mac 00:06:AA:F6:BB:B3))) (device (vif (backend 0) (script vif-bridge) (mac 00:06:AA:F7:BB:B3))) (device (vbd (backend 0) (dev sda1) (uname file:/vm/vm_images/vosg/GK/osg-gk-base.img) (mode w) ) ) (device (vbd (backend 0) (dev sda2) (uname file:/vm/vm_images/vosg/GK/swap1.img) (mode w) ) ) (device (vbd (backend 0) (dev sda3) (uname file:/vm/vm_images/vosg/GK/osg-gk-grid.img) (mode w) ) ) (device (vbd (backend 0) (dev sda4) (uname file:/vm/vm_images/vosg/GK/osg-gk-app.img) (mode w) ) ) (state --p---) (shutdown_reason poweroff) (cpu_time 0.0) (vcpus 1) (online_vcpus 1) (up_time 397.679515123) (start_time 1134596694.65) (store_mfn 773182) (console_mfn 773181) ) ================ XM LIST --LONG (AFTER UNPAUSE) ====================== (domain (domid 0) (uuid 00000000-00000000-00000000-00000000) (ssidref 0) (vcpus 2) (vcpu_avail 3) (cpu_weight 1.0) (memory 1019) (maxmem 1019) (name Domain-0) (on_poweroff destroy) (on_reboot restart) (on_crash restart) (state r-----) (shutdown_reason poweroff) (cpu_time 113.247016938) (vcpus 2) (online_vcpus 1) ) (domain (domid 1) (uuid fdd737fd-63b3b254-562d4b14-94c311e7) (ssidref 0) (vcpus 1) (vcpu_avail 1) (cpu_weight 1.0) (memory 396) (maxmem 396) (name osg-gk) (on_poweroff destroy) (on_reboot restart) (on_crash restart) (image (linux (kernel /boot/vmlinuz-2.6.12-xenU) (ip XXX.YYY.67.232:1.2.3.4:XXX.YYY.67.253:255.255.254.0::eth0:off) (root ''/dev/sda1 ro'') (vcpus 1) (vcpus 1) (boot c) (nics 2) ) ) (device (vif (backend 0) (script vif-bridge) (mac 00:06:AA:F6:BB:B3))) (device (vif (backend 0) (script vif-bridge) (mac 00:06:AA:F7:BB:B3))) (device (vbd (backend 0) (dev sda2) (uname file:/vm/vm_images/vosg/GK/swap1.img) (mode w) ) ) (device (vbd (backend 0) (dev sda3) (uname file:/vm/vm_images/vosg/GK/osg-gk-grid.img) (mode w) ) ) (device (vbd (backend 0) (dev sda4) (uname file:/vm/vm_images/vosg/GK/osg-gk-app.img) (mode w) ) ) (state r-----) (shutdown_reason poweroff) (cpu_time 6.079421914) (vcpus 1) (online_vcpus 1) (up_time 473.439181089) (start_time 1134596694.65) (store_mfn 773182) (console_mfn 773181) ) -- ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: Borja Sotomayor, University of Chicago Ph.D. Student, Department of Computer Science Hinds 016, 5734 South Ellis Avenue, 60637 Chicago, IL GT4 Tutorial: http://gdp.globus.org/gt4-tutorial/ BorjaNet: http://www.borjanet.com/ borja@borjanet.com ····························································· "Dis maschine vill run und run!" -- Kurt Gödel (on the Turing Machine) ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Borja Sotomayor
2005-Dec-15 01:43 UTC
Re: [Xen-users] Device 2049 (vbd) could not be connected. Backenddevice not found.
Hi,> What version do you use? > CD images? Source download?I''m using the 3.0.0 source download.> In Xen 3.0 CD which is download in 25 Nov., if one domain is constructed > using two and more than images, these VBDs conflict at same loop device. > This problem is fixed at lock mechanism before two weeks. > > If would you like to download new patches ?My guess is that the problem is not in Xen itself, but in some configuration issue (I''m also wondering if the udev and hotplug packages included in SLES9SP2 are a bit dated). Take into account that I have no problem creating a domain with multiple images using the same Xen 3.0.0 on a different machine (with Ubuntu Breezy Badger, which is relatively recent). Cheers! -- ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: Borja Sotomayor, University of Chicago Ph.D. Student, Department of Computer Science Hinds 016, 5734 South Ellis Avenue, 60637 Chicago, IL GT4 Tutorial: http://gdp.globus.org/gt4-tutorial/ BorjaNet: http://www.borjanet.com/ borja@borjanet.com ····························································· "Dis maschine vill run und run!" -- Kurt Gödel (on the Turing Machine) ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Satoshi Uchida
2005-Dec-15 02:56 UTC
RE: [Xen-users] Device 2049 (vbd) could not be connected. Backenddevice not found.
> > My guess is that the problem is not in Xen itself, but in someconfiguration issue> (I''m also wondering if the udev and hotplug packages included inSLES9SP2 are> a bit dated). Take into account that I have no problem creating adomain with multiple> images using the same Xen 3.0.0 on a different machine (with UbuntuBreezy Badger> , which is relatively recent). >Hi. Ok, I miss my comprehension.>Dec 14 15:45:00 tp-c120 logger: /etc/xen/scripts/block: > Writing backend/vbd/1/2050/node/dev/loop1 to xenstore.>Dec 14 15:45:00 tp-c120 logger: /etc/xen/scripts/block: losetup > /dev/loop1/vm/vm_images/vosg/GK/osg-gk-base.img failed But, I see that missing to allocate image file into loop back device. your log shaw VBD 2050 ( /vm/vm_images/vosg/GK/swap1.img ) is allocated loop1 and, VBD 2049 ( file:/vm/vm_images/vosg/GK/osg-gk-base.img ) is trying to allocate loop1. Thank you. Satoshi UCHIDA _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Brian Lavender
2005-Dec-15 03:57 UTC
RE: [Xen-users] Device 2049 (vbd) could not be connected. Backenddevice not found.
On Thu, 2005-12-15 at 11:56 +0900, Satoshi Uchida wrote:> > > > My guess is that the problem is not in Xen itself, but in some > configuration issue > > (I''m also wondering if the udev and hotplug packages included in > SLES9SP2 are > > a bit dated). Take into account that I have no problem creating a > domain with multiple > > images using the same Xen 3.0.0 on a different machine (with Ubuntu > Breezy Badger > > , which is relatively recent). > > > > Hi. > > Ok, I miss my comprehension. > > >Dec 14 15:45:00 tp-c120 logger: /etc/xen/scripts/block: > > Writing backend/vbd/1/2050/node > /dev/loop1 to xenstore. > >Dec 14 15:45:00 tp-c120 logger: /etc/xen/scripts/block: losetup > > /dev/loop1 > /vm/vm_images/vosg/GK/osg-gk-base.img failed > > But, I see that missing to allocate image file into loop back device. > your log shaw VBD 2050 ( /vm/vm_images/vosg/GK/swap1.img ) is allocated > loop1 and, > VBD 2049 ( file:/vm/vm_images/vosg/GK/osg-gk-base.img ) is trying to > allocate loop1. >When the domain fails, try this. for i in /dev/loop* do losetup $i done If you don''t have any VM''s running, and you see a loopback device connected, it could be that somewhere in Xen (I believe xenstored) is not releasing the losetup connection to the image file. I have seen some problems with this in testing. You can manually delete the connection using losetup -d /dev/loop<number that is connected to your image file> brian -- Brian Lavender <brian@xensource.com> _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Borja Sotomayor
2005-Dec-15 21:18 UTC
Re: [Xen-users] Device 2049 (vbd) could not be connected. Backenddevice not found.
Hi,> When the domain fails, try this. > > for i in /dev/loop* > do > losetup $i > done > > If you don''t have any VM''s running, and you see a loopback device > connected, it could be that somewhere in Xen (I believe xenstored) is > not releasing the losetup connection to the image file. I have seen some > problems with this in testing. You can manually delete the connection > usingMy problem was different, because there were loopback devices that weren''t being connected at all when creating a new domain. However, you did nudge me in the right direction. losetup revealed that my system has 16 /dev/loop''s (loop0...loop15), but the kernel only supports the default 8. When creating the domain, Xen correctly connected to loop0 and loop1, but when checking for the "next available loop device", Xen picked loop10 (instead of loop2), which resulted in the error. Setting max_loop to 16 solved my problems :-) Thanks! Cheers, -- ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: Borja Sotomayor, University of Chicago Ph.D. Student, Department of Computer Science Hinds 016, 5734 South Ellis Avenue, 60637 Chicago, IL GT4 Tutorial: http://gdp.globus.org/gt4-tutorial/ BorjaNet: http://www.borjanet.com/ borja@borjanet.com ····························································· "Dis maschine vill run und run!" -- Kurt Gödel (on the Turing Machine) ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users