Hi XEN Users and Developers, xend does not start into a clean state. The process is running...> #ps -efl | grep xen > 1 S root 20 19 0 70 -5 - 0 xenwat Mar06 ? 00:00:00 [xenwatch] > 1 S root 21 19 0 70 -5 - 0 xb_rea Mar06 ? 00:00:00 [xenbus] > 5 S root 12663 1 0 76 0 - 1274 - 09:56 ? 00:00:00 xenstored --pid-file /var/run/xenstore.pid > 5 S root 12667 1 0 84 0 - 3552 stext 09:56 ? 00:00:00 xenconsoled > 1 S root 12671 1 0 79 0 - 13265 wait 09:56 ? 00:00:00 python /usr/sbin/xend start > 5 S root 12673 12671 0 76 0 - 15412 184466 09:56 ? 00:00:00 python /usr/sbin/xend start...but no virtual machine can be started and xm answers e.g with this:> # xm list > Error: Unable to connect to xend: No such file or directory. Is xend running?In the xend.log there is the following error:> [2007-03-07 09:56:13 xend 12659] WARNING (XendAuthSessions:27) python-pam is required for XenAPI support. > [2007-03-07 09:56:13 xend 12673] INFO (SrvDaemon:322) Xend Daemon started > [2007-03-07 09:56:13 xend 12673] INFO (SrvDaemon:326) Xend changeset: unavailable. > [2007-03-07 09:56:13 xend 12673] INFO (SrvDaemon:333) Xend version: Unknown. > [2007-03-07 09:56:13 xend.XendDomainInfo 12673] DEBUG (XendDomainInfo:121) XendDomainInfo.recreate({''max_vcpu_id'': 3, ''cpu_time'': 11 > 74593321820L, ''ssidref'': 0, ''hvm'': 0, ''shutdown_reason'': 0, ''dying'': 0, ''online_vcpus'': 4, ''domid'': 0, ''paused'': 0, ''crashed'': 0, ''r > unning'': 1, ''maxmem_kb'': 17179869180, ''shutdown'': 0, ''mem_kb'': 3985144, ''handle'': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], > ''blocked'': 0, ''name'': ''Domain-0''}) > [2007-03-07 09:56:13 xend.XendDomainInfo 12673] INFO (XendDomainInfo:135) Recreating domain 0, UUID 00000000-0000-0000-0000-00000000 > 0000. at /local/domain/0 > [2007-03-07 09:56:13 xend.XendDomainInfo 12673] WARNING (XendDomainInfo:149) /local/domain/0/vm is missing. recreate is confused, tr > ying our best to recover > [2007-03-07 09:56:13 xend.XendDomainInfo 12673] DEBUG (XendDomainInfo:1714) Storing VM details: {''on_xend_stop'': ''ignore'', ''shadow_m > emory'': ''0'', ''uuid'': ''00000000-0000-0000-0000-000000000000'', ''name'': ''Domain-0'', ''on_reboot'': ''restart'', ''on_poweroff'': ''destroy'', '' > on_xend_start'': ''ignore'', ''on_crash'': ''restart'', ''xend/restart_count'': ''0'', ''vcpus'': ''4'', ''vcpu_avail'': ''15'', ''memory'': ''3892'', ''max > mem'': ''16777216''} > [2007-03-07 09:56:13 xend 12673] ERROR (SrvDaemon:344) Exception starting xend ((28, ''No space left on device, while writing /vm/000 > 00000-0000-0000-0000-000000000000/xend/restart_count : 0'')) > Traceback (most recent call last): > File "//usr/lib64/python/xen/xend/server/SrvDaemon.py", line 336, in run > servers = SrvServer.create() > File "//usr/lib64/python/xen/xend/server/SrvServer.py", line 226, in create > root.putChild(''xend'', SrvRoot()) > File "//usr/lib64/python/xen/xend/server/SrvRoot.py", line 40, in __init__ > self.get(name) > File "//usr/lib64/python/xen/web/SrvDir.py", line 82, in get > val = val.getobj() > File "//usr/lib64/python/xen/web/SrvDir.py", line 52, in getobj > self.obj = klassobj() > File "//usr/lib64/python/xen/xend/server/SrvDomainDir.py", line 41, in __init__ > self.xd = XendDomain.instance() > File "//usr/lib64/python/xen/xend/XendDomain.py", line 1399, in instance > inst.init() > File "//usr/lib64/python/xen/xend/XendDomain.py", line 116, in init > dom0 = XendDomainInfo.recreate(dom0info, True) > File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 174, in recreate > vm._storeVmDetails() > File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 1716, in _storeVmDetails > self._writeVm(to_store) > File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 672, in _writeVm > return xstransact.Write(self.vmpath, *args) > File "//usr/lib64/python/xen/xend/xenstore/xstransact.py", line 302, in Write > complete(path, lambda t: t.write(*args)) > File "//usr/lib64/python/xen/xend/xenstore/xstransact.py", line 354, in complete > result = f(t) > File "//usr/lib64/python/xen/xend/xenstore/xstransact.py", line 302, in <lambda> > complete(path, lambda t: t.write(*args)) > File "//usr/lib64/python/xen/xend/xenstore/xstransact.py", line 87, in write > self._write(key, d[key]) > File "//usr/lib64/python/xen/xend/xenstore/xstransact.py", line 75, in _write > (''%s, while writing %s : %s'' % > RuntimeError: (28, ''No space left on device, while writing /vm/00000000-0000-0000-0000-000000000000/xend/restart_count : 0'') > [2007-03-07 09:56:13 xend 12675] WARNING (XendAuthSessions:27) python-pam is required for XenAPI support.Does somebody know a cure for this behaviour? With regards, Oli -- ///////////////////////////////// Oliver Heil DKFZ - Deutsches Krebsforschungszentrum Im Neuenheimer Feld 515 69120 Heidelberg Tel (+49 6221) / 42 4708 Email o.heil@dkfz.de WWW www.dkfz.de ///////////////////////////////// -----BEGIN PGP PUBLIC KEY BLOCK----- Version: GnuPG v1.0.6 (MingW32) mQGiBDyzHxERBACFUsv9ZzMAstEUKpV7ZI+//CqMwNWteCdcet8ONYQkE+meq7qp L2yQgqdGUdlRz/O0Ix9ERuMw+6qnRYxXCGvRjWX6t+wDSALfFjjEIUVecedsMKGv 7bINfUQEJQ+7hZAonOJ/iEL3adh8fvl1Bl0yNYxATbqp+q1YGawsHLu4TwCg10VR AkS9zZnv1gsGBg2rDm8f2CcD/0siN0AUvjvj38Hzxr7CCG22c5QYHZLhFvM7+jSA OF2qTNGtFJ0fVkIlqzkd1D21efmM2xKo9Jzlhb5D1Dr1GQ7epxvpcGV/n8vHtwOv +N8UoOaNfrJIoM9lV4IXKTAnwlTcyy61Bthx4L+dKNTH0ejbPk3/SIUlYAju6V9y uqOCA/0euPYCj0GQ4YfdULUu/ZTLOAX3X+fUXqSQThBUt8BpTv/wqNqSifURttH9 ERb+Q22E9ck5mYs2zZmd2xvmxU6swFqJJVZMBzs1/iQcc7mC1v84Riddcuj+C4YA whph8LxHAdFPN+ZHoaQTIu8vC0KdtHJX88KNxvl7HVYL8x9yW7QcT2xpdmVyIEhl aWwgPG8uaGVpbEBka2Z6LmRlPohXBBMRAgAXBQI8sx8RBQsHCgMEAxUDAgMWAgEC F4AACgkQ87yOFfCjArRQUgCgqL5irUC6wdZntArL+8n19RWQWrMAn32Dee0njHm1 2qTW27TcBSPAWGwSuQINBDyzH18QCADC8Em8AKrM1QrBR1qpmb9kN+KmF5FZDwXP 4ioO4k0ee2cfcriUiZJXeNo/TpUvGpGRhINMyCKDpiXeU970Gxhy6ksU+q+hEqFZ nAqYuq00xQQ+SxMb+TgGytgmq3Jz8q8jJFokkd3Mc0mcicPKlKNv4Sq+sGAzHXTk in/XJRTB7RA7zl7K+iz+u4lnXyyhM+cT82EtduQ2XvPGYjQwymLMcIRhoWlbvB1w 1uGGCaOQSdlH36LEG+P41TwMjB4Eq2dhk9PL+6pUpbRXrtWtnUzfQuEFwHlMwITv RCit6Pp+5f2USyWeUP0oc/39I0iTOqINZ7S031FD7xa61vtMK01bAAMGCAC2sX9z vormmdWvHWyH5+d9PiYUS86JNy8ncVqgoLtin6fqpy0UlGMHH2ewfpExfVip8CWj Cp7vzpmzhsu+wIDyVdAVx3OldKFUH7Al3PDdMe4hNkfB0om+DDAYPO33C7PhpcUL +Muz14fc6ti5PKs8zNcqENGyL0sv/Uun28Ebb9cOHWfEcgJ8ZbfmLiW8b2vXYFPj F6T0+tOTCSNDuWGBAWfvumOhZB6sFokPUxs+5CqibGTgGtCUYW1VXVTu+d3bugMN NcuQBucGKdq7v6rREgfNfqvfodyk77Zy9w9wbGvPROUe77PaYubdRviiCiMYfQQc 501JeSIwwARsSt5viEYEGBECAAYFAjyzH18ACgkQ87yOFfCjArSc9ACdEY4YqlqM lIuB4uWb6XZ3DTTUs3QAoL+a4UJp9OLLVtjTsijb/hpu3uhO =I3OA -----END PGP PUBLIC KEY BLOCK----- _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Oliver Heil
2007-Mar-08 13:56 UTC
Re: [Xen-users] SOLUTION FOUND FOR xend does not start "No space left..."
Hi, I finally found a solution for my problem. I still do not know, what the exact problem was. The symptoms are, that the xen dom0, Domain-0 does not start and that e.g. xm asks, if xend is running, but xend is in the process list. >> # xm list >> Error: Unable to connect to xend: No such file or directory. Is xend >> running? The xend.log file reports error messages like > RuntimeError: (28, ''No space left on device, while writing /vm/00000000-0000-0000-0000- > 000000000000/xend/restart_count : 0'') but every device and memory has plenty space available. The system is a openSUSE 10.2 on 2 processor XEON 64bit machine with 4Gbyte RAM. XEN is build from source xen-3.0.4_1 To workaround and to get xen back again I just removed the directories /var/lib/xen, /var/lib/xend and /var/lib/xenstored. In /var/lib/xen/save was 1 Gbyte file named like the only guest virtual machine I am running on the host: /var/lib/xen/save/vm01 This is propably a memory image of the guest vm and I believe that the existance of this file was the cause of the trouble. There are a few other threads unanswered in the net, which maybe solved the same way. Here are some of the subjects I found with google: [Fedora-xen] Xen0 not booting into Login console [Fedora-xen] Exception when starting xend: error while reading /vm/00000000-0000-0000-0000-000000000000/ssidref Sincerelly, Oli Oliver Heil wrote:> Hi XEN Users and Developers, > > xend does not start into a clean state. The process is running... > >> #ps -efl | grep xen >> 1 S root 20 19 0 70 -5 - 0 xenwat Mar06 ? >> 00:00:00 [xenwatch] >> 1 S root 21 19 0 70 -5 - 0 xb_rea Mar06 ? >> 00:00:00 [xenbus] >> 5 S root 12663 1 0 76 0 - 1274 - 09:56 ? >> 00:00:00 xenstored --pid-file /var/run/xenstore.pid >> 5 S root 12667 1 0 84 0 - 3552 stext 09:56 ? >> 00:00:00 xenconsoled >> 1 S root 12671 1 0 79 0 - 13265 wait 09:56 ? >> 00:00:00 python /usr/sbin/xend start >> 5 S root 12673 12671 0 76 0 - 15412 184466 09:56 ? >> 00:00:00 python /usr/sbin/xend start > > ...but no virtual machine can be started and xm answers e.g with this: >> # xm list >> Error: Unable to connect to xend: No such file or directory. Is xend >> running? > > In the xend.log there is the following error: >> [2007-03-07 09:56:13 xend 12659] WARNING (XendAuthSessions:27) >> python-pam is required for XenAPI support. >> [2007-03-07 09:56:13 xend 12673] INFO (SrvDaemon:322) Xend Daemon started >> [2007-03-07 09:56:13 xend 12673] INFO (SrvDaemon:326) Xend changeset: >> unavailable. >> [2007-03-07 09:56:13 xend 12673] INFO (SrvDaemon:333) Xend version: >> Unknown. >> [2007-03-07 09:56:13 xend.XendDomainInfo 12673] DEBUG >> (XendDomainInfo:121) XendDomainInfo.recreate({''max_vcpu_id'': 3, >> ''cpu_time'': 11 >> 74593321820L, ''ssidref'': 0, ''hvm'': 0, ''shutdown_reason'': 0, ''dying'': >> 0, ''online_vcpus'': 4, ''domid'': 0, ''paused'': 0, ''crashed'': 0, ''r >> unning'': 1, ''maxmem_kb'': 17179869180, ''shutdown'': 0, ''mem_kb'': >> 3985144, ''handle'': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], >> ''blocked'': 0, ''name'': ''Domain-0''}) >> [2007-03-07 09:56:13 xend.XendDomainInfo 12673] INFO >> (XendDomainInfo:135) Recreating domain 0, UUID >> 00000000-0000-0000-0000-00000000 >> 0000. at /local/domain/0 >> [2007-03-07 09:56:13 xend.XendDomainInfo 12673] WARNING >> (XendDomainInfo:149) /local/domain/0/vm is missing. recreate is >> confused, tr >> ying our best to recover >> [2007-03-07 09:56:13 xend.XendDomainInfo 12673] DEBUG >> (XendDomainInfo:1714) Storing VM details: {''on_xend_stop'': ''ignore'', >> ''shadow_m >> emory'': ''0'', ''uuid'': ''00000000-0000-0000-0000-000000000000'', ''name'': >> ''Domain-0'', ''on_reboot'': ''restart'', ''on_poweroff'': ''destroy'', '' >> on_xend_start'': ''ignore'', ''on_crash'': ''restart'', ''xend/restart_count'': >> ''0'', ''vcpus'': ''4'', ''vcpu_avail'': ''15'', ''memory'': ''3892'', ''max >> mem'': ''16777216''} >> [2007-03-07 09:56:13 xend 12673] ERROR (SrvDaemon:344) Exception >> starting xend ((28, ''No space left on device, while writing /vm/000 >> 00000-0000-0000-0000-000000000000/xend/restart_count : 0'')) >> Traceback (most recent call last): >> File "//usr/lib64/python/xen/xend/server/SrvDaemon.py", line 336, in >> run >> servers = SrvServer.create() >> File "//usr/lib64/python/xen/xend/server/SrvServer.py", line 226, in >> create >> root.putChild(''xend'', SrvRoot()) >> File "//usr/lib64/python/xen/xend/server/SrvRoot.py", line 40, in >> __init__ >> self.get(name) >> File "//usr/lib64/python/xen/web/SrvDir.py", line 82, in get >> val = val.getobj() >> File "//usr/lib64/python/xen/web/SrvDir.py", line 52, in getobj >> self.obj = klassobj() >> File "//usr/lib64/python/xen/xend/server/SrvDomainDir.py", line 41, >> in __init__ >> self.xd = XendDomain.instance() >> File "//usr/lib64/python/xen/xend/XendDomain.py", line 1399, in >> instance >> inst.init() >> File "//usr/lib64/python/xen/xend/XendDomain.py", line 116, in init >> dom0 = XendDomainInfo.recreate(dom0info, True) >> File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 174, in >> recreate >> vm._storeVmDetails() >> File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 1716, in >> _storeVmDetails >> self._writeVm(to_store) >> File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 672, in >> _writeVm >> return xstransact.Write(self.vmpath, *args) >> File "//usr/lib64/python/xen/xend/xenstore/xstransact.py", line 302, >> in Write >> complete(path, lambda t: t.write(*args)) >> File "//usr/lib64/python/xen/xend/xenstore/xstransact.py", line 354, >> in complete >> result = f(t) >> File "//usr/lib64/python/xen/xend/xenstore/xstransact.py", line 302, >> in <lambda> >> complete(path, lambda t: t.write(*args)) >> File "//usr/lib64/python/xen/xend/xenstore/xstransact.py", line 87, >> in write >> self._write(key, d[key]) >> File "//usr/lib64/python/xen/xend/xenstore/xstransact.py", line 75, >> in _write >> (''%s, while writing %s : %s'' % >> RuntimeError: (28, ''No space left on device, while writing >> /vm/00000000-0000-0000-0000-000000000000/xend/restart_count : 0'') >> [2007-03-07 09:56:13 xend 12675] WARNING (XendAuthSessions:27) >> python-pam is required for XenAPI support. > > Does somebody know a cure for this behaviour? > > With regards, > > Oli >-- ///////////////////////////////// Oliver Heil DKFZ - Deutsches Krebsforschungszentrum Im Neuenheimer Feld 515 69120 Heidelberg Tel (+49 6221) / 42 4708 Email o.heil@dkfz.de WWW www.dkfz.de ///////////////////////////////// -----BEGIN PGP PUBLIC KEY BLOCK----- Version: GnuPG v1.0.6 (MingW32) mQGiBDyzHxERBACFUsv9ZzMAstEUKpV7ZI+//CqMwNWteCdcet8ONYQkE+meq7qp L2yQgqdGUdlRz/O0Ix9ERuMw+6qnRYxXCGvRjWX6t+wDSALfFjjEIUVecedsMKGv 7bINfUQEJQ+7hZAonOJ/iEL3adh8fvl1Bl0yNYxATbqp+q1YGawsHLu4TwCg10VR AkS9zZnv1gsGBg2rDm8f2CcD/0siN0AUvjvj38Hzxr7CCG22c5QYHZLhFvM7+jSA OF2qTNGtFJ0fVkIlqzkd1D21efmM2xKo9Jzlhb5D1Dr1GQ7epxvpcGV/n8vHtwOv +N8UoOaNfrJIoM9lV4IXKTAnwlTcyy61Bthx4L+dKNTH0ejbPk3/SIUlYAju6V9y uqOCA/0euPYCj0GQ4YfdULUu/ZTLOAX3X+fUXqSQThBUt8BpTv/wqNqSifURttH9 ERb+Q22E9ck5mYs2zZmd2xvmxU6swFqJJVZMBzs1/iQcc7mC1v84Riddcuj+C4YA whph8LxHAdFPN+ZHoaQTIu8vC0KdtHJX88KNxvl7HVYL8x9yW7QcT2xpdmVyIEhl aWwgPG8uaGVpbEBka2Z6LmRlPohXBBMRAgAXBQI8sx8RBQsHCgMEAxUDAgMWAgEC F4AACgkQ87yOFfCjArRQUgCgqL5irUC6wdZntArL+8n19RWQWrMAn32Dee0njHm1 2qTW27TcBSPAWGwSuQINBDyzH18QCADC8Em8AKrM1QrBR1qpmb9kN+KmF5FZDwXP 4ioO4k0ee2cfcriUiZJXeNo/TpUvGpGRhINMyCKDpiXeU970Gxhy6ksU+q+hEqFZ nAqYuq00xQQ+SxMb+TgGytgmq3Jz8q8jJFokkd3Mc0mcicPKlKNv4Sq+sGAzHXTk in/XJRTB7RA7zl7K+iz+u4lnXyyhM+cT82EtduQ2XvPGYjQwymLMcIRhoWlbvB1w 1uGGCaOQSdlH36LEG+P41TwMjB4Eq2dhk9PL+6pUpbRXrtWtnUzfQuEFwHlMwITv RCit6Pp+5f2USyWeUP0oc/39I0iTOqINZ7S031FD7xa61vtMK01bAAMGCAC2sX9z vormmdWvHWyH5+d9PiYUS86JNy8ncVqgoLtin6fqpy0UlGMHH2ewfpExfVip8CWj Cp7vzpmzhsu+wIDyVdAVx3OldKFUH7Al3PDdMe4hNkfB0om+DDAYPO33C7PhpcUL +Muz14fc6ti5PKs8zNcqENGyL0sv/Uun28Ebb9cOHWfEcgJ8ZbfmLiW8b2vXYFPj F6T0+tOTCSNDuWGBAWfvumOhZB6sFokPUxs+5CqibGTgGtCUYW1VXVTu+d3bugMN NcuQBucGKdq7v6rREgfNfqvfodyk77Zy9w9wbGvPROUe77PaYubdRviiCiMYfQQc 501JeSIwwARsSt5viEYEGBECAAYFAjyzH18ACgkQ87yOFfCjArSc9ACdEY4YqlqM lIuB4uWb6XZ3DTTUs3QAoL+a4UJp9OLLVtjTsijb/hpu3uhO =I3OA -----END PGP PUBLIC KEY BLOCK----- _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users