Xen users - May 2010 - xml files in xend database corrupted and xend failed to start solved now but read to diagnose

Hi,
 I am posting this message because past 36 hours have been nightmare for me.

I finally have solved the problem I by temporarily moving put  the xml
files  from /var/lib/xend directories reboot then xend started.
And DomU''s were failing to start the above xml files I kept back and
got all DomU''s back.

I am very very surprised.I do not have any logical answer to this some
one who is reading this thread and has understood and can suggest some
thing as why
 did after I moved  xml files in /var/lib/xend direcotries to /tmp
xend was able to start and
then I again put those files back I got all my DomU''s back.
The solution to above is mentioned here
http://forums.novell.com/novell-product-support-forums/suse-linux-enterprise-server-sles/sles-virtualization/343286-unable-connect-xend.html



Here is the complete description of what had happened and steps I took.

My xen server had crashed yesterday all of a sudden.
I am using Debian Lenny and xen-3.2.1
Yesterday I rebooted my system and found the xen no more was working.

I started searching on net mailed on the list.I did not had any clue
as what is happening.The message I got was similar to a Debian bug
number bug no 502849.

Error: Unable to connect to xend: No such file or directory. Is xend running?
server:~# /etc/init.d/xend start
Starting XEN control daemon: xendFATAL: Could not create tdb file
/var/lib/xenstored/tdb: Input/output error
failed!

but I was unable to understand what to then
I digged a bit more looked at /var/log/messages and found
an error [ 0.377064] map irq failed
it was repeated at many places in the file.After reboot xen fails to start.

So from above message I inferred xend was unable to create the tdb file.


Then looked at  XenStore database
http://wiki.xensource.com/xenwiki/XenIntro#head-4e27782aeef9debb00d812d9041a8e180ca414f0

yes the disk was full at the time I gave this message.
I saw that /var/xen/lib/save consumed all the space with respective
DomU''s
here is a log before I deleted the /var/xen/lib/save

Traceback (most recent call last):
  File "/usr/lib/xen-3.2-1/lib/python/xen/xend/XendDomain.py", line
1134, in domain_restore_fd
    return XendCheckpoint.restore(self, fd, paused=paused)
  File "/usr/lib/xen-3.2-1/lib/python/xen/xend/XendCheckpoint.py",
line 239, in restore
    forkHelper(cmd, fd, handler.handler, True)
  File "/usr/lib/xen-3.2-1/lib/python/xen/xend/XendCheckpoint.py",
line 361, in forkHelper
    raise XendError("%s failed" % string.join(cmd))
XendError: /usr/lib/xen-3.2-1/bin/xc_restore 4 4 1 2 0 0 0 failed


there was not enough space to create the xenstore.
I did freed it up and 2.3 GB is free but it this message was still there.


But still
 /etc/init.d/xend start I get error
Starting XEN control daemon: xend failed!



then I read the novell link deleted all xml files in directories /var/lib/xend
and again did /etc/init.d/xend start it started

I was able to see xend running xm info was  giving output.



But voilla I ran into another problem DomU''s did not started
Code:

server:~# xm create /etc/xen/domu1.cfg
Using config file "/etc/xen/domu1.cfg".
Error: Device 0 (vif) could not be connected. Hotplug scripts not working


Then I put those files back which on web searches were mentioning that
xend database was corrupt and got all Doms back.So what had exactly
happned.
I want to reach the root cause of the problem.

Thanks

-- 
Tapas

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Tue, May 11, 2010 at 7:57 PM, Tapas Mishra <mightydreams@gmail.com>
wrote:
> So what had exactly
> happned.
> I want to reach the root cause of the problem.
Your disk was full. It causes lots of problem. Never let that happen.

SInce you mentioned /var/lib/xen/save was full, it''s possible your
system is setup by default to save domUs on dom0 shutdown. While it''s
a useful feature, it requires lots of disk space, and might add
significant time to dom0 shutdown process. I usually turn it off on my
systems. See /etc/sysconfig/xendomains or /etc/default/xendomains,
look for XENDOMAINS_SAVE. Change it to empty string.

-- 
Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users