Xen users - Apr 2010 - domu does not starts after dom0 reboot

Hi,
I am having a Xen dom0 running created 5 DomU and they all were
running successfully.
I rebooted 3 of DomU were up.2 did not come up.This is a frequent
situation with me when I reboot some dom''s do not start. While some
do.
What is the root cause of this I know I can always open command prompt
and type xm create domu.cfg
some where on Xen wiki I read to autostart domains always you need a
softlink /etc/xen/auto
 I do not have any soft link neither do I want to have.
Then in this case why did some domains started and some failed to
start.The behavior of all doms should be same.Either after reboot they
don''t start any one or all are start.I do not want a fix for this I
want to know the root cause of why is it happening.
-- 
Not sent from my iPhone or my Blackberry or anyone else''s

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> Hi,
> I am having a Xen dom0 running created 5 DomU and they all were
> running successfully.
> I rebooted 3 of DomU were up.2 did not come up.This is a frequent
> situation with me when I reboot some dom''s do not start. While
some
> do.
> What is the root cause of this I know I can always open command prompt
> and type xm create domu.cfg
> some where on Xen wiki I read to autostart domains always you need a
> softlink /etc/xen/auto
>  I do not have any soft link neither do I want to have.
> Then in this case why did some domains started and some failed to
> start.The behavior of all doms should be same.Either after reboot they
> don''t start any one or all are start.I do not want a fix for this
I
> want to know the root cause of why is it happening.
Hello,

In my thoughts "xm create" does not create a "permanent"
machine, I use
always "xm new " and "xm start" to create (permanently) and
launch a new
VM.

regards 



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Okay thanks nothing better.



-- 
Not sent from my iPhone or my Blackberry or anyone else''s

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Fri, Apr 30, 2010 at 5:28 PM, Tapas Mishra <mightydreams@gmail.com>
wrote:
> Hi,
> I am having a Xen dom0 running created 5 DomU and they all were
> running successfully.
> I rebooted 3 of DomU were up.2 did not come up.
Did the 3 domUs restarted, or are they completely unaware that dom0
was rebooted? My guess is it''s the second.
> This is a frequent
> situation with me when I reboot some dom''s do not start. While
some
> do.
> What is the root cause of this I know I can always open command prompt
> and type xm create domu.cfg
> some where on Xen wiki I read to autostart domains always you need a
> softlink /etc/xen/auto
>  I do not have any soft link neither do I want to have
It''s like going to the docter and say "doctor, my head is
bleeding,
but I don''t want you to stich it".

While it''s perfectly your right to do that, it''ll reduce the
chance
that others will want to help you. And even those who wants to will
probably be unable to due to your predefined restrictions.
> Then in this case why did some domains started and some failed to
> start.The behavior of all doms should be same.Either after reboot they
> don''t start any one or all are start.I do not want a fix for this
I
> want to know the root cause of why is it happening.
Here''s some things you might want to look at:
- some domUs startup are managed by files/sysmlinks to their config
file in /etc/xen/auto. Others are managed by xend.
- some distros default to shutdown domU and start it later, while
others default to SAVE and restore domUs later.

My guess is that you have xend-managed domUs, defaults to save domUs
during dom0 shutdown, but don''t have the necessary disk space to save
all domUs memory. xend.log should give you an idea of what''s wrong,
look for entries around the time dom0 is restarted.

-- 
Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Mon, May 3, 2010 at 9:07 AM, Fajar A. Nugraha <fajar@fajar.net>
wrote:
> On Fri, Apr 30, 2010 at 5:28 PM, Tapas Mishra
<mightydreams@gmail.com> wrote:
>> Hi,
>> I am having a Xen dom0 running created 5 DomU and they all were
>> running successfully.
>> I rebooted 3 of DomU were up.2 did not come up.
>
> Did the 3 domUs restarted, or are they completely unaware that dom0
> was rebooted? My guess is it''s the second.
I could not get your second suggestion.
DomU''s did restart or not I am not clear I was able to SSH into the
three 3DomUs immediately after Dom0 was in a position to give me an
SSH login.
I am accessing all Dom0 and DomU''s by ssh only.
>> This is a frequent
>> situation with me when I reboot some dom''s do not start. While
some
>> do.
>> What is the root cause of this I know I can always open command prompt
>> and type xm create domu.cfg
>> some where on Xen wiki I read to autostart domains always you need a
>> softlink /etc/xen/auto
>>  I do not have any soft link neither do I want to have
>
> It''s like going to the docter and say "doctor, my head is
bleeding,
> but I don''t want you to stich it".
>
> While it''s perfectly your right to do that, it''ll reduce
the chance
> that others will want to help you. And even those who wants to will
> probably be unable to due to your predefined restrictions.
Its not that I don''t want to stitch the problem is when I installed
Xen then there was not auto directory created.I don''t have any problem
what  I wanted to convey is I wanted to handle the current thing as it
is.
>> Then in this case why did some domains started and some failed to
>> start.The behavior of all doms should be same.Either after reboot they
>> don''t start any one or all are start.I do not want a fix for
this I
>> want to know the root cause of why is it happening.
>
> Here''s some things you might want to look at:
> - some domUs startup are managed by files/sysmlinks to their config
> file in /etc/xen/auto. Others are managed by xend.
> - some distros default to shutdown domU and start it later, while
> others default to SAVE and restore domUs later.
>
> My guess is that you have xend-managed domUs, defaults to save domUs
> during dom0 shutdown,
This seems to be relevant how do I make sure this.
> but don''t have the necessary disk space to save
> all domUs memory. xend.log should give you an idea of what''s
wrong,
> look for entries around the time dom0 is restarted.


xend.log has only the following there is one more file xend.log.1

[2010-04-29 06:38:17 3305] DEBUG (XendDomainInfo:1008) Storing domain
details: {''image/entry'':
''18446744071564165120'', ''console/port'':
''2'',
''image/loader'': ''generic'',
''vm'':
''/vm/75a4a665-3b0b-d9d8-81f9-f521fe52fe6a'',
''control/platform-feature-multiprocessor-suspend'':
''1'',
''image/guest-os'': ''linux'',
''image/features/writable-descriptor-tables'':
''1'', ''image/virt-base'':
''18446744071562067968'', ''memory/target'':
''2097152'',
''image/guest-version'': ''2.6'',
''image/features/supervisor-mode-kernel'':
''1'', ''console/limit'':
''1048576'', ''image/paddr-offset'':
''0'',
''image/hypercall-page'':
''18446744071564197888'',
''image/suspend-cancel'': ''1'',
''cpu/0/availability'': ''online'',
''image/features/pae-pgdir-above-4gb'': ''1'',
''image/features/writable-page-tables'': ''1'',
''console/type'':
''xenconsoled'',
''image/features/auto-translated-physmap'':
''1'', ''name'':
''software'', ''domid'': ''4'',
''image/xen-version'': ''xen-3.0'',
''store/port'': ''1''}
[2010-04-29 06:38:17 3305] DEBUG (XendCheckpoint:218)
restore:shadow=0x0, _static_max=0x80000000, _static_min=0x0,
[2010-04-29 06:38:17 3305] DEBUG (balloon:132) Balloon: 2488304 KiB
free; need 2097152; done.
[2010-04-29 06:38:17 3305] DEBUG (XendCheckpoint:235) [xc_restore]:
/usr/lib/xen-3.2-1/bin/xc_restore 4 4 1 2 0 0 0
[2010-04-29 06:38:17 3305] INFO (XendCheckpoint:373) xc_domain_restore
start: p2m_size = 80800
[2010-04-29 06:38:17 3305] INFO (XendCheckpoint:373) Reloading memory
pages:   0%
[2010-04-29 06:38:21 3305] INFO (XendCheckpoint:373) ERROR Internal
error: Error when reading page (type was 0)
[2010-04-29 06:38:22 3305] INFO (XendCheckpoint:373) Restore exit with rc=1
[2010-04-29 06:38:22 3305] DEBUG (XendDomainInfo:1897)
XendDomainInfo.destroy: domid=4
[2010-04-29 06:38:22 3305] DEBUG (XendDomainInfo:1914)
XendDomainInfo.destroyDomain(4)
[2010-04-29 06:38:22 3305] ERROR (XendDomainInfo:1926)
XendDomainInfo.destroy: xc.domain_destroy failed.
Traceback (most recent call last):
  File "/usr/lib/xen-3.2-1/lib/python/xen/xend/XendDomainInfo.py",
line 1921, in destroyDomain
    xc.domain_destroy(self.domid)
Error: (3, ''No such process'')
[2010-04-29 06:38:22 3305] DEBUG (XendDomainInfo:1534) No device model
[2010-04-29 06:38:22 3305] DEBUG (XendDomainInfo:1536) Releasing devices
[2010-04-29 06:38:22 3305] DEBUG (XendDomainInfo:1542) Removing vif/0
[2010-04-29 06:38:22 3305] DEBUG (XendDomainInfo:590)
XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/0
[2010-04-29 06:38:22 3305] DEBUG (XendDomainInfo:1542) Removing vbd/51713
[2010-04-29 06:38:22 3305] DEBUG (XendDomainInfo:590)
XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/51713
[2010-04-29 06:38:22 3305] DEBUG (XendDomainInfo:1542) Removing vbd/51714
[2010-04-29 06:38:22 3305] DEBUG (XendDomainInfo:590)
XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/51714
[2010-04-29 06:38:22 3305] DEBUG (XendDomainInfo:1542) Removing console/0
[2010-04-29 06:38:22 3305] DEBUG (XendDomainInfo:590)
XendDomainInfo.destroyDevice: deviceClass = console, device console/0
[2010-04-29 06:38:22 3305] ERROR (XendDomain:1136) Restore failed
Traceback (most recent call last):
  File "/usr/lib/xen-3.2-1/lib/python/xen/xend/XendDomain.py", line
1134, in domain_restore_fd
    return XendCheckpoint.restore(self, fd, paused=paused)
  File "/usr/lib/xen-3.2-1/lib/python/xen/xend/XendCheckpoint.py",
line 239, in restore
    forkHelper(cmd, fd, handler.handler, True)
  File "/usr/lib/xen-3.2-1/lib/python/xen/xend/XendCheckpoint.py",
line 361, in forkHelper
    raise XendError("%s failed" % string.join(cmd))
XendError: /usr/lib/xen-3.2-1/bin/xc_restore 4 4 1 2 0 0 0 failed



-- 
Not sent from my iPhone or my Blackberry or anyone else''s

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Mon, May 3, 2010 at 11:14 AM, Tapas Mishra <mightydreams@gmail.com>
wrote:
>> Did the 3 domUs restarted, or are they completely unaware that dom0
>> was rebooted? My guess is it''s the second.
> I could not get your second suggestion.
> DomU''s did restart or not I am not clear I was able to SSH into
the
> three 3DomUs immediately after Dom0 was in a position to give me an
> SSH login.
> I am accessing all Dom0 and DomU''s by ssh only.
If you access your domU and run "uptime", it''s possible that
domU''s
uptime is higher than dom0s. This might happen because domUs can be
migrated and save-restored.
> [2010-04-29 06:38:17 3305] DEBUG (XendCheckpoint:235) [xc_restore]:
> /usr/lib/xen-3.2-1/bin/xc_restore 4 4 1 2 0 0 0
> [2010-04-29 06:38:17 3305] INFO (XendCheckpoint:373) xc_domain_restore
> start: p2m_size = 80800
> [2010-04-29 06:38:17 3305] INFO (XendCheckpoint:373) Reloading memory
> pages:   0%
> [2010-04-29 06:38:21 3305] INFO (XendCheckpoint:373) ERROR Internal
> error: Error when reading page (type was 0)
This looks similar to the logs I get when xend tries to a restore an
incomplete-saved domain.

On RHEL, check /etc/sysconfig/xendomains (might be present on other
distros as well), there''s a line

XENDOMAINS_SAVE=/var/lib/xen/save

that will save domUs on dom0 shutdown on that directory. I prefer to
simply comment-it-out to tell xend to shutdown domUs instead of saving
them.

You can start by checking if that file exists, or see if your distro
has /etc/init.d/xendomains (or similar) and see what config file it
uses.

-- 
Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> This looks similar to the logs I get when xend tries to a restore an
> incomplete-saved domain.
>
> On RHEL, check /etc/sysconfig/xendomains (might be present on other
> distros as well), there''s a line
Well I am using Debian and there is no directory named sysconfig :)
any other idea
> XENDOMAINS_SAVE=/var/lib/xen/save
> that will save domUs on dom0 shutdown on that directory. I prefer to
> simply comment-it-out to tell xend to shutdown domUs instead of saving
> them.
>
> You can start by checking if that file exists, or see if your distro
> has /etc/init.d/xendomains (or similar) and see what config file it
> uses.
In the script /etc/init.d/xendomains there was a variable  XENDOMAINS_SAVE
but where is it defined.
> --
> Fajar
>


-- 
Not sent from my iPhone or my Blackberry or anyone else''s

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Mon, May 3, 2010 at 11:35 AM, Tapas Mishra <mightydreams@gmail.com>
wrote:
>> On RHEL, check /etc/sysconfig/xendomains (might be present on other
>> distros as well), there''s a line
> Well I am using Debian and there is no directory named sysconfig :)
> any other idea
> In the script /etc/init.d/xendomains there was a variable  XENDOMAINS_SAVE
> but where is it defined.
try looking in /etc/default/xendomains

-- 
Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> try looking in /etc/default/xendomains
 yes there was XENDOMAINS_SAVE=/var/lib/xen/save
You mean to say I should disable this.
But even then I am not able to get to the root of problem.If I assume
what you suggested is right.
I have 8 GB of memory and before creating VMs I had calculated memory
and swap for all the VMs.So to me it does not appear that the memory
is unavailable to Doms to restart.
The dom0 while rebooting is shutting down and trying to save VMs that
is what I understand from this discussion and it is not able to create
guest Doms
In /var/log/xen there are following files

/var/log/xen# ls
domain-builder-ng.log  xend-debug.log  xend.log  xend.log.1  xen-hotplug.log
tell me if any of these can help I will pastebin them.





-- 
Not sent from my iPhone or my Blackberry or anyone else''s

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Mon, May 3, 2010 at 11:54 AM, Tapas Mishra <mightydreams@gmail.com>
wrote:
>> try looking in /etc/default/xendomains
>  yes there was XENDOMAINS_SAVE=/var/lib/xen/save
> You mean to say I should disable this.
> But even then I am not able to get to the root of problem.If I assume
> what you suggested is right.
> I have 8 GB of memory and before creating VMs I had calculated memory
> and swap for all the VMs.So to me it does not appear that the memory
> is unavailable to Doms to restart.
I never said there was not enough memory. I said you "don''t have
the
necessary disk space to save
all domUs memory".

Again, this is still just a guess, but most likely your /var (or
whatever partition /var/lib/xen/save is on) has less than 8 GB free
space.

-- 
Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Ok thanks for your help.

On Mon, May 3, 2010 at 10:38 AM, Fajar A. Nugraha <fajar@fajar.net>
wrote:
> On Mon, May 3, 2010 at 11:54 AM, Tapas Mishra
<mightydreams@gmail.com> wrote:
>>> try looking in /etc/default/xendomains
>>  yes there was XENDOMAINS_SAVE=/var/lib/xen/save
>> You mean to say I should disable this.
>> But even then I am not able to get to the root of problem.If I assume
>> what you suggested is right.
>> I have 8 GB of memory and before creating VMs I had calculated memory
>> and swap for all the VMs.So to me it does not appear that the memory
>> is unavailable to Doms to restart.
>
> I never said there was not enough memory. I said you "don''t
have the
> necessary disk space to save
> all domUs memory".
>
> Again, this is still just a guess, but most likely your /var (or
> whatever partition /var/lib/xen/save is on) has less than 8 GB free
> space.
>
> --
> Fajar
>


-- 
Tapas

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users