CentOS virt - Aug 2008 - Cannot start xen domUs anymore, domUs hang on kernel startup, happens after a long dom0 uptime

Hello!

I've noticed this problem two times now.. last time I fixed it by rebooting
the (centos 5.1 x86 32b) xen host/dom0.

Symptoms:

- Already running domUs (debian 2.6.18-6-xen-686 32b PAE) continue running
  and working OK 

- Cannot start new domUs (debian 2.6.18-6-xen-686).. kernel bootup just
hangs before running initrd. Same domU with the exact same xen domU cfgfile
worked
earlier. 

- This problem starts to happen after a "longer" uptime.. atm uptime
for the
dom0-host is 174 days. Cannot say what the actual "limit" is.. last
time I
used the dom0 everything worked fine, now it doesn't.. 


"xm list" shows the domain as "-b----" and the Time(s) for
the domain does
not increase.. it stays at 1.9 or so. It might increase to 2.0 after 15 mins
or so..  it looks like nothing happens in domU kernel.

"xm console" shows the bootup has stalled.. destroying the domU and
restarting it might show the hang in a bit different place 
(+- a couple of lines). It always hangs before actually executing initrd
image. 

Example console output:

checking if image is initramfs... it is
Freeing initrd memory: 12028k freed
Grant table initialized
NET: Registered protocol family 16
SMP alternatives: switching to SMP code
<hangs here, nothing happens anymore>

Like said, it could be a couple of lines later or earlier where it hangs.. 

I tried changing the domU to use just a single vcpu.. doesn't help. I tried
changing (lowering and increasing) the amount of memory.. didn't help
either.

Any ideas how to debug this? 

-- Pasi

Pasi K?rkk?inen wrote:
> Hello!
> 
> I've noticed this problem two times now.. last time I fixed it by
rebooting
> the (centos 5.1 x86 32b) xen host/dom0.
> 
> Symptoms:
> 
> - Already running domUs (debian 2.6.18-6-xen-686 32b PAE) continue running
>   and working OK 
> 
> - Cannot start new domUs (debian 2.6.18-6-xen-686).. kernel bootup just
> hangs before running initrd. Same domU with the exact same xen domU cfgfile
worked
> earlier. 
> 
> - This problem starts to happen after a "longer" uptime.. atm
uptime for the
> dom0-host is 174 days. Cannot say what the actual "limit" is..
last time I
> used the dom0 everything worked fine, now it doesn't.. 
> 
> 
> "xm list" shows the domain as "-b----" and the Time(s)
for the domain does
> not increase.. it stays at 1.9 or so. It might increase to 2.0 after 15
mins
> or so..  it looks like nothing happens in domU kernel.
> 
> "xm console" shows the bootup has stalled.. destroying the domU
and
> restarting it might show the hang in a bit different place 
> (+- a couple of lines). It always hangs before actually executing initrd
> image. 
> 
> Example console output:
> 
> checking if image is initramfs... it is
> Freeing initrd memory: 12028k freed
> Grant table initialized
> NET: Registered protocol family 16
> SMP alternatives: switching to SMP code
> <hangs here, nothing happens anymore>
> 
> Like said, it could be a couple of lines later or earlier where it hangs.. 
> 
> I tried changing the domU to use just a single vcpu.. doesn't help. I
tried
> changing (lowering and increasing) the amount of memory.. didn't help
either.
> 
> Any ideas how to debug this? 
> 
> -- Pasi
> _______________________________________________
> CentOS-virt mailing list
> CentOS-virt at centos.org
> http://lists.centos.org/mailman/listinfo/centos-virt
Sometimes xenconsoled dies after a while.  Is it running when you 
attempt this?

-- 
Christopher G. Stach II